Evaluation
Guidance for planning project evaluation
1 Introduction
All good projects include evaluation to ensure that value for money, and real outcomes, can be shown.
This guidance will help you consider why, when, and how you should evaluate, it includes some general advice on techniques, and links to some tools which can be used. Further examples of evaluation tools can be found associated with some case studies on the food vision website.
2.Background
2.1 What is evaluation?
Evaluation is about determining the effectiveness of a planned intervention in achieving its stated aims and objectives. It generally involves the collection of baseline data and comparing this with data collected after the intervention but evaluation should also be an integral part of a project rather than simply a task carried out at the end. This is to allow an ongoing awareness of success and to ensure objectives can be tailored as necessary.
2.2 Why is evaluation important?
Proper, thorough, evaluation of a project is very important because it determines which areas were successful or not, and why, and can also provide suggestions for alternative approaches. This is useful if you want to repeat, or continue, the project. Evaluation is also important if you are seeking funding to continue a project as you will need to demonstrate to funding bodies that the pilot/project was successful.
3. What are the differences between evaluation, monitoring and audit?
There are distinct differences between evaluation, monitoring, and audit although they all involve using some of the same tools e.g. questionnaires, data collection and interviews.
3.1 Evaluation
Evaluation is a systematic examination of a planned, ongoing, or completed project. It aims to answer specific questions and to judge the overall value of a project. It can also produce lessons learned to improve future actions, planning, and decision-making.
Evaluation aims to determine the efficiency, effectiveness, impact, sustainability and the relevance of the project. An evaluation should provide information that is credible and useful, offering concrete findings to help partners and funders make decisions.
3.2 Monitoring
Monitoring is the process of following up activities to ensure that they are progressing according to plan. It involves an ongoing assessment of the progress of a project in order to identify problems and to modify and improve the activities as appropriate.
The regular collection and analysis of information assists timely decision making, ensures accountability and provides the basis for evaluation and learning. It is a continuing process that uses methodical collection of data to provide early indications of progress and achievement of objectives.
3.3 Audit
An audit is the systematic, critical analysis of the quality of programmes and services. An audit determines whether, and to what extent, the activities and organisational procedures conform to the standards and criteria set out in advance. It is a management tool that can be used to measure performance and aims to identify if a project is failing to reach a certain standard.
An audit is a cycle which aims to provide continuous improvement. Audits can also involve the verification of the legality and regularity of the implementation of resources, carried out by independent auditors. An audit helps an organisation accomplish its objectives by bringing a systematic, disciplined approach to evaluate and improve the effectiveness of processes.
4. When should I evaluate?
It is important to consider when, as well as how, you will evaluate your project. It is important to evaluate through the projects timetable as well as at the end. Your project may have definite phases after which it would be sensible to evaluate.Evaluation may be driven by political or financial timetables as it may be necessary to show positive progress to secure further funding or support, or evaluation may only be possible at the end of the project. However it is important that when considering the method to be used, it is appropriate to the timing of the evaluation, certain methods may be more appropriate as the project progresses where as a different method could be used as the concluding evaluation.
5. Evaluation methods
Evaluation can be carried out at various different levels, from number collection to in-depth interviews and focus groups. It is a skilled process that needs to be integrated into project planning at the outset. When considering the type of evaluation it is important to ensure that the proportion of the budget spent on the method is proportionate to the total funding of the project.Each evaluation method has pros and cons and the choice will depend on what the data will be used for and who the project target group are. The amount of time, effort, and money that is put into carrying out the evaluation will also depend on how the results will be used.
Evaluation methods can be both quantitative and qualitative and may include:
• Questionnaires
• Surveys/interviews
• Data collection
• Physical measurements
• Journals/Participants’ observations / comments
• Rapid appraisal
• Tests / exams
It is very important that the evaluation methods to be used should be determined in the planning stages of your project, at the same time as your aims and objectives are set. It is important at this stage to consider what information you want to collect and how.
Examples of aims and objectives you may want to achieve are:
• Changes in behaviour, knowledge and skills
• Increased consumption of fruit and vegetables
• Increased availability and wider choice of healthy foods
Examples of the types of information you might need to gather are shown below:
• Number or participants in an event and satisfaction levels amongst participants
• Changes in consumption of healthy foods
• Knowledge increases before and after an intervention
• Number of outlets selling healthy foods
5.1 Using questionnaires
The main advantages of questionnaires are that they are an efficient use of time and their analysis is easy.
The main disadvantage with a qualitative evaluation, e.g. questionnaires, is that the data is often superficial and you can be uncertain about the honesty of responses. In most cases response rates to questionnaires are small compared to the total number of beneficiaries, which makes it difficult to assess whether these are representative of the target audience, or whether there is a particular reason why certain participants did not respond.
5.1.2 Designing questionnaires
It is important that your questionnaire is well designed to ensure that you gather the information you need and that the results are not ambiguous. For example, specific questions are better than general ones, e.g. rather than just asking people what fruit and veg they buy, show a list of fruit and veg and ask them to tick the items they purchased last week.Closed questions are usually better than open questions, i.e. rather than asking what factors affect people’s choice about whether to buy fruit and veg, give a list of options – e.g. price, knowing how to cook, being able to carry it home.
Providing lists of options that people tick rather than writing individual answers also makes the results easier to read and analyse. You should also offer a ‘no opinion’ or ‘other’ option so that participants always tick something rather than leaving the question blank.
5.1.3 Increasing response rates to questionnaires
Some factors which increase response rates are collecting the evaluation information at the time that a project is delivered (and making time available to do this) and having someone organise the administration and collection of the evaluation data.
If your initiative involves giving participants certificates of attendance or an award, you could make it a condition that the evaluation has to be completed before they receive the certificate/award.
Other tips to maximise the response rate for questionnaires are to:
• Make the form simple to understand and easy to complete
• Don’t make them too long – try to keep to one or two sides of A4
• Offer an incentive for completion – e.g. a money-off voucher
• Ensure responses are confidential / anonymous
• Work through questionnaires with respondants (although anonymity is lost using this method)
5.1.4 Example: FACET five a day questionnaire
The FACET questionnaire was developed for the evaluation of the pilot Five a Day initiatives. It was also used in a major national evaluation, commissioned by the Big Lottery Fund, of the 66 Big Lottery Fund-funded initiatives. It provides a simple, practical tool for assessing change in fruit and vegetables consumption in a population. This tool can also be used to evaluate the effectiveness of other 5 A DAY interventions. A guidance document has been prepared to assist in using the questionnaire.
5.2 Rapid appraisal
5.2.1 What is rapid appraisal and when should it be utilised?
Rapid appraisal, as the name suggests, is a much quicker way of gathering information and so is good for certain activities where people do not have time to complete a written questionnaire.
Rapid appraisal is a more interactive method of evaluation and can be very useful when working with certain groups who may find it difficult to fill in questionnaires e.g. people with literacy problems, children, and those who first language is not English.
5.2.2 What methods can be used for rapid appraisal?
Rapid appraisal can be carried out using a variety of different tools.
For example, if you want to survey people attending a farmers market or other event you could have large laminated sheets with questions on and then ask customers to stick dots on the appropriate answers. People are also often much happier answering questions this way rather than face-to-face, especially if they are quite personal e.g. relating to age, earnings, or average amount spent on produce.
When doing a rapid appraisal with children you could use even more interactive methods e.g. running to one side of the room or another to answer questions, or doing drawings to demonstrate what they’ve learned about healthy eating.
5.2.3 Example: Rapid appraisal tool for health impact assessment
Rapid appraisal tool developed by Institute of Health Sciences, Oxford
5.3 Using qualitative evaluation
5.3.1 What is qualitative evaluation and when should it be used?
Qualitative evaluation is used for collecting, analysing and interpreting data that can’t be summarised as numbers. It is not so important how many people benefited, but how much they learnt and what their experiences were.
5.3.2 What methods can be used for qualitative evaluation?
Qualitative evaluation often has smaller samples than quantitative methods but more in-depth analysis e.g. focus groups usually only involve 4-8 participants but may involve talking to them for 30 mins – 1 hour.
Qualitative methods include:
• Focus groups
• In-depth interviews
• Observation
• Case studies
• Diaries
Qualitative evaluation is often valuable when it is particularly important to understand what a participant feels about a particular topic/project or aspect of the project. However its key disadvantage is that data is subjective and results cannot be generalised.
5.3.3 Example: FSA NZ & Australia qualitative assessment of food labelling
Report and tools used in a qualitative study on consumer attitudes to food labelling in Australia and New Zealand
6. Analysis of results
After evaluation information has been collected, it needs to be analysed to compare results to aims and objectives. This is often very time consuming, especially if you are dealing with a large number of responses. Using IT tools can help to make it easier to analyse data e.g. by entering results into a spreadsheet or database than can be used to make calculations. However this is not feasible with qualitative data.
7. Longer term evaluation and planning
When carrying out your evaluation you should also include any additional outcomes or benefits that were not thought of in your original aims, which arose from delivery of the project. These may provide the impetus for new or expanded projects, which wider ranging objectives. If your project had long term aims, you may also wish to consider more evaluation after say 6 or 12 months to see if your aims are still being achieved – this might be particularly important if the initiative is designed to be self sustaining after you have withdrawn from it.
Guidance posted 16 August 2007
Other Resources: