If you're new to evaluation, short on time, or just need a quick guide, then try our one-page Framework summary. It'll tell you the basics and help you get started on your evaluation journey.
By following the steps outlined in Sport England’s M&E framework, you will be able to measure and evaluate the success of your own project against the success criteria that you define in collaboration with the project’s stakeholders.
By setting out clear outcomes for the project and deciding on the best way to measure success against those outcomes at the outset of the project, you can collect the relevant data and information to enable a robust evaluation of the project’s success.
Measurement is the routine and systematic collection of data and information that enables us to determine attributes about things such as people or activities. Measurement helps us to answer questions such as ‘who?’, ‘how many?’, ‘how much?’ and ‘how long?’.
Evaluation uses the data and information provided through measurement, combined with other sources of information, to make judgements about the value or success of a project, activity, service or organisation.
A key aspect of evaluation is using information to learn and make improvements to what we do or how we do things.
Measurement & evaluation is the responsibility of the project or funding stream manager, the wider project team and its stakeholders.
Depending on the size and scale of the project, there may be dedicated resource for managing aspects of the measurement and evaluation process, such as data collection, analysis and reporting. In smaller-scale projects, measurement and evaluation tasks will need to be undertaken by members of the project team.
The key is to identify at the outset of the project what M&E tasks need to be done and assign clear roles and responsibilities – guidance for this is provided in Step 4 and 5 of the M&E framework.
A measurement and evaluation framework is a structured approach to measuring and evaluating the success of projects and investments. A framework provides guidance to individuals and organisations so they are able to effectively measure and evaluate their own projects to establish evidence of impact and outcomes; and learn from their activities to drive improvements in the design and delivery of future projects and investments.
A successful evaluation is one that effectively answers the M&E questions that were set at the beginning of the project (see Step 2 in the M&E framework).
There are no specific criteria that define a successful evaluation, as every project will have different requirements and resources – so the key is to ensure that the M&E objectives and priorities are clearly defined up-front and are used to guide all aspects of M&E for the project.
In the broadest terms, good evaluations should be both accurate in what they measure and useful in that the information they generate informs decision-making and guides future projects.
A Logic Model is a tool that can be used to help conceptualize what a project is trying to achieve – and how it is trying to achieve it. To create a logic model you will need to articulate your understanding of the current situation, the changes you hope to bring about as a result of the project, the activities that are planned to contribute towards these changes, and the resources that are needed to successfully implement the planned activities.
The key benefit of using a logic model is that by articulating all of the inputs, activities, outputs and outcomes of the project, you can test the assumptions you are making within the planning of the project. For example, it will help you to answer questions such as:
Will we be able to deliver the planned activities with our available resources?
Will the planned activities really deliver our intended outputs (e.g. participants, throughput)?
Will the outputs we deliver (e.g. participants, throughput) realistically lead to the outcomes we are aiming to achieve?
It can sometimes be difficult to distinguish between project outputs and project outcomes.
Outputs describe the direct, immediate results of a project’s activities – often related to participants, throughput (visits), services and facilities. For example:
192 unique participants attended ParkRun sessions during the summer programme
6 training workshops were conducted, with a total of 27 co-ordinators receiving training
The new multi-games centre had an average of 3,921 visits per month in its first 12 months
Outcomes describe the actual impact or change that has come about as a result of the project’s activities – they answer the question: what difference has it made? Outcomes relate back to the original aims and objectives of the project or investment.
Outcomes are usually expressed in terms of an increase or improvement in desired behaviours, attitudes or perceptions. For example:
The project drove a 47% increase in once a week sport participation amongst project participants
Improvements in psychological well-being were reported by 29 of the 40 participants
Incidents of crime and anti-social behaviour dropped by 31% in the project intervention area
Identifying the outcome(s) of your project or investment is essentially just expressing what you want to achieve, in a way that can be evidenced through some form of direct or indirect measurement.
The key is to start with your over-arching objective, then attempt to express it in the most focused and specific way possible. For example, the project’s over-arching aim might be ‘to get more people in my local community more active’. To define this as an outcome, you need to ask questions such as:
Which people are you trying to impact – anybody, or a specific group?
Are you trying to get active people to participate more regularly; or are you trying to get inactive people to take up activity?
Are you focusing on a particular sport or type of physical activity?
In the example above, the resulting outcome for the project might be: “To increase the number of young women aged 16-25 who participate in organised club sports at least once a week”.
A number of factors will influence the data that you need to collect to evaluate your project. The key is to start by defining the rationale of the project, identifying clear outcomes and agreeing the indicators that will be used to measure the project’s success, before thinking about the data you will need.
Steps 1-4 in the M&E framework are designed to help you define exactly what data is appropriate and relevant to collect based on the aims and scope of your project.
It is good practice to collect different types of data from a range of sources and compare the findings in order to generate the best understanding of impact, however, the primary purpose of data collection should always be to collect the information that best enables you to test if the project is working as expected and is achieving the planned outcomes.
An outcomes hierarchy is a visual model that helps to identify the indicators that will provide the most accurate and relevant measure of the impact of your project.
By breaking down a projects intended outcomes into component parts, the changes that need to be brought about to deliver the project’s outcomes can be identified; and a way of measuring progress against each of these can be developed. These measures are referred to as ‘indicators’ and are usually the primary measures that will be utilised as part of the project evaluation to assess the success of the project.
Indicators are variables that are used to measure progress towards outcomes. Outcome indicators are important in the evaluation of projects as they enable the quantifiable measurement of changes or impact over time resulting from a project’s activities.
The reason most projects need to define indicators (often referred to as 'Key Performance Indicators' or 'KPIs') is because the intended outcomes of many projects or interventions are quite abstract and broad e.g. 'improving perceptions of physical activity amongst teenage girls'. This can make it unclear exactly how success can be measured and evaluated.
The key to defining an outcome indicator is to find things that are specific and measurable, which will also provide evidence of whether the outcomes of the project have been achieved.
For example, using surveys to measure the % of girls aged 11-16 who agree with the statement “physical activity can be an enriching experience" would provide a good indication of the perception of physical activity amongst teenage girls - and could be used to track these perceptions over time.
Outcomes cannot always be measured directly. However, if your outcomes have been clearly defined, you should be able to develop a way to measure the success of your project.
There are a variety of limitations to measuring outcomes such as cost, resources, time and access that can make it difficult to measure all of the changes you are attempting to bring about.
You need to consider what you can realistically measure that will provide a good indication of progress towards the project’s intended outcomes. These measures are called ‘indicators’ and they are used on most projects as a way of measuring the success of activities and interventions in achieving their aims and objectives.
The data collection methods you need to use will be dictated by the type and amount of data you need to collect. Guidance on choosing the most appropriate data collection methods based on the requirements and objectives of your project is provided in Step 4 of the M&E framework.
Process evaluation focuses on the implementation of a project or investment to establish whether it is progressing as planned and assess the effectiveness of its design and delivery. Process evaluation thus enables issues or problems to be identified while a project is in progress, allowing improvements to its design and/or delivery to be made. It also provides invaluable learning about the aspects of a project or investment that have gone well, the things that could be improved and how improvements might be made in future initiatives which have similar objectives.
Outcome (or impact) evaluation focuses on the impact a project or investment has made. Intended outcomes should always be clearly defined at the outset of a project, so that appropriate data can be collected enabling the project’s impact to be assessed.
Outcome evaluation usually concludes at the end of a project or investment and can use a variety of methods – participant data, surveys, interviews - to evaluate whether the outcomes have been achieved.
Quantitative data is numerical, usually expressed in numbers and percentages. Quantitative data is useful because it enables the measurement of specific variables, such as how many people are taking part or what percentage of participants are female. By collecting this type of data at different points in time (e.g. at the start and end of a project) we can use the resulting numerical data to identify changes in behaviour or attitudes of particular populations.
Qualitative data uses words, descriptions, images and quotes. The information contained within qualitative data is more detailed and in-depth than quantitative data, however, it is not suitable for measuring trends or patterns of large groups. Qualitative data does, however, help to understand people’s feelings, experiences and motivations in detail, which can help to answer questions about why people act or think in certain ways and how things could be changed or improved.
Frequently Asked Questions
