Step 2 is about identifying the main priorities for your measurement and evaluation (M&E). This will enable you to focus your efforts on the things you will benefit from the most and ensure that your approach is appropriate for the scale and scope of your work.
Process
In this step you will need to talk to stakeholders to decide on the most important outcomes that will show the impact of your work. You need to think about who will use the findings and what type of information they will require. Follow the steps below to work your way through Step 2.
1. DECIDE ON YOUR M&E OBJECTIVES
Discuss and agree your M&E objectives with your key stakeholders. It’s useful to think about these objectives as questions that you want your M&E to help you answer. Once you know the key questions you want to answer, it is much easier to design an approach that will provide you with what you need.
You need to think about how measurement and evaluation can help you to:
show the impact of your work
learn and improve
Identifying the benefits that are most important to you will help you to define your M&E priorities. Here are some of the main benefits of conducting measurement and evaluation:
TIP: Work closely with your stakeholders on this and try to agree a few key priorities to focus on - don’t try to do too much.
Types of Evaluation
There are two broad types of evaluation: process evaluation and outcome/impact evaluation. Both of these are important in measuring and evaluating your project.
Process evaluation looks at how well the funding stream or project is being carried out and how it could be improved.
Outcome/impact evaluation looks at whether the project has achieved its objectives and produced the intended outcomes.
Process evaluation is carried out while the funding stream or project is still in progress, to understand the extent to which it is progressing as planned. It helps you to know how impact is being achieved and means that successful projects can be replicated. It uses information collected from project monitoring and stakeholders to measure and understand the activities that have been conducted, how many people have been turning up, whether they are from the target group etc. The data collected might be qualitative (descriptive), quantitative (numbers) or a mixture of both.
Every Sport England funding stream and project needs to have some basic process evaluation, to monitor whether delivery is working as expected.
When you want to learn from a new or untested approach – e.g. a pilot initiative – a thorough process evaluation is the best way to find out how well it has been carried out and what could have been improved. You can look at this information alongside the findings from impact evaluation, and make comparisons with other approaches to determine the best way to achieve the planned outcomes. This will help you decide on future investment and methods of delivery.
Outcome/impact evaluation is done while the funding stream or project is in progress and after it has come to an end. It looks at the short, medium and long-term effects and examines the extent to which the intended outcomes have been achieved. It nearly always involves quantitative methods of data collection (numbers), often combined with qualitative (descriptive) methods. Sport England often commissions an independent research organisation when we want to carry out a rigorous outcome/impact evaluation.
Rigorous impact evaluation uses a lot of resources, so for some funding streams and projects a basic level of impact measurement will be sufficient. The next section (Step 3) of this guide provides advice on how to decide what level of impact measurement you will require.
Economic evaluation is a form of outcome/impact evaluation that compares the cost of a funding stream or project with the value of the outcomes achieved (the ‘benefits’) by looking at its value for money (VFM). You can assess the benefits either in their natural units (‘cost per outcome’), or converted into monetary values (typically shown as a ratio of benefits to costs or a statement that “£1 of investment generates £x of value”). Cost Benefit Analysis (CBA), Return on Investment (RoI) and Social Return on Investment (SRoI) are all types of economic analysis.
For Sport England funding streams and projects you should think about including economic analysis if the evaluation objectives include, for example, demonstrating the economic value of an approach to a potential funder or commissioner.
Sport England has developed some standard tools to support economic analysis. You can use the MOVES Tool to work out the value of the health outcomes of a sport or physical activity intervention and the Local Economic Model to value the contribution of sport and physical activity to a local economy.
Recommended Action Write out your measurement and evaluation objectives in the ‘M&E Key Questions’ section of the M&E Priorities Form.
2. Define the M&E audience
Talk to project stakeholders about who the main users of any measurement and evaluation findings will be. You should think about:
what type of evidence they would want
what data or reports they would expect to receive
what they will do with the findings
Find out whether there are any specific things that you need to report on or provide to stakeholders. This might include:
progress reports
sharing data
updates at particular points in time (e.g. to inform important decisions)
TIP: Don’t forget to include your own team, organisation or colleagues when considering the measurement and evaluation audience for your project – a well conducted evaluation will help you improve what you do and help you to show the impact of your work to others.
Recommended Action 1. Write all of the potential audiences and users of your measurement and evaluation findings in the ‘M&E Users’ section of the M&E Priorities Form. 2. Write any particular requirements these users have specified in the ‘Reporting Requirements’ section of the M&E Priorities Form.
3. Identify impact measurement priorities and outcome indicators
Depending on how many outcomes you identified for your project or funding stream (see Step 1), you may not be able to measure them all. In this case think about:
which outcomes are most relevant and important for you to measure
which outcomes you can reasonably expect to influence the most
You need to define and agree a set of outcome indicators with stakeholders – these are things that you can measure over time that will show progress towards your intended outcomes and which you can use as evidence of impact. Outcome indicators should be:
directly related to the outcome you are trying to achieve
something that you can measure accurately using either qualitative or quantitative methods
something that will be useful for informing decision making
Here are some examples of outcome indicators:
OUTCOME
OUTCOME INDICATORS
Improve job prospects of out of work 16-24 year olds
- % in employment, education or training - # of interviews attended in last 4 weeks
Increase older female membership at local sports clubs
- # of new female club members each month - % of all club members who are 55+ females
Improve tailored local sports provision for BME males
- # of weekly sessions targeting BAME males - % BME males who rate local provision as 'good/excellent'
Setting out your planned outcomes in an ‘Outcomes Hierarchy’ (see below) can help you to identify the best indicators to use to measure the impact of your funding stream or project. This will help you to think about what realistic things you can measure in the short term that will provide evidence of progress towards your broader, long-term objectives or 'primary outcomes'.
You can download the 'Outcomes Hierarchy' template via the links on the right, or view a completed example by downloading one of our case studies.
You may decide or be required to have targets for one or more of your outcome indicators (sometimes called Key Performance Indicators – KPIs) that you aim to achieve within a set time.
TIP: A good target:
is realistic and achievable
considers past performance as well as future aspirations/forecasts
has a clear ‘line of sight’ to the outcomes and does not distract from them
is agreed with and owned by the people who will be responsible for its achievement
Recommended Action 1. Use the Outcomes Hierarchy template and the guidance in this section to create your own Outcomes Hierarchy 2. Write your chosen indicators in the ‘Outcome Indicators’ section of the M&E Priorities Form.
4. Identify learning priorities
Think about the most important things that you want to learn from the delivery of your project and how you might collect the data that will give you this information. This might include:
how your team / organisation could improve how you engage participants and deliver services
where problems have occurred – and how to avoid them in future
what worked well and what worked less well
if an untested approach has worked and how it could be improved in future
You should focus on filling gaps in what you already know about the approach being used, as that can help you to improve how similar funding streams and projects are developed and implemented in future.
Recommended Action Write the priorities you have identified in the ‘Learning Priorities’ section of the M&E Priorities Form.
TIP: Before you move on, refer back to the M&E objectives you set out at the start of this section. Check to make sure that your outcome indicators and the learning priorities you have identified will achieve your M&E objectives. If you’re happy that they do, you’re ready to move on to the next step.
Insight Team Support
The Sport England Insight Team is available to help you if you need extra help or advice.
Purpose
Process
