Step 3

Decide the level of evaluation

Why is this step important?

  • Ensures M&E arrangements are relevant and proportionate to the scale and scope of your work
  • Aligns your M&E approach with the intervention's objectives
  • Ensures sufficient resource and budget arrangements are in place

What does this step involve?

  • Decide what level of impact measurement is required
  • Decide whether you need an independent evaluation supplier
  • Decide how the evaluation will be paid for

      Click to read more...

Tools and resources

outputs

  • Level of impact measurement

helpful resources

  • Level of Measurement Decision Tree
  • NESTA 'Standards of Evidence'

     Click to see all tools and resources...

Purpose

This step makes sure the measurement and evaluation (M&E) arrangements you choose are right for your funding stream or project. You don’t need to measure everything, only what is important to fulfil the objectives you’ve set out and what is feasible with the resources you have available.

Process

In Step 3 you need to decide how complex your impact measurement needs to be and whether you need to involve an independent evaluation supplier. You also need to work out whether you need a budget to pay for the evaluation and, if so, who will manage it.

1. decide the level of impact measurement

You need to decide how robust your evidence of impact needs to be. This will depend on:

  • whether you’re trying something new and untested
  • the size of the investment
  • the resources and skills available to collect and analyse data

It’s important to work out what level of measurement your M&E approach will deliver because:

  • Sport England (or other funders and stakeholders) may require you to provide a particular level of information to show what you have done and the impact it has had
  • the way you measure changes you are trying to bring about will determine how sure you can be that your project or funding stream has caused them

A higher level of impact measurement uses more rigorous methods, meaning you can be more confident in the findings - although it doesn’t guarantee the same approach would work in the same way somewhere else.

Less demanding methods can be cheaper and easier, but offer less certainty that you have brought about the changes that your M&E results identify.

Questions to ask yourself:

LEVEL OF MEASUREMENT DECISION TREE

We have developed a 'Level of Measurement Decision Tree' as a guide to help you work out the level of measurement that would be expected for Sport England funding streams or projects. Click on the image below to download the full size version.

     

TIP: You don’t have to limit your evaluation to the level that’s expected, but remember to consider the burden on the people who will be doing the activities, such as collecting and analysing data. Higher isn't always better - the majority of Sport England evaluation is at levels 1 and 2. It's better to aim for a level that you can realistically do thoroughly and to a high standard, than aim for a higher level and do it badly.

2. LEVELS OF MEASUREMENT

Sport England has developed five levels of impact measurement that are linked to the NESTA Standards of Evidence. Sport England will only require some projects to describe what they do and why it matters by collecting basic information (Level 1), while others will need to show they have caused changes by comparing the information they have collected with what has happened to a similar group or setting outside of the funding stream or project (Level 3). 

Here is the definition of each level:

LEVEL 1

You will need to describe what you do and why it matters, logically, convincingly and coherently.

LEVEL 2

You will need to collect data that shows positive change, but you don’t need to be able to confirm you caused this.

LEVEL 3

You will need to show you have caused changes by comparing the data you have collected with what has happened to a control group (a similar group or setting outside your funding stream or project).

LEVEL 4

You will need at least one other independent Level 3 evaluation of a funding stream or project that has used the exact same approach as yours and produced results that confirm your conclusions.

LEVEL 5 

You will need to have manuals, systems and procedures in place to ensure the approach is carried out in exactly the same way so that it results in the same positive impact.

You can view an example of a level 1, level 2 and level 3 measurement by downloading the 'Project Measurement Examples' document from the links on the right.

The table below shows what information you will typically need to collect for each level:

 TYPE OF DATA COLLECTION LEVEL 1  LEVEL 2  LEVEL 3-5 
Written progress reports from project/delivery teams  yes  yes  yes
Recording basic characteristics of projects and the people involved e.g. gender, age, disability  yes  yes  yes
Recording the number of participants and attendance figures at the start of the intervention ('baseline')  yes  yes  yes
Recording outcome measures at the start of an intervention ('baseline') e.g. activity level, subjective wellbeing   yes yes
Measuring basic outputs achieved e.g. participants, throughput (total attendances) yes yes yes
Measuring short-term outcomes at the end of the intervention e.g. activity level, subjective wellbeing   yes yes
Measuring medium and long-term outcomes after the intervention e.g. sustained activity level     yes
Tracking one or more control groups or settings (a similar group / setting outside your funding stream or project)     yes
Using an independent evaluation supplier    maybe yes

There can be good reasons to collect other types of information in addition to the minimum you need for your chosen level of impact measurement, for example:

  • to have evidence of impact to attract future funding
  • to help promote your activities
  • to develop and share learning

Recommended Action
Use this guidance and the ‘Level of Measurement Decision Tree’ to decide what level is right for your funding stream or project.

3. decide if you need an independent supplier

The higher the level of impact measurement, the more complex it is, which makes it more likely you will need to involve an independent person or team that specialises in evaluation (a ‘supplier’). Here is some guidance to help you decide if you will need one:

level 1

Sport England funding recipients should be able to collect and report the data they need for Level 1 impact measurement as part of their regular monitoring, without needing extra resource or an external evaluation supplier.

level 2

Whether an organisation will need support for Level 2 depends on their knowledge, capability, systems and capacity, as they will have to follow-up with participants and/or carry out surveys designed to measure and calculate impact across everyone who has taken part.

levels 3, 4 and 5

You will usually need a specialist evaluation supplier for these more complex studies. Their independence will help to make any claims that a funding stream or project has caused a particular outcome more credible.

Independent evaluation suppliers may be involved with Sport England funding streams and projects in a number of different ways:


finding an evaluation supplier

There are a wide range of agencies and organisations that specialise in evaluating the impact of investment programmes and projects. The supplier you choose will depend on the type of investment or project you are developing, the outcomes you are trying to achieve, the level of evidence you want to produce and practical considerations such as the budget you have at your disposal.

If you are unsure where to start your search for an independent evaluation supplier, the Market Research Society (MRS) have produced an online research buyers guide and search tool designed to help organisations find research and evaluation suppliers that meet their needs.


Recommended Action
1. Decide whether you will need to involve an independent evaluation supplier.
2. Find an appropriate evaluation supplier.

4. Paying for the evaluation

EVALUATION BUDGET AND PROCUREMENT

When a Sport England funding recipient collects and reports the data themselves, they won’t usually need a separate evaluation budget unless they lack the capability and/or capacity to carry out the evaluation. In that situation, Sport England may provide dedicated funding to help an organisation to develop their internal measurement and evaluation processes and a budget for measurement and evaluation can be included in the funding award.

If an independent evaluation supplier is required, there will need to be a dedicated budget to pay for the evaluation. The method of procurement and management of an independent evaluation will depend on the type of evaluation that is needed:

Single supplier for multiple Sport England funded projects: it usually makes sense for Sport England to procure, manage and pay for the work delivered by the evaluation supplier.

One or more suppliers evaluating a single project: it could make sense for either Sport England or a funding recipient to procure, manage and pay for the evaluation. If a funding recipient will manage the evaluation, Sport England can include a budget for the evaluation in the award, where appropriate. The best arrangement will depend on the specific circumstances. Managing the evaluation provides full ownership and oversight, but it can be resource intensive (in terms of time and expertise) to do effectively.

Recommended Action
Agree how the evaluation is going to be paid for and/or resourced.

Recommended Outputs

Output icon

Level of Impact Measurement

Before you move on you need to have decided on the level of impact measurement for your M&E approach. Ensure you are clear on what this means in terms of the data you'll need to collect before you select your data collection methods (Step 4).

Level of Measurement Decision TreeOpens in new window

Framework overview