Step 3

Decide the level of evaluation

Why is this step important?

  • Ensures M&E arrangements are relevant and proportionate to the scale and scope of your work
  • Aligns your M&E approach with the intervention's objectives
  • Ensures sufficient resource and budget arrangements are in place

What does this step involve?

  • Decide what level of impact measurement is required
  • Decide if you need an independent evaluation supplier
  • Decide how the evaluation will be paid for

      Click to read more...

Tools and resources

Outputs

  • Level of impact measurement

Helpful resources

  • Level of Measurement Decision Tree
  • NESTA 'Standards of Evidence'

     Click to see all tools and resources...

Purpose

This step makes sure the measurement and evaluation (M&E) arrangements you choose are right for your funding stream or project. You don’t need to measure everything, only what is important to fulfil the objectives you’ve set out and what is feasible with the resources you have available.

Process

In Step 3 you need to decide how complex your impact measurement needs to be and whether you need to involve an independent evaluation supplier. You also need to work out whether you need a budget to pay for the evaluation and, if so, who will manage it.

1. decide the level of impact measurement

You need to decide how robust your evidence of impact needs to be. This will depend on:

  • whether you’re trying something new and untested
  • the size of the investment
  • the resources and skills available to collect and analyse data
  • the requirements of your funders and stakeholders
  • the potential value of the evidence to your organisation

Working out your level of impact measurement is important because the way you measure your impact will determine how sure you can be in the findings, and in whether your project or funding stream is what caused them.

A higher level of impact measurement uses more rigorous methods, meaning you can be more confident in the findings - although it doesn’t guarantee the same approach would work in the same way somewhere else.

Less demanding methods can be cheaper and easier, but offer less certainty that you have brought about the changes that your M&E results identify.

Questions to ask yourself:

LEVEL OF MEASUREMENT DECISION TREE

We have developed a 'Level of Measurement Decision Tree' as a guide to help you work out the level of measurement that would be expected for Sport England funding streams or projects. Click on the image below to download the full size version.

     

TIP: You don’t have to limit your evaluation to the level that’s expected, but remember to consider the burden on the people who will be doing the activities, such as collecting and analysing data. Higher isn't always better - the majority of Sport England evaluation is at levels 1 and 2. It's better to aim for a level that you can realistically do thoroughly and to a high standard, than aim for a higher level and do it badly.

2. LEVELS OF MEASUREMENT

Sport England has developed five levels of impact measurement that are linked to the NESTA Standards of Evidence. Sport England will only require some projects to describe what they do and why it matters by collecting basic information (Level 1), while others will need to show they have caused changes by comparing the information they have collected with what has happened to a similar group or setting outside of the funding stream or project (Level 3). 

Here is the definition of each level:

LEVEL 1

You will need to describe what you do and why it matters, logically, convincingly and coherently.

LEVEL 2

You will need to collect data that shows positive change, but you don’t need to be able to confirm you caused this.

LEVEL 3

You will need to show you have caused changes by comparing the data you have collected with what has happened to a control group (a similar group or setting outside your funding stream or project).

LEVEL 4

You will need at least one other independent Level 3 evaluation of a funding stream or project that has used the exact same approach as yours and produced results that confirm your conclusions.

LEVEL 5 

You will need to have manuals, systems and procedures in place to ensure the approach is carried out in exactly the same way so that it results in the same positive impact.

You can view an example of a level 1, level 2 and level 3 measurement by downloading the 'Project Measurement Examples' document from the links on the right.

The table below shows what information you will typically need to collect for each level:

 TYPE OF DATA COLLECTION LEVEL 1  LEVEL 2  LEVEL 3-5 
Written progress reports from project/delivery teams  yes  yes  yes
Recording basic characteristics of projects and the people involved e.g. gender, age, disability  yes  yes  yes
Recording the number of participants and attendance figures at the start of the intervention ('baseline')  yes  yes  yes
Recording outcome measures at the start of an intervention ('baseline') e.g. activity level, subjective wellbeing   yes yes
Measuring basic outputs achieved e.g. participants, throughput (total attendances) yes yes yes
Measuring short-term outcomes at the end of the intervention e.g. activity level, subjective wellbeing   yes yes
Measuring medium and long-term outcomes after the intervention e.g. sustained activity level     yes
Tracking one or more control groups or settings (a similar group / setting outside your funding stream or project)     yes
Using an independent evaluation supplier    maybe yes

There can be good reasons to collect other types of information in addition to the minimum you need for your chosen level of impact measurement, for example:

  • to have evidence of impact to attract future funding
  • to help promote your activities
  • to develop and share learning

Recommended Action
Use this guidance and the ‘Level of Measurement Decision Tree’ to decide what level is right for your funding stream or project.

3. decide if you need an independent supplier

The higher the level of impact measurement, the more complex it is, which makes it more likely you will need to involve an independent person or team that specialises in evaluation (a ‘supplier’). Here is some guidance to help you decide if you will need one:

level 1

Sport England funding recipients should be able to collect and report the data they need for Level 1 impact measurement as part of their regular monitoring, without needing extra resource or an external evaluation supplier.

level 2

Whether an organisation will need support for Level 2 depends on their knowledge, capability, systems and capacity, as they will have to follow-up with participants and/or carry out surveys designed to measure and calculate impact across everyone who has taken part.

levels 3, 4 and 5

You will usually need a specialist evaluation supplier for these more complex studies. Their independence will help to make any claims that a funding stream or project has caused a particular outcome more credible.

If you think you need a supplier, consider what you want them to do, and how many you might need. In most cases one is enough, but if your project is very complex you might want to appoint multiple suppliers to specialise on different elements. Similarly, if you have several projects going on at once, consider whether you want a single supplier across them all, or one for each project.

If you decide to commission an independent supplier, you will need to prepare an evaluation brief. This is a short document giving information about your intervention, your evaluation, how you want a supplier to help, and how they can bid for the work. A good brief is one that enables suppliers to propose an approach that matches your requirements, making it easier for you to choose which one to work with. Click here for our guidance on what to include in an evaluation brief.

Once you have your brief, you can move on to finding, commissioning, and ultimately working with a supplier.

Finding, Commissioning and Working with Evaluation Suppliers

 

Recommended Action
1. Decide whether you will need to involve an independent evaluation supplier.

2. Find an appropriate evaluation supplier.

4. Paying for the evaluation

EVALUATION BUDGET AND PROCUREMENT

When a Sport England funding recipient collects and reports the data themselves, they won’t usually need a separate evaluation budget unless they lack the capability and/or capacity to carry out the evaluation. In that situation, Sport England may provide dedicated funding to help an organisation to develop their internal measurement and evaluation processes and a budget for measurement and evaluation can be included in the funding award.

If an independent evaluation supplier is required, there will need to be a dedicated budget to pay for the evaluation. The method of procurement and management of an independent evaluation will depend on the type of evaluation that is needed:

Single supplier for multiple Sport England funded projects: it usually makes sense for Sport England to procure, manage and pay for the work delivered by the evaluation supplier.

One or more suppliers evaluating a single project: it could make sense for either Sport England or a funding recipient to procure, manage and pay for the evaluation. If a funding recipient will manage the evaluation, Sport England can include a budget for the evaluation in the award, where appropriate. The best arrangement will depend on the specific circumstances. Managing the evaluation provides full ownership and oversight, but it can be resource intensive (in terms of time and expertise) to do effectively.

Recommended Action
Agree how the evaluation is going to be paid for and/or resourced.

Recommended Outputs

Output icon

Level of Impact Measurement

Before you move on you need to have decided on the level of impact measurement for your M&E approach. Ensure you are clear on what this means in terms of the data you'll need to collect before you select your data collection methods (Step 4).

Level of Measurement Decision TreeOpens in new window

Framework overview