Decide the level of evaluation

Purpose

This step makes sure the measurement and evaluation (M&E) arrangements you choose are right for your funding stream or project. You don’t need to measure everything, only what is important to fulfil the objectives you’ve set out and what is feasible with the resources you have available.

Process

In Step 3 you need to decide how complex your impact measurement needs to be and whether you need to involve an independent evaluation supplier. You also need to work out whether you need a budget to pay for the evaluation and, if so, who will manage it.

1. decide the level of impact measurement

You need to decide how robust your evidence of impact needs to be. This will depend on:

whether you’re trying something new and untested
the size of the investment
the resources and skills available to collect and analyse data
the requirements of your funders and stakeholders
the potential value of the evidence to your organisation

Working out your level of impact measurement is important because the way you measure your impact will determine how sure you can be in the findings, and in whether your project or funding stream is what caused them.

A higher level of impact measurement uses more rigorous methods, meaning you can be more confident in the findings - although it doesn’t guarantee the same approach would work in the same way somewhere else.

Less demanding methods can be cheaper and easier, but offer less certainty that you have brought about the changes that your M&E results identify.

Questions to ask yourself:

LEVEL OF MEASUREMENT DECISION TREE

We have developed a 'Level of Measurement Decision Tree' as a guide to help you work out the level of measurement that would be expected for Sport England funding streams or projects. Click on the image below to download the full size version.

TIP: You don’t have to limit your evaluation to the level that’s expected, but remember to consider the burden on the people who will be doing the activities, such as collecting and analysing data. Higher isn't always better - the majority of Sport England evaluation is at levels 1 and 2. It's better to aim for a level that you can realistically do thoroughly and to a high standard, than aim for a higher level and do it badly.

2. LEVELS OF MEASUREMENT

Sport England has developed five levels of impact measurement that are linked to the NESTA Standards of Evidence. Sport England will only require some projects to describe what they do and why it matters by collecting basic information (Level 1), while others will need to show they have caused changes by comparing the information they have collected with what has happened to a similar group or setting outside of the funding stream or project (Level 3).

Here is the definition of each level:

LEVEL 1

You will need to describe what you do and why it matters, logically, convincingly and coherently.

LEVEL 2

You will need to collect data that shows positive change, but you don’t need to be able to confirm you caused this.

LEVEL 3

You will need to show you have caused changes by comparing the data you have collected with what has happened to a control group (a similar group or setting outside your funding stream or project).

LEVEL 4

You will need at least one other independent Level 3 evaluation of a funding stream or project that has used the exact same approach as yours and produced results that confirm your conclusions.

LEVEL 5

You will need to have manuals, systems and procedures in place to ensure the approach is carried out in exactly the same way so that it results in the same positive impact.

You can view an example of a level 1, level 2 and level 3 measurement by downloading the 'Project Measurement Examples' document from the links on the right.

The table below shows what information you will typically need to collect for each level:

TYPE OF DATA COLLECTION	LEVEL 1	LEVEL 2	LEVEL 3-5
Written progress reports from project/delivery teams	yes	yes	yes
Recording basic characteristics of projects and the people involved e.g. gender, age, disability	yes	yes	yes
Recording the number of participants and attendance figures at the start of the intervention ('baseline')	yes	yes	yes
Recording outcome measures at the start of an intervention ('baseline') e.g. activity level, subjective wellbeing		yes	yes
Measuring basic outputs achieved e.g. participants, throughput (total attendances)	yes	yes	yes
Measuring short-term outcomes at the end of the intervention e.g. activity level, subjective wellbeing		yes	yes
Measuring medium and long-term outcomes after the intervention e.g. sustained activity level			yes
Tracking one or more control groups or settings (a similar group / setting outside your funding stream or project)			yes
Using an independent evaluation supplier		maybe	yes

There can be good reasons to collect other types of information in addition to the minimum you need for your chosen level of impact measurement, for example:

to have evidence of impact to attract future funding
to help promote your activities
to develop and share learning

Recommended Action
Use this guidance and the ‘Level of Measurement Decision Tree’ to decide what level is right for your funding stream or project.

Insight Team Support

The Sport England Insight Team can help colleagues work through these considerations to decide what level of impact measurement is appropriate for different funding streams and projects.

Not Sport England staff?

3. decide if you need an independent supplier

The higher the level of impact measurement, the more complex it is, which makes it more likely you will need to involve an independent person or team that specialises in evaluation (a ‘supplier’). Here is some guidance to help you decide if you will need one:

level 1

Sport England funding recipients should be able to collect and report the data they need for Level 1 impact measurement as part of their regular monitoring, without needing extra resource or an external evaluation supplier.

level 2

Whether an organisation will need support for Level 2 depends on their knowledge, capability, systems and capacity, as they will have to follow-up with participants and/or carry out surveys designed to measure and calculate impact across everyone who has taken part.

levels 3, 4 and 5

You will usually need a specialist evaluation supplier for these more complex studies. Their independence will help to make any claims that a funding stream or project has caused a particular outcome more credible.

If you think you need a supplier, consider what you want them to do, and how many you might need. In most cases one is enough, but if your project is very complex you might want to appoint multiple suppliers to specialise on different elements. Similarly, if you have several projects going on at once, consider whether you want a single supplier across them all, or one for each project.

If you decide to commission an independent supplier, you will need to prepare an evaluation brief. This is a short document giving information about your intervention, your evaluation, how you want a supplier to help, and how they can bid for the work. A good brief is one that enables suppliers to propose an approach that matches your requirements, making it easier for you to choose which one to work with. Click here for our guidance on what to include in an evaluation brief.

Once you have your brief, you can move on to finding, commissioning, and ultimately working with a supplier.

Finding, Commissioning and Working with Evaluation Suppliers

There are a wide range of agencies and organisations that specialise in evaluating the impact of investment programmes and projects. The supplier you choose will depend on the type of investment or project you are developing, the outcomes you are trying to achieve, the level of evidence you want to produce and practical considerations such as the budget you have at your disposal.

If you are unsure where to start your search for an independent evaluation supplier, the Market Research Society (MRS) have produced an online research buyers guide and search tool designed to help organisations find research and evaluation suppliers that meet their needs.

How you commission your evaluation supplier will depend on your internal systems, as well as any project or funder requirements, so check these before you start. However, the usual way is through a competitive tender.

This involves inviting several suppliers to tender (bid) for your evaluation, then assessing each application and appointing one supplier to carry out the work. This allows you to compare different approaches and costs to get optimal value for money – particularly important for large evaluations.

Some companies use a formal framework to advertise their brief to suitable suppliers and manage the procurement process. Otherwise, you can follow the steps below:

Identify suitable suppliers, e.g. via recommendations or word of mouth, or through researching online. Aim to make the pool large enough to be competitive (at least three), but not so big as to be unmanageable.
Invite the suppliers to bid for the work by sending them your evaluation brief. A simple way to do this is to send an email Bcc’ing each supplier, with the brief attached.
Assess the applications against the criteria and weightings outlined in your brief. Ideally, at least two people should independently assess each bid, then moderate and compare results, and agree on a preferred supplier. You may sometimes need a second round of assessment – for example asking a shortlist of suppliers to attend an interview or give a presentation.
Appoint your preferred supplier and finalise terms. You may also wish to offer feedback to unsuccessful bidders; they will appreciate it.

You will need to ensure this process is fair and transparent, so follow your internal systems if you have them, and be sure to formally document the process and your decisions.

An alternative way to commission is to approach a single supplier to carry out your evaluation, and agree terms with them directly. This is usually quicker and more informal than a competitive tender, and works best for small evaluations when you have a reliable supplier in mind who you know will do a good job. However, it isn’t always possible (or desirable) to take this route. For example, some funders may require you to follow a competitive tender process; while for larger evaluations it is usually wiser to have a competitive tender so you can compare different approaches and secure good value for money.

Once you’ve appointed a supplier, arrange an inception meeting with them to discuss the practicalities and agree next steps. At this stage you should also share any other factors that might affect the evaluation, e.g. around staffing, funding, project delivery, IT systems, or stakeholder relationships.

Establishing a good rapport with the supplier will help you enormously, so spend time developing a relationship. Set up clear and practical communication channels, and agree regular progress checks such as weekly email updates or phone calls. It’s best to have a single point of contact to liaise with them, but consider who else they might need to talk to and how – e.g. a steering group or wider stakeholders.

If it’s a long or multi-stage evaluation, hold occasional review meetings (internally, and with the supplier) to discuss how things are going. Don’t be afraid to change things that aren’t working, and let the evaluation evolve if it needs to, provided it will still achieve the objectives.

At the end of the contracted work, evaluate the evaluation! Reflect on, log and share what worked well, and what you might do differently next time.

Recommended Action
1. Decide whether you will need to involve an independent evaluation supplier.
2. Find an appropriate evaluation supplier.

4. Paying for the evaluation

EVALUATION BUDGET AND PROCUREMENT

When a Sport England funding recipient collects and reports the data themselves, they won’t usually need a separate evaluation budget unless they lack the capability and/or capacity to carry out the evaluation. In that situation, Sport England may provide dedicated funding to help an organisation to develop their internal measurement and evaluation processes and a budget for measurement and evaluation can be included in the funding award.

If an independent evaluation supplier is required, there will need to be a dedicated budget to pay for the evaluation. The method of procurement and management of an independent evaluation will depend on the type of evaluation that is needed:

Single supplier for multiple Sport England funded projects: it usually makes sense for Sport England to procure, manage and pay for the work delivered by the evaluation supplier.

One or more suppliers evaluating a single project: it could make sense for either Sport England or a funding recipient to procure, manage and pay for the evaluation. If a funding recipient will manage the evaluation, Sport England can include a budget for the evaluation in the award, where appropriate. The best arrangement will depend on the specific circumstances. Managing the evaluation provides full ownership and oversight, but it can be resource intensive (in terms of time and expertise) to do effectively.

Recommended Action
Agree how the evaluation is going to be paid for and/or resourced.

Required Action

Within Sport England, the team responsible for the funding stream or project typically holds the evaluation budget. Any Sport England colleagues who manage an evaluation contract should follow the standard procurement procedures. This includes obtaining authorisation from the finance and procurement teams and the relevant budget holder.

Not Sport England staff?

Recommended Outputs

Level of Impact Measurement

Before you move on you need to have decided on the level of impact measurement for your M&E approach. Ensure you are clear on what this means in terms of the data you'll need to collect before you select your data collection methods (Step 4).

Level of Measurement Decision Tree

Why is this step important?

What does this step involve?

Tools and resources

Outputs

Helpful resources