step 4

select data collection methods and tools

why is this step important?

Determines the data you need to collect and how to collect it
Helps you select the most appropriate methods and tools to measure your outcomes
Ensures consistent data collection supporting accurate impact measurement, benchmarking and comparisons

what does this step involve?

Decide what data you're going to collect
Choose the methods and tools you'll use to collect your data
Work out how you're going to collect the data

Click to read more...

Tools and resources

Outputs

Measurement & Evaluation Plan

helpful resources

Data Collection Guide
'Accessible Evaluation' Guidance and Case Studies
'Standard Questions' Banks for Adults and Children

Click to see all tools and resources...

purpose

Clearly defining the data that you need to collect will enable you to effectively measure the impact of your funding stream or project and ensure that the same things are always measured in the same way. This way, you can have clear and consistent benchmarks (points of reference) and make fair comparisons.

Process

In Step 4 you will decide what data you need to collect and select the best methods and tools to use to collect it.

1. decide what you are going to collect

You need to collect data that will tell you if your funding stream or project is working as expected. To do this you will need to think about what information will be most effective to:

inform how things are going and what could be improved (process evaluation)
show whether the intended outcomes are being or have been achieved (impact evaluation)

The data you collect for impact evaluation should be directly linked to the ‘outcome indicators’ that you defined and agreed in Step 2.

You will get the best understanding of impact if you:

look at and compare different kinds of information from a range of sources
collect a mixture of quantitative (numbers) and qualitative (descriptive) information

TIP: Qualitative data can include images, videos and quotes as well as written accounts.

Think about whether there is any existing data that you could use to support your process evaluation and impact measurement. This might give you some of the information you need and save you having to collect it yourself. One example is Sport England's Active Lives Survey, which collects and publishes data on sport and physical activity participation in England.

For Sport England investment, funding recipients will need to collect some standard data and report it back to Sport England. What they need to report will depend on the level of evaluation decided for that funding stream or project (see Step 3). Sport England will produce investment guides setting out measurement and evaluation requirements for specific funding streams.

Recommended Actions
1. Decide what data you need to collect for your process evaluation.
2. Decide what data you need to measure the impact of your funding stream or project.
3. Identify any existing data that you plan to use as part of your measurement and evaluation (M&E) approach.

2. choose your data collection methods and tools

There are lots of different ways you can collect data. You’ll need to choose the data collection methods and tools that are most suited to your funding stream or project based on the requirements you have set out in Steps 1, 2 and 3. If possible, speak to your funders, stakeholders, and your target audience and involve them in choosing suitable methods. This is particularly important if your audience has communication or accessibility needs. Examples of data collection methods include surveys, interviews, focus groups and participant observation.

We have developed an information pack which gives an introduction to the most commonly used data collection methods and tools, such as 'Surveys and Questionnaires':

This information pack will help you to decide on the best way to get the data that you need to collect - to download it click on the image above or use the links on the right.

We also provide guidance on designing evaluation that is accessible for audiences with communication or accessibility needs.

Sport England’s Question Banks

Sport England wants to work with other organisations to standardise the questions that are used to obtain some commonly collected data, such as the characteristics of project participants and how physically active people are. We have developed an adult question bank that sets out the standard questions that you should use when collecting this data from adults and another question bank to be used to collect data from children.

Sport England expects these standard questions to be used when this type of information needs to be collected, in all projects that involve our investment.

The questions are suitable for most adults and children, but you may need to adapt them for particular groups such as adults or children who might have difficulty understanding a question (e.g. where there are language barriers or people with impairments such as learning difficulties). See our Accessible Evaluation resources for more guidance on this.

Funding recipients will also need to think about the tools and methods to use to collect any additional data required to measure progress of outcome indicators, obtain information for learning, or to report to other funders, partners and stakeholders.

Recommended Actions
1. Choose the questions that are relevant for your M&E approach from the Sport England Question Banks.
2. Decide and document the methods and tools you will use to collect the data you need in the 'M&E Plan' template.

3. decide how to collect the data

The final thing you need to decide on are the processes and systems you will use to collect and store the data. There are some bespoke systems that are designed to support the collection of M&E data, however, simple tools (such as Excel) are often all you need.

Just make sure you factor in time and resource for any manual data entry required - transcribing interviews or entering survey responses from printed forms into a spreadsheet can be a time consuming task.

TIP: It's easy to let a system that’s available dictate what you collect and end up without the information you’re looking for. Be sure to find the best system for collecting the data you need.

The key is to think about how much and what type of data you need to collect and what you will need to do with it. This will help you decide which data collection processes and systems will best suit the needs of your funding stream or project.

If you plan to use existing data sources, such as the Active Lives Survey, spend time working out exactly how you plan to use this data and what you hope it will tell you. Most national surveys are designed to track very large populations and therefore may not be suitable for measuring the impact of a funding stream or project on particular groups or local areas. In addition, most official statistics are only released once a year, so they won’t give a short-term view of impact or change.

Sport England funding recipients currently have to report measurement data to us in a standard format, usually via our online portal. Sometimes they also have to use other systems and templates, which means manually keying the information in as the data doesn’t transfer automatically into Sport England’s records. We’re seeking to develop systems that will allow this to be automated in the future.

Recommended Action
Use the 'M&E Plan' template to document how you plan to collect the data that will enable you to measure all of your outcome indicators.

4. think about ethics and data protection

While you are deciding what tools and methods to use for data collection you will also need to think about ethics and data protection. Start by identifying any potential ethical considerations or challenges, and how you might overcome them. For example:

Obtaining informed consent from research participants. If your research is with children, young people or vulnerable adults then informed consent must come from a parent or guardian
Telling research participants how their data will be used
Protecting research participants’ anonymity and confidentiality
Ensuring that research subjects can exercise their rights in relation to their personal data (to access, correct, withdraw or delete the data)
Minimising any risk of harm to the research participants and researchers
Avoiding deceptive practices by providing honest and complete information to participants about what to expect during the research.

The General Data Protection Regulation (GDPR) came into effect on 25^th May 2018. Everyone is responsible for using personal data carefully and needs to follow strict rules or ‘Data Protection Principles’. The requirements of the GDPR are encompassed in six Data Protection Principles stating that personal data shall be:

processed fairly, lawfully and in a transparent manner
collected for specified, explicit and legitimate purposes
used in a way that is adequate, relevant and limited to what is necessary
accurate and where necessary up to date
kept in a form that allows identification of individuals for no longer than necessary
processed in a manner that ensures the appropriate security of the personal data, including against unauthorised or unlawful processing.

Guidance on Personal Data

Personal data is any information that could lead to the identification of an individual, and includes Direct and Indirect identifiers.

Direct identifiers are data that will directly identify the participant and includes their name, postcode, on-line identifiers and contact details.

Indirect identifiers are data which when linked with other information could identify someone e.g. gender, age, ethnicity and disability.

All personal data, Direct and Indirect, is subject to the provisions of GDPR.

Only collect the personal data that you need for the research, and make sure you limit access to direct identifiers to just those people working on the project.

It is good practice to anonymise data by either removing or generalising the data.

Example 1: Separate direct identifiers like name and address from the rest of the dataset. If you need to link the data in the dataset with the direct identifiers then you can use ID numbers.

Example 2: If your dataset includes participant postcodes, you could transfer this data into the corresponding IMD decile and then remove (or separate) the postcode data from the main dataset. This ensures participants cannot be identified.

You are responsible for all personal data being securely kept. For example, if you have paper surveys with personal data you will need to keep these locked in a secure cabinet. For electronic files, store them in a secure location and provide access to the project team only.

You should also take precautions when transporting personal data, for example when taking it on a train or posting it.

You must not keep personal data for longer than you need it. You will need to think about, and justify, how long you keep it . This may be until the end of your project, or for a set period of time after the end of the project.

External Resources:

Recommended Action
Identify any ethical or data protection considerations relating to your evaluation, and set out the necessary steps for dealing with them.

Insight Team Support

The Sport England Insight Team is available to help colleagues decide what data to collect and what tools and systems to use to collect that data. If you're going to appoint an external evaluation supplier you should also involve the supplier in making those decisions.

Not Sport England staff?

recommended outputs

M&E plan

Before you move on you should be clear about what data you need to collect and the methods and tools you will use. Creating an 'M&E Plan' will help with your planning to ensure you are collecting the data you need to measure your indicators.

M&E Plan Template