Gathering and using evidence
Gather different types of evidence, combine your findings and use a theory of change to guide decisions.
Use evidence to guide decision-making in every phase of the test and learn process. It can:
- reduce bias
- show where your assumptions are right or wrong
- reduce risk by drawing on what’s already known
-
highlight uncertainty by revealing what’s not known
- show what does and does not work
- strengthen your rationale for what to do next
By combining different types of evidence, you can build a clearer, more convincing case for the choices you need to make.
Using evidence to test assumptions
An assumption is something you believe but have not proved with evidence. The test and learn process relies on teams identifying and validating assumptions.
The process runs in a cycle:
- Use evidence to identify assumptions.
- Test these assumptions by gathering more evidence.
- Make decisions based on the evidence, which will raise new assumptions to test.
Keep repeating this cycle as you move towards your outcome, using learnings to update your theory of change.
Read the guidance on creating a testing plan.
You can collect evidence by:
- reviewing existing evidence
- talking to people with lived experience
- talking to subject matter experts
- conducting structured research
Each evidence type helps you build a case for your decisions and agree on what to do next. Use your findings to explain the reasoning behind your choices.
Before you start, you can read the Nesta practice guide on using research evidence.
Reviewing existing evidence
Look at existing evidence in the early phases of test and learn to understand your challenge area.
Review ideas tested in different fields and think about how to apply them to your work. For example, you can learn about:
- running safety checks from air travel
- streamlining repeatable tasks from the fast food industry
- keeping users engaged from video games
Existing evidence gives you a framework for decision making. It cannot prove what will work for you because every situation is different. If you rely too much on what other people have done in different contexts, you risk making false comparisons.
Use existing evidence to:
- learn what has worked well elsewhere
- learn from mistakes made elsewhere
- create a logical framework for your assumptions
- think about your challenge area from a different angle
- find innovative ideas from different fields
You can then use these insights to form assumptions, for example an approach:
- worked in a similar context, so it might work for your challenge area
- improved outcomes in a different challenge area, so you should explore if it’ll work for you
Talking to people with lived experience
Talk early and often to the people affected by issues in your challenge area. Their insights will help you design interventions that meet their needs.
Structured research is the main way to talk to people with lived experience. People may also share stories with you informally. Listen to them with empathy and respect, but do not treat informal conversations as research data.
One person’s experience may be typical or unusual. You need to do structured research to find that out. If you rely too much on evidence from a small number of people, you risk designing interventions that only work for a minority.
How to use lived experience insights
Talk to people with lived experience of an issue to:
- identify friction and frustrations
- understand possible motivations for behaviour
- raise assumptions that you can test later
- collect stories and quotes to help explain an issue
Use these insights to form assumptions, for example if one person:
- finds a step in a service frustrating, others may also drop out at this point
- says an incentive motivates them, you should check whether it’s been used elsewhere
Talking to subject matter experts
Subject matter experts help you make sense of complex or technical problems. Even if you have experts in your team, no single person can tell you everything you need.
Expert input will reflect individual judgement and experience. Experts may have:
- personal biases
- outdated knowledge
- knowledge that does not apply elsewhere
- conflicts of interest
Always balance expert views with other evidence to avoid relying on authority or keeping things as they are.
How to use expert insights
Talk to experts to:
- build trust and credibility with stakeholders
- guide decisions in fields where your team’s expertise is limited
- explore strategies and assumptions
- understand technical findings through an expert lens
- identify potential weaknesses in test ideas
Use these insights to form assumptions, for example:
- an expert identified a technical risk, so you should design your test to reduce it
- frontline service staff reported a bottleneck, so you can assume the process is contributing to delays
Conducting structured research
Structured research is when you collect data through a recognised research method, such as:
- surveys
- experiments
- interviews
Unlike informal conversations, structured research is planned in advance and aims to produce repeatable results. If someone uses your method, they should find similar or comparable results.
When to use structured research
You can conduct structured research into services or with people, including experts or those with lived experience.
You’re less likely to use structured research at the start of your test and learn work. You’re more likely to use it as you move through design and into testing.
Types of data
Structured research collects 2 types of data:
- quantitative data – numerical information that helps identify trends or show if an action causes a result
- qualitative data – insights into people’s experiences that explain why something is happening
Strengths and risks of structured research
Structured research helps you assess the extent of an issue and reduces bias.
Setting your scope can limit what you find. Be aware you could miss findings outside your research area, lose important detail or narrow your focus too much. For example, relying too much on quantitative data could make you focus on changing numbers rather than meeting user needs.
How to use evidence from structured research
You can use structured research to:
- understand the size or frequency of what you’re interested in
- identify trends and patterns
- test assumptions generated by other forms of evidence
- prove that something is or is not working
- understand why something is or is not working
Turn these insights into assumptions, for example:
- quantitative data shows two-thirds of users experience a problem, indicating a widespread issue you need to resolve
- qualitative data shows people struggle to access a service on mobile phones, which may explain why case numbers have dropped after a website relaunch
Use a range of evidence to support your decisions. There are 4 main types of evidence you can use:
- analogical – to compare with a similar or related situation
- anecdotal – to understand user stories and experiences
- testimonial – to provide expert views and opinions
- statistical – to show scale, measure impact and identify trends using numerical data
Using a mix of these evidence types helps build a clear and convincing rationale for decision making.
Combining evidence from different sources
Test and learn teams include people with a range of skills and perspectives. This is known as a multidisciplinary team.
Each role uses distinct methods to gather evidence. For example, a user researcher and a data analyst will approach the same question from different angles and find out different things.
Work with the range of evidence they generate to understand an issue, rather than relying on one person’s findings. Use the mixed methods research practice to combine qualitative and quantitative evidence.
Cross-checking results
Check your findings against different sources of evidence. For example, if you identify a trend in existing evidence, ask people affected by the issue if it matches their experience.
Finding the same pattern across sources gives you higher confidence in your results. This is known as triangulation.
Filling gaps in your understanding
Use different types of evidence to explore different aspects of an issue. For example, if structured research shows a service feature is not working, talking to service users or experts could explain why.
Combining these insights gives you a broader understanding than one source would provide. This is known as complementarity.
Bringing findings together
Bring your evidence together to get a clearer understanding of a problem, rather than keeping the sources separate. For example, you can use:
- data about problems in one service to inform discussions with users of another
- interview insights to design tools that collect numerical data
Combining your evidence leads to deeper insights. This is known as integration.
A theory of change visually maps:
- how your activities lead to the outcomes you want to achieve
- the assumptions you’re making about how that change will happen
Developing a theory of change provides a foundation for your test and learn project, setting out the logic for:
- how your intervention will work
- what you should test along the way
Benefits of using a theory of change
A theory of change helps you:
- agree on what you want to achieve through testing
- check that your activities, expected results and evidence are linked
- identify and challenge your assumptions
- spot unintended consequences
- avoid solution-first thinking
Solution-first thinking happens when you choose an idea because you like it. Base your decisions on evidence that an idea will improve outcomes.
Draft a theory of change for your intervention once you know the outcomes you want to achieve.
Creating a theory of change
Involve stakeholders by running an online or in-person workshop. Working together helps to:
- build shared understanding
- invite constructive challenge
Start by listing the impact and outcomes you want to achieve. Then map out the activities and resources needed to reach them.
Use a draft that is partly filled in to help stakeholders who have:
- limited time
- different levels of knowledge
You do not have to finish your theory of change in one session. Developing it over time gives people a chance to reflect between sessions.
Before you start, you can read the Government Analysis Function theory of change toolkit.
Iterating your theory of change
Your theory of change should be a living document.
As you move through the test and learn phases, you’ll learn more about how your intervention works. Update your theory of change regularly using evidence of what works and what does not. This process gives you a strong rationale for how each iteration supports your intended outcomes.