Testing, learning and iterating (phase 4)
Create a test plan, choose testing methods, measure success and decide whether to continue, iterate, pause or stop.
In this phase, you test your ideas in a structured way and use the learnings to decide whether and how to iterate. Testing helps you learn quickly and make better decisions about how to reach an outcome.
Main aims of this phase
The main aims of this phase are to:
- develop a test plan
- test your idea
- gather evidence you can learn from
- use evidence, assumptions and your theory of change to guide your decisions
- iterate what you’re testing, along with your theory of change, assumptions and decisions
- know whether you want to continue testing and iterating, pause, scale or stop
You may develop a monitoring and evaluation framework to record longer-term learning.
Outcomes of this phase
At the end of this phase you should have:
- completed at least one test cycle to understand how your idea works in practice
- used your findings to guide decisions on what to do next
- updated your theory of change based on evidence
Testing is a way of collecting evidence to guide decisions that help you reach an outcome.
Testing gives you a structure for learning about an idea.
It involves:
- an assumption and related hypothesis, covering what you believe is true and the prediction you want to test
- an activity, setting out what you do to test the hypothesis
- a way to observe and record results, so you can capture learnings
Good practice when testing
When you’re testing ideas, you should:
- work as a multidisciplinary team so you benefit from the right skills
- focus on assumptions with greater uncertainty or risk
- focus on shared, measurable outcomes
- build an evidence-based understanding of the test idea
- combine different types of evidence to make decisions as you test
- use learnings to decide whether to iterate, scale, pause or stop
- work in the open
Why you should test
Testing moves a team from abstract ideas to making practical design and delivery decisions. It gives you the information you need to decide what to do next.
Testing helps you:
- reduce risk by learning what works before investing more time and money
- develop better services using feedback from local people and partners
- build enthusiasm by bringing ideas to life
- challenge assumptions by seeing how an idea works in practice
Before running a test, invest time as a team in agreeing what you hope to achieve. Only test when you’re clear how the results could change your next steps.
Make a plan to give structure and purpose to your activities.
Your plan should include:
- the aim of your test
- assumptions and hypotheses to check
- testing methods you’ll use
- measures of success
- risks and ethical considerations
- decisions you’ll make based on results
Setting your test objectives
Test and learn lets you start small and make changes to design and delivery before scaling an intervention.
Starting small means:
- testing the simplest working version of an idea so early feedback informs your next steps
- focusing on your riskiest assumptions first, so you check what matters most before investing further
Set out what you want to achieve from your test and why the test and learn approach suits your challenge area.
Example: setting objectives
The Test, Learn and Grow (TLG) team worked with Liverpool City Council and other partners to plan a test for a community healthcare intervention. The aim of the test plan was to prevent and reduce respiratory illness in children aged 5 and under.
The plan identified 3 objectives, to:
- see if an intervention used elsewhere in the region could work in Liverpool
- check if the intervention could be adapted to the health needs of a new location
- find ways to fund and maintain the intervention beyond the test period
Agreeing decision criteria
List the decisions you need to make and when you need to make them, using your theory of change as a guide.
This list will help you:
- choose your testing method
- collect the right evidence
- involve the right people
Set your decision criteria early so you know how to act on findings. Review your progress regularly and adapt your plan as you gather evidence.
Decide as a team what evidence you need to:
-
continue or scale – when an idea works and is ready for developing or a larger test
-
iterate – when an idea shows promise, but needs changes before you test again
-
pause – when you need more evidence, capacity or a change in conditions before deciding
-
stop – when evidence shows the idea is not working
Setting criteria at the start protects against 2 common biases:
- giving early positive signals more weight than mixed or negative ones
- continuing an approach because you put time and effort into it
Make sure the evidence you plan to collect matches the size of the decision.
Identifying assumptions
An assumption is something you believe is true, but have not yet proved with evidence.
When developing an idea, you usually make assumptions about:
- people’s needs or behaviour, for example “people will use this tool”
- feasibility, for example “this idea is technically possible”
- impact, for example “earlier conversations will keep people in employment”
List all the assumptions your team is making.
Prioritising assumptions
Focus your testing on your riskiest assumptions. These are the ones that have the biggest impact on your outcomes and the least evidence to support them. You can test the rest later.
To prioritise your riskiest assumptions, you can:
- plot them on a grid showing certainty and impact
- score them against your decision criteria
Read guidance on prioritising assumptions based on evidence and impact in Chapter 5 of the Magenta Book Test and Learn Annex.
Example: prioritising the riskiest assumption
In Liverpool, the TLG team worked on a test plan to see whether pop-up clinics and service referrals would improve the respiratory health of children and young people.
They used the riskiest assumptions in their theory of change to design the test. For example, the team identified an assumption that referring families to one service with a high impact on health, like housing support, would be more effective than offering several services.
This assumption was considered to be:
- high impact – it affected whether the intervention succeeded
- low certainty – there was little evidence to prove or disprove it
To test the assumption, the plan focused on offering referrals across multiple services.
Turning assumptions into hypotheses to test
A hypothesis takes your riskiest assumption and turns it into a prediction you can test.
Writing hypotheses down helps your team agree on what to test, which is an approach known as hypothesis-driven design.
One way to write a hypothesis is:
If we… [do something]
Then… [this will happen]
Because… [reason based on your theory of change or existing evidence]
The change you predict can be short term or long term.
You should build your test plan on the decisions you need to make. It will define:
- your testing methods
- the evidence you need
- who you test with
Collecting the right amount of evidence
The evidence you need changes as your service or intervention develops. What counts as ‘good enough’ depends on your context and testing stage. Some decisions need greater certainty and evidence of outcomes. Agree as a team on the type and amount of evidence you need for the stage you’re at.
In early stage testing, your goal is to reduce uncertainty and risk around design decisions. You need less evidence at this stage because the cost of being wrong is low. You have not committed resources or scaled. Look for early signs of impact, including the possibility that your idea might make things worse.
In later stage testing, you need stronger evidence to support decisions that have a big impact on people’s lives.
This evidence helps you decide whether to:
- scale an existing service in an area
- expand to new places
- invest further in an intervention
Read guidance on testing tools, methods and trial design in Chapter 6 of the Magenta Book Test and Learn Annex.
Choosing a testing method
Match your method to what you need to learn.
If you’re exploring ideas and do not yet have a prototype to test, read about using participatory methods for design.
If you have an idea and want to test your assumptions about it, use qualitative interviews, short surveys or user journey mapping.
To learn what works as you build your service or intervention, you can use:
- testing with users
- qualitative sprints
- A/B testing
- randomised controlled trials
Example: planning test methods
In Liverpool, the Test, Learn and Grow test plan focused on adapting a successful local health intervention to a new location.
The plan included activities that would quickly find out which:
- families the test should focus on in the new location
- local support services would most benefit families in the area
These activities involved:
- analysis of existing health data to identify relevant families in one neighbourhood
- co-design workshops with staff and service users to prototype and iterate the initial service offer
Deciding who to test with
You should test with:
- people affected by or using your intervention
- practitioners who will deliver it
In early testing you can learn a lot from a small number of well-chosen people. If you’re using qualitative methods, the Service Manual recommends 4 to 8 participants for each round of research.
For quantitative methods, aim for a sample of at least 30 people. Make sure your sample reflects the people the intervention is designed for, matching characteristics such as age or gender.
When you’re evaluating your service, testing with representative participants will give you stronger evidence of impact.
Read the Service Manual guidance on:
Planning a comparison group
Later in the process, you may need evidence that your intervention caused a specific change.
To measure impact, do not include all eligible users in early testing or delivery.
Plan early who you’ll test with, so you can form a comparison group. You can then measure differences in outcomes between those who have used the intervention and those who have not.
Testing is only useful if you use what you learn. Plan your learning before you start testing. Think about:
- your measures of success
- how you’ll capture data during testing
- how your team will review findings
- what decisions you need to make based on learnings
Your test idea might not have the effect you want – this is still useful learning.
Defining measures of success
Decide how you’ll measure success before you start testing. What you measure may change as you learn more.
Your measures should help you understand the impact of an idea or whether your assumptions are correct. Writing hypotheses makes it easier to choose what to track.
Good measures will:
- show whether something has happened
- help you decide what to do next
Measures are not pass or fail scores. Tracking them consistently over time creates the data need for evaluation.
Read about developing a monitoring and evaluation framework.
Once you’ve chosen your measures, set out how you’ll collect data to track them. Agree as a team on the level and type of evidence you need at each stage of testing. Combining qualitative and quantitative evidence helps you understand how well an test idea works.
Read about collecting the right amount of evidence.
Example: measuring success
In Liverpool, the Test, Learn and Grow team’s test plan aimed to reduce repeat visits to A&E and GPs for respiratory problems by introducing regular clinic appointments.
A short initial test would be unlikely to show changes in these top-level numbers.
Instead the team’s evaluation plan focused on early signs that symptoms were improving and services were dealing with the underlying causes of the illness. For example, the plan set out to measure whether:
- people needed less emergency medication after visiting the clinic
- children and caregivers took up referrals to support services
Establishing learning routines
Agree how your team will learn from the data you collect and insights you generate. You could agree:
- the way you document what you learn, such as a learning log
- regular routines, such as synthesis sessions to review findings as a team
Managing risks and ethical considerations
Test in a way that is safe, respectful and responsible. Consider and document:
- potential risks to participants and team members
- ethical practice and how testing will meet standards
- data protection
- safeguarding, if you test sensitive topics or work with people in vulnerable situations
- steps to reduce any risks you identify
Read the guidance on conducting research safely.
Agreeing on your approach with partners and your team
Testing plans can become complex, particularly for methods such as randomised controlled trials. Include evaluation advice in your project to identify complex methods that need extra planning.
Agree testing decisions with your team and partners before writing a detailed plan. For example, you could share a summary that covers:
- what you want to learn
- how you plan to test
- what you’ll measure
- the decisions you hope to make
You could also run a workshop with decision makers to agree your approach.
Use different ways to collect evidence and see how your intervention works for users, so you can make decisions with confidence.
Check new findings by combining them with existing evidence. Comparing evidence is especially important later in testing, when you need confidence to make big decisions.
Read the guidance about combining different types of evidence.
You do not need to wait for later evaluation to collect data. Running small-scale surveys can strengthen your early testing.
To get the most from your evidence:
-
find out what quantitative data the service already holds, including administrative data, and make sure you have the right data protection documents to use it
-
plan how you’ll collect new evidence to avoid asking the same people to participate again
- map your findings in a team session to see what each test showed and how they fit together
- review evidence as a team to agree on findings and next steps
Read the guidance on collecting the right amount of evidence.
Record your findings and review what you learn. Use what you learn to inform your next steps. A learning framework helps you keep your wider context in mind alongside day-to-day, small-scale testing. Use it to map evidence against your:
- theory of change
- assumptions and hypotheses
- decisions
You should also keep a change log to record big decisions and changes.
Iterating your theory of change
Update your theory of change as you test if:
- new evidence confirms or challenges your assumptions
- your activities, resources or outputs change – for example, if you improve a service and need to reflect how you deliver it
- your intended outcomes change, which may require a new theory of change
Review your theory of change regularly during planning cycles to:
- record what has changed after testing
- check that the link between your activities and outcomes still makes sense
Building evaluation into your work
In test and learn, evaluation is not separate from testing. It’s a structured way to measure changes in:
- impact
- process
- implementation
- value for money
Base your evaluation on data that tracks how your activities lead to outcomes.
Before you start testing, decide how and when users will receive the intervention. Making delivery plans early ensures you have the right data to prove your impact later, for example by setting up a comparison group. Read about deciding who to test with.
As you move from testing ideas to proving what works, you’ll need stronger evaluation and evidence to show your intervention caused a change. Read about defining measures of success.
Developing a monitoring and evaluation framework
Work with evaluation colleagues to create your monitoring and evaluation framework, using your theory of change as the starting point. The framework helps you:
- identify the outputs and outcomes you want to track
- write hypotheses to test across the project
Decide how you’ll collect data on your main outcomes from the start. Track this data throughout testing, using:
- direct measures – to get the exact information you need
- substitute measures – to get related information
- your agreed measures of success – to check if you’ve met your overall goals
Continue evaluation once the intervention is set up. This will help you decide whether to scale, iterate, pause or stop. Agree your decision criteria early to make this process easier.