Agreeing the challenge (phase 1)
Work with partners to agree a challenge area, choose what to improve, write a problem statement and find opportunities for testing.
In this phase you narrow a broad challenge area down to a starting point you can work with.
A main principle of test and learn is that it focuses on outcomes. It’s about the difference an intervention can make to people’s lives. This phase is about getting clear on what you want to achieve, before you start looking for solutions.
By the end of this phase, you should have agreed:
- a high-level outcome measure
- a problem statement that links to that measure and can guide test and learn activity
Main aims of this phase
The main aims of this phase are to:
- understand the strategic context of the challenge area
- narrow the challenge area down to a problem statement and outcomes
- build consensus among the team, main partners and stakeholders
- identify the big decisions you need to make and the deliverables your research will support
- find opportunities for ideas to take into testing, for example from stakeholder engagement, desk research or data analysis
You may need to do some work before this phase to:
Outcomes of this phase
At the end of this phase you should have:
- a high-level understanding of the problem area, who is impacted and initiatives that currently exist
- engaged main stakeholders, such as policymakers, service leaders, community organisations and think tanks
- agreed which partners to involve in decision making, including on scope
- agreed an overarching outcome to begin exploring test ideas
A challenge area is the policy problem or systemic issue you want to improve.
You can apply test and learn principles to many policy and delivery challenges by researching user needs and getting early feedback.
But test and learn is most valuable when:
- the stakes are high, or outcomes clearly need improving
- there is little evidence about what works
- you do not know how residents will react to or use a service
- you can measure outcomes or proxy outcomes quickly
- there is a realistic way to scale what works
For example, there are many practical delivery questions that come with a shift to neighbourhood-centred healthcare. How do you identify people earlier? How do you organise partners around neighbourhoods? How do you use data safely?
Test and learn methods can help you build early evidence of what works before changing whole systems.
Narrowing down the challenge
Your challenge area may be broad to begin with. Use research and work with your partners to:
- agree on a more tightly defined outcome measure and problem statement
- set boundaries for test and learn
These agreements should be specific enough to begin to identify opportunities for testing.
As your focus narrows, it’s important to build:
- a clear rationale for how the problem links to the overarching outcome
- support and agreement among decision makers and team members
This phase is rarely linear. In practice, the scope may narrow and then widen again as you learn more.
Example: creating a problem statement
Challenge area: helping young people with special educational needs and disabilities (SEND) aged 16 and over move from school into further education, training or work.
Overarching outcome: reduce the number of young people with SEND who are not in education, employment or training (NEET).
Problem statement: how can we identify young people with SEND who are most at risk of becoming NEET and offer them support?
Use evidence from different sources to narrow down the challenge area. You may find several problems that could be a good fit for test and learn. Be led by:
- a clear overarching outcome
- stakeholder priorities
- energy and momentum
Throughout this phase, you can build a backlog of problems to revisit later.
Questions to explore:
- Why are services not delivering the intended outcomes?
- What has already been tried and what works?
- What initiatives are running and where is there momentum?
- What does local data show about service gaps or specific user groups?
Desk research and policy analysis
Review strategies, white papers, data and research to understand:
- the ambition and commitments made, including targets, outcomes or delivery priorities
- existing initiatives, mapping current services to identify gaps and avoid duplication
- the gap between current and desired states, and where doing things differently could make an impact
- urgent priorities, including milestones or deliverables suited to a test and learn approach
Read guidance on reviewing existing evidence.
Stakeholder engagement
Relationship building is an important part of test and learn. Create a stakeholder map and engage stakeholders to:
- understand the challenge through different perspectives
- build collective commitment to prioritising the problem
- identify who needs to make final decisions on scope – aim for the simplest governance model
User research
Explore real-life experiences of the challenge area through user research to:
- understand the main issues facing those affected
- see how services work from the user’s perspective
- create problem statements that reflect experiences of the main user groups
Analysis
Explore existing research and evaluation in your challenge area to:
-
see what evidence says about how to achieve your intended outcomes
- review local, national and international efforts in the area
- create a log of ideas that have been tested elsewhere
Explore existing data in your challenge area to:
- find measures and data for your intended outcomes
- build an understanding of the issue by analysing public and internal datasets
- identify specific groups of people you could test with
- check what data is held locally, and make sure it’s linked and ready for when you need it
- check what data is held nationally or by others, such as charities, universities and businesses, so you can plan for sharing it later
Agreeing on your focus
In complex challenge areas there will be various opportunities for test and learn. You will need to balance the desire to learn more about a challenge with the necessity of agreeing on a narrower scope.
Your timings and approach will depend on your local context. For example, relationships, organisational priorities and ownership of the challenge will influence how quickly you progress.
You can help build momentum and move the work forward by recommending opportunities that are backed by evidence-based rationale.
Example: defining the challenge in your local context
Early in the Test, Learn and Grow (TLG) programme, demonstrator teams were set up to prove the concept. They ran 12-week projects in partnership with central and local government, showing how to put the approach into practice. Teams had to agree quickly, both on the challenge area and the focus for test and learn, prioritising speed over depth.
At a later stage of TLG, accelerator teams had different priorities. They were tasked with specific challenge areas, such as economic inactivity or violence against women and girls. These areas represent significant national challenges and are priorities for both central and local government.
These teams spent 3 to 6 months narrowing the challenges into test and learn opportunities.
They focused on:
- building relationships with local partners
- looking at challenge areas from different perspectives
- understanding local contexts
- getting support for the test and learn approach