Research and analysis

Health Transformation Programme evaluation strategy

Updated 13 July 2023

DWP research report no. 1030

Crown copyright 2023.

You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence. To view this licence, visit The National Archives or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email psi@nationalarchives.gov.uk

This document/publication is also available on GOV.UK.

If you would like to know more about DWP research, email socialresearch@dwp.gov.uk

First published May 2023.

ISBN 978-1-78659-529-4

Views expressed in this report are not necessarily those of the Department for Work and Pensions or any other government department.

Executive summary

The Health Transformation Programme (HTP) is transforming the Personal Independence Payment service, introducing a simpler application process, including an option to apply online, improved evidence gather and a more tailored journey for customers. It is also developing a new single Health Assessment Service for all benefits that use a functional health assessment.

The HTP is a large, complex and agile programme which means that our approach to evaluation must develop as the Programme does. This high-level evaluation strategy provides an overview of our plans to evaluate the Programme. The evaluation is built around four interchangeable layers that will help the Programme build a robust evidence base. It also includes a Theory of Change, an internal and externally commissioned research programme; a test and learn strategy; ongoing evaluation of services; and a performance framework.

Health Transformation Programme Aims

The Health Transformation Programme (HTP) is modernising benefit services to vastly improve customer experience, build trust in our services and the decisions we make, and create a more efficient service for taxpayers. The Programme is developing a new Health Assessment Service and transforming the Personal Independence Payment (PIP) service over the longer term.

The HTP will also support testing and implementation of proposals set out in the Transforming Support: The Health and Disability White Paper^{[footnote 1]}.

The Programme’s key strategic outcomes are:

• increased trust in services and decisions
• a more efficient service with reduced demand for health assessments
• increased take up of wider support and employment
• improved customer experience with shorter journey times
• transformed in-house data and IT infrastructure that is secure

The HTP Theory of Change (ToC) provides a high-level overview of what we could change, how we could change it, and how we expect these changes to support the Programme’s strategic outcomes. It will inform what the Programme is testing and the changes we take forward when scaling the service. The Theory of Change section looks at the ToC in more detail.

This strategy sets out our current high-level approach for evaluating and tracking the performance of the HTP as a whole.

Health Transformation Programme Scope and Approach

In this section, we provide more detail on the new services that the HTP is developing, the difference these new services will make and the Programme’s approach to developing them.

The Programme is transforming the entire PIP service, from finding out about benefits and eligibility through to decisions and payments. As part of this, the programme will make improvements to existing ways to claim and introduce an optional online service, providing customers with a greater choice over the channels they use to interact with the Department for Work and Pensions (DWP).

The transformed PIP service will deliver a simpler application process for customers with more information and support available to those who need it, including helping them decide whether applying for PIP is right for them. Improved evidence gather will also enable the Programme to better tailor the service to the customer’s circumstances. For example, they will only have a face-to-face assessment if this is the most appropriate method and it may be unnecessary for the customer to have an assessment at all.

As part of this, we are exploring a case management model: a personalised approach for customers from initial contact and throughout the application, including signposting to other benefits and services. This approach will help the Programme to better understand our customers, tailor their service, and help to build customer confidence and trust that their case is being progressed appropriately.

Simplifying and automating the PIP journey and tailoring how the customer is assessed will mean many customers get a decision on their claim much quicker. Customers will be able to apply, upload evidence, track progress and see payment details online. Communications and notifications will be simpler and easier to understand (and for those on the digital journey, they will be available to receive and submit electronically) so that customers understand their claim decision and how the evidence they provided supported it.

The programme is also creating a single new Health Assessment Service for all benefits that use a functional health assessment. This will eventually replace the different services we and our assessment providers use to undertake health assessments across all benefits, including new IT and processes.

The Health Assessment Service will be fully integrated with other systems, including the transformed PIP service, to create a seamless customer experience. By improving how we gather evidence and by enabling the re-use of information, the new integrated service will provide DWP agents and Healthcare Professionals with easier access to relevant information. This will reduce the burden on customers to provide complex information and reduce the need for them to provide it more than once.

With around 2 million health assessments conducted each year, the nature and scale of the HTP’s work is complex and ambitious. The new services represent a major change to how the Department manages functional health assessments and how we support those who are applying for PIP. In recognition of these complexities, and the need to protect vulnerable customers, we are developing the new services at a small scale in a safe and controlled environment known as the Health Transformation Area. The Health Transformation Area is where we are exploring, adapting and learning from new ideas and processes. It provides the space to develop and build our services safely, before carefully and gradually growing the service.

The Programme is procuring new assessment provider contracts, which will start operating from 2024. These Functional Assessment Service contracts represent an important step in delivering our ambitious longer-term service reforms. They will provide the foundation for the new Health Assessment Service, replacing the current Health and Disability Assessment Services and PIP assessments with joint services under a single contract for each geographic area. They will ensure continuity of service for customers while we safely develop the new service, providing the flexibility to introduce the new Health Assessment Service gradually, before we roll it out nationally, from 2029. The Functional Assessment Service contracts will support white paper reform proposals and a gradual move towards phasing out the Work Capability Assessment. Once the new contracts are in operation, customers will experience more consistency, interacting with the same Health Assessment Service provider regardless of which benefit they are being assessed for.

Overall, the Programme is taking a long-term, phased approach that will move away from the current ways we are operating, gradually and carefully. The programme is currently focussed on building the foundations and creating the environment to enable the Programme to begin to transform these services for the future.

Evaluation Approach

Evaluation is a systematic assessment of the design, implementation, and outcomes of an intervention. For a major programme of change such as the HTP, it is essential that comprehensive, robust, and proportionate evaluation is undertaken to ensure the Programme is delivering positive outcomes and value for money. The content of this strategy utilises evaluation guidance from the Green Book and Magenta Book and builds on the DWP Evaluation Strategy which sets out the approach for ensuring that DWP evaluations deliver high quality evidence, on the right questions, at the right time.

We are drawing on advice from other major DWP programmes, and cross-government experts to build a programme of evaluation and monitoring which supports the agile nature of the HTP across the life of the Programme. As well as the work detailed in this document, our evaluation strategy will be complemented by ongoing day-to-day learning and iterating that forms an integral part of an agile programme. This will include the insights and learning that we collect as part of our usual processes, such as that undertaken by user researchers, Digital service designers and business analysts.

Evaluation Aims

The evaluation of the HTP will develop over the life of the Programme to incrementally build a picture of: 

• the context, influencing factors and levers which relate to the outcomes of the HTP as set out in the strategic outcomes, ToC and Programme business case
• how successful the HTP interventions are at achieving these outcomes and delivering on the HTP’s business case

The evaluation will provide evidence to inform the HTP’s decision-making and support the DWPs’ wider Health and Disability strategic outcomes. The core principles of the evaluation are to: 

• establish a robust and detailed evidence base to inform decisions about service improvements and how the HTP can drive better customer experience 
• provide person-centred insight on what works for whom to ensure a broad range of needs are considered when transforming our services 
• provide assurance that the business case has been met
   

The remainder of this strategy will describe how the evaluation will deliver on these aims and principles.

Evaluating an agile and evolving programme

The HTP evaluation will align to the agile approach to design and delivery in the Programme. We have developed an evaluation approach which ensures we are consistently providing timely evidence and iterating our learning plans to feed into key decisions across the life cycle of the Programme, not just at the end. This approach facilitates evidence-based decisions and allows the Programme to adjust direction where necessary, based on up-to-date evidence.

To deliver this approach, our evaluation strategy works across 4 layers interchangeably:

• Planning and Scoping
• Cross-Cutting Insight
• Test and Learn
• Evaluation of Services

Planning and Scoping

The Planning and Scoping layer ensures that we are developing and iterating our evaluation approach with consistent principles and understanding.

This involves:

• developing a ToC (see the section on Theory of Change) that sets out the vision for the HTP by describing the inputs it can change, the mechanisms it can use to bring about the desired change, and the outcomes it is aiming to achieve
• iterating an evaluation strategy that provides a clear forward looking evidence plan
• working with delivery teams to understand emerging evidence needs

Cross-Cutting Insight

The Cross-Cutting Insight layer ensures that we are continuously learning about the context, levers and influencing factors that relate to the HTP.

This involves:

• conducting research about a broad range of issues that affect our customers and services to fill evidence gaps and provide up-to-date learning (see the section on Test and Learn strategy)
• reporting against a set of performance and results indicators to build a picture of whether the Programme is achieving what it set out to achieve (see the section on Data Sources)

Test and Learn

The Test and Learn layer ensures that when we test new ideas, we are testing the right things at the right times and feeding the learning into Programme feedback loops to influence design and delivery decisions (see the section on Evaluation of Services).

This involves:

• designing a process for logging, prioritising, and scoping new ideas for testing
• implementing tests that build our understanding of new ideas whilst protecting customers and staff
• ensuring we get robust learning from our tests by implementing proportionate evaluation and reporting

Evaluation of Services

The Evaluation of services layer ensures that, as services develop, we can understand to what extent the HTP has successfully achieved its outcomes (see the section on Measuring Performance).

This involves:

• Conducting ongoing evaluation work which will scale as the Programme scales to give an indication of impacts and the effectiveness of delivery, and inform key decisions

As required, evaluation activity will include:

• impact evaluations which tell us what effect the service has on customers and outcomes
• process evaluations which tell us whether the delivery of the service is successful and sustainable over time
• value for money assessments which tell us if the service delivers good use of tax-payer money and delivers on the business case set out for the HTP

It is important to note that the strands of work in the HTP will not move linearly across these four layers but will move back and forth between layers as we build the evidence base about what works. For example, we might undertake some evaluation and then move back to test some new initiatives based on what we learnt.

Theory of Change

How we use Theory of Change to structure our evaluation

The HTP ToC has been developed to set out the high-level vision for the Programme. A ToC is a way of demonstrating what we can change (inputs), how we could change these inputs (mechanisms), and what we want to achieve when we’ve made the changes (outcomes). By highlighting the links between these three things, the ToC surfaces the assumptions that sit behind the HTP’s activity. Understanding these assumptions allows the Programme to assess the evidence base and highlights where we have unevidenced assumptions that need to be tested. This forms the basis for our evaluation strategy by guiding what we will test, how we will test it, and what we need to measure to understand if we are achieving our goals. The ToC is intended to be a live document that will evolve and be updated as the Programme develops.

The ToC will be used across the different stages of the Programme to guide transformation and implementation. In the Programme’s current stage, the ToC supports development of our evidence base by identifying assumptions and knowledge gaps which we will then address with research and analysis. This allows the Programme to build the foundations for the right kind of transformation in the future. As HTP moves to the next stage, the ToC will be used to identify opportunities to test innovative solutions to improve services. In the future, we will use the evidence built around the ToC to underpin our decisions about what changes we take forward when scaling the service and delivering the Programme’s outcomes.

Theory of Change Logic Model

The HTP ToC Logic Model is illustrated here, and the full ToC can be found in annex A.

Operating Environment

Inputs	Mechanisms	Outcomes
Communication Job Design Data Processes Estates Contracts IT and Digital Tools Staff Capability Policy	Intelligent use of data Transparency Information Gathering Case Manager and HCP Behaviour Accessibility Person-centred approach Accuracy	Increased customer trust in services and decisions More efficient service with reduced demand for health assessments Increased take-up of wider support and employment Improved customer experience with shorter journey times Transformed in-house data and IT infrastructure that is secure

Research Programme

To ensure that the iterative design and implementation of the HTP is built on a strong evidence base, we have put in place an ongoing programme of internal and external research. Through rapid evidence reviews and the design and commissioning of new research projects, we are ensuring that the strategic design and delivery of the HTP is rooted in evidence.

Our research programme involves a combination of in-house and externally commissioned projects using a variety of qualitative and quantitative methodologies to gather robust insight into priority areas for the Programme to address. We are undertaking research both within the DWPs’ business as usual services and within the Programme’s test and learn sites to understand more about customer and staff experiences, as well as looking at more cross-cutting topics relating to health and disability. We are taking proactive steps to ensure that our research practices are inclusive, and we seek input from a diverse range of disabled people.

We are developing our 2023 to 2024 research programme, ensuring that new research addresses key gaps in our evidence base and seeking views from a range of stakeholders. Further details of a project that is currently underway can be found below.

Example: Fluctuating health conditions

Research Programme example: Understanding fluctuating health conditions

For the Programme to successfully meet its strategic outcome around increasing customer trust in services and decisions, it is critical that the service meets the needs of all of its prospective users. We know from wider feedback that current assessments don’t always fully capture the impact of fluctuating conditions and it can be difficult for some people with fluctuating conditions to answer questions about how their health is for the majority of the time. To be able to address this in future service design, we need to start by building our understanding of how fluctuating health conditions are experienced and described by individuals.

To support this, we have commissioned an exploratory piece of research with a panel of individuals who consider themselves to have health conditions or disabilities that fluctuate over time. We have designed a multi-phase mixed method research project to address these questions: a survey exploring the variety of ways in which conditions fluctuate and impact upon people’s lives; a month-long diary study for individuals to articulate their experiences of fluctuations in real-time; and in-depth interviews to reflect and probe further. From this a typology of fluctuations will be developed, which will in turn be used to provide evidence-based recommendations to support future changes to how the application and assessment processes capture evidence of fluctuation.

Test and Learn Strategy

The HTP has used test and learn guidance from other major DWP Programmes such as Universal Credit and the Government Digital Service to build on best practice when designing our approach to developing and testing new ideas in the HTP. The Health Transformation Area provides a test and learn environment that enables the Programme to safely test new ideas, quickly providing insight about what works and building the evidence base that sits behind our ToC. This enables the Programme to make informed changes to the way we are designing and delivering transformation.

Test and Learn Principles

To ensure our test and learn activities produce high quality, innovative insight which drives behavioural, system and process changes whilst protecting our customers and staff, we use 5 principles to guide the HTP test and learn activity:

• testing needs to be informed by the ToC
• build engagement with a diverse range of stakeholders in from the beginning​
• take a consistent approach to test identification and design​
• protect staff & customers from risk​
• use a consistent and appropriate system of governance

Testing needs to be informed by the ToC

The ToC sets out the kinds of change needed to drive success and should remain the overarching reference point for all testing.

Build engagement with a diverse range of stakeholders in from the beginning

Engaging with a broad range of internal stakeholders and consultation with external stakeholders (including small numbers of customers) at an early stage ensures that we are driving the right behavioural and process changes and are building the right thing for the people who need it.

Take a consistent approach to test identification and design​

By following a consistent prioritisation process, we can decide what we will test and how we will measure successful outcomes from tests to ensure that testing drives quality learning.

Protect staff & customers from risk​

Staff involved in test and learn activity will be trained to deal with difficult situations that could occur when working with customers, meaning that tests adhere to the same standards for supporting customers and staff that are applied to all DWP services

Use a consistent and appropriate system of governance

Sufficient oversight is in place to scrutinise all the strands of the Programme and give a clear picture of what we are learning. Testing will be carefully managed to maximise available capacity while avoiding any potential contamination caused by running multiple tests at the same time.

Prioritising and Designing Test and Learn

As well as using the test and learn principles, our growing evidence base, policy priorities and operational deliverability will help the Programme to prioritise what gets tested and when.

Our test and learn activity will take different forms depending on the kind of learning that is required. We will use a range of methods from exploratory research, to structured proofs of concept, and robust trials, depending on what is appropriate and what level of evaluation and monitoring is proportionate. These decisions about method will be informed by the scale of the decision that the learning will inform, the risks associated with the intervention, and the existing evidence base.

The types of things we will explore through test and learn, that relate to the HTP ToC and strategic outcomes, include:

• how should we clearly communicate with our customers to build their trust in our services and decisions?
• how can we use data more effectively to decrease journey times and deliver better value for money?
• how can we ensure our staff have the right skills and training to best support improved customer outcomes?
• how could our processes be more person-centred to accommodate different needs and improve customer experience?
• how can DWP’s IT and digital tools help the HTP services to be more flexible to future customer and service needs?

Note – these are simplified examples and not exhaustive.

The Health and Disability White Paper identifies several areas as having the potential to drive better experience and outcomes for people who apply for health benefits. As we continue to evolve the Programme’s approach to Test and Learn, we will look into the best options for progressing the testing of these new initiatives.

Evaluation of Services

Ongoing evaluation will provide evidence for key decision points as the HTP services develop. This will enable the Programme to accurately determine whether it is achieving its strategic outcomes. Evaluation will tell the Programme if HTP services deliver:

• improved outcomes for customers, including a clear understanding of what works for whom, and how the services work for claimants with protected characteristics
• effective delivery mechanisms
• value for money for the taxpayer

This ongoing evaluation will draw on 3 methods at different stages of the HTP to assess these things.

Impact Evaluation

We will use impact evaluation to assess what changes have occurred at least partly as a result of the HTP’s changes, the scale of those changes, and the extent to which they can be attributed to the HTP.

The impact evaluation will assess this in a number of ways including: ​

• impact assessment which uses quasi-experimental analysis of administrative data to determine the causal effects of the HTP services on customer outcomes. It does this by comparing to customers with similar characteristics but who didn’t receive the HTP service (a comparison group)
• customer surveys gathering perspectives on a range of aspects of their experience.
• qualitative research with customers and staff to understand impact of the HTP services on experience

Based on the HTP’s strategic outcomes, examples of evaluation questions we will seek to answer are:

• do customers report having a better overall customer experience when using the HTP service than when using DWP business-as-usual services?
• are customers less likely to dispute their benefit decision when using the HTP services than when using DWP business as usual services?
• is there reduced need for health assessments in the HTP services than in DWP business as usual services?

Process Evaluation

We will use process evaluation as the HTP services develop to assess if changes are being implemented as intended, whether implementation is consistent across sites, and whether there have been unexpected factors influencing delivery.

This will consist of a range of methods including: ​

• qualitative research with staff and customers to understand consistency and effectiveness of delivery ​
• surveys with staff and customers to understand their experience of using the service
• data collection from our systems to understand how the service is being delivered

Based on the HTP’s strategic outcomes, examples of evaluation questions we will seek to answer are:

• do customers experience any barriers to accessing the services?
• what worked well and less well for which customers when using services?
• do staff have the right resources to deliver the service as intended?

Value for Money Assessment

We will use Value for Money assessments of the HTP services to weigh up the total monetisable benefits against the total cost of implementing and delivering the service. This involves factoring in economic, social, and environmental costs and benefits, whether intended or unintended.

Value for Money assessments will utilise the findings from the impact and process evaluation strands. For example, if the process evaluation shows that a service is more productive due to greater efficiency, the value for money assessment will determine the monetary value of that increased productivity and add it into the calculation of total costs and benefits.

This process provides assurance that public money is being spent in a way that maximises the value to the taxpayer and delivers against the cost of funding the Programme. As set out in the Green Book, this kind of economic appraisal is at the heart of any programme business case and as such this process provides fundamental information required for the HTP to demonstrate that it has delivered against the commitments it made in its business case.

Any of the HTP services that are tested at scale will be assessed using this method and will be compared to DWP business as usual services when it is appropriate and proportionate to do so.

Examples of evaluation questions we will seek to answer are:

• has the HTP service been cost effective?
• what were the costs of delivering these services?

An example of our approach: Apply for PIP

An example of our approach to evaluation: Apply for PIP

As part of the transformation of the end to end Personal Independence Payment (PIP) journey, a new online service to claim PIP is being developed. This is being designed, built and iterated over time and the service will gradually be scaled up to be available for all citizens. To support this, we are delivering a complementary evaluation approach that provides robust evidence to inform design and scaling decisions at the right points in time.

During the programme’s initial testing phase, we have delivered early process and impact evaluation to explore how Apply for PIP has affected customer experience, engagement in the application process and application journey times. Looking ahead to phased roll-out via gov.uk, we are working to develop a geographical-based implementation approach that will enable us to deliver robust and timely evaluation evidence whilst ensuring that the service can continue to iterate and scale.

Examples of evaluation questions we will seek to answer in this next phase of the evaluation are:

• does a self-serve online application option remove barriers for citizens applying for PIP? If so, how?
• are claim processing times of the digital form quicker than for other application routes? How do staff experiences of processing claims differ by application route?

Measuring Performance

Developing a performance framework

As the HTP develops it is important that we can monitor progress and understand if the Programme is on track to deliver its outcomes. To support this, we are developing a performance framework that links success factors to performance and results indicators. The framework will include balancing measures, to ensure that the HTP does not drive unintended outcomes, and contextual measures, which account for factors that the HTP cannot control directly, but may explain changes seen elsewhere. The indicators in the framework will develop as the Programme and its data sources develop to provide more detail and highlight any new measurements that are needed.

The aims of the performance framework are to:

• provide a consistent approach to measuring progress over time and across the different strands of the HTP
• align with DWP-wide approaches to measuring performance to support (where appropriate and possible) robust comparisons between the HTP services and DWP business as usual services
• support the agile approach by feeding robust and consistent data into the Programme regularly. This will enable the HTP to identify areas that need attention
• allow the Programme to tell a consistent story of progress to our internal and external stakeholders as the Programme develops

It will not be appropriate to compare new HTP services to DWP business as usual services while they are still in development (i.e. before services are stabilised to a point where they can be scaled). This is because:

• the number of customers involved in tests is small to draw statistically accurate comparisons
• the population of customers in the test areas is not representative of the whole population (e.g. geographical differences such as urban/rural)
• data accuracy issues arise during the development of new services
• service design and implementation are not comparable

Key Performance Indicators

There are 9 top level key performance indicators in the performance framework. These have been identified as a set of quantifiable indicators to aid monitoring the HTP performance against its strategic outcomes:

• Health assessment quality
• Customer journey time by milestone
• Percentage of cases cleared within benchmark
• Contracted staff attrition
• Average unit cost of service
• Health assessment capacity compared with demand
• Percentage of actual productivity to funded productivity
• Customer query resolved at first contact
• Engagement with employment support

Key performance indicators and the strategic outcomes are interrelated and can be mapped to each other. These relationships can be bi-directional and one strategic outcome could be linked to multiple key performance indicators.

Data Sources

As set out in the ToC, the HTP aims to use data more intelligently in order to improve customer experience and service efficiency. To that end, as the HTP services develop, new and better data will become available. We will use the best available data source at any given time to evaluate the HTP and report on its performance. This will involve consideration of data reliability, ability to compare the HTP services to DWP business as usual services, and data availability. As part of the planning and scoping layer we are working to ensure new services meet data needs for future service delivery, guarantee current data capabilities, future proof our data sources, and support evaluation.

The data we pull together to evaluate the HTP services will include:

• data and management information about claims, held by DWP administrative systems
• systems data which tells us how our services are performing
• data from bespoke HTP evaluation surveys of customers and staff
• customer experience data

Conclusion and Next Steps

This evaluation strategy sets out the HTP’s plans for how it will build and draw from a robust evidence base and monitor and evaluate progress. This strategy is part of an ongoing programme of evaluation and will further iterate as the Programme’s plans develop to ensure that evaluation and monitoring continues to meet both Policy and Programme needs.

In line with Office for Statistics Regulation Code of Practice for Statistics, we will start to publish Management Information from Autumn 2023 and will continue to develop our plans for publishing metrics.

We expect to further refine the Programme’s ToC, performance measurement approach and evaluation strategy as the Programme matures. As set out, the Programme is in its early stages and is currently creating the right environment to transform services. We will look to publish information about our performance and evaluation at points when the service is suitably developed and robust, and where publishing will provide a representative picture of the HTP’s progress.

Research reports from projects commissioned as part of the HTP’s research programme to address key gaps in our knowledge will be published in line with the Government Social Research Publication Protocol.

1. Transforming Support: The Health and Disability White Paper ↩