WorkWell Pilots Evaluation Feasibility Study

Question 1

Acknowledgements

Accepted Answer

The authors would like to thank the team in the DWP and DHSC Joint Work and Health Directorate for their guidance and support throughout this project, including their contributions to the stakeholder interviews and theory of change workshop. Particular thanks are owed to Emma Heffernan and Amy Skates for their support in managing the delivery of the feasibility study.

Thanks are also due to colleagues in the National Support Team for their time and input throughout the study.

Finally, we would like to thank the representatives from the Pilot sites that gave their time to speak to us as part of this research.

Question 2

Author details

Accepted Answer

The report was authored by researchers at IFF Research:

Angus Tindle (Director)
Lorna Adams (Director)
Catherine O’Driscoll (Associate Director)
Tom Bradley (Research Manager)
Iona Gallacher (Senior Research Executive)

In partnership with researchers at York Health Economics Consortium (YHEC):

Jo Hanlon (Project Director)
Barbara Uzdzinska (Research Consultant)

And researchers at the Centre for Evaluation of Complexity Across the Nexus (CECAN), part of the University of Surrey:

Ben Shaw (Deputy Director)
Stuart Astill (Director, ICILA)

The report was prepared for publication by researchers at the Department for Work and Pensions:

Emma Heffernan (Project manager)
Victoria Leech (Research Officer)

Question 3

Glossary of terms

Accepted Answer

Biopsychosocial interventions	Interventions that take a holistic view of the barriers an individual experiences through their physical health, their psychological situation and their social situation.
Early intervention	WorkWell prioritises intervening at the earliest possible point, as evidence shows this is the most effective way of helping people to stay in work or go back to work.
Employer representative bodies	These are organisations that represent networks of employers belonging to different trades and industries, such as the Chambers of Commerce.
Fit note	Issued by healthcare professionals to provide evidence of the advice they have given about an individuals’ fitness for work. They may be issued to individuals who are unwell and cannot work for more than 7 days, including weekends and bank holidays.
Integrated Care Board (ICB)	NHS organisations responsible for planning health services for their local populations. They manage the NHS budget and work with local providers of NHS services, such as hospitals and GP practices.
Jobcentre Plus (JCP)	JCP is a funded by the Department for Work and Pensions, whose aim it is to help people of working age find employment in the UK. It is also responsible for delivering services that help people claim working age benefits.
Joint Work and Health Directorate (JWHD)	JWHD is a directorate within DWP and DHSC, who are responsible for delivering better work outcomes and work experiences for disabled people and people with health conditions.
Learning and Change Manager	A stipulated role in each pilot area, the Learning and Change Manager will be employed by the pilot area and will work as part of the NSO, to raise awareness of WorkWell and contribute to cultural change towards local work and health integration.
Local Authority (LA)	Also known as councils, LAs provide public services, including work and health support. They operate independently of central government.
Participant Management Information (MI)	The data that pilot areas are expected to submit to the JWHD monthly to satisfy the conditions of their funding. This will support the local and national evaluation of WorkWell.
Grant Management Information (MI)	It is a requirement of the grant conditions to submit quarterly MI and financial expenditure returns. The MI is used to monitor the performance and progress of the ICBs. It consists of starts achieved and customer satisfaction scores. Key performance indicators outlined in the grant agreement are set against these measures. Therefore, the quarterly returns inform how the ICB is performing and enable performance management actions to be taken where appropriate. They also inform on the collective outputs of the grant scheme.
Multidisciplinary teams (MDTs)	This is where services are delivered by two or more members of staff from different disciplines, which may include a mix of non-clinical and clinical roles. The overall composition of the MDT is to be determined by individual pilot areas according to local needs.
National Support Offer (NSO)	A support offer attached to the WorkWell programme, including provision across three core tiers nationally (National Support Team), regionally (Regional Programme Advisers), and locally (Learning and Change Managers). The NSO was set up to support pilots to meet their delivery targets as they compete with other high-level priorities in their local areas. The learning shared will support pilot areas to adjust their services appropriately and contribute to national learnings about which models work.
National Support Team (NST)	The NST will work as part of the NSO as a national team of experts, working closely with Regional Programme Advisors, to encourage evidence-based planning and delivery, and support a programme of national and regional cross-system learning.
Onward referral route	Once an individual has been referred to and received support through WorkWell, this is the way in which they are passed onto other local services to receive the personalised support they need.
Outcome	An outcome is recorded by pilot areas as part of their management information submission, stating whether a participant has started at work, returned to work, remained in work, completed a plan or stopped participating before completion of an agreed plan.
Personal budget	An allocation of money directly paid to an individual for use on services to help them with their needs, to give them more choice about how their funding is spent. In the context of WorkWell, it would be expected that the personal budget is spent on work and/or health services.
Pilot area	The 15 ICB areas that were awarded funding to deliver WorkWell are referred to as pilot areas. Note such areas were previously called ‘vanguard partnerships’.
Primary care services	Primary care services provide the first point of contact in the healthcare system, acting as the ‘front door’ of the NHS. Primary care includes general practice (GP), pharmacy, dental and optician services.
Referral route	The way in which an individual is initially passed into WorkWell services from local partners in work and health or via self referral. The service will have multiple, clear referral routes for people both in work or who have recently fallen out of work.
Regional Programme Advisor	Regional Programme Advisors will work as part of the NSO to support pilot areas with their joined-up approach to work and health.
‘Return to work’ plan	Aimed at individuals who have recently fallen out of work due to a health condition. It will include clear objectives that address the service users’ biopsychosocial needs to support them to get back into suitable work.
Target beneficiaries	The target population for the service i.e., the individuals who are intended to receive support through WorkWell.
The Prospectus	National government guidance for Local System Partnerships of Integrated Care Boards (ICBs), local authorities (LAs) and local Jobcentre networks who wished to apply for funding to deliver WorkWell services. The document describes the background of WorkWell, expectations of Vanguard Partnerships and describes the support package for Vanguard Partnerships.
Third sector organisations	Organisations including charities, social enterprise and community groups which deliver essential services to support communities and deliver person-centred services.
‘Thrive in work’ plan	Aimed at individuals at risk of falling out of work due to a health condition. It will include clear objectives that address the service users’ biopsychosocial needs to support them to stay in work, or find alternative work (if their existing employment doesn’t accommodate their health needs).
Triage	Once an individual is referred to the service, some pilot areas will undertake an assessment of eligibility and suitability to decide whether an individual is appropriate for WorkWell. Once someone is deemed eligible and suitable, WorkWell may connect participants into the rest of the local work and health infrastructure through signposting and referral.
Work and health assessment	Evidence-based, person-centred, low intensity work and health assessments that support individuals with their low-level occupational health needs and to overcome barriers to work. Undertaken by a work and health coach.
Work and health coach	A stipulated role in each pilot area, the work and health coach will be the first point of contact (POC) for individuals receiving support through WorkWell. They will deliver one-to-one coaching.
Work and Health Strategy	Pilot areas developed work and health strategies, or built on existing ones where they are already in place, to support and drive a strategic approach to integrating work and health services at local level.

Question 4

Abbreviations

Accepted Answer

The following abbreviations are used in this report:

Abbreviation	Definition
CECAN	The Centre for the Evaluation of Complexity across the Nexus
DHSC	Department of Health and Social Care
DWP	Department for Work and Pensions
ICB	Integrated Care Board
JCP	Jobcentre Plus
JWHD	Joint Work and Health Directorate
LA	Local Authority
MDTs	Multidisciplinary teams
MH	Mental health
MI	Management information
MSK	Musculoskeletal
NHS	National Health Service
NSO	National Support Offer
NST	National Support Team
POC	Point of contact
ToC	Theory of Change
YHEC	York Health Economics Consortium, part of the University of York

Question 5

Executive Summary

Accepted Answer

Recognising that good health and employment outcomes cannot be achieved by local services acting in siloes, WorkWell aims to introduce a joined-up approach. This approach aims to bring together a range of organisations at a local and national level, to provide a bespoke work and health offer that meets the needs of people in local communities.

At an individual level, WorkWell will provide locally designed and delivered, low intensity work and health assessments, triage, and integrated referral. This will aim to get the available support to people early enough to help them stay and thrive in work, or move back into work quickly if they fall out. Central to WorkWell services will be an assessment of a person’s work and health needs. This produces a ‘return to work’ or ‘thrive in work plan’ with clear objectives that address the service users’ biopsychosocial needs.

The pilot of WorkWell is intended to be delivered through 15 Integrated Care Boards (ICBs) as leaders of local system partnerships, which may include Local Authorities and the local Jobcentre Plus, amongst others. Though there are some standard elements to WorkWell that all pilots must follow (such as the eligibility criteria, the numbers of individuals to be offered support and the requirement that each individual have a work and health assessment, access to a thrive in work or return to work plan, then some follow-up engagement), beyond this, the 15 pilots have some flexibility in how they deliver WorkWell, to suit their local populations (a ‘black box’ approach). A National Support Offer (NSO), including support from the National Support Team (NST), has been developed to support pilot areas in their delivery of WorkWell (as well as providing wider sharing of learning for non-pilot sites). JWHD commissioned IFF Research, York Health Economics Consortium (YHEC) at University of York, and CECAN Ltd. to assess the feasibility of evaluating the WorkWell pilots.

Objectives

This report assesses the feasibility of evaluating the WorkWell pilot programme. This programme aims to support up to 56,000 disabled people and people with health conditions get into and on at work. It links work and health and acts as a gateway into local services.

The report describes the evidence reviewed to arrive at this assessment of feasibility. The report then sets out the possible options for evaluation approaches and required data for these. This encompasses evaluations of the programme’s impacts, costs and benefits, implementation and local system integration using participatory systems mapping. Participatory systems mapping will capture how local systems work and factors affecting the delivery of desired outcomes.

Findings

This feasibility study has been written alongside the process of the 15 pilots developing their local offer. This has created opportunities for the evaluation (for instance, making it possible to embed standardised Management Information into the delivery of the pilots). However, it also creates some challenges in designing an evaluation. This is because there continues to be refinement about exactly how the individual pilots will be delivered in practice and how similar or different they will be, in terms of:

How referrals into WorkWell will take place
The likely balance in participants between those who are in-work and out-of-work (and relatedly the balance between those who will be existing benefit claimants and those who will not)
How challenging it will be to reach the target number of WorkWell participants (and hence whether or not there is a possibility that they will be under or over-capacity)
The exact nature of services that will be provided to people referred into WorkWell
The demographic profile of WorkWell participants

With the aim to have the outline of the evaluation in place by Autumn 2024, when the earliest pilots start to deliver services, it was important to carry out a Theory of Change (ToC). This will outline how the activities that WorkWell pilots are going to undertake will lead or contribute to the longer-term impacts it intends to have.

The ToC summarises the problem to be addressed, sets out the inputs, activities, outputs, and intended outcomes and impacts of the WorkWell programme. It also presents the underlying assumptions (i.e., necessary conditions for change).

Alongside this, a review of evidence from similar initiatives was undertaken to identify the evaluation approached being used in these programmes, outcomes achieved and methodological lessons learned.

The pilot was designed prior to delivery beginning in October 2024. It incorporates local flexibility as a core element, alongside evidence of uncertainty around referral routes, target populations and challenging mobilisation timescales for pilot sites. Therefore, it is unlikely that an RCT can be implemented in order to be able to evaluate the impact of the programme. However, useful information will be gathered to help inform the possibility of using such methods on similar programmes in the future. This is the context in which this report sets out proposals for the evaluation design.

Approach for a process evaluation

In order to understand how the programme has been implemented and delivered and how the pilots operate, we recommend carrying out a process evaluation. This will test and explore the mechanisms and assumptions in the Theory of Change (ToC). It will also explore how and why outcomes and impacts are achieved (or not); and help us understand which elements of the programme are more effective for which groups of people. It will explore whether the pilot has been implemented as intended, the challenges faced in delivery and how these have shaped delivery over time.

We suggest gathering the perspectives of the following audiences within the process evaluation:

Participants who’ve received support from WorkWell pilots, with data gathered from analysis of Management Information submitted by the pilots, longitudinal participant surveys and longitudinal qualitative interviews. These will, examine the demographic characteristics of WorkWell participants and those who disengage; explore the participant ‘journey’ and support experience and capture suggested improvements.
WorkWell coaches who support participants, via qualitative focus group discussions. This will explore how onward referral decisions are made, approaches to developing and following up on individual plans, the involvement of Multidisciplinary Teams (MDTs) and barriers and enablers to onward referral.
Employers, via qualitative discussions to examine employer experiences of engaging with WorkWell, their perceptions of its effectiveness and suggested improvements.
Senior stakeholders, via a ‘cost and resources’ survey and qualitative longitudinal interviews with the pilot partnerships and delivery partners. Qualitative interviews would include up to three partnership members, including at least one representative of the Integrated Care Board (ICB) and the Local Authority. The qualitative interviews would take place early in delivery, and again around 12 months on. This will, explore approaches to governance, experiences of resourcing WorkWell and WorkWell’s impacts on the ‘whole system’ of health, employment and wider community place-based services.

As each pilot is implementing WorkWell in its own way, the ideal would be to take a case study approach. This would mean collecting data from a mix of the above audiences, for all 15 pilots. This design may be adapted once delivery of WorkWell is further developed.

Approach for impact evaluation

The WorkWell pilots intend to achieve impacts both on individual participants and on local systems around employment and health. The evaluation of these needs to be approached separately.

Impacts on individual participants:

WorkWell seeks to support individuals with a health condition or disability who fall into one of two groups:

Out of work (looking to find work or economically inactive); and whose health condition is impacting their ability to find work. For these individuals, the primary outcome will be moving into good quality work^{[footnote 1]}.
Already working; and whose health condition or disability is impacting their ability to perform in their current job. For these individuals, the primary outcome will be sustaining appropriate employment (with their current or a different employer).

Both groups must be considered in the impact evaluation design.

When evaluating an intervention, there is the possibility that any outcomes and impacts observed among participants may have happened anyway, without the intervention. For example, some WorkWell participants may have received support with work and health anyway, in the absence of WorkWell. We therefore need to consider causality. This is the extent to which we can provide evidence that the observed outcomes and impacts are caused by the intervention, rather than by other factors.

We can do this by comparing the treatment group to a counterfactual group. The counterfactual group looks at the contribution the WorkWell pilots have made to the observed outcomes and impacts. It explores what would have happened in the absence of the WorkWell support. A counterfactual group needs to be as similar as possible to the treatment group who received the intervention. Ideally, the only difference would be that the treatment group accessed the pilot support, and the counterfactual group was not able to access the support. By comparing outcomes and impacts for the two otherwise similar groups, we can conclude that any outcomes or impacts observed in our treatment group but not in our counterfactual group, are attributable to the WorkWell pilots.

The most robust impact evaluation design is generally considered to be a Randomised Control Trial (RCT).This involves eligible individuals being randomly assigned to the treatment group (who receive WorkWell support) or a control group (who receive ‘business as usual’ services). However, an RCT was not possible to implement due to the limited knowledge of referral pathways and target populations, alongside challenging delivery timescales. Instead, we suggest impact can be estimated using a quasi-experimental design (QED) in which we identify a group of individuals who closely resemble WorkWell participants, as an indication of the outcomes that would have been achieved in the absence of WorkWell support.

The participant Management Information (MI) template that all pilots will complete, includes personal contact details and National Insurance Numbers (enabling participants to be identified in other datasets). It also includes information about their employment and health. This information could be used to identify a comparison group of similar individuals:

From within WorkWell pilot areas. However, eligible non-participants in pilot areas are likely to differ from WorkWell participants in ways that cannot easily be controlled for, which would bias the results of the evaluation.
From a synthetic control group that blends non-WorkWell areas in such a way that pre-WorkWell trends in relevant datasets match the pilot areas exactly. However, it is unlikely this approach would feasibly allow for disaggregation of results to look at impacts for each pilot individually, as this would require the formation of synthetic control groups for each pilot site and subgroup. This would lead to smaller sample sizes and therefore reduce statistical power. Additionally, forming synthetic controls for each pilot site and subgroup would be substantially time and resource intensive.
From within a group of comparison areas chosen as good matches for each of the individual WorkWell pilots, this is the most likely approach.

Relative changes over time in key measures will be compared for the treatment group (receiving support) and the counterfactual group (receiving ‘business as usual’ support). This will involve establishing measures at a baseline point (as soon as possible after joining WorkWell) and at follow-up points. These are suggested to be around 6 months later (by which point WorkWell activities are likely to have been completed) and 12 months (a recommended timeframe in studies of similar health and work interventions, included in the evidence review). The data required for this can be drawn from:

Secondary data; which will give access to larger sample sizes as, with some datasets, it should be possible to use all WorkWell participants and individuals matched to them. However, it will be possible to look at only a small subset of the outcomes in the Theory of Change. In addition, one of the datasets that can be accessed with a shorter time-lag, and that gives more scope to examine the impact of WorkWell on earnings, covers only benefit claimants (this is Real Time Information, ‘RTI’).
Survey data; which will involve smaller sample sizes (because it will not be possible to survey all WorkWell participants).However, it will be possible to define the outcomes collected by the survey, enabling coverage of more of the Theory of Change outcomes.

For the survey, the most robust source of a comparison sample for claimants (both those in and out of work) is likely to be DWP benefit records. Finding a potential sample source for a comparison group of non-claimants is more challenging. Taking account of the resources available for the evaluation, the only viable approach seems to be using an online panel provider to screen large numbers of panellists. This will identify a sample of individuals of working age who have health conditions and are not claiming benefits. Conversations with panel providers indicate that the maximum sample achievable would be 600 baseline and 300 12-month follow-up survey.

For participants and the comparison groups of DWP claimants, the best balance between cost and response rates is likely to be achieved by a sequential online and telephone approach, using contact details drawn from the participant MI. This will likely sample 16,000 WorkWell participants, with a starting sample of 5,333 per participant group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants. This sample size will enable volumes of 1,600 per group at baseline, 800 per group for the 6-month survey and 400 per group for the 12-month survey to be achieved (see Table 1.1 below).

Whilst, within the available resources, this distribution of surveys across the various participant and comparison groups would be the ideal in order to look separately at outcomes for those in work and those out of work, the make-up of actual WorkWell participants is as yet unknown. The survey sampling approach may need to adapt to this.

The analysis of this survey data will involve matching individuals across participant and counterfactual groups. This may affect final sample sizes available for analysis (i.e. if some individuals in the counterfactual group prove to be too different to individuals in the participant group). However, for the survey, matching can be conducted in two steps, with the first round of matching being carried out based on administrative data to identify the counterfactual group to survey. This should minimise any reductions in counterfactual group sample sizes available for analysis. However, it is unknown what proportion of the sample will be lost in this way until the matching is carried out.

A range of potential statistical approaches may be drawn on to calculate impact across the data sources and outcome measures discussed above. These may include regression analysis; difference-in-difference analysis; an interrupted time series analysis; and may involve propensity score matching. The statistical approach chosen will be informed by the data sources that it proves possible for the evaluation to access. Different approaches may need to be applied to the three levels at which impact evaluation is conducted (i.e. amongst all WorkWell participants; amongst WorkWell participants who are benefit claimants; and amongst WorkWell participants who complete the participant survey). The quantitative impact evaluation findings will also need to be triangulated with the qualitative process evaluation findings, to provide narrative and context for the impacts observed.

Further detail on secondary data sources considered for the impact evaluation, and on the survey design, is included in Chapter 7.

Impacts on local systems around employment and health:

Participatory Systems Mapping and Qualitative Comparative Analysis may be used to examine the complexity of the ‘whole system’ around employment and health, and how this influences outcomes and impacts achieved.

Participatory Systems Mapping brings stakeholders together in facilitated sessions to collaboratively develop a causal systems map. The mapping process encourages participant reflection on how the local system works and the factors affecting delivery of desired outcomes. The map generated is a snapshot of the system at a moment in time. By repeating the exercise, further mapping can explore whether and how the system is evolving, while encouraging further participant reflection on the challenges of achieving system change.

Qualitative Comparative Analysis is a structured method for case-based comparative analysis. It involves conceptualising ‘cases’ (for example, individual WorkWell pilots, or groups of pilot participants with similar characteristics), scoring each case for the presence or absence of factors of interest (e.g. pilot geographical characteristics, delivery approaches or systems change approaches) and then comparing how the ‘cases’ perform in achieving an outcome or impact of interest.

By examining the combinations of factors present when the outcome or impact of interest is achieved (or not), it’s possible to identify which factors are consistently present when a certain type of outcome or impact occurs. This helps us to assess how factors converge to potentially cause the outcome or impact – a ‘causal recipe’). This analysis will need to be conducted later in the study to allow time for the outcome or impact of interest to have occurred. Further detail is included in Chapter 7.

Approach for an economic evaluation

For this evaluation, costs associated with providing the WorkWell programme will be obtained from the quarterly expenditure schedule MI that pilots provide to JWHD. The total cost will be divided by numbers of participants to determine a cost per participant per pilot. The detail given in the quarterly expenditure schedule MI will enable analysis of where costs are being incurred. If the cost per participant varies widely between pilots, delivery plans could be evaluated to identify differences between pilots that may be driving this.

An additional ‘costs and resources’ survey could be used to supplement the expenditure schedule MI, amongst both the 15 pilots (capturing, for example, existing staff time contributed to WorkWell) and amongst central teams (capturing, for example, the costs associated with working with the 15 pilots).

Alongside this, the economic evaluation will incorporate:

Changes in earnings and benefits at different time points, using earnings information for benefit claimants from RTI, RAPID^{[footnote 2]} and, potentially, for non-claimants from the Annual Population Survey (counterfactual group only).
Differences in sick leave – monetising each day of sick leave by estimating the daily salary of individuals based on earnings from their job. This could be evaluated for the participant survey sample.
Self-reported health measures for the participant survey sample, by mapping self-reported health measures to utility values that can be converted to quality-adjusted life years (QALYs), using the Green Book value of £70,000 per QALY, found in The Green Book (2022) - GOV.UK.
Changes in healthcare resource use – monetised using publicly available cost from, for example, the Personal Social Services Research Unit (PSSRU) and the national cost collection.

Table 1.1. Sample for evaluation approaches

Participants	Initial qualitative interviews	60 (4 per site)
Participants	Follow-up qualitative interviews	25 (at least one per site)
Participants	Baseline survey	1,600 per group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants). 600 surveys with non-claimant comparison group of non-participants, using an online panel.
Participants	6-month survey	800 per group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants)
Participants	12-month survey	400 per group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants). 300 surveys with the same non-claimant comparison group of non-participants, using an online panel.
Participants	MI analysis	All
WorkWell coaches	Focus groups with WorkWell coaches	15 (up to 6 WorkWell coaches in each)
Employers	Focus groups with employers	15 (one per pilot)
Senior stakeholders	Initial in-depth interviews with senior stakeholders	15 (up to 3 stakeholders in each)
Senior stakeholders	12-month in-depth interviews with senior stakeholders	5 (up to 3 stakeholders in each)
Senior stakeholders	Costs and resource survey	15 (one per pilot)

Recommendations

This report highlights that the pilot, as designed prior to delivery beginning in October 2024, incorporates local flexibility as a core element, alongside evidence of uncertainty around referral routes, target populations and challenging mobilisation timescales for pilot sites. Therefore, it unlikely that an RCT can be implemented in order to be able to evaluate the impact of the programme. We therefore recommend using a quasi-experimental design (QED). In this approach, we identify a comparison group of individuals who closely resemble WorkWell participants, in order to show the outcomes that would have been achieved in WorkWell’s absence.

Within this, participant Management Information (MI) can be used to identify WorkWell participants in other datasets and to profile WorkWell participants (allowing us to identify similar individuals to form a comparison group). We recommend that our comparison group consist of a group of comparison areas chosen as good matches for each of the individual WorkWell pilots. This allows disaggregation of findings for individual pilots and reduces the risks associated with eligible non-participants in WorkWell areas being different from those who do participate.

The impact evaluation will examine relative change over time in key outcome measures for our treatment group (receiving WorkWell support) and our comparison group (receiving ‘business as usual’ services). This will be done by establishing measures at a baseline and at a follow-up point of 12 months. The data for this can be drawn from secondary data (including Real Time Information and RAPID) and survey data. We recommend that the impact evaluation draws on both secondary data (for larger sample sizes) and survey data (for wider coverage of our outcome measures).

We recommend that the impact evaluation draws on multiple secondary datasets, and is conducted at three levels:

Using all WorkWell participants (so as to evaluate employment status, earnings, benefit caseload and sick leave from RAPID and the Annual Population Survey for participants and the comparison group respectively)
Using WorkWell participants who are benefit claimants (so as to evaluate employment status, earnings, benefit caseload and spend, and sick leave from Real Time Information)
and using WorkWell participants (and comparison group) who complete the survey (so as to cover a range of other outcome measures, including self-reported health and resource use).

To address the complexity of the ‘whole system’ around employment and health, and how this influences outcomes and impacts achieved, ideally the evaluation would include:

Participatory Systems Mapping (to reflect on the mechanisms of change from a system perspective and explore how systems change over time)
and Qualitative Comparative Analysis (to identify the combinations of factors, or ‘causal recipes’, accounting for success, or the lack of it, in different WorkWell areas)

We strongly recommend that an economic evaluation be included within the evaluation design, examining costs by using the quarterly expenditure schedule MI that pilots will provide to the JWHD; combined with a ‘cost and resources’ survey. Alongside this, examining and monetising, for example, changes in earnings and benefits; differences in sick leave; changes in self-reported health; and changes in healthcare resource use.

We also strongly recommend that a process evaluation be included within the evaluation design. This would ensure that qualitative insight into the delivery and experiences of those involved is captured. The ideal would be to adopt a case study approach, collecting data from a mix of the above audiences, for all 15 pilots. This would give us scope to provide qualitative findings for each pilot individually, thus helping provide narrative and context for the scale of impacts observed in each pilot. This may prove important, if the 15 pilots adopt individual delivery approaches, further detail is given in Chapter 6.

Question 6

1.  Introduction to WorkWell

Accepted Answer

This chapter introduces the WorkWell programme. It discusses how the programme is to be delivered, the information that the local pilots are expected to share, and the National Support Offer (NSO) that has been developed to support the pilots.

Background

What is the WorkWell programme?

The WorkWell programme aims to support up to 56,000 disabled people and those with health conditions by linking work and health, acting as a gateway into local services. Recognising that good health and employment outcomes cannot be achieved by local services acting in silos, WorkWell introduces a joined-up approach that brings together various organisations at local and national levels to provide a bespoke work and health offer tailored to local communities. At an individual level, WorkWell provides locally designed and delivered, low-intensity work and health assessments, triage, and integrated referrals. This gets support to people early, helping them stay and thrive in work or quickly return if they fall out.

Central to WorkWell services is an assessment of a person’s work and health needs, producing a ‘return to work’ or ‘thrive in work plan’ with clear objectives addressing the service users’ biopsychosocial needs. The WorkWell prospectus (WorkWell prospectus: guidance for Local System Partnerships - GOV.UK) was developed to provide guidance for Local System Partnerships of local Integrated Care Boards (ICBs) who wished to enter a competitive tender process to become one of the 15 pilot sites who would receive grant funding to deliver WorkWell services in their area. Integrated Care Boards (ICBs) leading local system partnerships, are locally driven by a partnership of organisations across work and health, varying by pilot area in services offered, referral pathways, and delivery of mandated roles. The learnings from this pilot aim to inform future rollout decisions to more ICB geographies across England.

How will the WorkWell programme be delivered?

There are some standard elements to WorkWell that pilots must follow. These are:

Eligibility: Delivery to certain groups based on broad eligibility criteria:
- Anyone with a disability or health condition who needs support to remain in work, needs support managing a health condition to return to work from sickness absence, or needs support to start work
- Whose home address or address of their GP/ local Jobcentre falls within the pilot area
Volume: The number of individuals offered support, based on the pilot’s estimates of how many people they could deliver to
Type of support:
- A work and health assessment delivered by a work and health coach
- Access to a thrive in work or return to work plan after the work and health assessment
- Some level of follow-up engagement with the service user after the plan is developed
Roles: A work and health coach that sits within a multi-disciplinary team (MDT)
Governance: A Work and Health Strategy produced by pilot areas as part of their funding agreement
Sharing learning: Engagement with the National Support Offer to share best practice across pilot and non-pilot areas

Though funding will be delivered to ICBs, they are expected to co-produce WorkWell services with lead partners, including other ICBs, Local Authorities, and Jobcentres, to join up services. The 15 pilot areas have flexibility in delivering WorkWell to suit local needs, with no stipulated length of engagement or follow-up. Learnings from the pilot will identify effective models. Pilot sites believe the services they will be delivering through WorkWell are new or additional, providing support participants would not otherwise receive. Some existing services may be expanded, and individuals may be referred to existing provisions after initial support, enhancing linkages between local employment and health support. Outcomes will likely result from both initial and signposted support.

All 15 pilot areas had submitted their interim delivery plans to the Joint Work and Health Directorate (JWHD), expanding on their original bids for funding to demonstrate progress. This plan includes information about key project timelines, the governance structures established to manage the local WorkWell service, their data and information management strategy, recruitment plans, details of the support and intended outcomes for individuals, and referral routes into such services. They were expected to submit a final delivery plan by the end of August and to be ready to go live with WorkWell on 1 October 2024.

To support the national evaluation, pilot areas are required to submit participant management information (MI) to the JWHD monthly, and performance/grant MI on a quarterly basis. The evaluation and performance/grant MI will be collected at the individual participant and pilot area level (see Table 1.2) to measure the impact of WorkWell.

Table 1.2: MI collected at the individual participant and pilot area level

Individual participant – submitted monthly	Personal information and demographics
Individual participant – submitted monthly	Information at first appointment, including current employment status and barriers to work
Individual participant – submitted monthly	Outcome information
Pilot area – submitted quarterly	Governance
Pilot area – submitted quarterly	A breakdown of expenditure costs
Pilot area – submitted quarterly	Referral routes
Pilot area – submitted quarterly	Starts, interventions and outcomes
Pilot area – submitted quarterly	Satisfaction scores

How will pilots be supported?

A National Support Offer (NSO) has been developed to support pilot areas in their delivery of WorkWell, providing support at the national, regional and local level. This involves:

The National Support Team (NST), named WorkWell Together, identifies support needs across the pilot sites and shares learning to develop policy guidelines
The Regional Programme Advisors, work with existing work and health leads to enhance capacity and support cross-system integration, particularly through WorkWell, but also on other key work and health programmes
A Learning and Change Manager in each local pilot site is responsible for raising awareness, engaging stakeholders, and promoting cultural change towards work and health integration

The NSO aims to share real-time pilot progress information and address delivery gaps to support the success of WorkWell. It is intended that non-pilot sites will benefit from the learning shared across the NSO, so that they see the benefits of the joined-up approach to work and health and feel encouraged to design services in a similar way in their area. For instance, the NST will develop a toolbox of shared learning which non-pilot sites can benefit from.

Evaluation Objectives

The aim of the proposed evaluation is to provide the JWHD with evidence on the delivery and impact of the WorkWell pilots, including whether they represent value for money. The evidence generated from the evaluation will help understand the case for any wider rollout of the WorkWell programme. It will develop the evidence base on light touch occupational health therapies and programmes, particularly non-clinical interventions; and will explore how WorkWell has affected joint working across organisational boundaries on work and health service integration.

The evaluation will consist of three strands:

Impact evaluation: Impact evaluation will assess the changes that have been brought about by the WorkWell Pilots, over and above what would have happened anyway
Process evaluation: The process evaluation will build an in-depth understanding of how the pilots are operating on the ground, how they are achieving impact, the challenges they have faced and how these have shaped delivery over time
Economic evaluation: The aim of the economic evaluation will be to attach monetary values to the impacts that have been achieved, to help the JWHD determine whether the pilots offer value for money

In addressing these requirements, the evaluation is likely to explore the following research questions:

How do the WorkWell pilots affect work, health, and wellbeing outcomes at the individual beneficiary level? 
What mechanisms at the local level lead to successful employment and/or health outcomes for the individual, and to successful implementation and delivery of the WorkWell pilots?  
How do local partnerships work to deliver their pilot? What impact does delivering the WorkWell pilot through a local partnership have on work, health and wellbeing outcomes?
What are the main costs and benefits associated with the implementation and delivery of WorkWell? Are these as expected?

Methodology

In order to meet the feasibility study aims, and answer the above research questions, IFF Research, York Health Economics Consortium (YHEC) at University of York, and CECAN Ltd. undertook the following activities:

Systematic evidence review.
Stakeholder interviews amongst both Joint Work and Health Directorate colleagues and representatives of the 15 pilot sites.
A Theory of Change (ToC) workshop with IFF Research, YHEC, CECAN Ltd., and Joint Work and Health Directorate stakeholders.

Alongside this, IFF Research reviewed the pilot documentation that underpins WorkWell, including the grant applications for the 15 pilots. They also reviewed documentation setting out the Joint Work and Health Directorate’s initial thinking about evaluation design and draft templates for Management Information (MI) to be shared regularly by the 15 pilots.

IFF Research conducted in-depth interviews with key stakeholders to address knowledge gaps remaining after the evidence review, and to explore stakeholder views on potential evaluation approaches.

A total of 20 in-depth interviews were conducted with stakeholders, as follows:

Table 1.3 Sample breakdown of stakeholder interviews

Audience group	Number of interviews	Example roles of participants
Joint Work and Health Directorate (DWP / DHSC) policy team staff	6	Senior Responsible Officer for the WorkWell Programme, WorkWell and Local Systems Team, WorkWell Policy
Stakeholders from ICBs	15 discussions covering 40 individuals	Deputy Chief Officer, Executive / Associate Director, Operational Manager, (Senior) Project Manager, (Senior) Development Manager, Strategic Lead

Semi-structured interviews were conducted around discussion guides – one for Joint Work and Health Directorate policy team staff and one for pilot stakeholders – with the specific coverage in each interview adapted to individual participants’ areas of expertise. Discussions explored:

The background to and rationale for the WorkWell pilots: What is the problem that the WorkWell pilots are trying to solve? Who are its intended beneficiaries?
Intended outcomes: What are the aims of the WorkWell programme? What are the aims of the pilots locally, and the main mechanisms through which these aims will be achieved?
WorkWell pilots’ implementation plans: What is being delivered by each pilot, and how? What organisations are involved? What assumptions is delivery based on? How ‘ready’ for delivery do pilots feel? What other local initiatives or interventions may interact with the pilots? To what extent are there commonalities and differences in approach between individual pilots?
Potential evaluation approaches and data collection/availability: What should the mainstage evaluation look to achieve? What possible issues might arise when conducting the mainstage evaluation, and how could these be addressed? Exploring thoughts on specific metrics to be collected, including details of data to be captured locally, approaches to exploring whole system change in the local health and work landscape around the pilot, and considerations for the Value for Money analysis.

Question 7

2.  Theory of Change for WorkWell

Accepted Answer

This chapter presents the Theory of Change (ToC) for WorkWell, accompanied by a narrative description.

Introducing the ToC

A Theory of Change (ToC) outlines how a programme’s activities will lead or contribute to the longer-term impacts it intends to have. It summarises the problem to be addressed, sets out the inputs, activities, outputs, and intended outcomes and impacts of a programme. It also presents the underlying assumptions (i.e., necessary conditions for change).

Theory of Change workshop

The ToC workshop was a forum for IFF Research, YHEC, CECAN Ltd., and Joint Work and Health Directorate stakeholders to collaborate on the further refinement of this draft ToC. In the session, the problem statement, ToC and assumptions underpinning the programme were presented to workshop participants. Participants then worked through each element of the ToC (problem statement, inputs, activities, outputs, outcomes, impacts, assumptions) and discussed whether and how each of these should be refined. This collaborative process aimed to ensure that the ToC accurately reflected how stakeholders view the WorkWell programme and its intended aims, and ultimately what the national evaluation should focus on.

Figure 1 Structure of a Theory of Change

The Joint Work and Health Directorate (JWHD) developed a draft ToC model prior to the evaluation feasibility study being commissioned. IFF Research refined this draft in light of findings from the document review and stakeholder discussions. This ToC was then discussed and further refined during a ToC workshop on 25 July 2024.

For the purposes of the WorkWell Pilot, the ToC has six elements:

The inputs and resources that are required to deliver the programme
The core activities that are carried out with those resources
The outputs that are the products that turn the activities into outcomes
The outcomes (short and mid-term changes resulting from the activities)
The impact of the programme and the ultimate effects of the combined outcomes

The components that make up each element of the ToC form a crucial step in designing the evaluation approach. Ultimately, the ToC shapes what we explore through the evaluation to allow us to demonstrate the programme’s value and helps us to interpret what we learn from the evaluation. The ToC will be reviewed and monitored throughout the evaluation and an updated version will be incorporated into the final evaluation report. Figure 2.1 presents the final Theory of Change for the WorkWell Pilot, with a more detailed narrative provided in Appendix 1.

Figure 2.1 Theory of change for the WorkWell Pilot

Question 8

3.  Evaluations of similar initiatives

Accepted Answer

This chapter presents the findings from the evidence review of evaluations of similar initiatives. It discusses the types of programmes delivered and outcomes achieved, the evaluation approaches used for these programmes, and any methodological lessons learned. Methods used to identify and analyse the evidence were in line with Government Social Research guidelines on conducting Rapid Evidence Assessments (Rapid evidence assessment toolkit - Civil Service).

Method

Eligibility criteria

Studies meeting the following eligibility criteria were eligible for this review:

Population: Adults over the age of sixteen living with a disability or long-term health condition who need support to get in or get on at work. Studies were eligible whether or not the participants were claiming benefits.

Interventions: Integrated, locally-delivered programmes designed to offer holistic support to people with a disability or long-term health condition who are in work, returning to work or absent from work due to sickness. Holistic support may include a combination of referral and signposting of services, work coaching, low-intensity occupational health support, multi-disciplinary team (MDT) support, employer liaison or managing biopsychosocial barriers to work.

Comparators: The programmes were compared to either no intervention, or to other interventions designed to support those returning to, or remaining in, work.

Outcomes: Studies reporting measurement of impact of the following outcomes were eligible:

Employment, including employment retention
Changes to sickness and absence
Changes in productivity (e.g. return-to-work, or moving towards return-to-work)
Improved participant health and wellbeing
Changes to economic inactivity
Changes in healthcare service use (e.g. less requirement for healthcare visits, or the need for further interventions)

Limits: Case reports, pre-prints, editorials and news items were excluded. Only studies reported in English were eligible

Identification of studies

Pragmatic, targeted searches were undertaken to identify evidence on integrated programmes to support people with a disability or long-term health condition to return to or remain in work. Peer reviewed literature was sought via the Ovid MEDLINE, Ovid Social Policy and Practice and King’s Fund Library databases. We also searched key websites to identify grey literature. The primary search strategy for Ovid MEDLINE can be found in Appendix 3.

Search results were downloaded and de-duplicated in EndNote. The results were uploaded to the Covidence software package for screening by a single reviewer. The title, abstracts and full texts were reviewed in stages and included if they met the eligibility criteria. The evidence review flow diagram can be found in Appendix 4. Electronic or paper copies of potentially relevant full papers, meeting the eligibility criteria, were obtained using both local access routes and the University of York subscription services.

Data extraction

To meet project resources and timescales, data extraction was prioritised by only extracting data from studies published since 2014. Data extraction was high level and limited to the key data of relevance.

Key data for each relevant study was extracted into a searchable Excel template. The data to be extracted was in a narrative format. The following elements were extracted from the eligible studies:

Bibliographic details: (authors, title, journal, and year of publication)
Study country and geographic area (if available): (setting)
Study design
Study objective
Description of the programme or intervention
Programme duration
Participant employment status at study start
Methods used to evaluate programmes
Measurement of programmes including data sources
Outcomes, including impact/effect size
Lessons learnt

Studies included

1,426 records were retrieved from the database searches and 8 from other sources. 58 duplicates were removed and 1,376 references were screened for inclusion. 54 studies met the eligibility criteria, and 31 studies were de-prioritised for data extraction because they were published prior to 2014 (i.e. more than 10 years old). 19 studies reported in 23 papers were included in the evidence review (again, see Appendix 4 for a flow diagram showing the identification and screening process).

Eight of the 19 studies were randomised controlled trials, four were systematic reviews, three were process evaluations, two were prospective cohort studies. There was one non-randomised controlled study and one narrative review of case studies. The geographic distribution of the studies can be found in Table 3.1.

Studies took place in Belgium, Denmark (Silkeborg and Copenhagen), Norway (Inlandet and Trondelag), Sweden (Uppsala), Switzerland (Geneva) and the United Kingdom (London, Hampshire and Greater Manchester). While studies outside of the UK may have differences in local systems that make them less relevant, there is still useful learning that can be obtained from them.

Table 3.1: Geographical distribution of the studies

Study name/number	Setting
Aili 2022 [1]	Sweden
Berglund 2018 [2]	Sweden, Uppsala
Bernaers 2023 [3]	Multiple (systematic review)
Brendbekken 2017 [4]	Norway, Inlandet
Centre for Regional and Economic Knowledge 2022 [6]	UK, Greater Manchester
Cullen 2018 [7]	Multiple (systematic review)
Dudley 2016 [8]	UK, London and Hampshire
Fisker 2022 [9]	Denmark, Copenhagen
Gloster 2018 [10]	UK
Ibrahim 2019 [11]	Switzerland, Geneva
Langagergaard 2021 [12]	Denmark, Silkeborg
Learning and Work Institute 2021 [15]	UK, London
Moll 2018 [16]	Denmark, Silkeborg
Momsen 2016 [17]	Denmark
Pedersen 2018 [18]	Denmark, Silkeborg
Sabariego 2018 [20]	Europe (systematic review)
Schepens 2024 [21]	Belgium
Skagseth 2020 [22]	Norway, Trondelag
Vogel 2017 [23]	Multiple (systematic review)

Eight studies ([1], [4], [10], [12], [16], [19], [21], [22]) had participants who were already in work but on sick leave from their place of employment. Five studies ([2], [6], [9], [11], [17]) included both participants who were on sick leave from their place of employment plus those who were unemployed with a long-term health condition or disability. One study only included those who were unemployed but with a long-term health condition [15].

Summary of Findings

What type of programmes have been delivered to date? What innovations are taking place?

The programmes were multidisciplinary and involved a range of professionals delivering services to people on long-term sick leave due to musculoskeletal disorders, chronic pain or mental health issues.

The programmes varied in duration, from three to four weeks ([11], [22]) to twelve months [2]. One programme [18] had an indefinite duration, and provided support until return-to-work was achieved or the participant was signed off work permanently. Three to four months was the most frequently reported duration.

Interventions provided by the programmes included training on coping strategies [1], cognitive behavioural therapy ([7], [21]) relaxation and body-awareness training [1], pain management [17] and physical and occupational therapy ([1], [7], [16], [17], [21]).

Most of the programmes also included provision of an individually tailored return-to-work plan ([1], [2], [3], [4], [6], [8], [10], [16], [18], [22], [23]).

Some interventions were provided in the workplace. These included meetings between the patient, manager or employer and a member of the rehabilitation team ([1], [12], [18], [22], [23]), or changes to the workplace conditions, environment or equipment ([3], [7], [21], [23]) or counselling on return to work [3].

Service development interventions, designed to better coordinate the delivery of, and access to, services to assist return-to-work within and involving the workplace were reported by one systematic review [7]. Service coordination involves attempts to improve communication within the workplace or between the workplace and the healthcare providers. Examples include development of return-to-work plans, case management and education and training.

Innovations included:

One UK-based study [6] attempted a recruitment pathway through small-and-medium sized enterprises (SMEs) who may not have access to occupational health services. The outcomes were limited however, as the number of referrals via this pathway fell well below target. Recommendations to improve this were to design and commission a provision based on evidence of SME diversity, needs and demand. Engagement with employers and establishing partnerships with local authority business support schemes were seen as crucial to success.
Demand and Ability Protocol (DAP). One study reported on the deployment of the Demand and Ability Protocol, an intervention embedded in a return-to-work programme in Sweden [1]. The DAP is an intervention based on the Dutch Functional Ability List and knowledge about disability in working life. It is linked to the International Classification of Functioning, Disability and Health (ICF). The DAP intervention aimed to help patients and managers assess the patient’s workability and make adjustments to promote a successful return to work.
Two studies ([2], [22]) reported on the integration of Acceptance and Commitment Therapy (ACT) into a return-to-work programme. ACT is psychological therapy that uses cognitive behavioural therapy (CBT) and acceptance and mindfulness strategies, together with behavioural strategies, to increase function and quality of life.
One study [4] reported on the use of a novel educational tool, the Interdisciplinary Structured Interview and a Visual Educational Tool (ISIVET), to establish an overall picture of the patient’s situation through visualization. The underlying hypothesis was that this design could introduce a new cognitive approach to cope with health problems. This might strengthen the motivation of patients to go through with changes, thereby improving the actual coping and resuming work.
One study [21] reported on a multidisciplinary biopsychosocial rehabilitation (MBR) programme, provided in thirty-six sessions. This combines education and physiotherapy, with different forms of cognitive-behavioural psychology to address participants’ unhelpful beliefs about their pain, reduce ‘fear-avoidance’ behaviours and catastrophic thinking and improve mood. This decreases disability and improves function. Treatments provided by this programme included: functional and psychosocial evaluation (including questionnaires), information/education (biopsychosocial influencing factors), ergonomics (including work-related adjustments if applicable) and an individualized exercise program.

How can improvements in outcomes be achieved?

Key findings in terms of improving outcomes for participants included:

Actively involving workplaces in providing return-to-work interventions may offer more positive outcomes.
Providing individually tailored workplans improves the success of return-to-work interventions.
An employee’s current physical and mental health, as well as change in health over time, was consistently significantly associated with employment status and willingness to engage.
Those with stronger job relations (i.e. job security and job satisfaction) tended to have better outcomes in terms of return-to-work and the speed of return-to-work than those with weaker job relations.
Addressing physical issues such as mobility may have an impact on employment outcomes. This could be done by either embedding intensive clinical provision into the programme or by having clear pathways to referral.

Key findings in terms of service delivery and partnerships included:

Creating and sustaining multi-agency working partnerships is an important part of devolving responsibility for employment to the local level, allowing for integrated and holistic ways of working.
Co-ordinated design and delivery of health and employment support can be an effective means of producing employment outcomes: often more so than delivering such interventions separately.
Co-location of services may be beneficial where feasible. This encourages communication, networking between services and awareness for referral. One recommendation was to embed programme provision and/or caseworkers into directly into GP services.
Use and quality of personal advisors and individual case managers made a significant difference to the likelihood of securing an employment outcome.
Keeping referral numbers at a high level was seen as a factor in keeping local services engaged.
Where possible, a single-tiered approach to support was most effective. If a single-tiered approach is not feasible, multi-tiered models should be flexible to the needs of the client and be client-led.
Initial assessment processes should be designed to identify client needs at the earliest possible stage, to enable quicker triage into appropriate support.
Targets should be feasible and subject to external scrutiny to ensure that they are realistic.
There is a need for the referral process to be formal, rather than just signposting.
One barrier in terms of partnership working was staff turnover within services, as it may take time to re-establish relationships and identify the key personnel that can provide support. This was seen as a crucial limitation in one of the programmes [14] as it meant that referrals to the programme were low.
For those in work, referrals may come through employers. One recommendation given was to involve representative bodies such as the Chamber of Commerce and Federation of Small Business as potential delivery partners to increase reach into, and engagement from, the business community.

Key findings in terms of service devolution included:

Service devolution outcomes were only explored in two studies.
One study [6] emphasised the importance of aligning the geographical boundaries of services (e.g. to ensure that there was consistency with the primary care structures and boundaries).
For local devolution, time, resource and expertise need to be invested in developing governance structures and relationships with key health partners of appropriate seniority. Integrating programmes within the health system requires buy-in from public health teams, health commissioners and practitioners.
There was a need to design provision around the needs and ecosystem of support and delivery within individual localities at a granular level.
Local Integration Boards (LIBs) were seen to play a useful role in fostering networks; increasing mutual understanding of organisational practices; facilitating referrals; and identifying signposting opportunities.
One study [14] found that a limitation of the programme was that there may be competition from other local services which may impact on referrals.
Engaging local authorities was seen as crucial to success. This provides opportunities to raise awareness of local provision and provides numerous communication channels to raise awareness of the programme. Where caseworkers were linked in with existing local employment programmes, they could find out more about local services and job opportunities that were available.

What methods and measures are being used to assess the impact of programmes?

Nine of the studies assessed used a control group to assess the impact of programmes, where the control group received either no intervention, or a modified, brief version of the intervention. Control groups were generally recruited at the same time as the intervention groups. Both groups were most often referred to the service by a GP ([4], [9], [12], [16], [19], [21], [22]) or job centre service ([9], [13], [17]) but some were also identified via registry data ([2], [22]). The participants were then assigned to intervention or control by the researchers.

Five studies were reviews: four were systematic and one was a narrative review. These generally also included controlled studies which used a control group to measure the effectiveness of the programme.

Three were process evaluations, one of these (Working Well Early Health [6]) reported use of a counterfactual from the Labour Force Survey Five Quarters Longitudinal Panel. This estimates the extent to which outcomes would have been achieved without the programme. It also estimates how important the interventions were to outcomes over and above the influence of other factors, interventions, or changes. The other two process evaluations did not report use of a counterfactual.

The Labour Force Survey Five Quarters Longitudinal Panel counterfactual was used to identify respondents who had recently fallen out of work for a health reason (within the last 6 months). It also identified those who were in work and at risk of falling out due to a health reason, with a limiting condition that was likely to keep them out of work for 12 months. The main concern with this approach was that the latter group may not have been similar enough to the programme participants. A matching was undertaken between the counterfactual and programme participants: age, gender, ethnicity, and highest education were matched to find the “nearest neighbour”. But there was concern that there was the possibility that some important but unobservable variables were not taken into account in this matching. There was also a very restricted number in the counterfactual (93) that could be matched against the 983 participants in the programme, which was a major limitation.

Two studies were cohort studies, which measured return-to-work and data before and after the intervention.

All the studies measured employment data including: return-to-work, duration of sick leave, amount of sick leave.

Only one study looked at changes to healthcare utilisation (GP visits, outpatient visits, emergency department visits).

Studies also considered health-related outcomes to measure the impact of the programme. There was a variety of outcomes measured in this category, including pain, depression and anxiety levels, disability levels and quality of life.

Baseline data collected also included gender, ethnicity, age and levels of educational attainment.

What sources of data have been used to measure impacts of programmes?

A range of data sources have been used. These include:

Self-reported sickness/absence by study participants
Organisational records including: duration of sick leave, work status and recurrence of sick leave and absence
Registry data from national registers on benefits claimed or insurance claims
Work Ability Index (WAI) – where participants self-reported their ability to work on a scale compared to their lifetime best ability
Work Participation Score (WPS), defined as a fraction with number of weeks working as the numerator, and number of weeks receiving social transfer payments plus number of weeks working as the denominator. The score is between 0 and 100. A higher score represents a higher work participation
Patient-reported outcome measures for health and well-being, including:
- Hospital anxiety and depression scale (HAD)
- EQ5D
- The General Self-Efficacy Scale (GSE)
- SF-36
- The Lower-Back Pain Self-Rated Scale
- Fear-Avoidance Beliefs Questionnaire
- The Örebro Musculoskeletal Pain Questionnaire (ÖMPQ)
- Oswestry Disability Index (ODI)
- The Tampa Scale for Kinesiophobia (TSK)
- The Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS)
- ONS Life Satisfaction
- The Patient Activation Measure (PAM) (helps to measure the spectrum of skills)
- GAD7 (an assessment of Generalised Anxiety Disorder)
- PHQ9 (used to monitor the severity of depression and response to treatment)
- MSK-HQ (Musculoskeletal Health Questionnaire)
The tools reporting patient-reported outcome measures were varied, and dependent on the type of participant and their reasons for sickness absence. There was no indication that any were better than others in detecting impacts. The most frequently reported were: EQ5D, hospital anxiety and depression scale (HAD), Oswestry Disability Index (ODI) and The Tampa Scale for Kinesiophobia (TSK)

What outcomes have been observed?

The included studies reported mixed results on return-to-work outcomes.

The only study which reported using a counterfactual [6] reported 38% in work after 26 weeks, as opposed to 17% of the counterfactual. One study [10] reported 65% of participants in work after a 3 month programme, but this study did not report a counterfactual for comparison. One study [11] reported 66% of participants had recovered their ability to work, this study combined physical, occupational and psychological counselling and lasted for four weeks. However, this study also reported a very high number of drop-outs with 47% of participants lost to follow-up. One of the studies from the UK [15] reported that 13% of the participants were in work after 6 months, with age being the greatest barrier to finding sustained work. In this study, over half of participants were over 50, but only 9% of these gained sustained employment at 6 months.

In general, there was no obvious link between programme duration and the impact of the interventions. However, one of the systematic reviews [23] reported on programme duration and found that programmes that supported participants indefinitely until return-to-work had positive results. Shorter programmes (3 weeks duration) had less positive results. However, even here results were mixed. The only programme which individually tailored the length of the programme to the needs of the participant did not achieve positive results in return-to-work or sustaining work. However, some shorter programmes (3 to 4 months) did have positive outcomes.

Four studies reported a positive difference in employment outcomes for multidisciplinary return-to-work programmes: including fewer missed workdays and fewer sick leave periods ([2], [3], [7], [10]). One study found that a multidisciplinary return-to-work programme helped people return to work faster [4].

Three studies reported no difference between participants on integrated, multidisciplinary return-to-work programmes compared to a brief intervention or no intervention ([1], [9], [12]). One study found that the programmes did not decrease the time to return-to-work [3]. A systematic review [23] found that there was no impact of return-to-work programmes on cumulative sickness absence, the proportion of participants at work at end of the follow-up, or the proportion of participants who had ever returned to work at short-term, long-term or very long-term follow-up.

One systematic review [3] found that adding interventions in the workplace to a multidisciplinary programme had a beneficial effect. Actively involving the workplace also seemed to have a beneficial effect in a further study [4]. Although one study found that a workplace intervention consisting of an intervention where a meeting was held between employer, participant and a member of the rehabilitation team made no difference [22].

A systematic review [20] reported three studies with positive return-to-work employment outcomes, and two with no difference.

The only study to measure healthcare utilisation [17] found that a multidisciplinary programme which referred participants to specialists depending on need, did not lead to any change in utilisation of services including GP visits, outpatient visits and visits to emergency departments.

In terms of wellbeing outcomes for participants, several studies reported a positive difference in patient-reported outcomes, including anxiety ([1], [4]), depression ([1], [4]) and sleep [1]. A systematic review [23] reported a small positive improvement in patient-reported outcomes. An evaluation of the Working Well Early Help programme [6.] found it was more effective in supporting health and wellbeing than employment outcomes.

Those in work but on sick leave when they entered the studies appeared to have better return-to-work outcomes than those who were unemployed with a long term health condition, however this was only compared and reported in two studies. One study [6] reported better health and wellbeing outcomes for those referred via the GP who were in work but on medical leave than unemployed individuals referred via job centres. A further study [17] reported lower return-to-work amongst those who were unemployed at baseline compared with those who were employed but on medical leave. The strength of job relations (weak or strong) in terms of perceived job security was also a factor in one of the studies [19]. Participants with strong job relations had better outcomes than those with weaker job relations.

Are there any key methodological lessons learnt to inform evaluation of these types of programmes?

A number of methodological lessons learnt were reported:

Many of the studies had low numbers of participants and referrals, this was often attributed to lack of awareness of the programmes by those who had the ability to refer potential participants.
High rates of drop-out were common, either because of continuing health issues or issues with workplace relations. The number of participants dropping out varied, from 25% [9] to 47% [11].
Work functioning and cost data were not collected as often as data associated with lost time. One of the studies recommended that all three should be collected in order to get a complete picture of programme effectiveness [8].
Studies should have follow-up of at least twelve months in order to measure programme effectiveness.
Some studies found that there was insufficient follow-up of employee outcomes in the workplace.
Definitions need to be clearly stated: i.e. what constitutes long-term sickness and absence.
Collecting registry data can be problematic if there are changes in legislation which affect outcomes (for example, a Swedish study found that legal changes to the number of days of unemployment required before collecting benefits midway through the study had an impact on the registry data that could be collected [18]).
Account should be taken of confounding factors which may affect programme success, such as educational attainment levels of participants, or complex needs. One study found that those referred through job centres tended to have more complex needs than those referred via GP services. Differing programme success rates were observed as a result [10]. Complex needs were variously defined across the studies but included: older people with caring responsibilities and older people who may perceive age discrimination in job seeking/retention, people who were not as in-touch with referring services such as job centres, people with co-morbidities who may need more than one assessment from different specialties, people with complex social barriers (poor or insecure housing, low incomes or financial problems such as debt). These complex needs could be exacerbated by the length of time the individual had spent out of work and not having a foothold in work.

Are there any examples of whole system change in coordinating activity in the work and health landscape?

Only one study (on Working Well Early Help [6]) reported on systems change. Systems change proved to be overambitious for this programme; a single programme failed to influence whole systems change in a complex local health system. However, there was some tentative evidence of systems change with GPs in one area of Greater Manchester, which was not sustainable due to GP workloads and the time-limited nature of the programme.

Key lessons learned

Multidisciplinary programmes featuring a range of professional support for people with long-term illness or disabilities had mixed results in terms of employment and health outcomes.

Key findings from the evaluations appear to be that involving workplaces, providing individualised support via return-to-work plans and co-ordinating and sustaining multi-agency partnerships may be key to achieving successful employment outcomes. Strength of job security, job satisfaction and physical mobility may also be factors in the success of these programmes.

Data used to measure impact varied, but use of a control group was common. Data collected during evaluations included return-to-work data such as amount and duration of sick leave. Many studies also collected a variety of evidence on health-related outcomes, including pain, depression and anxiety levels, disability levels and quality of life.

Several studies reported positive health-related outcomes for participants undergoing the programmes. There were some key learning points regarding evaluation methods:

Drop-out rates can be substantial, in some case up to 47%
Poor rates of referral to services were common
Future studies need adequate follow-up (at least twelve months)

Some barriers that were non-health related were reported. These included perceived age discrimination, and individuals with complex social barriers including insecure housing, financial debt and low incomes. Studies which reported on these barriers emphasised that the delivery model should be individualised to client needs and flexible to meet these challenges [13].

Question 9

4.  The WorkWell Pilots

Accepted Answer

This chapter provides detail on the local WorkWell pilots. It describes their target beneficiaries, the geography of the local pilots, the stakeholders collaborating in delivering them, and their delivery models. It also explores the existing and prior programmes that provide the local context for the pilots and their approaches to whole systems change. Information referenced in this chapter is based on delivery plans and interviews conducted in Summer 2024, therefore some details may have since changed.

Target beneficiaries

Definition of target beneficiaries

As defined in the WorkWell Prospectus (WorkWell prospectus: guidance for Local System Partnerships - GOV.UK), the target beneficiaries for the programme are:

1. People in work with a disability or health condition who are either:

a. Struggling with health barriers, or

b. On sick leave and at risk of falling out of work

2. People out of work with a disability or health condition, and either or both of the following:

a. Low level needs

b. Recently out of work

WorkWell is intended to also act as a gateway through which those who require more intensive support may be referred on to services such as Connect to Work, Access to Work, Individual Placement and Support in Primary Care or Restart.

Mental health (MH) conditions and musculoskeletal (MSK) conditions are two of the key drivers of ill-health related economic inactivity. Therefore, the Joint Work and Health Directorate (JWHD) expect (but have not mandated) that these groups will form a significant part of the target population for the service.

Whilst sites have retained the broad definition above as their target beneficiaries at the time of commencing delivery, most of the pilots had further specified priority groups for referral to WorkWell. Typically, these cohorts were based on analysis of local data including prevalence of long-term physical and mental health conditions, indices of multiple deprivation, life-expectancy, and unemployment and inactivity levels, in addition to consultation locally with Jobcentre Plus offices and/or district councils. Additional categories of priority groups included:

Specific sub-groups of disabled people and people with health conditions:

For example, recency of fit note issued, or number of fit notes issued
Types of condition (e.g. respiratory conditions)
Whether accessing other services (e.g. if accessing social prescribing, or on a waiting list for community health or elective care)

Duration of time out of work

For example, newly unemployed, unable to work for between 5 and 13 weeks, unemployed in last 8 weeks before referral where poor health contributed to unemployment

Demographic characteristics

For example, Women, ethnic minorities, men aged 50+, young people, those living in rural areas and those living in areas of deprivation

Disadvantaged groups

For example, care experienced young people, ex-offenders, digitally excluded people and people with learning difficulties

Employment characteristics

Those from specific sectors, including where there are employment skills gaps and those working in lower income occupations

A small number of pilots have specified that they will have different target groups of beneficiaries within different areas of their pilot. However, for all pilots, the sub-groups that have been specified, and duration of time they may focus on these groups, are subject to change, as the roll-out and delivery of pilots commences.

Geography of WorkWell pilots

Twelve of the WorkWell pilots are in “Predominantly Urban” areas, with three “Significantly Rural”, as defined by the Rural Urban Classification - GOV.UK. The majority (10) of the WorkWell pilots will cover the entire ICB area. Of the remaining sites, the stated reasons during evaluation scoping for geographically targeting delivery of WorkWell were based either on evidence of higher need in certain areas, or the desire to select a variety of locations within an ICB with a range of levels of need, to allow the service to be piloted in different local contexts. Pilots in which the Local Authority area aligned with the ICB area tended to highlight this as a particular advantage in their delivery of WorkWell. Some of the other ICBs which do not have congruence between LA areas and their ICB geography, are targeting their WorkWell pilot so that it does not operate across the whole ICB footprint.

Membership of WorkWell pilots

WorkWell pilots are typically a collaboration between ICBs and Local Authorities, with additional support from a much larger group of other stakeholders including Jobcentre Plus, third sector organisations and employer representative bodies. In terms of governance, the Senior Responsible Owner (SRO) of the project usually sits within the ICB. They are ultimately accountable to a board within their ICB. Project management roles are also usually held by individuals employed by the ICB, although there are some exceptions, with the project manager for these locations sitting within the Local Authority.

Partnerships are of differing levels of maturity, based on the extent of previous experience of collaboration. Amongst the most mature joint working of this type has been taking place for over 30 years. Other partnerships have only more recently (e.g. within the past five years, or as a direct result of WorkWell) formally collaborated on initiatives.

Delivery models

At the time of conducting the feasibility and scoping exercise, the pilot sites were still in relatively early stages of the development of how their WorkWell service would operate. As illustrated in the Theory of Change (Chapter 2), all pilots will be delivering core elements including:

A mechanism for generating, recording and triaging referrals to the service
An assessment of an individual’s support needs
Input from a multidisciplinary team
Development of a personalised action plan
One-to-one coaching
Onward referral to other local support services

In-house vs outsourcing

Most pilots plan to deliver the WorkWell service “in-house”, by recruiting new employees and/or by seconding staff based within the ICB (or wider NHS e.g. Foundation Trusts, General Practices etc.) or Local Authority. Other sites were in the process of appointing an external delivery partner. All pilot sites felt that the set-up times for the programme were ambitious. This was particularly the case for those ‘outsourcing’ delivery, and these areas tended to be less further along (at the point of scoping) in the development of their services as a result.

However, those appointing external suppliers did note that they anticipated their delivery partners to be able to “hit the ground running”. They planned to contract organisations with pre-existing links to their local communities, and who would already have a workforce with staff able to deliver the role of the WorkWell coach. Some pilots have adopted a hybrid model of a blend of “in-house” staff supplemented by additional contracted capacity.

Pilots that intended to directly employ staff to provide support to individuals felt that a benefit of this approach was the level of oversight they would have to ensure that they appointed suitable staff members as coaches. Also, being able to reactively train and retrain them to deliver their roles as the delivery model evolves.

Referrals and triage

All pilots intend for referrals into WorkWell to come from a variety of sources including primary care networks, NHS secondary care trusts, Jobcentre Plus offices, VCSE organisations, employers, Local Authorities, and self-referrals, although a couple of pilots indicated that they expect the majority of their referrals to be sourced via GPs/primary care. Some pilots referenced plans to seek referrals from more specific sources, such as education and training providers, social housing providers, the National Careers Service and business representative bodies (e.g. Chambers of Commerce).

Some areas spoke of plans to build on existing referral routes, such as those in place for social prescribing, or for other work and health focused programmes, such as the Individual Placement and Support in Primary Care initiative. A few pilots were keen to emphasise that they felt the current gaps in their work and health support systems were mainly for those in-work, but struggling, including those on sick leave. Therefore they expected that a substantial proportion of referrals would come from this group, potentially following the participant meeting with the GP about receiving a fit note.

Triage was described as consisting of assessment of two components; eligibility and suitability. The WorkWell eligibility criteria (as defined above in the service prospectus) is considered to be fairly broad, and therefore some sites reference plans to prioritise potential participants based on suitability. Mechanisms to assess suitability were still being refined by pilot sites. However, pilots indicated that they would be assessing how likely the individual would be to benefit from the service, through measures such as the Patient Activation Measure^{[footnote 3]}, as a way of understanding motivation and readiness for change. The key to the success or otherwise of this process for pilots is mapping and facilitating effective onward signposting to more suitable offers, where WorkWell isn’t deemed the right fit.

Assessment of support needs, development of an action plan, the role of the MDT and one-to-one coaching

These aspects of service delivery tended to be amongst the least developed at the point the evaluation scoping was undertaken in Summer 2024. Aspects the pilots agreed upon were the holistic and person-centred nature of the assessment and action plan.It also agreed that the service is most suitable for those with low-level needs, serving as an early intervention function.

One-to-one coaching is a required element of the WorkWell programme, however within pilots, the individuals that will provide this type of support will have different job titles, such as “WorkWell Navigator”, “Work and Health Coach” or “Journey Guide”. For simplicity, we refer to those doing this role as “WorkWell coaches” for the remainder of the report. In some locations the WorkWell coaches will perform the triage function, whilst in others, this work will be supported by other staff. Some sites specified that their coaches will use motivational interviewing techniques.

The terminology used for action plans also varies; for example, “WorkWell Support Plan”, “Strive” or “Thrive” action plans or “Personalised Care Plans”. Commonalities include the plans being directed by, or co-produced with individuals, with regular reviews alongside a WorkWell coach.

Onward referral to other local support services

Other services suggested for onward referral included:

Debt advice providers
Jobcentre Plus facilitated programmes
Social prescribing
NHS Talking Therapies
Adult social care
Community physiotherapy
Education or training providers

Some areas are conscious of the impact of creating increased demand amongst onward referral organisations. One area specified that they would facilitate a programme of grants in order to boost capacity of local VCSE organisations. Several stated that their delivery of WorkWell included provision for “funding that follows the person”, a personal budget that would allow individuals to access further support.

Engagement with employers

Pilots had different visions for how they planned to engage with employers through WorkWell, building on the core elements specified in the WorkWell prospectus. As a minimum, the WorkWell prospectus states that WorkWell coaches should provide advice to participants on workplace adjustments. Additionally, with participant consent, they should contact employers to share work related aspects of their personal plan and provide advice.

Some pilots spoke of wider plans beyond engagement related to specific participants, including:

Provision of strategic support, advice and guidance to upskill employers on aspects of employment such as inclusive recruitment, coaching and mentoring, having challenging conversations, and wellbeing at work
- Some sites specifically said they planned to create employer-focused training packages to, for example, upskill line managers
Further promotion of pre-existing initiatives including the Disability Confident employer scheme and Mental Health First Aid training
Intentions to influence employers to create job opportunities that are accessible to those with health conditions

As mentioned above, it was common for pilots to have secured some form of representation from employer groups (e.g. Chambers of Commerce, Local Enterprise Partnerships) on their wider WorkWell Partnership delivery group. Pilots planned to leverage these channels as one means of reaching employers, in addition to some developing employer-focused communications plans.

A few sites did not elaborate on engagement with employers beyond the individual participant focused activities. One stated their plans were still emergent, depending on the extent of help employers required with creating “healthy workplace strategies”.

Role of technology in delivery

The emphasis on the role of technology in referral, triage and support provision varied between pilot sites. Some detailed their plan to use a single website as a gateway for all referrals, both self-led and by third parties (e.g. employers, GPs, VCSEs). Another intended to create a separate professional-facing portal. One pilot stated they intend to explore the use of AI and/or some degree of automation in assessing referrals. Some pilots emphasised that their support would be multi-channel: in-person, by telephone and online.

Existing and prior programmes

WorkWell is one initiative which fits into a larger picture of work and health provision across the pilot areas. Each pilot area’s context is unique, but programmes that will be live in parallel with WorkWell, mentioned by some, included:

Individual Placement and Support in Primary Care (IPSPC)
Employment Advisors in Talking Therapies (EAs in TT)
Local Supported Employment (LSE)
Restart
Social prescribing
Shared Prosperity Fund support
Connect to Work (previously known as Universal Support)
Thrive to Work

Locations which had in place the IPSPC initiative felt that this was particularly complementary to the WorkWell service. They saw this as an alternative referral route for individuals who may be suitable for WorkWell, but whose needs are more complex. Several pilot sites mentioned that the end of funding streams for other individual programmes, and the potential changes brought about by the recent change in government, were a potential barrier to their successful delivery of WorkWell. This is because these factors would change the landscape that WorkWell operates within. A few pilot sites also referenced programmes they had previously delivered which they felt had a similar design and/or remit to WorkWell. These included: Working Win and Working Well Early Help. These sites felt better positioned to mobilise as they were able to draw on learnings from their previous initiatives.

Whole systems change

Many of the pilot sites felt that their partnership members shared common goals but had been working towards these in silos to different degrees. These goals, such as early intervention, reduction in health inequalities, and inclusion of marginalised groups, were seen to be aligned with the intended goals of WorkWell, and in particular, the content of the Work and Health Strategies that WorkWell funding has supported. Pilot sites thought that it was likely that local organisations have been missing out on opportunities to collaborate on both efficiencies and innovations in this space.

Pilot sites hoped that systems-wide change, supported by WorkWell, might manifest through:

Clear allocation of responsibilities and having a transparent governance structure
Increased leadership capacity
Better understanding of current work and health provision and gaps, and alignment of existing offers
Culture change to embed work within health services, influenced by senior leadership buy-in

Question 10

5.  Designing the evaluation of the WorkWell pilots

Accepted Answer

This chapter discusses the approach to designing an evaluation of the WorkWell pilots. It briefly describes the context in which the evaluation has been designed, and the challenges and constraints created by this.

This feasibility study has been written alongside the process of the pilot sites developing their offer. They put in their initial bids for funding by January 2024 and received notification that they had been successful in May 2024. The evaluation scoping was undertaken in Summer 2024.

This has created opportunities for the evaluation in terms of making it possible to embed standardised Management Information into the delivery of the pilots. However, it also created some challenges in designing an evaluation as there continues to be uncertainty about exactly how the individual pilots will be delivered in practice and how similar or different they will be. In some cases, their bids detail their ambition for WorkWell but they have waited until confirmation of funding to design much of the practical detail.

To some extent this is often the case when new initiatives are at the early stages of development. The issue is exacerbated here by the ‘black box’ nature of the intervention. As explained in the previous chapter, in this case, this meant that at the time of developing this report, most sites were still finalising their thinking. There is thus some uncertainty about:

How referrals into WorkWell will take place
The likely balance in participants between those who are in-work and out-of-work (and relatedly the balance between those who will be existing benefit claimants and those who will not)
How challenging it will be to reach the target number of WorkWell participants (and hence whether or not there is a possibility that they will be over or under-capacity)
The exact nature of services that will be provided to people referred into WorkWell
The demographic profile of WorkWell participants

It is also the case that contracts with pilot sites had already been signed prior to the start of this feasibility study. Contracts encouraged pilot sites to co-operate with evaluation. However, they did not stipulate the type of participation requirements that would typically have large service-design implications and potentially costs. So, for example, sites were not expecting to put in place Randomised Control Trials (RCTs) or to seek to generate more referrals than their target number of participants, to generate a waiting list for evaluation purposes.

With the aim to have the outline of the evaluation in place by Autumn 2024 (when the earliest pilots started to deliver services), it did not feel realistic for the core evaluation design to rely on experimental techniques. These would require considerable logistical and perceived ethical barriers to be overcome, as well as having potential cost and resource implications for the pilots. In addition, the ‘black box’ delivery approach means that, essentially, any experimental design would need to be built into 15 different pilots, with participant characteristics and referral routes across pilot areas still being refined at this stage.

The following chapters of this report lay out our thinking around evaluation design within this context. We include sections on:

Process evaluation. The aim of the process evaluation will be to understand how the pilots are operating on the ground, how they are achieving impact, the challenges that they have faced and how these have shaped delivery over time.
Impact evaluation. The aim of the impact evaluation will be to measure the changes that have been brought about by the WorkWell pilots, over and above what would have happened anyway.
Economic evaluation. The aim of the economic evaluation will be to attach monetary values to the impacts that have been achieved.

Question 11

6.  Process Evaluation

Accepted Answer

Purpose of the process evaluation

The purpose of the process evaluation is to understand how the programme has been implemented and delivered and how the pilots operate. The process evaluation will test and explore the mechanisms and assumptions in the Theory of Change (ToC), as well as capture information about how and why outcomes and impacts are achieved or not achieved. The process evaluation will also help us understand ‘what works for whom’, for example, which elements of the programme are more effective for which groups of people. It will explore whether or not the pilot has been implemented and delivered as intended, as well as the challenges the pilots have faced and how these have shaped delivery over time.

Scope of the process evaluation

As outlined in the Theory of Change, the WorkWell pilot programme is expected to create change:

at an individual level, primarily for participants
within organisations in pilot areas, for example amongst statutory and voluntary sector staff and employers
and at a wider system level, for example in the nature of relationships between partnership members

The overarching research objectives will be to understand:

Is WorkWell being implemented as expected?
Is the design working?

Specific research questions relevant for each audience are detailed in the table below (Table 6.1). These are broad research questions, to inform the design of research instruments; they are not intended to be respondent-facing. Primary audiences, those who will supply key information, for research questions are denoted with an ‘x’, secondary audiences, those who may be able to provide supplementary information, are detonated with a ‘(x)’. Please note that the Participatory Systems Mapping (PSM) and the Qualitative Comparative Analysis (QCA) methods shown below are discussed separately in Chapter 7, as components of the impact study, that will also provide insight into process evaluation element.

Table 6.1: Process evaluation research questions, by audience and research activity

	Participants	Participants	Participants	WorkWell coaches	Employers	Senior stakeholders
	MI analysis	Participant surveys	Participant qual	WorkWell coaches qual	Employer qual	Pilot partnership qual
Participants
1.    What are the demographic characteristics of those who successfully complete WorkWell? Are any groups more likely to drop-out of the programme, or not achieve their planned outcomes?	Primary audience	Primary audience	not applicable	Secondary audience	not applicable	Secondary audience
2.    How was the journey through WorkWell experienced by participants, both overall, and at specific stages (referral, triage, assessment, development of a personalised plan, ongoing support, referral to other services)?	not applicable	Primary audience	Primary audience	Primary audience	Primary audience	not applicable
3.    Were overall aims of creating one central referral point for work and health support in pilot areas achieved? How ‘joined-up’ did the process feel?	not applicable	Primary audience	Primary audience	Secondary audience	not applicable	Secondary audience
4.    To what extent was the support multidisciplinary or holistic?	not applicable	Primary audience	Primary audience	Primary audience	not applicable	Secondary audience
5.    For those in work, how did WorkWell engage with their employer?	not applicable	Secondary audience	Primary audience	Secondary audience	Primary audience	not applicable
6.    How satisfied were participants with their experience of support?	not applicable	Primary audience	Primary audience	not applicable	not applicable	not applicable
7.    Are there aspects of the service participants feel could be improved?	not applicable	Secondary audience	Primary audience	not applicable	not applicable	not applicable
Organisations within pilot areas
8.    How do staff decide suitability to refer to WorkWell?	not applicable	not applicable	not applicable	not applicable	not applicable	not applicable
9.    What are the motivators to selecting WorkWell at referral stage?	not applicable	not applicable	not applicable	not applicable	not applicable	not applicable
10. What are the barriers to referring to WorkWell?	not applicable	not applicable	not applicable	not applicable	not applicable	Secondary audience
11. How were participants individual plans developed and monitored?	not applicable	not applicable	not applicable	Primary audience	not applicable	not applicable
12. How effectively are MDT used/involved?	not applicable	not applicable	not applicable	Primary audience	not applicable	Secondary audience
13. How well equipped did WorkWell coaches feel to refer to other providers? Barriers and facilitators to onward referral	not applicable	not applicable	not applicable	Primary audience	not applicable	not applicable
14. How could the service be improved?	not applicable	not applicable	not applicable	Primary audience	not applicable	Secondary audience
15. What types of engagement did employers have with WorkWell?	not applicable	not applicable	not applicable	not applicable	Primary audience	not applicable
16. How useful or effective was this support/engagement for employers?	not applicable	not applicable	not applicable	not applicable	Primary audience	not applicable
17. How satisfied were employers with WorkWell?	not applicable	not applicable	not applicable	not applicable	Primary audience	not applicable
18. Are there aspects of the service employers feel could be improved?	not applicable		not applicable	not applicable	Primary audience	not applicable
19. How do WorkWell providers manage the quality of the service, demonstrating appropriate oversight and clinical governance?	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
20. For those delivering ‘in-house’: what has been the experience of resourcing WorkWell, particularly staffing and recruitment? Any particular challenges experienced, impacts of challenges and how difficulties have/could be overcome.	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
21. For those procuring delivery partners: what has been the experience of procuring service providers to deliver WorkWell? Any particular challenges experienced, impacts of challenges and how difficulties have/could be overcome.	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
22. How has service demand aligned with expectations?	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
Systems level
23. How does WorkWell develop an integrated whole system response between health, employment and wider community place-based services?	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
24. How does WorkWell improve service integration/local networks?	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
25. How does WorkWell improve access to specialist work and health assessments?	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience
26. How does WorkWell improve access to treatment, dependent on capacity and skills mix over time?	not applicable	not applicable	not applicable	not applicable	not applicable	Primary audience

Methodological approach

Scale of case study approach

Each pilot is implementing WorkWell in their own way, and so we can expect the experiences of all of the above audiences to differ by pilot area. There will be variation within pilot areas too, where pilots are tailoring their approach within smaller geographic areas. To understand how different elements of delivery interact and what that means for participants, it would be ideal to take a case-study approach to all pilot areas. This would allow for evidence for each pilot to be triangulated. This would allow us to answer the process study questions comprehensively for each pilot. However, due the limited qualitative sample size, this option will not be feasible. Instead, a broader coverage across all 15 pilots will be used, with the ability to add context for individual pilot areas where applicable to help interpret the impact findings.

Approach by audience

Participants

To address the research questions focused on individuals (1 through 7, in Table 6.1 above), we suggest drawing on a combination of MI analysis, survey research and qualitative in-depth interviews. The MI will should provide an overview of demographic characteristics for the total participant population. Survey data will be collected simultaneously with the evidence for the impact evaluation, with baseline, 6-month and 12-month data collection from non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants. A baseline and 12-month survey will also be conducted for the non-claimant comparison group of non-participants.

For the qualitative research amongst participants, we suggest data is ideally collected through longitudinal in-depth interviews, over two rounds, so we can understand whether and how short to medium term outcomes manifest. Aiming to recruit 4 participants per pilot case study, so 60 participants in total, would provide both a good sample size to understand participants’ experiences across the pilot sites, and to mitigate attrition on follow-up, allowing for 25 follow-up interviews.

The initial in-depth interview should take place once an individual has been referred onto, and started receiving support from WorkWell. This will allow us to understand their situation immediately prior to WorkWell, and their initial impressions of the service. The second should take place a few months after they complete the intervention so that we can understand early emerging outcomes.

A purposive sample of participants would be suitable to capture deliberately contrasting characteristics.This includes those in employment and not in employment, and those with a range of different types of disability/health condition.

Table 6.2 Suggested sample structure for participants

	Sampling 15 pilots
First wave of qualitative interviews	60 (4 per pilot) WorkWell participants
Second wave of qualitative interviews	25 WorkWell participants
Baseline survey	1,600 per group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants). 600 surveys with non-claimant comparison group of non-participants.
6-month survey	800 participants per group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants).
12-month survey	400 participants per group (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants). 300 of the same non-claimant comparison group of non-participants.

WorkWell coaches

Research questions 8 to 14 concern the views of those making referrals into WorkWell, and the staff most closely involved in providing support to participants on an individual basis, the WorkWell coaches.

WorkWell coaches are likely to be a highly motivated and engaged audience. With this group it would be suitable to conduct focus groups of 5-6 WorkWell coaches working in the same pilot areas. This would allow the evaluation to draw out the commonalties and differences of experience within pilot areas.

Table 6.3 Suggested sample structure for WorkWell coaches

	Sample
12-month focus group with WorkWell coaches	15 focus groups (with up to 6 WorkWell coaches in each),1 focus group per site.

Employers

As for WorkWell coaches, to capture the views of employers (primarily to address research questions 15 to18), it is recommended that focus groups of employers within each case study area would be an appropriate way of achieving the depth of insight needed about employers’ experiences of the programme.

It should be noted that the arrangements for obtaining permission to share contact details for employers with an evaluator, were not in place at the point the scoping research was undertaken. To maximise the potential to obtain a robust sample size of employers, the evaluator, JWHD, the pilot partnership members, and sub-contracted delivery partners would need to collaborate to ensure such mechanisms are created.

Table 6.4 Suggested sample structure for employers

	Sample
12-month focus group with employers	15 focus groups, 1 per site.

Senior Stakeholders

To address research questions 19 to 26, we suggest two further phases of qualitative research with key members of pilot partnerships. We would expect to include at least one representative from the area’s ICB, and one from the Local Authority. This element of the research would comprise paired/triad interviews, ideally with the same individuals who took part in the scoping phase interviews. This is so that the evaluator can explore how and why delivery in practice has evolved from the intentions stated during the scoping phase. These interviews will allow the evaluator to understand system and strategic outcomes and impacts of the delivery of WorkWell, and how different delivery models have influenced these outcomes.

We recommend these interviews would ideally be undertaken in every pilot area, to capture the full range of ways in which WorkWell has been operationalised and the resultant impacts.

To support the economic evaluation, we also recommend a ‘cost and resources’ survey is undertaken, with the purpose of obtaining a more granular and precise understanding of the inputs needed to deliver each WorkWell pilot. The audience for this survey would be either the in-house Project Manager of WorkWell within a partnership, or the outsourced Delivery Partner, if relevant. This is discussed further in Chapter 8.

Table 6.5 Suggested sample structure for senior stakeholders

	Sample
Initial focus group with senior stakeholders	15 focus groups (up to 3 stakeholders in each).
12-month focus group with senior stakeholders	15 focus groups (up to 3 stakeholders in each).
Costs and Resource survey	15 responses, 1 from each site.

National Support Team

During the scoping phase, the role of the National Support Team in providing the National Support Offer was still emergent. However, we think that ideally there could be value in drawing on outputs that the NST are producing as part of their programme of work. We understand potentially the NST will create fortnightly summaries of the content of peer learning sessions. It may be useful for the evaluator to review these documents (2 per month) and synthesise learnings of relevance to the evaluation into the wider analysis.

Question 12

7.  Impact Evaluation

Accepted Answer

This chapter explores the possible approaches for an impact evaluation of the WorkWell pilots, in order to evaluate the impacts of the WorkWell programme identified in the Theory of Change. It discusses how the treatment and comparison group might be defined. It considers experimental and quasi-experimental impact evaluation options, potential secondary data sources that might be used and how these might be supplemented by primary participant surveys. It describes how impact might be calculated. Finally, it explores potential approaches for exploring the whole systems change aspect of the pilots’ work.

As demonstrated by the Theory of Change shown earlier in this report, the WorkWell pilots are looking to achieve impacts both on individual participants but also on the local systems around employment and health. Measurement of impact on individuals and impact on systems change need to be approached separately.

Defining who is eligible

Those eligible for the WorkWell pilot will be individuals with a health condition or disability that is impacting either on their ability to find work or their ability to perform in their current job. These two groups are likely to receive different services and the outcomes that WorkWell is looking to achieve are slightly different. With those currently looking to find work, the primary outcome desired is moving into good quality work while for those already working it will be sustaining appropriate employment (whether with their current employer or not).

In some cases, the interventions might be similar for the two groups (e.g. support around managing a health condition or disclosing to employers how a condition affects them) but in others they might be different (e.g. for those in work there might be support in negotiating reasonable adjustments with employers while for those not in work there might be support in identifying jobs that are a good fit and other job search skills). In the long term, outcomes may also be sustained differently between those in work and out of work^{[footnote 4]}. This means the impact evaluation has to consider both groups as part of its design.

The importance of a counterfactual group

When evaluating an intervention, there is the possibility that any outcomes and impacts observed among participants may have happened anyway, without the intervention. For example, some WorkWell participants may have received support with work and health anyway, in the absence of WorkWell. We therefore need to consider causality. This is the extent to which we can provide evidence that the observed outcomes and impacts are caused by the intervention, rather than by other factors. We can do this by comparing the treatment group to a counterfactual group who did not have access to the intervention.

The counterfactual group looks at the contribution the WorkWell pilots have made to the observed outcomes and impacts. It explores what would have happened in the absence of the WorkWell support. A counterfactual group needs to be as similar as possible to the treatment group who received the intervention. Ideally, the only difference would be that the treatment group accessed the pilot support, and the counterfactual group was not able to access the support. By comparing outcomes and impacts for the two otherwise similar groups, we can conclude that any outcomes or impacts observed in our treatment group but not in our counterfactual group, are attributable to the WorkWell pilots.

Experimental designs

The most robust design for an impact evaluation is generally considered to be a randomised control trial (RCT). With this design eligible individuals are randomly allocated to the treatment group (where they receive WorkWell) or a control group (where they receive the ‘business as usual’ service). The outcomes of both groups can be tracked over time and the differences in outcomes (if statistically significant) can be reasonably attributed to having taken part in WorkWell.

As mentioned in Chapter 5, it is not feasible to establish a large-scale RCT at this stage of the rollout of the pilots, with so many unknowns around referral routes and participant characteristics. This is not what pilot sites have been led to expect through the process of bidding for funding, and they are not currently building their service design with mechanisms for randomisation.

Quasi-experimental designs

It is also possible to estimate impact using a quasi-experimental design (QED). With these approaches, we need to identify a group of individuals who closely resemble the WorkWell participants. They can be used to provide an indication of the outcomes that would have been achieved in the absence of WorkWell support. There are a number of different ways in which this can be achieved but generally they will place less burden on the pilots themselves, as can be done retrospectively through datasets.

As part of the set-up of the WorkWell pilots, a participant Management Information form has been developed which is intended to be used across all pilots. If completed correctly then this provides us with a good level of detail about each participant which can be used for evaluation purposes. The MI template includes personal contact details for each participant and their National Insurance Number (NINO). This provides scope for participants to be identified in other datasets, which include variables of interest identified by the Theory of Change, such as earnings and benefit information. The MI template also contains information about their employment and health context. This can be used for identifying a group of similar individuals to form a counterfactual group. We could use this information to identify a comparison group of individuals:

From within the WorkWell pilot areas
From within a group of comparison areas chosen as good matches for each of the individual WorkWell pilots
Using a synthetic control group composed from a blend of non WorkWell areas, in such a way as to ensure that the pre-WorkWell trends in relevant data match the pilot areas exactly

Drawing a good comparison group from within pilot areas poses some challenges. Most pilots are envisaging delivering their service across the whole ICB area, so there are not particular geographies within the pilot ICBs where the service will not be available. Eligible non-participants from within pilot areas therefore could, in theory, have chosen to access WorkWell. These include those who have been referred but declined to participate, those who declined to be referred and those who were never considered for referral. The reason they have chosen not to participate in WorkWell may make them different from WorkWell participants in ways that cannot easily be identified or controlled for, which would bias the results of the evaluation. There would also be the potential for those identified as a good match to then go on to be referred into WorkWell at a later date.

Hence an approach using matched comparison areas or a synthetic control feels preferable. A synthetic control might provide a closer match for the pilots at a combined/overall level. However, in order to look at impacts for each pilot on its own (or for a group of pilots), a matched comparison area for each individual form of WorkWell implementation will be required to gain the ability to disaggregate. Additionally, to evaluate the impact for each sub-group (e.g. those in work and those out of work at the start of WorkWell participation), separate synthetic control groups would be required for each sub-group of interest. Given that we are not yet completely clear on how different delivery might be in each pilot or how sub-groups of participants might be impacted differently, this ability to disaggregate feels like it might be useful. It would enable us to understand whether greater impacts are achieved under certain approaches compared to others, or for different participants.

The formation of synthetic control groups for each pilot site and subgroup, in order to disaggregate, would lead to smaller sample sizes and therefore reduce statistical power. Additionally, forming synthetic controls for each pilot site and subgroup would be substantially time and resource intensive. For this reason, creating a counterfactual from a group of comparison areas chosen as good matches for each of the individual WorkWell pilots, is the most likely approach. Further discussion of the comparison group approaches that were considered are included in Appendix 5.

However we define our comparison group, we would then want to compare the relative change in key measures over a relatively long period of time, given that some outcomes from the Theory of Change, for example improved health and wellbeing, might take some time to realise. This would involve establishing measures at a baseline point and then at follow-up points.

It seems reasonable to set these follow-up points at 6 and 12 months. By the 6 month point it seems reasonable for individuals to have completed all activities related to their involvement with WorkWell (including referrals to other services). The 12 month point will allow us to look at medium term outcomes. These follow-up points also line up well with the overall timeframe available for the evaluation, and also with the minimum follow-up points recommended from studies of other health and work interventions. (see Chapter 3).

For a QED approach, we can collect the necessary data at both the baseline and follow-up points from secondary data sources and/or through a survey of participants and a comparison group. In broad terms:

Using secondary data will generally allow us to look at larger sample sizes because we can use all participants (that we are able to match using unique identifiers such as NINOs) and individuals that we match to them. However, we can probably only look at a relatively small number of the outcomes shown in the Theory of Change.
Using survey data will inevitably involve smaller sample sizes, due to not being able to survey all participants. However, we can define the outcomes that we collect meaning that we would expect to cover a much larger proportion of outcomes.

Hence it seems likely that we will want to consider including both options in the impact analysis. We discuss each of these in more detail below.

Impact estimates from secondary data

Potential data sources

There are various potential data sources which could be used to collect the data necessary for the evaluation. These are listed in Table 7.1. All the reported datasets have been reviewed, to understand the quality, format and whether any outcomes relating to employment and health are collected.

Some of the datasets identified have been ruled out from use in the evaluation. For example, the Fit Note dataset is unlikely to be used in the evaluation, as it only provides aggregate level information at Integrated Care System (ICS) level. Additionally, primary care datasets will not be used due to access limitations.

Table 7.1 gives an overview of each dataset considered, along with the advantages and disadvantages of using it for the evaluation. Further discussion is provided below.

Table 7.1: Data Source Information

Dataset	Overview	Advantages	Disadvantages
RAPID	Dataset using HMRC and DWP linked data on employment and benefit history for anyone who has a national insurance number (NINO).	WorkWell participants can be identified using NINO, and a counterfactual group of non-participants could be created; Covers 100% of the population; Includes employment and benefit outcomes which can be used to capture the impact of WorkWell; Includes data allowing assessment of impacts which are included in the dataset beyond the 12-month point; DWP have access to the dataset and can match non-participants for the counterfactual group.	Annual release a few months after the end of the financial year which could delay analysis of outcomes; Earnings are aggregated across the tax year. This will make it difficult to identify the impact of WorkWell on earnings, especially for participants who start the programme part-way through the financial year; Health condition data is only available for those who are claiming disability benefits.
RTI (Real Time Information)	HMRC data which includes JSA, PIP and UC claimants since 2013, includes monthly earnings information.	Lagged only a few months from the current date; Earnings information provided for individual months. The impact of the WorkWell programme on earnings will be able to be identified; Includes data allowing assessment of impacts which are included in the dataset beyond the 12-month point; DWP have access to the dataset and can match non-participants for the counterfactual group.	Only covers UC/JSA/PIP claimants (including previous claimants of these), so there is the potential for a large proportion of WorkWell participants to not be in this dataset.
Labour Force Survey (LFS)	A household survey of employment circumstances in the UK, conducted by the Office for National Statistics (ONS). A longitudinal dataset is available for employment over time.	Includes granular information on employment, hours of work, occupation, training and health conditions; A ready dataset which would not require additional work to collect information.	A sample-based survey which only covers approximately 0.13% of the UK population. The longitudinal dataset is even smaller and only includes around 4,000 samples per quarter; The dataset with small enough geographical variables to identify people not living in WorkWell sites is difficult to access.
Annual Population Survey (LFS)	A survey which builds on the Labour Force Survey to increase the number of responders. A longitudinal dataset is available with outcomes at entry to the survey and after 12 months. This would only be used for the counterfactual group, and not for WorkWell participants.	Includes granular information on employment, hours of work, occupation, training, sickness and health conditions; A ready dataset which would not require additional work to collect information; The longitudinal dataset includes a larger sample size than the Labour Force Survey.	The dataset with small enough geographical variables to identify people not living in WorkWell sites is difficult and time consuming to access; While larger than the sample in the longitudinal Labour Force Survey, the survey is still a sample, and the size will be reduced when identifying those eligible for WorkWell in non-pilot areas. This could end up being a small sample overall; The survey may collect or define variables differently to the administrative datasets such as RAPID and RTI, this inconsistency may cause bias.
Fit Note dataset	Aggregate data on the number of fit notes issued in each ICS.	Information on the number of fit notes issued in each ICS.	Only aggregated data at ICB level is available, no person level data.
Participant Management Information (MI)	Management information template to be filled out by pilot sites for each participant in the programme. Includes a wide variety of outcome measures including employment, reason for referral, health conditions and work plans.	Includes many required outcome measures to evaluate the impact on employment and health of the WorkWell programme; Completed by pilot sites and returned monthly so timely information for the evaluation.	Complete data only collected for WorkWell participants; Those referred but who do not participate may also have reason for referral and eligibility recorded.
Participant Survey	To be completed by WorkWell participants to understand their experience before starting the programme, while receiving support and upon completion of the programme. The comparison group survey will collect information on participants health and how it effects their ability to work.	All relevant outcomes can be measured.	Small sample size.

The Labour Force Survey and the Annual Population Survey are being considered, to identify a counterfactual group. The Labour Force Survey is the largest household study in the UK and is the official measure of employment and unemployment. The Annual Population Survey is based on the Labour Force Survey and boosts the sample for statistical inference to be made at the local level. The larger sample size makes the Annual Population Survey the preferred survey to use.

Both have a larger sample size than the participant and counterfactual group survey which will be carried out for the evaluation. The two-year Annual Population Survey longitudinal dataset is proposed to be used, since it contains outcomes for the same individuals at two points in time, one year apart. This matches the timeframe for data obtained from the participant and control surveys. The datasets are released quarterly, and multiple editions can be used in the evaluation to more closely match the timings of when WorkWell participants start the programme.

The multiple editions of these datasets each have different access requirements. The less restricted datasets have ‘region’ as the lowest geographical variable. This is so it would not be possible to identify a counterfactual group in non-WorkWell areas, as the WorkWell pilots operate within smaller geographies than this. The more restricted datasets, while still anonymised, contain geographical variables at the local authority level, so areas with WorkWell pilots could be excluded from the formation of a counterfactual group. However, these datasets can take a long time to access, so it may not be feasible to use in the evaluation.

The UK Data Service indicates that from application, access to data can be gained in four months, if no changes need to be made to the application for approval. It is likely that additional time would be required to prepare the application and potentially to make amendments to gain approval. However, given the evaluation period is four years, attempting to access this dataset should be strongly considered. Annual Population Survey may underreport benefit claimants (i.e. some people who are benefit claimants are not listed as such in the dataset). Therefore, extra consideration will have to be given when deciding on whether subgroup analysis using benefit claim status should be conducted.

Additionally, there may be some people who live outside of WorkWell pilot areas who can still be referred to the programme if their GP or JCP is in a WorkWell pilot area. There is a risk these people could be identified as living in non-WorkWell areas and included in the counterfactual group. However, the proportion of the counterfactual group with this specific geography is expected to be small. Since the datasets are anonymised, these can only be used to evaluate outcomes for the counterfactual group formed from non-WorkWell pilot areas. Therefore, if national insurance numbers (NINO) are missing from the MI template for participants, there will be no impact on this approach.

DWP datasets, such as the Registration and Population Interaction Database (RAPID) and the Real Time Information (RTI) database, contain identifiable information, such as NINO. This will enable participants in the intervention group to be identified. Therefore, additional outcomes could be evaluated for the intervention group, as well as excluding them from the formation of a counterfactual group. If NINOs are not available for all participants, it is anticipated that items of personal identifiable data could be used to identify WorkWell participants in administrative datasets. This is subject to confirmation of the granularity of date of birth and may result in some participants not being able to be identified with certainty if the personal data is out of date, incomplete or truncated.

RAPID contains data for everyone with a national insurance number, which would allow a larger counterfactual group to be identified. However, this dataset is only released annually, and earnings are provided in aggregate for the full financial year. Health condition information is only provided for those who need to declare these for claiming benefits. The impact of WorkWell on earnings will be difficult to identify from annually aggregated data, especially for participants who do not start WorkWell at the beginning of the financial year and finish the programme at the end of the financial year.

The RTI database contains monthly earnings information, which could be used to determine the impact of the WorkWell programme on earnings for participants, and compare this to non-participants in the counterfactual group. However, the RTI database only includes Universal Credit (UC), Personal Independence Payment (PIP) and Job Seekers Allowance (JSA) claimants and previous claimants. This will limit the scope of the evaluation, as there will be systematic differences between claimants and non-claimants. If the proportion of WorkWell participants who are claimants is high, this will not be a large problem.

However, if the majority of WorkWell participants are not claimants, granular earnings outcomes will not be available for a large proportion of participants or controls in non WorkWell areas. This is a large risk for the evaluation, the impact of which will not be known until the programme starts and the proportion of claimants and non-claimants can be determined. However, no datasets have been identified which can evaluate detailed earnings in the long-term for non-claimants.

WorkWell pilots will be providing JWHD management information (MI) data on a regular basis. Participant MI data are expected to be returned monthly, and expenditure MI data are expected on a quarterly basis. The participant MI has been developed by JWHD to collect participant information at the start of the programme, as well as their outcomes at the end of the programme. This information will be used in the evaluation to assess the short-term impact of WorkWell on participants. It will also provide information on the make-up of participants in the programme. For example, what proportion of participants are in work and struggling and what proportion are out of work due to their health conditions.

Impact estimates from survey data

The WorkWell MI provides a good sample source for a survey of WorkWell participants. In line with the approach for the impact estimates from secondary data, the matched comparison group of eligible non-recipients is likely to be best drawn from non-WorkWell areas (perhaps ideally matching each pilot to a comparable geographical area).

For DWP claimants, the most robust approach to sampling a counterfactual group will be DWP’s own databases of benefit claimants which will, by definition, include a complete record of all eligible non-recipients who are claimants. These records can be used as a sample for survey research using the ‘public interest’ provision under GDPR.

Sampling a counterfactual group of individuals who are not benefit claimants is more challenging. The suggested approach to gaining a sample for a non-claimant counterfactual group of non-participants would be through sourcing via an online survey panel. Online panels are comprised of individuals who have expressed interest in taking part in research projects and who are rewarded financially for their participation. Individuals are recruited from a wide range of sources. Large numbers of panel members can be screened cost effectively to find eligible participants. Initial investigations with panel providers suggest that it might be possible to recruit a baseline survey of 600 individuals, and to conduct 1 follow-up survey with them at 12 months (which would be likely to generate around 300 complete interviews).

The specification for the evaluation for the WorkWell pilots, stated a desire to achieve an end-point sample with a confidence interval of +/- 5% at the 95% confidence level. Given the different needs and expected outcomes for the two different groups, we want to aim for at least this volume of interviews in both the groups of claimants who are working and not working at the point of entry to WorkWell. This would point towards trying to achieve the interview volumes outlined below (Table 7.2). This assumes that in-work DWP claimants count for a sufficient proportion of all participants for it to be worth looking at impacts on them specifically.

For the non-claimant comparison group, the sample is limited to the maximum achievable number of interviews. The differences needed for statistical significance shown in Table 7.2 are for a ‘worst case’ scenario of findings at around the 50% mark and also for findings at around the 10% mark where smaller differences would be necessary. In addition, smaller differences would be detectable by combining the various participant and comparison samples (as shown in the ‘total samples’ columns). This would be possible for survey questions asked of all audiences.

Table 7.2: Suggested sample sizes for survey and associated differences needed for significance

	Non-claimants	Non-claimants	Non-claimants	Non-claimants
	Participant	Comparison (from online panel)	Difference for statistical significance c.50%	Difference for statistical significance C10%
Starting sample	5,333	not applicable	not applicable	not applicable
Baseline survey	1600	600	4.7pp	2.8pp
6-month survey	800	not applicable	not applicable	not applicable
12-month survey	400	300	7.5pp	4.5pp

	DWP Claimants (in work)	DWP Claimants (in work)	DWP Claimants (in work)	DWP Claimants (in work)
	Participant	Comparison	Difference for statistical significance c.50%	Difference for statistical significance C10%
Starting sample	5,333	5,333	not applicable	not applicable
Baseline survey	1600	1600	3.5pp	2.1pp
6-month survey	800	800	4.9pp	2.9pp
12-month survey	400	400	6.9pp	4.8pp

	DWP Claimants (out of work)	DWP Claimants (out of work)	DWP Claimants (out of work)	DWP Claimants (out of work)
	Participant	Comparison	Difference for statistical significance c.50%	Difference for statistical significance C10%
Starting sample	5,333	5,333	not applicable	not applicable
Baseline survey	1600	1600	3.5pp	2.1pp
6-month survey	800	800	4.9pp	2.9pp
12-month survey	400	400	6.9pp	4.8pp

	Total samples	Total samples	Total samples	Total samples
	Participant	Comparison	Difference for statistical significance c.50%	Difference for statistical significance C10%
Starting sample	not applicable	not applicable	not applicable	not applicable
Baseline survey	4800	3800	2.2pp	1.3pp
6-month survey	2400	1600	3.2pp	1.9pp
12-month survey	1200	1100	4.1pp	2.5pp

Table 7.3 shows differences needed for statistical significance for findings that would be reported as a percentage (for example the proportion of individuals entering work). However, some of the key metrics that we are likely to use for assessing impact will be standardised scales reported as a numeric value. Table 7.3 shows examples of the minimum detectable effects (MDE) that would apply for a continuous outcome using the EQ-5D as an example.

Table 7.3: Minimum detectable effects for suggested sample sizes (using EQ-5D as an example)^{[footnote 5]}

	Participant	Comparison (from online panel)	MDE (95% confidence level and 80% statistical power)
Starting sample	5,333	not applicable	not applicable
Baseline survey	1600	600	0.0115
6-month survey	800	not applicable	not applicable
12-month survey	400	300	0.0184

	Participant	Comparison (from online panel)	MDE (95% confidence level and 80% statistical power)
Starting sample	5,333	5,333	0.0047
Baseline survey	1600	1600	0.0085
6-month survey	800	800	0.0121
12-month survey	400	400	0.0171

	Participant	Comparison (from online panel)	MDE (95% confidence level and 80% statistical power)
Starting sample	5,333	5,333	0.0047
Baseline survey	1600	1600	0.0085
6-month survey	800	800	0.0121
12-month survey	400	400	0.0171

	Participant	Comparison (from online panel)	MDE (95% confidence level and 80% statistical power)
Starting sample	not applicable	not applicable	not applicable
Baseline survey	4800	3800	0.0052
6-month survey	2400	1600	0.0078
12-month survey	1200	1100	0.0101

Within the available resources, this distribution of interviews across the various participant and comparison groups would be the ideal in order to look separately at outcomes for those in work and those out of work. However, the make-up of actual WorkWell participants is as yet unknown, and the survey sampling approach may need to be revised.

For the surveys of participants / non participants who are claimants, The table above assumes response rates of:

30% to the baseline survey
50% to each of the follow-up surveys

These are comparable with the response rates achieved for longitudinal surveys of other health and work interventions (e.g. Evaluation of Group Work / Jobs II and the evaluation of Employment Advisors in IAPT/Talking Therapies).

The analysis of this survey data will involve matching individuals across participant and counterfactual groups. This may affect final sample sizes available for analysis (i.e. if some individuals in the counterfactual group prove to be too different to individuals in the participant group). However, for the survey, propensity score matching can be conducted in two steps, with the first round of matching being carried out based on administrative data to identify the counterfactual group to survey. This should minimise any reductions in counterfactual group sample sizes available for analysis.

In terms of administering the surveys of these groups, we suggest that:

It takes a sequential online and telephone approach ensuring that it is possible to benefit from the cost-effectiveness of online interviewing, but also the higher response rates that telephone interviewing achieves. A mixed-mode approach is also good for survey accessibility. The MI template includes both e-mail addresses and telephone numbers, therefore assuming both fields are well-populated, this approach will be possible.
We offer an incentive of £5 for survey completion helping us to maintain a high response and completion rate. Incentives would be paid as a shopping voucher sent electronically.
Taking this approach, we anticipate around half of the surveys being conducted online and half by phone.

For the comparison survey of non-claimants, the survey would need to be conducted entirely online as this is the only option offered by online panel providers. The incentive would be determined by the online panel provider based on their agreement with panellists.

Individuals will be starting on WorkWell between November 2024 and March 2026, therefore recruiting for the baseline survey will continue across this full period. This approach enables a gradual start to the programme. It also allows us to draw the survey sample from participants throughout the entire duration of WorkWell. This helps mitigate seasonal effects and ensures the service has adequate time to reach a steady state. It is estimated that up to 56,000 individuals will participate in WorkWell over this period and, as shown in the table above, we would only need to sample 16,000 participants as a starting sample, with 5,333 per participant group. We suggest that initially we put processes in place to receive contact details for all participants. If the volumes achieved are in line with expectations then we can draw a subset of participants at random for the survey. We assume that there will be a month’s lag in transferring participant details to us (meaning that we have a slightly imperfect baseline).

Ideally, we would also spread the comparison group surveys over the same period (to avoid our estimates of impact being influenced by seasonal effects). However, it does not feel realistic to draw a comparison group sample for claimants on a monthly basis. Our suggestion would be to draw comparison group samples quarterly; so maybe in:

January 2025
April 2025
July 2025
October 2025
January 2026
April 2026

For each draw, we would be looking to select a sample volume and profile to match that of participants who are DWP claimants from the first two months of each quarter. The process would involve identifying the relevant participants in DWP benefit records (matching on NINO or via fuzzy matching of name/date of birth/address). Then, a comparison sample would be drawn, using Propensity Score Matching, using the following variables for matching:

Benefit type
Length of claim
Health condition
Employment history
Family composition
Gender
Age

We appreciate slightly different variables may need to be used for matching depending on the benefit claimed (UC, ESA, PIP) and would anticipate refining this nearer the time. The claimant comparison group samples would be given a synthetic start date of the draw month. This would be their reference point for asking about baseline circumstances and for timing of the follow-up surveys. For the comparison group survey of non-claimants through an online panel, we would similarly need to administer the survey in a reasonable number of batches. This could perhaps be two batches/waves considering the smaller sample size.

Summary of data sources available

Reflecting on the primary and secondary data sources discussed above, the table below (Table 7.4) summarises the types of measures of interest for the impact evaluation and the data sources from which data for these could be drawn.

Table 7.4: Data Specification

Measure	Data required	Sources of data/ information for WorkWell participants
Demographic data	Gender, ethnicity, age, education level, IMD quintile, disability status.	Gender, ethnicity and education level collected in JWHD participant level MI. Age and IMD quintile can be determined from JWHD WorkWell identifier and NINO MI. Disability status to be determined through participant survey.
Change in employment status	Employment status before and after WorkWell support. Information on people moving into work, remaining in work and leaving unsuitable work.	JWHD participant level MI for WorkWell participants. RAPID/RTI datasets for longer term employment outcomes.
Change in ‘productivity’ measures	Duration of sick leave, amount of sick leave, time to return to work, presenteeism	Participant survey.
Change in earnings	Earnings per year and month.	RAPID/RTI datasets.
Change in benefits	DWP benefits caseload and spend.	RAPID/RTI datasets for benefit caseload and value.
Change in health	Reported health status before and after contact with WorkWell. Measures to be confirmed but could be EQ-5D or SF-36. Mental health can be measured using PHQ-9 or GAD-7. Health conditions experienced by WorkWell participants.	Participant survey for reported measures. JWHD participant level MI identifies the participant’s lead and secondary health related barriers to work.
Change in mental wellbeing	Reported mental wellbeing before and after contact with WorkWell, measured using either SWEMWBS (WEMWBS: 14-item vs 7-item scale) or ReQoL (ReQoL: OVERVIEW), to be confirmed.	Participant survey for reported measures. JWHD participant level MI identifies the participant’s lead and secondary health related barriers to work.
Changes in healthcare resource use	As a result of external referrals and health coaching.	Participant survey. Expert elicitation with local healthcare providers e.g. GP, physiotherapy, Talking Therapies (discussed below), for longer-term impacts. Potential for local evaluation findings/data collection subject to local plans.

The participant survey will thus be used to supplement outcome measures for the impact analysis which are not collected, or available in the administrative or MI datasets. The outcome measures which have not been identified in administrative datasets are health related outcome measures and healthcare resource use. These measures will only be available for a smaller sample of WorkWell participants and a limited counterfactual group, that can be identified and contacted (from DWP records, for WorkWell participants and claimant non-participants).

There are various health and wellbeing measures which could be included in the participant survey to determine the impact of WorkWell on the health and wellbeing of participants. The EQ-5D is the preferred measure in health economics evaluations, but it is known to be less sensitive to changes in mental health outcomes. SF-36 could be used instead and may be more sensitive to changes in the health of WorkWell participants, as it incorporates questions on both physical and mental aspects of health.

However, it is a longer survey which may impact on the cognitive burden of the participant survey. Mental health measures such as GAD-7 or PHQ-9 could be used alongside EQ-5D to evaluate the impacts on both physical and mental health. Additionally, mental wellbeing measures such as REQoL and SWEMWBS could be considered, though these are less widely used in health economics. If monetary value is to be applied to any observed changes in health status, the outcome measures must be able to be mapped to EQ-5D.

The Theory of Change suggests that in the long-term, WorkWell will reduce the burden on the NHS due to early intervention and improved health literacy. In the shorter term, the demand on local resources (NHS services such as physiotherapy and other public services such as employment advice) may increase. The WorkWell programme can provide onward referrals for participants, to access the care they need to help them back into work or to thrive in work.

Therefore, in theory, access to local service monitoring of resource data would allow the change in resource use to be evaluated. However, at this stage it is unknown what local data collection is being planned by pilot sites, and whether this would be accessible for the evaluation. Additionally, it is unlikely that non-pilot sites would be collecting the same information or be willing to share this with the evaluation team. This makes it hard to determine what the impact of WorkWell is on resource/service use. The participant survey could be used to evaluate this with a smaller sample size, if any locally collected data cannot be accessed, by asking participants and the counterfactual group responders about their use of healthcare resources. However, this will be limited to the shorter term, as the last follow up participant survey is expected to be conducted 12 months after people begin participating in the WorkWell programme.

To assess the longer-term impact on resource use, expert elicitation with local service providers such as GPs, talking therapies services and physiotherapy services could be conducted, if project resources allow. To validate this anticipated long-term impact of the programme, we could conduct two ‘expert elicitation’ focus groups with healthcare professionals who have referred participants to a WorkWell service (e.g. general practitioners, physiotherapists, Talking Therapies practitioners). A top-level literature search and review of the ongoing evaluation work would be conducted to develop a semi-structured topic guide. The questions may include, for instance, whether WorkWell participants’ health statuses and service needs had changed to date, were anticipated to change following referral and the magnitude of this change (if any). The findings of these focus groups would provide an indicative qualitative assessment of the programme’s long-term impact from the perspective of healthcare professionals.

Development of the baseline, 6 and 12-month questionnaires for the participant survey will be designed to fill in gaps in the secondary data. An indication of the broad question areas likely to be necessary to cover in the three surveys is as follows:

Table 7.5 Suggested questionnaire coverage

	Baseline	6 and 12 month survey
Screening and profiling	Yes	Yes
Overall wellbeing (physical, mental, financial)	Yes	Yes
Previous work experience / current work status	Yes	Yes
Types of benefit being claimed	Yes	Yes

If out of work:	Baseline	6 and 12 month survey
Drivers to work / undertake work-related activity (e.g., looking for vacancies at Jobcentre / in newspapers / online / in apps; approaching employers; number of jobs applied for)	Yes	Yes
Barriers to work, including confidence and motivation to find work / enter work / stay in work	Yes	Yes
Level of job search activity; work preparation skills and behaviours	Yes	Yes

If in work:	Baseline	6 and 12 month survey
Relationship with employer and overall job satisfaction	Yes	Yes
Issues around presenteeism / difficulties in performing to full ability	Yes	Yes
Disclosure of health condition and management of this	Yes	Yes

For participants only	Baseline	6 and 12 month survey
How they were referred to WorkWell and initial expectations	Yes	No
Perceptions and overall experiences of WorkWell	Yes	Yes
Structure and types of support received through WorkWell	Yes	Yes
Use of healthcare services	Yes	Yes
Perceived impact of WorkWell on movement towards work / enjoyment of work if employed and wider health/wellbeing	No	Yes

The survey can assess individuals’ quality of work. In addition to relationship with employer and overall job satisfaction (included in the table above), it may be valuable to include a measure of skills under-utilisation, that is; whether the individual has qualifications and skills more advanced than required for their current job role.

The disability employment gap measures the difference in the rate of employment for disabled people and non-disabled people. The Theory of Change indicates that the WorkWell programme is expected to reduce the disability employment gap. The Measuring disability for the Equality Act 2010 harmonisation guidance – Government Analysis Function describes an individual as disabled if they have a health condition which ‘is expected to last 12 months or more’ and ‘reduces their ability to carry out day-to-day activities, a little or a lot’.

An impact on the disability employment gap could be estimated using baseline disability status from the Annual Population Survey (if access is gained), and employment outcomes, compared to at 12 months. If the disability employment gap decreases more or increases less in WorkWell areas than in non-WorkWell areas, this would imply that the WorkWell programme will narrow the employment disability gap. However, it may be the case that not enough people use the WorkWell programme to have a measurable impact on the disability employment gap. If access to the Annual Population Survey is not gained, it might not be possible to estimate the change in the disability employment gap due to the WorkWell programme. Using data from the participant survey sample alone, only inference on the impact on benefit claimants could be made, rather than the wider impact due to the WorkWell programme.

Calculating impact

This section outlines the potential statistical approaches to take in the evaluation of impact. It also discusses the feasibility of each approach, given the options for data acquisition and the creation of a counterfactual.

Different statistical approaches can be used to form a counterfactual group and estimate the impact of WorkWell on the outcomes of interest. Due to the data available for the evaluation, not all approaches are applicable for all subgroups identified. The approaches which have been explored for use in the analysis are described in this section.

Approaches to defining the comparison group

A synthetic control group could potentially be formed using data from administrative datasets such as RAPID, for use in the statistical analysis. A synthetic control group is formed from a pool of other potentially similar population groups. Data on relevant outcome predictors is used to assign weights to areas in the synthetic control group. This is so that prior to the intervention, the weighted synthetic control group matches the trend in the intervention group. After the intervention is delivered, the weighted average outcome in the synthetic control group can be compared to the outcome in the intervention group.

Aggregate data from non-WorkWell ICB areas would be used to create the pool of control groups. Relevant outcome predictor variables will be used to create the weights to apply to the control group studies in order to generate a weighted synthetic control group which had an identical outcome measure trend as the WorkWell pilot areas. This approach requires outcome measures to be evaluated at multiple time points before and after the intervention, to evaluate trends.

Therefore, for the WorkWell evaluation, this approach only allows for the use of variables in administrative data to be used in the formation of the synthetic control group. This is because the other sources of data identified only contain outcome measures at two time points. Predictors used for weighting would be identified once the data become available, to empirically choose variables that most-strongly predict outcomes. From the administrative databases, this could include earnings and employment status. However, since multiple data points are needed before and after the intervention, outcomes which will only be in the participant survey, such as health outcomes, would not be able to be used, as these will not include any data points before the intervention. Therefore, pre-intervention trends cannot be determined, and a synthetic control group cannot be formed.

This additionally causes issues for the non-claimant subgroup, since administrative datasets available to DWP will not contain eligibility information for this subgroup. Therefore, this approach would only be appropriate for the benefit claimants subgroup. There is a risk that the non-WorkWell ICB areas will be too different and that it won’t be possible to form a pre-intervention trend match from the administrative data available, whether this is done for each pilot site or for all sites combined. This approach is therefore not recommended due to the inability to evaluate non-claimants and risks associated with forming synthetic control groups with the available data.

Propensity score matching could be used to identify the closest match of comparison for WorkWell participants by controlling for variables that would predict WorkWell participation, matching each WorkWell participant to a non-participant. This would be followed by statistical testing to determine the impact of the intervention. Propensity score matching could be used for both the benefit claimant and non-claimant group, with different datasets used for both.

Since the benefit claimant group has more variables available, including health condition information for some, a more robust group may be able to be formed than for non-claimants. Propensity score matching would rely on there being a sufficient number of non-participants with characteristics similar enough to participants in the available datasets. Additionally, if any unobservable characteristics exist which are strong predictors of both WorkWell participation and outcomes, propensity score matching will provide biased impact estimates. This is particularly a problem for the non-claimant comparison group, which would have to be matched on characteristics in RAPID, which will not include health condition information for non-claimants.

For the survey participant and counterfactual group, propensity score matching can be conducted in two steps. First, matching can be carried out based on administrative data to identify the counterfactual group to survey. Secondly, after the survey is completed, a second round of propensity score matching can be carried out to match participants and counterfactuals based on variables collected in the survey. This approach is likely to give more robust estimates of impact, as variables which are strong predictors or WorkWell participation and outcomes, such as health related outcome measures, are not available in the administrative data.

A simpler approach to matching can also be used to define a counterfactual group which will then be used in a regression analysis. Variables available in administrative data, such as age, gender and employment status can be used to form a counterfactual group that have similar characteristics to the intervention group. This matching step would allow the regression to generate more relevant results, by removing individuals who are not a good match to the intervention. For example, age can be used to ensure a similar mix of working age individuals are included in the counterfactual group, since earnings tend to increase with age.

Approaches to establishing effect size

Regression analysis can be used to determine the impact of the WorkWell programme. A generalised linear mixed model would suit the data available, which has a mix of continuous and binary variables, such as earnings and employment, respectively. This approach also does not require propensity score matching to be carried out to determine a counterfactual group, as any imbalances between the control and intervention group are adjusted for in the analysis. Multiple independent variables can simultaneously be tested to determine the impact of WorkWell, whilst controlling for the effect of other independent variables that may be different between the intervention and counterfactual group. However, a counterfactual group which has been matched on some variables, such as gender and age, can be used within this regression approach to remove individuals from the counterfactual group which are quite different to the WorkWell intervention group.

Difference-in-difference analysis is an option for the analysis. This statistical technique compares the intervention and comparator groups before and after the intervention, to determine the treatment effect. Panel data, which includes observations over time for the same individuals, is required for this evaluation design. The main assumption underlying difference-in-difference estimation is of parallel trends. That is, in the absence of the intervention, outcomes in the intervention and counterfactual groups would follow the same trend.

Matching can also be done in addition to difference-in-difference when defining the counterfactual group, but it is not a necessary step for this approach. This design may be used for different sub-groups in the analysis. The whole WorkWell population could be evaluated if access to the Annual Population Survey is obtained, as it is a panel dataset with variables to determine eligibility for the WorkWell programme for the counterfactual group. Difference-in-difference analysis can also be used to estimate outcomes for benefit claimants only, by comparing to the counterfactual group formed using the RTI database. This will have longer term employment and benefit outcomes. The outcomes from the participant survey can also be evaluated using this design. Since difference-in-difference analysis requires data for individuals at two points in time, this approach can be used for all sub-groups in the analysis.

Another option for the evaluation design is conducting an interrupted time series analysis. Interrupted time series analysis looks at the trend in an outcome of interest over time, before and after the implementation of WorkWell. This approach assumes that the trend in outcomes before the implementation of WorkWell would continue in the absence of the programme. By comparing the expected outcomes with the observed outcomes after the implementation of WorkWell, the impact of the intervention could be estimated. This approach also allows for a counterfactual group to be incorporated into the regression, to incorporate outcomes after the implementation of the WorkWell pilot. This is a controlled interrupted time series design.

This analysis would be able to evaluate short- and long-term impacts, by examining the change in the outcome at the point of the intervention and also the trend. For the WorkWell programme, a multiple baseline design could be used to account for participants starting the programme at different times. However, since this design relies on multiple observations on the outcomes of interest before the intervention, this limits the analysis to outcomes which are available in routine datasets available to DWP.

Benefit claimant outcomes could be evaluated using this approach, but not outcomes for non-claimants since measures which will be obtained from the Annual Population Survey for the counterfactual group or participant survey will not have enough observations before the intervention to establish trends in outcomes. The participant survey will not collect information before individuals participate in WorkWell, so no pre-intervention trends can be established. Similarly, the Annual Population Survey only contains two data points, one at ‘baseline’ and one 12 months later. The data necessary to carry out this approach are only available for some subgroups, limiting where in the analysis this can be used.

If there are problems accessing the Annual Population Survey, it will be difficult to determine a counterfactual group for WorkWell participants who are not benefit claimants. If a counterfactual cannot be determined for non-claimants, the evaluation for this group will be limited to a before and after analysis, using an interrupted time series analysis approach with data from administrative datasets. This will not be able to determine the impact due to the WorkWell intervention alone, but it would give an idea of what happens to WorkWell participants after the programme.

The preferred approach for this evaluation would be regression analysis with simple matching to identify a counterfactual group. Ideally, a controlled interrupted time series would be used to determine the impact of WorkWell in both the short and long term. However, due to the data necessary for this approach, this can only be conducted for the benefit claimant group. For the remaining sub-groups which only have data at two points in time for the intervention and counterfactual groups, an approach of simple matching followed by a difference-in-difference would be preferred.

Analysis

The impact analysis will focus on the outcome measures described in Table 7.6. Descriptive statistics and outcomes will be reported from the analysis for any sub-groups evaluated. Due to the likely approach to the counterfactual being different for benefit claimants and non-claimants, a different evaluation design may have to be chosen for the two groups, or different sets of outcomes could be considered. Therefore, it may not be possible to report the impact on outcomes for all WorkWell participants as a whole, and it may be necessary to report as different subgroups.

Additionally, it is possible to take a sub-group analysis approach to evaluating different approaches to the WorkWell programme taken by pilot sites. Pilots could be grouped by the way they deliver the WorkWell programme to evaluate if the approach taken has an impact on participant outcomes. If there are no commonalities between pilots, making grouping for sub-group analysis difficult, a case study approach could be taken to compare the approach in specific pilots.

Early delivery plans indicate that there may be a difference in the targeted population for the programme between pilot sites. Delivery plans indicate that WorkWell services may be targeted based on different types of health conditions, duration of time out of work, demographic characteristics, disadvantaged groups and employment characteristics. Comparing pilot sites targeting different sub-populations may provide insights on the effectiveness of the programme for different groups of people. Ways to group pilot sites together or deciding which pilots to evaluate on a case study basis, would be determined at a later stage, when more information is available about the way pilots are providing the WorkWell service locally.

The Theory of Change indicates that some outcome measures may change over time from the beginning of WorkWell participation. For example, resource use of public services, including healthcare, may initially increase due to additional referrals and signposting from the WorkWell programme. However, in the long term, as health literacy improves, WorkWell is expected to reduce the burden on the NHS. Therefore, it will be important to report outcomes at different time points, and ensuring the maximum possible time period (‘horizon’) is considered to capture the impact of the earliest WorkWell participants.

It is recognised that some sites are already implementing similar programmes to WorkWell, and will be adapting those for the WorkWell programme. Therefore, a limitation of the analysis may be that the potential for impacts may be lower in these areas. It is advised that the quantitative impact evaluation is triangulated with the qualitative evaluation, to provide narrative and context for the scale of impacts observed. The evaluation’s ability to do this will be affected by the scale of the qualitative process evaluation research that can be resourced (see Chapter 6).

Summary of suggested approach to impact evaluation

Based on the information outlined above, we suggest that we consider the impact evaluation at 3 different levels:

All WorkWell participants
WorkWell participants who are benefit claimants
WorkWell participants who complete the participant survey

The outcome measures which could be evaluated for these participant groups will be different, due to the ability to acquire data. Table 7.6 outlines the outcome measures which could be evaluated for each group and our suggested data sources for each. The most complete evaluation in terms of the outcomes assessed is likely to be survey participants. Additionally, these groups will be evaluated over different time horizons, due to data collection frequency and length of follow-up. Figure 7.1 shows how the participant and comparison groups have been identified, depending on eligibility, geographical area and individual characteristics. Figure 7.2 shows the main dataset for each participant and counterfactual group, and which outcomes can be evaluated for them.

Table 7.6: Proposed evaluation groups, data sources, and outcome measures

WorkWell participant group	Primary data source	Outcome measures	Time horizon
All WorkWell participants	RAPID	Employment status; Earnings; Benefit (caseload and spend)	Length of the evaluation
All WorkWell participants	Annual Population Survey	Employment status; Earnings; Benefit (caseload, not spend); Sick leave; Health condition	Baseline and 12 months due to follow-up in the Annual Population Survey
WorkWell participants who are benefit claimants (or have been claimants previously)	RTI	Employment status; Earnings; Benefit (caseload and spend)	Length of the evaluation
WorkWell participants who complete the participant survey	Participant survey	Outcome measures which will be included in the survey are to be confirmed, but health measures and resource use should be included as a minimum as these are not in any other datasets identified.	Baseline, 6 and 12 months

Figure 7.1 The origins of the subgroups used for the WorkWell groups and counterfactual groups

Dotted lines represent individuals who are only represented in the ‘All WorkWell participants’ group and not the intervention/comparator subgroups.

Participants in the flowchart are sifted based on their eligibility, location, participation, and benefit status. Those eligible for the WorkWell pilot will be individuals with a health condition or disability that is impacting either on their ability to find work or their ability to perform in their current job. If an individual is not eligible for WorkWell, they are excluded at the first sift. If an individual is eligible, the next sift will identify if the individual’s home address is in a WorkWell area, and/or if their Local jobcentre plus (JCP) or General Practice (GP surgery) is in a WorkWell area. If the participant doesn’t live in a WorkWell area, or if their JCP or GP are not in a WorkWell area, they will be eligible to form part of the counterfactual group. If the individual has completed the control group survey they will form part of the counterfactual group of non-WorkWell participants who complete the control group survey. If they don’t complete the counterfactual group survey but claim benefits, they will form part of the counterfactual group of non-WorkWell participants who are benefit claimants. If they don’t claim benefits they will also be included in the counterfactual group of all non-WorkWell participants. If an individual lives in a WorkWell area, or their JCP or GP is in a WorkWell area but they don’t participate in WorkWell, they are excluded from the sample. However, if they do participate in WorkWell and have completed the participant group survey, they will form part of the treatment group of WorkWell participants who complete the participant group survey. If they don’t complete the survey and don’t claim benefits they will form part of the WorkWell participant treatment group. If the individual doesn’t complete the participant survey but does claim benefits they will form part of the treatment group of WorkWell participants who are benefit claimants.

Figure 7.2: Datasets and outcomes for each of the WorkWell and counterfactual groups.

All non-WorkWell participants in these subgroups are also counted in the ‘All Non-WorkWell participants’ master group

** All WorkWell participants in these subgroups are also counted in the ‘All WorkWell participants’ master group

Participant and counterfactual groups are constructed through either participant MI data, the annual population survey, the participant survey, Registration and Population Interactions Dataset (RAPID) or Real Time Information (RT)I data. Firstly, all WorkWell participants can be constructed through participant MI data, which will be used to identify any changes in employment status, and RAPID data, which will collect data on changes in employment status, changes in benefits status/claims and changes in earnings. The counterfactual group of non-WorkWell participants can be constructed through the annual population survey, which collects data on sick leave, changes in employment status, changes in benefits and changes in earnings. If a participant belongs to the WorkWell participant group who complete the participant group survey, or if they belong to the counterfactual group of non-WorkWell participants who complete the counterfactual group survey, they can be constructed through the participant survey. The participant survey can be used to collect data on changes in healthcare use, changes in productivity measures, changes in health, changes in mental wellbeing, sick leave and changes in employment status. The group of WorkWell participants who are benefit claimants and the counterfactual group of non-WorkWell participants who are benefit claimants can be constructed using RAPID and RTI data. RTI data will identify changes in employment status, changes in benefits and changes in earnings.

Impact on systems change in the WorkWell areas

To complement the impact and process evaluations we are likely to need a component that addresses the WorkWell evaluation from a whole system perspective. This would:

Address the developmental nature of individual WorkWell pilots, i.e. that they will be developing and adapting during the evaluation
Acknowledge that they are interventions in complex social systems each with different actors and contexts
Contribute to the aim of WorkWell pilots to drive systems change in the support that individuals receive to gain and remain in work

Essentially, the impacts that are found to arise from WorkWell will be a result of complex causality. We suggest using a combination of Participatory Systems Mapping and Qualitative Comparative Analysis.

Participatory Systems Mapping is the foundation of good systems understanding and complexity sensitive analysis and remains central. Participatory Systems Mapping should be run early in the evaluation delivery, with workshops then being reconvened with further analysis once delivery has matured, late 2025, and again around the end of pilot funding, early 2027, to gain insight to the systems change over time.

Qualitative Comparative Analysis (QCA) would build on the Participatory Systems Mapping, Theory of Change and other evaluation evidence to generate insight into what configurations of factors in the intervention’s delivery and context are associated (or not) with success.

Value added of whole system approach

In the review of evaluations of similar interventions (see Chapter 3) only one study was found that reported on systems change. As noted in the Whole Systems Change section on the WorkWell Pilots (see Chapter 4) an aspiration is indicated that WorkWell will support system-wide change. The work of this evaluation will add value by explicitly exploring how systems can be strengthened, and systems change achieved within the timescale of the pilots and ideally sustained following them. Value will also be added methodologically through contributing to the currently limited number of studies exploring whole system approaches.

The whole system approach will complement and help weave together insights from the impact and process studies extracting further value from the data collected. It will provide complexity-sensitive analysis that can go further than the ‘individual’ (statistical) impact assessment, the derived programme level impact and/or qualitative analysis of the mechanisms for achieving change. The impact study can only look at impact on individuals. Most counterfactual impact studies are designed to examine the impact of an intervention on individuals (and at aggregate levels). They are less suited to examining specific causality, especially in complexity sensitive contexts. The Participatory Systems Mapping and further complexity analysis can extract further value from impact data and results to examine system change. Qualitative Comparative Analysis (QCA) specifically examines complex configurations, built on knowledge drawn from the Participatory Systems Mapping and evidence drawn from process and impact evaluations, to explain differential impacts.

In the wider evaluation picture, coupling complexity approaches with consideration of how the 15 areas are learning, adapting, and capturing and sharing insight into what works to create impact in different circumstances, will give a valuable synthesis of process and impact evidence with complexity sensitivity. Actionable insights for WorkWell pilots should result from the whole system approach, derived from both the analysis done for it and the involvement of participants in the process.

Whole System Approaches and Complexity

The causality behind the outcomes (rather than if, or what, impact occurs) is the main focus of complexity sensitivity methods in general, for example, the combination of context, mechanism and outcome in realist evaluation, the configuration of factors associated with outcomes in QCA, the complex causality behind shifting (Bayesian Belief Networks) likelihoods with Outcome Likelihood Causal Analysis (OLCA), or understanding the system as a whole (PSM - Participatory Systems Mapping). Our approaches are choices building on the approaches articulated in the CECAN-authored supplementary guidance to the HMT Magenta Book Handling Complexity in Policy Evaluation.

We explain further below, the two specific methods that seem most suitable for this evaluation, describing what they are, the rationale for using them and practicalities of delivery in terms of resources and inputs required to conduct them.

Participatory Systems Mapping

To respond to the evaluation questions there is a need to understand the complexity of the ‘whole system’, and its relevance to changing processes and achieving outcomes and impacts (in the light of varying delivery of the WorkWell interventions and varying local contexts). We are proposing to use Participatory Systems Mapping to do this.

What is Participatory Systems Mapping?

Participatory Systems Mapping brings stakeholders together to, with facilitation, collaboratively develop a causal system map of an issue or challenge of interest. The size and complexity of the mapping process varies according to evaluation need and resources, but typically, a small group of (c.8-12) stakeholders are convened and develop the causal system map of an issue over the course of a half to full-day, in-person workshop with a follow-up, online session. As well as the output of a systems map, the process generates rich conversation and qualitative data on how a system works and factors affecting delivery of desired outcomes.

Participatory Systems Mapping is an engaging process for those involved, and the resulting system map and accompanying narrative can be used by all participants involved to think strategically and practically about how to better steer, develop and evaluate the system they work in. In particular, it can help align participants to a common shared objective and in understanding the perspectives of different actors in the system. It should be noted the mapping process maps the system context and the environment of delivery first, adding interventions at a later stage. This means it can avoid a programme-centric approach overlooking the context of operation and means it complements Theory of Change processes.

In terms of resource and inputs to the process, the key input is access to stakeholders who can bring a diversity of perspectives on the system of interest, and clarity on the ‘focal’ problem, in order to explore it from a systems perspective. The process can be conducted rapidly, but the greater the involvement of stakeholders the more the map and subsystems within it can be refined and actions from it explored.

Repeating Participatory Systems Mapping to understand systems change

Participatory Systems Mapping generates a systems map that is a snapshot of the system at a moment in time, typically, the system ‘as it is now’. This can be analysed by inspection and/or formal network analysis to identify the strengths and weakness in the system. Areas that may need to be developed in future can also be identified, to deliver desired outcomes – the system ‘as desired’.

To understand system change achieved by WorkWell, we can revisit the Participatory Systems Mapping in the same pilot areas over time, by conducting further in-person and online mapping sessions. This will allow us to explore whether and how the system is evolving over time; and the challenges of achieving system change for stakeholders within it the system.

Limitations

The Participatory Systems Mapping process has been found to be a valuable means of understanding how systems work, and engaging participants in discussions on whole system approaches. While large inclusive processes can be designed to gain wide stakeholder input, these are time and resource intensive to conduct and beyond the scope of this evaluation. We are proposing a smaller process which will be used to generate insight but will represent the views of those directly involved in the process, an intersubjective process, and not necessarily the wider stakeholders. Choice of stakeholders is an important part of the process, as is defining the focal problem. Setting too broad a focal problem doesn’t allow a meaningful exploration of the mechanisms at work and too narrow a focal problem risks missing the bigger picture.

Qualitative Comparative Analysis (QCA)

The Participatory Systems Mapping will build an understanding of the factors affecting outcomes of interest in each of the WorkWell areas, and their causal relationships. The systems maps will be specific to each area, although it is highly likely transferable insights will be generated that are relevant across all the WorkWell pilots. To build a complexity-appropriate cross-case comparison of the factors accounting for the delivery of WorkWell outcomes, we therefore propose conducting a Qualitative Comparative Analysis (QCA).

This will build on the Theory of Change and Participatory Systems Mapping elements of the evaluation, and their focus on mechanisms of impact and combine it with data generated elsewhere in the evaluation (e.g. management information and surveys). This will create a systematic cross-case comparison of factors in delivery and context that account for outcomes.

What is QCA and why is it useful?

QCA is a structured method for case-based comparative analysis. It uses qualitative data but structures it in a systematic way to allow replicable generalisation of case-specific findings and can be conducted with a small number of cases, as low as three.

As Befani (2016) notes in the report, Pathways to Change: Evaluating development interventions with Qualitative Comparative Analysis (QCA), ‘At its core, QCA requires conceptualising cases (for example, projects, or groups of projects) as combinations or “packages” of characteristics that are suspected to causally influence an outcome’. ‘Once the characteristics of the cases are known, together with their outcomes, a systematic cross-case comparison is carried out to check which factors are consistently associated with a certain type of outcome (e.g. success of the intervention) and can potentially be considered causally responsible for it. This allows for a potentially quick, simultaneous testing of multiple theories of change.’

We can create much of the required data by synthesising the process and impact evaluation findings along with other evidence. This means we can derive greater value from the data already likely to be collected.

QCA has the ability to handle complexity, particularly equifinality, that is, where multiple ‘routes’ can lead to the same outcomes. This is an extension of the notions of necessary and sufficient conditions. WorkWell has this kind of complexity inherent within its design, i.e. there are multiple routes to delivery of successful outcomes.

To conceive the QCA it is necessary to:

Confirm and understand the complex causality at work (different paths to an outcome) – system mapping is a key tool; also using process evaluation evidence and the WorkWell Theory of Change.
Identify cases to compare (e.g. groups of participants with similar outcomes or WorkWell pilots as a whole) – using process evaluation evidence and statistical analysis that will be part of the impact evaluation.
Gain good understanding of each case – using process and impact evaluation evidence.
Through the understanding of the cases, find similarities and differences in cases, that can be used for comparison – using process evaluation evidence; wider evaluation evidence, insight from system mapping.

The QCA technical step then takes the specifics of each case with the assessment of similarities and differences to establish configurative causality (combinations of causes). At this level, QCA is a method that translates data into a truth table format to compare different cases systematically and extract the causal recipes that bring about an outcome, through identifying combinations of necessary and sufficient conditions.

What is needed to conduct QCA?

The choice of how to construct size and type of ‘cases’ (units of observation for a QCA) will be a balance between the accuracy of the impact measure and the ability to meaningfully attribute ‘factor’ characteristics. The envisaged analysis would:

Agree a research question for the QCA, likely based on an outcome from the Theory of Change. This might have an organisational effectiveness focus, for example ‘Which combinations of factors account for effective integration of work and health services?’. Or, it may have a participant focus, for example ‘What factors are associated with different groups of participants experiencing similar outcomes?’
Choose ‘implementation’ units of observation. The 15 individual WorkWell pilots are the clear units of observation for system integration objectives and represent a bounded whole system in each of the areas; while groups of participants with similar outcomes within a pilot might be another option.
Synthesise a low-resolution (aggregated) impact measure for these units. The measure needs to be one in which an assessment of success and failure is possible, and which has had time to occur in the period of the evaluation.
Identification of factors whose presence or absence in the programme or context of operation may have contributed to the outcome, identified from the Theory of Change and system mapping, and characterisation of pilots and their geography, constituent memberships, and attributes of delivery models (see Chapter 4).
Data collection/collation on factors of interest to allow scoring of each factor across each of the cases, either on a presence/absence basis (Crisp Set QCA) or strength of factor (Fuzzy Set QCA).
Carry out the complexity analysis on these units, using QCA software, with ‘factors’ being important intervention differences, contextual factors, and, importantly, the ‘system’ factors of “service delivery and partnership”, “service devolution” and other ‘system’ issues.

There is a long-standing debate in the application of QCA, on the specification of the appropriate ratio of cases to factors. For example, Marx, A., and A. Dușa (2011)^{[footnote 6]} proposed benchmark tables for setting the ratio. With 15 cases (i.e. the 15 pilots), their tables indicate considering three factors and an outcome as appropriate. More recent work questions this limitation on the number of factors it is appropriate to consider (Thiem and Mkrtchyan, 2023)^{[footnote 7]}.

In this evaluation, QCA is likely to go beyond considering three factors but limitations on resource and data will prevent going too far beyond this. However, both these authors noted, and the wider literature highlights, the importance of positioning QCA findings in a strong theory-based framework and validating (or rejecting) mechanisms identified in QCA through expert input. The approach here would include several rounds of integrating computer analysis and cross-case comparison, within case analysis and ToC evidence, to generate final findings.

The QCA analysis will need to be conducted later in the study to allow time for the impact of interest to have occurred. Some consideration will need to be given to the choice of impact, to ensure it is one that is likely to occur withing the timescale of the evaluation. The detailed design and data collection requirements will need to be developed early in the study to ensure data is available for each condition of interest, ready for the QCA analysis to be done later in the evaluation. We recommend that, ideally, Participatory Systems Mapping be conducted in conjunction with QCA. This would help understand the WorkWell mechanisms of change from a systems perspective and how systems change over time. It would also identify the range of causal mechanisms accounting for success, or lack of it, in the different WorkWell areas. QCA would allow us to explore the combinations of factors affecting outcomes, and variations across the 15 pilots.

Question 13

8.  Economic Evaluation

Accepted Answer

This chapter discusses possible approaches for an economic evaluation of the WorkWell pilots, including how the cost of the WorkWell programme will be evaluated and how the impact of the programme on outcomes will be monetised.

Tables 8.1 and 8.2 outline the data necessary and available for evaluating the costs of the WorkWell programme. The costs associated with providing the WorkWell programme will be obtained from the expenditure schedule MI that pilot sites are required to provide to JWHD quarterly. A cost per participant will be determined by dividing the total cost of providing the service by the number of participants. The additional granularity provided in the MI expenditure schedule with cost categories, will allow for analysis on where costs are being incurred, and any differences between pilots.

If there is wide variation in costs per participant between pilot sites, delivery plans could be evaluated to identify differences in programmes which may drive cost differences. For example, more mature programmes may have lower set up costs than brand new programmes. An additional costs and resources survey could also be carried out with pilot sites to supplement the MI data and delivery plans. For example, by looking at the amount of JWHD funding passed over to local services to increase their capacity, and existing staff time contributed to the programme.

Additional resource is also being used by central teams to set up the WorkWell pilot, and for the continuous support provided to pilot sites. Failure to consider the costs associated with the central teams would underestimate the costs associated with setting up and running the national WorkWell programme. The additional survey work for costs and resource use could be extended to these teams, to obtain estimates of the costs associated with supporting WorkWell sites.

Table 8.1: Data specification for costs of the WorkWell programme

Data required to measure the cost of setting up and delivering the WorkWell programme	Source of data
Information on the following: Staff time required in hours (e.g. work and health coaches, local authority leads), training new and existing staff; Setting up new systems and infrastructure for referrals and appointments, time per task in hours; Community engagement to support service development, promotional materials, time in hours; Project management time in hours.	JWHD to provide funding information for pilot sites through the expenditure schedule MI; Costs and resources survey work with WorkWell pilots (and central teams) to supplement MI data where necessary.

Table 8.2: Datasets for costs of the WorkWell programme

Dataset	Overview	Advantages	Disadvantages
Expenditure Schedule MI	Quarterly expenditure information with granularity about administration, staffing, training and other costs completed by pilot sites.	Required to be completed by pilot sites so would expect timely reporting and high adherence.	May not be granular enough to determine the reasons behind differences in cost per participant. This is likely due to broad scale of cost categories at an aggregate level.

Changes in earnings and benefits will be incorporated into the economic evaluation. This could be evaluated at different time points, to determine the economic costs and benefits of the WorkWell programme in the shorter and longer term. In the short-term, earnings information will be used for benefit claimants from RTI. For non-claimants, this could only be evaluated if the Annual Population Survey is accessed, since otherwise a counterfactual group cannot be formed for non-claimants. Longer term impacts will be evaluated for claimants only.

Differences in sick leave could also be monetised by estimating the day salary of an individual, based on their earnings from their job and assigning this value to each day of sick leave. This is a measure of productivity and would be a benefit to employers and the wider society. This part of the evaluation would be limited to the survey sample, as sickness is not recorded in the databases available to JWHD.

Self-reported health measures for the survey sample group could also be included in the economic evaluation. The selected health measures will be mapped to utility values which can be converted into quality-adjusted life years (QALYs). These can be monetised at the current Green Book value of £70,000 per QALY, taken from The Green Book (2022). This would be limited to WorkWell participants who complete the participant surveys.

Changes in healthcare resource use will be included in the economic evaluation. These will be monetised using publicly available costs from sources such as the PSSRU Unit Costs of Health and Social Care 2023 Manual - Kent Academic Repository and the national cost collection, taken from NHS England » 2022/23 National Cost Collection Data Publication.

Question 14

9.  Conclusions and recommendations

Accepted Answer

The evaluation approach will seek to achieve a complete picture of WorkWell’s delivery, its outcomes, impacts and the value for money it represents. The chosen evaluation encompasses sampling from all 15 pilots including, 15 focus groups with Work Coaches and 15 focus groups with employers.

Additionally, a total of 30 interviews will be conducted with senior stakeholders across 2 separate occasions. Qualitative interviews will also be conducted with WorkWell participants. The impact evaluation will include 3 rounds of surveys amongst 5 cohorts of participants and non-participants, with 2 rounds of surveys with a non-claimant comparison group of non-participants. Participatory Systems Mapping and Qualitative Comparative Analysis will also be included. Furthermore, the Economic evaluation will include analysis of the quarterly MI expenditure schedule and a Cost and Resource survey.

Impact Evaluation

In terms of impact evaluation approaches, a Randomised Control Trial (RCT) is unlikely to be feasible as an overall design, due to the large degree of local flexibility around delivering the pilot, and uncertainty around referral routes and target populations; alongside challenging delivery timescales. We therefore recommend using a quasi-experimental design (QED) in which we identify a comparison group of individuals who closely resemble WorkWell participants. This is in order to show the outcomes that would have been achieved in WorkWell’s absence.

Within this QED approach, the participant Management Information (MI) can be used to identify WorkWell participants in other datasets and to profile WorkWell participants, allowing us to identify similar individuals to form a comparison group. The comparison group may be drawn from within WorkWell pilot areas, from a synthetic control group or from within a group of comparison areas chosen as good matches for each of the individual WorkWell pilots. We recommend the latter approach – a group of comparison areas chosen as good matches for each of the individual WorkWell pilots. It allows disaggregation of findings for individual pilots and reduces the risks associated with eligible non-participants in WorkWell areas being different from those who do participate.

The impact evaluation will examine relative change over time in key outcome measures for our treatment group (receiving WorkWell support) and our comparison group (receiving ‘business as usual’ services), by establishing measures at a baseline and at follow-up points (likely to be at 6 and 12 months on). The data for this can be drawn from secondary data and survey data. We recommend that the impact evaluation draws on both secondary data (for its larger sample sizes) and survey data (for its wider coverage of our outcome measures).

Within this, the impact evaluation may draw on multiple secondary datasets, each of which has distinct advantages and drawbacks (including differences in their coverage of our audiences of interest). We recommend that the impact evaluation draws on multiple secondary datasets, and is conducted at three levels. The first being amongst all WorkWell participants (so as to evaluate employment status, earnings and benefit caseload) Another level being amongst WorkWell participants who are benefit claimants (so as to evaluate employment status, earnings, benefit caseload and spend from Real Time Information). And the third level being amongst WorkWell participants who complete the participant survey (so as to cover a range of other outcome measures, including self-reported health and resource use). The statistical approach to calculating impact will be determined by the data sources that it proves possible to access; and different approaches may need to be applied to the three levels at which impact evaluation is conducted.

To address the complexity of the ‘whole system’ around employment and health, and how this influences outcomes and impacts achieved, ideally the evaluation would include:

Participatory Systems Mapping (to reflect on the mechanisms of change from a system perspective and explore how systems change over time)
and Qualitative Comparative Analysis (to identify the combinations of factors, or ‘causal recipes’, accounting for success, or the lack of it, in different WorkWell areas)

Economic Evaluation

For an economic evaluation, the costs associated with WorkWell may be obtained from the quarterly expenditure schedule MI that pilots will provide to the JWHD; combined with a ‘cost and resources’ survey. This will capture the use of existing resources within each pilot as well as the costs and resources used by central teams in working with the pilots. Alongside this, the economic analysis will examine and monetise, for example, changes in earnings and benefits; differences in sick leave; changes in self-reported health; and changes in healthcare resource use. We strongly recommend that an economic evaluation be included within the evaluation design.

Process Evaluation

The impact and economic evaluation could be accompanied by a process evaluation to understand how the programme has been delivered and how the pilots operate, including testing the mechanisms and assumptions in the programme’s Theory of Change (ToC). This will likely take the form of qualitative in-depth interviews and focus groups, complemented by surveys and analysis of Management Information (MI), amongst individual participants, WorkWell coaches, employers and senior stakeholders (particularly Integrated Care Board and Local Authority representatives).

There are options for the scale of the qualitative fieldwork, but we strongly recommend that a process evaluation be included within the evaluation design. This would ensure that qualitative insight into the delivery and experiences of those involved is captured. The ideal would be to adopt a case study approach, collecting data from a mix of the above audiences, for all 15 pilots. This would, if commissioned at sufficient scale, give us scope to provide qualitative findings for each pilot individually. This would therefore help provide narrative and context for the scale of impacts observed in each pilot. This may prove important, if the 15 pilots adopt individual delivery approaches.

Table 9.1: Summary of the chosen evaluation approach.

Element of evaluation	Assumptions
Process Evaluation	Assumes a process is based around sampling from all 15 pilots: one round of focus groups with coaches (15 focus groups, one per pilot) and employers (15 focus groups, one per pilot). Paired interviews or triad interviews with pilot senior stakeholders cover all 15 pilots (two rounds; 15 discussions initially and 15 discussions on the second occasion). Longitudinal participant interviews consist initially of 60 in-depth interviews, leaving 25 in-depth interviews for wave 2.
Impact Evaluation: participant survey (also covering claimant comparison groups)	A survey, amongst five distinct groups (non-claimant participants; in work DWP claimant participants and non-participants; out of work DWP claimant participants and non-participants), on three occasions (baseline of 1,600 participants per group; 6-month survey of 800 participants per group; 12-month survey of 400 participants per group). At 12 months, this allows us to detect differences of +/-6.9 percentage points between participant and non-participant samples.
Impact Evaluation: non-claimant comparison group survey	To address the lack of a non-claimant comparison group of non-participants, using an online panel to recruit non-participating individuals who in theory would be WorkWell-eligible, England-wide, and surveying them twice (a baseline of 600 interviews and a 12-month survey of 300 of the same individuals).
Impact Evaluation: impact on systems change	Participatory Systems Mapping, bringing together stakeholders from multiple pilots in facilitated sessions to develop a causal systems map, as a ‘snapshot’ of the system at a moment in time. By repeating the process, the mapping can capture how the system is evolving; the mapping would take place three times in total. QCA will be conducted alongside Participatory Systems Mapping to explore the combinations of factors affecting outcomes, and variations across the 15 pilots.
Economic Evaluation	Quarterly MI Expenditure schedule and cost and resource survey.

Question 15

Appendices

Accepted Answer

Appendix 1: Theory of Change (ToC).

Inputs

The inputs column on the left-hand side of the ToC describes the resources – funding, policy and stakeholders. They are required to deliver the key activities of the WorkWell Pilot, and which are necessary to bring about the desired outcomes and impacts.

The policy, delivery and analysis teams at the JWHD have been instrumental in the planning and delivery of the WorkWell Pilot so far, and will continue to feed into the development of the policy at a national level, learning from feedback and suggestions from Pilot sites. The WorkWell Pilot will be funded by the JWHD. To support shared learning a digital Future NHS platform will be provided. This platform will provide Pilot areas, and eventually non-Pilot areas, with the opportunity to communicate and share best practice in regard to their WorkWell delivery. The delivery of WorkWell relies on existing local resources like the workforce, stakeholder partnerships, networks, and support offers. A key aim is to integrate health and work services locally, making existing provisions crucial for the programme’s success.

The core activities column summarises how the inputs will be utilised to bring about the programme’s intended outcomes and impacts. The programme activities are broadly delivered by three key groups: the JWHD; Pilot areas; and the National Support Team. The service provided by Pilot areas is anticipated to be the most important driver of outcomes, and so this will be a key focus of the evaluation.

Before this evaluation was commissioned, the JWHD completed some work to ensure the smooth delivery of the WorkWell Pilot, including policy development (see WorkWell prospectus: guidance for Local System Partnerships - GOV.UK) and the selection of Pilot areas (after they had submitted bids for funding). The JWHD will support the wider success of the Pilot by building engagement and awareness among local systems, and providing support to pilot sites by producing advice and guidance focussed on delivery and programme management.

Pilot areas will deliver a WorkWell service which includes the first steps for service users, which is referral to the WorkWell service, assessment of need in relation to work and health, and triage to the appropriate services. This three-step process will involve the input of the required multi-disciplinary team (MDT), including the work and health coach and the learning and change manager. The Pilot area will then deliver the service to the service user, including the development of a personalised action plan, one-to-one coaching, and an onward referral network to local services.

Pilots will carry out additional activities to ensure the delivery of their WorkWell service is successful. They will create a partnership between the Integrated Care Board (ICB), Local Authority (LA) and the other local services and organisations delivering WorkWell. To support this, Pilots will create a mandated Work and Health Strategy to ensure such organisations are strategically aligned. In addition to this, Pilots will map existing service provision in their area to identify the work and health services that are already available. This will identify gaps so their WorkWell service can meet the varied needs of individuals.

In regard to supporting referrals, Pilots will engage with key referral organisations to develop referral pathways into WorkWell, ensuring individuals have access to the service and supporting Pilot areas to meet their referral targets. Pilots will also market the service to raise awareness, including the various referral pathways that can be used to access the service. Additionally, some Pilot areas have expressed intent to carry out awareness and knowledge building activities with employers and frontline staff about work and health issues. Pilots will also recruit and train a workforce to deliver WorkWell, including the mandated work and health coach and learning and change manager.

Finally, the National Support Team will support Pilot areas to design and deliver services by coordinating shared learning across the Pilot area, supporting the sharing of best practice.

Outcomes (and the outputs that will translate the activities to outcomes)

Short-term outcomes

In the short-term, by creating local partnerships, mapping existing services, and creating a Work and Health Strategy, it is anticipated that formal governance structures linking the ICB and LA teams will be established, and that new or improved working practices or relationships will be developed between local partners. It is required that all 15 Pilot sites will produce a Work and Health Strategy. For Pilot areas, this should lead to an increased understanding of local communities at risk of unemployment through ill health, increase co-operation to work towards mutual work and health objectives, and give greater clarity and accountability for local work and health objectives.

Through the development of referral pathways into the service and awareness raising around the service (including referral pathways), it is expected that up to 56,000 individuals will take up support. In the shorter term, it is predicted that this will create temporary increased demand on local services in each Pilot area, such as the National Health Service or local advice services, as service users begin to take up the support on offer.

It is intended that individuals receiving support through WorkWell will see many health and work-related benefits, such as:

Health related: Increased knowledge of how and where to seek appropriate help for work and/ or health issues, hopefully leading to increased ability to manage own health condition
Work related: Increased understanding of job roles they can successfully fulfil, increased work-related motivation, confidence and resilience, and potentially the development of new skills and qualifications

Pilot sites will recruit experienced individuals, including dedicated work and health coaches. Training will help frontline professionals understand the roles of statutory and voluntary groups, referral routes, and how to discuss work and health. In the short term, this aims to enhance understanding of local work and health support and increase confidence in related conversations.

In delivering WorkWell, some Pilot areas are planning to engage employers to help them achieve their aims of supporting people with a health condition or disability to get in and on work. For example, they might share (with participant permission) an individual’s ‘return to work’ or ‘thrive in work’ plan with an employer that includes suggestions on how they can support their employee in work. It is intended that this engagement helps employers provide workplace support and reasonable adjustments to employees with health conditions.

The National Support Team will support Pilots to design and deliver their WorkWell services. They will also set up infrastructure to encourage Pilots to share learning and best practice about delivery in their area, such as the learning and change network and the digital Future NHS Platform. It is the intention that this leads to a greater understanding of models delivered in other areas. This then links back to activities, as it is the intention that this greater understanding feeds into the agile adaptation of service design and delivery.

Mid-term outcomes

In the mid-term, it is anticipated that each pilot’s increased understanding of their local community, increased cooperation towards mutual work and health objectives, and greater clarity and accountability around such objectives, leads to improved integration and coordination of services locally. This streamlining should increase the capacity of services through the efficient allocation of resources to those most in need. Furthermore, it is anticipated that this will fill gaps and remove duplication in services. For Pilot areas, it is likely that these outcomes together will lead to a more joined up work and health landscape that is easier to navigate for populations in Pilot areas, an outcome which is anticipated to occur in the shorter- and mid-term.

It is anticipated that having a dedicated WorkWell work and health coach in post will lead, in the mid-term, to the development of learnings that could support future workforce development. Furthermore, since statutory and voluntary sector staff should have a greater understanding of the local work and health support available and how to refer to such services, it is intended that referrals between services become more effective. This will ensure people get the right help at the right time. It is anticipated that this, along with the intention that statutory and voluntary sector staff have increased confidence to have employment and health conversations, then leads to the mid-term individual outcomes.

These mid-term individual outcomes include being able to get back into suitable work if they have recently fallen out due to a health condition, to stay in work if they’re at risk of falling out of work due to a health condition, or to find alternative work if their existing employment doesn’t accommodate their health needs. It is also anticipated that individuals become more productive in work, sickness absence days reduce, general health and wellbeing improves, and that this leads to a reduction in benefit claims. It is anticipated that individual short-term outcomes (discussed above, i.e. increased knowledge of work and health support available, increased ability to manage their own health condition, and increased understanding of suitable job roles) will also lead to these mid-term individual outcomes.

In those Pilot areas engaging with employers, it is anticipated that employers’ increased ability to provide workplace support and reasonable adjustments to employees with health conditions will lead, in the mid-term, to a recognition of the benefits associated with employees having access to health support. It is intended this will ultimately increase staff retention and workforce productivity, if individuals recognise their employer is supportive to disabled people and people with a health condition.

Impacts

There are three key levels where impacts have been identified in the WorkWell ToC. At the Pilot area level, it is anticipated that local integration and joining up of work and health services will be sustained beyond the funding period of WorkWell. It is also anticipated that through delivering WorkWell, sites will have experience of devolved decisions around work and health, which will be beneficial considering the increased devolution of work and health support that is anticipated. It is intended that the learnings around the future work and health workforce development (from the dedicated work and health coach) will lead to some learning about which core functions are most effective to support future workforce development.

At the national level, it is anticipated that the individual and employer benefits achieved in the short- and mid-term lead to sustained work of two or more years for people with health conditions or a disability. In turn, this reduces economic inactivity in pilot areas. Ultimately, this should reduce the DWP benefits caseload and spend and reduce the disability employment gap. From a health perspective, while it is anticipated that local services will initially see increased demand, it is intended that improved health through WorkWell ultimately leads to a reduction in demand for health services, reducing the burden on the NHS. Furthermore, it is anticipated that shared learnings around WorkWell delivery, facilitated by the National Support Team, ensures that work and health policies, programmes and systems are evidence-based in the longer term.

Where Pilot areas are engaging with employers, at the employer level, it is intended that their engagement with joined up work and health support will lead to them being more responsive to work and health needs. This ultimately improves workplace culture in their organisation. Ideally, this would lead to economic benefits such as improved retention and the recruitment of talent.

Although the evaluation will aim to capture data to evidence impacts, it may not be possible to include all of these within the scope of the evaluation. It may also be that some of these impacts are not achieved within the timescales of the evaluation.

Assumptions

The process of developing the ToC identified six key assumptions that will need to hold true, in order for the outcomes and impacts shown in the ToC to occur. Understanding the assumptions underpinning the programme’s theory will also help us to understand why an outcome may not have been achieved. The six key assumptions are that:

There will be sufficient volumes of participants eligible to be referred and willing to engage with the programme.^{[footnote 8]} Across the 15 pilots the assumption is that the total number of target beneficiaries will be 59,000.
WorkWell pilot sites can recruit sufficiently skilled staff to deliver new roles created by the service.
Pilot sites are able to form and mobilise partnerships within the project set-up timescales and have the capacity and skills to deliver on their bid.
Professionals targeted to refer individuals will have the time and motivation to help people to access the programme, and support participants to complete their actions in a timely manner.
Participants will be motivated to complete actions identified in their personalised plans in a timely manner.
There are services with sufficient capacity within local areas, that will be able to accept onward referrals and provide the support needed by participants.

Appendix 2: Evidence Review – References to Included Studies

Aili K, Hellman T, Svartengren M, Danielsson K. Including a three-party meeting using the demand and ability protocol in an interdisciplinary pain rehabilitation programme for a successful return to work process. Int J Environ Res Public Health. 2022;19(24):16614.
Berglund E, Anderzén I, Andersén Å, Carlsson L, Gustavsson C, Wallman T, et al. Multidisciplinary intervention and acceptance and commitment therapy for return-to-work and increased employability among patients with mental illness and/or chronic pain: a randomized controlled trial. Int J Environ Res Public Health. 2018;15(11):2424.
Bernaers L, Cnockaert E, Braeckman L, Mairiaux P, Willems TM. Disability and return to work after a multidisciplinary intervention for (sub)acute low back pain: a systematic review. Clin Rehabil. 2023;37(7):964-974.
Brendbekken R, Eriksen HR, Grasdal A, Harris A, Hagen EM, Tangen T. Return to work in patients with chronic musculoskeletal pain: multidisciplinary intervention versus brief intervention: a randomized clinical trial. J Occup Rehabil. 2017 Mar;27(1):82-91.
Brendbekken R, Vaktskjold A, Harris A, Tangen T. Predictors of return-to-work in patients with chronic musculoskeletal pain: a randomized clinical trial. J Rehabil Med. 2018;50(2):193-199.
Centre for Regional and Economic Knowledge. Working Well Early Help: annual report 2022. Available at: https://www.greatermanchester-ca.gov.uk/media/6763/wweh-2022-annual-report.pdf, (accessed 14 June 2024).
Cullen KL, Irvin E, Collie A, Clay F, Gensby U, Jennings PA, et al. Effectiveness of workplace interventions in return-to-work for musculoskeletal, pain-related and mental health conditions: an update of the evidence and messages for practitioners. J Occup Rehabil. 2018;28(1):1-15.
Dudley C, McEnhill L, Steadman K. Is welfare to work, working well? Improving employment rates for people with disabilities and long-term conditions. 2016. Available at: https://englishbulletin.adapt.it/wp-content/uploads/2016/06/405_Work-Programmes.pdf, (accessed 12 June 2024).
Fisker A, Langberg H, Petersen T, Mortensen OS. Effects of an early multidisciplinary intervention on sickness absence in patients with persistent low back pain - a randomized controlled trial. BMC Musculoskelet Disord. 2022;23(1):854.
Gloster R, Marvell R, Huxley C. Fit for work: final report of a process evaluation. 2018. Available at: https://assets.publishing.service.gov.uk/media/5b23ca4940f0b634b73dbf5b/fit-for-work-final-report-of-a-process-evaluation.pdf, (accessed 13 June 2024).
Ibrahim ME, Weber K, Courvoisier DS, Genevay S. Recovering the capability to work among patients with chronic low Back pain after a four-week, multidisciplinary biopsychosocial rehabilitation program: 18-month follow-up study. BMC Musculoskelet Disord. 2019;20(1):439.
Langagergaard V, Jensen OK, Nielsen CV, Jensen C, Labriola M, Sørensen VN, et al. The comparative effects of brief or multidisciplinary intervention on return to work at 1 year in employees on sick leave due to low back pain: a randomized controlled trial. Clin Rehabil. 2021;35(9):1290-1304.
Learning and Work Institute. Greater Manchester Working Well: early impact assessment. 2018. Available at: https://www.gov.uk/government/publications/greater-manchester-working-well-early-impact-assessment, (accessed 18 June 2024).
Learning and Work Institute. Learning from four years of Working Capital: informing the development and delivery of future services. 2019. Available at: https://learningandwork.org.uk/wp-content/uploads/2020/05/Learning-from-Four-Years-of-Working-Capital.pdf, (accessed 14 June 2024).
Learning and Work Institute. Working Capital: final evaluation report. 2021. Available at: https://learningandwork.org.uk/resources/research-and-reports/working-capital-final-evaluation-report, (accessed 14 June 2024).
Moll LT, Jensen OK, Schiøttz-Christensen B, Stapelfeldt CM, Christiansen DH, Nielsen CV, et al. Return to work in employees on sick leave due to neck or shoulder pain: a randomized clinical trial comparing multidisciplinary and brief intervention with one-year register-based follow-up. J Occup Rehabil. 2018;28(2):346-356.
Momsen AH, Stapelfeldt CM, Nielsen CV, Nielsen MB, Aust B, Rugulies R, et al. Effects of a randomized controlled intervention trial on return to work and health care utilization after long-term sickness absence. BMC Public Health. 2016;16(1):1149.
Pedersen P, Nielsen CV, Jensen OK, Jensen C, Labriola M. Employment status five years after a randomised controlled trial comparing multidisciplinary and brief intervention in employees on sick leave due to low back pain. Scand J Public Health. 2018;46(3):383-388.
Pedersen KKW, Langagergaard V, Jensen OK, Nielsen CV, Sørensen VN, Pedersen P. Two-year follow-up on return to work in a randomised controlled trial comparing brief and multidisciplinary intervention in employees on sick leave due to low back pain. J Occup Rehabil. 2022;32(4):697-704.
Sabariego C, Coenen M, Ito E, Fheodoroff K, Scaratti C, Leonardi M, et al. Effectiveness of integration and re-integration into work strategies for persons with chronic conditions: a systematic review of European strategies. Int J Environ Res Public Health. 2018;15(3):552.
Schepens C, Bouche K, Braeckman L, Rombauts P, Linden P, Parlevliet T. The multidisciplinary biopsychosocial rehabilitation programme for patients with chronic spinal pain: outcomes with work status as the primary focus. J Rehabil Med Clin Commun. 2024;7:5250.
Skagseth M, Fimland MS, Rise MB, Johnsen R, Borchgrevink PC, Aasdahl L. Effectiveness of adding a workplace intervention to an inpatient multimodal occupational rehabilitation program: a randomized clinical trial. Scand J Work Environ Health. 2020;46(4):356-363.
Vogel N, Schandelmaier S, Zumbrunn T, Ebrahim S, de Boer WE, Busse JW, et al. Return-to-work coordination programmes for improving return to work in workers on sick leave. Cochrane Database Syst Rev. 2017;3(3):CD011618.

Appendix 3: Evidence Review – Search Strategy

A MEDLINE (OvidSP) search strategy was designed to identify studies of integrated multidisciplinary programmes, to support those with long-term health conditions or disabilities. It is presented in Figure 1. The search strategy is highly targeted and pragmatic to suit the project resources and context.

Search strategy for Ovid MEDLINE:

1 exp chronic disease/ (644704)

2 exp disabled persons/ (75252)

3 exp pain/ (476484)

4 exp “wounds and injuries”/ (1035049)

5 (disabl* or disabilit* or incapacit* or handicap*).ti,ab,kf. (337295)

6 ((long-term or longterm or long-standing or longstanding or continu* or endur* or persist* or lifelong or life-long) adj5 (health* or pain* or injur* or sick*)).ti,ab,kf. (120320)

7 ((chronic* or incurable or recur* or reoccur* or re-occur) adj5 (condition or disease* or disorder* or syndrome* or health or pain* or ill)).ti,ab,kf. (717121)

8 or/1-7 (2907629)

9 return to work/ (3819)

10 rehabilitation, vocational/ (9823)

11 ((work* or employ* or unemploy* or job* or labour* or labor* or vocation* or occupation) adj5 (return or back* or engag* or reengag* or capacit* or incapacit* or participat* or enter* or entry or reenter* or reentry or resum* or sustain* or retain* or retention or maintain* or continu*)).ti,kf. (23421)

12 ((work* or employ* or unemploy* or job* or labour* or labor* or vocation* or occupation) adj5 (health or wellbeing or well being or wellness or welfare or rehab* or fit or fitness)).ti,kf. (79235)

13 (vocation* adj2 rehab*).ti,kf. (1681)

14 *employment/ (27385)

15 *unemployment/ (3943)

16 *productivity/ (5762)

17 productiv*.ti,kf. (21601)

18 *absenteeism/ (4505)

19 *presenteeism/ (402)

20 (absenteeism or absenteism or presenteeism).ti,kf. (3084)

21 (absen* adj3 (rate or rates)).ti,kf. (97)

22 *sick leave/ (3871)

23 (sick* adj5 (absen* or leave* or list* or day*)).ti,kf. (4449)

24 ((disabilit* or illness* or medical) adj5 (absen* or leave*)).ti,kf. (970)

25 or/9-24 (163950)

26 delivery of health care, integrated/ (14550)

27 intersectoral collaboration/ (2629)

28 *patient care teams/ (29375)

29 ((integrat* or coordin* or co-ordin* or combin* or merg* or assimilat* or cohesiv* or collaborat* or communicat* or intersector* or inter-sector) adj5 (service or program* or strateg* or support* or intervention* or resourc* or team* or care* or caring)).ti,kf. (48666)

30 ((multidisciplin* or multi-disciplin* or interdisciplin* or inter-disciplin) adj5 (service or program* or strateg* or support* or intervention* or resourc* or team* or care* or caring or stakeholder*)).ti,kf. (11782)

31 ((integrat* or coordin* or co-ordin* or combin* or merg* or assimilat* or cohesiv* or collaborat* or communicat) and stakeholder).ti,kf. (947)

32 ((workplace* or work-place* or employer) adj5 (collabor or intervention*)).ti,kf. (948)

33 ((work* or employ) and (predict or barrier* or facilit*)).ti. (7443)

34 (service* or program* or strateg*).ab. (2911248)

35 33 and 34 (2154)

36 26 or 27 or 28 or 29 or 30 or 31 or 32 or 35 (99862)

37 *occupational health services/ (8034)

38 ((occupation* or employee) adj3 (health service or assist*)).ti,kf. (1331)

39 37 or 38 (8512)

40 36 and 39 (255)

41 8 and 25 and 36 (802)

42 (fit for work or working well).ti,ab,kf. (647)

43 8 and 42 (100)

44 40 or 41 or 43 (1117)

45 exp animals/ not humans/ (5227826)

46 (news or editorial or case reports).pt. or case report.ti. (3372920)

47 44 not (45 or 46) (1077)

48 limit 47 to english language (971)

Key to Ovid symbols and commands:

* Unlimited right-hand truncation symbol

ti,ab,kf. Searches are restricted to the Title (ti), Abstract (ab), Keyword Heading Word (kf).

adj Retrieves records that contain terms next to each other (in the shown order)

adjN Retrieves records that contain terms (in any order) within a specified number (N) of words of each other

/ Searches are restricted to the Subject Heading field

exp The subject heading is exploded

* The subject heading is searched as a major descriptor only

pt. Search is restricted to the publication type field

.fs. Term is searched as a floating subheading

or/1-7 Combines sets 1 to 3 using OR

Appendix 4: Evidence Review – Flow Diagram

Flow diagram showing the number of records included and excluded at each stage of the evidence review:

Firstly in the identification stage, 1,426 studies were identified from databases, with an additional 8 references being identified from other sources, including 7 from citation searching and 1 from grey literature. The second step in the identification phase included removing 58 duplicate studies. Following this, a screening stage took place with 1,376 references being screened, of which 1,217 references were excluded. Next, the remaining 159 references were sought for retrieval, of which 26 references were not retrieved. The remaining 129 references reported in 133 papers were assessed for eligibility. Of these studies, 110 were excluded due to; 3 including the wrong outcome; 23 including the wrong intervention; 36 including the wrong study design; 17 including the wrong patient population and 31 papers not being prioritised due to them being published pre-2014 or a shorter-term absence. Finally, a total of 19 studies, across 23 papers were included in the review.

Appendix 5: Identifying the counterfactual group

A number of different options for identifying the counterfactual group were considered, as follows:

Data collected by unsuccessful pilot sites
Excluded areas from successful pilot sites
Those who do not take up the programme after referral
Those refused after a referral
Those referred who are put on a waiting list to access services
UC claimants from the RTI dataset
Eligible people from DWP administrative data in non-WorkWell pilot areas
Eligible people from the Annual Population Survey in non-WorkWell pilot areas

Each option was appraised, taking into consideration the availability of data, how representative the data are expected to be, and the feasibility of implementation within the WorkWell programme. The table below shows a summary of the options appraisal; and further discussion follows this.

Counterfactual group options

Counterfactual group option	Representativeness of WorkWell participants	Availability of data	Feasibility of the approach	Comments
Data collected by unsuccessful pilot sites	Medium	Low	Unknown at this stage	At this stage it is unknown what data might be collected, and if data would only be collected from people accessing other work and health programmes.
Excluded areas from successful pilot sites	Low	Low	Low	Very different population groups in the excluded areas to the included areas.
Those who do not take up the programme after referral	Low	Low	Low	The reason for not taking up the referral may affect outcomes being evaluated. This is also likely to be a small sample size.
Those refused after a referral	Low	Unknown at this stage	Low	The reason for being refused into the programme will likely make this group different to participants. This group is also expected to have a small sample size.
Those referred who are put on a waiting list to access services	High	Unknown at this stage	Low	This is likely to be a very small sample size. Pilot sites have suggested they won’t be ‘over capacity’ for a long time. People would also have to be on a waiting list long enough to evaluate impacts.
UC claimants from the RTI dataset	Unknown at this stage	High	High	It is unknown if this group will be representative of WorkWell participants but access to data and contact information is feasible.
Eligible people from DWP administrative data in non-WorkWell pilot areas	Medium	Medium	Low	A counterfactual group of WorkWell eligible people in areas without a WorkWell pilot can only be obtained from the RTI database. This will only include benefit claimants. RAPID does not contain any health condition information for people who have not had to report this for benefit claim reasons. Therefore, it is not possible to determine who is eligible for WorkWell from the larger population pool of non-claimants.
Eligible people from the Annual Population Survey in non-WorkWell pilot areas	High	Unknown at this stage	Medium	It is unknown at this stage if can access the data to identify individuals who are living in non-WorkWell areas. A counterfactual group from this dataset would include those in work and out of work, as well as benefit claimants and non-claimants.

Using data collected by unsuccessful pilot sites to form a conterfactual group for WorkWell participants is not recommended. Firstly, there is no requirement for these areas to be collecting any information about people who would be eligible for the programme. Therefore, these data may not exist for use in the evaluation. Additionally, there is no requirement for these areas to engage with the WorkWell evaluation, so if data do exist, there is no guarantee that it would be shared with the evaluation team. This combination of factors rules out unsuccessful pilot site data collection as an option for creating the counterfactual.

People living in ICB areas delivering the WorkWell pilot, but in excluded geographical areas are also not considered to be a suitable counterfactual group. The pilot site delivery plans indicate that only a few of the pilots will be excluding parts of their geography from the programme. This would leave the evaluation with a small sample size of eligible individuals who cannot access the programme. Additionally, the areas excluded from service provision by pilots tend to be more economically prosperous, have lower unemployment rates and have a lower need for the WorkWell service. Therefore, this group would not be representative of WorkWell participants.

There are also various options to construct a counterfactual from people who are referred into the WorkWell programme:

Those who do not take up the programme after referral
Those who are refused the programme after referral
Those who are put on a waiting list after referral into the programme

The reason for not taking part in the programme or being refused, make this group of people fundamentally different from WorkWell participants. This makes them unsuitable to form a counterfactual, as the results of the evaluation may be biased due to unobservable differences between participants and non-participants.

Discussions with the pilots during the stakeholder interviews also suggested that those who do not take up the programme will not consistently be recorded (as they will sometimes be signposted with no formal referral), thus potentially introducing further bias to the individuals that it is possible to obtain data for, within this group. Additionally, this is likely to be a small sample size, which will likely result in statistically non-significant findings.

WorkWell pilots have indicated that they do not expect to be at maximum capacity initially, and for a long time from the beginning of the programme. Therefore, it is not feasible to rely on people referred into the programme being put on a waiting list for long enough to collect outcomes and compare with the outcomes of participants. Using people eligible for the WorkWell programme in WorkWell areas but not referred is not recommended, as the reason these people are not referred could be a confounding factor which impacts on their outcome measures, biasing the results.

A counterfactual group could be formed using the RTI dataset, to only include benefit claimants who are living in non-WorkWell pilot areas. Until the WorkWell programme begins, it is unknown if this will form a representative counterfactual group of all WorkWell participants. People who are not claiming benefits may be a large proportion of WorkWell participants, thereby making this a poor way of forming a counterfactual group for the whole participant pool. However, this would be representative of benefit claimants. At this stage it is not possible to assess what proportion of WorkWell participants will be covered by this sub-group.

Using people in non-WorkWell areas from the RAPID dataset will only allow for people who are eligible for the programme who are benefit claimants to be identified, since information on health conditions is only available for those who have had to declare them to claim certain benefits. Therefore, the RAPID dataset cannot be used to form a counterfactual of people who are eligible for WorkWell which includes non-benefit claimants.

The Annual Population Survey contains variables on health conditions and whether these impact individual’s work. Therefore, a counterfactual group which includes benefit claimants and non-claimants could be constructed from people not in WorkWell areas. If access to the more restricted dataset can be obtained, this counterfactual group will be preferred to evaluate the impact on all WorkWell participants. Additional impacts such as healthcare outcomes and healthcare resource use are not included in this dataset, so some measures will still only be able to be evaluated for the subgroup of benefit claimants who complete the survey.

Individuals who are economically inactive and not looking for work may be present amongst WorkWell participants, but would likely not be present in numbers sufficient to allow separate analysis of outcomes/impacts; so would be analysed within the ‘not working’ participant group. They would be identifiable at the baseline from the individual Management Information, thus giving the option of removing them from the impact analysis if there were concerns that they might be ‘diluting’ the impact of WorkWell. ↩
A dataset using HMRC and DWP linked data on employment and benefit history for anyone who has a national insurance number (NINO). ↩
A measure of a person’s knowledge, skills and confidence to manage their own health and wellbeing. The PAM tool is made up of 13 questions, which determines the person’s ‘activation’ score (1, 2, 3 or 4). ↩
Individuals who are economically inactive and not looking for work may be present amongst WorkWell participants, but would likely not be present in numbers sufficient to allow separate analysis of outcomes/impacts; so would be analysed within the ‘not working’ participant group. They would be identifiable at the baseline from the individual Management Information, thus giving the option of removing them from the impact analysis if there were concerns that they might be ‘diluting’ the impact of WorkWell. ↩
These calculations use a ‘pwr.t2n.test’. A pooled standard deviation has been calculated using the findings from the Health-Led Employment Trial Evaluation: 12-month outcomes report: Estimating the impact of IPS over 12 months. 2022. ↩
(Marx, A., and A. Dușa. 2011. Crisp-set qualitative comparative analysis (csQCA), Contradictions and consistency benchmarks for model specification. Methodological Innovations Online 6:97–142) ↩
Alrik Thiem and Lusine Mkrtchyan 2023 Case-to-factor Ratios and Model Specification in Qualitative Comparative Analysis  Field Methods 2024, Vol. 36(1) 52–6. ↩
As mentioned in Chapter 4, across the 15 pilots the assumption is that the total number of target beneficiaries will be . ↩

Cookies on GOV.UK