Policy paper

Mental Health and Employment Partnership Evaluation: Life Chances Fund - Summary report

Published 24 March 2023

Applies to England

This briefing summarises the key findings of the first report from the evaluation of the Mental Health and Employment Partnership (MHEP) projects commissioned under the Life Chances Fund (LCF). The LCF is a £70 million outcomes fund to support locally commissioned social impact bonds (SIBs), launched by the Department for Culture, Media & Sport (DCMS) in 2016. There are 5 MHEP SIBs contracted under the LCF, each of which supports the delivery of an intervention known as ‘Individual Placement and Support’ (IPS) to help people experiencing mental health issues or learning disabilities to find and remain in competitive, paid work. The first report adopts a mixed methods approach, drawing on qualitative interview data and aggregate quantitative performance data.

Key findings

Based on the qualitative research with service providers and other stakeholders, the evaluation identified three key routes through which MHEP provides additional value compared to traditional commissioning:

1. Additional financial and human resources: MHEP was able to unlock additional funding for projects through the Life Chances Fund. This was understood to bring additional financial and human resources.

2. Additional performance management function: MHEP provided a dedicated performance management function that was seen to drive an additional focus on achieving outcomes, supported by data analysis and intelligence.

3. Collaborative working: MHEP supported a more intentional, collaborative working culture within each local partnership.
   
   Early performance data finds that MHEP SIBs were performing below initial high-case scenario targets in the period up to December 2021. However, these targets were set prior to the pandemic, and job outcome and sustainment rates have improved since the end of 2021. Amongst participants with severe mental illness, the job outcome rate up to December 2021 was 29% which is similar to the lower-end rates seen in the IPS implementation literature (generally 30-50%). This means an average of one new job start for every 3 to 4 people who engage in the programme.

MHEP was established in 2015 to drive the expansion of high-quality employment support programmes, based on the Individual Placement and Support (IPS) model, for people with mental health issues. It is a special purpose vehicle^{[footnote 1]} run by Social Finance, backed by investment from Big Issue Invest. The Life Chances Fund provides funding for five MHEP impact bonds, the key characteristics of which are provided in the table below.

Table 1: Key characteristics of the LCF MHEP projects

Location	Haringey and Barnet	Shropshire	Enfield	Tower Hamlets Mental Health	Tower Hamlets Learning Disabilities
Policy focus	Severe mental illness	Severe mental illness	Severe mental illness	Severe Mental Illness	Learning disabilities
Service delivery launch	April 2019	April 2020	April 2020	April 2020	July 2020
Local commissioner	London Borough of Haringey and Barnet	Shropshire Council (local authority)	Enfield Council (local authority)	Tower Hamlets Clinical Commissioning Group	Tower Hamlets Council (local authority)

In the MHEP SIBs’ outcomes contract payments are contingent on the achievement of 3 pre-specified, measurable outcomes:

1. Engagement: individual engages with the programme and completes the vocational profile.

2. Job start: individual spends one full day (or 4 hours for part-time work) in paid competitive employment.

3. Job sustainment: individual sustains paid competitive employment for at least 13 weeks.

MHEP is being evaluated as part of the LCF’s supplementary evaluation, which involves in-depth studies into certain projects, looking to directly compare the use of SIBs to alternative commissioning mechanisms. MHEP provides a useful project to enable this comparison, given that it delivers a tightly defined intervention (IPS) which has also been delivered through other commissioning models. Through this approach the evaluation aims to assess the extent to which the SIB model contributed to the outcomes achieved.

The aims of this first report in the MHEP evaluation are to:

• generate theories of change and outline contextual factors (barriers and facilitators) which serve as external influences on the MHEP projects
• explore the distinctive contribution of MHEP
• analyse performance data of the key outcome metrics through time and across different sites and providers

MHEP’s theory of change

MHEP’s theory of change was developed through a combination of interviews, documentary analysis, observation, and theory of change workshops, with the MHEP team, local commissioners and service providers. The theory of change connects inputs to eventual outcomes, and also unpacks mediating mechanisms which help MHEP transform its inputs into both short-term and long-term outcomes. Figure 1 summarises the theory of change elements that were shared across all three stakeholder groups (see the full report for all theory of change models developed by each stakeholder group).

Figure 1: Collated theory of change for MHEP’s interaction with project stakeholders

Overall aim: to support individuals with severe mental illness or learning disabilities into employment through high-quality IPS services.

A theory of change for the interaction between project stakeholders and MHEP

Inputs	→	Mediating mechanisms – Intermediary outcomes	→	Short-term outcomes	→	Long -term outcomes
Convening and advocacy input on applying for and unlocking funding through the Life Chances Fund and SIB model, as well as connecting project stakeholders to each other and new organisations. Operational input by MHEP connecting project to an IPS specialist, solution-orientated and contracting support. Analytical input around contract design, contract monitoring, data analysis and performance management.		Additional financial and human resources boost local capacity to deliver and expand high quality IPS services. Additional performance management, contract monitoring and scrutiny from MHEP boosts analytical capacity for providers and commissioners, whilst allowing for benchmarking against and learning from other services. Collaborative working with providers and commissioners based on strong and professional relationships.		Clients are supported into appropriate employment. Greater number of clients can be supported into employment than before. Clients benefit from a high quality service which addresses their needs and preferences adequately.		Clients become self-reliant and experience positive changes to their lives and wellbeing. Clients are supported into sustainable employment and are able to build careers.

Providers and commissioners identified three main inputs provided by MHEP:

1. Convening and advocacy: The MHEP team signposted relevant opportunities and funding, led multi-stage LCF applications and engaged in wider advocacy for IPS as an approach.

2. Operational: The MHEP team provided operational support through an IPS implementation specialist to support performance, a solution-oriented approach that adapted to meet the needs of specific projects, and additional commissioning and contracting support.

3. Analytical: MHEP offered analytical support in the form of performance monitoring, enhanced data quality and access, and benchmarking against comparable IPS services.

However, despite identifying these inputs, there was considerable variation in stakeholders’ perceptions about the distinctiveness of MHEP compared to traditional commissioning. While research participants generally acknowledged that MHEP’s assistance in applying for and unlocking LCF funding had been key, there was disagreement between the interviews over whether the other identified inputs were markedly different from, or additional to, existing practices and performance management procedures under non-SIB commissioning models.

Stakeholders were also asked to identify the barriers or facilitators which hindered or supported the delivery of MHEP. While different stakeholder groups identified a range of different factors, there were a number of facilitators and barriers which were common to all.

The common facilitator identified across all stakeholders was that the MHEP contracts were aligned with wider IPS commissioning. This included alignment to KPIs in previous contracts and the national IPS rollout, which ensured that these were familiar to providers and aided the smooth delivery of the programme.

While this was the only facilitator identified across all stakeholders, additional facilitators were discussed by different stakeholder groups, such as commissioners, for example, stating that providers were able to access more support than they would have been without the MHEP SIB (see full report for complete list of facilitators and barriers).

A common barrier identified by stakeholders was the impact of cohort differences, namely that learning disabilities services required different expertise and intensiveness of support compared to severe mental illness services, particularly due to additional in-work support. Furthermore, this cohort represented a relatively fixed population, limiting referral numbers.

Another common barrier was that the social impact bond arrangement was perceived as complex in both payment (in order to blend funding from the LCF and local commissioners while balancing outcomes-based and block payments) and design (with second job starts not classed as payable outcomes, but instead covered under sustainment outcomes) compared to traditional contracting arrangements.

The final common barrier was that COVID-19 brought challenges including low referral numbers, reduced employment opportunities as a result of the impact of the pandemic on key sectors including hospitality and retail, and the reliance on digital engagement which limited opportunities for valuable face-to-face engagement with users. This also led to delays in launching the MHEP SIB projects.

Three mediating mechanisms – the routes through which MHEP is perceived produce positive changes in IPS implementation and outcome achievement – were outlined by all interviewees:

1. Additional financial and human resources. MHEP provided access to SIB funding which providers may not have been able to receive otherwise, boosting local capacity and enabling additional employment specialists to be hired.

2. Collaborative working. MHEP represented a “three-way partnership” with a sense of shared purpose, and despite requiring considerable work, this was viewed as hugely beneficial.

3. Additional performance management function. This took the form of regular and rigorous scrutiny from the MHEP team, as part of the SIB contract, and internal organisational processes. It was grounded in strong relationships, aligned objectives, frequent communication, as well as analytical capacity and data – although there were some concerns around heightened reporting requirements.

Outcome performance

Across the MHEP SIB projects, performance is measured against four outcomes, which are:

• referral (not linked to payment)
• engagement (intermediary outcome)
• job start (primary outcome)
• job sustainment (secondary outcome)

Early performance data finds that MHEP SIBs are performing below initial high-case scenario targets. However, these targets were set prior to the pandemic, and job outcome and sustainment rates have improved since the end of 2021. Changes were made to support projects through the pandemic such as COVID-related activity payments (see full report for description of these changes).

Amongst participants with severe mental illness, the job outcome rate was 29% up to the end of 2021, which is similar to the lower-end rates seen in the IPS implementation literature (generally 30-50%). This means an average of one new job start for every 3 to 4 people who engage in the programme.

Job outcomes for MHEP have been costlier (£5,248 on average) than expected (£4,123), partly due to price and payment changes during the pandemic. However, job outcome rates are on an upward trajectory since the end of lockdown.

Overall, an analysis of outcome rates against targets indicates that:

• quarterly performance appears to be below expectations, often at around 50% of anticipated high-scenario targets
• the success rates in meeting job start targets has generally remained at a similar level over time, although Shropshire and Enfield projects both show signs of closing the gap between actual and intended job outcomes
• with respect to outcomes composition, more than 65% of all achievements for the severe mental illness cohort to date have been on engagements
• performance against targets is likely to have been significantly affected by COVID disruptions

Tables 2 and 3 below present a summary of outcomes against targets for the MHEP projects working with severe mental health illness and learning disability cohorts respectively.

Table 2 – Success rates against targets by end of 2021 (severe mental illness cohort)

	Haringey and Barnet	Tower Hamlets	Enfield	Shropshire	MHEP Total
Referrals (actuals)	672	830	184	298	1,984
Referrals (success rate)	80%	52%	59%	110%	66%
Engagement (actuals)	496	508	99	204	1,307
Engagement (success rate)	76%	64%	43%	116%	70%
Job Start (actuals)	147	145	36	56	384
Job Start (success rate)	52%	54%	49%	68%	54%
Job sustainment (actuals)	69	78	17	23	187
Job Sustainment (success rate)	43%	43%	47%	48%	44%
Outcome Payment (£ actuals)	£1,081,038	£955,306	£294,319	£560,946	£2,891,609
Outcome Payment (success rate)	40%	78%	83%	114%	61%

Total outcome payment for ‘target’ ignores any changes to pricing and payment which happened as a result of COVID and follows the pre-defined targets and prices as set under Tier 1 arrangement.

Table 3 – Success rates against targets by end of 2021 (Tower Hamlets - learning disability cohort)

	Actuals	Success rate
Referrals	36	18%
Engagement	15	8%
Job Start	8	9%
Job Sustainment	5	25%
Outcome Payment	£606,759	61%

Future research

The current report was Phase 1 of a three-stage evaluation of MHEP services. Future research will build on the analysis offered in this report to investigate whether the MHEP SIB projects are associated with different levels of outcomes achievement compared to more conventionally-funded services.

Phase 2 will involve further work to identify counterfactual data to compare with MHEP outcomes. It also aims to run a “cohort analysis” using individual-level data on MHEP participants, and to deepen the process evaluation around the implementation of the programme.

Phase 3 will deliver a full quantitative impact evaluation, using a quasi-experimental design, and including an economic analysis around the costs and benefits of the SIB approach.

This summary, and the report which underpins it, was prepared by the Government Outcomes Lab as part of the supplementary evaluation of the Life Chances Fund. For more information, you can read the main report and technical annexes.

1. A special purpose vehicle is a legal entity (usually a limited company) that is created solely for a financial transaction or to fulfil a specific contractual objective. Special purpose vehicles are sometimes used in the structuring of impact bonds. ↩