Research and analysis

Evaluation of the Creating Opportunities Forum (COF) programme

Published 13 February 2025

Applies to England and Wales

Executive Summary

The Creating Opportunities Forum (COF) programme was funded by a joint HM Treasury bid by the Home Office and the Department for Work and Pensions. COF aimed to bridge the gap to employment for vulnerable young people by providing wraparound support, offering employment-related opportunities, helping to build confidence among young people on their abilities and talent to become ‘work ready’, and consequently deter them from violent crime. COF operated in 12 local authority (LA) areas and sought to achieve 3 key outcomes:

• reduction in violence and crime among individuals in the programme
• engagement in employment-related activity among individuals in the programme
• improvements in overall physical and mental wellbeing of the individuals involved

The programme worked with young people aged 16 to 24 deemed to be ‘at risk’ based on a range of criteria (see Section 3.4.1 for details). Young people identified as ‘at risk’ could opt in to the support offered by COF, through referrals by local agencies or by self-referral.

A national provider consortium delivered COF, comprising Catch22, The Prince’s Trust and Apprentice Nation, through a single services contract with Catch22 as the lead provider. The role of Catch22 and The Prince’s Trust was to work in partnership to deliver the programme in LAs. Apprentice Nation was responsible for signposting to recruitment events and local opportunities, and offering a curriculum of content to young people through its online platform.

In September 2021, the Home Office commissioned RSM UK Consulting to evaluate the COF programme (October 2021 to March 2023). The aims of the evaluation were to:

• conduct a robust process and impact evaluation of activity undertaken by the consortium of delivery organisations
• continuously feed into the programme of services offered by the consortium and work collaboratively with them to ensure best practice, and implement any changes needed to achieve the aims of the COF

The mixed-methods evaluation, underpinned by a programme-level Theory of Change (ToC), included a literature review, qualitative research with COF participants and various stakeholders including delivery staff, review and analysis of monitoring information (MI) collected by the consortium through Catch22’s online case management system, Aptem, and a pre-/post-participant survey.

Key findings

Key overarching findings included:

• delivery staff and external stakeholders described the programme as a unique service, able to respond to the needs of participants across the 12 LA areas
• the programme delivered an ongoing and holistic programme of one-to-one tailored support including: financial, job-related and soft-skills development; and external signposting to services such as housing, health and wellbeing
• although not attributable to COF alone, this holistic programme of support resulted in positive outcomes for participants identified in the programme ToC. At an aggregate level, survey results found statistically significant improvements (using 95% confidence intervals) on 2 of the 3 outcomes – life outlook and access to employment – although there are limitations to these findings (see full report)
• however, one of its biggest shortcomings was its limited success in targeting the key participant group of interest – young people with a history of, or at risk of, serious violence. The programme engaged 203 at-risk young people (21% of programme cohort). Of these, 91 stayed on the programme and 112 disengaged from COF activities (that is, 55% of all at-risk participants disengaged from the programme)
• the programme struggled to reach its intended audience due to a lack of capacity in building strong connections with organisations working with ‘hard-to-reach’ young people, such as Youth Offending Teams (YOTs) and prison services
• as a result, it was not possible to assess impact on crime and offending behaviour due to the low sample size of the at-risk cohort of young people and a lack of disclosure about offending behaviour
• in addition, staff turnover and burnout were 2 factors lowering the provision of a high-quality service to participants. The lack of capacity meant the programme was not sufficiently resourced to engage with and address specific needs of participants over an extended period, such as between 3 to 6 months, depending on the individual needs. This led to a high proportion of disengaged participants (38% in total), including those at risk of violence or with a history of offending (12% of the total programme cohort; one-third of all participants disengaged)

Other findings relating to delivery of the programme included:

Roles and responsibilities

• staff roles and responsibilities were generally perceived as clear and provided staff with autonomy to tailor support to the needs of the participants, although some staff found the roles blurred in certain regions, affecting the quality of support for participants. Others felt responsibilities could have been delegated more across teams

Cooperation and communication

• there was very limited evidence on coordination between the 2 delivery partners – Catch22 and The Prince’s Trust, with regional staff feeling siloed from other provider teams within and across the different regions. There were also administrative challenges within the consortium, which impacted the programme activities and evaluation data collection
• pre-existing relationships with local stakeholders were key for successful partnerships, but these varied across the programme, creating inefficiencies when they were insufficiently leveraged due to a lack of knowledge or communication between delivery partners

Programme implementation

• although programme staff deemed the referral process fit for purpose, external stakeholders believed that it lacked sufficient clarity on the main benefits for participants and the type of support offered to young people
• additional support through group-based activities was identified as an area for further improvement

Recommendations

The following recommendations emerged from the process and impact evaluation.

Recommendations for national providers

On roles and responsibilities:

• although there were 2 roles related to programme delivery (regional coordinator and career coach), further streamlining responsibilities within provider teams could minimise the loss of quality support offered to participants by enabling coaches to focus on the core job, rather than dealing with multiple tasks
• collaborating with each other at a regional level, as opposed to taking an individual provider-led approach, to ensure effective joint programme delivery
• clearer guidance, planning and management in relation to delivery team size, individual workloads, and distribution across geographical regions/LAs to avoid staff turnover or burnout, and any knock-on effects on programme delivery

On programme activities:

• more group activities besides one-to-one support as a core offering to young people attending the COF programme and forming participant groups for peer-to-peer learning and engagement
• further engagement with the Apprentice Nation platform through greater clarification and support in relation to job seeking, and effective navigation
• further expansion of skills/interest areas covered by the programme, for example, in relation to the music and arts, to better meet participant needs
• further streamlining of the process allocating career coaches to programme participants by matching them with coaches that have knowledge of and links with job or industry types that were targeted by young people, or based on participants’ preferred engagement patterns (for example, groups versus one-to-one or in-person versus virtual) to facilitate continued engagement

On relationships with external stakeholders:

• clearer messaging and communication with referral partners on key benefits and types of support available, and the time and processes for making referrals onto the programme
• more resources specifically dedicated towards building relationships and communication channels with external organisations working with hard-to-reach young people at risk of violence (such as prison services)
• more direct relationships with employers and engagement on a continued basis
• more joined-up working with services supporting at-risk participants with ‘adjunct’ needs such as food and fuel banks, and accommodation providers

On delivery processes:

• improve participant onboarding processes through additional ways of verifying identity (for example, national insurance numbers), which was part of the onboarding process for verifying participants’ age and enabling job applications
• standardising processes between different delivery partners to clarify what, when and how wraparound support can be provided to maximise programme impact
• improving target participant identification through building better connections with referral partners, including meeting regularly to discuss delivery processes and how these might be improved
• clearer focus on targeting the at-risk group from the beginning to meet the funding aims and objectives

Recommendations for the Home Office

Recommendations for the Home Office include:

• defining the target group clearly within the provider contract to ensure that future funding targets the intended beneficiaries fully instead of in part
• build in joint working structures between all national providers in the service contract, such as mutual data sharing agreements to enable joint case management processes, ahead of programme rollout. Providing sufficient resources is key to facilitate the building of such structures
• considering including key external organisations (such as prison services) that work with young people at risk of serious violence into the single service contract
• clarifying expectations around target numbers from the outset and setting variable delivery targets based on variations in the calendar year
• accounting for more realistic timeframes into the programme delivery milestones for building a presence of the programme within the local community, including relationships with local and national stakeholders. Sufficient resources need to be provided to enable this to happen

Acknowledgements: authors and contributors

The evaluation was conducted by RSM UK: Emilio Torrini, Mahima Hoffmann, Michael Greig, Aditi Mehrotra, Nazia Chowdhury, Bijun Qin, and James Cockett.

With special thanks to academic advisors Professor Darrick Jolliffe (Royal Holloway University) and Dr Gilly Sharpe (University of Sheffield).

1. Introduction

In March 2020, the Home Office and the Department for Work and Pensions (DWP) submitted a joint proposal to His Majesty’s Treasury (HM Treasury) Shared Outcomes Fund to support the work of the Creating Opportunities Forum (COF), which was established following the Serious Violence Summit held in December 2019.

In July 2020, the Shared Outcomes Fund team confirmed that the Chief Secretary had approved the request for £3.799 million of funding for the delivery of the programme.

1.1 COF rationale and aims

Introducing the April 2018 Serious Violence Strategy was a significant milestone in the government’s aims to limit and reduce the upward trends in cases of serious youth violence. The Strategy placed a particular emphasis on the pivotal role that early intervention plays in helping young people to make positive choices, deterring them from criminal behaviour and violence.

The rationale for the COF programme was based on a multi-dimensional approach for early intervention, which comprised 4 main strands: mentoring, work coaching, life skills and counselling. Overall, research shows that programmes that combine these 4 types of support have generally been more successful at reducing re-offending than individual intervention approaches (MoJ, 2018). In addition, evidence from life-course criminology has demonstrated that employment can have a positive impact on desistance from crime through several mechanisms. A steady job brings not only economic gains, but may also foster maturity, responsibility (Sampson & Laub, 2003), and identity change (Maruna, 2001). Lastly, a “good job offers opportunities to invest in human and social capital and can generate turning points that allow offenders to disconnect from criminal pathways” (Van der Geest et al., 2011) as life-course studies put it. In fact, employment can provide a stake in conformity that motivates offenders to avoid (routine) activities involving criminal opportunities.

Focusing on those most at risk of violence (see Section 3.4.1 for definition), the COF programme aspired to empower vulnerable young people to make positive life choices and address barriers to engaging in work-related activities through developing employability skills and taking steps to become work ready.

COF aimed to support young people through their journey towards employment by:

• providing wraparound support
• offering employment-related opportunities
• helping to build confidence in abilities and talent

Such opportunities were designed to offer young people a chance to develop their work-related skills, aspirations and career pathways, consequently deterring them from violent crime.

There was mixed evidence on collaboration between COF delivery partners – Catch22, The Prince’s Trust and Apprentice Nation. It operated in 12 LA areas (see Section 3.2 for list) that the Home Office identified by using place-based serious violence, social cost of crime, and alternative claimant count (an unemployment metric) for 16 to 24 year olds as key metrics. COF sought to achieve 3 key outcomes for individuals engaged in the programme:

• reduction in violence and crime
• engagement in employment-related activity
• improvements in overall physical and mental wellbeing

On the ground, regional coordinators and career coaches employed by Catch22 and Prince’s Trust implemented the programme, with Apprentice Nation used their online platform to signpost participants to recruitment events and opportunities in their local area.

1.2 COF Theory of Change

A ToC is an overall narrative that explains how a programme aims to achieve its intended objectives. It traces the programme inputs through to its intended impacts (Better Evaluation, 2022).

The figure below provides a ToC diagram for the COF programme, illustrating the key components of the ToC. The logic model includes references to the assumptions that are relevant to each strand. Full discussion of these assumptions, indicated by circled letters in the logic model figure, are in Appendix 1.

This logic model was developed following a collaborative workshop with representatives from RSM, the Home Office and Catch22 to build a shared understanding of the COF programme. In addition to the workshop, the model includes findings from the evaluation team’s review of programme documentation as well as learnings from familiarisation interviews with Catch22.

1.3 Report structure

The remaining chapters of this report are structured as follows:

• chapter 2 summarises the evaluation methodology, with further detail presented in Appendix 1
• chapter 3 outlines the key findings related to the delivery of the COF programme, including key barriers and enablers to delivery
• chapter 4 provides the key findings related to benefits to programme participants. The chapter is structured around the intended impacts of the fund, as outlined in the ToC. It considers progress towards intended COF outcomes and the direction of travel towards longer-term outcomes, and key barriers and enablers for COF to contribute to relevant outcomes
• chapter 5 outlines the conclusions, answering the overarching evaluation questions, and the recommendations

2. Evaluation and aims methodology

2.1 Aims

In September 2021, the Home Office commissioned the independent research agency RSM UK Consulting to evaluate the COF programme. This report presents the findings of evaluation activity conducted during the programme delivery, from October 2021 through to March 2023. RSM UK’s independent evaluation of COF had the following aims:

• to carry out a robust process and impact evaluation of activity undertaken by a consortium of delivery organisations – Catch22, Prince’s Trust and Apprentice Nation
• to continuously inform the services provided by the consortium and work collaboratively with the consortium to ensure best practice, and implement any changes needed to achieve the aims of the COF using the evaluation findings

The evaluation, underpinned by a programme-level ToC, gathered evidence through a range of approaches, outlined in the following sections.

2.2 Methodology

The evaluation involved a theory-based approach to assess the achievement of the COF’s outputs, outcomes and impacts. The evaluation focused on establishing the impact of the programme and the processes used to deliver COF activities, understanding any local variation, perceived and realised benefits among participants, and identifying and sharing best practice. These activities included a combination of both quantitative and qualitative data collection methods, as well as consideration of evidence from secondary sources and monitoring data captured through Catch22’s case management system, Aptem. The following sections outline these activities in more detail.

As this evaluation ran in parallel to implementing COF programme activities, it was too early to carry out an analysis of some of the envisaged longer-term outcomes and impacts. This limitation is mitigated by the use of Mayne’s contribution analysis (CA) approach (Mayne, 2008), a theory-based approach that considers results achieved to date and the direction of travel toward desired impacts by assessing the causal links between the programmes’ activities and its intended outcomes. Theory-based design refers to an evaluation design based on the programme’s own ToC and explores the causal links between the programmes’ inputs (activities), outputs and outcomes. CA explores the contribution of observed outcomes to the COF programme as opposed to other explanations and contextual factors.

2.2.1 Inception phase

During the scoping stage (September to November 2021), a variety of activities were carried out to inform the main stage evaluation approach:

• 2 inception meetings
• a desk-based review of programme and policy documentation
• a literature review led by Dr Gilly Sharpe, Lecturer in Criminology at the University of Sheffield
• development of a programme-level ToC

Professor Darrick Jolliffe, Royal Holloway (University of London) led a feasibility study to examine the suitability of different approaches for conducting a counterfactual analysis. The study found that the possibility of using experimental and quasi-experimental evaluation designs, including randomised control trials (RCTs), was deemed not feasible (the full results of the feasibility study are in Appendix 1.

2.2.2 Main stage evaluation activities

The evaluation used a mixed-method approach, comprising quantitative data sources to assess the programme’s impact, and qualitative sources, allowing for the exploration of the programme’s implementation and contextual factors that have contributed to the achievement or non-achievement of outcomes. This approach included the analysis of programme monitoring data and a survey alongside qualitative interviews. The full methodology is listed below:

• qualitative data collection in all 12 LAs where COF was operating. Data were collected through 145 online and face-to-face one-to-one semi-structured, in-depth interviews from:
  • 89 programme participants (young people)
  • 27 programme delivery staff (operations, career coaches and regional coordinators)
  • 29 external stakeholders (referral organisations, local and national employers; third-sector organisations, LA staff, other local partners)
• quantitative data using a bespoke survey tool delivered by BMG Research, offered to all COF participants at 3 time-points using a double baseline design. The survey was identical across the 3 points and was piloted in 4 LAs (Liverpool, Manchester, Lambeth, Croydon), supported by 2 careers coaches, across 2 regions
• the review and analysis of monitoring information (MI) collected by the consortium through Catch22’s Aptem. This MI data was shared with RSM weekly for the duration of the evaluation and provided vital information on the delivery of programme activities. This included data about the demographic characteristics such as age, gender, sexual orientation, ethnicity, and also intervention relevant data such as referral pathways, engagement with services and details of benefits that the participants might have been receiving. This report contains a descriptive analysis of Aptem data from participants from 11 November 2021 up to 13 February 2023

2.2.3 Analysis and synthesis

Survey data: RSM created a version to be completed by paper as well as an online version, to support individual respondents’ preferences. All surveys were administered by career coaches, with physical questionnaires copied and shared with BMG Research and RSM.

The double baseline approach was intended to be used by administering the survey at 3 data collection points: at the start of the programme; after one month of engagement with programme activities; and at the end of the programme. The benefits of using this type of design are two-fold: firstly, by allowing more time for participants to develop a trusted relationship with programme delivery staff it elicits more honest answers. These are more likely to be captured on the second baseline (for example, when asked questions about frequency of criminal activity). Secondly, the double baseline design can increase causal inference. This assumes a hypothesis that the intervention’s benefit is exposure time-related (a reasonable assumption in most forms of behavioural intervention).

The original double baseline design was dropped at analysis stage for 2 main reasons:

• most participants completed the first survey 4 to 6 weeks after the start of COF activities. This invalidated the principle upon which the double baseline is based (that is, the first survey should be completed at the very first point of contact with participants with a programme/intervention)
• the sample of participants having completed the second survey was too small to infer any statistical significance to the results

As a result, the evaluation methodology presented in this report ranks at Maryland Scale Level 2 (What Works Growth, 2015) – this means the evaluation draws on pre-/post-observations between results of the first and third surveys. In total, the evaluators received 125 of matched pre-/post-surveys, out of a possible 584, giving a 21.4% response rate.

The response rate was the number of usable completed pre- and post-matched surveys returned, divided by the number of participants taking part in the activities (this is assumed to be the same as the number of surveys that were distributed – that is, all participants would be eligible to take part in the survey and thus all should have been asked to complete a survey, although this was down to the programme staff distributing them this way). This is expressed in the form of percentage. This figure excludes 357 participants who were not ‘actively’ engaged in the programme on 13 February. In addition, the evaluation – in agreement with the Home Office – excluded 41 young people (recruited by The Prince’s Trust) who had been exposed to The Prince’s Trust support activities linked to COF before a delivery contract was signed between The Prince’s Trust and Catch22.

All survey findings are subject to a margin of error, and confidence intervals (CIs) are used to express the degree to which any given answer might differ from that observed in the population of interest. For this evaluation, the CI is constructed at the 95% confidence level. This implies that there is confidence that 95 out of 100 times the estimated answer will fall between the upper and lower values specified by that CI, should the survey be run again or the participants be sampled in the same way. However, there are specific elements of the survey where more caution should be taken when interpreting the figures reported:

• some young people may have completed the questionnaire with the help of their career coach to help with comprehension of the most difficult questions. Although this might have helped to boost response rates, this introduced a social desirability bias into the quantitative evidence gathered as part of this evaluation, that is, the respondents might have wished to please the career coach and tried to provide answers to the survey they thought were ‘right’ in the eyes of this career coach
• the survey may have been completed by more engaged young people, given how the survey was administered (that is, via career coaches)
• results may be subject to positive bias and do not consider any substitution or displacement effects that may have taken place (that is, they are self-reported figures)

As a result, such data should be treated with a degree of caution.

Qualitative data: The research team recorded, transcribed, systematically inputted and coded all the interviews into Nvivo 14, a qualitative data analysis program. There was one grid for each type of group of participants consulted (programme participants, programme delivery staff, external stakeholders). The Nvivo 14 analysis framework followed the structure of the topic guides, enabling the identification of relevant quotes for each element of the impact and process evaluation. A thematic analysis approach (Braun & Clarke, 2006) was implemented to identify, analyse and interpret patterns of meaning (or ‘themes’) within the qualitative data, which allowed the evaluation to explore similarities and differences in perceptions, views, experiences and behaviours. Once the team had inputted all the data, Nvivo brought together evidence for each outcome and key delivery theme in a second analysis matrix to triangulate the evidence and assess its robustness.

Quotes in this report are verbatim and are used to illustrate the views expressed by interviewees and highlight key points and common themes (quotes that contained personal information were anonymised.). The findings rely on multiple interviews and analysis of these, but they are not the only source of evidence. Quotes from delivery staff encompass both career coaches and regional coordinators due to low numbers and risk of identification.

Synthesis: It was necessary to bring together the evaluation strands of data collection for a systematic analysis. Therefore, all evidence sources were mapped against the Key Evaluation Questions (KEQs) highlighted in the evaluation matrix in Appendix 1. The Key Evaluation Questions are presented in Table 1.

Table 1: Key evaluation questions

Process evaluation
1. To what extent and in what ways are the local authority areas implementing the interventions to reflect the local area and cohort profile?
2. What intervention resources (mentoring, counselling, life skills, employment skills) are found most engaging and useful by the young people participating in the interventions?
3. How are young people being engaged in the interventions?
4. Are young people involved in any other interventions from other programmes? Has this impacted on their progress towards the outcomes?
5. What are the motivators and barriers to the employment-related opportunities being provided?
6. How is the national provider working with local partners to define, identify and form an understanding of the ‘at-risk cohort’ for involvement in the interventions?
7. What employment-related opportunities are being provided by local partners?
8. How sustainable are the employment-related opportunities?
9. What benefits and added value will be/has been derived from the interventions and employment-related opportunities, both for individuals involved in the interventions and for the local partner businesses?
10. How many individuals are being reached in each area? How are they being identified, and do they meet the selection criteria?
11. What are young people’s experiences of interventions and employment-related opportunities?
Impact evaluation
1. Do those who receive (up to 3 types of) support report a demonstratively better life outlook?
2. Is there evidence that those receiving (soft/employability skills) support report a significantly better life outlook?
3. Is there evidence that those self-reporting high levels of positive engagement with their career coach report a demonstratively better life outlook?
4. Do those who receive (up to 3 types of) support do demonstratively better on access to employment?
5. Is there evidence that those receiving (soft/employability skills) support do demonstratively better on access to employment?
6. Is there evidence that those self-reporting high levels of positive engagement with their career coach do demonstratively better in relation to access to employment?
7. Do those who receive (up to 3 types of) support report demonstratively lower criminal intent/activity?
8. Is there evidence that those receiving (soft/employability skills) support report demonstratively lower criminal intent/activity?
9. Is there evidence that those self-reporting high levels of positive engagement with their career coach report demonstratively lower criminal intent/activity?
10. What individual characteristics typify those who receive significant levels of support, but do not appear to benefit from them?

Nvivo then triangulated the evidence by mapping the findings to the evaluation framework and its KEQs. Where the evidence produced highlights contrasting results (such as differences between views of stakeholders or between qualitative and quantitative sources), these differences were reconciled by weighting the evidence collected by quality, consistency and its broader context (such as the likely interests of different stakeholders) and where possible using objective data gathered from monitoring information. Secondary data and monitoring data shared by the programme were analysed to extract key findings related to the achievement of outputs and outcomes. Furthermore, the contribution analysis approach demonstrated the complementarity of findings between the individual data sources. Taken in the round, this suggested that a medium degree of confidence in the accuracy of the evidence collected can be placed, except for a few notable limitations, which are described below.

2.2.4 Methodological limitations

• MI data incomplete: There was a 38% attrition rate for the whole programme participation. However, the evaluation was unable to explore how varied attendance might have impacted outcome realisation at the participant-level, for example, through a dose-effect analysis as suggested in the feasibility study (see Appendix 1 for more details). This is because individual-level data were incomplete or inaccurate
• participant self-selection biases for interviews with wider stakeholders: interview participants could decide for themselves whether they wanted to take part in evaluation activities. Therefore, the sample may not have included a range of responses that reflects the wider target population
• attribution issues: COF is one of several projects delivered by both Catch22 and The Prince’s Trust. Participating young people may have taken part in other programmes before, during or after their engagement with COF. Although each project/programme aims to develop a specific set of skills, they are all designed to enhance young people’s personal skills and employability. This created an attribution challenge for the evaluation, whereby it is difficult to be sure whether the outcomes achieved by participants are due to their participation in COF
• survey design: Some of the validated scales used were not used in their entirety, and this may have affected their validity (that is, they would have been validated on the full scale). This approach was used to reduce survey length and adjust timeframes to the programme delivery timescales
• survey sample size: Not all participants taking part in the programme completed a survey. Findings are based on relatively small sample sizes, which may not fully reflect the broader range of COF participants
• lack of counterfactual group: the feasibility of a comparative counterfactual analysis was explored during the inception phase: a comparison group was not identified. Due to the absence of a comparator group, the evaluation cannot make a robust assessment to determine whether the changes (both positive and negative) in any intended outcomes are directly attributable to COF programme activities, as opposed to other factors

3. Enablers and barriers to delivery

3.1 Introduction

This chapter summarises the key findings related to the process evaluation. Data for this chapter draws primarily on the findings from analysing programme MI and interviews. It starts with an overview of local needs, how process and governance were set up and managed between the consortium, and how participants were identified and referred onto the programme. It then considers what implementation approaches appeared to work well and what the key challenges were. Lastly, it examines the relevance and sustainability of the employment opportunities offered through COF.

3.2 Design and relevance

This section examines local contexts and needs in which the programme was implemented, whether the programme was successful in addressing these needs, and any duplication of existing provisions.

The COF programme was designed to work with young people aged 16 to 24 most at risk of violence (referred to as participants when engaged with the programme; see definition of ‘at risk’ in Section 3.4.1) in the LA areas of Birmingham, Bradford, Croydon, Hackney, Kingston upon Hull, Lambeth, Leeds, Liverpool, Manchester, Newham, Sandwell and Wolverhampton. These LAs were grouped into 6 regions based on geographical location:

• region 1. Birmingham, Wolverhampton and Sandwell
• region 2. Liverpool and Manchester
• region 3. Newham and Hackney
• region 4. Lambeth and Croydon
• region 5. Bradford and Leeds
• region 6. Kingston upon Hull

One regional coordinator and several career coaches were assigned to each region. Section 3.3 provides further details of the delivery structures, and staff roles and responsibilities of the delivery staff.

3.2.1 Local needs

The evaluation identified a diverse range of issues concerning the COF participant target group of (young people most at risk of violence aged 16 to 24) present in the COF delivery areas. The type and nature of these issues varied according to the stakeholders interviewed. For example, the most common need identified by delivery staff and stakeholders is that many young people are lacking in personal attributes and transferable skills that can enable them to progress into work, where: “Young people are very isolated and do not have the confidence and social skills ability” (Delivery staff). A lack of confidence was also frequently linked with a subsequent negative mindset as a barrier to applying for jobs in the first place.

“Some young people say that when they are applying for jobs, ‘No, they will never give me an interview’. So they don’t even try.” (Delivery staff)

Delivery staff and stakeholders also felt that issues around mental health, frequently either a misdiagnosis or lack of diagnosis, was cited as a major local issue. In fact, Figure 1 shows that 31% of participants had involvement with mental health services or counselling at the programme start. Delivery staff perceived that many of the young people they worked with, who often come from challenging or complex family backgrounds, exhibited signs of anxiety and Attention Deficit Hyperactivity Disorder (ADHD). These health issues, coupled with a lack of diagnosis often driven by a lack of adequate access to support services, presented a considerable barrier to employment for many young people.

“The biggest barriers I have come across are mental health issues. That is what most of the young people I meet come across too. Usually, that is in the form of anxiety, social anxiety, and depression, and some of them have experienced family troubles, so I think these mental health issues stem from a variety of different forces.” (Delivery staff)

Delivery staff and stakeholders reported a lack of available employment opportunities for young people in the COF regions. This was exacerbated by a lack of qualifications where, “A lot of the young people that we work with have either dropped out of school, or not really achieved many qualifications. You have got a massive shortage of opportunities for this client group in particular”, (delivery staff). The lack of opportunities meant that there was frequently a high rate of youth unemployment in the COF regions.

“So, for 16 to 24 year olds we have got the highest unemployment rates across the country, which is not a good place to be.” (External stakeholder)

Interviews with young people echoed this finding. The most common issue raised was their area had few opportunities available to them, or that there were opportunities, but the salaries for them were very low and there was a great deal of competition:

“Let’s say you are trying to find some work, it is not easy to find let’s say jobs and that. There are too many people applying. It is so hard to find one.” (COF participant)

Figure 1: Involvement with services at programme start

Source: COF programme data

Base: 941

Both external stakeholders and delivery staff mentioned the challenging and complex issues facing young people in the family setting as a barrier. They cited examples of young people often coming from a culture within the familial setting that had a negative attitude towards work. Breaking from that familial cycle and achieving a positive outcome, such as education or employment, was deemed a major challenge.

“So, in the area of Wolverhampton that I work in, a lot of the issues around the attitude to work and training is almost a cultural thing.” (External stakeholder)

The coalescence of these issues around confidence and social skills, mental health, a lack of opportunities and complex family settings were identified as contributing factors in driving young people towards gang activity, and the risks of serious violence that come with it. External stakeholders highlighted that a lack of opportunities can lead young people to joining gangs to earn ‘easy’ money, and as a result many of them might engage in violence and knife crime.

“These young people want sustainability, they want money, they want to have their own their own houses. Unfortunately, looking at jobs in our area, that appears to them not to be the best avenue to achieve that.” (External stakeholder)

Interviews with young people supported this, with some feeling that antisocial behaviour and high crime rates were major issues facing their local areas. Others responded that they felt unsafe in their local area and could see how gang membership was an attractive offer for some of their peers.

“It is quite unsafe… you have got to watch yourself wherever you are going. Who you are hanging around with. The area is really unsafe.” (COF participant)

Interviews with young people shed light on another issue not mentioned by delivery staff or stakeholders. In fact, young people cited a barrier was having to rely on public transport in their local areas, making it difficult for them to get anywhere without a car, where: “Travelling is always the worst thing. You know, when you do not have a car, you have to get a bus everywhere” (COF participant).

3.2.2 Does COF duplicate any existing provision?

There was consensus amongst delivery staff and external stakeholders that the COF programme, while similar in some ways to existing provision, did not duplicate any existing local or national provision. The programme’s approach of providing tailored one-to-one wraparound support to young people at risk of serious violence was described as a unique service for young people.

“I would say the unique selling point is that it helps young people with a criminal record try and find them employment. From what I know, this is the only one that I am aware of that support with something like that.” (External stakeholder)

However, delivery staff and external stakeholders acknowledged some overlap with a mix of employability / violence reduction programmes aimed at young people being delivered by other organisations. The table below showcases the range of projects they mentioned as similar, by region.

Table 2: Other youth support programme in COF local areas

Region 1	Region 2	Region 3	Region 4	Regions 5 and 6
Birmingham Youth Promise	Barclays Life Skills Programme	Prospects in Hackney	No specific programme mentioned	Cat Zero
Street League	Weapons Down, Gloves Up	St Giles
Impact Project		Flatmates
		Bounceback

Source: Stakeholder and delivery staff interviews

3.3 Processes and governance

Key to effective delivery of a large-scale programme split across 2 national providers are well-defined structures, clear roles and responsibilities, and effective working relationships. This section covers how working arrangements within COF delivery partners, and between these and external partners worked in practice and what could be improved.

3.3.1 Roles and responsibilities

The Home Office stipulated a single services contract with the consortium, with Catch22 as the lead provider. The programme was implemented and delivered in partnership by Catch22 and The Prince’s Trust (see Section 3.4.1 for division of work).

Apprentice Nation was responsible for signposting participants to recruitment events and opportunities in their local area, as well as supporting with the re-engagement of young people with COF. The Apprentice Nation online platform also offered a curriculum of content to young people who signed up to it. Topics include life and work, mental health and awareness, career planning and personal branding. The platform also offered a bi-annual, six-week live webinar programme and the opportunity to meet with professionals working in the technology, business and creative industries.

At Catch22’s central level, COF was overseen by an operations manager, with staff supported by a welfare and safeguarding officer, a data and compliance officer and the national account manager. At The Prince’s Trust, instead, it was overseen by a contract manager, supported by the head of contract performance.

On the ground, 2 groups of delivery staff employed by both Catch22 and The Prince’s Trust – regional coordinators and career coaches – implemented the programme.

The main responsibilities of regional coordinators included:

• organising, facilitating and delivering recruitment events, including job fairs, roundtables and employer summits
• establishing referral pathways with relevant partners
• navigating and coordinating the local landscape of available support for young people
• engaging with local businesses to identify employment-related opportunities
• facilitating the matching of young people with employment-related opportunities
• line managing career coaches in their region

Regional coordinators were recruited based on their experience in partnership management, recruitment and account management, as well as the ability to engage and communicate with a range of external stakeholders and programme delivery partners.

The main responsibilities of career coaches included:

• engaging, onboarding and maintaining communication with the participant during their time on the programme
• completing activities through their one-to-one support, such as CV development, job searches, arranging interviews and job applications with participants
• supporting participants to develop their mindset, self-confidence and overall esteem to help them succeed in the workplace
• helping place participants into career-focused employment or education opportunities that met their skills and matched what they wanted to do

Career coaches were expected to work with a range of young people and help them build achievable and aspirational outcomes. They needed to have experience of successfully engaging the harder-to-reach young people, build strong rapport with them, and be able to positively challenge mindsets.

These roles and responsibilities were generally perceived as clear, both by regional coordinators and career coaches. The latter felt a positive aspect of the role was having a degree of autonomy, allowing them to deliver their tailored support flexibly. Having clearly defined roles enabled career coaches to explain to participants what their role entailed, in terms of the type and frequency of support provided during their time on the programme.

The separation of responsibilities between the 2 roles was well-defined at the beginning of the programme, when full teams were in place.

“I do all the recruitment, I do all the outreach, I do all the partnerships, and the coaches can just concentrate on the young people, they do not need to worry about numbers.” (Delivery staff)

As the programme went along, however, experiences from delivery staff tended to diverge. Regional coordinators commented that, while their role was well established, some reported that certain responsibilities could have been delegated. Examples of this included having a dedicated staff member focusing on employer engagement and another dedicated staff member focusing on administration (participant enrolment, ID checks).

Interviews with delivery staff echoed this finding and indicated that the role of regional coordinators and career coaches could sometimes be blurred, or that they experienced crossover between the roles. For example, a few career coaches ended up having to temporarily take over responsibilities of regional coordinators, in addition to their usual responsibilities. This happened twice over the programme life, when regional coordinators resigned in post.

“We are wearing more than 2 hats because we have to be career coaches, we have to be regional coordinators and we have to provide one-to-one support, help young people with employability skills, reach out to referral partners…” (Delivery staff)

One cause of crossover was at the beginning of COF’s delivery, when a few regions did not have regional coordinators in post, therefore requiring the career coaches in those regions to fulfil a dual role in the interim. It caused challenges in outreach to referral and partner organisations, and also in providing support to participants engaged in the programme. Another reason for the crossover of roles was that some staff, at the onset of the programme, interpreted the role specifications differently and did not have a full understanding of what each role entailed, resulting in a crossover of responsibilities.

This resulted in a different interpretation of roles across regions, where some regions kept the 2 roles separate and well-defined, and others that took (driven by necessity) an inconsistent approach to the roles. At programme level, this was considered as a barrier to delivery, and led to frustrations due to a perceived lack of uniformity across regions, often to the detriment of the quality of support to participants.

“I can see that in a lot of other regions, their numbers are often much better than mine. I reckon that probably part of the reason is that they do just have more resources.” (Delivery staff)

Another barrier highlighted by delivery staff interviewed was lack of clarity around work processes and inconsistency across regions and delivery organisations. This included differences in the operation and guidelines between Catch22 and The Prince’s Trust. In relation to the former, staff indicated that the lack of clarity is because work processes have changed multiple times since the beginning of the programme. These changes included additional compliance and administrative and reporting processes, which may not have been applied consistently across all the regions. While career coaches recognised the need for compliance and reporting processes, they felt these took valuable time away from each participant. This particularly made it difficult for staff to get used to work processes and often created inefficiencies as they constantly had to adapt to new ways of working in relation to administrative procedures and reporting requirements.

“I think that the processes are too many… They have been ever-changing, and nothing has been set in stone from the beginning.” (Delivery staff)

3.3.2 Cooperation and communication within the consortium

There was mixed evidence on collaboration between COF delivery partners – Catch22, The Prince’s Trust and Apprentice Nation. However, there was some evidence of collaboration amongst delivery partners – Catch22 and The Prince’s Trust. For example, if a Prince’s Trust staff member joined a Catch22-led region, they felt supported in settling in and navigating the various systems and processes. Catch22 and The Prince’s Trust held weekly meetings, which delivery staff noted were useful for resolving issues between the 2 organisations during the programme delivery.

However, discussions with delivery staff indicated a lack of inter-regional communication as a barrier, with staff within each region feeling isolated from the rest of the programme. Delivery staff from both Catch22 and The Prince’s Trust frequently commented that they were not aware of what caseload other career coaches had, what approach to recruitment was being used, what stakeholders each organisation were reaching out to, and whether their staff could use further support/ had capacity to provide support.

This heightened a perceived lack of team building opportunities across the wider programme, and sometimes within regions that had a mix of Catch22 and Prince’s Trust staff. This need was particularly evident at the beginning of the programme, where shared objectives and approaches could have been agreed upon and clearly understood by all delivery staff. Some stakeholders mentioned developing and establishing a steering group in regions before the commencement of the programme could have mitigated this.

“I may have set up some sort of steering group for Wolverhampton, to engage relevant partners, and we could have helped inform who those relevant partners might be, and as the programme developed, we might have engaged other people.” (External stakeholder)

While Catch22 and The Prince’s Trust broadly operated similarly, staff felt that there were differences in the delivery of COF. For instance, how communication with the participant was handled (particularly during disengagement) and how participants were supported in finding employment.

“In my point of view, we might do similar things, but we do work in different ways.” (Delivery staff)

Efforts to resolve these challenges often ran into their own difficulties, highlighting even further a lack of clear synergies in place between the 2 organisations delivering the programme. In fact, both of them had their own governance and management models, which extended into the delivery of COF. As an example of this, Prince’s Trust staff working on a Catch22-led region did not report directly to the Catch22 regional coordinator of that specific region. This was seen as a barrier and led to confusion and uncertainty around reporting responsibilities and procedures across staff. As a result, some members of staff cited a breakdown in the expectations that were placed on them.

“I am supposed to check in with them. But they have their own management, and they are responsible for their side of things. We just check in with them. But we have found that the career coach is just not attending appointments, meetings and they are just not emailing back.” (Delivery staff)

These issues indicated the need for more collaboration between delivery partners at a regional level, as opposed to taking an individual provider-led approach, to enable effective joint programme delivery. Holding frequent, face-to-face forums and meetings where staff could engage with one another and foster a sense of belonging was suggested as a potential way to mitigate this issue.

Lastly, there were delays regarding the Data Sharing Agreement (DSA) between Catch22 and The Prince’s Trust. The DSA allowed the 2 organisations to collect, share and retain data on COF participants. Without a DSA in place, Prince’s Trust staff did not have access to Aptem and, as a result, were not able to onboard young people on the programme. Issues were resolved in early April 2022.

This delay prevented Prince’s Trust staff from delivering COF activities and engaging with the evaluation activities until the DSA was in place, and also prevented access to Aptem (the administration system), for data protection reasons. This also had major implications for the survey rollout, as Aptem automatically generates unique reference numbers used to match survey responses. It also delayed engagement with the other evaluation activities, such as interviews with both participants and stakeholders.

3.4 Implementation

Catch22 and The Prince’s Trust established a set of steps in the COF implementation model. These broadly include:

• identification and engagement with young people through a combination of direct, targeted referral pathway and open self-referral routes
• assessment of participants’ specific needs to provide them with an individualised support structure
• provision of wraparound support based on their specific needs identified based on the assessment(s) carried out previously

This section analyses each implementation stage in turn, examining what worked well at each stage, followed by what did not work so well and areas for improvement. Evidence provided by the interviews with stakeholders and programme delivery staff informed the analysis (differentiated between regional coordinators and career coaches where necessary).

3.4.1 Identification and engagement

This section assesses the progress towards achievement of the programme targets by deep-diving into the enablers and barriers to the effectiveness of the referral processes. It also considers the sources of referrals and how adequate staffing levels were considered to achieve the targets.

Referral process

Participants were referred to the programme by relevant local agencies or via self-referral. This was carried out through a combination of a direct, targeted referral pathway and open self-referral routes. Targeted referrals took place through local organisations that formed a partnership with the delivery organisations (that is, Catch22 and The Prince’s Trust) based on their targeted in-person and virtual outreach. Referrals also came from other national and local training providers who may have worked with young people affected by youth violence. Both young people who self-referred onto the programme and those referred by partner organisations had to complete an online referral form through the COF website.

Broadly, most participants, staff and stakeholders interviewed found the whole referral process, including completion of the form, to be fit for purpose. They described it as quick, user-friendly and straightforward. In particular, several highlighted the ease of the referral process due to it being completed via an online form.

“… the referral process could not be any easier if you tried. And I have got to be honest, it is a really short, straightforward referral process.” (External Stakeholder)

However, a few external stakeholders raised a perceived lack of clear communication around the referral process, which included the main benefits for participants and the type of support provision.

“The communication needs to be very clear … you need to signpost that this service is free, this is how you can refer young people, this is when you can refer them from, this is what the help will look like.” (External stakeholder)

This led to a perception amongst some referral organisations that the referrals were delayed due to COF staff not contacting participants in a timely manner, and frustration around having to chase COF staff for processing the referrals. A minority of stakeholders also noted that the amount of work that was associated with the referral process was unexpectedly more than they had originally anticipated. Therefore, more work could be done to give stakeholders who intended to refer onto the programme a realistic understanding of the amount of time and resources needed when providing referrals.

In addition, delivery staff identified the requirement for participants to provide their identification as part of onboarding to be a barrier. ID was in fact needed to enable participants to access jobs and other opportunities offered by the programme. Particularly, how long participants took to share their identification delayed the onboarding process, while some participants who did not possess official identification required extra support in acquiring it to enable their onboarding.

“It will get to the point where I have started the onboarding and then it takes weeks to send me a copy of their passport. And I can’t actually put them as a start.” (Delivery staff)

**

Programme eligibility criteria

The COF programme aimed to work with participants aged 16 to 24 at the time of referral. The consortium of delivery providers had to identify young people suitable for COF working with local agencies. The Home Office indicated that the young people should have come from the following target groups (also referred to as ‘at risk’):

• young people in contact with YOTs or in Youth Offender Institutions
• young people in alternative provision
• young people persistently truanting
• young people in care
• young people seen in A&E with knife injuries
• young people engaged by the Supporting Families programme

Analysis of MI programme data showed COF engaged 203 at-risk young people (21%). Of these, 91 stayed on the programme and 112 disengaged from COF activities (see Section 3.4.2 for more details), outlining that delivery providers had a limited degree of success in identifying young people from the target groups above.

A participant was marked as ‘disengaged’ following 4 weeks without any contact with their career coach. Coaches were then required to check in with disengaged participants every fortnight up to 8 weeks from the point of recorded disengagement.

At programme start, all participants were asked a series of questions as part of their onboarding process. Analysis of this data shows that the main barrier faced by COF participants at the outset was related to employment, or lack thereof, as opposed to most at risk of violence. Almost three-quarters of participants (593) were NEET (not in education, employment or training), with specific barriers related to attaining employment or receiving further education or training.

Furthermore, Figure 2 shows that only 21% had prior involvement with the criminal justice system (CJS), 9% was involved with gang activity and 2% depended on alcohol/drugs (considered the ‘hardest-to-reach’ sub-group).

Figure 2: Self-reported offending and substance abuse at programme outset

Source: COF programme data

Base: 941

Notes:

1. Multiple response options in the Aptem data meant that these categories are not mutually exclusive.

Interviewed delivery staff and stakeholders shed light on why the identification process used in the delivery model sometimes struggled to reach intended audiences. A key weakness participants identified related to challenges and lack of resources and capacity to build connections with professionals working with harder-to-reach young people. These included YOTs and the prison service. Delivery staff commented that the prison service (across several regions) would have been a particularly valuable source of referrals. In fact, COF staff could have met young people directly when they were about to be released from custody and begun providing support immediately.

However, no interviews with delivery staff reported that relationships with these types of partners had been fully established. They felt that there were often short timeframes for staff indicating a lack of capacity and resources to building trust and close connections with these organisations. Interviewees felt the COF could do better at targeting at-risk young people. This could be achieved through several suggested means, such as reaching out to schools to get young people involved in COF even before they become school leavers.

The struggle to engage harder-to-reach young people is also evidenced by further analysis of MI programme data (Figure 3) showing that only 13% participants were referred by YOTs; a further 9% came from alternative provision schools (APS) or pupil referral units (PRUs), although these only came from 5 out of 12 COF LA areas; and 1% from probation services, A&E services and violence reduction units (VRUs). Internal referrals (Catch22 or Prince’s Trust referrals from another programme within their portfolio) accounted for 10% of participants.

VRUs provide a central focus for violence prevention in their areas, where a public health approach is adopted for reducing serious and violent criminal activity. VRUs were considered a key stakeholder for the COF programme, boasting the programme’s knowledge of the local landscape, referral routes and helping to leverage local opportunities. Only Manchester had participants referred by VRUs and only 2 COF LA areas (Manchester and Birmingham) had participants from A&E services.

Internal referrals from Catch22 services included from courses such as Care Support Worker, Health and Safety, First Aid at Work, and Manual Handling. Prince’s Trust referrals came from services supporting personal development through programmes such as Groundwork.

Figure 3: Referral source

Source: COF programme data

Base: 941

In terms of the self-referrals, participants reported that they found out about COF through social or community services, or through local organisations. These services included family counselling, work coach at job centres, social worker, support worker, care leaver personal adviser; or through women’s aid, local community centres, local events/festivals, Career Connect and YMCA among others. In addition, they also mentioned they previously heard of COF from friends, family or social services but had not engaged with them.

When asked about their motivations and aspirations in joining COF, all participants interviewed also felt COF would bring certain benefits they would be unable to achieve on their own. While some indicated they were convinced by family members, others were referred by friends who were already taking part in the programme. Meanwhile, several young people mentioned their initial contact with delivery staff put them at ease about joining to achieve specific goals in relation to employment (for example, CV support, job search, networking opportunities with employers through events), training (for example, Construction Skills Certification Scheme (CSCS) card), starting their own business, or simply to figure out next steps in their career path.

“I was very open about taking part, just you know anything to help me find something. Since I’ve not got a lot of experience with this type of thing.” (COF participant)

Outreach activities

When commenting on the quality of relationships with referral partner organisations (outreach activities), one of the key success factors relating to developing partnerships with local organisations was having pre-existing relationships with partners. Most regions had pre-existing links, but the extent of their leverage varied across regions. Some of these partnerships included:

• Movement to Work (MTW) (this facilitates quality vocational employment and work placement opportunities and linking unemployed young people with employers)
• Youth Offending Teams
• care Leaving Teams
• children and Young People teams in local authorities
• Jobcentre Plus (DWP)
• The Prince’s Trust had established programmes with TK Maxx and Marks & Spencer

Delivery staff also identified in-person outreach efforts or physical presence as a key enabler when engaging with stakeholders. They indicated that compared to virtual outreach or communication via email or telephone, when they met with stakeholders in-person or attended networking events, they were more successful at forging a fruitful relationship with them as referral partners.

“I would definitely say that physical presence is a massive thing … that day [when we met in-person] I probably received 12 referrals, as opposed to 1 or 2 when I have done virtual meetings.” (Delivery staff)

This was further reinforced by external stakeholders, who felt they had a good understanding of how the programme worked due to these efforts made to build positive working relationships with them.

“We have developed a good understanding of how our young people work, how COF works, how they can benefit each other, how we can fit in as well.” (External stakeholder)

Delivery staff pointed out that to achieve these strong relationships, they proactively ran events to encourage new stakeholders to refer participants onto the programme. Region 2 in particular mentioned the success they had in developing positive relationships with public sector organisations and youth support organisations.

Another enabler to recruitment of local organisations was the contacts given by the Home Office. Staff indicated that this approach allowed them to quickly progress with forging relationships with referral partners and getting more young people referred onto the programme.

“… actually being fed those contacts directly from the Home Office was helpful and more of that would have been really good.” (Delivery staff)

While the partnership model and ways of working had been clearly laid out in the proposal, this did not always translate into a clear model during programme delivery. Although mentioned by only a handful of delivery staff, one barrier to recruiting local programme partners was lack of information sharing and communication, resulting in both Catch22 and Prince’s Trust staff frequently followed their own approach to outreach when speaking to potential partner and referral organisations. This led to a duplication of effort creating confusion and inefficiency, where staff in one region separately reached out to community centres, youth centres, charities and local employers, unaware that these organisations had already been contacted. Interviews with delivery staff found that this created hesitancy when reaching out to potential partner organisations in the community, as they may have already been contacted.

“I am not aware if Catch22 have already made that contact … I think that information, we could really utilise from each other and utilise each other’s contacts a little bit more [to] make this programme much more successful.” (Delivery staff)

The approach to outreach could have been improved through closer integration, communication and delivery between Catch22 and Prince’s Trust. This could have achieved an improved understanding of what the key eligibility criteria established by the Home Office were and, therefore, lead to less divergent approaches to outreach. As evidenced elsewhere in this report, delivery staff from Catch22 and The Prince’s Trust frequently viewed the other partner organisation as delivering their own programme separately under the banner of COF, rather than acting as part of a wider, integrated consortium. This corresponded with different approaches to outreach and application of the eligibility criteria (including the prioritisation of employment status) when outboarding participants.

There was consensus that a more joined-up process would also result in greater impact. One example was the approach taken to reach out to the Job Centre, when there was no crossover in contacting this stakeholder and referrals were balanced across the 2 delivery organisations. Activities like this were found to be a good example for strengthening the relationship between The Prince’s Trust and Catch22, complimented by effective communication.

“We took a joint approach in trying to be more present with the Job Centre and also, we have booked some workshops for the new year as a joint approach with The Prince’s Trust. So, I think it is a relationship that’s definitely strengthening and will benefit the young people.” (Delivery staff)

While the overall level, frequency and responsiveness of communication with stakeholders was felt to be positive, they cited a few specific elements as areas for improvement. External stakeholders felt that the programme could be more proactive in providing feedback about the participants they had referred onto the programme, such as whether a participant had achieved an outcome, was still on the programme, or had disengaged entirely. Stakeholders commented they had to reach out to the programme directly if they wanted an update on their referrals.

“Just to keep the referrers in the loop, so they know actually what help they are getting and what is going on.” (External stakeholder)

Staffing levels and targets

At delivery level, each COF region had a dedicated regional coordinator and several career coaches, depending on the size of the region and engagement targets over the programme’s lifespan. Table 3 shows staff allocation and actual number of participants starting the programme compared to the targets set out by the delivery providers, by COF region. While none of the regions achieved their targets, Liverpool and Manchester emerged as the region with the highest target achievement rates, followed by Newham and Hackney.

Table 3: Staff allocation, referral targets versus actual starts by region

—	Region	Regional coordinator	Career coaches	Target	Actual	Percentage
1	Birmingham/ Wolverhampton/ Sandwell	One (Catch22)	Four in total: 2 from Catch22 2 from Prince’s Trust	432	262	60%
2	Liverpool/Manchester	One (Catch22)	Three in total: 2 from Catch22 one from Prince’s Trust	328	272	82%
3	Newham/Hackney	One (Catch22)	Two in total: one from Catch22 one from Prince’s Trust	216	148	68%
4	Lambeth/Croydon	One (Catch22)	Two in total: one from Catch22 one from Prince’s Trust	216	116	53%
5	Bradford/Leeds	One (Prince’s Trust)	Two from Prince’s Trust	216	109	50%
6	Kingston upon Hull	One (Prince’s Trust)	One from Prince’s Trust	104	34	32%
—	TOTAL	6	14	1,512	941	62%

Source: COF programme data

Base: 941

When looking at the programme uptake over time, Figure 4 shows that, despite establishing a set monthly target, the actual number of participant starts varied considerably from one month to the next. This trend was even more evident during summer holiday periods (that is, June and July 2022) as well as December, when a low number of programme starts was recorded.

Figure 4: Programme uptake

Source: COF programme data

Base: 934

Providing a high-quality service to participants while achieving targets and balancing a large caseload proved a challenge at times. The main issues they cited were staff turnover, too ambitious targets and area coverage.

The evaluation team found that staff turnover was a major issue faced by the programme, where 55 staff left the programme between October 2021 to February 2023. Staff left the programme for a number of reasons, but a deciding factor was burnout. Staff leaving the programme had a knock-on impact on delivery, where small teams lacked in-depth capacity to cover caseloads as replacement staff were brought into post.

The programme did implement steps to mitigate the impact of staff turnover and minimise disruption of support being provided to participants. It introduced regional caseload reviews when a staff member left, where regional coordinators would introduce themselves to the participants on a caseload and inform them a new career coach would be in place soon, as well as conducting check-ins with those participants. The programme’s strategic stakeholders felt this was an effective mechanism for reducing participant dropout and ensuring continuity of support when delivery staff left the programme.

“But it [dropout] has been very minimal. But that is because, obviously, we have been working with a young person.” (Delivery staff)

The programme introduced longer notice periods to mitigate the disruption caused by staff leaving. It increased notice periods from 2 to 4 weeks for career coaches, and from 6 to 12 weeks for regional coordinators. This gave the programme more time to identify and recruit new staff and, when needed, prepare participants to transfer to a new career coach.

Staff burnout might also have been caused by excessive caseloads. Although there was no clear guidance or indication of what an expected caseload should be per career coach, evidence collected through interviews with delivery staff suggested that staffing levels were broadly perceived as acceptable only when a caseload was kept below 50 participants per career coach at any given time. Most career coaches also added that a caseload of 40 to 45 participants allowed them to devote sufficient time to currently enrolled participants and actively pursue disengaged ones.

“I think if we had 50 people on the caseload and they were all engaged, we would know that that caseload would be too big, but we have factored in the fact that they are not all going to engage at the same time.” (Delivery staff)

Relatively small teams, coupled with shifting staff levels, meant that a few career coaches struggled to manage their caseloads. This, in turn, led to excessive workloads for career coaches, who were reportedly working past their working hours in order to deliver a good quality service to participants. Staff turnover – especially when multiple members of staff left at once – inevitably caused significant disruption to delivery and affected the continuity of work with participants. If larger teams were in place, the impact of turnover could be mitigated by re-allocating caseloads to existing team members.

This meant that participants reported, in a few instances, a lack of continuous support instead of having a dedicated team focused on their needs. Interviews with regional coordinators also highlighted that, while their role was well established, certain responsibilities could have been outsourced to other staff members. Particular examples of this included having a separate staff member focusing on employer engagement, and another one focusing on administrative tasks (participant enrolment, ID checks).

External stakeholders reinforced this by indicating a lack of availability with career coaches to meet the needs of participants when they started engaging in the COF. While they were aware of career coaches having multiple responsibilities and having other young people on their caseload, they felt more availability to tend to the needs of young people would have led to more beneficial engagement. Echoing the sentiment that the success of young people engaged in COF depends upon their specific needs and barriers, a minority of stakeholders mentioned that the success of referrals made into the COF programme is slow to become visible, hence, the programme needs longer engagement with participants.

“I think some of those advisers and their availability maybe. And I think some of our young people when they connect, they might want to see more presence [from advisers].” (External stakeholder)

Additionally, when a single member of staff was allocated multiple LAs, as with most regions, both regional coordinators and career coaches felt it became challenging to build effective contacts in each LA, and that they would have preferred to focus on a single area.

“So, having one careers coach to manage 2 boroughs, 2 challenging boroughs, absolutely not. We needed one person in one borough and another person in the other.” (Delivery staff)

External stakeholders also perceived this, mentioning the large geographical coverage of the programme in each region as a barrier, which they believed stretched the capacity of delivery staff to the detriment of the quality of support, and negatively affected their ability to build rapport with more localised stakeholders.

“I think the actual aims and objectives of the programme are fantastic … but they’re just not present enough to raise the profile of the programme and delivery.” (External stakeholder)

Another key barrier highlighted was a lack of clear understanding about programme objectives and targets. Specifically, career coaches mentioned not having received clear guidance from their leads and management regarding the monthly referral target, rather they were instructed to simply “bring in referrals” using a ‘blanket approach’ to setting delivery targets. Specifically, they explained that dividing up the programme-level total target by the number of months in a year ultimately disregarded natural fluctuations in incoming referrals. Many of the staff iterated that monthly referral targets should have been lower or more flexible. In addition, staff highlighted seasonal fluctuations in incoming referrals as a barrier to reaching targets. For instance, the holiday period spanning December to January yielded little success in securing referrals.

“I thought, ‘That’s quite unrealistic’, because the target is the same every single month, but external plus internal factors are not taken into consideration to know, actually, this could really affect the numbers.” (Delivery staff)

3.4.2 Assessment of participants’ needs

Once enrolled onto the programme, career coaches undertook a series of assessments to better understand each participant’s specific needs and be able to personally tailor the offer of support. MI data indicated that the following assessments were carried out:

• one-to-one and digital inductions: Through one-to-one discussions with participants, career coaches aimed to understand their motivations and aspirations in joining the programme. Each participant also received an induction on Catch22’s Aptem case management system, demonstrating the digital support and training available, as well as being introduced to the Apprentice Nation platform
• identification and assessment of each participant’s needs, including Special Education Need or Disability (SEND): Career coaches conducted a need assessment, including prior attainment of the participant and functional skills assessment. This assessment was then used to develop a Bespoke Action Plan (BAP) for each participant. In addition, this included identifying participants with a SEND
• RAG rating: Participants also had their needs Red-Amber-Green (RAG) rated based on an assessment of their needs and distance from their career goals. At onboarding, all participants were RAG-rated red, indicating that all required support in achieving their goals. The RAG rating informed the level of wraparound support each participant received, as well as the activities carried out in the BAP

Many of the delivery staff identified this multi-step, and often long, onboarding process as a barrier to sustainable engagement of participants onto COF. Particularly, delivery staff found it challenging to ensure completion of multiple forms and acquiring identification from young people. This long onboarding process sometimes:

• encroached upon time that could be used for support provision

“It’s definitely paperwork heavy in terms of the application. There’s also a bit of a delay between getting a young person signed up on Aptem and then finish[ing] their application.” (Delivery staff)

• made it difficult to manage expectations of participants, who expected quick results in terms of employment and training

“Getting them to sign up onto the programme was a little bit trickier … if we could just take the forms away or if we could take away the fact that they have to give us their ID … [would] make it easier for that young person to sign up initially.” (Delivery staff)

In relation to assigning a RAG rating to participants, programme data in Figure 5 shows a snapshot of data on 13 February 2023 (note that Aptem allowed for overrides, hence it was not possible to capture progression over time at an individual level). One in 4 (28%) participants on the programme remained on a red rating, with 11% moved to amber rating, meaning they are one barrier away from job readiness. Five percent of participants moved to a green rating, indicating they were closer to achieving their goals and job-ready. More than a third of young people (38%) became disengaged or ‘off-programme’. And 18% of participants had ‘in work’ as their current status. Aptem did not record when the participant had their RAG rating changed on the system, so it was not possible to see the exact time taken to move between ratings.

Figure 5: RAG rating and status of participants (programme level)

Source: COF programme data

Base: 941

Participants at-risk of violence

The programme engaged 203 at-risk young people (21% of programme cohort). Of these, 91 stayed on the programme and 112 disengaged from COF activities (that is, 55% of all at-risk participants disengaged from the programme).

When looking at the total number of participants who disengaged (357) with or went off-programme, analysis showed that almost a third (31%, 112) were participants at risk of violence or with a history of offending. This suggests that although delivery partners strived to target the ‘hardest-to-reach’, the programme was not sufficiently equipped to engage with and address their specific needs over an extended period of time. Further analysis presented in Table 4 provides a breakup of the different types of at-risk young people as a proportion of:

• the total number of participants who disengaged/went off-programme (357)
• the total number of at-risk participants who disengaged (112).

This breakdown of at-risk groups shows that the most vulnerable young people did not tend to stay on the programme, particularly those involved with police and other criminal justice services, those with a current or previous sentence, or those with a caution or reprimand in the last 12 months. Although it is important to acknowledge that there will be other factors contributing to such participants’ disengagement from the COF programme (such as, their motivation or reachability), the high levels of disengagement indicated that the programme design should be modified to allow career coaches to engage and support them more closely. This could be through lower caseloads and stronger and direct links with youth offending teams, and prison and probation services.

Table 4: Subset analysis of at-risk participants that went off-programme/ disengaged

At-risk sub-group	Number of at-risk participants who disengaged (total cohort = 112)	% of at-risk participants of total disengaged cohort (of 357)	% of at-risk participants of total at-risk disengaged cohort (of 112)
Involved with the police/ criminal justice/ legal/ probation services	99	28%	88%
Currently or previously in custody/ probation/ subject to any other sentence	83	23%	74%
Conviction/ caution/ reprimand/ warning in the last 12 months	75	21%	67%
Drug/alcohol dependence (past or present)	29	8%	26%

Source: COF programme data

Base: 357 (total disengaged participants)

Notes:

1. Multiple response options in the Aptem data meant that these categories are not mutually exclusive, and sub-group percentages do not add up to 100% because a young person can fall across multiple risk categories at once.

Throughout the course of the programme, each participant was assessed on whether they were outcome ready and prepared to start work/training. This involved further support from their career coach, including follow-up calls. If the participant did well in the role, they proceeded to their exit review. If they did not perform well, participants underwent a work/training review and re-entered the stage of ongoing support. This stage also included re-engagement with participants who disengaged with the programme.

3.4.3 Provision of wraparound support

Once onboarded, participants were matched with a dedicated career coach who acted as a trusted adviser to them throughout their time on the programme. The COF delivery model was designed to deliver an ongoing and holistic programme of support to enable participants to achieve one of the programme’s outcomes: employment, education and training. The participant received ongoing support and communication from their career coach throughout the duration of the programme, but at this stage focused on:

• financial support
• one-to-one tailored support and activities
• signposting to further support, both internal and external

Financial support

This included the provision of financial support as a £300 bursary, providing a personalised budget for each participant. Its aim was to help participants overcome some of the financial barriers they faced (as discussed in Section 3.2.1). In particular, it was used to:

• identify and access available job opportunities
• access better transport links
• improve IT skills
• obtain employability documentation (multiple career coaches mentioned the CSCS was particularly popular with participants)
• obtain ID
• fund training courses

Besides the bursary, COF participants also had access to a laptop loan scheme intended to support especially those experiencing digital poverty. MI data show that 10 laptops were given as part of the scheme. By receiving laptops, participants had greater access to the internet, could improve their IT skills and more easily access further support and services.

“[Career coach] has helped me out a lot with my passport, my birth certificate. She has also got me a laptop sorted. So it has been really good.” (COF participant)

Delivery staff agreed that this suite of financial and IT support undeniably contributed to make participants more employment-ready.

“A lot of the times young people have a stumbling block in regard to finances, that is, not having a laptop … So I talk to them about the positive opportunities that can develop from this programme, not only to mentor them, but actually give them tools.” (Delivery staff)

However, according to some delivery staff, the administration and reporting of processes around the bursary and laptop loan scheme were perceived as burdensome and confusing. There was confusion on how to record and keep track of spending due to different processes used by The Prince’s Trust and Catch22.

“The admin and the reporting procedures … how to get bursaries for example, there’s a lot of confusion about that and then how to report things properly.” (Delivery staff)

One-to-one tailored support and activities

A wide array of tailored, one-to-one support activities were provided to meet each participant’s individual needs. The figure below shows the number of participants who were supported by career coaches, broken down by activity/ type of support.

Figure 6: Uptake of COF activities

Overall, participants found the support and activities provided by career coaches beneficial, indicating that they benefitted from interacting with career coaches, who they described as a positive presence in their life. Being able to relate to the young people taking part in the programme on a personal level and share their own lived experiences was also deemed extremely important.

When probed on reasons, participants frequently referred to one-to-one sessions creating a safe environment where they were encouraged to learn and progress with their lives. Relationships and trust were built. Attending sessions with career coaches had a positive effect in terms of being able to socialise, talk and spend quality time, especially for more socially isolated groups.

COF-supported activities were key enablers for increasing confidence in themselves and employability skills through:

• in work/training support, which often-combined CV review and mock interviews into one longer session. Having the participants’ CVs reviewed by a professional, and the reassurance that came with it, were felt to make a real difference

“We have been through my CV and stuff like that. He also helped get me clothes for interviews.” (COF participant)

• mock interviews were likewise felt an effective way of building up the participants’ confidence to show that they could go through with real interviews

“I would do another mock interview with them just to find out if they understood what was said the first time. Again, it’s just filling their confidence so that they are confident to answer the questions with minimal issues.” (Delivery staff)

• self-assurance contributed to developing transferrable soft skills, such as increasing confidence and developing communication skills

“I’ve seen my confidence just shoot up. I put that down to this programme.” (COF participant)

• mentoring, which was found particularly beneficial by participants with previous criminal convictions. COF activities were described as instrumental in helping to develop leadership skills among participants who often felt more confident and capable to find a job

“We go in and we give them that confidence that there [are] other things out there, so it has been really good for young people who, the youth offending side of things, to get to meet with them and give them that hope that you can still get a job, that it is not the end of the line for you.” (Delivery staff)

• skills training, both in-house skills development workshops and external

“[Career coach] helped me start on the Digital Leap [course], and that has been really, really helpful for getting into IT.” (COF participant)

• language, literacy and numeracy (LLN) skills training, both in-house and external support towards building up IT skills (for example, online safety, Microsoft Word, setting up email account) and English language skills

“It was a Fundamentals Basic IT course. It lasted a few months, and it was highly beneficial.” (COF Participant)

Delivery staff and participants indicated that the wraparound support model was unique and offered flexibility to fit the needs of participants. In relation to the flexible nature of the support, they mentioned a few positive aspects as:

• frequent and regular catchups with participants made the support provided very effective

“Having that frequent meeting with their adviser and being able to pick up the phone just for a bit of advice and guidance, that is key. And that’s really quite pivotal to this programme.” (Delivery staff)

• matching participant skills to their needs and interests

Nearly all participants interviewed reported receiving ‘customised’ forms of support, which was highlighted as contrasting with the other employability programmes attended by some of them. Coaches also provided them with information and opportunities that were relevant to their interests. Several participants also highlighted the personalised guidance and feedback provided to them by coaches that helped them identity the areas that needed to be further developed. They attributed this to be a key enabler of positive outcomes.

“[My coach said], ‘I’m going to get you on that course.’ There was a lot of people going for it, and she said, ‘I’ll make sure, because I know how enthusiastic you are about this.’” (COF participant)

“It is a personal relationship. It is not just like they are reading off a sheet. I’ve had that before when they are doing that, and not customising things to you.” (COF participant)

• the role of mentor and adviser, providing guidance as and when needed by participants

Most participants reported being consistently nudged and encouraged by coaches, which boosted their confidence for applying to jobs and provided them with a sense of purpose. Delivery staff also reported such forms of positive impact.

“[Career coach] finds courses that are best suited to me, and she asks me what I want … it is always just a reassurance.” (COF participant)

• the provision of one-to-one personalised support provided to participants based on personal circumstances, which was crucial to building rapport with career coaches and resonated with participants who were not comfortable in group settings

Several participants also reported receiving additional support from the programme based on their unique personal needs. Such support took the form of infrastructure that could enable participants’ educational or job search activities, such as a laptop, bus pass or travel fund to pay for interviews. Other forms of support included help with obtaining ID documentation, such as passports or birth certificates or writing offending disclosure letters for potential employers. A third form of personal support was related to wellbeing, such as housing support, or health and fitness or mindfulness courses to help participants’ anxiety issues.

“A lot of the young people missed their formative years of being in a group setting. So being able to meet them on a one-to-one basis and focus on their needs is just fantastic.” (Delivery staff)

“[Coach] has helped me write a disclosure letter for convictions for when I apply for a job. She has also helped me with getting PPE [Personal Protective Equipment] for a job.” (COF participant)

• specific support to 16 to 18 year olds

Several participants from the younger cohort reported receiving personalised employment-related support provided by COF. They highlighted how the support received from career coaches enabled them to achieve greater success at securing employment because of learning how to talk to employers about specific abilities, strengths and what they might offer as an employee during interviews. Delivery staff interviews echoed this finding and highlighted that support for this age group is usually not available through other employability programmes.

“The 16 to 18 year olds are not captured anywhere. If they’ve left education, it seems to have lost them. So, I see quite a positive response from the 16 to 18 year olds as we are investing in training and support for them.” (Delivery staff)

• the flexibility and ease of engagement through in-person presence of COF delivery staff during face-to-face drop-in sessions with participants at the initial stages helped them transition onto the programme

Several participants reported coaches let them choose how to engage (for example, via in-person meetings, virtual sessions, phone calls, text messages or a mix of these), and set the pace of engagement based on their needs or life circumstances (such as family commitments, illness or travel). Delivery staff also noted such flexibility as key to positive impact.

“[Career coach] has been very good with trying to meet me and come to the area that I live, so, there’s not been much difficulty there.” (COF participant)

“If had some sort of cohort or something of young people who met every month so that they could speak to each other about what each other’s career coaches are doing or something.” (Delivery staff)

However, a handful of participants felt they had not been offered certain types of support through the programme. Particularly, these young people mentioned a lack of group activities, with participants often only engaging in one-to-one support with their career coaches. While this suited the needs of many participants, there was nonetheless a feeling among a minority of career coaches that there could have been more group activities.

Signposting to further opportunities

Overall, career coaches used signposting to raise awareness of further support and activities available locally, regionally and nationally. Table 5 shows the number of participants they signposted to further types of support, either internal or external organisations, if in-house support remained outside of remit.

Table 5: Internal and external signposting

Internal signposting	Number of participants signposted
Business and entrepreneurial programmes	4
Health and Wellbeing programmes	12
Employability programmes	48
Other internal programmes	69

External signposting	Number of participants signposted
Housing	26
Functional skills	24
Financial	19
Drug/Alcohol misuse	18
ESOL	16
Gang prevention	16
Gambling	15
Legal	15

The evaluation found that signposting participants to different types of support and services, was cited by both stakeholders and delivery staff to be something that worked in the delivery model. Stakeholders felt this element of the delivery model was particularly important for young people who were at risk of, or had been affected by, serious youth violence. It was also another element of the tailored, personalised approach used by the programme, whereby a participant could be signposted to services that suited their individual needs.

“It seemed like it is quite holistic and the whole point is that, you know, they work with young people that are affected by serious youth violence.” (External stakeholder)

Delivery staff likewise felt that the signposting of services meant that participants could access further, more specialist support easily available to them that allowed participants to continue moving towards positive outcomes.

“So the signposting has been great, because that is what the young person [needs] to get to the next level, and that’s what we’ve seen quite a lot.” (Delivery staff)

Relevance, quality and sustainability of opportunities

Career coaches provided wraparound support to participants with the overall aim of getting them ready for employment. To this end, once job-ready, the programme matched participants with potential employers and began the job application process. As an alternative, they also completed education or training that was directly relevant to the jobs sought.

The evaluation found that the relevance and quality of opportunities offered through COF was positive, with 171 participants achieving an outcome relevant to their needs. When discussing the relevance of different types of opportunities offered, delivery staff highlighted that participants found support towards getting them job-ready or help them start a career in a specific field to be most relevant. For instance, participants interested in the creative sector, such as music, acting or arts, were directed to the Apprentice Nation platform, which was found beneficial as a first step in their career.

“We talked about Apprentice Nation as well and Catch22 and Prince’s Trust programmes. Some of them might be connected to music, so it is easier to put them onto Apprentice Nation.” (Delivery staff)

Also, according to delivery staff, the COF matched participants with jobs fitting their specific skills and interests. For instance, delivery staff supported many of the participants without GCSE qualification enter construction, health and social care sectors following an upgrade of their English and maths skills through local colleges.

“Construction is a big one, and health and social care. We’re getting a lot of young people who might have left school without their GCSEs … a lot of colleges will say, ‘Yes, we can take them to get their maths and English’, it is really great.” (Delivery staff)

Most of the interviewed young people who received job-specific training or apprenticeship through COF also found it to be of high quality and relevant in their employment journey. Speaking of the relevance of the training offers received, many participants highlighted sector-specific training in construction (for example, CSCS and forklift training) and the service sector (for example, food safety and preparation, customer service and beautician training). Several indicated the relevance of training received in the creative arts and media sector (for example, teaching assistant in creative arts, music production and media training), as well as in health and social care (for example, Prince’s Trust courses for work in the NHS and animal care).

Some participants also found entrepreneurship-specific training courses signposted to by their career coach, such as the Enterprise programme offered by The Prince’s Trust, to be particularly beneficial in starting or further developing their businesses. Overall, delivery staff were positive about the sustainability of the opportunities provided through COF. In comparison, stakeholders held split views while young people held the least positive outlook on the opportunities they were offered.

The approach the programme took to finding opportunities for participants involved matching them with roles and jobs that suited (as far as was feasible) their skills and interests. Some delivery staff in particular felt that this approach meant that the chances of the participant remaining in that role increased if it matched their interests and they enjoyed the role. The programme also predominantly matched participants in roles that offered full-time positions rather than temporary ones. This included permanent roles within the NHS, the construction industry, customer service roles, fitness, technology (IT) and volunteering roles that offered the opportunity to move into a job.

Roles in health and social care, for example, received growing interest for participants. While the programme did match participants with some roles that were more short-term, focusing on roles that provided lifelong career aspects and interested the participants contributed to the sustainability of the programme’s opportunities.

“A lot of them have gone into the NHS, some have gone into customer service jobs, some are doing volunteering to move into a possible job outcome. So, the majority have jobs that would allow them to develop and move up at some point, so they’re more sustainable jobs.” (Delivery staff)

The relationships the programme has developed with employers and stakeholders by conducting outreach and building partnerships have also contributed to the sustainability of the opportunities offered by the programme. If COF was to run again, it could build from the foundation of relationships in place to continue identifying suitable opportunities for participants.

“It’s got ground now, and the foundation’s been set, so I think provided it’s continued, it can make a positive effect.” (Delivery staff)

However, both participants and stakeholders discussed the challenges affecting the sustainability of opportunities provided. Despite participants expressing an interest in roles that could offer sustainability and eventual career advancement, the actual availability of long-term roles in some of the regions meant the opportunities on offer did not always match what the programme participants were looking for. Building meaningful relationships with employers, such as those who could offer long-term and sustainable opportunities, also took time. Therefore, some of these opportunities may not have been in place when the programme started.

“I think there could be more options. We could have a bit more of a list of different opportunities.” (Delivery staff)

Finally, the participant’s attitude and work readiness were a large part of whether an employment opportunity was sustainable. A minority of stakeholders mentioned participants’ attitudes and engagement towards their work, including whether the participant needed ongoing support throughout their employment journey. Without the programme continuing to provide further mentoring support or checking in with the participants, maintaining participants who may have struggled with their behaviour in employment could prove challenging.

“I think unemployment is down to yourself, if you want a job that bad, there [are] jobs out there.” (COF participant)

4. Benefits to participants

4.1 Introduction

This chapter addresses the contribution of COF in delivering benefits to programme participants, that is, young people, by assessing the extent to which it has:

• improved participants’ life outlook by considering their progress towards 5 life outlook measures – self-esteem, impulsivity, self-efficacy, interpersonal relations and mental wellbeing
• increased participants’ access to employment by considering their progress on 2 job measures – work aspirations and job search self-efficacy
• reduced participants’ crime and offending by considering their progress in relation to 9 types of behaviours

The analysis is based on a pre-/post-survey without a counterfactual, with significant fluctuations in programme participation across the evaluation period (see details in Chapter 2, Section 2.2.4). It is, therefore, challenging to assess whether the impact reported can be confidently attributed to the COF programme. However, qualitative evidence complementing the statistical analysis can demonstrate support for the impact being linked to the programme, among other factors.

It was not possible to assess the impact of programme participation on crime and offending because of its limited success in targeting young people with a history, or at risk of, serious violence. Additionally, a low rate of offending disclosures was observed among survey respondents, which prevented a robust analysis of the impact on offending behaviours to be undertaken. Surveys, instead, reported a higher response rate on questions linked to non-offending. Although the data was subsequently analysed, which showed statistically significant improvements in non-offending among participants when comparing pre- and post-participation scores, it is not a proxy for assessing the impact of the programme on offending. The subsection on offending in this chapter, therefore, relies primarily on qualitative evidence to assess the impact on participants.

Table 6 summarises the evidence on COF outcomes based on 125 survey respondents (21.4% response rate). It reports statistically significant improvements at the 95% CI on:

• life outlook, particularly self-esteem, self-efficacy (including leadership self-efficacy) and mental wellbeing
• access to employment, particularly job search self-efficacy

The full list of significance testing tables linked to this chapter can be seen in the survey analysis subsection of Appendix 1. It is important to highlight that although the analysis relies on tests of statistical significance as a measure of impact, these results need to be interpreted with caution, especially where sample sizes are lower than 30. Further, any limitations emerging from biases due to participant characteristics, self-selection in survey participation, and levels of disclosure on specific survey questions have been discussed in detail in Chapter 2, Section 2.2.3.

Table 6: Overview of outcomes achievement

Logic model impact measure	Interlinked logic model outcome	Evaluation evidence
Life outlook Improved mental wellbeing of participants	Self-esteem	Achieved: Aggregate level: statistically significant improvements at the 95% CI Individual measure level: statistically significant improvements on 3 of 6 individual measures of self-esteem when comparing pre-/post-COF programme
	Mental wellbeing (including self-efficacy)	Achieved: Aggregate level: statistically significant improvements Individual measure level: statistically significant improvements on 5 of 11 individual measures of mental wellbeing (including self-efficacy) when comparing pre-/post-COF programme
	Impulsivity and problem-solving	Inconclusive: Aggregate level: positive but statistically non-significant improvements Individual measure level: positive but statistically non-significant improvements on 2 individual measures of impulsivity and problem-solving when comparing pre-/post-COF programme
	Interpersonal relations	Inconclusive: Aggregate level: positive but statistically non-significant improvements Individual measure level: positive but statistically non-significant improvements on 2 individual measures of impulsivity and problem-solving when comparing pre-/post-COF programme
Access to employment Participants demonstrate progress in the distance travelled towards employment Increased engagement in employment-related opportunities	Work aspirations	Partially achieved: Aggregate level: positive but statistically non-significant improvements Individual measure level: statistically significant improvements on one of 5 individual measures of work aspirations when comparing pre-/post-COF programme
	Job search self-efficacy	Achieved: Aggregate level: statistically significant improvements. Individual measure level: statistically significant improvements on 2 of 6 individual measures of job search self-efficacy (including self-efficacy) when comparing pre-/post-COF programme
Crime reduction Reduced engagement in crime and offending Reduction in risk factors and increase in protective factors for serious violence Reduced involvement in gangs and gang activity	Offending behaviours in the last month	No evidence: there is no evidence to assess this outcome. Some qualitative process evaluation evidence is used to assess potential for impact.
	Risk and protective factors for serious violence	No evidence: there is no evidence to assess this outcome.
	Gang involvement	No evidence: there is no evidence to assess this outcome.

Notes:

1. Wherever referenced in Table 6, statistical significance at the aggregate level refers to t-tests conducted across groups of statements capturing a given measure, for example, self-esteem, reported at the 95% CI.

Each chapter subsection begins with an assessment of progress made towards intermediate outcomes set out in the ToC related to life outlook, access to employment, and crime and offending at an aggregate and then, measure-by-measure level. Following this, each subsection also considers any variations in outcomes achievement between specific participant sub-groups by examining 3 sets of characteristics:

• personal characteristics – age, gender, ethnicity
• employability characteristics – disability, participant’s second language is English
• life circumstances – involvement in police/CJS/probation/legal, involvement with housing support, region

4.2 Impact on life outlook

Life outlook was a composite outcome comprising 3 measures – self-esteem, mental wellbeing, and impulsivity and interpersonal relations. Each of these was assessed through sets of individual statements that were examined separately and at an aggregate level to evaluate the impact of the COF programme.

The survey analysis showed statistically significant increases at the 95% CI (see Appendix 1 for detailed t-test analyses) on the following indicators assessing young people’s life outlook before and after the COF programme:

• 3 of 6 measures of self-esteem
• 5 of 11 measures of mental wellbeing
• none of 4 measures of impulsivity and interpersonal relations

4.2.1 Impact on self-esteem

Survey analysis showed statistically significant increases (denoted by a yellow triangle in the charts wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant) at the aggregate self-esteem level, and on 3 of the 6 individual measures based on the Rosenburg Self-esteem scale (RSE) (Rosenberg, 2006).

Figure 7 shows the percentage who agreed or strongly agreed with the 6 individual statements measuring self-esteem. The average positive shift in outlook reported across these measures was 9.5 percentage points. This reflects participants on the programme experiencing greater self-esteem as measured by the RSE scale in the post-survey compared to the pre-survey.

Figure 7: Agreement with statements (strongly agree or agree) on self-esteem

Source: COF Survey (Q1 to Q6)

Base: 122 to 125 respondents, depending on number of responses to each statement

Notes:

1. Negative worded statement (for example, ‘I certainly feel useless at times’) scores show respondents who disagreed/strongly disagreed. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

This was further reinforced by qualitative data collected through interviews with participants and delivery staff. Participants reported the completed activities served as an important catalyst for increased positive feelings about themselves. For example, they felt better equipped to externalise aspects of their personalities and be more open to taking on new opportunities they would not have come across had they not been on the programme. This contributed to improving their ability to look positively at the future, as well as increasing their attitude towards making a positive shift in their circumstances.

“It’s changes in terms of vocalising things about myself, and working on opportunities to take advantage of them, it’s given me a lot of insight about me.” (COF participant)

It was also reported that specific training and educational opportunities were important in increasing participants’ self-esteem. For example, by increasing their sense of confidence about their abilities and believing that they were well capable of achieving progress towards their life goals, which led to a sense of pride.

“I went for a training course for 4 weeks, and I think it went really well because they helped me with my confidence. At first, I thought I wasn’t going to be able to do it, but I achieved something which I’m proud of.” (COF participant)

A final way in which the COF impacted self-esteem was through helping participants identify the skills they already had but were not previously aware of, and thus not fully utilised. Coming to such a realisation strengthened the desire among participants to strive towards personal goals.

“I have found out that there are a lot of skill sets that I’ve got within me, which I’m not fully utilising, and I just want to get out there, really.” (COF participant)

4.2.2 Impact on mental wellbeing

Results from the participant survey showed statistically significant increases at the aggregate mental wellbeing level, and on 3 of 7 individual measures drawn from the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS) (Stewart-Brown et al., 2009) and 2 of 4 measures drawn from the Intermediate Outcomes Measurement Instrument (IOMI) (Liddle et al., 2019) and Leadership Self-efficacy Scale (LSE) (Bobbio & Manganelli, 2009). Note that mental wellbeing was assessed at the aggregate through 3 different constructs – wellbeing, self-efficacy and leadership self-efficacy – and each of these showed statistically significant increases at the aggregate level at the 95% CI.

Together, these wellbeing measures covered 3 distinct categories: positive outlook, positive decision-making and self-efficacy. On positive outlook, participants reported average positive shifts of 11 percentage points across all 4 measures (Figure 8). This reflects programme participants experiencing a more positive outlook in the post-survey as compared to the pre-survey.

Figure 8: Agreement with statements (strongly agree or agree) on mental wellbeing (positive outlook)

Source: COF Survey (Q15, 16, Q17, Q20)

Base: 123 to 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

On decision-making, participants reported average positive shifts of 5 percentage points across the 3 measures (Figure 9) – indicating greater levels of positive decision-making among programme participants on the post-survey as compared to the pre-survey.

Figure 9: Agreement with statements (all of the time or often) on mental wellbeing (positive decision-making)

Source: COF Survey (Q18, Q19, Q21)

Base: Minimum 123, maximum 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

These positive shifts were further corroborated in participant and delivery staff interviews. Attending sessions and working with career coaches on their mental health provided participants with the impetus to shift the old habits that tied them to inaction and stagnated progress towards their goals. Many reported coaches helped them cope with anxiety and other forms of mental health issues and helped improve their social communication skills through group work, mock interview sessions, or walkarounds in the local area to speak with potential local employers.

This resulted in positive shifts in participants’ mindsets, such as feeling deserving of positive outcomes in life and feeling more easily able to communicate their career aspirations with family members. For example, investing in their wellbeing led participants to regain the motivation to take incremental steps forward by getting simple tasks done.

“[Career coach] has helped me with my mental health. Any problems, just talk through them. It has showed me things to do to help me calm down.” (COF participant)

Focusing on wellbeing also enabled participants to improve how they presented themselves to the outside world, which was a prerequisite to taking the next steps towards finding employment.

“The past couple of months, if you saw me, I looked drained and I did not even go for a haircut. But if you saw me now, I’m dressing well, grooming myself. It’s a big step that I’ve taken.” (COF participant)

Devoting efforts to their mental health also reinforced their sense of belief in their own self.

“Yes, even though it is only a tiny bit, I believe in myself a little bit more.” (COF participant)

On self-efficacy, participants reported average positive shifts of 8 percentage points across all 4 measures (Figure 10), reflecting an increase in programme participants’ feelings of self-efficacy on the post-survey as compared to the pre-survey.

Figure 10: Agreement with statements (agree or strongly agree) on mental wellbeing (self-efficacy)

Source: COF Survey (Q9, Q10, Q13, Q14)

Base: 122 to 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

The above shifts in self-efficacy were also reported in the qualitative evidence gathered. Participants suggested that the support received through the COF helped them be more self-reliant by strengthening their self-belief in their competences and ability to accomplish tasks and produce favourable outcomes; for example, by engaging in a process of self-appraisal when encountering novel situations and being able to make decisions without relying on external sources of help.

“I have learned quite a lot about myself with this programme, I’ve learned how to assess yourself in some ways.” (COF participant)

“I feel like I have matured a lot more than my age. I feel like I’ve become more independent in myself. I am doing stuff on my own without asking for help.” (COF participant)

“I have got more confidence than what I used to have, so I’m able to push myself so I’m not afraid so I know what I want.” (COF participant)

The COF programme enabled participants to make decisions and definitive steps in the journey towards employment, something that they reportedly held back from doing in the past.

“I did not want to go for a job in case something went wrong, and now I have pushed myself to get a job that I will enjoy.” (COF participant)

“I know how to go and attempt other challenges; I can fulfil them and do what I need to do rather than shying away as such.” (COF participant)

Although the survey measures assessed shifts in participants’ leadership self-efficacy, there was little to no qualitative evidence on how such shifts manifested in participants’ behaviour or actions before and after attending the COF programme.

4.2.3 Impact on impulsivity and interpersonal relations

Young people demonstrated positive but statistically non-significant shifts at the aggregate level, and on 4 individual measures of impulsivity and problem-solving and interpersonal trust drawn from the IOMI. However, some positive improvements were noted, as suggested by the qualitative evidence collected through the interviews.

For example, when asking about how participating in COF activities impacted their outlook towards life, participants reported being more thoughtful and deliberate in decisions and actions, as opposed to the often-unconsidered approach that they previously took.

“I feel like I’m more aware of surroundings. I just don’t jump to conclusions, and I take everything into consideration before making a decision, no matter how small or big it is.” (COF participant)

In interviews, participants also reported shifts in interpersonal trust in their relationships with others in general, and career coaches specifically. Participants reported that the time spent with career coaches and other facilitators (such as, through the services participants were signposted to) served as a formative experience that enabled them to build trust with significant others and model future relationships on. This directly impacted their sense of self and any actions taken towards future goals. For example, by reinforcing their belief that help was available when requested and reassuring them when they felt worried, which encouraged them to continue pursuing the goals they had set out to achieve.

“I feel like this has contributed to my whole idea of, ‘If you have the will do something, people around you will help you as much as they can for you to achieve this goal of yours.’” (COF participant)

“[The relationship] keeps you positive because you have got somebody who is very encouraging, and saying, ‘do not worry, just keep applying and the work will pay off.’” (COF participant)

Positive interpersonal relationships also reinforced participant’s belief in their own abilities because of the trust placed in them by career coaches. This pushed them to be confident in their abilities and seek help and advice when needed.

“She gives me that boost because I do not have any confidence in myself, but she has always been the one to give it to me, especially when we first met.” (COF participant)

“I know that [coach] wants the best for me. So, I am open to taking advice from him. And I put my trust in him because I know that he is trying to do something good for me.” (COF participant)

4.2.4 Variation in life outlook outcomes by participant groups

To consolidate the individual sets of measures and report trends at the aggregate level of interest (life outlook, access to employment, and crime and offending), RSM UK built an index for each theme and conducted t-tests to estimate statistically significant differences in outcomes at a sub-group level (see Appendix 1 for a fuller explanation).

This analysis examined the number of survey respondents who agreed/ strongly agreed with each individual measure. The index reports the consolidated responses to all statements relative to a given measure (for example, for life outlook, this would include self-esteem, mental wellbeing, and impulsivity and interpersonal trust) and reports the difference pre-/post-survey. A positive score reflects a greater number of participants reporting a positive shift on the post-survey as compared to the pre-survey. Higher scores represent a greater number of participants reporting the positive shift as compared to lower scores.

The index-based analysis for life outlook showed variation among different participant groups such that overall, the largest statistically significant improvements at the 95% CI across all participant groups were reported by:

• black/black British participants (+0.17 index points)
• male participants (+0.13 index points)
• 16 to 19 year old participants (+0.13 index points)

This analysis was completed by creating a ‘life outlook’ index that combined the results of individual scales and measures assessing life outlook into one indicator. Further details on this approach are in Appendix 1.

Figure 11 provides a snapshot of life outlook outcomes at the aggregate level for different participant groups. Although several results are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30. See Appendix 1 for details.

Figure 11: Variation in life outlook outcomes by participant groups: Aggregate differences between pre- and post-surveys (statistically significant results are denoted by an *)

Source: COF Survey (Q1, 2, 3, 9, 13, 15, 19, 20)

Base: 117 respondents, depending on the number of responses to each statement. For sample sizes of sub-groups, please see Appendix 1, Table A13

Notes:

1. The index scores are presented at 2 decimal places for stylistic purposes. Some bars with the same values may appear different in size due to the underlying difference in decimal points.
2. Although several results reported in the sub-group analysis are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30 (indicated by an ^). See Appendix 1 for details.

4.3 Impact on access to employment

Access to employment was a composite outcome comprising 2 measures – aspirations for work and job search self-efficacy. RSM assessed each of these through sets of individual statements that were examined separately and at an aggregate level to evaluate the impact of the COF programme.

Survey analysis showed statistically significant increases at the 95% CI (see Appendix 1 for detailed t-test analyses) on the following indicators assessing young people’s access to employment before and after the COF programme:

• one of 5 measures of aspirations for work
• 2 of 6 measures of job search self-efficacy

4.3.1 Impact on aspirations for work

Participants reported statistically non-significant improvements at the aggregate level, but statistically significant improvements on one of the 5 measures of aspirations for work as drawn from the Longitudinal Study of Young People in England (LSYPE) (DfE, 2011).

Figure 12 shows that the average positive shift in overall outlook among participants across the 5 measures was 1.4 percentage points, reflecting more positive work aspirations among programme participants on the post-survey as compared to the pre-survey.

Figure 12: Percentage of respondents who rated measures of aspirations for work to be important/very important

Source: COF Survey (Q22 to Q26)

Base: 123 to 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

The qualitative data collected from participant and delivery staff interviews provided strong evidence in support of the above shifts in work aspirations. When talking about how their engagement with the COF has impacted their work-related goals, it was apparent that the programme increased participants’ job aspirations to higher levels than previously held. For example, by enabling participants to develop a clearer understanding of the value of holding a job and its place in their wider life, and the professional standards that one needs to abide by when working.

“[I understood] how work life works, for example, thinking rationally and in a responsible way, balancing work and home life. I have realised what the real world of work is.” (COF participant)

It also prompted participants to think about the longevity and sustainability of employment, something they may not have necessarily focused on in the past.

“I feel more serious about what I’m doing, I have no time to procrastinate like I used to. This job is something that could be sustainable in the future, if I just put in the work and take myself seriously.” (COF participant)

Participating in COF activities also helped participants obtain skills training that improved their ability to earn a higher income as compared to before. This increased their confidence about getting a better job in the future and helped focus their efforts towards finding highly skilled jobs.

“I am not going to work for £9.50 an hour. The amount of training that I have now had, I could be on a lot more money so I am not going to lower myself and work for less when I know I could be earning more.” (COF participant)

Finally, for some participants, feeling enabled in their own professional journey because of the COF programme increased their ambitions and drive to help others in the future.

“Now I want to provide opportunities for people who feel like there are no opportunities for them. Similar to what I’m on right now.” (COF participant)

4.3.2 Impact on job search self-efficacy

Participants reported statistically significant improvements at the aggregate level, and on 2 of 6 measures of job search self-efficacy drawn from the Job Search Self-efficacy scale (JSSE) (Saks et al., 2015).

Figure 13 shows that the average positive shift in outlook relating to job search self-efficacy across all 6 statements was 4 percentage points. This reflects participants on the programme experiencing greater self-efficacy in relation to finding jobs on the post- as compared to the pre-survey.

Figure 13: Percentage of respondents reporting confident/highly Confident on job search self-efficacy

Source: COF Survey (Q27 to Q32)

Base: 123 to 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

The qualitative evidence gathered reflected that completing job-related activities with their career coaches or through being signposted to external agencies provided participants with the impetus to take deliberate steps towards finding employment. Participants reported completing apprenticeships (for example, bricklaying, beauty therapy or hospitality work) and attending skills workshops (for example, on money management, IT, positive body language or confidence). They also reported looking for jobs themselves instead of relying on career coaches.

“I am now looking for jobs myself – the programme gave me a kick to look myself.” (COF participant)

Attending the programme enabled participants to participate in training and qualification schemes that they had been hoping to attend but were unable to before the programme. This gave them a sense of recognition about their skills and the confidence to find related employment quickly.

“A lot of young people were interested in construction cards like the CSCS [Construction Skills Certification Scheme] course. The coach would sit down with the material and get them ready for the test by doing mocks with them. This was well received because it is actually quite hard to pass that test.” (Delivery staff)

“I got the CSCS [Construction Skills Certification Scheme] card, I have achieved something.” (COF participant)

Finally, the COF programme equipped participants with communication and networking skills that increased their confidence to put themselves forward for future opportunities.

“I feel able to ask for the help that I need, and I am better at networking as well. Just because I have had to speak to and meet so many more people.” (COF participant)

Most participants also indicated that the opportunities offered to them through COF matched their skills and interests. A key success factor was the effort made by career coaches to identify and signpost them to relevant opportunities. In-depth discussions with their career coach allowed participants to gain a better understanding of the overall employment process. Several participants indicated that their career coach helped them make a clear step-by-step plan to overcome their barriers (such as having to upgrade their current skillset or attaining new skills through relevant training) to gaining a suitable employment.

“[Career coach] helps with young people to get into work experience, and then they have a job interview at the end, and it’s a guaranteed job. So, I can definitely see this helping me.” (COF participant)

However, delivery staff highlighted several factors that prevented a higher number of participants from moving towards employment and skills:

• participant engagement: this included issues such as difficulty obtaining confirmatory evidence when participants achieved education or employment outcomes, low levels of engagement by the hardest-to-reach participants relative to the quick turnaround time in relation to expected outcomes, and the significant shifts often required in participants’ lifestyles for them to engage with the programme, which varied from participant to participant

“Sometimes I’m not getting evidence until a month after, or even 2 months after in some cases.” (Delivery staff)

• complex needs: it was challenging to tailor opportunities for participants with complex personal circumstances, such as those who could not travel to certain areas of the city or only work certain hours due to family responsibilities. Staff also found it difficult to manage participant expectations on entering highly competitive sectors or jobs that required highly specialised skills

“We had young people that couldn’t travel to certain areas of the city, or we had young people that couldn’t work certain hours, so that side of it was difficult.” (Delivery staff)

• staff related factors: as highlighted in Section 3.4.1, these related to: high levels of staff turnover and burnout due in part to excessive caseloads; staff unable to provide consistent levels of support to participants due to Consortium targets that career coaches perceived as overambitious; and their inability to build enough highly engaged connections with external stakeholders that worked with hard-to-reach young people

“It was challenging, just the turnaround and trying to build up those pathways and get the turnaround in numbers and to get them into jobs. That’s a long process. Generally, you work with somebody who’s the hardest to reach in the community, and then in 4 weeks’ time you needed to get 4 job outcomes. It’s like, ‘Well, actually, I will get this person a job outcome, but it might be in 5 months. It’s not going to be in 3 weeks.’ It is completely participant dependent.” (Delivery staff)

• processual factors: as highlighted in Sections 3.4.2 and 3.4.3, delivery staff considered the multi-step and long onboarding process as administration-heavy, and reporting to some forms of wraparound support (for example, bursaries) as often burdensome and confusing
• employer-related factors: regional coordinators struggled to develop closer links with larger employers as they deemed this a time-consuming process with no guarantee of success. This meant that without pre-established relationships with employers, regional coordinators spent time speaking with larger employers only to find out they were not the right fit for the programme

“So, it’s more time-consuming but in the long run, why waste time with a partner that’s not going to be beneficial for what we’re doing?” (Delivery staff)

4.3.3 Variation in access to employment outcomes by participant groups

The employment outcomes index score examined statistically significant differences in outcomes relating to access to employment among participant sub-groups (see Appendix 1 for a fuller explanation of our approach). Higher scores represented a greater number of participants reporting the positive shift as compared to lower scores. Overall, the largest statistically significant improvements at the 95% CI across all groups were reported by male participants (+0.11 index points).

Figure 14 provides a snapshot of employment outcomes at the aggregate level for different participant groups. Although several results reported are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30. See Appendix 1 for details.

Figure 14: Variation in access to employment outcomes by participant groups: Aggregate differences between the pre- and post-surveys (statistically significant results are denoted by an *)

Source: COF Survey (Q26 to 28)

Base: 119 to 123 respondents, depending on the number of responses to each statement. For sample sizes of sub-groups, please see Appendix 1, Table A14

Notes:

1. The index scores are presented at 2 decimal places for stylistic purposes. Some bars with the same values may appear different in size due to the underlying difference in decimal points.
2. Although several results reported in the sub-group analysis are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30 (indicated by an ^). See Appendix 1 for details.

4.4 Impact on crime and offending behaviours

‘Crime and offending’ were a composite outcome comprising 9 individual statements capturing violent and criminal behaviour, drugs consumption and antisocial behaviour. The analysis proposed to assess these at an individual and aggregate level to evaluate the impact of the COF programme. However, 2 limitations prevented such analysis from being undertaken. First, the programme’s limited success in targeting young people with a history of, or at risk of, serious violence (see Figure 2, Section 3.4.1) meant the programme cohort was not appropriate for the intended analysis because of the low numbers of participants with an offending history. Second, very low rates of offending disclosures were observed among survey respondents that did have a background of offending, which prevented a robust analysis of impact on offending behaviours to be undertaken.

The following analysis therefore focuses on qualitative evidence gathered as part of the process evaluation to provide a limited assessment of the impact on crime and offending behaviours among programme participants. An additional indicative piece of analysis of non-offending behaviours among programme participants is also presented. This is not a proxy for assessing the impact of the COF programme on offending.

4.4.1 Impact on violent and criminal behaviour

Evidence collected through interviews with delivery staff and stakeholders shows that COF made a positive impact on participants’ risk factors in relation to offending behaviours. Regional coordinators stressed that despite low referrals of at-risk young people, the majority benefitted from the support offered by the programme, due to the correlations between mental health and lack of access to sustainable employment that can lead to criminal behaviour (for example, see Frisby-Osman & Wood, 2020 and Van der Geest, 2011), even if everyone on the programme may not have a criminal record.

“So, there are multi-factor issues that are going on which contribute to a young person being involved in serious youth violence. COF tackled most of these.” (Strategic stakeholders)

It was also felt that COF made a positive impact in addressing underlying factors such as lack of direction and awareness of opportunities, as well as low confidence and self-esteem. The personal support provided to participants as open discussions about their past criminal activities, showing them positive pathways and helping them realise their capabilities were perceived as key enablers.

4.4.2 Impact on drugs consumption and antisocial behaviour

There was limited evidence collected through interviews on drugs consumption and antisocial behaviour. A few career coaches indicated it was difficult to identify the impact of COF activities on participants who have previously exhibited criminal behaviour because there was no mechanism for monitoring their activities. Due to their complex needs, any visible impact could take some time. However, anecdotal evidence provided by some career coaches and regional coordinators indicated that the programme was able to reduce antisocial behaviour by giving young people the opportunity to access support and employment.

“We’ve had direct feedback. You can tell, the young people that are now in these jobs, they’ve stopped doing certain behaviours because they are now working 9-5, so you know they wouldn’t be employed if they were still, you know, using drugs or whatever it is.” (Delivery staff)

Some stakeholders supported this finding, where they felt the programme was useful to participants known to the CJS and that the programme was a mechanism to support them from re-offending. Re-offending can be linked to a lack of access to employment (and money), therefore difficulty in accessing employment can exacerbate the likelihood of re-offending. Stakeholders viewed the direct one-to-one support offered by the programme in finding participants suitable and sustainable employment reduced the likelihood of re-offending.

4.4.3 Impact of the COF on non-offending among participants

As mentioned previously, it was not possible to assess the impact of COF programme participation on reductions in crime and offending due to the mostly ineligible programme cohort (with no previous offending history) and low rates of offending disclosures among survey respondents. However, it is possible to use these questions to assess ‘non-offending’. This analysis compared those that reported having ‘never offended in the last month’ between the pre- and post-surveys to understand shifts in patterns of non-offending before and after the COF programme. Note that this analysis of non-offending is not a proxy for assessing the impact of the programme on offending behaviours, as it could have been driven by factors outside of the COF programme, such as participants’ increased trust in career coaches and willingness to disclose this information with greater ease in the post- rather than pre-survey.

The analysis is presented across Figure 15 and Figure 16, which show statistically significant improvements in non-offending at the aggregate level, and on 4 of 9 measures of crime and offence drawn from the Edinburgh Study of Youth Transitions and Crime (EYSTC) (Smith et al., 2001) when comparing pre- and post-participation scores.

Figure 15: Percentage of respondents who reported NO violent, criminal and theft behaviour in the last month

Source: COF Survey (Q33, Q37)

Base: 123 to 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

Figure 16: Percentage of respondents who reported to NOT have offended in relation to drug consumption and antisocial behaviour in the last month

Source: COF Survey (Q34, Q36, Q40)

Base: 123 to 125 respondents, depending on the number of responses to each statement

Notes:

1. A yellow triangle is used wherever the pre- and post-survey mean score differences for a given statement were found to be statistically significant.

5. Conclusions and recommendations

This chapter presents conclusions on the overall contribution of the COF programme as an intervention focused on deterring violence and crime among those most at risk of violence through enhancing soft and employability skills. The chapter also provides recommendations to the Home Office and national providers for running similar programmes in the future.

5.1 COF contribution

The COF approach of providing tailored one-to-one wraparound support to those at risk of serious violence was described as a unique service that did not duplicate any existing provision in the 12 local authority (LA) areas. At an aggregate level, the survey indicated improvements by the COF participants on 2 anticipated impact areas identified in the ToC – life outlook (particularly on measures self-esteem and mental wellbeing including self-efficacy), and access to employment (particularly on measures of job search self-efficacy).

Besides the ToC impact measures achieved above, the unique contributions made by the COF programme in improving participant outcomes included:

• its responsiveness to local needs linked to young people’s employment
• the diversity of activities conducted, specifically, to improve life outlook and enable job-related self-efficacy through one-to-one sessions with career coaches
• the tailored facilitation provided by career coaches, particularly their mentorship, and flexible and continued engagement with participants
• personalised forms of support, particularly the high level of matching of skills development with participant needs, age and interests, including any unique personal circumstances
• partnership working with external organisations, including establishing a presence locally through outreach and building on existing relationships while forging new relationships, some with organisations linked to crime and offending

Although there were several process-related challenges faced by the programme, as outlined elsewhere in this report, the biggest limitation of the COF programme was its lack of targeting to the key participant group of interest – young people with a history of, or at risk of, serious violence. Another key shortcoming was a lack of coordination between the service providers, which impacted not only programme activities but also the nature and timing of data collected by the evaluation itself. Both severely limited the programme’s impact on the demographic of most interest to the Home Office and for which it provided the funding.

5.2 Recommendations

The following recommendations emerge from the process and impact evaluation conducted.

5.2.1 Recommendations for national providers

On roles and responsibilities:

• although there were 2 roles related to programme delivery (regional coordinator and career coach), further streamlining roles and responsibilities within provider teams to minimise the loss of quality support offered to participants (for example, through additional dedicated roles relating to employer/stakeholder engagement and relationship management, programme administration, young person recruitment and onboarding, and conducting outreach activities)
• improving collaboration between the 2 delivery partners at a regional level to ensure effective joint programme delivery. For example, through team building, articulation of shared objectives, better use of the programme steering group to feed back into programme design or delivery, and regular communication to share challenges and good practices
• clearer guidance, planning and management in relation to delivery team size, individual workloads, and distribution across geographical regions/LAs to avoid staff turnover or burnout, and any knock-on effects on programme delivery

On programme activities:

• more group activities, besides one-to-one support, as a core offering to young people attending the COF programme, such as CV and confidence-building sessions, and forming participant groups for peer-to-peer learning and engagement
• further engagement with the Apprentice Nation platform through greater clarification and support in relation to job seeking, and effective navigation
• further expansion of skills/interest areas covered by the programme, for example, in relation to the music and arts, to better meet participant needs
• further streamlining of the matching process between career coaches and participants to align with young people’s job interests and preferred engagement patterns

On relationships with external stakeholders:

• clearer communication with external stakeholders referring participants onto the programme on key benefits and types of support available, and the time and processes for making referrals onto the programme
• more resources dedicated towards building connections and relationships with external organisations working with hard-to-reach young people at risk of violence (such as prison services), and stronger and regular two-way communication channels that keep all stakeholders updated on the progress being made
• more direct relationships with employers (as opposed to going through an employment agency or online jobs platform)
• more joined-up working with organisations supporting at-risk participants with ‘adjunct’ needs, such as food and fuel banks and accommodation providers, to give due recognition to the complex circumstances facing those at risk that need to be addressed simultaneous to any employability efforts made

On delivery processes:

• increasing efficiencies when onboarding programme participants through additional ways of verifying identity, such as using national insurance numbers where passports are not available
• standardising processes between different delivery partners to clarify what, when and how wraparound support can be provided (especially bursaries and laptops), strengthen provider relations to maximise programme impact, and avoid any confusion between programme priorities and approaches for reaching them
• improving target participant identification through building better connections with external organisations, potentially including them as part of a consortium that meets regularly to discuss delivery processes and how these might be improved
• clearer focus on targeting the ‘at-risk’ group from the beginning to provide justice to the funding aims and objectives

5.2.2 Recommendations for the Home Office

Recommendations for the Home Office include:

• defining the target group clearly within the provider contract to ensure that future funding targets the intended beneficiaries fully instead of in part
• facilitating communications between national providers and, ultimately, consider delaying the full programme rollout until all national providers involved have signed a DSA
• including joint working structures between all national providers in the service contract, such as mutual data sharing arrangements, to enable a joint case management process for a given participant through, for example, referrals and signposting using a coordinated list of resources shared between all providers
• considering including key external organisations working with young people at risk of serious violence into the single service contract
• clarifying expectations around target numbers from the onset and setting variable delivery targets based on variations in the calendar year
• accounting for more realistic timeframes into the programme delivery milestones for building a presence of the programme within the local community, including relationships with local and national stakeholders

Appendix 1: Methodology note

Strengths

• expert inputs: The evaluation team worked collaboratively with experts – Dr Gilly Sharpe and Professor Darrick Jolliffe – to validate the evaluation approaches and research tools
• breadth and depth of qualitative data: The evaluation included programme participants, project staff and external stakeholders, which contributed to a well-rounded analysis of the programme’s activities and is a key strength of this evaluation
• triangulation of findings: A ToC model underpinned all aspects of the data collection, analysis and reporting (for example, the structure of research tools, including depth interview topic guides) to allow for effective triangulation and quantification of evidence. A variety of data collection methods were used to triangulate evidence against each evaluation question, including quantifiable and objective output and outcome measures where feasible
• representative sample for young people interviews: The interview sample represented the target population (young people), and this minimised selection bias. The team selected the interview participants on their background characteristics (local authority (LA), age, gender, ethnicity, Red-Amber-Green (RAG) rating) and not based on the high level of interactions with the programme activities
• strong communication: Communication between delivery staff and the evaluation team allowed for a transparent and honest relationship which further strengthens the credibility of the evaluation itself
• statistical significance of survey findings: Despite a 21.4% response rate (see Section 2.2.4 Limitations), survey findings were extracted from data and statistically tested for significance using paired t-test at aggregate level. The sample (125 cases) provided reliable estimates of statistical significance on differences of group means on 15 questions out of 44
• social desirability bias: As delivery staff distributed the survey there is a risk of bias (the respondents might wish to please the delivery staff and try to provide answers to the survey they thought were ‘right’ in the eyes of this key reference person/ delivery staff). This has been mitigated by designing the questions using existing and relevant validated surveys, as well as allowing participants to self-complete, while having staff there to support them if issues of literacy arise
• independence of research teams: Different members of the evaluation team conducted different fieldwork activities, analysis and reporting, ensuring that a limited evaluator bias was introduced to this study

Feasibility study

The evaluation team completed a feasibility study, led by Professor Darrick Jolliffe, Royal Holloway (University of London), to examine and assess the options for conducting a quantitative impact analysis. Findings presented in the evaluation plan concluded it was not possible to identify a suitable comparison group to examine the impact of the COF programme.

The evaluation method that would provide the strongest evidence of the impact of the COF programme would be an RCT. This method would involve randomly allocating young people to either the COF programme, to business as usual, or what the individuals would receive if COF did not exist. By design, this method would equate individuals on all measured and unmeasured characteristics and maximise the likelihood that any differences observed in later outcomes would result from COF and not pre-existing differences between the groups. To conduct an RCT, significant resources would need to be invested before the commencement of the intervention to establish the appropriate protocols (for example, point of randomisation, what would happen to those allocated to not receive support, ethical clearance). Unfortunately, this was not available in this instance.

The next highest level of evaluation would involve some form of quasi-experiment where statistical methods are used to create 2 groups (one who received COF, and another who did not) equivalent on important measured characteristics such as age, ethnicity, job readiness, qualifications. The evaluation team tried to identify a suitable comparison group, but one could not be found. There were a few reasons why they did not consider the Justice Data Lab (JDL) an appropriate measure of impact in this instance:

• despite the primary aim of COF being to reduce engagement in crime and offending risk factors, it was not the sole aim, so JDL would only capture some of the relevant outcomes
• JDL’s methodology is not perfect – the statistical approach violates the conditional independence assumption required to conduce propensity score matching
• not all those receiving COF would be ‘offenders’ (so not possible to match)
• JDL would only allow for an assessment of the impact of referral to the COF programme on later re-offending (because it would not be possible to determine the re-offending of those who did or did not engage with the COF programme)

The team considered other data matching, such as MoJ-DfE and MoJ-DWP datasets, but rejected these as the complexity and time sensitivity (that is, waiting list for access) of obtaining access to this data and the unanswered questions about the extent to which the key variables needed for matching, particularly for assessing the key outcomes (for example, obtaining jobs skills, increased awareness of training opportunities).

Furthermore, both these linked datasets and the JDL have a long (circa 2 years) time lag on employment and offending data, which is after the evaluation period ends. As a result, they could not be used as a data source for outcomes during the timeline of the evaluation. Additionally, the target cohort of young people could not be identified in the employment databases at that time, as there were no flags on the system to enable this.

RSM and Professor Jolliffe also explored undertaking a Waiting List RCT. This would involve:

• conducting an initial assessment with all young people referred to COF
• random allocation to either: (1) immediate support, or (2) going on a waiting list for 6 to 8 weeks
• after 6 to 8 weeks, Group 1 (immediate support) would have their outcomes assessed, and group 2 (waiting list group) would have their assessment updated and commence COF support

This approach would allow for a very strong assessment of the impact of COF after 6 to 8 weeks. However, there were some challenges to consider:

• the approach would not provide insight into longer-term outcomes because eventually all young people will have received some support
• previous studies suggest that young people allocated to the waiting list group were more likely to drop out before the programme begins
• logistical challenges for Catch22 to implement this approach as it added a critical step to their already existing delivery model

They concluded it was not possible to identify a suitable comparison group to examine the impact of the COF programme. This was because it would have been a considerable practical challenge for Catch22, and the impact of COF could only be observed for the waiting list period. For example, if individuals were randomly allocated to the waiting list for 6 weeks, the impact of COF could only be assessed at 6 weeks when the waiting list began receiving COF.

As a result, the approach the team considered the most feasible was a dose-response study. In the absence of a suitable comparison or control group, the most feasible approach to examine the impact of COF was developing a series of evaluation questions to examine a dose-response relationship between COF and outcomes for individuals with different characteristics. They recognised that the dose-response approach would not address issues such as selection effects, but this approach could have allowed for the team to assess the potential impact that these effects might have on the results (for example, by comparing the impact for those with various levels of engagement).

The monitoring information (MI) the evaluation team had access to was incomplete, so the evaluation could not explore how varied attendance might have impacted outcome realisation at the participant-level using dose-effect analysis, as planned in the feasibility study. As a result, the analysis conducted was a pre/post on outcomes of interest, coupled with breakdowns on demographics.

Key evaluation questions

Process

Evidence was collected to answer the following process evaluation questions:

• to what extent and in what ways are the local authority areas implementing the interventions to reflect the local area and cohort profile?
• what intervention resources (mentoring, counselling, life skills, employment skills) are found most engaging and useful by the young people participating in the interventions?
• how are young people being engaged in the interventions?
• are young people involved in any other interventions from other programmes? Has this impacted on their progress towards the outcomes?
• what are the motivators and barriers to the employment-related opportunities being provided?
• how is the national provider working with local partners to define, identify and form an understanding of the ‘at-risk cohort’ for involvement in the interventions?
• what employment-related opportunities are being provided by local partners?
• how sustainable are the employment-related opportunities?
• what benefits and added value will be/ has been derived from the interventions and employment-related opportunities, both for individuals involved in the interventions and for the local partner businesses?
• how many individuals are being reached in each area? How are they being identified, and do they meet the selection criteria?
• what are young people’s experiences of interventions and employment-related opportunities?
• how was the data for the impact evaluation collected and used to assess impact, and what challenges were associated with this?

Impact

The goal of the impact evaluation was to assess the overall impact of the COF programme, the factors contributing to the observed impacts, and the extent to which different groups of participants were impacted in different ways. These areas of analysis were used to develop a list of 10 specific questions that examined the impact of 3 programme focus areas (that is, employment-related activities, skills development, and engagement) on 3 specific measures of interest – life outlook, access to employment and crime reduction.

The individual questions were also framed to allow observations of impact on different participant groups categorised by their personal characteristics, employability characteristics and individual life circumstances. The matrix table below lists these questions. While it was possible for the evaluation to answer questions 1a to 10a, we could not analyse 1b to 10b and 1c to 6c due to data limitations, as all analysis was dependent on obtaining sufficient data.

—	Activities	Skills	Engagement
Life outlook Improvements in physical and mental wellbeing Source: Not possible to assess physical wellbeing Life outlook assessed through survey questions: - self-esteem (Q1-6) - impulsivity/problem-solving (Q7-9) - leadership self-efficacy (Q10-14) - mental wellbeing (Q15-21)	1a. Do those who receive (up to 3 types of) support report a demonstratively better life outlook? Source: Aptem support types: signposting (internal and external), wraparound services, Apprentice Nation, bursary, and credential building support - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 1b. What aspect of the support received (for example, relevance or duration) contributes to a better life outlook (that is, only looking at those demonstrating a better life outlook)? Source: First young person (YP) interview Section 3, second YP interview Section / combined interview Section 4 1c. What does an improved life outlook for those who receive more types of support look like? Source: Second or combined YP interview Section 4	2a. Is there evidence that those receiving (soft/employability skills) support report a significantly better life outlook? Source: Aptem subset of support related to soft/ employability skills development - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 2b. What types of training/ skills development support contribute to the development of a better life outlook (that is, only looking at those demonstrating a better life outlook)? Source: Second YP interview Section 3/ combined YP interview Section 4 2c. What does an improved life outlook for those receiving soft/employability skills support look like? Source: Second or combined YP interview Section 4	3a. Is there evidence that those self-reporting high levels of positive engagement with their career coach report a demonstratively better life outlook? Source: survey questions on career coach engagement (that is, comparing survey Qs with survey Qs) - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 3b. What types of positive engagement with career coaches contribute to the development of a better life outlook (that is, only looking at those demonstrating a better life outlook)? Source: First YP interview Section 5, second YP interview Section 2, combined YP interview Section 5 3c. What does an improved life outlook for those with high levels of positive engagement with their career coach look like? Source: Second or combined YP interview Section 4
Access to employment - progress in the distance travelled towards employment Source: Survey questions on work aspirations (Q22-26) - engagement in employment-related opportunities Source: survey Qs on job search self-efficacy (Q27-32)	4a. Do those who receive (up to 3 types of) support do demonstratively better on access to employment? Source: Aptem support types: signposting (internal and external), wraparound services, Apprentice Nation, bursary, and credential building support - variation based on activity type, relevance, and duration for which received - variation by personal characteristics - variation by employability characteristics 4b. What aspect of the support received (for example, relevance, or duration) contributes to improved access to employment (that is, only looking at those demonstrating improved access to employment)? Source: First YP interview Section 3, second YP interview Section 1/ combined interview Section 4 4c. What does improved access to employment for those who receive more types of unique support look like? Source: First YP interview Section 4, second YP interview Section 3, combined YP interview Section 4	5a. Is there evidence that those receiving (soft/employability skills) support do demonstratively better on access to employment? Source: Aptem subset of support related to soft/ employability skills development - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 5b. What types of training/ skills development support contribute to improved access to employment (that is, only looking at those demonstrating improved access to employment)? Source: Second YP interview Section 3/combined YP interview Section 4 5c. What does improved access to employment for those receiving soft/employability skills support look like? Source: First YP interview Section 4, second YP interview Section 3, combined YP interview Section 4	6a. Is there evidence that those self-reporting high levels of positive engagement with their career coach do demonstratively better in relation to access to employment? Source: survey questions on career coach engagement (that is, comparing survey Qs with survey Qs) - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 6b. What types of positive engagement with career coaches contribute to improved access to employment (that is, only looking at those demonstrating improved access to employment)? Source: First YP interview Section 5, second YP interview Section 2, combined YP interview Section 5. 6c. What does improved access to employment for those achieving high levels of positive engagement with their career coach look like? Source: First YP interview Section 4, second YP interview Section 3, combined YP interview Section 4
Crime reduction - engagement in crime and offending Source: assessed through survey Qs on whether offended in the last month (Q33-41) - reduction in risk and increase in protective factors for serious violence Source: Second or combined YP interview Section 4 - reduced involvement in gangs and gang activity Not possible to assess	7a. Do those who receive (up to 3 types of) support report demonstratively lower criminal intent/activity? Source: Aptem support types: signposting (internal and external), wraparound services, Apprentice Nation, bursary, and credential building support - variation based on activity type, relevance, and duration for which received - variation by personal characteristics - variation by employability characteristics 7b. What aspect of the support received (for example, relevance, or duration) contributes to lower criminal intent/activity (that is, only looking at those demonstrating improved access to employment)? Source: First YP interview Section 3, second YP interview Section 1/ combined interview Section 4	8a. Is there evidence that those receiving (soft/employability skills) support report demonstratively lower criminal intent/activity? Source: Aptem subset of support related to soft/ employability skills development - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 8b. What types of training/ skills development contribute to lower criminal intent/activity (that is, only looking at those demonstrating a lower criminal intent/activity)? Source: Second YP interview Section 3/ combined YP interview Section 4	9a. Is there evidence that those self-reporting high levels of positive engagement with their career coach report demonstratively lower criminal intent/activity? Source: survey questions on career coach engagement (that is, comparing survey Qs with survey Qs) - variation by personal characteristics - variation by employability characteristics - variation by life circumstances 9b. What types of positive engagement with career coaches contribute to lower criminal intent/activity (that is, only looking at those demonstrating lower criminal intent/activity)? Source: First YP interview Section 5, second YP interview Section 2, combined YP interview Section 5

10a. What individual characteristics typify those who receive significant levels of support, but do not appear to benefit from them? Source: Aptem data on disengaged and off-programme - what characteristics typify those who do not benefit from improvements in life outlook? - what characteristics typify those who do not benefit from access to employment? - what characteristics typify those who do not benefit from reductions in criminal activity/behaviour? 10b. What about the support or career coach engagement offered prevents individuals from benefitting? Source: First, second and combined YP interview questions on what did not work well for them 10c. What could be improved to enable individuals that do not appear to benefit from the programme to be better supported? Source: First, second and combined YP interview questions what could be improved about the programme

Notes:

1. Ideally, the evaluation would need to find the level of overlap between those completing surveys and interviews – how big is the group that has done both? This will be useful in answering the ‘b’ and ‘c’ type questions listed under questions 1 to 10 above.
2. Q1 ‘types of unique support’: the Aptem dataset covers 12 unique types of signposting (including internal signposting and Apprentice Nation), 10 forms of wraparound support (including bursary) and credential building courses. This covers 6 different types of support as follows: internal signposting, external signposting, Apprentice Nation platform, wraparound support, bursary and credential building.
3. For Q1b, duration of support received can also be calculated by total length of stay in the COF programme, using the Aptem start date and date of completion of the last survey.
4. Q2 ‘employability/ soft-skills development’: the Aptem dataset covers specific support types (signposting or wraparound support) that can be linked to employability and soft skills.

Survey design

The survey contained 44 Likert scale questions, covering 3 pages of A4 paper (when printed). The survey included items from the Rosenberg Self-esteem Scale (RSES) (Rosenberg, 2006), a widely used scale to measure self-esteem first used on high school students in New York, USA. There were also several questions from the Intermediate Outcomes Measurement Instrument (IOMI) toolkit developed to measure the impact of interventions that work with offenders (Liddle, 2019). Questions on relationship with staff which form part of this toolkit were also included. These were followed by questions from the Leaderships Self-Efficacy scale (LSE) (Bobbio & Manganelli, 2009). All these questions were asked on a five-point Likert scale from strongly disagree to strongly agree.

The next set of questions made up the Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS) (Stewart-Brown et al., 2009), a wellbeing measure commonly used in healthcare settings which has been validated on both young people (aged 15 to 21) and adults. The timeframe across which this section refers to was changed to one month (rather than 2 weeks) to align with the set of offending questions (covered later in this section).

In addition, the survey also comprised employment-related questions, a key outcome for the programme. It included questions on aspirations for work derived from the Longitudinal Study of Young People in England (LSYPE), the major study tracking young people across time, funded by the Department for Education (DfE, 2011). The second scale, the Job Search Self-efficacy scale (JSSE) (Saks, 2015), asked questions on young people’s belief or confidence that they can successfully accomplish particular job search activities.

The final set of questions on offending were all originally sourced from the Edinburgh Study of Youth Transitions and Crime (EYSTC), a longitudinal study to examine the causes and consequences of young people’s involvement in crime and antisocial behaviour (University of Edinburgh, nd). The questions were asked across one month, rather than a year as it is in EYSTC.

Across a few survey measures, wording has been simplified following a pilot study to aid young people in understanding the questions. In addition, not all questions from a specified scale were included, to reduce burden.

Survey analysis

Analysis is presented at 2 levels – aggregate and sub-group. The aggregate level estimates, per statement (COF Survey Q1 to Q44), whether pre- and post-survey differences are statistically significant for the sample. The evaluation team used a t-test statistic at the 95% confidence level to determine significant differences. At the sub-group level, only statements with statistically significant results at the aggregate level were included to analyse the data distribution among different factors (age, sex, ethnicity), and any significant differences for any sub-groups. At each level, the team maintained a minimum sample of 30 to estimate the t-tests.

T-test calculations

Aggregate level

Paired sample t-tests were estimated for 10 scales and the individual statements they are comprised of (see Table A1). Therefore, at the aggregate level, t-test scores were calculated for 10 scales plus 44 questions on the survey.

To calculate the t-test for scales, the team combined questions pertaining to each scale into a single scored variable. For example, the RSES covered statements Q1 to Q6 of the COF survey. As the response options of each statement were between 1 and 5, when combined, scores between 1 and 30 were obtained. The closer the score to 30, the higher the reported self-esteem. The team calculated these scores for the first and second surveys to obtain paired scores for each participant.

They used paired sample means t-tests to estimate the significance of the difference between the first and second score. This test hypothesised that the sample mean difference was not statistically different to zero. In other words, it hypothesised that there was no difference between the mean scores obtained in the first survey and the second survey. Using the 5% significance level, p-values of the test statistic were used to reject or not reject the hypothesis.

Table A1: Scales by survey questions

Scale	Survey question number
Rosenburg self-esteem scale	Q1 to Q6
Impulsivity and problem-solving	Q7 to Q8
Self-efficacy	Q9 to Q10
Trust in others	Q11 to Q12
Leadership self-efficacy	Q13 to Q14
SWEMWBS	Q15 to Q21
Aspiration for work	Q22 to Q26
Job search self-efficacy	Q27 to Q32
Offended last month	Q33 to Q41
Staff relationship	Q42 to Q44

While t-tests for scales were calculated based on scores developed by combining relevant questions, the t-tests for individual statements or questions (one to 44) were simply calculated using the average response (between one and 5 or one and 4) and the sample means without development of any combined scores.

Sub-group level

Chapter 4 Benefits to participants is divided into 3 themes – life outlook, access to jobs, and crime and offending behaviours. Each theme presents a sub-group analysis for scales and statements with statistically significant results at the aggregate level.

Analysis is presented on 3 sets of sub-groups:

• personal characteristics – age, gender, ethnicity
• employability characteristics – disability, participant’s second language is English
• life circumstances – involvement in police/ Criminal Justice System (CJS)/ probation/ legal, involvement with housing support, region

To provide a concise analysis of the data on each of the themes and sub-groups, the team built an index score per theme. The index allowed the use of t-tests to estimate whether statistically significant differences existed at a sub-group level among participants.

The Life Outlook Index comprised any statistically significant statements between survey questions one to 21. The Access to Jobs Index covered statistically significant statements between questions 22 to 32, and on Crime and Offending Behaviours Index on questions 33 to 41. Statistical significance was assessed by testing whether the difference between the first and second survey of the statement was statistically significant at 95% confidence level using t-test statistic

The Life Outlook Index combined the results of individual scales to lend to a broader analysis in a meaningful and rigorous way. To build the Index, survey responses were converted into binaries (agree/strongly agree = one; others = zero), equal weights were applied to each binary variable in the index calculation, and the values were summed. The weights were assumed to be equal in the absence of sufficient knowledge of what questions may be more important to the respondents or similar groups. The index values lay between zero and one, and were unique for pre- and post-survey questions. The average difference for each break variable was calculated by deriving the mean pre- and post-index value for each sub-category. The value of the difference itself did not need to be interpreted, but the magnitude, direction and significance were critical to drawing an inference. A higher index score (or the average difference of score) relative to a sub-category implied greater improvement on a measure. The t-tests were conducted on these average differences.

The Access to Jobs Index was calculated in the same manner, including only questions with a statistically significant mean difference at an aggregate level. Survey question 23 was transformed into a binary variable coding important/very important as one and all other responses as zero. Survey questions 23 and 30 were transformed into binary variables with the categories confident/very confident coded as one and the rest as zero. To derive the index score, the same steps were followed as discussed under the Life Outlook Index above.

The Crime and Offending Behaviours Index was calculated in the same manner, including only questions with a statistically significant mean difference at an aggregate level. Survey questions 33, 37, 39 and 40 were transformed into binary variables with categories confident/very confident coded as one and the rest as zero. The same steps as discussed under the Life Outlook Index above were followed to derive the index score.

Due to the distribution of the data, some categories reported less than 30 responses (see Tables A14 to A16) which led to a small sample for running linked t-tests. Test results are more readily interpreted for statistical inference when samples are larger; however, the validity of the underlying assumptions becomes more critical when interpreting the statistics in small samples. Thus, to ensure the robustness of the t-test results on small samples (N<30), the team also performed visual checks to test normality of the data – one of the core assumptions of a t-test (Kim, 2015).

The team considered estimating the Shapiro-Wilk test statistic to test the normality of data, but its results tend to be unreliable in smaller samples. The visual checks on data using histograms sufficed to establish that the data on the sub-categories (N<30) were not normally distributed. As the data do not hold one of the main assumptions of t-tests for some cases (where N of sub-categories <30), the test of significance should not be interpreted using the p-values.

A note on the analysis of the crime and offending behaviours outcome

While the team needed to analyse offending behaviours to assess the impact of the COF programme, they did not deem it appropriate considering the low disclosure rates among survey participants. Therefore, it was not possible to evaluate this question; however, the survey data on offending was heavily skewed to non-offending behaviours. Note that this was not a proxy for measuring the impact of the programme on offending.

This analysis is based on responses to questions on offending behaviours in the month before the survey, to which the majority in both pre- and post-surveys chose ‘never in the last month’. Across all questions on offending behaviours, the response rate was 96% to 100% of the total sample (N=125). On average, 87% (N=107) of respondents reported they did not offend in the pre-survey, and this rose to 94% (N=114) in the post-survey. Those who reported having offended accounted for an average 3% (N=4) of all respondents on the pre-survey and 2% (N=2) on the post-survey. Respondents who chose ‘do not want to answer’ accounted for 9% (N=11) on average on the pre-survey, which fell to 4% (N=5) on the post-survey.

Table A2 below lays out the results on offending questions by presenting the percentage of responses obtained for each sub-category of response. This shows a shift in behaviours as a larger number of respondents reported not offending in the last month on the post-survey compared to the pre-survey. However, it is necessary to caveat that the results are prone to bias as they come from self-reported data. As before, it is also important to remember that an analysis of non-offending is not the same as an analysis of offending behaviours.

Table A2: Breakdown of responses of offending survey questions

—	Never offended in the last month			Offended once/ 2-4 times/ 5+ times			Do not want to answer
	Survey1	Survey2	Diff	Survey1	Survey2	Diff	Survey1	Survey2	Diff
Q33	85%	91%	6%	5%	5%	0%	10%	4%	-6%
Q34	89%	94%	6%	5%	2%	-3%	7%	4%	-2%
Q35	89%	95%	7%	2%	1%	-2%	9%	4%	-5%
Q36	87%	93%	7%	7%	3%	-3%	7%	3%	-3%
Q37	88%	95%	7%	3%	1%	-2%	9%	4%	-5%
Q38	90%	94%	4%	0%	2%	2%	10%	4%	-6%
Q39	80%	90%	10%	6%	2%	-3%	15%	7%	-7%
Q40	88%	96%	8%	3%	0%	-3%	9%	4%	-5%
Q41	92%	96%	4%	0%	0%	0%	8%	4%	-4%

Analysis of results

The following tables illustrate results from the aggregate level and sub-group level analysis. In Tables A3 to A13, the rows represent the scales (and sometimes statements/survey questions). The columns present the mean values obtained in Survey1 and Survey2 (rounded-off), the difference between the mean values, the p-value of t-test statistic, and the number of observations. Tables A14 to A16 present the index-based results.

Similarly, the tables present the mean index scores in the first and second surveys, the mean index score difference, p-values of the t-test statistic, and the number of observations per sub-group in each row. Tables A4 to A10 (aggregate) and A14 (Life Outlook Index) inform the discussion in Section 4.2; Tables A11 and A12, and A15 (Access to Jobs) lend to Section 4.3; and Tables A13 and A16 to Section 4.4.

Table A3: Difference in mean scores, t-test p-values and sample size by scales

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
Rosenburg Self-esteem scale	21	22	1.3	0.006*	0.37 2.21	111
Relationship with staff	4	4	0.1	0.209	-.83 1.23	84
Impulsivity and problem-solving	3	3	-0.1	0.445	-.09 .22	114
Agency/self-efficacy	4	4	0.2	0.029*	.01 .31	110
Interpersonal trust	4	4	0.1	0.468	-.11 .25	110
Leadership Self-efficacy Scale	4	4	0.2	0.028*	.01 .32	115
Mental Wellbeing Scale (SWEMWBS)	21	22	1.0	0.021*	0.15 1.85	111
Aspirations for work	16	16	0.1	0.603	-0.30 0.51	111
Job search self-efficacy	15	16	1.1	0.005*	0.32 1.80	111
Whether offended in the last month	0.21	0.10	-0.1	0.025*	-0.19 -0.01	94

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A4: RSES – difference in mean scores, t-test p-values and sample size (Q1 to Q6)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
On the whole, I am satisfied with myself	3.5	3.8	0.3	0.003*	.11 .53	123
I am able to do things as well as most other people	3.8	4.0	0.3	0.006*	.07 .43	124
I feel I do not have much to be proud of	2.8	2.5	-0.3	0.027*	.51 .30	124
I certainly feel useless at times	2.9	2.7	-0.2	0.058	-.41 .00	122
All in all, I feel that I am a failure	2.3	2.1	-0.1	0.218	-.33 0.07	124
I take a positive attitude toward myself	3.6	3.8	0.1	0.115	-.03 .33	122

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A5: Impulsivity and problem-solving scale – difference in mean scores, t-test p-values and sample size (Q7 to Q8)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
I often do the first thing that comes into my head	2.9	2.9	0.0	0.934	-.20 .18	123
I often do things without thinking of the consequences	2.4	2.2	-0.1	0.153	-.32 .05	124

Table A6: Agency and self-efficacy – difference in mean scores, t-test p-values and sample size (Q9 to Q10)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
I make good decisions	3.6	3.8	0.2	0.011*	.04 0.36	121
I am confident that I can cope with unexpected events	3.4	3.6	0.1	0.098	-.02 .32	122

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A7: Interpersonal trust – difference in mean scores, t-test p-values and sample size (Q11 to Q12)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
There are people who really understand me	3.5	3.6	0.1	0.404	-.11 .27	121
There are people I can turn to when I have a problem	3.8	3.9	0.1	0.282	-.08 .30	122

Table A8: Relationship with staff – difference in mean scores, t-test p-values and sample size (Q42 to Q44)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
I feel able to trust them	4.3	4.4	0.1	0.310	-.07 .24	84
They do what they say they’ll do	4.4	4.5	0.1	0.244	-.07 .28	84
They have listened to me	4.5	4.6	0.1	0.346	-.07 .22	84

Notes:

1. The CIs are reported as per the statistical software which subtracts Survey1 values from Survey2 thus has differences in negatives, unlike the calculation in the third column. They are both correct and the reader, for convenience, can focus on the absolute values.

Table A9: Leadership self-efficacy – difference in mean scores, t-test p-values and sample size (Q13 to Q14)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
I am able to change things in a group even if they are not completely under my control	3.4	3.6	0.2	0.011*	-.05 .38	124
I can motivate others within a group	3.6	3.8	0.2	0.072	-.01 .37	124

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A10: Short Warwick-Edinburgh Mental Wellbeing Scale – difference in mean scores, t-test p-values and sample size (Q15 to Q21)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
I’ve been feeling positive about the future	3.3	3.6	0.3	0.003*	-.09 0.47	122
I’ve been feeling useful	3.1	3.3	0.2	0.117	-.05 .44	122
I’ve been feeling relaxed	3.2	3.2	0.0	0.682	-.15 .23	122
I’ve been dealing with problems well	3.4	3.4	0.0	0.659	-.14 .22	122
I’ve been thinking clearly	3.4	3.6	0.2	0.046*	-.00 0.34	122
I’ve been feeling close to other people	3.2	3.5	0.3	0.004*	.08 .44	121
I’ve been able to make up my own mind about things	3.7	3.9	0.2	0.082	-.02 .33	120

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A11: Aspiration for work – difference in mean scores, t-test p-values and sample size (Q22 to Q27)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
To have a job you can keep for as long as possible?	3.3	3.4	0.1 0.273	-.05	.20	122
Earning a lot of money?	3.2	3.3	0.1	0.015*	.20 .25	121
A job that leaves you with a lot of free time?	3.0	3.0	0.0	0.608	-.16 .09	121
Finding a job which interests you?	3.6	3.6	0.0	0.783	-.13 .10	121
Finding a job in which you can help others?	3.1	3.1	0.0	0.828	-.13 .16	122
Using networking or personal contacts in your job search?	2.4	2.6	0.2	0.050*	.00 .39	120

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A12: Job search self-efficacy – difference in mean scores, t-test p-values and sample size (Q28 to Q32)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
Using the internet in your job search?	2.9	3.0	0.1	0.294	-.07 .23	121
Recognising your skills related to a job?	2.6	2.8	0.2	0.054	-.0 .35	121
Writing a good CV or application form for a job application?	2.3	2.7	0.3	0.000*	.17 .50	121
Finding information about companies before an interview?	2.6	2.7	0.0	0.624	-.12 .20	121
Presenting yourself well in an interview?	2.6	2.7	0.1	0.066	-.00 .30	121

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A13: Whether offended in the last month – difference in mean scores, t-test p-values and sample size (Q33 to Q41)

—	Survey1	Survey2	Diff	p-value	[95% CI]	N
Have you hit, kicked or punched someone on purpose? Not brothers, sisters or play-fighting	1.5	1.2	-0.3	.015*	-.05 -.4	122
Have you damaged something that did not belong to you on purpose for example, windows, cars or streetlights?	1.3	1.1	-0.2	0.094	-.34 .02	121
Have you stolen something from a shop or somewhere else?	1.4	1.2	-0.2	0.058	-.41 .00	122
Have you been noisy or disruptive in a public place so that people have complained, or you got into trouble?	1.3	1.2	-0.2	0.051	-.36 .00	120
Have you carried a knife or other weapon with you to keep you safe or in case it was needed in a fight?	1.4	1.1	-0.2	0.015*	-.41 -.04	122
Have you used force, threats or a weapon to steal money or something else from somebody?	1.4	1.2	-0.2	0.053	-.39 .00	122
Have you taken any drugs?	1.7	1.3	-0.4	0.002*	-.13 -.59	121
Have you set fire or tried to set fire to something on purpose?	1.4	1.1	-0.3	0.011*	-.44 -.05	119
Have you gone into or broken into a house, shop or car to try and steal something?	1.3	1.1	-0.2	0.058	-.34 .00	119

Notes:

1. An asterisk (*) against a p-value denotes that the reported average difference is statistically significant at the 95% confidence level.

Table A14: Life outlook, mean index scores, p-values and sample size

—	—	Survey1	Survey2	Average diff	p-value	[95% CI]		N
Gender	Female	0.48	0.54	0.06	0.0656	-.00	.14	46
	Male	0.53	0.66	0.13	0.0001*	.07	.16	70
Age	16-19	0.50	0.63	0.13	0.0001*	.07	.21	58
	20-25	0.52	0.60	0.08	0.0247*	.00	.13	58
Ethnicity	Asian/ Asian British	0.51	0.58	0.07	0.1139	-.02	0.17	26
	Black/ black British	0.51	0.68	0.17	0.0004*	.08	.27	35
	Other ethnicities	0.47	0.66	0.19	0.0016*	.09	.31	13
	White	0.53	0.57	0.04	0.3372	.04	.11	43
Disability	No disability	0.50	0.61	0.11	0	.06	.17	98
	Disabled	0.56	0.63	0.06	0.4327	-.07	.16	19
English language	English is not second language	0.49	0.60	0.11	0	.06	0.16	98
	English is second language	0.61	0.69	0.09	0.1782	-.04	.24	19
CJS	Not involved with the CJS	0.51	0.59	0.07	0.0039*	.02	.12	96
	Involved with the CJS	0.50	0.72	0.23	0	.15	.35	21
Housing support	Not involved with housing support	0.50	0.60	0.09	0.0002*	.04	.15	101
	Involved with housing support	0.56	0.71	0.15	0.0083*	.04	.25	16
Region	Birmingham, Sandwell, Wolverhampton	0.55	0.68	0.13	0.0098*	.03	.25	23
	Liverpool and Manchester	0.55	0.62	0.07	0.2508	-.05	.19	13
	Newham and Hackney	0.47	0.63	0.16	0.0005*	.08	.28	36
	Croydon and Lambeth	0.47	0.60	0.13	0.0532	-.00	.26	18
	Bradford and Leeds	0.50	0.49	-0.01	0.7371	-.13	.09	21
	Kingston upon Hull	0.69	0.71	0.02	0.7646	-.13	.16	6

Notes:

1. An asterisk (*) against a p-value denotes the test is statistically significant at the 95% confidence level. Some sub-groups do not add up to the complete survey sample size of 125 due to missing information. Although several results reported in the sub-group analysis are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30.

Table A15: Access to jobs, mean index scores, p-values and sample size

—	—	Before	After	Average diff	p-value	[95% CI]		N
Gender	Female	0.6	0.7	0.1	0.1844	-.02	.13	45
	Male	0.6	0.7	0.1	0.0047*	.03	.16	72
Age	16-19	0.7	0.8	0.1	0.0623	-.00	.13	59
	20-25	0.6	0.7	0.1	0.0554	-.00	.16	58
Ethnicity	Asian/ Asian British	0.6	0.7	0.1	0.0675	-.00	.25	26
	Black/ black British	0.7	0.7	0.0	0.5446	-.06	.11	36
	Other ethnicities	0.7	0.8	0.1	0.1039	-.02	.18	12
	White	0.6	0.7	0.1	0.1242	-.02	.17	43
Disability	No disability	0.6	0.7	0.1	0.0489*	.00	.11	97
	Disabled	0.6	0.7	0.2	0.0287*	.01	.29	21
English language	English is not second language	0.6	0.7	0.1	0.0506	-.00	.11	97
	English is second language	0.7	0.8	0.1	0.0239*	.02	.28	21
CJS	Not involved with the CJS	0.6	0.7	0.1	0.0197*	.01	.13	96
	Involved with the CJS	0.7	0.8	0.1	0.1336	-.03	.21	22
Housing support	Not involved with housing support	0.6	0.7	0.1	0.0037*	.02	.14	102
	Involved with housing support	0.8	0.8	0.0	1	-.13	.13	16
Region	Birmingham, Sandwell, Wolverhampton	0.7	0.7	0.1	0.4484	-.10	.22	21
	Liverpool and Manchester	0.6	0.7	0.1	0.3356	-.08	.22	14
	Newham and Hackney	0.6	0.7	0.1	0.0039*	.04	.23	37
	Croydon and Lambeth	0.7	0.7	0.0	0.3313	-.21	.07	18
	Bradford and Leeds	0.6	0.7	0.1	0.0952	-.01	.22	22
	Kingston upon Hull	0.6	0.7	0.1	0.465	-.18	.35	6

Notes:

1. An asterisk (*) against a p-value denotes the test is statistically significant at the 95% confidence level. Some sub-groups do not add up to the complete survey sample size of 125 due to missing information. Although several results reported in the sub-group analysis are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30.

Table A16: Offending behaviours, mean index scores, p-values and sample size

—	—	Before	After	Average diff	p-value	[95% CI]		N
Gender	Female	0.98	0.99	0.01	0.6599	-.02	.03	44
	Male	0.95	0.98	0.02	0.0276*	.00	.08	51
Age	16-19	0.96	0.98	0.02	0.0567	-.00	.07	41
	20-25	0.97	0.98	0.00	0.4105	-.01	.04	54
Ethnicity	Asian/ Asian British	0.98	0.99	0.01	0.1623	-.01	.05	21
	Black/ black British	0.97	0.99	0.02	0.1033	-.00	.07	28
	Other ethnicities	0.93	0.98	0.05	0.4405	-.08	.17	11
	White	0.97	0.97	0.00	0.744	-.03	.04	36
Disability	No disability	0.96	0.98	0.02	0.0244*	.00	.05	82
	Disabled	0.98	0.97	-0.01	0.583	-.08	.05	14
English language	English is not second language	0.96	0.98	0.01	0.0882	-.00	.05	77
	English is second language	0.99	1.00	0.01	0.3306	-.01	.04	19
CJS	Not involved with the CJS	0.97	0.98	0.00	0.2537	-.01	.04	84
	Involved with the CJS	0.92	0.99	0.06	0.0388*	-.00	.16	12
Housing support	Not involved with housing support	0.97	0.98	0.01	0.1087	-.00	.04	83
	Involved with housing support	0.94	0.98	0.04	0.337	-.0.4	.12	13
Region	Birmingham, Sandwell, Wolverhampton	0.95	0.95	0.00	0.1894	-.02	.13	14
	Liverpool and Manchester	1.00	1.00	0.00	.			10
	Newham and Hackney	0.95	0.99	0.04	0.096	-.00	.09	30
	Croydon and Lambeth	0.95	0.96	0.01	0.7192	-.08	.11	15
	Bradford and Leeds	0.99	1.00	0.01	0.3293	-.01	.03	21
	Kingston upon Hull	1.00	0.96	-0.04	0.3632	-.14	.06	6

Notes:

1. An asterisk (*) against a p-value denotes the test is statistically significant at the 95% confidence level. Some sub-groups do not add up to the complete survey sample size of 125 due to missing information. Although several results reported in the sub-group analysis are statistically significant, they need to be interpreted with caution wherever the sample sizes are lower than 30.

Theory of Change assumptions

The ToC is based on several assumptions for each of the Activities, Outputs, Outcomes and Impacts. It is necessary to consider these assumptions, as the extent to which they are realised will determine the overall impact of the COF programme. These assumptions are outlined below, categorised by the components of the ToC they underpin. The letters next to each assumption map to the logic model illustrated in Section 1.2.

Programme level

A.  COF programme is not duplicating work of existing programmes.

B.  Young people are committed to desist from crime, offending and gangs.

C.  Employment and training-related opportunities are the right intervention.

Activities to outputs

A.  Referral routes onto the programme are effective, both from third parties and self-referrals.

B.  Assessment ensures the programme targets key groups.

C.  Accurate assessment of participant needs, including SEND needs.

D.  Underrepresented groups identified and engaged across local authorities.

E.  Support packages are flexible and tailored.

F.  Trust is built between careers coaches and young people.

G.  Wraparound support covers the breadth of participants’ needs.

H.  Accurate signposting to relevant support.

Outputs to outcomes

A.  Young people involved in design of programme.

B.  Young people engage with career coach and Apprentice Nation platform.

C.  Cross-departmental working.

D.  Quality employment opportunities are created and offered to young people.

E.  Staff are able to engage young people.

F.  Young people are retained on the programme.

G.  Employers are engaged and commit to working with vulnerable young people and harder-to-reach groups.

H.  Programme is effective in increasing the skills of young people.

I.  There are jobs available for young people to access.

J.  Communities recognise personal change in young people.

K.  Staff trained to effectively engage and refer young people.

L.  Appropriate number of careers coaches and staff available to provide young people with personal support.

Outcomes to impacts

A.  Creation of employability and skills-based opportunities with local/national businesses across areas.

B.  Programme is effective in encouraging young people’s desistance from crime and offending.

C.  Programme is effective in encouraging young people’s desistance from gangs.

D.  COF programme can bring about identity change to young people involved in offending or gangs.

Interviews

RSM researchers collected qualitative evidence between March 2022 and November 2022 (see Table A17). Topic guides were tailored to each respondent group from a template aligned with key delivery and outcome questions.

Table A17: Qualitative research activities

Participant group	Research method	Sampling approach
Delivery staff	27 online interviews	All sample
Participants	- 7 face-to-face interviews - 82 online interviews	Purposive. Main criteria used for selection: - age - gender - ethnicity - RAG rating
External stakeholders	29 online interviews	Snowball

Impact measures definitions

The team developed the definitions below in agreement with the team at Home Office Serious Violence, Crime and Policing Analysis Unit. The evaluation logic model and methodology (that is, use of surveys as a data collection tool) were designed to rely on quantitative evidence as the primary metric for assessing programme impact, with the qualitative evidence providing context and supplementing the impact analysis findings.

Achievement Measure	Definition
Achieved	There is evidence to conclude that the output/outcome has been achieved based on: - statistically significant results at the aggregate level - statistically significant results for at least one individual measure - quantitative data confirm outcome achievement.
Partially achieved	There is evidence to infer that some of the output/outcome may have been achieved based on: - no statistically significant results at the aggregate level - statistically significant results for at least one individual measure - quantitative data evidence outcome achievement.
Inconclusive	The evidence indicates that this outcome has not been achieved based on: - no statistically significant results at the aggregate level - no statistically significant results at the individual measure level
No evidence	There is no evidence to assess this outcome or to suggest that the output/outcome has been achieved.

Appendix 2: Profile of COF participants

This section contains a descriptive analysis of Aptem data from participants from 11 November 2021 to 13 February 2023. In this timeframe, 941 participants were onboarded on COF; of these, 157 disengaged and 200 dropped out.

If a participant disengaged or contact is lost, career coaches followed a focused process of attempting to re-establish communication with the participant. A participant was marked as ‘disengaged’ after 4 weeks without any contact with their career coach, who were required to check-in every fortnight up to 8 weeks from the point of disengagement. If successful, the participant had a re-engagement review to understand why they disengaged from the programme. If re-engagement was unsuccessful, the participant was moved to ‘off programme’.

Key demographics

Approximately two-thirds (68%) of participants in the programme were male and one-third (32%) female.

Source: COF programme data (as at 13 of February 2023)

Base: 882

While the programme was targeted at young people aged 16 to 24, there were a few participants among those aged 25 (5%) and 26 (1%). A substantial proportion of participants was in the younger end of this age range. In total, 20% of participants were aged 16 and 19% were aged 17, followed by participants aged 18 (12%) and participants aged 19 (10%). Of the remaining participants, 15% were aged 20 to 21 and 18% were 22 to 24.

Age group

Source: COF programme data

Base: 906

The participants came from a diverse range of ethnic backgrounds. Black/black British and Asian/Asian British make up 22% and 17% of participants, respectively. Only 8% of participants reported dual heritage, with the same proportion reporting their ethnicity as other ethnic groups. Almost half (45%) of participants reported white as their ethnicity.

This is not particularly surprising, as an analysis of the Annual Population Survey covering the period from October 2021 to September 2022 estimated that 61% of all eligible participants aged 16 to 24 in the LAs covered by COF were from a white ethnic background (data from the Annual Population Survey from October 2022 to February 2023 was unavailable at the time of writing). This figure ranges from a low of 26% in Newham to a high of 88% in Leeds.

Note that the Annual Population Survey groups individuals aged 16 to 19, 20 to 24 and 25 to 29. As the eligible age range of participants for the COF programme was 16 to 24, only figures of individuals aged 16 to 24 are presented from the Annual Population Survey.

Ethnic group

Source: COF programme data

Base: 811

Table A18: Population aged 16 to 24 with a white ethnic background, by COF local authority

Local authority	Population aged 16-24 – white	COF participants aged 16-24 – white	COF participants aged 16-26 – white
Birmingham	47%	30%	30%
Croydon	44%	16%	16%
Lambeth	46%	13%	12%
Hackney	51%	15%	19%
Sandwell	55%	50%	49%
Manchester	53%	55%	52%
Bradford	80%	48%	50%
Leeds	88%	57%	55%
Wolverhampton	63%	46%	44%
Liverpool	81%	88%	88%
Kingston upon Hull, City of	80%	100%	95%
Newham	26%	21%	20%

Source: Annual Population Survey (ONS) October 2021 to September 2022, COF programme data

Base: 763 (COF participants aged 16 to 24), 811 (COF participants aged 16 to 26)

Eleven percent of participants considered themselves to have a disability and more than one in 4 (28%) considered themselves to have mental health challenges or psychological disorders.

Local authority

Source: COF programme data

Base: 941

Employment and education

Almost all (94%) participants were not working all at programme start. The remaining 6% comprised participants working 16 hours or fewer per week (5%), being unable to work due to illness, disability, caring or asylum (1%). Fewer than 5 participants were working more than 16 hours per week.

Employment status

Source: COF programme data

Base: 830

Four in 5 (84%) participants were not in education. Among those in education, more were in education full time (more than 14 hours per week) than part time (less than 14 hours per week), at 9% and 7% respectively.

Education status

Source: COF programme data

Base: 830

Of those out of work, over a quarter (28%) were defined as long-term unemployed (LTU), which is out of work for more than 12 months. A further one in 5 (22%) had been out of work between 6 and 12 months, with some at risk of becoming LTU on the duration of the COF programme; 12% of participants had been out of work between 3 and 6 months; with the remaining 38% out of work for less than 3 months, and it is these participants who have the closest labour market attachment.

Duration of unemployment

Source: COF programme data

Base: 764

Overall, among 34 sectors recorded by all participants (N=345), construction stood out as the most desired sector of interest, with 23% of all participants interested in working here, followed by retail (16% of all participants). The remaining participants were interested in a variety of sectors. Female participants were most interested in working in other sectors (12%) and retail (11%). A large proportion (28%) of male participants, however, were interested in working in construction, followed by retail (13%).

Among the participants, there was a relatively low level of educational attainment, with half having no formal qualifications (29%) or to lower secondary education only (16%). A further 12% and 21% respectively had Level one or Level 2 qualifications with just 21% having qualifications at Level 3 or above. Combining the first 3 categories means at least 57% of participants did not have a pass at English or maths GCSE, a minimum requirement for many employment opportunities. Since 94% of participants had been unemployed at programme start, that group strongly dominates the above breakdown by qualification. For those in work at programme start (5%), however, the average qualifications are higher (64% holding Level 2, 3 and above qualifications).

Qualifications

Source: COF programme data

Base: 830

Barriers and risk factors

This subsection details other issues which may be a barrier to employment or contribute to offending behaviour. A quarter (22%) of participants who started the programme had difficulties with school attendance. A similar proportion had problems with everyday reading and writing and everyday maths, each at 21%. Technological illiteracy was also common, with 16% of participants having problems with computers. Problems with relationships within the family and in their household were also fairly common, with participants at 12% and 8%, respectively. The following chart shows 42% of participants presented with no such problems.

Does the participant have any problems…

Source: COF programme data

Base: 73 to 203

References

Better Evaluation (2022) Manager’s guide to evaluation: Describe the theory of change. [Accessed 30 May 2024]

Bobbio, A. & Manganelli, A. M. (2009) ‘Leadership self-efficacy scale: A new multidimensional instrument.’ TPM-Testing, Psychometrics, Methodology in Applied Psychology, vol. 16(1), pp. 3 to 24. [Accessed 30 May 2024]

Braun, V. & Clarke, V. (2006) ‘Using thematic analysis in psychology’. Qualitative research in psychology, vol. 3(2), pp. 77 to 101.

CORC (2008) Short Warwick-Edinburgh Mental Wellbeing Scale (SWEMWBS). Child Outcomes Research Consortium. [Accessed 30 May 2024]

DfE (2011) Longitudinal Study of Young People in England – User Guide to the Datasets: Wave 1 to Wave 7. Department for Education. [Accessed 30 May 2024]

Frisby-Osman, S. and Wood, J. (2020) ‘Rethinking How We View Gang Members: An Examination into Affective, Behavioural, and Mental Health Predictors of UK Gang-Involved Youth.’ Youth Justice, vol. 20(1-2), pp. 93 to 112

Kim, T.K. (2015) ‘T test as a parametric statistic.’ Korean Journal of Anaesthesiology, vol. 68(6), pp. 540 to 546.

Liddle, M. et al. (2019). Intermediate Outcomes Measurement Instrument (IOMI) Toolkit: Guidance Notes. HM Prison & Probation Service. [Accessed 30 May 2024]

Maruna, S., (2001) Making Good: How Ex-Convicts Reform and Rebuild Their Lives. Washington, DC: American Psychological Association Books.

Mayne, J. (2008) Contribution analysis: An approach to exploring cause and effect. ILAC Briefing 16.

MoJ (2018) Justice Data Lab analysis: re-offending behaviour after receiving treatment from Key4Life. Ministry of Justice. [Accessed 30 May 2024]

Rosenberg, M. (2006) Rosenberg Self-Esteem Scale (RSES). American Psychological Association. [Accessed 30 May 2024]

Saks, A.M., Zikic, J. & Koen, J. (2015) ‘Job search self-efficacy: Reconceptualizing the construct and its measurement.’ Journal of Vocational Behavior, vol. 86, pp. 104-114. [Accessed 30 May 2024]

Sampson, R. and Laub, J., (2003.) ‘Life-Course Desisters? Trajectories of Crime Among Delinquent Boys Followed to Age 70.’ Criminology, vol. 41(3), pp. 555 to 592.

Smith, D.J., McVie, S., Woodward, R., Shute, J. and McAra, L. (2001) ‘The Edinburgh Study of Youth Transitions and Crime: Key findings at ages 12 and 13.’ The Edinburgh Study of Youth Transitions and Crime, Research Digest number one.

Stewart-Brown, S., Tennant, A., Tennant, R., Platt, S., Parkinson, J. and Weich, S. (2009) ‘Internal construct validity of the Warwick-Edinburgh mental well-being scale (WEMWBS): A Rasch analysis using data from the Scottish health education population survey.’ Health Qual Life Outcomes, vol. 1(7), pp. one to 8. Scale [Accessed 30 May 2024]

University of Edinburgh (nd) Edinburgh Study of Youth Transitions and Crime. [Accessed 30 May 2024]

Van der Geest, V., Bijleveld, C. and Blokland, A. (2011) ‘The Effects of Employment on Longitudinal Trajectories of Offending: A Follow-Up of High-Risk Youth from 18 to 32 Years of Age.’ Criminology, vol. 49(4), pp. 1195 to 1234.

What Works Growth (2015) The Maryland Scientific Methods Scale (SMS). [Accessed 30 May 2024]