Research and analysis

Youth Employment Initiative – Impact Evaluation

Published 2 March 2022

DWP research report no. 1011

A report of research carried out by Ecorys UK on behalf of the Department for Work and Pensions.

Crown copyright 2022

You may re-use this information (not including logos) free of charge in any format or medium, under the terms of the Open Government Licence

Or write to the:

Information Policy Team
The National Archives
Kew
London
TW9 4DU

Email: psi@nationalarchives.gov.uk

This document/publication is also available on our website

If you would like to know more about DWP research, email socialresearch@dwp.gov.uk

First published March 2022

ISBN 978-1-78659-360-3

Views expressed in this report are not necessarily those of the Department for Work and Pensions or any other government department.

Acknowledgements

This report was commissioned by the Department for Work and Pensions (DWP). The evaluation was part-funded by European Social Fund (ESF) technical assistance. We would like to thank Nicholas Campbell and Benjamin Ashton in DWP’s ESF Evaluation Team for their guidance and contributions throughout the project.

We would also like to thank all the staff and participants who gave up their time to take part in the study. Without them of course, the research would not have been possible.

Authors’ credits

This report was authored by Ian Atkinson and Matthew Cutmore of Ecorys UK. Ecorys UK are experienced in conducting research into European programmes aimed at supporting individuals into employment and bringing about social inclusion. They have contributed to a number of studies for the DWP, other UK Government Departments and the European Commission around these themes.

Glossary of terms

Term Definition
Counterfactual impact evaluation (CIE) CIE is a type of impact evaluation using a counterfactual analysis approach. Counterfactual analysis compares the real observed outcomes of an intervention with the outcomes that would have been achieved had the intervention not been in place (the counterfactual). A CIE can involve use of a randomised controlled trial (RCT) methodology, also referred to as an ‘experiment’, or quasi-experimental methods that seek to mimic an experiment, often through construction of a comparison group to compare outcomes with the treatment group receiving an intervention. The CIE for the Youth Employment Initiative impact evaluation uses a quasi-experimental approach.
Difference in differences (DiD) DiD analysis is an impact evaluation technique, seeking to estimate the effect of a treatment on an outcome of interest through assessing change over time in an outcome for a treatment group relative to the average change over time for a control/comparison group.
European Social Fund (ESF) The ESF is the European Union’s (EUs) main financial instrument for supporting jobs, helping people get better jobs and ensuring fairer job opportunities for EU citizens. The European Commission works with countries to set the ESF’s priorities and determine how it spends its resources.
(ESF) Managing Authority (MA) The Department for Work and Pensions (DWP) assumes the role of the ESF Managing Authority for England. It has overall responsibility for administering and managing the ESF and reporting to the European Commission.
(ESF) Operational Programme (OP) Operational Programmes describe the priorities for ESF activities and their objectives at national or regional levels within the European Union[footnote 1].
Local Enterprise Partnerships (LEPs) LEPs are voluntary partnerships of local authorities and businesses with responsibility for deciding on general economic priorities at the local level.
European Structural and Investment Funds (ESIF) Sub-Committee Each LEP area has a sub-committee that provides implementation advice to the Managing Authorities for the ESIF Growth Programme in England. Their role is to advise Managing Authorities on local growth conditions and priorities with regard to project call specifications, funding applications and implementation.
Monitoring Information (MI) Data that is collected regularly by YEI providers following a set template. All data is forwarded to DWP, which collates and analyses the information.
Nomenclature of Units for Territorial Statistics (NUTS) NUTS areas are geographical territories identified through a standard developed and regulated by the EU in order to reference the sub-division of countries for statistical purposes.
Propensity score matching (PSM) PSM is a statistical technique used to estimate the impact of an intervention on a set of specific outcomes. It mimics an experimental research design by comparing outcomes for a treatment group and a statistically generated comparison group, which is similar to the treatment group in its composition.
Theory of change Theory of change is an evaluation methodology drawing on work developed in the United States to evaluate community and social programmes. The approach involves identifying the logic behind an intervention in terms of its rationale and aim, key objectives, inputs, activities and short, medium and long term outcomes and testing this ‘intervention logic’ through a range of evaluative methods.

Executive summary

Introduction

This summary presents the findings of an impact evaluation of the Youth Employment Initiative (YEI), undertaken by Ecorys. The YEI represents part of the European Commission’s (EC’s) policy response to the social and economic challenges stemming from the financial crisis of 2007-2008, and is implemented in England as part of the European Social Fund (ESF). The YEI impact evaluation commenced in April 2017 and ran until late 2019. It follows a previous process evaluation published by the Department for Work and Pensions (DWP) in 2017[footnote 2]. The impact evaluation focuses on examining the effectiveness, efficiency and impact of the YEI.

Methodology

A mixed-method approach was adopted to evaluate the effectiveness, efficiency and impact of the YEI. The findings presented below are drawn from:

  • a desk-based review of YEI and related documentation
  • secondary analysis of data, including YEI Management Information (MI), results from the separate ESF and YEI Leavers Survey commissioned by DWP, and additional statistical data from the Office for National Statistics (ONS)
  • interviews with high-level stakeholders from the ESF Managing Authority (MA)
  • primary research across 10 delivery areas, including interviews with YEI provider staff, European Structural and Investment Funds (ESIF) sub-committee representatives, and YEI participants
  • a counterfactual impact evaluation (CIE) strand, comparing outcomes for YEI participants with a comparison group

The evaluation findings also include a cost-benefit analysis (CBA), undertaken by DWP analysts based on the results of the impact evaluation strand.

Key findings

Effectiveness

To date, the YEI has proved generally effective in terms of targeting participants, the delivery models developed to implement provision, and the delivery of provision itself. The programme is broadly on target in terms of anticipated engagement numbers and in respect of the anticipated gender split of participants. Key factors supporting effectiveness in targeting include: effective use of local data and intelligence; use of delivery partners’ existing networks; focusing extensively on outreach activity; developing partnerships with Jobcentre Plus to encourage referrals; and engaging in co-location with other services.

The delivery models developed also appear to be effective. Notable aspects to this include the use of ‘key worker’ roles to provide consistency and an overview of individuals’ support needs, the development of partnerships and referral routes to offer a wide-ranging support offer, and the establishment of effective governance procedures. The available evidence also suggests a positive picture of the effectiveness of YEI delivery itself in terms of the provision on offer and its implementation. The range of support available, and the extent it is tailored to individual needs, are key factors promoting effectiveness.

Several aspects of the YEI support offer consistently emerged as being effective and important. These included: ‘wraparound’ support designed to address individuals’ personal and often deep-seated challenges and barriers, commonly facilitated through a key worker role; short, sharp interventions to address a small employment need or gap in a young person’s CV; English and Mathematics provision, a lack of competence and qualifications in respect of which was seen as a key barrier to finding work; training linked to employment route-ways; and community-based activities and volunteering, such elements being key in reducing isolation and increasing confidence as part of moving towards employment.

While the majority of YEI delivery and provision can be assessed as effective, in particular contexts or for particular participants some elements appeared to be less so. For example, more structured provision in a classroom setting was cited as discouraging the engagement of young people in some cases, potentially due to prior negative experiences at school.

The positive overall impression of effectiveness was also apparent when considering the quality of the employment and training offers participants received. The YEI Leavers Survey provided the main evidence for this assessment. Nine in ten respondents accessing traineeships felt that their traineeship would improve their chances of getting a job. In terms of job offers, around half of the 45 per cent of respondents reporting that they were in work six-months after leaving the YEI were on a permanent employment contract and a further 14 per cent on a contract lasting 12 months or more. This suggests the majority of jobs gained by YEI leavers were fairly stable and long term. Finally, almost two-thirds of YEI leavers receiving job offers between starting on the programme and six months after leaving rated the quality of offers received as either ‘very good’ (27 per cent) or ‘good’ (35 per cent). Less than one in ten rated their offers as either ‘poor’ or very poor’.

Efficiency

The available evidence suggests that organisations delivering the YEI are concerned with ensuring efficiency and hence, ultimately, offering value-for-money. Common examples given by provider representatives of how they sought to ensure efficient delivery included: close control of staffing numbers; ensuring appropriate caseloads; focusing on staff progression, retention and training to reduce recruitment costs; reducing transaction costs between delivery partners; and reducing overheads where possible. Other examples included reducing costs through re-use and recycling, using account holder discounts when purchasing provision and courses, and avoiding overlaps or duplication with other provision available locally.

While the general impression was one of efficient delivery, there are limitations to assessing the efficiency of specific types of YEI provision and activities. There is a lack of data to facilitate such assessment, with providers not generally identifying costs and throughput of individual activities (in terms of numbers participating). In addition, it was evident that the inter-related nature of many YEI activities naturally makes such data gathering and assessment problematic.

Accepting these limitations, work around personal development, such as building confidence, was seen as intensive though ultimately cost-effective when balanced against the positive effects this was often seen as having for young people. Short courses leading to required qualifications were also cited from this perspective, in that a relatively small outlay could fill a gap in a CV and unlock opportunities.

The case study research was used to identify aspects that promoted or hindered efficiency. A focus on unit costs in developing provision was widely cited as a factor supporting efficiency. This was noted by provider representatives as resulting from DWP as the ESF MA requesting details of such costs in the application process. Effective partnership working and governance also evidently plays a key role in enhancing efficiency, whether through ensuring that the particular skills and organisational capacities can be efficiently deployed, or that performance can be managed by the lead partner to promote efficiency across delivery partnerships.

Impact

Evidence shows that the YEI is having a positive impact on the employment prospects of participants. Compared to a similar comparison group constructed from administrative data, results from the CIE show that, on average, YEI participants were in employment for an additional 56 days in the twelve months following support. Effects on likelihood of claiming benefits are less clear, with there being no statistically significant effect on the likelihood of claiming Jobseekers’ Allowance (JSA) or Employment and Support Allowance (ESA) following YEI support. As explained in the main report, it was not possible to provide estimates on the likelihood of claiming Universal Credit (UC) due to data limitations in the context of UC roll-out during the YEI’s delivery period.

At the overall programme level, a CBA based on the results of the CIE and conducted by DWP analysts estimates that while the programme has a social return in the range of £1.50 to £1.55 per £1 spent, the fiscal return is in the range of £0.13 to £0.17 per £1 spent[footnote 3]. It is acknowledged, however, that these are likely to be underestimates with the true value for money being higher.

The positive estimates of employment impacts generated by the CIE are mirrored by the evidence available from the MI data and ESF Leavers Survey. While the former shows that just under a third of participants are recorded as entering employment immediately on leaving the programme, on the basis of respondents’ self-reporting the Leavers Survey indicates that just under one in two are in employment six months after leaving. Taking account of the YEI client group, for whom the evidence indicates labour market disadvantage and multiple barriers to work are common, the mutually reinforcing evidence around employment impacts from the CIE, MI and survey data can be considered very positive.

The MI data also enables an assessment by gender and disadvantaged status, using the definitions of disadvantage given in the ESF guidance. The data shows little variation in employment outcomes by gender. However, positive employment outcomes are considerably lower for participants recorded as having an additional labour market disadvantage (in addition to being NEET), compared to those for whom a disadvantage is not recorded. Amongst those with a disadvantage, just over a quarter (26 per cent) were recorded as entering employment on leaving the YEI, with the equivalent figure for those with no disadvantage being just over a third (37 per cent).

Evidence of education and training impacts is also available from the MI data and the Leavers Survey. As at September 2019, the MI showed that just under one in five YEI leavers moved into education or training, with just under one in ten gaining a qualification on leaving. Similar evidence is available from the YEI Leavers Survey, with 16 per cent of respondents reporting that they were in education or training at the six-month point after leaving. When adding respondents in training or education six months after leaving to those reporting that they were in work at this point (45 per cent), the survey evidence presents a very positive impression of just under two-thirds of participants achieving positive destinations in the period after support. For a programme seeking to reduce numbers of young people not in employment, education or training (NEET), this is further evidence of the key aims of the YEI being met to a considerable extent.

As well as the ‘harder’ impacts evident from the YEI in terms of entry into work or learning, evidence suggests that the programme frequently has a wide range of beneficial ‘softer’ impacts, such as enhancing self-confidence. Consistent proportions of around four-fifths of respondents to the Leavers Survey, for example, reported that the YEI had helped with a range of such outcomes, including communication skills, self-confidence around work, ability to do things independently, motivation to find a job or do more training, and team working. Similarly, improved confidence, aspirations and motivation emerged as particularly strong and consistent themes in participants’ discussions of the impact of support during case study interviews.

A range of new or enhanced skills resulting from engagement with the YEI were also frequently cited by participants, including improved leadership skills, employability skills, skills around budgeting and managing finances, enterprise skills, and subject-based skills, including those related to Mathematics and English. The case study evidence also indicated that such softer outcomes are central to the achievement of broader YEI impacts, for example around reducing barriers to, and entering, work and learning. As such, they help explain, and reinforce, the positive impacts discussed above around levels of re-entry to employment and learning amongst the NEET young people supported through the YEI.

In addition to impacts on YEI participants, evidence indicates that the programme is having a number of broader impacts. The clearest and most consistent involve beneficial impacts for provider organisations. These include improved delivery systems and structures gained from insights from working with partners, enhanced ability in terms of delivering ESF-funded provision, and the development of new links and relationships with agencies and organisations working to support young people. Evidence around impacts on employers and local communities was more anecdotal and piecemeal, though still positive in cases where concrete examples were offered.

Finally, it was also evident that the experience of designing and delivering the YEI has generated broader ‘learning’ impacts. Typically, these were described in terms of helping to inform or crystallise policy insights, and/or as providing lessons for employability and skills initiatives targeted at young people. Specific examples included insights on the importance and frequent necessity of addressing mental health in the context of employability support, the need to focus more on young people further from the labour market as youth employment rates improve, and the potential gap that will be left following the end of the YEI, along with the implication that similar provision is likely to continue to be required in future.

Conclusion

Evidence across the different evaluation strands presents a positive impression of the effectiveness, efficiency and impacts generated by the YEI. This demonstrates that interventions of this type, specifically focused on targeting and supporting NEET young people, can be effective and lead to a range of positive outcomes. The support on offer is clearly welcomed by young people, with the flexibility and wraparound elements being central to participants’ engagement with the YEI and the positive results they gain from it.

Looking at the theory of change developed to structure the assessment of the programme, it is apparent that the majority of intended outcomes can be assessed as being achieved or exceeded. In particular, results in relation to the core aim of the programme, around supporting young people to (re-)enter employment, education or training, hence contributing to a reduction in NEET levels, can be viewed as very positive. In addition, evidence indicates that the YEI had a positive effect on many of the broader, intermediate outcomes likely to have contributed to these ultimate impacts, particularly in terms of enhanced confidence and interpersonal skills, alongside reduced labour market barriers. The relative success of the programme is also notable in light of the nature of the YEI client group, suggesting learning can usefully be drawn for future initiatives aimed at reducing NEET levels.

1. Introduction

This report presents the findings of an impact evaluation of the Youth Employment Initiative (YEI), undertaken by Ecorys on behalf of the Department for Work and Pensions (DWP). The evaluation adopted a mixed-method approach, combining qualitative case study research with a counterfactual impact evaluation (CIE), supplemented by cost-benefit analysis (CBA) and examination of additional secondary data sources. This impact evaluation follows on from a previous process evaluation of the YEI, published by DWP in 2017. This introduction provides an overview of the YEI before briefly detailing the aims, objectives, and methodology for the evaluation. Subsequent chapters examine the range of evidence gathered to assess and make judgements on the YEI’s effectiveness, efficiency and impact respectively.

1.1. Youth Employment Initiative overview

The YEI forms part of the European Commission’s response to high levels of youth unemployment in the wake of the financial crisis of 2007-2008. Designed to complement other national and ESF provision, the YEI provides support to those under the age of 25, or 29 in some cases, residing in European Union (EU) regions particularly affected by youth unemployment[footnote 4]. Reflecting this geographical targeting, 90 per cent of YEI funding is channelled to regions where the youth unemployment rate in 2012 was higher than 25 per cent, or where youth unemployment was more than 20 per cent but had increased by more than 30 per cent in 2012. YEI provision typically includes support to access apprenticeships, traineeships, job placements and further education, amongst other employability assistance combined with wrap-around support for participants.

As reflected in the ESF Operational Programme (OP) 2014-2020 for England, the overall objective of the YEI is to support the sustainable integration of young people into the labour market, in particular those not in employment, education or training including young people at risk of social exclusion and young people from marginalised communities. The specific objectives of the YEI are:

  • “To support the rise in the participation age by providing additional traineeship and apprenticeship opportunities for 15-29 year old NEETs in YEI areas, with a particular focus on 15-19 year old NEETs.
  • To engage marginalised 15-29 year old NEETs in YEI areas and support them to re-engage with education or training, with a particular focus on 15-19 year olds.
  • To address the basic skills needs of 15-29 year old NEETs in YEI areas so that they can compete effectively in the labour market.
  • To provide additional work experience and pre-employment training opportunities to 15-29 year old NEETs in YEI areas, with a particular focus on those aged over 18.
  • To support 15-29 year old lone parents who are NEET in YEI areas to overcome the barriers they face in participating in the labour market (including childcare).[footnote 5]

The OP emphasises that support for NEET young people is already available through a variety of other provision, and that the YEI should be additional and complementary to existing measures – for example, through providing more intensive support[footnote 6].

The YEI in England is being delivered through 24 projects spread across the areas eligible for funding under the initiative. The projects are led by a mixture of public, private and voluntary and community sector (VCS) organisations, with these lead organisations typically working with a range of delivery partners to support young people. As anticipated in the OP, YEI projects deliver a range of support to young people with the aim of assisting them into employment, education or training. This includes providing access to training and qualifications, volunteering activities, support to access apprenticeships, ‘wraparound’ support aimed at building confidence and softer skills required for employment, enhanced careers advice and guidance, and working with employers to broker work placements and facilitate employment opportunities for young people.

Further detail on the support offer and delivery models adopted by YEI providers can be found in the previous process evaluation of the initiative[footnote 7].

1.2. Evaluation aims and objectives

The overall purpose of the evaluation was to provide an assessment of effectiveness, efficiency and impact in respect of the YEI. Within this, three main aims of the evaluation were specified as follows:

  • To assess the extent to which the YEI has achieved its objectives (Sustainable integration into the labour market of young people not in employment, education or training, including those at risk of social exclusion and young people from marginalised communities)
  • To evaluate how efficient the YEI has been (in terms of achieving its objectives at the minimum cost and without duplicating existing provision) and which elements of the programme were most cost-effective
  • To gather evidence on the impact of the YEI and the extent to which observed outcomes can be considered an effect of the programme.

More broadly, the evaluation also aimed to explore what works well or less well in supporting young people into quality and sustainable employment, education or training. Given the focus on assessing the YEI’s impact, the study also sought to identify the effect of the support given on generating the key intended results of the YEI, notably supporting individuals to enter employment, education or training, as well as on a series of ‘softer outcomes’. These include, for example, increased confidence, reduced barriers to engagement in work or learning, and improved wellbeing.

In line with the theory of change developed for the initiative in the context of the previous YEI process evaluation[footnote 8], the research also aimed to assess the YEI’s effect on some broader outcomes. These included outcomes for the organisations delivering the provision, effects on encouraging more joined-up services at local levels, and any impacts on local communities that could be discerned. A diagram summarising this theory of change is included at Appendix A for reference.

1.2.1. Research questions

The aims and focus of the evaluation outlined above are reflected in a series of research questions included in the specification for the study developed by DWP. These reflect the evaluation criteria of effectiveness, efficiency and impact as follows:

Effectiveness

  • How and to what extent did YEI achieve its objective of sustainable integration of young people into the labour market?
  • How and to what extent did YEI contribute to addressing the problem of NEETs?
  • Were YEI funds spent on those most in need of support? Were the specific target groups reached as planned?
  • Which types of interventions were the most effective, for which groups and in which contexts?
  • Which kinds of support were applied to which sub-groups (e.g. young people leaving education without a qualification, unemployed graduates etc.)? Were they relevant to the specific needs of the participants?
  • What does the evidence suggest about ‘what works’ to help disadvantaged groups move closer to the labour market?
  • What was the quality of the offers received by the participants? (collected through the ESF and YEI Leavers Survey)[footnote 9]
  • Were there any barriers to effective implementation of the YEI programme?
  • Which projects displayed best practice and how can this be shared?

Efficiency

  • What were the unit costs per type of operation and per target group?
  • Which types of operations were the most efficient and cost-effective?
  • What are the direct and indirect benefits and costs to projects? Individuals? Local stakeholders such as employers?

Impact

  • What is the impact of the YEI support for young unemployed people on their future employment chances? How big is the effect of the YEI support on entering the labour market? What would have been individuals’ employment status in the absence of the support?
  • What was the contribution of the YEI to changes in the youth employment/ unemployment/activity rates in the geographical areas covered by the YEI?
  • Which specific elements of the intervention contributed to the impact observed?
  • What were the unintended effects of the intervention (if any) and how significant were they?
  • Were there any structural impacts (new partnerships between public/private/voluntary sectors, changes in education system, vocational training system, Public Employment Services)?
  • Was YEI responsible for broader outcomes for individuals, including:
    • improved interpersonal and basic skills
    • improved attitudes and behaviours
    • reduced barriers to re-engaging in work and learning
    • improved access to apprenticeships and traineeships
    • improved understanding of support services available
    • enhanced access to and competitiveness in the labour market
    • improved health and wellbeing including more self-sufficiency and empowerment
    • greater financial security and earning power
  • Did YEI lead to wider outcomes such as:
    • Positive ‘knock-on effects’ on peer groups/families, such as improved attitudes or reduced barriers to re-engaging in work and learning
    • Organisational and policy learning – e.g. ‘what works’
    • Economic benefits to the Exchequer and local communities
    • Improved inter-generational awareness/understanding

1.3. Evaluation methodology

In summary, the mixed-method approach to the evaluation encompassed:

  • A desk-based review of YEI and related documentation
  • Secondary analysis of YEI MI data[footnote 10] and additional contextual statistical data (e.g. that available from the Office for National Statistics (ONS))
  • Secondary analysis of data from the ESF and YEI Leavers Survey
  • Six interviews with high level stakeholders, principally from the ESF Managing Authority (MA), to explore and understand the programme context
  • Primary, case-study based, qualitative research across 10 YEI delivery areas conducted in late 2017 (including c.10 interviews in each area with a range of stakeholders including YEI lead provider staff, YEI delivery partner staff, YEI participants, and European Structural and Investment Funds (ESIF) sub-committee representatives)
  • A CIE, comparing outcomes for the YEI treatment group to a statistically generated comparison group, using propensity score matching and difference-in-difference techniques drawing on YEI MI and administrative datasets.
  • A CBA estimating the return on investment offered by the YEI, informed by the results of the above CIE but undertaken in-house by DWP using existing cost-benefit analysis modelling tools.

Evidence informing this final report is drawn from all research strands outlined above, with evidence from each strand being synthesised to generate the overall findings and conclusions presented. The first evaluation criterion addressed, that of effectiveness, is primarily informed by the qualitative case study and interview research, supplemented by use of evidence from the YEI MI and Leavers Survey. Insights into efficiency were primarily generated by the case study research, with some reflections on the results of the CBA also informing the analysis. Findings concerning the impact of the YEI draw on the CIE and CBA in respect of employment impacts, with the YEI MI and Leavers Survey also informing this along with other impacts including those relating to entry into education and training. The qualitative case study research was also used to explore ‘softer’ impacts, such as those on participants’ confidence, as well as some of the broader outcomes of the programme, including those on providers delivering support.

1.3.1. Limitations to the study approach

The initial design for the CIE element of the evaluation aimed to use administrative datasets relating to education, firstly to construct a statistically similar comparison group against which to compare the impact of support for a ‘treatment’ group of YEI participants, and, secondly, to include education and training outcomes, as well as employment outcomes, in the estimates produced concerning the YEI’s impact. However, anticipated access to these datasets proved not to be possible in the evaluation timescale.

As a consequence, some adaptations to the CIE model and approach were required. Specifically, lack of access to education datasets restricted the CIE analysis to employment outcomes, rather than including education and training outcomes as was initially felt to be possible. Lack of available education data also limited the amount of variables, notably those relating to educational history and qualifications, available for the construction of a comparison group to compare against YEI participants. However, as explained in more detail in the technical annex accompanying this report, we believe the lack of these variables would have had a limited effect on the employment estimates generated.

It should also be noted that full data from the YEI MI and Leavers Survey, covering the whole delivery lifetime of the programme, is not yet available. The YEI is still being delivered and hence the MI available is restricted to validated data available at the point of drafting this report. Findings from the Leavers Survey, being conducted by IFF Research on behalf of DWP, are likewise based only on fieldwork waves completed and available at the time of reporting. It should be recognised that these figures are subject to change by the time the YEI programme is finished. However, both the MI and survey sources cover the majority of the programme delivery period. Additionally, the Leavers Survey is based on a relatively large sample size of 2,213 respondents, while the MI data captures 73,935 YEI participations.

Data availability and the final research design also affected how far it was possible to address a small number of the research questions detailed above in section 1.2.1. In particular, the questions below, developed in light of the European Commission’s guidance on YEI evaluation, were not possible to address fully in light of the MI being collected and available for analysis, along with the final evaluation design adopted. Brief explanations for this are included next to the questions concerned.

  • What were the unit costs per type of operation and per target group? (This was not possible to address due to this data not being collected at the project level, as explained in more detail in the efficiency section of this report).
  • What are the direct and indirect benefits and costs to projects? Individuals? Local stakeholders such as employers? (This question was developed before a decision was made not to undertake an enhanced CBA, the potential for which was briefly examined at the outset of the evaluation. Primarily for reasons of comparability with other programmes, it was seen as preferable to use the cost-benefit model available internally to DWP analysts; this does not deal with costs and benefits in a way that would be required to address this question, and data to inform such an assessment was not gathered).
  • What was the contribution of the YEI to changes in the youth employment/ unemployment/activity rates in the geographical areas covered by the YEI? (While it is possible to provide overall figures for YEI participants entering employment, presented in the report chapter on impact, it was recognised that using a CIE to estimate additional (net) employment impacts occasioned by the YEI was a more robust approach to estimating the programme’s contribution in this area. As covered in the technical annex to the report, sample sizes available for the YEI treatment group used in the CIE would not be sufficient to generate robust estimates of additional, or net, effects at the NUTS area levels within which the YEI was delivered).

1.4. Report structure

The remainder of the report is structured as follows:

  • Chapter two assesses the effectiveness of the YEI in terms of targeting and engaging participants, the delivery of support, and the quality of the employment and training offers received by participants.
  • Chapter three examines the efficiency of YEI delivery from the perspective of assessing the extent that inputs, financial or otherwise, translate to outputs and outcomes at the minimum possible cost.
  • Chapter four examines the impact of the programme, considering both ‘hard’ impacts on participants in terms of, for example, its employment effects, and ‘softer’ impacts on participants such as confidence. It also assesses a series of broader impacts, including those on providers delivering support, along with estimating the programme’s economic benefits.
  • Chapter five concludes the report by presenting some concluding reflections in the results of the evaluation, including revisiting the theory of change developed to help guide and structure the analysis.

A more detailed discussion of the methodological approach to the CIE element of the study is presented in a technical annex accompanying this report.

2. Effectiveness

Assessing the effectiveness of the YEI implies making judgements over whether, and the extent to which, the processes, mechanisms and activities put in place by YEI providers support the objectives of the initiative in terms of helping participants move closer or in to employment, education or training. In line with this focus, this chapter first considers how effective the YEI has been in targeting and engaging participants. It then examines the effectiveness of provision, both in terms of the types of support put in place and how effective they were in moving participants towards, or into, work or learning. Effectiveness is then considered from the perspective of available evidence on the quality of YEI employment and training offers.

2.1. Targeting of participants

Evidence from the sources reviewed for this report indicates that YEI delivery has been relatively effective to date in terms of targeting and engaging NEET young people, including those from specific sub-groups such as those facing labour market disadvantages. As detailed below, a review of programme MI, data from the ESF Leavers Survey, and evidence from the qualitative case study visits to ESF providers indicates that, while targeting and engaging participants has been challenging in some cases (including for some sub-groups), the programme as a whole has been broadly successful in engaging the numbers anticipated to date.

Analysis of programme MI available at September 2019 shows that of the 73,935 participants recorded as joining the programme thus far, two-fifths (40 per cent) are female and three-fifths (60 per cent) are male (Table 2.1). When compared to the targets in the refreshed ESF Operational programme (OP) published in October 2018, wherein the target for female participation is 43 per cent, this shows that the programme is broadly on target in terms of the anticipated gender split of participants. The figure of 73,935 participants engaged to date should be viewed in light of a lag in the submission and validation of programme MI. The real figure will thus be higher than this, though it does illustrate that the programme appears on course to meet the overall YEI participant engagement target until 2023 of 110,480[footnote 11].

Table 2.1 Participation to date by gender

Gender Number Percentage
Female 29,348 40%
Male 44,379 60%
Other/ Undisclosed 208 0%
All 73,935 100%

Source: YEI programme MI as at September 2019

Programme MI can also be used to illustrate the degree to which the YEI is engaging participants targeted by the ESF that experience some form of additional disadvantage (i.e. in addition to being NEET)[footnote 12]. As Table 2.2 shows, of the 73,935 participants joining the programme to date, almost three-quarters (74 per cent) were recorded as fitting one or more of the definitions of disadvantage used for the YEI. Self-reported additional disadvantage by survey respondents, including drugs and alcohol dependency and ex-offenders, was even higher (82 per cent). These findings suggest that YEI projects are successfully targeting and engaging NEET young people who face additional, and potentially multiple, disadvantages in respect of accessing the labour market.

Table 2.2 Participation to date: disadvantage / no disadvantage recorded

Recorded Number Percentage
No disadvantage recorded 19,358 26%
Disadvantage recorded 54,577 74%
All 73,935 100%

Source: YEI programme MI as at September 2019

Further insights around the effectiveness of the YEI in targeting those facing labour market disadvantage are available from the YEI Leavers Survey, along with the qualitative case study research undertaken in late 2017. The survey data reveals that, at the point of engagement with support, just over two-fifths (41 per cent) reported living in a household where nobody worked, while nearly one in four (23 per cent) reported having a physical or mental health condition expected to last twelve months or more. In addition, more than a quarter of respondents (27 per cent) reported being from an ethnic minority background[footnote 13].

This suggests that the programme is engaging considerable proportions of participants falling into several of the main categories of disadvantage targeted by the YEI, including those living in a jobless household, having a disability, or coming from an ethnic minority background.

While smaller proportions or respondents were from groups facing other labour market disadvantages, such as having childcare or caring responsibilities or being homeless, the survey data does indicate that such groups are also being engaged onto provision. Specifically, 14 per cent of respondents reported that they had childcare responsibilities when joining the provision[footnote 14], six per cent that they had caring responsibilities on joining, and three per cent reported they were either homeless or living in hostel accommodation when joining.

It is also worth noting in this context that the YEI Leavers Survey suggests that the programme is engaging and supporting a significant proportion of the long-term unemployed (defined as being out of work for six months or more). Of those respondents stating that they were unemployed on joining YEI provision, more than half (53 per cent) reported being out of work for more than six months or never having had a job. In turn, this indicates that the objective of targeting harder to reach young people, or those more marginalised and/or isolated in other terms, is likely to be being fulfilled.

Evidence from the case study visits provides some insights into the reasons behind the broadly effective picture of targeting and engagement suggested by the MI and survey data. Firstly, it was apparent that most lead YEI providers are using local data and intelligence to help with targeting, including reviewing data on an ongoing basis to track changes in NEET levels / youth unemployment. Other common approaches reported to aid targeting and engagement included use of delivery partners’ existing networks, focusing extensively on outreach activity, developing partnerships with Jobcentre Plus to encourage referrals, and engaging in co-location with other services. This latter approach is considered in more detail in the following section on YEI delivery models.

Referrals from Jobcentre Plus appeared to vary in terms of their prevalence across the areas visited, in part due to differences in levels of engagement on the part of local Jobcentre Plus offices. In some cases, local labour market conditions also potentially appeared to play a role. Some provider interviewees felt that falls in youth unemployment had reduced the numbers of Jobcentre Plus referrals, while increasing the necessity of outreach activity to engage those not in contact with such services.

In respect of such activity, having a high street presence was reported to have encouraged self-referrals, including parents bringing young people in. Such an approach, along with word of mouth from peers having positive experiences of provision, was reported as particularly important in engaging NEET young people who are more disengaged from mainstream services – those sometimes referred to as ‘hidden NEETs’. At the same time, several interviewees acknowledged that this group, along with economically inactive NEET young people and lone parents, has proved more challenging to identify and engage. As one YEI provider representative explained:

The issue has been around finding the young people who are economically inactive … We already recognise that they can be quite challenging to reach because they might not necessarily engage with the traditional means of targeting them… so essentially it requires a more innovative and, quite frankly, a tenacious approach. (YEI delivery staff representative)

In respect of lone parents, it was similarly noted by a provider representative that it

…takes time to build up trust and they have more complex barriers.” As discussed in section 2.2, co-location with services supporting (young) parents was seen as an effective response to challenges around engaging this sub-group.

Effectively targeting more ‘hidden’ NEETs was likewise cited as important in a context of falling youth unemployment; to ensure good engagement levels several projects reported having to focus more on this group than anticipated. In turn, several interviewees advanced the perspective that the YEI was engaging those further from the labour market in many cases, with participants from this group often requiring more intensive and longer periods of support. Specific challenges related to this were also cited: for example, accessing and sharing data on inactive young people and ‘hidden NEETs’. While seen as challenges to effective delivery, such issues were generally not viewed as notably compromising the effectiveness of targeting and engagement. Again, co-location to facilitate data and intelligence sharing was seen as a helpful solution in this context.

Finally, it is worth noting that identifying marginalised young people was often seen as less of a challenge than getting them to positively engage with provision and maintain that engagement, suggesting that this group might be ‘harder to support’ rather than ‘hard to reach’ or ‘hard to find’. Linked to this, the voluntary nature of the YEI was seen as promoting engagement amongst more marginalised young people, and encouraging that to be maintained, rather than hindering it. Several provider representatives positively contrasted the YEI with programmes mandating attendance from this perspective. Similarly, participants interviewed also welcomed the voluntary nature of the provision and felt that this had encouraged, rather than discouraged, them to engage with the programme and maintain this.

2.2. Enablers and barriers to YEI delivery

Part of assessing effectiveness involved considering which YEI delivery models, or elements therein, appeared to promote effective support to NEET young people, as well as any aspects that appeared to hinder this. There were few aspects to the design of delivery models that appeared to hinder effectiveness. Conversely, several aspects to delivery models emerged as key to YEI’s effectiveness in supporting those engaged. Evidence for this assessment was drawn from provider and participant perspectives gathered through the project visits, along with the results of the YEI Leavers Survey. Elements reported as key to supporting effective delivery were, in particular:

  • The co-location of YEI delivery with other related services supporting young people
  • The development of partnerships and referral routes to offer holistic support to young people and address their particular issues
  • The establishment of effective governance procedures, including mechanisms to engage and benefit from the input of Local Enterprise Partnership (LEP) representatives.

Co-location with other services was frequently referenced as having a number of benefits for effectiveness. As well as supporting the targeting of lone parents as noted above, co-location in children’s centres was cited as effective in supporting joint working with other services, facilitating data sharing and a ‘joined-up approach’, and encouraging and facilitating cross referrals of young people. More broadly, co-location in hubs providing a range of community services was cited as effective in several instances for similar reasons.

In the majority of cases, YEI projects utilised a ‘hub and spoke’ delivery model, enabling a wide support offer to be provided to participants across lead and partner delivery organisations. This approach was widely seen as effective in ensuring that young people could access additional and specialised support as required. Combined with the key worker role adopted by most projects, examined in more detail in the following section, this delivery model was reported as being effective in supplementing dedicated, case managed, support for participants (the ‘hub’) with referral to specialist support as required (the ‘spokes’).

Whether or not the additional and specialised support available was delivered by organisations within the formal YEI delivery partnership appeared less important than ensuring a range of support could be accessed. Thus, in instances where a lead provider alone formed the delivery model, such organisations were still able to refer to specialist support as required. It was noted, however, that including specialist providers within the formal delivery partnership offered the chance for lead providers to manage their delivery partners more closely.

2.3. Effectiveness of YEI provision

In addition to the effectiveness of YEI delivery models, the actual activities and forms of support offered to YEI participants also appear to have been largely effective. Both the case study visits to projects and findings from the YEI Leavers Survey provide a number of insights into the relative effectiveness of different activities and types of support, and the relevance of the support provided in terms of meeting the needs of participants and particular sub-groups within the overall YEI cohort. The range of support available, and the extent it was tailored to individual needs through an ‘action planning’ process, was cited by both YEI participants and provider staff as a key factor in its perceived effectiveness. Equally, the adoption of key worker models, providing dedicated, case managed support to participants, emerged as a consistent theme in discussions of effective types of support.

2.3.1. Effective types of support

Based on evidence gathered from provider staff and participants through the case study visits, many of the activities offered through the YEI appear to be effective. However, their relative importance or effectiveness was acknowledged by provider staff as being likely to vary according to the particular individual being supported, along with other contextual factors such as local labour market conditions. Forms of support most consistently cited as effective by staff tended to mirror those seen as most effective from a participant perspective. In particular, these included:

  • The adoption of ‘key worker’ models to provide consistency of support and an overview of case management for each participant.
  • ‘Wraparound’ support designed to address individuals’ personal and often deep-seated challenges and barriers to (re-)engaging with work and learning, in particular those relating to confidence, attitudes, and aspirations, along with physical or mental health conditions.
  • Short, sharp interventions (usually to address a small employment need/gap in a young person’s CV or qualifications), like a Construction Skills Certification Scheme (CSCS) qualification, which can lead to further enhanced qualifications.
  • English and mathematics provision (in terms of these being basic requirements for better jobs, and young people being motivated to get these qualifications in light of this, as evidenced by several interviews with participants)
  • Training linked to an employment route-way (for example, in colleges or sector-based work academies; or training that has been delivered with the support of an employer). It was noted that seeing an employment opportunity at the end of a qualification or training is an effective motivator, particularly for young people closer to the labour market.
  • Social or community-based activities and volunteering, cited as helping young people to regain/build up their feelings of engagement in the local community and helping the community to change their perceptions of NEET young people.

Amongst the above elements, the key worker role, typically used by projects to facilitate the ‘wraparound’ support delivered, was particularly widely referenced. Depth interviews with YEI participants conducted as part of the case study visits strongly indicated the importance of this aspect of support. Displaying trust and faith in participants as individuals, flexibility in responding to needs, offering support in a sensitive and compassionate way, and working at a pace suited to their needs were common themes expressed in discussing the role played by key workers. The comment of one young person sums up several of these aspects:

To say helpful is an understatement, because [key worker] has got me in touch with new contacts and she is always inspiring… She is approachable, compassionate and always interested. She’s the only one who truly has faith in what I can do. (YEI participant)

While also being seen as effective in many cases, other elements of the YEI offer were seen as more nuanced in their effects and/or as challenging to successfully and consistently offer. Work experience and work placements fall into this category. When suited to participants, and a good match with aspirations achieved, it was apparent that such support could be highly effective in building confidence, adding to individuals’ CVs, making participants believe that they could find work, and, in a number of cases discussed on the case study visits, being key in participants successfully entering employment.

However, in a small number of cases participants’ experience of placements had been less positive, generally due to the placement being different from expectations and/or the individuals concerned struggling with the work environment for a number of reasons. Equally, from a staff perspective, encouraging employers to provide tasters and work experience was seen as challenging in some cases, due to a perceived lack of incentives to facilitate such activities. Despite this, it was apparent that some YEI providers were particularly effective in securing opportunities through, for example, dedicated employer engagement teams. Equally, just over half of YEI Leavers Survey respondents reported that their programme provided a work placement or work experience (51 per cent), suggesting that this form of support was relatively widely available.

Again, while effective in many cases, support to access traineeships and apprenticeships was another area where there was some variation evident in the relative effectiveness of support. Limited numbers of suitable apprenticeships in some contexts was referenced by provider staff as a challenge. Equally, there was a perception from both provider staff and some participants that, in many cases, apprenticeships are more beneficial for the younger age range within the YEI cohort rather than for some older people. For example, it was noted that the latter can have family responsibilities; in this and some other contexts apprenticeships were seen as not paying enough. Conversely, where suitable apprenticeships were available, and young people were interested, a number of examples were given about how young people had been supported and given the confidence to access them, as well as support being provided to identify suitable opportunities in the first instance.

2.3.2. Effectiveness in meeting participant needs

There was a clear consensus amongst all stakeholders interviewed as part of the case studies, including participants, that the range and flexibility of provision was a particular feature of the ‘YEI offer’. These aspects were seen as ensuring that the provision could effectively meet participants’ varied support needs. Linked to the previous discussion of delivery models, tailoring support to individual needs in a flexible way, rather than putting young people through a set support pathway or course, was commonly highlighted as important. Several participants positively contrasted this approach with previous, more structured and less flexible, employability support programmes they had experienced.

Both provider staff and participants also felt that the tailored and flexible approach evident was likely to contribute to the sustainability of any outcomes achieved. As one participant discussed, the support provided by a key worker had enabled him to focus on thinking in terms of a career, rather than just taking the type of low-paid and precarious employment he had previously been used to. Addressing more deep-seated issues putting young people at a labour market disadvantage was also referenced. Many interviewees cited the importance of addressing confidence issues, anxiety, lack of aspirations, and mental or physical health barriers as ‘first order priorities’, before moving onto more employability or learning-focused support. Direct evidence from participants, and examples given by provider staff, suggest such an approach is important, effective, and likely to generate more sustainable effects.

In terms of flexibility, the opportunity to combine elements of financial support with intensive confidence building, and other support focused on breaking down personal barriers to work and training, was highlighted in several contexts. For example, the prevalence of a reluctance to travel for work and training amongst those more isolated and marginalised was frequently discussed as a significant barrier. The ability to offer support to address this, combining financial assistance, for example to buy travel passes, with more ‘hands on’ support, in terms of accompanying young people the first time they travelled a particular route, was seen as both important and effective in meeting the needs of many young people.

Particularly innovative provision was also evident in several cases, with the driver for such innovation again linked by provider staff to the desire to meet young people’s needs, in particular through providing support that was engaging and likely to encourage on-going contact. Examples included training to restore and repair bicycles that young people then kept, training to produce items involving 3-D printing, and using sport as a lever for engagement and as a hook to get young people to engage with employability skills without necessarily recognising it as such.

The apparent effectiveness of the YEI in meeting participant needs appears to be confirmed by the results of the YEI Leavers Survey. Amongst respondents, 89 per cent were satisfied with the guidance and information about the support they would receive and 88 per cent were similarly satisfied with the guidance and feedback received during support, with these elements typically being a core part of the key worker role. The overall satisfaction rate with the support similarly reinforces the impression of effective delivery. Just under half (47 per cent) were ‘very satisfied’ overall whilst a further 38 per cent reported being ‘fairly satisfied’. Less than 1 in 20 respondents were ‘fairly dissatisfied’ (3 per cent) or ‘very dissatisfied’ (3 per cent) with the support received.

Combined with the case study evidence this presents a persuasive case that the nature and delivery of YEI support is, in most cases, effective in addressing the issues and challenges young people face.

The following case study, based on a depth interview with a YEI participant[footnote 15], helps illustrate the range of support provided, the significance of the flexible and tailored approach outlined, and some of the effects this had.

Case study – YEI provision (Gareth)

Gareth left college as he was not enjoying his courses and was signposted to the YEI through a family contact. The idea of adding to his CV was appealing, although he had few expectations. Once Gareth started, he really enjoyed the opportunity to meet new people and to establish new social connections. Gareth was introduced to his 1-1 key worker who helped him with finding an apprenticeship. He spoke of the individual support he received:

It was all done from the point of view of how they can help us. I had an interview with my coach [key worker]. We came up with a plan for how we could move forward towards our goals. It was all based on what we wanted.

The support offer from the YEI included: CV writing support, help with job searches, applications and gaining work placements, and working on the YEI provider’s reception. He was also later supported by his key worker to research, access and ultimately begin an apprenticeship. Gareth noted how the YEI had exceeded his expectations and hopes:

It’s been way better than I expected, I have learned lots of new things… the employability support will help me the most in the future.

For Gareth, the YEI support differed from a number of other employability programmes he had accessed in the past. He said that the YEI had “put him on track” and that “nobody else was able to do that”. Additionally, it was evident that there were softer outcomes that were generated for this participant. In terms of improvements in confidence, overcoming personal challenges and improving wellbeing:

It’s built my confidence, when I first came I was the shyest person – I had anxiety. This has really helped me, my mental health is much better now.

2.3.3. Less effective aspects of support

While the evidence suggests the majority of YEI provision is generally effective, and meets young people’s needs, some forms of provision appeared to be less effective in some instances. More structured provision in a classroom setting was cited as discouraging the engagement of young people in some cases, potentially due to prior negative experiences at school, with this being reflected in the views of some participants interviewed as part of the case study visits. Partly to address this, some providers had experimented with different settings and with combining classroom settings to discuss employability skills with other activities, including sports.

In a minority of cases evident in the context of the case study visits, there was also some evidence of providers ‘holding on’ to young people they had engaged, when provision elsewhere in a partnership or progressing young people onto other support would have been more beneficial. This was reported by lead partners as a problem in a limited number of instances, despite efforts to mitigate it through messages to the partnership and the manner in which contracts and memoranda of understanding detailing expectations had been designed.

2.4. The quality of YEI employment and training offers

In line with the evaluation guidance for the YEI published by the European Commission[footnote 16], a further aspect to assessing effectiveness involves the quality of YEI employment and training offers received by participants. While individual examples of participants accessing what were reported as good quality jobs and apprenticeships were revealed through the case study visits, the best evidence to address effectiveness in this area comes from the YEI Leavers Survey. Overall, this suggests a positive picture in terms of the quality of training offers received and, where applicable, forms of learning that YEI participants were engaged in six-months after leaving. Similarly, most evidence from the Leavers Survey points to a broadly positive picture in terms of the quality of jobs/job-offers received by participants.

Looking at the evidence in more detail, of those respondents to the survey who reported having done a traineeship as part of their YEI provision (19 per cent), 82 per cent reported that the duration of their traineeship was ‘about right’ with only one in ten (10 per cent) feeling that it was too short. Perhaps more importantly in terms of assessing quality, 9 in 10 (89 per cent) felt that their traineeship would improve their chances of getting a job; of this group over half felt that their chances had improved to a ‘large extent’ (53 per cent) and a further 36 per cent felt their chances had improved to ‘a little extent’.

In terms of satisfaction, over half of the respondents accessing a traineeship (54 per cent) were ‘very satisfied’ with it overall in terms of the work experience gained and how they have benefitted since. A further 35 per cent reported being ‘fairly satisfied’, with a minority (7 per cent) being either ‘fairly’ or ‘very’ unsatisfied. Combined with the perceived role of traineeships in supporting them to find work, these high satisfaction levels from participants suggest a positive view of the quality of the traineeships available through the YEI. Data available in respect of those in education or training six months after leaving also appears positive in the sense that, amongst respondents in this position, over four-fifths (83 per cent) reported that this would lead to a nationally recognised qualification.

While the data relating to employment available through the YEI Leavers Survey relates to those respondents in work at the six-month point after leaving provision, and hence respondents may have moved jobs or moved into work after leaving YEI, the findings available can nonetheless be used to draw some conclusions as to the quality of jobs that YEI support helped lead to (either immediately on leaving or soon after). Of the survey respondents in work at the six-month point after leaving (45 per cent of total respondents), half (50 per cent) were on a permanent employment contract and a further 14 per cent on a contract lasting 12 months or more. This suggests the majority of jobs gained by YEI leavers were fairly stable and long term.

However, just under a quarter (24 per cent) of those in work at the six-month point were either on temporary or zero-hours contracts, demonstrating less stable forms of work for a notable minority of YEI leavers. This should, though, be considered in the context of part-time, temporary, and zero hours working in the UK having increased relative to permanent employment in the last few years. In this context, the survey finding that just over seven out of ten participants in work at the six-month point were in full-time jobs (70 per cent) can also be viewed as relatively positive from a job-quality standpoint. Equally, it should be noted that approaching half (46 percent) amongst the 29 per cent in part-time work did not wish to work full-time.

Further positive evidence in terms of job quality can be gained from the nearly six in ten (58 per cent) YEI leavers responding to the survey that had received a job offer between starting on the provision and six months after leaving. Of this group, almost two-thirds (62 per cent) rated the quality of job offers received as either ‘very good’ (27 per cent) or ‘good’ (35 per cent). While a further 26 per cent rated their offers as ‘reasonable’, fewer than one in ten (9 per cent) rated their offers as either ‘poor’ or very poor’. Allied to the above findings on the nature of the jobs accessed, this appears to confirm the impression of generally good quality jobs/job-offers resulting from engagement with the programme.

3. Efficiency

Assessing the YEI’s efficiency requires consideration of the degree to which inputs, financial or otherwise, translate to outputs and outcomes at the minimum possible cost. The extent to which duplication of activity with other provision is also relevant from this perspective. This implies a focus on examining how, and the extent to which, YEI providers have sought to minimise costs and provide value-for-money. In assessing efficiency, this chapter thus examines evidence on provider approaches to value-for-money, along with considering insights that can be gained around the relative efficiency of particular activities and the factors that help or hinder the efficient delivery of support.

3.1. Approaches to ensuring value for money

The impression from the visits to YEI providers was that organisations delivering the YEI are concerned with ensuring efficiency and hence ultimately offering value-for-money. It was noted on several occasions that doing so is part of the general ethos of the organisations concerned, and in their own interest. From this perspective, representatives cited that failing to operate efficiently and/or offer value will result in winning less employability and training contracts, losing money on delivery, and ultimately compromising their organisations’ viability. Most provider representatives were thus able to offer concrete examples of where they had sought to ensure value-for-money through their delivery approach.

While a range of such examples were offered, a number of common themes and groups of related examples emerged. These can be summarised as follows:

  • Focusing on reducing internal costs, including:
    • Close control of staffing numbers and ensuring appropriate caseloads
    • Focusing on staff progression, retention and training to reduce recruitment costs and ensure that staff are delivering as efficiently and effectively as possible
    • Reducing transaction costs between delivery partners by developing effective systems of referrals, data sharing, IT solutions, and communication
    • Reducing overheads such as accommodation, venue hire costs, and electricity costs where possible
    • Reducing costs through re-use and recycling, for example of materials used in activities, with some interviewees linking this to their focus on sustainability as an ESF cross-cutting theme
  • Seeking to ensure value in terms of the activities offered, for example by tapping into and/or expanding existing provision to ensure cost-effectiveness, or using account holder discounts when purchasing provision and courses
  • Designing activities to ensure value-for-money, including careful consideration of where group work would be an appropriate and efficient solution for example
  • Avoiding any overlaps or duplication with other provision locally so as to avoid the potential for the available funds to be wasted
  • Lead YEI providers focusing on ensuring value amongst delivery partners, through provision of guidance, development of memoranda of understanding, oversight by finance officers, and, for example, requiring partners to consider and justify particular expenditure, such as that related to supporting transport costs amongst participants.

While typically offering several examples of how value-for-money is promoted, it was also common for provider representatives to reflect on the nature of the YEI target group and the level of support often needed in this context. In particular, interviewees commonly argued that the individualised, wraparound support frequently provided necessarily has a cost attached, and that meeting complex needs can, in some cases, be relatively expensive, even while their delivery overall strives for value-for-money. In some instances, the point was made that delivering what participants need is the core focus, and a strength, of the programme, rather than having to ensure the lowest possible delivery cost in all cases. As one interviewee commented from this perspective, “…this way of working takes longer but the results do come”, while a lead provider representative noted:

…V-f-M is fundamental. We always test the value, ask providers if they can do things at a better price … But we want quality so [there is] a balance to be struck about value and quality. (YEI lead provider representative)

3.2. Efficiency of activities

While the case study visits offered a broadly positive impression of efficiency, there are some limitations to precisely assessing the efficiency of specific types of provision and activities. This relates principally to a lack of data being collected that would facilitate such an assessment, with providers not generally identifying costs and throughput of individual activities (in terms of numbers participating) in this way. In addition, it was evident that the inter-related nature of many YEI activities naturally makes such data gathering and assessment problematic.

Reflecting this, providers generally struggled to identify which specific activities were most beneficial from a cost-effectiveness or efficiency perspective, in part because the nature of the holistic, wraparound support typically offered by the YEI meant it was hard to disentangle individual elements to consider their cost-effectiveness. In addition, a lack of hard data or evidence on which to, reliably, base such an assessment was also frequently referenced.

Where interviewees offered views on the relative cost effectiveness of provision, these tended to relate to those activities seen as having the most effect on the young people supported. Therefore, work around personal development, such as building confidence, was seen as intensive though ultimately cost-effective when balanced against the positive effects this was often seen as having for young people. As noted in the previous chapter, such work was seen as important in generating outcomes in terms of young peoples’ confidence, aspirations, and attitudes, but also in terms of its key role in progressing participants towards work or learning outcomes, and opening the way to more of a direct focus on achieving these.

Short courses leading to required qualifications were also cited from this perspective, in that a relatively small outlay could fill a gap in a participant’s CV and unlock opportunities. The perceived cost effectiveness of such short, sharp interventions was thus stressed in several cases.

In addition, several interviewees made the point that most YEI support could be cost effective in terms of costs avoided. A number of specific examples were cited of individuals securing access to apprenticeships, traineeships or being supported into work who otherwise might engage in criminality or impose other societal costs based on their previous behaviour. Ex-offenders were seen as a particularly relevant client group from this perspective, though it was noted that a number of participants had histories of more low-level anti-social behaviour.

Similarly, the perceived cost-effectiveness of activities was often related to the avoidance of any duplication of provision. In several cases, provider representatives described how they had undertaken mapping of local provision targeting young people in the course of designing their YEI offer, so as to complement and not duplicate this. The stress on avoiding duplication in the YEI guidance, and throughout the bid process, was also seen as supporting this end. Most provider representatives had a high degree of confidence, therefore, that duplication of provision with other non-YEI support for young people was being minimised.

The view of provider representatives was almost universal in terms of arguing that there were not really any forms of provision or specific activities that were inefficient per se. It was noted in a number of cases that delivery organisations are typically very experienced at offering the type of support provided through the YEI; therefore, the perspective was that any activities that did not offer efficiency and value in the past have been discontinued. The point made above concerning the view that the nature of the support, and responding to specific needs, implies a certain level of cost, was also made in this context. From this perspective, while ensuring efficiency and value was seen as important, addressing the particular needs of participants, within reason, was viewed as even more so.

While only limited indications of the relative efficiency of activities can be derived from the available evidence, more specific findings can be offered as to aspects promoting or hindering efficiency. These are covered in the following sub-sections.

3.2.1. Aspects promoting efficiency

Several providers cited a focus on unit costs in developing their provision as a factor supporting efficiency. This was noted as resulting from DWP as the ESF MA requesting details of such costs in the application process. This was felt to have encouraged, as one interviewee put it, a ‘sharp focus’ on costs and hence to promote efficient delivery. Partnership working and effective governance was likewise a key theme in this area. This was seen as aiding efficiency from several perspectives: in terms of ensuring that the particular skills and capacities of organisations within the delivery partnership were efficiently and effectively deployed, that efficient mechanisms could be established – for example in terms of cross referral – and that performance could be managed by the lead partner to promote efficiency. Finally, as noted above, organisational experience and commitments to delivering efficiently were, in themselves, seen as promoting efficiency within YEI delivery.

3.2.2. Challenges to efficiency

While the overall impression was that YEI support was being delivered efficiently, provider representatives and other stakeholders did acknowledge that some elements of the programme posed challenges to this. Although probably only accounting for only a small part of YEI projects’ overheads, interviewees cited revisions in guidance, the data portal, and levels of evidence requirements needed for the Managing Authority’s financial assurance processes as a burden for delivery.

In several cases, while not a challenge to efficiency directly, providers also noted that anticipated unit costs for provision may have been under-estimated in light of the nature of the client group being supported. Partly due to improving labour market conditions, the perception of some provider staff was that the YEI is supporting a group of young people more distant from the labour market than was maybe expected. In particular, a series of complex, multiple barriers to engagement in work or learning, and high levels of mental health concerns, were reported to be common features of the client group supported. This was linked to the view noted earlier that the individual, often complex needs, of YEI participants necessarily implies dedicated support that can be expensive. The perspective was that such support, while it may appear less efficient in terms of throughput of participant numbers against costs, is nonetheless required.

Finally, while the general perception of case study interviewees was that the YEI does not duplicate other provision being delivered locally, there were isolated examples offered of some scope for duplication within the programme itself. Some provider representatives acknowledged that this has happened on occasion, including young people being signed up for similar provision by two different providers due to participants not being clear at the engagement stage that they had been, or were, already receiving support. While acknowledging that it did happen, this was not seen as common, however, and therefore not a major issue or challenge.

4. Impact

Assessing the impact of the YEI depends on combining several evaluation elements and evidence sources. This chapter first presents the results of the counterfactual impact evaluation (CIE) strand of the study, central to estimating the additional (or ‘net’) employment effects of YEI support. The chapter then considers supplementary evidence on employment impacts, including that drawn from YEI management information (MI) and the YEI Leavers Survey, along with qualitative case study research, with these sources also being used to assess education and training outcomes. Evidence concerning the range of softer impacts identified in the YEI theory of change underpinning this evaluation is then considered, along with broader outcomes beyond the immediate beneficiaries of support. These aspects draw principally on the case study evidence, along with insights from the Leavers Survey and secondary datasets.

4.1. Employment impacts of the YEI: estimates from the CIE

4.1.1. Introducing the CIE

Impact evaluation typically aims to compare the outcomes of an intervention against the outcomes that would have been achieved in the absence of that intervention – often referred to as the ‘counterfactual’. CIE approaches aim to estimate the causal effect of a specific intervention on an observed outcome, addressing the following main questions:

1. Has a change occurred? 2. Was the intervention under assessment responsible for this change?

The CIE approach adopted for the YEI impact evaluation relies on identifying a ‘counterfactual’ through which the programme’s impact can be estimated. In common with many impact evaluation designs, the approach adopted here involves comparing outcomes for a set of YEI participants with a similar ‘comparison group’ not subject to YEI support. In this case, available data on pre-intervention employment and benefit claim histories, along with background characteristics (encompassing a range of specific variables)[footnote 17], were used to create similar treatment (YEI) and comparison (non-YEI participant) groups.

In line with the YEI’s focus as an employability support intervention, the programme’s impact on time spent in work relative to being unemployed or economically inactive was the main outcome examined through the CIE, complemented by time off benefits[footnote 18]. However, as with all such programmes, these employment related outcomes could have been affected by factors other than the intervention. Impacts on employment can be driven by the selection of particular types of people into a programme, or have nothing to do with programme participation but be driven mainly by contextual factors. Such factors can include, for example, an individual’s distance from the labour market and local employment opportunities. The CIE undertaken aimed to exclude such alternative explanations where possible to estimate the ‘additional’ (or net) impact of the YEI.

4.1.2. Methodology

This section provides an overview of the methodology used to conduct the CIE. Firstly, the likelihood of allocation to treatment (YEI participation) is discussed; this forms the foundation of the propensity score matching (PSM)[footnote 19] technique used to construct matched treatment and comparison groups for the CIE. We then outline the approach taken to ensuring the likelihood of receiving treatment amongst the comparison group was as close as possible to the YEI (treatment) group. This involved using PSM to undertake statistical matching between a YEI treatment group, derived from YEI programme records, and a comparison group constructed from available administrative datasets, as outlined further below.

Additional detail on the methodological approach adopted can be found in the technical annex accompanying this report.

Allocation to treatment

To be eligible for YEI support an individual is required to be not currently in employment, education or training (i.e. NEET). This can include those claiming out of work benefits (e.g. Jobseekers’ Allowance (JSA) or Employment and Support Allowance (ESA)), as well as those not claiming benefits. Further to this criterion, the YEI targets young people aged 15 to 29.

The above eligibility criteria, alongside the following factors that may affect the outcomes of interest, were used in constructing the CIE model:

  1. Sex of the individual
  2. Levels of local socio-economic disadvantage (proxied with the Index of Multiple Deprivation (IMD)) where the individual resides
  3. Wider economic opportunities available to the individual (proxied with those English regions where the YEI is delivered)[footnote 20]
  4. Claiming Universal Credit (UC) and/or Income Support
  5. Whether the individual has a disability and/or caring responsibilities
  6. The date the individual was supported by YEI, to reflect labour market opportunities at the period in question.

Data sources

The key data sources used in constructing the CIE model were benefits data (DWP) and employment data (HMRC). A total of 3,276 YEI participants[footnote 21] were matched to DWP’s Customer Information System (CIS), which subsequently enabled linking to other administrative datasets. To ensure a reliable comparison group could be formed, the following samples were drawn from CIS:

  • A random sample of 50,000 from the population aged 15-29. This was to ensure a pool from which similar individuals to the unknown or ‘hidden’ NEETs (i.e. not claiming out of work benefits) in the treatment group could be identified.
  • Recognising a proportion of the treatment group were known NEETs (i.e. claiming out of work benefits), 50,000 individuals aged 15-29, claiming either JSA or ESA at any time from the 12 months prior to the YEI start, were also drawn as a random sample from the DWP CIS.

The above 100,000 records formed a ‘comparator pool’ from which individuals similar to those benefitting from the YEI could be identified, so as to form the comparison group.

Data on educational history and qualifications were not available. As a result, education and training impacts could not be examined in the analysis as outcomes of interest. Equally, variables relating to educational history could not be used to inform the development of a comparison group. This led to an adaptation of the CIE design, specifically an increased focus on selecting an analysis cohort prior to matching, as detailed below.

Selecting the analysis cohort

To ensure reliable pre- and post- YEI comparisons of the impact of YEI on supporting individuals into work could be made, it was necessary to:

1. Define an analysis cohort (both YEI treatment and comparison groups) who, as far as possible, could be confirmed as being actively engaged in / available for the labour market. 2. Account for the fact that labour market conditions, and thus the likelihood of positive employment outcomes, vary over time.

For the YEI treatment group, employment and benefit histories prior to programme engagement were used to indicate availability for work[footnote 22], while the distribution of start dates on the programme was used to determine patterns of engagement over the period the YEI has been running (hence allowing for a reliable comparison over time in light of changing labour market conditions).

For the comparison group, pseudo start dates were applied in a way that mirrored the distribution of entry to YEI support amongst the treatment group, hence enabling the model to account for changes in labour market conditions[footnote 23], with employment and benefit histories prior to these pseudo start dates being used in the same way as for the treatment group in terms of indicating availability for work. Pseudo leaving dates were also calculated for the comparison group, based on the mean average duration of YEI interventions amongst those YEI participants who formed the treatment group for the CIE model and analysis (88 days). This enabled analysis of outcomes after leaving the intervention for both the treatment and comparison group.

Based on the data available and programme design, employment and benefit claiming activity pre- YEI was determined to be a reliable way of identifying active engagement in, and availability for, the labour market for the reasons noted above. To mitigate the risk of assessing employment outcomes for those not actively seeking employment as far as possible, within the available data, the following rules/criteria were developed to select an analysis cohort:

1. In the 6 to 12 months prior to the intervention the individual had an employment spell of at least 30 days (indicating they were engaged with labour market and likely to be available for work in the 6 months immediately before the intervention). AND, in the 6 months immediately before the intervention, there was a gap in employment spells of at least 30 days (indicating they were out of work for a period and eligible for YEI support, or in the case of the comparison group theoretically eligible for support)[footnote 24].

2. In the 6 to 12 months prior to the intervention, the individual had an out of work benefit spell of at least 30 days (indicating they were likely to be looking for work and be available for work in the 6 months immediately before the intervention). AND, in the 6 months immediately before the intervention, there was also an out of work benefit spell of at least 30 days (indicating they were out of work and eligible for YEI support, or in the case of the comparison group theoretically eligible for support)[footnote 24].

If an individual in the treatment group or comparator pool met any of these criteria, they were retained for analysis. Following data cleaning, 20,516 formed the analysis cohort, of which 1,050 were in the treatment group.

Matching

Using variables associated with the aforementioned factors affecting allocation to treatment, matching was undertaken using PSM to construct the final treatment and comparison groups through which to estimate the YEI’s impact on employment. The process of constructing balanced treatment and comparison groups for analysis resulted in a final treatment group size of 960 and comparison group size of 2,677. Table 4.1 demonstrates how, on key matching variables, the process of matching resulted in reductions in differences between the treatment and comparison groups in the analysis cohort of 20,516 and the final treatment and comparison groups used to estimate YEI’s additionality in respect of employment effects.

The matching was thus successful in constructing a comparison group of non-YEI individuals with similar characteristics to the treatment group supported by YEI.

Table 4.1: Background characteristics for treatment and comparator group, before and after matching

Group: Comparison/Treated Before matching Comparison group Before matching Treatment group After matching Comparison group After matching Treatment group
Sample size 19,466 1,050 2,677 960
Percentage male 57% 64% 62% 64%
Percentage female 43% 36% 38% 36%
Avg. age 21.6 22.8 22.7 22.6
Avg. JSA days (pre-YEI) 237 467 448 456
Avg. ESA days (pre-YEI) 180 162 153 162
Avg. UC days (pre-YEI) 18 68 59 62
Avg. days employed (pre-YEI) 311 325 314 315

4.1.3. Impact estimates

Impact on days in employment

Table 4.2 provides results from the impact analysis concerning the effect of YEI treatment on days in employment. This includes the average days in employment in the 6 months prior to YEI, 6 months after YEI and 6-12 months after YEI for both treatment and comparison groups. Results from difference in differences analysis[footnote 25] are provided for the treatment groups (full treatment and comparison groups, and subsets of these based on those known to be claiming JSA and ESA prior to YEI). These results represent the additional or net impact of the YEI programme.

All impact estimates were positive and statistically significant. In the 12 months after treatment, there was an average impact of 56 additional days spent in work for those benefitting from YEI support relative to the comparison group. The impact for individuals known to be claiming JSA and ESA pre-YEI was 48 days and 37 days respectively. Considering individuals in these groups, typically, had fewer days in employment prior to YEI (indicating they were further away from the labour market) this should be interpreted as a positive result.

Table 4.2: Impact of the YEI on days in employment

Group: Comparison/Treated All Comparison group All Treatment group JSA Comparison group JSA Treatment group ESA Comparison group ESA Treatment group
Sample size 2677 960 725 260 415 142
Avg. days employed before YEI (6 months) 41.6 41.6 25.0 14.3 8.8 4.6
Avg. days employed after YEI (6 months) 55.4 86.9 47.3 64.8 16.8 35.1
Avg. days employed after YEI (6-12 months) 52.6 77.3 50.7 60.1 16.4 26.1
Avg. days employed after YEI (total over 12 months) 98.0 164.2 98.0 124.9 33.2 61.2
DiD estimate (6 months)   31.5   28.1   22.6
DiD estimate (6-12 months)   24.7   20.0 14.0  
DiD estimate (total)   56.1   48.1   36.6

Impact on benefit claiming

Table 4.3 shows the impact results on outcomes relating to benefit claiming. This includes the average number of days claiming JSA, ESA and UC in the 6 months prior to YEI and 6 months after YEI. Results from difference in differences analysis are also provided for the treatment group, which can be interpreted as the impact of the YEI programme. While there appeared to be a slight reduction, relative to the comparison group, for days claiming JSA and ESA as a result of YEI support, this was not statistically significant. Furthermore, there was a statistically significant increase in the average number of days the treatment (YEI) group claimed UC after participating in YEI.

It is possible that the UC result represents an uneven pattern in the roll out of this benefit between treatment and comparison groups, and/or the fact that it also represents a benefit that can be claimed while in work with some of those exiting into work claiming UC. It is therefore unreliable to use this as a measure of the YEI’s effect on benefit claiming. However, drawing on the JSA and ESA estimates, while the additional positive impact on employment as a result of YEI support is clear, there appears not to be an accompanying reduction in likelihood of claiming benefits as a result of the intervention.

Table 4.3: Impact of the YEI on benefit claiming

Group: Comparison/Treated Comparison group Treatment group
Sample size 2677 960
avg. days JSA before YEI 49.4 50.8
avg. days JSA after YEI (6 months) 28.6 27.2
DiD estimate (JSA)   -2.8
avg. days ESA before YEI 25.5 26.0
avg. days ESA after YEI (6 months) 24.3 20.6
DiD estimate (ESA)   -4.2
avg. days UC before YEI 30.8 30.3
avg. days UC after YEI (6 months) 41.9 62.4
DiD estimate (UC)   21.0

4.2. Supplementary evidence on employment and learning impacts

The YEI MI available at the time of reporting[footnote 26] provides a range of additional and complementary evidence on the employment and learning results generated for participants. Of the 73,935 YEI participants recorded in the MI, 52,993 records are available for those exiting the provision. As table 4.4 shows, just under 60 per cent of these relate to male participants leaving YEI support and just over 40 per cent female participants.

Table 4.4 YEI leavers to date (Programme MI)

Gender All leaving provision (n) All leaving provision (%)
Male 31550 59.5%
Female 21269 40.1%
Other/ Undisclosed 174 0.3%
Total 52993 100%

Source: YEI programme MI as at September 2019 (N.B. figures may not sum due to rounding)

As table 4.5 shows, MI data indicates that just under three in ten of all participants are employed on leaving provision (29 per cent), with a slightly higher number unemployed (45 per cent) and just over a quarter (26 per cent) economically inactive[footnote 27]. As the data also illustrates, this conversion rate of just under three in ten participants going into work on leaving YEI support is very similar irrespective of gender.

Table 4.5 YEI participant employment status on leaving by gender

Gender Employed on leaving Inactive on leaving Unemployed on leaving
Male 29% 25% 46%
Female 29% 27% 44%
Other/ Undisclosed 25% 11% 64%
All 29% 26% 45%

Source: YEI programme MI as at September 2019

Taking into account the insights into the nature of the YEI client group gained from the MI and Leavers Survey (see the earlier section 2.1 on those engaged), which suggest a group facing labour market disadvantage and often multiple barriers to work, such a conversion rate appears positive. It is also worth noting that the case study interviews with YEI participants similarly served to highlight the complex and often entrenched issues and barriers to work faced by those engaged by the YEI.

Linked to this discussion of employment results in the context of the nature of the YEI client group, it is significant that, while employment rates by gender are very similar, there are some notable differences in outcomes when disadvantaged status is considered[footnote 28]. Almost three-quarters (74 per cent) of the 73,935 participants recorded as starting YEI provision had a declared disadvantage, with the remainder having no disadvantage declared. Table 4.6 indicates that employment rates amongst the 52,993 participants recorded as leaving provision are notably higher amongst those with no disadvantage recorded (37 per cent) compared to those with a disadvantage (26 per cent). Given the prevalence of disadvantage amongst the YEI cohort, this is indicative of the challenge faced by the programme in terms of successfully supporting individuals into work who face significant, and often multiple, barriers to labour market engagement.

Table 4.6 YEI participant employment on leaving by disadvantaged status

Status Employed on leaving Inactive on leaving Unemployed on leaving
Disadvantaged declared 26% 27% 47%
No disadvantage declared 37% 24% 40%
Total 29% 26% 45%

Source: YEI programme MI as at September 2019

When considering employment outcomes by age amongst programme leavers, as table 4.7 shows there is no difference between those under 25 and those in the 25-29 age group in terms of the percentage employed on leaving (29 per cent). However, the 25-29 age group were more likely to move into unemployment on leaving (57 per cent of leavers) relative to those under 25 (43 per cent). Conversely, having a status of inactive on leaving was more common amongst the under 25s (29 per cent) than the 25-19 age group (13 per cent). This may be due to those at the younger age end of the cohort being more likely to move into education.

Table 4.7 YEI participant employment status on leaving by age group

Age Employed on leaving Inactive on leaving Unemployed on leaving
Under 25 29% 29% 43%
25-29 29% 13% 57%
Total 29% 26% 45%

Source: YEI programme MI as at September 2019

In terms of learning results from the MI, relative to employment outcomes being recorded on leaving there were fewer YEI participants moving into education or training on exiting the programme. Analysis indicates that just under one in five (19 per cent) were recorded as being in education or training on leaving, there being only minor fluctuations in this rate according to gender, disadvantaged status, or age group. The MI data also records the numbers leaving provision with a qualification, with just under 1 in 10 participants (8 per cent) being in this category.

As the above analysis shows, the MI data on programme leavers presents a broadly positive picture of employment outcomes in light of the nature of the cohort (a just under one in three conversion rate) and learning outcomes, with just under one in five going into education or training. Figures from the YEI Leavers Survey confirm this impression, in some cases suggesting results that are even more positive, with this difference in part relating to the fact that participants were interviewed six months after leaving rather than immediately leaving provision.

For example, at the six-month point after leaving, approaching half (45 per cent) of respondents reported that their main activity was being in employment, a further 16 per cent that they were in education or training, and a quarter (25 per cent) that they were unemployed. Adding those in work to those in education or training shows, therefore, that amongst respondents to the survey, six in ten (61 per cent) were either in work and/or learning six months after leaving support. Likewise, the data shows that almost two-thirds of all leavers surveyed had received at least one job offer between starting on the YEI and six months after leaving.

Of those 14 per cent of respondents to the Leavers Survey whose employment status was recorded as inactive (excluding education and training) six months after leaving, over a quarter (28 per cent) were not in employment due to a sickness or disability, while just over one in five (22 per cent) were ‘looking after the home or family full time’. However a substantial minority (18 per cent) were working in a voluntary, unpaid role or in an internship. Of the remainder, just under one in ten (9 per cent) were caring for family members or friends with a long-standing illness, disability or infirmity. There were also a small proportion of respondents who were unsure or could not recall the reason for their inactivity at the six-month point, or who were retired, while 11 per cent gave ‘other’ reasons for this status.

As with the MI on programme leavers, there were no notable differences between men and women in terms of the six-month point employment and learning outcomes reported by survey respondents. There was a small difference between male respondents (46 per cent) and female respondents (43 per cent), while the converse was true for being in education or training (15 per cent of male respondents, 18 per cent of female respondents). While a smaller proportion of 15-17 year old respondents reported being in employment (27 per cent), as might be expected given their age group, figures for 18-24 and 25-29 year olds were also similar (48 and 52 per cent reporting being in work respectively). Similar to the picture suggested by the MI data, the main difference in relative rates of being in work at the six-month point after leaving was between those with a disadvantage (41 per cent) and those without (54 per cent).

Effectively the converse is true of the age group differences in terms of being in work at the six-month point, as might be expected more of the 15-17 age range reported being in education or training (47 per cent) compared to 18-24 or 25-29 year olds (15 per cent and 8 per cent respectively). The survey data shows, however, that there was little or no difference in terms of the likelihood of being in education or training at this point whether respondents had a disadvantage (19 per cent) or did not (20 per cent).

The survey data similarly reveals small differences for those reporting a disability or long term health condition and those who did not (17 and 20 per cent being in education or training at the six-month point after leaving respectively). However, there were notable differences in terms of employment outcomes for those with disabilities or health conditions compared to those without. Only 34 per cent of the former group reported being in employment at the six-month point compared to 50 per cent of the latter. This employment gap is, however, reflective of the differential rates of employment in the United Kingdom between those with disabilities and the overall employment rate. At the time of writing, labour market statistics show that the non-seasonally adjusted employment rate for those defined as having a disability was 54.1 per cent, as against the overall seasonally adjusted employment rate (including the disabled and non-disabled) of 82.2 per cent[footnote 29].

The positive picture of the YEI’s employment and learning impacts suggested by the CIE results, along with the MI and survey data, is backed up in several ways by the evidence gained through the YEI project visit case studies. A number of participants outlined positive employment and learning outcomes resulting from the support provided, along with the importance of these to their sense of self-worth, confidence and how positive they felt about career prospects in future. Indeed, participants interviewed that had entered work or apprenticeships through the support offered were extremely effusive concerning the positive impact of this, being described as ‘life-changing’ in one instance. Amongst this group, the YEI was also typically seen as a key factor in helping them achieve a positive work outcome, particularly the one-to-one and intensive support they had been provided.

The following case study concerning a participant gaining training, qualifications and ultimately an apprenticeship is typical of a number of themes and stories emerging through the participant depth interviews undertaken.

Case study – YEI learning outcomes (Rebecca)

Rebecca initially heard about the YEI when in the final year of high school, through the school Careers Centre. Rebecca had experienced some challenges with her mental health, and did not feel prepared for sixth form at the time due to severe anxiety.

Rebecca’s support from the YEI entailed regular key worker meetings focused on general advice and guidance, alongside researching different career routes, interests and hobbies. Alongside this 1-2-1 support, Rebecca engaged with a Maths tutoring programme to gain the core GCSE qualification that she needed. Other training that Rebecca received included courses on beauty therapy and food hygiene.

As a result of the support received, Rebecca described how she has secured an apprenticeship at a vegan food business in the local area. Through this opportunity, Rebecca feels she has developed new skills and gained new knowledge, business competencies and retail skills. To her, combined with the 1-2-1 support, this achievement of gaining an apprenticeship place has all made a significant difference to how Rebecca feels and her hopes for the future:

It’s helped my confidence and my anxiety has improved a lot… I ended up with the support that I wanted and it’s given me opportunities.

Without the YEI support, Rebecca commented that she would not have had access to the apprenticeship opportunity, as this was something that the key worker had organised with the employer. Summing up her experience of YEI support overall, Rebecca commented that:

It’s given me hope for the future; it’s given me hope for everything.

4.3. Softer impacts on participants

While the preceding section focused on ‘hard’ outcomes in terms of entry into employment, education or training, evidence gathered illustrates how the YEI has generated a number of important ‘softer’ outcomes, sometimes discussed in terms of ‘distance travelled’ towards employment and learning outcomes. Both the data available from the YEI Leavers Survey and the case study visits to YEI providers illustrate the nature, breadth and importance of these ‘softer’ participant outcomes.

Reflecting a number of anticipated ‘softer’ and broader impacts for participants captured in the YEI theory of change produced to guide the evaluation[footnote 30], both the Leavers Survey and qualitative research sought to gather evidence against a range of potential results of engagement with the YEI.

As table 4.8 summarises, consistent proportions of up to four-fifths of respondents to the Leavers’ Survey reported that the YEI had helped with a range of outcomes, including communication skills, self-confidence around work, ability to do things independently, motivation to find a job or seek promotion, motivation to do more training, and team working. Positive responses against these outcomes ranged from 77 per cent for motivation to find a job or seek a promotion, and motivation to do more training, to 80 per cent for communication skills and 79 per cent for ability to do things independently and self-confidence about working. A slightly lower proportion (72%), felt that YEI provision helped with their problem-solving skills. As the table indicates, results were fairly consistent irrespective of gender, though females were more likely to report improvements in all but one of the outcomes.

Table 4.8 YEI Leavers’ Survey: respondents reporting provision helped with…

Skills Total Male Female
Communication skills 80% 78% 82%
Ability to do things independently 79% 77% 82%
Self-confidence about working 79% 80% 78%
Ability to work with other people in a team 78% 77% 80%
Motivation to find a job or seek a promotion 77% 76% 79%
Motivation to do more training 77% 76% 79%
Problem solving skills 72% 71% 75%

Source: YEI Leavers’ Survey data gathered by IFF Research, Waves 1-9

Evidence from the qualitative case study research serves to confirm the above impression of the YEI having a broadly positive impact across a range of softer outcomes. While the emphasis placed on different softer participant impacts varied between interviewees, the view that YEI was having a range of beneficial impacts overall, in terms of ‘distance travelled’ and softer outcomes for participants, was consistent amongst both provider representatives and the young people receiving support.

From the perspective of YEI provider representatives, the most common outcomes for participants were enhanced confidence and interpersonal skills, improved attitudes and behaviours, and the achievement of basic and transversal skills (including employability skills). The impacts most commonly discussed by participants reflected this, being around improved confidence, raised aspirations, and increased motivation to work or go into education or training. Improved confidence and aspirations emerged as particularly strong and consistent themes in participants’ discussions of impacts from engaging with the YEI, with the support offered by key workers frequently being cited as instrumental in this.

Participants also cited a range of new or enhanced skills that they felt had resulted from engagement with the YEI. Examples included improved leadership skills, employability skills, skills around budgeting and managing finances, enterprise skills, and subject-based skills, including Mathematics and English in particular. In terms of the latter, both provider representatives and young people felt that the achievement of Level 1 qualifications, providing access to GCSEs in these core subjects, and the achievement of GCSEs themselves, were a key impact of engagement. Equally, such developments in skills and qualifications, along with improved confidence, were frequently seen as central to the achievement of broader YEI impacts, namely those around reducing barriers to (re-) engagement in work and learning and supporting access to traineeships and apprenticeships.

It was also common for participants to cite that engagement had improved their knowledge of the options available, in terms of work and further learning, how they might go about achieving goals, and the additional support and provision they could access to achieve these. As one participant commented in this context:

It’s quite good to have that knowledge that these are the routes that I can take, these are the skills I need…” (YEI participant)

While outcomes around improved health, and greater financial security, were less frequently referenced in the case study research, this was principally due to such impacts only being relevant to certain participants and/or the support not being targeted at addressing such issues directly. However, from a broader health perspective, improved wellbeing or happiness in general was cited by several participants. Likewise, in some cases YEI support was key in addressing or managing mental health issues, such as anxiety. However, more specific physical health impacts were less apparent. In the minority of cases where those interviewed had a physical health condition or disability, this tended to be managed by other forms of support outside the programme and/or by the participant themselves.

In terms of financial security and earning power, while participants interviewed in the case study visits were confident that the support would have such impacts in the longer term, even those who had entered work generally felt it was too early to identify such impacts at the time of the interviews. However, it is worth noting that the YEI Leavers Survey results indicate that almost nine in ten respondents felt that the support received would help them find a job in future, with 52 per cent feeling that it would ‘help a lot’ and a further 36 per cent that it would ‘help a little’. From the participant perspective, therefore, the support appears to provide confidence that greater financial security and earning power is likely to result. Likewise, this finding, along with the qualitative case study research, suggests that in many cases the YEI is likely to have improved participants’ competitiveness in the labour market, as the results of the CIE strand likewise seem to indicate.

The following participant case study, drawn from one of the depth qualitative interviews with young people, illustrates the role of the programme in generating a range of ‘softer’ impacts and providing confidence that labour market competitiveness, and the likelihood of finding work, has increased.

Case study – YEI softer outcomes (Derren)

Before joining the YEI, Derren reported that he had experienced a ‘challenging’ relationship with his parents after dropping out of university and having to move home. At that time, he described how he was struggling to find employment and his confidence levels were very low. Upon registering with the YEI, Derren was assigned a 1-1 key worker and discussed his background. Derren described how his key worker “joined the dots straight away” and gave him some clarity on how to move forward from his current situation. A key part of this was the support the key worker provided to find volunteering opportunities for Derren to improve his softer employability skills and help restore his confidence again. For Derren, a significant memory was the initial meeting with his key worker:

The very first meeting sticks out, he [job key worker] told me that I wanted a career not a job. I have taken that with me through my whole journey here. I don’t want to just get by, I want to do something I enjoy.

Derren was reassured by the support that he received and very positive about its impact on him:

I was very happy, it was very different to anything I had received previously – the support had a personal touch.

For Derren, engaging with the YEI provision has been a great success. Alongside a volunteering opportunity as an employability tutor at the YEI project, Derren has accessed a teacher-training course to help him with finding employment and developing a career in the future. Derren described how the YEI activities and support were much better than he had anticipated, addressing his personal development and circumstances as well as directly putting him on a path towards what he hopes will be sustainable employment:

To see where I started to where I am now… my relationship with my Mum is much better… it has helped me to be independent, to be more confident.

4.4. Wider impacts of the YEI

Beyond its direct effects on participants, the YEI theory of change produced to guide the evaluation served to highlight a range of potential wider impacts to which the programme could be expected to contribute. Specifically, these included a reduction in NEET and youth unemployment levels, positive ‘knock-on effects’ on peer groups and families, organisational and policy learning, supporting more joined-up services at local levels, providing economic benefits to the Exchequer, and improving inter-generational awareness. The following sub-sections bring together the evaluation evidence, including that from the CIE, CBA, MI data, Leavers Survey and case study research as applicable, to assess these presumed impacts of the YEI in turn.

4.4.1. Effects on NEET and youth unemployment levels

The CIE results presented in section 4.1, allied to YEI MI data and that from the Leavers Survey, provide good evidence that the programme succeeded in having a positive effect on NEET and youth unemployment levels. As outlined in previous sections, it is clear that the programme resulted in a positive net employment impact, with YEI participants spending an average of 56 days more in employment over 12 months compared to a similar comparator group. Added to the survey finding that six in ten (61 per cent) respondents reported being either in work and/or learning six months after leaving support, and that almost a third of participants enter work on leaving support (programme MI), this suggests that the YEI has had a notable impact on reducing NEET and youth unemployment levels.

The significance of these figures can be placed in context using publically available official statistics on NEET levels. Estimates of the number of NEET young people aged 16-24 in England over the period of the YEI’s delivery, taking the final quarter of 2014 as the start point, show a gradual fall, with some seasonal fluctuations, from 786,000 in Q4 2014 to 651,000 in Q4 2018[footnote 31]. Given that the YEI is being delivered in only three NUTS[footnote 32] 2 areas (Merseyside, Tees Valley & Durham, and West Midlands) within 3 of the 9 English regions, along with several smaller NUTS 3 areas (Leicester, Nottingham, Kingston upon Hull, and Thurrock), it can be conservatively assumed that the total 16-24 NEET population eligible for YEI support over this period is likely to be no more than 150,000 and possibly under 100,000[footnote 33].

The data available from the YEI MI showing that 73,935 participations amongst those on the programme have been recorded to date suggests, in turn, that the initiative is likely to be successfully targeting a notable proportion of NEET young people in the areas it operates (accepting that some of these will be in the 25-29 age group, while the above figures quoted are for those aged 16-24). By extension, the proportions of participants achieving positive employment and learning outcomes noted above indicates that the YEI is likely to be having an observable impact on NEET levels within its delivery areas. If the pattern of around 1 in 3 YEI leavers exiting the programme into employment is maintained, along with those exiting to learning destinations, this suggests that the effect in the areas concerned will be notable in terms of the proportion of young people supported out of being NEET.

4.4.2. Economic benefits

This section sets out the results of the CBA of the YEI, conducted in-house by DWP analysts, drawing on the impact estimates presented in section 4.1 above. It covers a sample of 960 people in the treatment group identified for the CIE analysis, split across DWP out of work benefits. The analysis presented in the following sub-sections tracks the employment and off-benefit economic impact for these people for twelve months post-YEI participation.

Methodology

The methodology underpinning the CBA is detailed in the DWP Working Paper 86[footnote 34]. It is used across DWP employment programmes to provide a consistent, evidence-based approach to CBA. For the purpose of this analysis, reference is made to the exchequer and societal perspective to present the CBA[footnote 35].

Costs and benefits under consideration

Benefits

There are a number of benefits that accrue from moving someone into employment and off-benefit, as is the case in respect of the YEI. These include:

  • Increases in earnings to the individual
  • Increases in tax receipts to the government
  • Reductions in costs to the government from the individual being on benefit
  • Some health benefit from being in employment

There is a full discussion of these benefits in a DWP impact evaluation and cost benefit analysis of Sector Based Work Academies[footnote 36].

Programme Costs

As with all of the European Social Fund (ESF), there is an allocation that exists for YEI. Projects bid to the Managing Authority for a portion of that allocation, through which they then deliver support for a number of participants. Each project can be quite different in terms of the costs, and the costs per participant.

Therefore, to estimate the programme costs we have analysed the 20 YEI projects from which the treatment group was drawn. This gives a more representative idea of the actual costs involved in delivering the programme in these areas. Performing this analysis gives us an estimated average of around £2,600 per participant in a combination of ESF and “matched” costs.

This is the full cost of YEI, including employment and education or skills support, even though we can only currently estimate the employment and off-benefit impacts. YEI will deliver both employment and educational outcomes, but it is not possible in the data to exclude the education or skills element of the costs.

Estimating the scale of costs and benefits of the YEI

The impacts for the treatment group presented in section 4.1 are split-out into JSA, ESA and “all”. From this, we can derive the costs and benefits associated with the average person on these benefits, and therefore the average costs and benefits obtained by the outcomes estimated above.

In order to estimate the likely costs and benefits of YEI we have specifically focused on the 16-24 age group. This group tend to have different wage assumptions and, also, different rates of DWP benefits, so therefore have differing scales of costs and benefits.

These individuals are tracked for 52 weeks post intervention. As a result, it is likely that the full impact of the programme will be underestimated. The total programme cost is reflected in the calculations but only 12 months of impacts are included. Therefore, the estimates presented potentially do not capture the full value for money achieved.

In addition, the impact analysis does not consider educational outcomes (as explained in section 4.1). Nor is it possible to estimate the potentially added impact on earnings for YEI participants moving into employment, even if the employment outcome itself was not additional.

Impacts

The approach taken allows us to calculate monetary costs and benefits for additional days off-benefit and additional days in employment. These then form the basis of our CBA from the perspective of the exchequer and society.

The total cost for the programme participants in the sample is £2,500,000, while the total fiscal return is £334,000 and the total social return is £3,900,000[footnote 37]. These are made up of different components, such as; increases in taxes, government payments/savings, increases in output by participants and operational costs or savings. The net government saving from moving YEI participants off ESA and JSA, but spending longer on UC, and into employment is £16,000. The increase in taxes paid to the exchequer is £330,000; this is due to more people spending time in employment and earning more money, paying more money in income tax and Value Added Tax (VAT). Finally, the additional days participants spend in employment leads to increased total output of £1,660,000.

Results

The cost-benefit analysis gives a social return in the range of £1.50 to £1.55 per £1 spent and a fiscal return in the range of £0.13 to £0.17 per £1 spent[footnote 38].

As the participants are young (15-29), the fiscal and social returns are generally lower. This is because even those young people who move into work tend to be paid less compared to the average person, leading to relatively lower benefits in terms of income, tax receipts and other benefits.

As we have discussed above, a lack of educational data also means that we cannot estimate the impact YEI may have on increasing education or skills, and, therefore, the associated benefits. As a result, not only is this a direct benefit that is missing from our CBA results, but some participants may not be obtaining labour market outcomes because YEI is moving them into additional education spells. This would show as a negative in our results, when the reality should be a positive outcome and a movement towards the labour market, or a chance of obtaining better labour market outcomes.

In addition, our twelve-month tracking period also limits the potential results as outcomes may accrue after the 12-month period or be sustained beyond that point.

Conclusion

To conclude, the impacts show that YEI is having a positive effect on participants, in terms of additional days in employment compared to the counterfactual.

Our CBA shows that these impacts translate to a positive social return on investment and suggests that this return would be even bigger with data not available at the time of the analysis.

In particular, if we were able to obtain income data then we could see more clearly what the impact on earnings and tax was, rather than using our assumption about the average young person’s wage. This could further improve the return.

On top of that, costs may be higher than comparable forms of support due to the disadvantages experienced by those whom the YEI is targeting, along with the intensity of support required.

Additionally, YEI delivers educational outcomes that we cannot yet include in our cost-benefit analysis. Therefore, the full costs are being reflected but only some of the benefits. Finally, the tracking period is only for 12 months, the positive impacts will carry on longer than the 12 months even though we are not tracking them.

All of this means that our CBA shows a positive return on investment as a result of YEI, but one that is potentially an underestimate because of the benefits that are not captured and the nature of the costs that are already included.

4.4.3. Organisational and service delivery impacts

Provider representatives consulted through the case study visits were generally positive concerning the results of engaging with the YEI from an organisational perspective. Examples offered of perceived organisational benefits and impacts included:

  • Improved internal systems and processes, for example through gaining experience of delivering ESF provision and/or from learning from delivery partners
  • Improved reputations and track-records for delivering large programmes, including those involving European funding
  • The development of new links and relationships locally with agencies and organisations working to support young people.

In several cases, through co-location of services for instance, interviewees were confident that this latter aspect of improved cross-organisation and cross-service working was contributing, in turn, to more joined-up and improved services for young people locally. It was acknowledged that the full effects of this are difficult to quantify, and it should be noted that such effects were not universally noted across all case study contexts. Nonetheless, in several instances such impacts were cited as both evident and important. In one case, for example, these benefits were situated in the wider context of improved partnership and collaborative working felt to have developed locally as a result of the YEI. As the interviewee concerned noted:

Participating organisations are learning [about] how we work together, how, as a city, providers can come together and do something for young people. (YEI lead provider representative)

In general, those consulted in the case study visits found it harder to define or assess impacts on organisations beyond those they were directly working with (i.e. delivery partners or related services with which close relationships had been developed). As such, most interviewees felt it was hard to assess impacts on participating employers, for example, beyond being able to, anecdotally, report positive experiences of taking on YEI participants that had been relayed to them. Accepting this, a number of provider representatives did note that some employers have undoubtedly gained good employees as a result of the programme. Equally, some interviewees commented that, in some cases, employers may have had their expectations concerning NEET young people changed for the better as a result of their engagement, whether through providing work placements or providing participants with employment.

While the above findings can only be indicative, they do suggest that the YEI has had a number of beneficial impacts for both organisations delivering support and, in all likelihood, some local employers. In addition, it should be noted that while delivering support was cited as being challenging at times, there were no specific examples of negative impacts caused by involvement with the YEI offered by case study interviewees.

4.4.4. Impacts on local communities

Outside of the perceived benefits for local communities from the community-based and volunteering activities young people have engaged with as part of the YEI, results for local communities were seen as hard to define or evidence other than in hypothetical terms. Thus, for example, some provider representatives felt that, instinctively, the support offered should result in less marginalised young people likely to engage in anti-social behaviour. However, it was noted that this is not being measured formally and is therefore hard to quantify. Equally, several interviewees posited that economic contributions made by those entering work as a result of YEI support would have a benefit locally. Again, however, it was acknowledged that in the absence of a form of cost-benefit analysis seeking to quantify this robustly, hard evidence for such impacts is inevitably limited.

Similar views were offered in relation to the likely benefits of inter-generational awareness and understanding in local communities, felt to be accruing through, for example, young people working in volunteering contexts with older colleagues. While some individual examples of these impacts were offered, it was generally felt that a lack of comprehensive evidence meant that any claims of widespread or consistent impacts in this area would be difficult to estimate or confirm.

There were also few examples given of knock-on impacts from participant engagement for their wider peer groups or families. Where such examples were offered, they tended to be relatively specific to the individual participants concerned and their particular situations. For example, improved family relationships were cited as an outcome in a small number of cases, generally in situations where individuals’ lives had been stabilised or improved with this having additional benefits in their home lives. While not common, these examples do nonetheless show that YEI support does have the potential to offer some knock-on benefits.

Overall, therefore, impacts on local communities and those within them were generally seen as hard to evidence or quantify, but nonetheless were felt to be positive in instances where they could be observed.

4.4.5. Policy and programme ‘learning’ impacts

Provider representatives, along with those from ESIF sub-committees and other high-level stakeholders, felt that the YEI had also generated broader ‘learning’ impacts. Typically, these were described in terms of helping to inform or crystallise policy insights, and/or as providing lessons for employability and skills initiatives targeted at young people. While a range of examples were offered, the themes discussed tended to focus on the following insights or lessons:

  • The unexpected prevalence of mental health issues facing NEET young people, including anxiety and depression, was seen by several provider representatives as being a significant change from their past experience. A focus on integrating low/medium level support for common mental health conditions, such as anxiety, into future employability policy and programmes was commonly suggested therefore, along with the need to ensure adequate signposting and provision for those with more severe conditions.
  • In the current economic and employment context it was noted that (remaining) NEET young people are likely to be further from the labour market and have complex, multiple barriers or needs; this was seen by several stakeholders as highlighting the potential importance of developing specific provision to target and support this group.
  • The potential to review the balance between the level of evidence requirements needed for the Managing Authority’s financial assurance processes, evaluation, fraud prevention and detection, as against seeking to minimise any administrative burdens on organisations delivering support.
  • There was a common concern that a ‘vacuum’ of the type of support being offered through the ESF for NEET young people could result from the UK leaving the EU and the end of the current programme. This was seen as a priority to address, particularly in terms of ensuring the availability of tailored, intensive one-to-one support and provision, able to help those facing multiple barriers, of a type not currently offered through other mainstream employability programmes.

4.4.6. Unanticipated or unintended impacts

In general, provider representatives and other stakeholders offered few examples of unintended or unanticipated impacts in respect of the YEI. However, some providers expressed surprise at the things young people had been through in terms of their backgrounds and, in light of this, what they had overcome to achieve positive outcomes.

Similarly, participants tended to feel that the support received had not generated any unanticipated outcomes for them per se. However, in several cases, those interviewed as part of the case study visits did cite that the level of progress they had made, the significance of the support received, and/or the changes that this had led to in their situations, was unanticipated in terms of scale and importance. This suggests that the provision had the impacts participants expected, but that it was the scale, nature and level of these effects that were unanticipated in some cases.

5. Conclusions

This chapter concludes the report by summarising the key findings from the preceding analysis. It also revisits the theory of change developed to guide the evaluation, considering the extent to which the presumed links between YEI inputs and activities led to intended outcomes and impacts, along with the degree to which the programme’s stated aims and objectives were met as a result. The report ends with a brief discussion of some broader implications and considerations arising from the evaluation.

5.1. Key findings from the evaluation

Evidence gathered for this evaluation shows that delivery of the Youth Employment Initiative (YEI) has been generally effective and broadly efficient, while the programme has achieved a range of positive outcomes and impacts – most notably around its core aim of reducing levels of young people not in employment, education or training (NEET).

The YEI has been effective in targeting and engaging NEET young people, including those most disengaged from mainstream services (so-called ‘hidden NEETs’). Likewise, evidence shows that the delivery models adopted to engage and support young people have proved generally effective, particularly through the widespread adoption of key worker roles to coordinate and offer wraparound support tailored to individuals’ needs. The programme has also ensured that young people have been able to access support that, in general, is effective in helping them move towards, and into, employment and education. The YEI also appears effective in terms of the quality of traineeships and job offers that participants have been able to access.

The findings around effectiveness in delivery are mirrored in the available evidence concerning efficiency. Delivery organisations appear committed to ensuring efficient delivery and, ultimately, offering value for money for the funding used. While a lack of data precludes a full efficiency assessment, in terms of weighing costs against the throughput of specific activities, the general impression from the qualitative evidence is that efficiency and value for money have been key delivery considerations. Effective governance, a focus on (reducing) unit costs, and efficiently harnessing skills and organisational capacities were all key factors in the generally positive impression of efficiency evident.

The evaluation findings indicate that the effective and efficient delivery of YEI support is likely to have contributed to the range of positive outcomes and impacts apparent. The CIE strand of the study suggests that the programme has generated notable additionality in terms of its effects on the employment prospects of young people, whilst the wider evidence base shows that support has generated a number of broader outcomes and impacts. These span:

  • positive effects on participants, in terms of a range of softer outcomes likely to promote employability, including enhanced confidence, motivation and transversal skills such as team working, along with gaining qualifications and being supported to re-enter education and training;
  • beneficial outcomes for organisations delivering support, including increased understanding of what works in supporting young people, along with enhanced networks and partnership arrangements;
  • broader benefits in terms of policy learning and insights into effective delivery that can be used in designing future initiatives.
    While, as with any such programme, YEI support will not have been successful in supporting all young people engaged, and some elements of delivery will have worked better than others, the overall impression is that the programme has been effective in meeting its core aims, and that it will have had a notable positive impact on reducing NEET and youth unemployment levels in its target areas.

5.2. Revisiting the theory of change

As the above discussion of key evaluation findings should indicate, when reviewing the evidence base generated against the theory of change produced to guide the evaluation (included at Appendix A), the impression is that the YEI activities delivered have generally led to the outcomes and impacts anticipated. As a result, the main aims and objectives of the programme have been met to a significant degree, while the rationale for the YEI appears appropriate and the intervention necessary in retrospect.

In terms of aims and objectives, the programme can be judged as meeting its overarching aim around sustainable labour market integration, including amongst young people at risk of social exclusion, and its specific objectives. Case study evidence clearly shows that some of those engaged have been marginalised with complex needs, while the support delivered has addressed skills needs, provided training and work experience, and in many cases supported young people to overcome barriers to the labour market. In doing so it has added value to existing provision, with stakeholders’ generally situating their positive views of the programme in the context of the YEI offering the type of tailored, intensive support to NEET young people that would otherwise be limited in their locality.

The key programme inputs (financial and otherwise), along with the wide range of activities delivered, have led to the majority of the anticipated outcomes and impacts captured in the theory of change. Table 5.1 below summarises the quantifiable results targets detailed in the theory of change diagram at Appendix A, demonstrating that most have been achieved or exceeded.

Table 5.1: YEI Performance against results targets

Target Achieved
70% of participants completing the YEI supported intervention 70%
48% of participants who complete their participation receiving an offer of employment, continued education, apprenticeship or traineeship upon leaving 61%
48% of participants who complete their participation being in education/training, gaining a qualification, or in employment, including self-employment, upon leaving 65%
60% of long-term unemployed participants completing the YEI supported intervention 58%
38% of long-term unemployed participants who complete their participation receiving an offer of employment, continued education, apprenticeship or traineeship upon leaving 56%
38% of long term unemployed participants who complete their participation being in education/training, gaining a qualification, or in employment, including self-employment, upon leaving 59%
60% of inactive participants completing the YEI supported intervention 81%
33% of inactive participants who complete their participation receiving an offer of employment, continued education, apprenticeship or traineeship upon leaving 67%
33% of inactive participants who complete their participation being in education/training, gaining a qualification, or in employment, including self-employment, upon leaving 63%
15% of participants who complete their participation being in continued education, training programmes leading to a qualification, an apprenticeship or a traineeship six months after leaving 20%
34% of participants who complete their participation being in employment six months after leaving 48%
3% of participants who complete their participation being in self-employment six months after leaving 2%

Source: YEI programme MI

As discussed in the preceding chapter, the YEI Leavers Survey evidence also shows that high proportions of participants felt that the initiative had beneficial effects on the types of softer outcomes referenced in the theory of change (including around interpersonal and basic skills, along with behaviours and attitudes). The programme’s performance in supporting individuals into work, training and education likewise offers a strong indication that, for many of those engaged, it will have been successful in addressing labour market barriers, improving labour market competitiveness, and improving access to apprenticeships, traineeships and education. By extension, positive effects on financial security and earning power can also be assumed. Evidence on health and wellbeing outcomes is more limited, though some examples of how the YEI can have effects of this type was apparent through the case study visits, as was a greater understanding of, and access to, other services.

Evidence shows that the intended wider outcomes of the initiative have been met to a notable extent. As discussed in chapter four, the YEI has certainly contributed to a reduction in NEET and youth unemployment levels, while positive knock-on effects on peers and families was referenced in a number of case study interviews with participants and delivery staff. Organisational learning likewise emerged as a notable additional benefit for those delivering support.

The achievement of ‘more joined up services at local levels’ is harder to assess and quantify. However, findings around benefits for YEI providers in terms of enhanced networks, and that wide-ranging support offers for participants were apparent, bringing a range of wraparound support together, offer positive indications in this direction. There were also anecdotal examples of improved inter-generational awareness in particular delivery contexts, though these were very much dependent on the focus of activity (being particularly linked to volunteering opportunities). Finally, in terms of economic benefits, as described the CBA conducted by DWP analysts estimates that while the programme has a social return in the range of £1.50 to £1.55 per £1 spent, the fiscal return is only in the range £0.13 to £0.17 per £1 spent. As discussed in chapter four, however, it is acknowledged that these figures are likely underestimate the returns concerned with the true value for money achieved by the YEI being higher.

5.3. Concluding reflections and implications

This evaluation of the YEI’s effectiveness, efficiency, and impact highlights that targeted interventions, offering intensive and wraparound support, can be effective in addressing the challenge of significant numbers of young people being out of work, education or training. While, from the early 2010s onwards, official statistics reveal a gradual decline in NEET numbers and youth unemployment, the need to effectively support young people facing labour market barriers and who are not in education or training remains. The YEI shows that well designed interventions can support numbers of young people into work over and above levels that would be expected in their absence. They can also have positive economic effects, both at the individual level, from moving into work from unemployment, and for the Exchequer in terms of tax gains and reductions in benefit payments.

In addition, specific aspects of the ‘YEI offer’ can usefully be considered in future initiatives targeting NEET young people. The role of key workers in facilitating effective wraparound support, and coordinating assistance, is one such aspect. The importance of designing effective targeting mechanisms, combining outreach with effective referral partnerships and routes, including Jobcentre Plus as a core partner, is another. Finally, the YEI re-affirms the importance of developing individuals’ softer and transversal skills, particularly around confidence and motivation, as part of a pathway towards work and/or (re-)engagement in education and training.

6. Annex A: Theory of change

The theory of change diagram

Theory of change diagram

7. Annex B: Technical

Supporting materials documenting the methods used in compiling this report are available in the following locations:

8. Annex C: Topic guides

These topic guides have been adapted for publication and excludes information provided to the participant about the research, consent and data protection.

8.1. ESF Managing Authority and European Commission Staff

Part One: Introduction / background information on the YEI

1. Please outline how you have been involved with the YEI in England.

  • Cover main elements of role in relation to YEI: period, timing and nature of involvement

Part Two: Effectiveness of targeting

2. Thinking about the focus of the YEI, how far do you feel the initiative has effectively targeted those young people most in need of support?

  • In what ways has the initiative been effective in its targeting?
  • Are there any ways in which the YEI has proved less effective?

3. To what extent have the specific target groups the YEI aims to support been reached as planned?

  • (reminder of target groups: young lone parents; looked after children and care leavers; young carers; ex-offenders; those involved in gangs; and young people with learning difficulties and disabilities; unemployed or inactive graduates).
  • To what extent have YEI projects specifically targeted these groups as opposed to targeting the NEET population in general?
  • Are there any of the specific YEI target groups that you feel have not been targeted, or have been targeted less effectively, and why?

4. To what extent have young people at risk of social exclusion and those from marginalised communities been reached?

  • Probe on the reasons for the interviewee’s response

5. How far has the YEI managed to target NEET young people with complex needs and multiple disadvantages?

  • Probe on the reasons for the interviewee’s response

Part Three: Effectiveness of the support offered

6. Do you have a sense of which YEI activities and interventions have been effective and which less so (e.g. enterprise support; access to apprenticeships; employer placements etc.)? If so, which ones and why?

  • Did this vary between sub-groups or not (e.g. certain interventions being effective for particular groups)? If so, how and in what ways?
  • Probe for evidence for the interviewee’s perspective

7. To what extent do you feel the types of support offered to YEI participants have proved relevant to their needs and why?

  • Probe on whether specific support was effectively targeted at specific sub-groups (e.g. graduates as opposed to those with basic skills needs)

8. Can you provide any specific examples of effective practice in supporting the target groups of the YEI to move closer to the labour market?

  • Probe on why this is seen as effective practice and any evidence for this.

9. What barriers or challenges were faced in implementing the YEI?

  • How were these addressed and what learning can be drawn from this to inform future policy and interventions?

10. Overall, how effective would you say the delivery of the YEI has been to date, in terms of meeting its key objectives such as supporting sustainable integration into the labour market, and why?

11. What elements of the YEI have proved most effective and why?

12. What aspects of the YEI have proved to be less effective and why?

13. What are the main lessons that can be learned from delivery of the YEI that might inform future initiatives aiming to support NEET young people?

14. How would you describe the overall contribution of the YEI to addressing the policy challenge posed by NEET young people?

  • Probe on the reasons for the interviewee’s response

Part Four: Efficiency of the YEI

15. In terms of the procurement of the YEI provision, to what extent do you feel this ensured value-for-money and why?

  • Probe on the degree to which the process ensured that outputs are likely to be achieved for the minimum possible cost, use of cost benchmarks, unit costs etc.

16. To what extent do you feel YEI providers have sought to ensure that their provision is being delivered efficiently and provides value-for-money?

  • Probe for examples of how providers are seeking to provide value-for-money or otherwise

17. Do you have a sense of which types of activity or intervention are proving to be most cost effective and why? Equally, those which are proving less cost effective and why?

  • Prompt and probe using knowledge of project interventions and the list of activities in the theory of change (training; volunteering; support to access apprenticeships; wraparound support; careers guidance; self-employment/enterprise support; financial support to address barriers; post- entry to employment / training support)

Part Five: Outcomes and impacts from the YEI

18. In your view, what are the main outcomes and impacts that have resulted from the YEI?

  • Probe on outcomes for individuals, employers, YEI providers, wider communities, the Exchequer, policy and practice learning
  • In what ways would you say outcomes and impacts from the YEI are most apparent and why?

Ask and probe on the following as applicable depending on the response to Q.18 and the likely level of knowledge of the interviewee around YEI outcomes.

19. How would you describe the impact of the YEI on young people’s future employment chances?

  • Probe on reasons for the response, evidence and examples

20. How would you describe the broader outcomes for YEI participants (e.g. interpersonal skills, behaviours, attitudes etc.)?

  • Probe on reasons for the response, evidence and examples

21. Are you aware of any unintended or unexpected outcomes for YEI participants? If so, what?

  • Equally, have there been any knock-on effects from support – e.g. on participants’ peer groups, families etc.?

22. How would you describe the impact of the YEI in terms of reducing NEET levels and youth unemployment?

  • Probe on reasons for the response, evidence and examples

23. Are there any additional outcomes or impacts that can be ascribed to the YEI? If so, what are these?

  • Prompt that we are thinking of broader impacts – e.g. more joined up local services, increased partnership working locally, policy changes, cross-sectoral links (public/private/voluntary), education system, Jobcentre Plus practices etc.

  • Probe on reasons for the response, evidence and examples

24. Have there been any unintended or unexpected outcomes of the YEI as a whole? If so, what?

Part Six: Closing / overall reflections

25. What are the key learning points from the YEI programme that might inform future provision of this type or any successor provision to the current ESF programme?

26. Do you have any final thoughts or reflections concerning the YEI and its relative effectiveness, efficiency and impact?

8.2. ESIF (European Structural and Investment Funds) Sub-Committee representatives

Part One: Introduction / background information on the YEI

1. Please outline how you have been involved with the YEI in your area.

  • Cover main elements of role in relation to YEI: period, timing and nature of involvement; level of knowledge of YEI provision and involvement with implementation

Part Two: Effectiveness of targeting

2. Thinking about the focus of the YEI, how far do you feel the initiative has effectively targeted those young people most in need of support?

  • How far did this targeting draw on local data / intelligence – e.g. from the LEP
  • In what ways has the initiative been effective in its targeting?
  • Are there any ways in which the YEI has proved less effective in its targeting?

3. To what extent have the specific target groups the YEI aims to support been reached as planned?

  • (focus on specific target groups in the area from: young lone parents; looked after children and care leavers; young carers; ex-offenders; those involved in gangs; and young people with learning difficulties and disabilities; unemployed or inactive graduates).
  • To what extent has YEI provision effectively targeted those groups identified as priorities locally – e.g. through LEP strategies?
  • To what extent have YEI projects specifically targeted specific groups as opposed to targeting the NEET population in general?
  • Are there any of the specific YEI target groups identified locally that you feel have not been targeted, or have been targeted less effectively, and why?

4. To what extent have young people at risk of social exclusion and those from marginalised communities been reached?

  • Probe on the reasons and evidence for the interviewee’s response

5. How far has the YEI managed to target NEET young people with complex needs and multiple disadvantages?

  • Probe on the reasons and evidence for the interviewee’s response

Part Three: Effectiveness of the support offered

6. Do you have a sense of which YEI activities and interventions have been effective locally and which less so (e.g. enterprise support; access to apprenticeships; employer placements etc.)? If so, which ones and why?

  • Did this vary between sub-groups or not (e.g. certain interventions being effective for particular groups)? If so, how and in what ways?
  • Probe for evidence for the interviewee’s perspective

7. To what extent do you feel the types of support offered to YEI participants through local provision has proved relevant to their needs and why?

  • Probe on whether specific support was effectively targeted at specific sub-groups (e.g. graduates as opposed to those with basic skills needs)

8. Can you provide any specific examples of effective practice in supporting the target groups of the YEI to move closer to the labour market?

  • Probe on why this is seen as effective practice and any evidence for this.

9. What barriers or challenges were faced in implementing the YEI locally?

  • How were these addressed and what learning can be drawn from this to inform future policy and interventions?

10. Overall, how effective would you say the delivery of the YEI has been to date, in terms of meeting its key objectives such as supporting sustainable integration into the labour market, and why?

11. What elements of the YEI have proved most effective and why?

12. What aspects of the YEI have proved to be less effective and why?

13. What are the main lessons that can be learned from delivery of the YEI that might inform future initiatives aiming to support NEET young people?

14. How would you describe the overall contribution of the YEI to addressing the challenges posed in respect of NEET young people in the local area?

  • Probe on the reasons for the interviewee’s response

Part Four: Efficiency of the YEI

15. In terms of the procurement of the YEI provision, to what extent do you feel this ensured value-for-money and why?

  • Probe on the degree to which the process ensured that outputs are likely to be achieved for the minimum possible cost, use of cost benchmarks, unit costs etc.

16. To what extent do you feel YEI providers have sought to ensure that their provision is being delivered efficiently and provides value-for-money?

  • Probe for examples of how providers are seeking to provide value-for-money or otherwise

17. Do you have a sense of which types of activity or intervention are proving to be most cost effective and why? Equally, those which are proving less cost effective and why?

  • Prompt and probe using knowledge of project interventions and the list of activities in the theory of change (training; volunteering; support to access apprenticeships; wraparound support; careers guidance; self-employment/enterprise support; financial support to address barriers; post- entry to employment / training support)

Part Five: Outcomes and impacts from the YEI

18. In your view, what are the main outcomes and impacts that have resulted from the YEI? - Probe on outcomes for individuals, employers, YEI providers, local communities, the Exchequer, policy and practice learning

  • In what ways would you say outcomes and impacts from the YEI are most apparent and why? Ask and probe on the following as applicable depending on the response to Q.18 and the likely level of knowledge of the interviewee around YEI outcomes

19. How would you describe the impact of the YEI on young people’s future employment chances in the local area? - Probe on reasons for the response, evidence and examples 20. How would you describe the broader outcomes for YEI participants (e.g. interpersonal skills, behaviours, attitudes etc.)? - Probe on reasons for the response, evidence and examples 21. Are you aware of any unintended or unexpected outcomes for YEI participants? If so, what?

  • Equally, have there been any knock-on effects from support – e.g. on participants’ peer groups, families etc.?

22. How would you describe the impact of the YEI in terms of reducing NEET levels and youth unemployment?

  • Probe on reasons for the response, evidence and examples

23. Are there any additional outcomes or impacts that can be ascribed to the YEI? If so, what are these?

  • Prompt that we are thinking of broader impacts – e.g. more joined up local services, increased partnership working locally, policy changes, cross-sectoral links (public/private/voluntary), education system, Jobcentre Plus practices etc.
  • Probe on reasons for the response, evidence and examples

24. Have there been any unintended or unexpected outcomes of the YEI as a whole? If so, what?

Part Six: Closing / overall reflections

25. What are the key learning points from the YEI programme that might inform future provision of this type or any successor provision to the current ESF programme?

26. Do you have any final thoughts or reflections concerning the YEI and its relative effectiveness, efficiency and impact?

8.3. Lead Provider – Manager / Strategic level

Part One: Introduction / background information on role

1. Please outline your role in relation to the YEI within [insert provider name]. - Cover main elements of role in relation to YEI: period, timing and nature of involvement

Part Two: Effectiveness of targeting

2. How far, and in what ways, do you feel your project has effectively targeted local young people most in need of support?

  • How far, and in what ways, does this targeting draw on local data / intelligence – e.g. from the LEP, knowledge within your organisation, partners’ knowledge etc.

  • In what (other) ways would you say your project has been effective in its targeting?
  • Are there any challenges you have faced in effectively targeting those young people in need of support locally, and how have you sought to address these?

3. To what extent have the specific target groups your project aimed to support been reached as planned?

  • (focus on specific target groups the project focuses on: e.g. NEET population in general; young lone parents; looked after children and care leavers; young carers; ex-offenders; those involved in gangs; and young people with learning difficulties and disabilities; unemployed or inactive graduates).
  • To what extent has your project targeted those groups identified as priorities locally – e.g. through LEP strategies?
  • To what extent has your project targeted specific groups as opposed to targeting the NEET population in general, and why?
  • Are there any of the target groups your project sought to engage that have proved challenging, or you haven’t been able to reach effectively, and why?

4. How do you secure buy-in from participants and motivate them to engage with provision?

  • What effect does the voluntary nature of provision have on this?
  • Does the lack of conditionality, in terms of participants being mandated to engage as a condition of benefit receipt for example, affect your ability to engage and motivate participants? If so, how and in what ways?

5. How far would you say your project has effectively reached young people at risk of social exclusion and those from marginalised communities?

  • Probe on the reasons and evidence for the interviewee’s response

6. How far would you say your project has effectively reached NEET young people with complex needs and multiple disadvantages?

  • Probe on the reasons and evidence for the interviewee’s response

Part Three: Effectiveness of the provision

7. Within your project, which types of provision or activities have been effective and why?

  • Are there specific approaches or activities that have been particularly effective with certain groups (e.g. the most marginalised; those with disabilities etc.)? If so, which activities and groups and how is this evident?

8. Equally, are there types of provision or activities that have proved less effective than anticipated and why?

  • Did effectiveness vary between groups (e.g. certain interventions being less effective for particular groups)? If so, how and in what ways?
  • Probe for evidence for the interviewee’s perspective

9. To what extent has the provision you have offered proved relevant to the needs of the young people you have worked with and why?

  • How have you sought to vary your provision to meet the needs of particular groups or individuals? How effective has this been and in what ways?
  • Is there any provision you feel would have been relevant to the young people you have worked with that you weren’t able to offer? If so, what and why?

10. Can you provide any specific examples of what you regard as particularly effective practice in supporting the young people you have worked with to move closer to the labour market?

  • Are there any specific examples of what you feel has been effective with particular groups or individuals? If so, what are these and for which groups / types of young people?

11. What barriers or challenges have you faced in implementing your YEI project?

  • How were these addressed and what learning can be drawn from this to inform future policy and interventions?

12. In what ways have the governance and partnership working aspects of your project helped or hindered its effectiveness and why?

  • Since the start of the project, have any changes been made to governance or partnership working arrangements and processes to make them more effective? If so, what, and what has the impact of these changes been?

13. What are the main lessons that can be learned from delivery that might inform future initiatives, policy or projects aiming to support NEET young people?

14. How would you describe the overall contribution of the YEI to addressing the challenges posed in respect of NEET young people in the local area?

  • Probe on the reasons for the interviewee’s response

Part Four: Efficiency of provision

15. In terms of the procurement process for YEI provision, to what extent do you feel this ensured value-for-money and why?

  • Probe on the degree to which the process ensured that outputs are likely to be achieved for the minimum possible cost, use of cost benchmarks, unit costs etc.

16. Within your project, how have you sought to ensure that your provision is being delivered efficiently and provides value-for-money?

  • Probe for examples of how the provider has sought to deliver efficiently and provide value-for-money

17. Do you have a sense of which types of activity or intervention are proving to be most cost effective and why? Equally, those which are proving less cost effective and why?

  • Prompt and probe using knowledge of the project’s interventions and the list of activities in the theory of change (training; volunteering; support to access apprenticeships; wraparound support; careers guidance; self-employment/enterprise support; financial support to address barriers; post- entry to employment / training support)

Part Five: Outcomes and impacts from the YEI

  1. How would you describe the level and types of impact your project has had on young people in the following areas (probe for evidence and examples):
  • interpersonal skills, attitudes and behaviours
  • basic and transversal skills, including employability skills
  • reduced barriers to re-engaging in work and learning
  • enhanced access to and competitiveness in the labour market
  • improved access to apprenticeships and traineeships
  • improved understanding of support services available
  • improved health and wellbeing, including more self-sufficiency and empowerment
  • greater financial security and earning power?
  • In what areas would you say these outcomes and impacts are most apparent and why?
  • Do outcomes and impacts tend to be less apparent in any of the above areas and why?

19. Have there been any unintended or unexpected outcomes for the young people you have supported? If so, what?

  • Equally, have there been any knock-on effects from support – e.g. on participants’ peer groups, families etc. (probe for specific examples)?

20. What outcomes have resulted from delivering YEI provision for your organisation?

  • Probe on both positive and negative outcomes and how these are apparent

21. (Ask as applicable) What outcomes would you say there have been for your delivery partners, or for the delivery partnership as a whole, as a result of delivering YEI provision?

  • Probe on both positive and negative outcomes and how these are apparent
  • Probe on whether new or improved partnerships have resulted, including partnerships with Jobcentre Plus, other public bodies, local services etc.

22. How would you describe any outcomes for local employers that have resulted from project activity, both employers engaged specifically and local employers in general?

  • Probe on both positive and negative outcomes and how these are apparent

23. Would you say there have been any outcomes or impacts on the local community stemming from the delivery of YEI provision? If so, what have these been and how are they evident?

24. How would you describe the impact of the YEI in terms of reducing NEET levels and youth unemployment locally?

  • Probe on reasons for the response, evidence and examples

25. How would you describe the contribution of your project to meeting the objectives of the local LEP strategy?

26. Have there been any unintended or unexpected outcomes resulting from your project as a whole? If so, what?

Part Six: Closing / overall reflections

27. Is there any learning from the YEI programme that can inform future approaches to supporting NEET young people, either locally or at the national level, and, if so, what?

28. Do you have any final thoughts or reflections concerning your project or the YEI in general in terms of its relative effectiveness, efficiency and impact?

8.4. Lead Provider – Delivery staff

Part One: Introduction / background information on role

1. Please outline your role in relation to the YEI within [insert provider name].

  • Cover main elements of role in relation to YEI: period, timing and nature of involvement

Part Two: Effectiveness of targeting

2. How far, and in what ways, do you feel your project effectively targets local young people most in need of support?

  • How far, and in what ways, does this targeting draw on local data / intelligence – e.g. from the LEP, knowledge within your organisation, partners’ knowledge etc.
  • In what (other) ways would you say your project has been effective in its targeting?
  • Are there any challenges you have faced in effectively targeting those young people in need of support locally, and how have you sought to address these?

3. To what extent have the specific target groups your project aimed to support been reached as planned?

  • (focus on specific target groups the project focuses on: e.g. NEET population in general; young lone parents; looked after children and care leavers; young carers; ex-offenders; those involved in gangs; and young people with learning difficulties and disabilities; unemployed or inactive graduates).
  • To what extent has your project targeted those groups identified as priorities locally – e.g. through LEP strategies?
  • To what extent has your project targeted specific groups as opposed to targeting the NEET population in general, and why?
  • Are there any of the target groups your project sought to engage that have proved challenging, or you haven’t been able to reach effectively, and why?

4. How do you secure buy-in from participants and motivate them to engage with provision?

  • What effect does the voluntary nature of provision have on this?
  • Does the lack of conditionality, in terms of participants being mandated to engage as a condition of benefit receipt for example, affect your ability to engage and motivate participants? If so, how and in what ways?

5. How far would you say your project has effectively reached young people at risk of social exclusion and those from marginalised communities?

  • Probe on the reasons and evidence for the interviewee’s response

6. How far would you say your project has effectively reached NEET young people with complex needs and multiple disadvantages?

  • Probe on the reasons and evidence for the interviewee’s response

Part Three: Effectiveness of the provision

7. When delivering the YEI, which types of provision or activities have been effective and why?

  • Are there specific approaches or activities that have been particularly effective with certain groups (e.g. the most marginalised; those with disabilities etc.)? If so, which activities and groups and how is this evident?

8. Equally, are there types of provision or activities that have proved less effective than anticipated and why?

  • Did effectiveness vary between groups (e.g. certain interventions being less effective for particular groups)? If so, how and in what ways?
  • Probe for evidence for the interviewee’s perspective

9. To what extent has the support you have offered proved relevant to the needs of the young people you have worked with and why?

  • How have you sought to vary the support offered to meet the needs of particular groups or individuals? How effective has this been and in what ways?
  • Is there any support you feel would have been relevant to the young people you have worked with that you weren’t able to offer? If so, what and why?

10. Can you provide any specific examples of what you regard as particularly effective practice in supporting the young people you have worked with to move closer to the labour market?

  • Are there any specific examples of what you feel has been effective with particular groups or individuals? If so, what are these and for which groups / types of young people?

11. What barriers or challenges have you faced in delivering support to the young people you work with?

  • How were these addressed and what learning can be drawn from this to inform future policy and interventions?

12. In what ways have the governance and partnership working aspects of your project helped or hindered its effectiveness and why?

  • Since the start of the project, have any changes been made to governance or partnership working arrangements and processes to make them more effective? If so, what, and what has the impact of these changes been?

13. What are the main lessons that can be learned from delivery that might inform future initiatives, policy or projects aiming to support NEET young people?

14. How would you describe the overall contribution of the YEI to addressing the challenges posed in respect of NEET young people in the local area?

  • Probe on the reasons for the interviewee’s response

Part Four: Efficiency of provision

15. Within your project, how have you sought to ensure that your provision is being delivered efficiently and provides value-for-money?

  • Probe for examples of how the provider has sought to deliver efficiently and provide value-for-money

16. Do you have a sense of which types of activity or intervention are proving to be most cost effective and why? Equally, those which are proving less cost effective and why?

  • Prompt and probe using knowledge of the project’s interventions and the list of activities in the theory of change (training; volunteering; support to access apprenticeships; wraparound support; careers guidance; self-employment/enterprise support; financial support to address barriers; post- entry to employment / training support)

Part Five: Outcomes and impacts from the YEI

17. How would you describe the level and types of impact your project has had on young people in the following areas (probe for evidence and examples):

  • interpersonal skills, attitudes and behaviours
  • basic and transversal skills, including employability skills
  • reduced barriers to re-engaging in work and learning
  • enhanced access to and competitiveness in the labour market
  • improved access to apprenticeships and traineeships
  • improved understanding of support services available
  • improved health and wellbeing, including more self-sufficiency and empowerment
  • greater financial security and earning power?
  • In what areas would you say these outcomes and impacts are most apparent and why?
  • Do outcomes and impacts tend to be less apparent in any of the above areas and why?

18. Have there been any unintended or unexpected outcomes for the young people you have supported? If so, what?

  • Equally, have there been any knock-on effects from support – e.g. on participants’ peer groups, families etc. (probe for specific examples)?

19. What outcomes have resulted from delivering YEI provision for your organisation?

  • Probe on both positive and negative outcomes and how these are apparent

20. (Ask as applicable) What outcomes would you say there have been for your delivery partners, or for the delivery partnership as a whole, as a result of delivering YEI provision?

  • Probe on both positive and negative outcomes and how these are apparent
  • Probe on whether new or improved partnerships have resulted, including partnerships with Jobcentre Plus, other public bodies, local services etc.

21. How would you describe any outcomes for local employers that have resulted from project activity, both employers engaged specifically and local employers in general?

  • Probe on both positive and negative outcomes and how these are apparent

22. Would you say there have been any outcomes or impacts on the local community stemming from the delivery of YEI provision? If so, what have these been and how are they evident?

23. How would you describe the impact of the YEI in terms of reducing NEET levels and youth unemployment locally?

  • Probe on reasons for the response, evidence and examples

24. If you are familiar with the local LEP strategy, how would you describe the contribution of your project to meeting its objectives?

25. Have there been any unintended or unexpected outcomes resulting from your project as a whole? If so, what?

Part Six: Closing / overall reflections

26. Is there any learning from the YEI programme that can inform future approaches to supporting NEET young people, either locally or at the national level, and, if so, what?

27. Do you have any final thoughts or reflections concerning your project or the YEI in general in terms of its relative effectiveness, efficiency and impact?

8.5. Delivery partner: Manager / Strategic level

Part One: Introduction / background information on role

1. Please outline your role in relation to the YEI within [insert provider name].

  • Cover main elements of role in relation to YEI: period, timing and nature of involvement

Part Two: Effectiveness of targeting

2. How far, and in what ways, do you feel your delivery has effectively targeted local young people most in need of support?

  • How far, and in what ways, does this targeting draw on local data / intelligence – e.g. from the LEP, your organisation, lead and other delivery partners’ knowledge etc.
  • In what (other) ways would you say your delivery has been effective in its targeting?
  • Are there any challenges you have faced in effectively targeting young people in need of support locally, and how have you sought to address these?

3. To what extent have the specific target groups your organisation aimed to support been reached as planned?

  • (focus on specific target groups the organisation focuses on: e.g. NEET population in general; young lone parents; looked after children and care leavers; young carers; ex-offenders; those involved in gangs; and young people with learning difficulties and disabilities; unemployed or inactive graduates).
  • To what extent has your organisation, and the wider project you are involved in, targeted those groups identified as priorities locally – e.g. through LEP strategies etc.?
  • To what extent has your organisation and the wider project targeted specific groups as opposed to targeting the NEET population in general, and why?
  • Are there any of the target groups you sought to engage that have proved challenging, or you haven’t been able to reach effectively, and why?

4. How do you secure buy-in from participants and motivate them to engage with provision?

  • What effect does the voluntary nature of provision have on this?
  • Does the lack of conditionality, in terms of participants being mandated to engage as a condition of benefit receipt for example, affect your ability to engage and motivate participants? If so, how and in what ways?

5. How far would you say your organisation, and your project as a whole, have effectively reached young people at risk of social exclusion and those from marginalised communities?

  • Probe on the reasons and evidence for the interviewee’s response

6. How far would you say your organisation, and your project as a whole, have effectively reached NEET young people with complex needs and multiple disadvantages?

  • Probe on the reasons and evidence for the interviewee’s response

Part Three: Effectiveness of the provision

7. As part of your YEI delivery, which types of provision or activities have been effective and why?

  • Are there specific approaches or activities that have been particularly effective with certain groups (e.g. the most marginalised; those with disabilities etc.)? If so, which activities and groups and how is this evident?

8. Equally, are there types of provision or activities that have proved less effective than anticipated and why?

  • Did effectiveness vary between groups (e.g. certain interventions being less effective for particular groups)? If so, how and in what ways?
  • Probe for evidence for the interviewee’s perspective

9. To what extent has the provision you have offered proved relevant to the needs of the young people you have worked with and why?

  • How have you sought to vary your provision to meet the needs of particular groups or individuals? How effective has this been and in what ways?
  • Is there any provision you feel would have been relevant to the young people you have worked with that you weren’t able to offer? If so, what and why?

10. Can you provide any specific examples of what you regard as particularly effective practice in supporting the young people you have worked with to move closer to the labour market?

  • Are there any specific examples of what you feel has been effective with particular groups or individuals? If so, what are these and for which groups / types of young people?

11. What barriers or challenges have you faced in delivering your YEI provision?

  • How were these addressed and what learning can be drawn from this to inform future policy and interventions?

12. In what ways have the governance and partnership working aspects of your project helped or hindered its effectiveness and why?

  • Since the start of the project, have any changes been made to governance or partnership working arrangements and processes to make them more effective? If so, what, and what has the impact of these changes been?

13. What are the main lessons that can be learned from your delivery that might inform future initiatives, policy or projects aiming to support NEET young people?

14. How would you describe the overall contribution of the YEI to addressing the challenges posed in respect of NEET young people in the local area?

  • Probe on the reasons for the interviewee’s response

Part Four: Efficiency of provision

15. As part of your delivery, how have you sought to ensure that your provision is being delivered efficiently and provides value-for-money?

  • Probe for examples of how the provider has sought to deliver efficiently and provide value-for-money

16. Do you have a sense of which types of activity or intervention are proving to be most cost effective and why? Equally, those which are proving less cost effective and why?

  • Prompt and probe using knowledge of the project’s interventions and the list of activities in the theory of change (training; volunteering; support to access apprenticeships; wraparound support; careers guidance; self-employment/enterprise support; financial support to address barriers; post- entry to employment / training support)

Part Five: Outcomes and impacts from the YEI

17. How would you describe the level and types of impact your project has had on young people in the following areas (probe for evidence and examples):

  • interpersonal skills, attitudes and behaviours
  • basic and transversal skills, including employability skills
  • reduced barriers to re-engaging in work and learning
  • enhanced access to and competitiveness in the labour market
  • improved access to apprenticeships and traineeships
  • improved understanding of support services available
  • improved health and wellbeing, including more self-sufficiency and empowerment
  • greater financial security and earning power?
    • In what areas would you say these outcomes and impacts are most apparent and why?
    • Do outcomes and impacts tend to be less apparent in any of the above areas and why?

18. Have there been any unintended or unexpected outcomes for the young people you have supported? If so, what?

  • Equally, have there been any knock-on effects from support – e.g. on participants’ peer groups, families etc. (probe for specific examples)?

19. What outcomes have resulted from delivering YEI provision for your organisation?

  • Probe on both positive and negative outcomes and how these are apparent

20. (Ask as applicable) What outcomes would you say there have been for your delivery partners, or for the delivery partnership as a whole, as a result of delivering YEI provision?

  • Probe on both positive and negative outcomes and how these are apparent
  • Probe on whether new or improved partnerships have resulted, including partnerships with Jobcentre Plus, other public bodies, local services etc.

21. How would you describe any outcomes for local employers that have resulted from project activity, both employers engaged specifically and local employers in general?

  • Probe on both positive and negative outcomes and how these are apparent

22. Would you say there have been any outcomes or impacts on the local community stemming from the delivery of YEI provision? If so, what have these been and how are they evident?

23. How would you describe the impact of the YEI in terms of reducing NEET levels and youth unemployment locally?

  • Probe on reasons for the response, evidence and examples

24. If you are familiar with the local LEP strategy, how would you describe the contribution of your project to meeting its objectives?

25. Have there been any unintended or unexpected outcomes resulting from your delivery of the YEI as a whole? If so, what?

Part Six: Closing / overall reflections

26. Is there any learning from the YEI programme that can inform future approaches to supporting NEET young people, either locally or at the national level, and, if so, what?

27. Do you have any final thoughts or reflections concerning your delivery or the YEI in general in terms of its relative effectiveness, efficiency and impact?

8.6. Delivery partner: Delivery staff

Part One: Introduction / background information on role

1. Please outline your role in relation to the YEI within [insert provider name].

  • Cover main elements of role in relation to YEI: period, timing and nature of involvement

Part Two: Effectiveness of targeting

2. How far, and in what ways, do you feel your delivery of the YEI has effectively targeted local young people most in need of support?

  • How far, and in what ways, does this targeting draw on local data / intelligence – e.g. from the LEP, your organisation, lead and other delivery partners’ knowledge etc.
  • In what (other) ways would you say your delivery has been effective in its targeting?
  • Are there any challenges you have faced in effectively targeting young people in need of support locally, and how have you sought to address these?

3. To what extent have the specific target groups your organisation aimed to support been reached as planned?

  • (focus on specific target groups the organisation focuses on: e.g. NEET population in general; young lone parents; looked after children and care leavers; young carers; ex-offenders; those involved in gangs; and young people with learning difficulties and disabilities; unemployed or inactive graduates).
  • To what extent has your organisation, and the wider project you are involved in, targeted those groups identified as priorities locally – e.g. through LEP strategies etc.?
  • To what extent has your organisation and the wider project targeted specific groups as opposed to targeting the NEET population in general, and why?
  • Are there any of the target groups you sought to engage that have proved challenging, or you haven’t been able to reach effectively, and why?

4. How do you secure buy-in from participants and motivate them to engage with provision?

  • What effect does the voluntary nature of provision have on this?
  • Does the lack of conditionality, in terms of participants being mandated to engage as a condition of benefit receipt for example, affect your ability to engage and motivate participants? If so, how and in what ways?

5. How far would you say your organisation, and your project as a whole, have effectively reached young people at risk of social exclusion and those from marginalised communities?

  • Probe on the reasons and evidence for the interviewee’s response

6. How far would you say your organisation, and your project as a whole, have effectively reached NEET young people with complex needs and multiple disadvantages?

  • Probe on the reasons and evidence for the interviewee’s response

Part Three: Effectiveness of the provision

7. When delivering the YEI, which types of provision or activities have been effective and why?

  • Are there specific approaches or activities that have been particularly effective with certain groups (e.g. the most marginalised; those with disabilities etc.)? If so, which activities and groups and how is this evident?

8. Equally, are there types of provision or activities that have proved less effective than anticipated and why?

  • Did effectiveness vary between groups (e.g. certain interventions being less effective for particular groups)? If so, how and in what ways?
  • Probe for evidence for the interviewee’s perspective

9. To what extent has the support you have offered proved relevant to the needs of the young people you have worked with and why?

  • How have you sought to vary the support offered to meet the needs of particular groups or individuals? How effective has this been and in what ways?

  • Is there any support you feel would have been relevant to the young people you have worked with that you weren’t able to offer? If so, what and why?

10. Can you provide any specific examples of what you regard as particularly effective practice in supporting the young people you have worked with to move closer to the labour market?

  • Are there any specific examples of what you feel has been effective with particular groups or individuals? If so, what are these and for which groups / types of young people?

11. What barriers or challenges have you faced in delivering support to the young people you work with?

  • How were these addressed and what learning can be drawn from this to inform future policy and interventions?

12. In what ways have the governance and partnership working aspects of your project helped or hindered its effectiveness and why?

  • Since the start of the project, have any changes been made to governance or partnership working arrangements and processes to make them more effective? If so, what, and what has the impact of these changes been?

13. What are the main lessons that can be learned from delivery that might inform future initiatives, policy or projects aiming to support NEET young people?

14. How would you describe the overall contribution of the YEI to addressing the challenges posed in respect of NEET young people in the local area?

  • Probe on the reasons for the interviewee’s response

Part Four: Efficiency of provision

15. Within your organisation, how have you sought to ensure that YEI provision is being delivered efficiently and provides value-for-money?

  • Probe for examples of how the provider has sought to deliver efficiently and provide value-for-money

16. Do you have a sense of which types of activity or intervention are proving to be most cost effective and why? Equally, those which are proving less cost effective and why?

  • Prompt and probe using knowledge of the project’s interventions and the list of activities in the theory of change (training; volunteering; support to access apprenticeships; wraparound support; careers guidance; self-employment/enterprise support; financial support to address barriers; post- entry to employment / training support)

Part Five: Outcomes and impacts from the YEI

17. How would you describe the level and types of impact your support has had on young people in the following areas (probe for evidence and examples):

  • interpersonal skills, attitudes and behaviours
  • basic and transversal skills, including employability skills
  • reduced barriers to re-engaging in work and learning
  • enhanced access to and competitiveness in the labour market
  • improved access to apprenticeships and traineeships
  • improved understanding of support services available
  • improved health and wellbeing, including more self-sufficiency and empowerment
  • greater financial security and earning power?
    • In what areas would you say these outcomes and impacts are most apparent and why?
    • Do outcomes and impacts tend to be less apparent in any of the above areas and why?

18. Have there been any unintended or unexpected outcomes for the young people you have supported? If so, what?

  • Equally, have there been any knock-on effects from support – e.g. on participants’ peer groups, families etc. (probe for specific examples)?

19. What outcomes have resulted for your organisation from delivering YEI provision?

  • Probe on both positive and negative outcomes and how these are apparent

20. (Ask as applicable) What outcomes would you say there have been for your delivery partners, or for the delivery partnership as a whole, as a result of delivering YEI provision?

  • Probe on both positive and negative outcomes and how these are apparent
  • Probe on whether new or improved partnerships have resulted, including partnerships with Jobcentre Plus, other public bodies, local services etc.

21. How would you describe any outcomes for local employers that have resulted from project activity, both employers engaged specifically and local employers in general?

  • Probe on both positive and negative outcomes and how these are apparent

22. Would you say there have been any outcomes or impacts on the local community stemming from the delivery of YEI provision? If so, what have these been and how are they evident?

23. How would you describe the impact of the YEI in terms of reducing NEET levels and youth unemployment locally?

  • Probe on reasons for the response, evidence and examples

24. If you are familiar with the local LEP strategy, how would you describe the contribution of your project to meeting its objectives?

25. Have there been any unintended or unexpected outcomes resulting from your delivery of the YEI as a whole? If so, what?

Part Six: Closing / overall reflections

26. Is there any learning from the YEI programme that can inform future approaches to supporting NEET young people, either locally or at the national level, and, if so, what?

27. Do you have any final thoughts or reflections concerning your delivery or the YEI in general in terms of its relative effectiveness, efficiency and impact?

8.7. Participants

Part One: Introduction / background information

1. Can you tell me a little bit about your background and how you became aware of and got involved with [insert project / organisation name as applicable]?

2. What encouraged you to become involved with [insert project / organisation name as applicable] and what did you hope to get out of being involved?

  • Were there any issues or challenges for you that you hoped the project could help with?
  • Probe on whether the individual knew it was an European Social Fund supported project, either initially or later on, and how they became aware (e.g. from project staff as part of their first meeting when becoming involved, because of posters on display / the use of the logo on documents, at a later stage etc.)

Part Two: Support received

3. How well did the process of becoming involved with the project work?

4. Since you first started at [insert project / organisation name as applicable] what kinds of advice or support have you received?

  • Talk through the different activities and types of support involved

5. How helpful do you think the advice and support you have received has been and why?

  • Prompt as needed: Did the advice and support match with what you were hoping to get from [insert project / organisation name as applicable]? Why / why not?

6. Which parts of this advice or support were most helpful and why?

7. Were there any parts of this advice and support that were less helpful?

  • If so, why?

8. Have you received any similar support in the past – for example, around helping to move towards work, learn new skills, or gain qualifications?

  • What was this and what did it involve? [check whether this was through Jobcentre Plus – e.g. Work Programme, Work Choices, NEA, Work Experience]

9. How does [insert project / organisation name as applicable] compare to this other support?

10. Thinking about your experience at [insert project / organisation name as applicable] as a whole, are there any ways in which this could have been better?

  • Prompt as appropriate: Are there any kinds of advice or support that would have been helpful that you haven’t received, or that wasn’t available?

Part Three: Outcomes of support

11. [Refer to any issues or challenges for the participant that were noted at the start of the interview] How far has the project helped with these?

12. More generally, what would you say the main results from your involvement with [insert project / organisation name as applicable] have been?

Ask as open question, then prompt / probe around the following possible/achieved outcomes (e.g. what effect these outcomes have had for the young person etc.):

  • Learning new skills
  • Achieving a qualification
  • Different attitude – to work, training or life in general
  • More confidence, in general and/or in relationships with others
  • Feeling better in general and/or improved health
  • More likely to start a new course / start training
  • More likely to start an apprenticeship
  • More confident in finding work and/or better idea of the type of work wanted
  • Starting a new course or traineeship
  • Starting an apprenticeship
  • Finding a new job and / or starting work
  • More financial security
  • Better prospects for the future

13. What has been the most important thing to come out of getting involved with [insert project / organisation name as applicable] and why?

14. [Ask of those still receiving support] Do you think the support and advice received will help you move closer to getting a job or start a new course, traineeship or apprenticeship in future?

  • In what ways has it helped and to what extent?

15. [Ask of those in work or training] How important do you think the advice and support you had was in helping you to find and start work / start a course / apprenticeship / training

  • How or in what ways did the advice and support help?

16. [Ask of those progressing into work or training] What has starting a new job / new course / apprenticeship / training [as applicable] meant for you as an individual?

17. Do you have any other comments or final thoughts about [insert project / organisation name as applicable]?

  1. See European Social Fund England Operational Programme 2014-2020 

  2. Atkinson, I. et.al, (2017) Youth Employment Initiative Process Evaluation: Assessment of Strategic Fit, Design and Implementation, DWP Research Report 995 

  3. ‘Fiscal return’ refers to an assessment of costs and benefits as measured from the point of view of the Exchequer, including changes in benefit and tax credit payments, taxes received and programme operational costs. ‘Social return’ refers to an assessment of costs and benefits, such as changes in income, stemming from the impacts of employment programmes on different individuals and groups in society. 

  4. Through England’s ESF Operational Programme (OP), flexibility to deliver YEI support to those aged up to 29 was provided to areas eligible for YEI funding. Eligible areas are determined at the level of Nomenclature of Units for Territorial Statistics (NUTS) areas, NUTS being a standard developed and regulated by the EU in order to reference the sub-division of countries for statistical purposes. There are four NUTS 2 regions in England eligible for YEI funding: Inner London, Merseyside, Tees Valley & Durham, and West Midlands. In addition, the following NUTS3 areas are eligible: Leicester, Nottingham, Kingston upon Hull, and Thurrock. 

  5. DWP (2015), European Social Fund Operational Programme 2014-2020 

  6. Ibid. 

  7. Atkinson, I. et.al, (2017) Youth Employment Initiative Process Evaluation: Assessment of Strategic Fit, Design and Implementation, DWP Research Report 995 

  8. Ibid., see Appendix A. 

  9. The ESF Leavers Survey seeks to help assess the effectiveness of the delivery of the ESF Programme in England. The survey is commissioned by DWP and is being conducted by IFF Research over a series of waves. Interim results from the Leavers’ Survey presented throughout this report are based on responses from 2,213 YEI participants in the first nine waves of the survey, conducted between 2017 and 2019. Survey respondents were interviewed at least six months after they left a YEI course or support programme. Results have been weighted using YEI Management Information to be representative of the overall YEI participant population. 

  10. The MI data analysed in this report covers the period until September 2019. It should be noted that at the time of reporting the YEI programme is still in its delivery phase so the final figures will differ from those presented in this report. The data also includes only records that have been validated according to the requirements concerning ESF participant data set by the European Commission. 

  11. Figure derived by combining targets for the under 25 and over 25 age groups presented in the OP 

  12. Full definitions of disadvantage are included in DWP (2018), 2014-2020 European Growth Programme: Output and Result Indicator Definitions Guidance for the European Social Fund. In summary, those recorded as having a disadvantage fall into one or more of the following groups: participants who live in jobless households; participants who live in a single adult household with dependent children; migrants, people with a foreign background, minorities (including marginalised communities such as the Roma); participants with disabilities; and, ‘other disadvantaged’ (which includes those who are homeless, those lacking basic skills and those without qualifications). Throughout this report, where disadvantage is referred to in the context of the YEI MI, the above represent the ‘disadvantaged groups’ concerned. 

  13. Black and minority ethnic (BAME) status includes White Gypsy, Irish Traveller, or Roma. 

  14. This figure of eight per cent only relates to those who perceived that such caring responsibilities would limit the work they could do. 

  15. As with all case studies and specific details of participants provided in this report, the actual name of the participant concerned has been changed to help protect anonymity. 

  16. European Commission (2015), Guidance on the evaluation of the Youth Employment Initiative. 

  17. See section 4.1.2 along with the technical annex accompanying this report for further details. 

  18. As explained in the section on ‘study limitations’ (1.3.1), available data meant that it was necessary to restrict the analysis to employment outcomes, rather than also focusing on education and training outcomes as was initially considered. 

  19. PSM is a statistical technique used to estimate the impact of an intervention on a set of specific outcomes. It mimics an experimental research design by comparing outcomes for a treatment group and a statistically generated comparison group, which is similar to the treatment group in its composition. 

  20. Namely, East Midlands, East of England, North West, North East, West Midlands, Yorkshire and the Humber. 

  21. The 3,276 YEI participants available for analysis were drawn from YEI MI records. Fewer participants than the total recorded could be used for the analysis given the requirement for each participant to be matched to DWP’s CIS to enable linking to the other datasets used. 

  22. The rationale for using employment and benefit histories in this way was to seek to exclude people who may not be looking for work, and hence not likely to enter the labour market, due to, for example, being in full time education in the periods in question. 

  23. Pseudo start dates were assigned to all individuals in the comparator pool so that pre-intervention employment/benefit histories could be constructed (and subsequently used in the process of developing matched treatment and comparison groups). To facilitate this, pseudo start dates were generated to mirror the distribution of start dates for the treatment group and then allocated at random to individuals in the comparator pool. 

  24. Spells claiming UC were not used within this criterion, given that roll-out of the benefit was underway during the periods of interest for the analysis and the fact that UC claimants may be in work.  2

  25. Difference in differences (DiD) analysis estimates the effect of a treatment on an outcome of interest, which in the case of YEI was the number of days in employment, by comparing the average change over time in the outcome for the treatment group, relative to the average change over time for the control/comparison group. 

  26. MI figures presented are from available data as at September 2019. As the YEI is still being delivered, and data being collated at the time of writing, these figures are subject to change. 

  27. It should be noted that the MI records participant status at the point of finishing support. This partly explains why the MI figure for those in employment on leaving is lower than that reported in the Leavers Survey data for those in employment at the six-month point after leaving, as cited later in this section. This is likely to relate to time taken for the employment and training support received by YEI participants to translate into employment outcomes for them. 

  28. Full definitions of disadvantage are included in DWP (2018), 2014-2020 European Growth Programme: Output and Result Indicator Definitions Guidance for the European Social Fund. In summary, those recorded as having a disadvantage fall into one or more of the following groups: participants who live in jobless households; participants who live in a single adult household with dependent children; migrants, people with a foreign background, minorities (including marginalised communities such as the Roma); participants with disabilities; and, ‘other disadvantaged’ (which includes those who are homeless, those lacking basic skills and those without qualifications). Throughout this report, where disadvantage is referred to in the context of the YEI MI, the above represent the ‘disadvantaged groups’ concerned. 

  29. Data from the Labour Force Survey (LFS), February 2020 release, using the Government Statistical Service (GSS) harmonised standard definition of disability comparing those meeting this definition with those who did not (excluding those who did not state their health situation). 

  30. See Appendix A for the full theory of change, setting out anticipated broader outcomes for participants, including: improved interpersonal and basic skills; improved behaviours and attitudes; reduced barriers to re-engagement in work and learning; improved access to apprenticeships and traineeships; improved understanding of support services available; enhanced access to, and competitiveness in, the labour market; improved health and wellbeing, including more self-sufficiency and empowerment; and greater financial security and earning power. 

  31. Office for National Statistics (ONS) (2019), NEET statistics annual brief 2018 (Headline NEET estimates from the Labour Force Survey by age and official age 16-18 estimates from Participation in Education, Employment and Training) 

  32. NUTS refers to Nomenclature of Units for Territorial Statistics, a standard developed and regulated by the EU in order to reference the sub-division of countries for statistical purposes. 

  33. Figures from the ONS NEET statistics annual brief for 2018 show that the whole regions in which the largest YEI-delivery (NUTS 2) areas lie (North East, North West, West Midlands) only have an estimated 217,000 NEET young people in total. YEI delivery areas are only a sub-set of these overall regions. Even accounting for additional NEET young people in the smaller NUTS 3 areas, it is unlikely that NEET levels are more than 150,000 therefore, and the numbers concerned may well be considerably lower than this. 

  34. Fujiwara, D. (2010), The Department for Work and Pensions Social Cost-Benefit Analysis framework: Methodologies for estimating and incorporating the wider social and economic impacts of work in Cost-Benefit Analysis of employment programmes, DWP Working Paper No.86 

  35. For a discussion, see Ward, R. et.al. (2016) Sector Based Work Academies: A Quantitative Impact Assessment, DWP Research Report No.916, pp.36 

  36. Ibid., p.37 

  37. Fiscal return’ refers to an assessment of costs and benefits as measured from the point of view of the Exchequer, including changes in benefit and tax credit payments, taxes received and programme operational costs. ‘Social return’ refers to an assessment of costs and benefits, such as changes in income, stemming from the impacts of employment programmes on different individuals and groups in society. 

  38. Range given is based on sensitivity analysis of the estimated impacts.