Guidance

UKSPF: intervention-level evaluation feasibility report - executive summary

Updated 4 March 2024

Learning ‘what works’ from UK Shared Prosperity Fund interventions

The UK Shared Prosperity Fund (UKSPF) was launched by the UK government in April 2022 and is a central pillar of the government’s Levelling Up agenda. It provides £2.6 billion of funding for local investment by March 2025, with all areas of the UK receiving an allocation from the fund via a funding formula rather than a competition.^{[footnote 1]} UKSPF funding is allocated straight to local areas to invest in 3 priorities: People & Skills; Business Support; and Communities & Place.

UKSPF provides an opportunity to learn about the contribution local interventions make to Levelling Up, with a particular focus on:^{[footnote 2]}

• Pride in place: local perspectives about high streets and regeneration; culture, heritage and sport; community and society; and safety and security; and
• Life chances: education and skills; local economic and social environment; health and wellbeing; childhood and family; and crime and anti-social behaviour outcomes.

The Department for Levelling Up, Housing and Communities (DLUHC) has therefore designed a 3-tier UKSPF evaluation strategy. This will include: a programme-level evaluation, which will explore the overall impact and value for money of the UKSPF; a place based-level evaluation, which will produce 36 place-based case studies across the UK to provide evidence on how combinations of UKSPF-supported interventions within a locality work together; and an intervention-level evaluation, which will focus on generating robust evidence on what interventions work, or do not work, for whom and why, in delivering pride in place and life chances. The evaluation activity is currently expected to conclude by late 2025. However, DLUHC is exploring possibilities to extend the evaluation beyond 2025 to ensure a robust and comprehensive capture of the impacts.

This Feasibility Report has been prepared by Frontier Economics and BMG Research and relates to the intervention-level tier.^{[footnote 3]} The scope of this evaluation includes:

• A process evaluation: which will explore how learning can be generated about the design and planning; implementation and management; and monitoring and evaluation of sampled interventions;
• An impact evaluation: which will explore the extent to which changes in key outcomes of interest have been enabled by the interventions, for whom and under what conditions; and
• A value for money evaluation: which will explore the extent to which the interventions have made best use of public resources in terms of meeting local strategic objectives and delivering local benefits that exceed costs.

Focus of this feasibility report

This document presents the outcome of an in-depth scoping exercise to determine how the UKSPF interventions can be evaluated in line with the Magenta Book (HMT, 2020). The aim of the feasibility work was to design a proportionate evaluation that would maximise learning.

The intervention-level evaluation approach presented in this Feasibility Report has been designed under the following guiding principles: proportionality, co-creation, flexibility, inclusivity, practicality and rigour. The proposed approach seeks to find a balance between ensuring that lead local authorities (i.e. the authorities which have been allocated UKSPF funding) have the opportunity to engage with the evaluation activities and avoiding overburdening local areas as they deliver their UKSPF interventions.

As there are thousands of UKSPF-funded projects being delivered across the country, it is not proportionate (or possible) to evaluate all projects in a way which delivers sufficiently detailed evidence of what works, for whom and under what conditions. Nor would a random sample be appropriate as this could miss valuable opportunities to generate robust evaluation evidence. The approach therefore is founded on identifying groups of similar projects to focus on that will enable robust evaluation to be undertaken. Those groups of projects are, for the purposes of this evaluation, called intervention study groups because each study group represents a particular type of intervention.

As there are various definitions of what an ‘intervention’ is, for the purposes of this particular intervention-level evaluation, the term ‘intervention’ refers to projects that share similar core characteristics in terms of their outputs (tangible deliverables from the investment), intended beneficiaries and the mechanisms through which they are intended to bring about changes in outcomes. For this intervention-level evaluation, similar projects can be located anywhere in the country but share these core characteristics.^{[footnote 4]}

Conducting the evaluation at the study group level (where each study group represents a particular type of intervention), rather than focusing on individual projects, provides the opportunity to ‘pool’ the evidence from the projects within the study group to enhance learning about the important contextual factors that can affect what works, for whom and why. A theory of change, evaluation questions, indicators of change and metrics to measure those changes have all been clearly articulated for each study group.

Appropriate analytical methods to apply to assess the changes in outcomes attributable to projects within the study groups have also been developed.

Study groups were iteratively identified and refined based on information collated for this evaluation on UKSPF-supported projects.^{[footnote 5]} Selection criteria enabled a list of ten study groups to be recommended, primarily focusing on the extent to which existing gaps in evidence could be addressed as well as ensuring that a rigorous and proportionate evaluation could be conducted over the timescales for this evaluation.

The 10 study groups selected for the intervention-level evaluation are summarised in Figure 1. For the purposes of this evaluation, three local projects will be grouped together to form each study group. Therefore, evaluation evidence will be generated by drawing on 30 projects across the ten study groups. This number of projects was intended to balance the need to include enough projects to enable variations in context to be explored, while also ensuring that sufficient granular data could be proportionately collected on each project. The feasibility stage has gathered evidence on the likely projects within each group, although the exact projects may need to be slightly adapted over time if lead local authorities adjust their portfolio of projects.^{[footnote 6]} These study groups and the projects within them are not chosen to be able to make representative findings across the UKSPF as a whole but to add to the evidence base on what works, for whom and under what conditions.

Figure 1: The 10 study groups for the intervention-level evaluation

Alt text: The diagram above shows and briefly describes the ten study groups.

For the People & Skills interventions, the study groups are:

1. Helping local economically inactive people into employment
2. Helping local economically inactive young adults into employment (excl. through volunteering)
3. Helping local economically inactive young adults into employment through volunteering
4. Involving local businesses in helping local economically inactive people into employment

For the Business Support, the study groups are:

1. Supporting the digital development of local businesses
2. Providing grants to local businesses
3. Helping businesses decarbonise through decarbonisation plans and grants

For the Communities & Place:

1. Major refurbishment of community buildings (e.g. community hubs)
2. Large investments in sports pavilions or pitches
3. Significant improvements to, or the provision of new, playground equipment

The proposed evaluation approach in summary

Process evaluation

The process evaluation will seek to learn about design, planning, implementation, management and monitoring of the interventions. It will triangulate evidence from documents (such as UKSPF guidance, investment plans, etc.); in-depth interviews with a range of stakeholders including lead local authorities, delivery partners, local community representatives and local businesses; focus groups with stakeholders who deliver and receive the interventions; and site observations which provide an understanding of interventions on the ground.

Impact evaluation

Recognising the complexity of the interventions, an integrative evaluation approach is proposed. This would maximise learning by integrating several evaluation methods (therefore involving several data sources and analytical approaches to enhance rigour), with the aim of generating robust evidence on what works, for whom, how and under what conditions.

The evaluation methodologies and analytical methods incorporated within this integrative evaluation are:

• Descriptive analysis of the projects (within each study group) that are being delivered, using data from surveys such as the Community Life Survey (CLS) and the Your Community, Your Say (YCYS) survey that was designed to support the UKSPF evaluation strategy. This will be used to understand contextual factors, which are important both for attributing impacts to the projects being evaluated and for exploring how impacts vary across different contexts. In addition, the strategic objectives of the projects will be identified to inform an assessment of the extent to which those strategic objectives are met by the projects within the study groups.
• Before-versus-after analysis to indicate what has changed over time, using data collected in the CLS, YCYS survey and bespoke surveys designed for this evaluation, alongside administrative data and locally collected monitoring data where available. This method does not provide evidence on what is attributable to the interventions, but it does highlight what has changed, where and for whom since the time of the intervention.
• Quasi-experimental difference-in-differences analysis using synthetic controls, control groups where data allows, or analysis comparing outcomes at specified distances from the intervention compared with outcomes close to the intervention. Incorporating this method into the evaluation further bolsters rigour by generating evidence on what impacts can be attributable to the projects within each study group. This method will be applied for all of the Communities & Place study groups. This method is more challenging if attempted in the period to 2025 for the People & Skills and Business Support study groups, both because sufficient data is not likely to be available (administrative data needed for identifying control groups is published with a lag of up to one year) and outcomes may not be observable for some time (some outcomes may only be observed several years after participating in a project). However, where this analysis is feasible for individual projects, it will be included. Over a longer evaluation period post 2025 (for example, to 2028), administrative data sets become more feasible to use and hence this method may be plausible for some People & Skills and Business Support study groups in the longer term.
• Theory-based contribution analysis which will triangulate qualitative data from the process evaluation and bespoke fieldwork alongside the quantitative analysis from the before-versus-after and quasi-experimental methods described above. This will be used to articulate an evidence-based narrative that validates the extent to which it is reasonable to claim that the interventions represented by each study group have contributed to the observed outcomes and, if so, how this varies across cohorts of participants or contextual factors, and the reasons for variations.

How the components of the integrative approach are used for each study group is summarised in Table 1.

Table 1: Components of the integrative evaluation approach in each study group

Study group	Descriptive analysis	Before-versus-after analysis	Quasi-experimental analysis	Theory-based contribution analysis
People & Skills
1: Helping local economically inactive people into employment	✓	✓	✖	✓
2: Helping local economically inactive young adults into employment (excl. through volunteering)	✓	✓	✖	✓
3: Helping local economically inactive young adults into employment through volunteering	✓	✓	✖	✓
4: Involving local businesses in helping local economically inactive people into employment	✓	✓	✖	✓
Business support
5: Supporting the digital development of local businesses	✓	✓	✖	✓
6: Providing grants to local businesses	✓	✓	✖	✓
7: Helping businesses decarbonise through decarbonisation plans and grants	✓	✓	Potentially	✓
Communities & Place
8: Major refurbishment of community buildings (e.g. community hubs)	✓	✓	✓	✓
9: Large investments in sports pavilions or pitches	✓	✓	✓	✓
10: Significant improvements to, or the provision of new, playground equipment	✓	✓	✓	✓

Source:   Frontier Economics

Value for money

Value for money (VfM) is a vital consideration when making decisions about what to invest in and where in order to meet local strategic objectives. This Feasibility Report therefore describes an approach for delivering an assessment of whether, and under what conditions, the interventions represented by each study group deliver VfM. This includes (i) providing a qualitative assessment of the extent to which the interventions have met the local strategic objectives they were designed to achieve; and (ii) using both scenario-based analysis and break-even analysis to determine the extent to which social benefits could be expected to exceed costs for these types of interventions over their lifetimes (including both changes in outcomes that can be monetised and those that cannot).

Evaluation time period

In designing this innovative integrative evaluation approach, it is clear that the analysis that it is possible to deliver in the period to 2025 is constrained by the data available, given that some of the projects will not be delivered until near the end of the funding period, i.e. March 2025. Furthermore, there are some outcomes that will not be observed in that short time period. This Feasibility Report therefore describes an appropriate and proportionate integrative evaluation approach by considering two time periods:

1. Evaluation period from late 2023 to late 2025: This evaluation activity would add value by generating evaluation evidence to deliver early insights and learning on what outputs and short-term outcomes each intervention is delivering, for whom, how and under what conditions. This is the period for which DLUHC has commissioned this intervention-level evaluation to be delivered.

2. Evaluation over a longer time period beyond 2025 (taken illustratively to be late 2023 to late 2028): This evaluation activity would add value by drawing on longer-term data to enrich the insights on what outputs and outcomes study groups are delivering over a longer period, for whom, how and under what conditions. This longer time period opens the opportunity for more rigorous quasi-experimental analysis of the scale of impacts observed that can be credibly attributed to the projects within the study groups.

Reporting timelines

This Feasibility Report has been prepared with a view to the intervention-level evaluation delivering an interim report by spring 2024 (including baselining activity and early process evaluation findings) and a final evaluation report in late 2025. Reporting beyond this point would be dependent on whether the evaluation is extended post 2025. DLUHC is exploring possibilities to extend the evaluation beyond 2025 to ensure a robust and comprehensive capture of the impacts.

1. The fund’s interventions will be planned and delivered by councils and mayoral authorities across England, Scotland and Wales – ‘lead local authorities’. In Northern Ireland, the UK Government will have oversight of the fund. ↩

2. These definitions of pride in place and life chances from the Department of Levelling Up, Housing and Communities are consistent with the Levelling Up White Paper (HMG, 2022) ↩

3. There is a separate randomised control trial (RCT) component of the intervention-level special tier. This aims to provide a deeper understanding of impacts for a relevant subset of interventions (as appropriate for a RCT methodology) beyond the intervention-level component discussed in this Feasibility Report. ↩

4. Other evaluations may define the term ‘intervention’ slightly differently. An example is the Towns Fund evaluation, which considers an intervention to refer to projects within a particular geographical locality such as a town. ↩

5. This includes information provided to DLUHC for monitoring and evaluation as well as specific additional data requests and discussions for this evaluation. ↩

6. There may be minor adjustments to the focus of the study groups following the project selection. ↩