UK Shared Prosperity Fund: evaluation (4)
This page provides information on evaluation for the UK Shared Prosperity Fund.
Summary
4.1 We are committed to reducing funding programme bureaucracy and will ensure that minimal burden is placed upon those LLAs that are asked to participate in the evaluation process, in line with the Department’s simplification agenda. We will carry out intervention, programme and project evaluation centrally, but will rely upon some LLAs to support this work. Our evaluation contractors will develop further detail of the Evaluation Strategy through the feasibility stage of the process during the early part of 2023. LLAs in regional geographies will be responsible for providing such data and information as is required for the purposes of evaluation from each of the constituent local authorities in their regional geography.
4.2 This section should be read alongside the Evaluation Strategy which sets out in detail the Evaluation framework for the UKSPF. The objective of the Evaluation Strategy is to gather a robust evidence base for what works, in what context, and by what means. Working together with LLAs, we and our evaluation contractors will gather evidence to better inform future policy making at both local and national level and to ensure that future levelling up and local growth spending can be focused upon those interventions that deliver the greatest beneficial impact and represent good value for money.
4.3 Evaluation is integral to understanding the impact of the UKSPF. Due to the scale and size of the individual projects that the UKSPF may fund, we need to supplement local level evaluations to better understand the impacts of the UKSPF. We will coordinate national evaluation but will need LLAs to engage and support this centrally coordinated evaluation work. LLAs should continue to develop and conduct their own robust and credible evaluations. For those LLAs with larger allocations, the emphasis on proportionate and rigorous evaluation is increasingly important.
4.4 LLAs that are invited to participate in evaluation will be informed ahead of time and we will confirm to them what will be required in terms of data collection and interaction with evaluation contractors, if applicable. LLAs will not be asked for retrospective data but would ask that, where the LLA has conducted its own data gathering or evaluation, that information should be shared with our evaluation contractor.
4.5 Participation in DLUHC-led evaluation is encouraged; we believe that the benefits for both participant LLAs and for wider growth funding and levelling up policy will be substantial. We recognise that some local authorities may struggle to resource working with our evaluation contractors, so we will work with them to ensure that the burden is manageable.
The evaluation framework
4.6. The evaluation framework will contribute to understanding the Fund’s impact on pride in place and life chances through three tiers of evaluation together with a crosscutting analysis, which will address a number of underlying questions, including, but not limited to:
- Crosscutting evaluation: developing a series of surveys to build a better understanding of how pride in place and life chances varies across the UK, providing a baseline against which to assess UKSPF impacts at intervention, place and programme level. Data will be gathered through a combination of the DCMS-led Community Life Survey and UKSPF-specific pride in place and life chances surveys delivered by DLUHC.
- Intervention level evaluation: To understand the impacts of 10-15 specific interventions across the UKSPF’s three investment priorities, and how well they have been delivered at the place level, using a mix of quasi-experimental approaches with treatment and control groups. Where interventions are appropriate, we may also conduct randomised control trials (RCTs) to provide a deeper understanding of the impacts above and beyond that made possible by the quasi-experimental approach of the wider intervention impact evaluations.
- Place level evaluation: To develop a detailed understanding of the UKSPF’s effectiveness across different types of place, considering their unique local characteristics and challenges, and focusing on interactions between stakeholders, local decision making, process efficiency and interactions with other local growth funds (such as the LUF)
- Programme level evaluation: To understand the impact of the programme as a whole using a non-experimental, multilevel regression-based approach to isolate the effects of the UKSPF from other local growth funds, places’ baseline economic contexts and wider confounding factors. For example, to understand the impact and value for money of Multiply, how effectively it has been delivered by places and UKG, and to build the evidence base around best practice delivery of local adult skills interventions.
What this means for LLAs
Crosscutting surveys
4.7 Not all LLAs will be asked to participate in the local and intervention-level pride in place and life chances surveys – only those that are also participating in the intervention level impact evaluations and place-level case studies that the surveys will serve as a key data source for. Given this, we will aim to keep any additional survey-specific asks on places to a minimum by building on other evaluation components. In short:
- For the CLS local level boost, data will be collected from respondents directly in parallel with the core CLS, with no additional work required from LLAs.
- For the PiP and LC surveys, participating LLAs will be expected to assist in establishing sampling frames for selected interventions and within specific geographies.
- The local survey tool will not require any input from LLAs – though LLAs are encouraged to make use of it for their own surveys once available.
For area-based projects (i.e. those not delivered to a specific group of individual participants, like a new park), some additional work may be required from LLAs to assist in identifying the correct project catchment area and addresses of residents therein.
Intervention impact evaluations
4.8 We envisage 5-10 LLAs will form the treatment and control groups for each intervention to be evaluated. Control group LLAs will be selected based on their socioeconomic and demographic similarities compared to the treatment group. The specific expectations and requirements of treatment group LLAs will be finalized as part of the feasibility stage, but will include:
- Developing and testing robust theories of change for the interventions to be evaluated in collaboration with the contractor.
- Assisting in organising focus groups and interviews with key stakeholders and project staff.
- Assisting in establishing suitable sampling frames for the Intervention Types being evaluated to support the intervention-level pride in place and life chances surveys.
4.9 Existing datasets and reporting channels will be used to collect LLA-level data where possible. However, some primary data collection will be required and LLAs will be asked to help facilitate this. Specific data requirements will be drawn up as part of the feasibility stage and clearly communicated to LLAs. LLAs are not expected to start pre-emptively collecting data outside of wider UKSPF reporting channels until this has happened.
4.10 Treatment group LLAs may be asked to provide:
- Detail of specific projects within each Intervention Type beyond that contained in places’ investment plans and details of local delivery responsibilities and reporting lines.
- Detail of local data availability and data gaps, particularly regarding beneficiaries of UKSPF interventions.
- Details of other local growth and levelling up programmes delivering in their geographies, particularly if this overlaps with UKSPF investment priorities and/or is being used to match-fund UKSPF projects.
4.11. The requirements of control group LLAs will be more light-touch and developed further during the feasibility phase, but will similarly include additional detail on investment plans, delivery structures and other local investment currently in delivery.
Randomised controlled trials
4.12 We will invite LLAs to express an interest in running randomised controlled trials (RCTs) for one or more of the projects they are using the UKSPF to fund. Only certain interventions will be suitable for an RCT evaluation design. Interventions involving the active participation of individuals are more conducive to random allocation of participants to the control group than those involving changes to the physical environment. During the expression of interest phase, we will run feasibility workshops to help interested LLAs develop their ideas. We envisage running around 10 trials in total. The RCTs themselves will be funded by DLUHC.
4.13 We encourage LLAs to consider opportunities to undertake RCTs where possible and will aim to support any place that is considering RCTs. More information is included in the Evaluation Strategy. RCTs could be used to test whether a project has achieved its desired impact, or they could be used to test which is the more effective of two or more different ways of doing things. For example, an RCT was recently conducted to determine if it was more effect for police to do frequent shorter patrols or less frequent longer patrols.
Evaluation case studies
4.14 We will commission up to 36 place-based evaluation case studies (in England, Scotland, Wales and Northern Ireland), to understand how the UKSPF has worked in places. The case studies will specifically focus on the role of the places, interactions between stakeholders, decision making, the different interventions and local growth funds, and processes.
4.15 LLAs that have a total allocation of £60m or more should expect to be invited to take part in one or more of the commissioned place-based evaluation samples.
4.16 The place-based case studies will involve a range of data-collection and analysis methods, including surveys, interviews, focus groups and quasi-experimental approaches.
4.17 Our contractor will deliver the place-based case studies. Where an LLA is participating in multiple elements of the evaluation simultaneously, contractors will share place-specific information where possible to reduce duplication of effort by LLAs.
4.18 All reports created as a result of evaluation activity will be shared with participating LLAs and made public where possible. Specific LLAs will not be named in any reports, as these case studies are intended to build the evidence base and not as a performance management tool.
4.19 Case study LLAs could be asked to provide the following data (if not already provided through participation in other parts of the evaluation):
- Detail of projects contained in places’ investment plans, with a particular focus on local delivery structures, responsibilities, and reporting lines.
- Detail of local data availability and data gaps, particularly regarding beneficiaries of UKSPF interventions and local economic and labour market indicators past those held centrally by ONS.
- Details of other local growth and levelling up programmes delivering in their geographies, particularly if this overlaps with UKSPF investment priorities and/or is being used to match-fund UKSPF projects.
- Personal information of UKSPF beneficiaries for data matching purposes, if not already provided in support of the intervention level evaluation.
4.20 We will contact places directly following the feasibility stage to formally invite their participation in case studies, and in doing so will confirm whether they need to commence any additional data collection. If selected, LLAs are encouraged to participate; the case studies in particular will provide a wealth of good practice and lessons for future funding delivery that will be especially relevant and useful to those LLAs that participate.
Programme evaluation
4.21 For the programme level evaluation, we expect that the majority of input data will be either publicly available, collated from other government departments, or collected through wider reporting channels that LLAs use already.
4.22 However, as the regression model underpinning the programme level evaluation is developed further, it may be necessary to include additional data sources such as:
- Additional place-level economic and social indicators past those collated centrally by ONS
- Further detail of LLA funded spend on UKSPF-adjacent projects
- Further detail on wider LLA budgets and funding sources
4.23 The programme level evaluation will run to slightly slower timelines to the rest of the components, in line with the longer time horizons necessary to see programme-level impacts. LLAs will be contacted directly following further feasibility work with detail of any additional data collection requirements. In the meantime, places will not be expected to collect programme evaluation-specific data in advance.
Multiply evaluation
4.24 Multiply is a new £559m, 3-year adult numeracy programme that is a part of the UKSPF. A core part of the programme is to build the evidence base on ‘what works’ in improving adult numeracy.
4.25 In Scotland and Wales, DLUHC is the responsible delivery department; delivery will use the same geographical structures as the core UKSPF.
4.26 DfE is the lead department for Multiply delivery in England as well as the UK wide programme of evaluation for Multiply in England. Where possible, we will work with DfE to align the evaluation approach in Scotland and Wales to the evaluation approach in England, recognising the different skills infrastructures in different nations.
4.27 DfE and DLUHC will work with LLAs to test and refine the evaluation plan for Multiply through a series of roundtables during the feasibility phase, and DfE will prepare and disseminate regular research updates to LLAs as fieldwork begins to inform them of upcoming evaluation activity. LLAs will have several opportunities to review emerging findings as they are documented throughout the evaluation cycle commensurate with their level of participation in the different strands of the evaluation
4.28 As key delivery stakeholders, LLAs may be asked to participate in Multiply’s evaluation through:
- taking part in surveys in case studies to solicit views on Multiply’s delivery model, barriers to implementation, and perception of impacts
- helping to organise focus groups, interviews and site visits.
- Engaging with participants and stakeholders to assist with data collection, disseminate evaluation outputs, and solicit feedback on evaluation design throughout the feasibility stage
4.29 We expect LLAs in Scotland and Wales to ensure third party contractors collect information on the numbers of individuals who participate in any substantive Multiply intervention. LLAs are not expected to gather personal data from those who participate in Multiply programmes unless required for their own monitoring and evaluation purposes.
Further evaluation information
4.30 Input from LLAs that are invited to engage with evaluation activities set out in the Evaluation Strategy will be proportionate and should be funded from their administration budgets, unless otherwise stated in the Evaluation Strategy. The outline of the types of evaluation activity are set out in section 4.6.
4.31 Places are encouraged to undertake their own place-based evaluations of how the UKSPF has worked in their area, particularly process evaluation on individual projects, alongside the place-based case studies commissioned by DLUHC (see section 4.14). Proposed methods could include measuring outputs, contribution analysis, developing case studies, conducting surveys and interviews with stakeholders and beneficiaries. As per section 3, DLUHC has asked to be informed when this is taking place through routine reporting.
4.32 LLAs do not need to indicate to us what their local evaluation strategy will be as part of their investment plan [but it will form part of annual reporting].
4.33 DLUHC may engage with HMRC and the Department for Work and Pensions, in particular around the people and skills and supporting local business investment priorities. As with DfE an MoU and appropriate data sharing arrangement will be developed.
4.34 We continue to work closely with the DLUHC Data Protection Officer and will review these arrangements to ensure continual compliance with DPA 2018 and UK GDPR.
4.35 As and when required, appropriate arrangements for sharing of data by LLAs with DLUHC will be developed and set out, including privacy notices as and where appropriate.
Last updated 8 March 2023 + show all updates
-
Greater detail and clarification provided in line with the UKSPF Evaluation Strategy published at the same time as this update. Clarification on what DLUHC expects from LLAs with respect to collection of additional data for evaluation purposes.
-
Welsh added
-
Added translation