Research and analysis

Towns Fund: evaluation feasibility report

Published 11 January 2024

Applies to England

1. Executive summary

The UK government has identified that some towns and high streets are prospering while others are lagging. Lagging towns experience physical and digital disconnection from other towns and cities. They have outdated land use, insufficient skills and business support for high-paying jobs and limited strategic local economic management.

The Towns Fund (TF) is a Department for Levelling Up, Communities and Housing (DLUHC) fund that aims to address these issues, working alongside other government initiatives. The fund is composed of 2 separate sub-funds:

• Town Deals (TD): an allocated fund which aims to fund projects that drive towns’ economic regeneration to deliver long-term economic and productivity growth. In July 2021, 101 towns in England were selected based on their need to develop TD-funded projects. The selected towns then developed Town Investment Plans (TIPs), which included a suite of projects that addressed the issues they identified in their local area.
• Future High Streets Fund (FHSF): a competitive fund aiming to finance projects which renew and reshape town centres and high streets (‘places’) to drive growth, improve the experience of public space users, and ensure future economic sustainability. In December 2020, 72 places in England successfully secured funding. Unlike TD, places were chosen based on the strength of their business case rather than their level of need. The FHSF-funded projects in a given place primarily address a locally defined issue confined to a relatively narrow geography.

DLUHC commissioned Frontier Economics and BMG Research to assess the feasibility of process, impact and value-for-money (VfM) evaluations of the TF consisting of the TD and the FHSF. In addition, Frontier Economics and BMG Research have also been commissioned to deliver the evaluation. The overall evaluation is to be completed in early 2026.

Main challenges to be addressed within the evaluation

The evaluation of TF presents several expected challenges. The main ones include:

• Diversity of project timelines: Nearly 1,000 projects are funded by TF. Some have already been completed, and the vast majority are forecast to complete by the March 2026 funding deadline for Town Deals.^{[footnote 1]} However to form part of the evaluation under this contracted period projects must complete by April 2025, and about 34% of Towns Fund projects are expected to complete after April 2025 at present.^{[footnote 2]} Further delays will further increase the number of projects which cannot be evaluated.
• Diversity of project outcomes: The projects were categorised by local authorities (LAs) into primary themes. The preliminary assessment shows that projects within one theme can have many diverse outcomes leading to various impacts. An evaluation based on these themes may not be the most appropriate as it would mix projects with different outcomes.
• Requirement for granular data: Since the outcomes of TF projects are expected to occur in a local geographical area, the evaluation will require data at the right level of geographic granularity (town or even more granular) to identify any changes. In addition, the data need to be recorded frequently enough so that they can be used to assess changes in outcomes before and after the completion of the TF projects.
• Place-based evaluation considerations: Changes within targeted areas might also impact neighbouring areas. For example, footfall may increase in a funded high street, but that could be because footfall shifted from a nearby high street which did not receive funding. This effect is called ‘displacement’. The evaluation needs to properly account for displacement in order to avoid under or overestimating the impact of the intervention.
• Presence of other local growth funds: DLUHC and other departments fund other projects aimed at improving local growth and levelling up in areas with TF funding. In addition, some projects are co-funded through other DLUHC and governmental interventions. This makes attributing observed impacts to the TF alone challenging.

Conclusions of the feasibility assessment

Given these challenges, DLUHC commissioned a feasibility assessment for the TF evaluation, which considered how best to address the challenges and outlined a methodology for dealing with them. The conclusions of this feasibility assessment are as follows.

• Programme-level impact evaluation: Several experimental and quasi-experimental evaluation methods were considered, and most were ruled out as unfeasible for both the TD and the FHSF.^{[footnote 3]} Some quasi-experimental methods, in particular a difference-in-difference (DiD) econometric approach, are feasible. Sample sizes are currently sufficiently large, and it appears possible to identify similar enough control groups of towns/places using matching techniques. To assess the level of displacement around the treated areas, Frontier Economics will analyse changes in outcomes in neighbouring areas surrounding the area where one might expect to see the direct impact. Secondary data with relevant levels of geographical granularity and observation frequency are available. However, a detailed assessment of the feasibility of analysis for individual indicators will be conducted in the first implementation stage. Information on some impacts, in particular pride-in-place, is also limited for this type of analysis. The completion dates of some projects are after the end of this evaluation’s contracted period (that is, projects commencing in early 2026), and this means that the analysis will likely focus primarily on the impact of the TF on short- to medium-term outcomes (for example, changes in local skills, property market outcomes and investment). Long-term impacts are most likely to be realised after the end of this evaluation.^{[footnote 4]}
• Intervention-level^{[footnote 5]} impact evaluation: An ‘intervention’ refers to a group of similar projects within the same town. A theory-based evaluation that utilises a Realist Evaluation (RE) and Qualitative Comparative Analysis (QCA) for 20 intervention case studies is feasible. Each case study can draw on a mix of secondary data sources, administrative data, TF performance data, and primary data collected through surveys. The design of the primary survey data collection will be prepared at the early stages of the evaluation to ensure baseline data is collected where possible (that is, where the project has not been completed yet). Twenty case study projects, which are expected to be completed within the evaluation timeline and which cover the key intervention characteristics of interest for this evaluation, have been identified.^{[footnote 6]}The choice of projects will be finalised in the first stage of implementation and will include a further 10 reserve projects that can be used as substitutes, if required. The timeframe for the evaluation of each case study will be based on the timelines of the projects selected. For each case study, the most appropriate outcomes and relevant data sources will be identified.
• Value-for-money (VfM): the assessment is that undertaking a ‘break-even’ analysis to assess the likelihood that project benefits outweigh project costs, and a cost-effectiveness assessment to identify projects that provide higher impact per-pound-spent is feasible. A full cost-benefit analysis is unlikely to be feasible because of the difficulties in monetising many of the potential benefits of the TF and the difficulties detecting statistically significant project impacts within the evaluation timeframe. Since projects can be co-funded with non-TF funds, the results of the break-even and cost-effectiveness analyses will be for the projects as a whole and not for the TF component alone. As part of the assessment, where possible, consideration will be given to the relative importance of the TF component compared to all the project funding.
• Process evaluation: A qualitative approach to understanding how the delivery processes have worked to date is feasible. The process evaluation will be informed by interviews, surveys, and administrative data and will be deployed in parallel to the intervention-level evaluation in the 20 selected case study places to minimise the burden on LAs and maximise the use of resources.

The evaluation will include projects that will be completed by April 2025 to allow for at least some short-term outcomes to emerge before the completion of this evaluation in Q1 2026. This evaluation will also exclude an assessment of the impacts of projects in Pathfinder Towns, which are towns that have more flexibility in how various DLUHC fundings are used (see more information about the Pathfinder Pilot).

The methodology proposed for this evaluation has been chosen in order to maximise learning and to deliver an as robust as possible evaluation within the existing timeline. While this feasibility study has not directly assessed the feasibility of a longer-term evaluation beyond the current evaluation timeline, the existing evaluation design could be extended post-Q1 2026, to further explore the impacts of the TF. In particular, rerunning the programme-level econometric analysis might increase the robustness of the results (due to increased sample sizes of completed projects) and test longer-term impacts (which are not likely to be realised within the contracted timeframe of this evaluation).

Revisiting the intervention-level case studies and undertaking additional case studies of larger projects completed after April 2025 might reveal additional impacts, such as delayed effects and dynamic changes, which might not be picked up by the case studies conducted in this evaluation. Any future evaluation work should be conducted around 2028 to allow for larger projects to be completed and for longer-term impacts to be realised.

Limitations and risks to the evaluation

The assessment identified the following as the main limitations and risks to the TF evaluation:

• The evaluation levels might not be able to detect statistically significant changes in the long-term impacts of the TF. This is due to the number of projects expected to be completed close to, or after, the end of the current evaluation period, which does not allow enough time for long-term impacts to be realised (for example, changes in productivity or local deprivation). However, the evaluation levels will aim to identify short- and medium-term outcomes that are more likely to be affected shortly after the conclusion of the projects, and for which data can be gathered at the relevant frequency and geographical granularity level. Where possible, the evaluation will aim to combine evidence on identified short- and medium-term outcomes with a theory-based approach to infer potential long-term impacts. Additionally, long-term impacts could be evaluated more robustly if a future evaluation were to take place around 2028 to allow for larger projects to be completed and for longer-term impacts to be realised.
• It might be impossible to find statistically significant short-term outcomes in the programme-level evaluation due to relatively small sample sizes and possible project delays. The theory-based nature of the intervention-level evaluation will help to mitigate this risk and provide insight into short-and medium-term outcomes of TF.
• TF-funded projects are sometimes co-funded by other governmental interventions and private investment. In addition, other projects with similar outcomes might be deployed in the TF-funded areas. Consequently, the benefits attributable, specifically, to the TF may be impossible to fully separate from those arising as a result of TF in combination with other initiatives. The evaluation will focus on the impact of a project rather than the TF-funded component.

Evaluation plan

The current evaluation period is expected to conclude in Q1 2026. The first wave of the intervention-level case studies and the process evaluation will begin in early 2024. The next report will include early emerging findings from these activities and will be mainly focused on the process evaluation. These activities will also inform the remainder of the evaluation. The remainder of the intervention-level and process evaluations will be spread throughout 2024 and 2025. The next interim report is likely to present some initial findings and insights that can be shared with key stakeholders, including those involved in the UK Shared Prosperity Fund and Levelling Up Fund evaluations. The programme-level and VfM evaluations will primarily be undertaken in late 2025 to allow for more time for programme-level impacts to materialise – these findings will be presented in the final report of this evaluation.

2. Introduction

Frontier Economics was commissioned by the Department for Levelling Up, Housing and Communities (DLUHC) to undertake a feasibility study for the impact, value-for-money (VfM), and a process evaluation of the Towns Fund (TF) and carry out the evaluation. The TF includes 2 funds:

• Town Deals (TD): a £2.35 billion fund aimed at driving towns’ economic regeneration to deliver long-term economic and productivity growth. In 2019, 101 towns in England were selected.
• Future High Streets Fund (FHSF): a fund worth over £830 million aimed at renewing and reshaping town centres and high streets (‘places’) to drive growth, improve the experience, and ensure future economic sustainability. In 2020, 72 high streets were chosen for the FHSF.

The findings and recommendations, with respect to the feasibility study, are set out in this report.

2.1     Aims and objectives of the evaluation and the feasibility assessment

Government spending of public money should be evaluated to ensure transparency and accountability. Evaluations provide systematic, evidence-based frameworks that can help understand what works and what does not, whether the intervention’s impacts were realised, and if the spending was cost-effective. They are an essential tool for making evidence-based decisions about spending and provide learnings that help make future public spending more effective.

The evaluation of TF is divided into 3 complementary evaluations:

• The impact evaluation: aimed at understanding the size, scale, and cause of impacts resulting from interventions funded by the TF.
• Value-for-money (VfM): an assessment of whether the intervention created benefits that outweighed its costs or was cost-effective in delivering benefits.
• The process evaluation: aimed at understanding how efficient and effective were the funding process (and its management) and the delivery of the funded projects. It may also identify potential improvements to these processes going forward.

The TF is a complex intervention to evaluate. It includes 2 sub-funds, each providing funding to multiple areas across England for numerous projects across a variety of themes. The projects vary by size, focus, and how long it is expected to take to complete them. These factors make an impact evaluation particularly challenging, because it can be difficult to establish a robust counterfactual and be confident that sufficient data will be available to detect the effect of the TF – if one should exist. This is where this feasibility assessment comes in. It allows for an assessment of alternative approaches to the evaluation which weighs their pros and cons, and lands on a reasonable, proportionate, and effective evaluation option.

The feasibility assessment was undertaken in the summer of 2023. The evaluation is planned to run from the Autumn of 2023 to early 2026. While further evaluation work may be undertaken post-2026, this feasibility report is focused on the current evaluation period (from 2023 to early 2026).

2.2     Feasibility study framework

There are 5 stages to the feasibility study approach:

1. Building a knowledge foundation: this stage included reviewing DLUHC internal and public information about TF (such as business cases, application forms, Monitoring and Evaluation [M&E] reports, etc.). A rapid evidence review of evaluation best practice and governmental guidelines was conducted. Academic literature of similar evaluations, with regard to possible outcomes and impacts, was reviewed. Lastly, the availability of administrative, secondary, commercial, and open-source data which could be used for the evaluation was examined.

2. Reviewing and updating the existing Theory of Change (ToC): this stage began with a review of the ToC that DLUHC developed for the TF M&E report, which was refined and updated.^{[footnote 7]} The updated ToC was validated with DLUHC stakeholders through a workshop.

3. Developing an Evaluation Metric framework: this stage included identifying areas of further information needed to assess the feasibility of evaluation approaches. This included the creation of a typology of TF-funded interventions, mapping outcomes to potential data sources and assessing the fit of various analytical approaches.

4. Testing possible analytical approaches: both quantitative and qualitative approaches were considered. For the quantitative approaches, this stage included assessing if the time frame of the evaluation would allow the detection of changes in the foreseen outcomes. For the qualitative approach, a small case study was tested for the feasibility of the proposed approach.

5. Reporting: the results of the feasibility analysis were compiled into this report. The feasibility of the evaluation, preferred approaches, high-level plans for the evaluation, any primary data collection and fieldwork, and risks and mitigations for the next stages, were concluded.

2.3     Structure of the report

The rest of the report provides the following information:

• Section 3: an overview of the TF, including the ToC and logic model
• Section 4: the main challenges for the evaluation
• Section 5: the proposed approach for the impact evaluation
• Section 6: the proposed approach for the VfM assessment
• Section 7: the proposed approach for the process evaluation
• Section 8: the high-level evaluation plan and timeline
• Section 9: a list of key terms and acronyms used throughout the report

3. Overview of the Towns Fund

The UK government has identified that some towns and high streets are prospering while others are lagging.^{[footnote 8]} While different places face different sets of problems, those which are lagging often share similar issues, which reinforce each other, leading to a negative feedback cycle.

For these struggling towns and high streets, the government has identified that the following issues need to be addressed:

• physical and digital disconnection, especially to markets
• outdated land use and built environment
• insufficient skills and business support and
• lack of strategic, local economic management

The presence of one or more of these issues can be associated with wider economic and social problems, including intergenerational unemployment or deprivation, crime and anti-social behaviour, poor mental and physical health outcomes, poor social mobility, depopulation, and low pride-in-place and civic participation.

The Towns Fund (TF) is one of several government funding schemes aimed at addressing these issues. TF is aimed at stimulating growth in England by improving the economic performance of lagging areas. The fund is composed of 2 separate sub-funds: Town Deals (TD) and Future High Streets Fund (FHSF). Each is discussed below.

3.1     Town Deals

TD is an allocated fund which aims to fund projects which drive a town’s economic regeneration to deliver long-term economic and productivity growth. In July 2021, 101 towns in England were selected, based on their need, to develop TD. Of the 1,082 towns in England, the 50% most income-deprived were initially selected to form a shortlist. These 541 shortlisted towns were then scored and ranked, using a weighted formula, across 7 criteria to reflect local needs and growth potential while considering their geographical location. At the end of the selection process, 101 towns were chosen. Each recipient of a TD then developed an overall Town Investment Plan (TIP) comprised of multiple projects, and the Department for Levelling Up, Housing and Communities (DLUHC) confirmed and approved the list of projects for each town.^{[footnote 9]}

While projects are highly diverse in their type and expected outcomes, DLUHC originally grouped them under the following 3 overall themes:

• Urban regeneration, planning, and land use: these projects aim to ensure that towns are thriving places for people to live and work by increasing density in town centres and strengthening local economic assets.
• Connectivity (transport and digital): these projects aim to develop local transport schemes that complement regional and national networks, as well as support the delivery of improved digital connectivity.
• Skills and enterprise infrastructure: these projects aim to drive private sector investment and ensure towns have the space to support skills and small business development.

Projects funded by TD may cover one or more (and even all) of these themes. The fund also provided capacity funding, training, and support to the local authorities (LAs) managing the projects to develop their strategic and learning capabilities to deliver local growth programmes.^{[footnote 10]}

3.2     Future High Streets Fund

The FHSF is a competitive fund which aims to renew and reshape town centres and high streets (‘places’) to drive growth, improve the experience of public space users, and ensure future economic sustainability. Places were chosen through a two-phase application process. In phase 1, eligible places submitted an Expression of Interest (EoI), ^{[footnote 11]} which was scored against  6 questions with differing weights. In Phase 2, 101 shortlisted places were asked to submit business cases, which were scored against 3 criteria. In December 2020, 72 places in England were successful in their funding applications.

As part of the business cases submitted in phase 2, LAs provided an outline of the FHSF projects. DLUHC originally grouped the collection of FHSF-funded projects into the following themes:

• investment in physical infrastructure, including housing, commercial spaces, and public spaces
• acquisition and assembly of land, including support for new housing, workspaces, and the public realm
• improvements to transport access, traffic flow, and circulation in the area
• support in the change of built environment use, including (where appropriate) housing delivery and densification
• support in the adaptation of the high street in response to changing technology

Unlike TD, which included a revenue element, the FHSF provided only capital funding for the selected projects. As such, all of the FHSF project themes can be primarily mapped into the first 2 themes of the TD-funded projects: urban regeneration and connectivity (transport and digital). In addition, the FHSF-funded projects in a given place mainly fall under one of the themes as the collection of the project is aimed at one area of improvement. Similar to TD, the list of projects for a given place might have changed over time due to various reasons, such as changes in costs or priorities.   

3.3     Theory of Change and Logic Model

• The UK government’s Magenta Book recommends that a key first step of any evaluation is to develop a Theory of Change (ToC). The ToC captures how the intervention is expected to work by setting out the steps involved in achieving the desired outcomes, the assumptions made, and wider contextual factors. Developing a ToC involves considering the inputs (such as the investment, regulation, and actions that will take place) and the causal chain that leads to the expected outputs and outcomes. These causal chains are then tested through a synthesis of evidence, and a logic model is created. This model is a visual tool for representing the ToC so that it can be rapidly understood and disseminated.

The ToC for the TF outlined below is based on:

• a review of the initial logic model and ToC developed for the TF Monitoring and Evaluation (M&E) Strategy in 2021
• a desk review of TF documentation and data provided by DLUHC
• semi-structured interviews with internal DLUHC stakeholders and
• a workshop with internal DLUHC stakeholders to discuss and validate updates to the ToC

The successful implementation of the TF relies on several key inputs. Financial and non-financial support, in the form of capital and revenue funding from DLUHC, is needed to administer the funds and support recipients in delivering the projects. DLUHC staff (including the DLUHC central team, the TF programme team, area leads, the Towns Fund Delivery Partner [TFDP] team, and the High Streets Task Force) expertise and wider non-financial DLUHC support are needed to administer the fund and support recipients in project delivery. LA time and resources are also required to attract private investment, build business cases and M&E plans, manage TF projects, and leverage other public investment sources.

These inputs support the TF activities, which were placed into 5 categories. Activities in each category create direct outputs:

• LA capabilities: DLUHC, through the TFDP^{[footnote 12]}, provided support and training on the management of the TD fund, which provided new skills for LA staff.
• Transport connectivity: Projects aimed at new transport facilities (for example, cycling lanes) lead to new/renovated cycling, pedestrian, and public routes; improved road connections to town centres; electric vehicle charging points; and improved parking.
• Digital connectivity: Projects that create a wider and more reliable internet infrastructure lead to improved 5G, Wi-Fi hotspots, and broadband access for residents and businesses.
• Urban regeneration: Projects aimed at creating mixed-used spaces, restoring cultural facilities, and providing sustainable housing will improve local spaces and create arts, sports, heritage, and culture facilities.
• Skills and enterprise: enterprises receiving non-financial and skills support, new and renovated office spaces and education facilities, and increased enrolment in training and education programmes.

Short-term outcomes are expected to be seen soon after the project’s end. The direct changes will be observable due to the local population’s interaction with the projects’ outputs. Short-term outcomes are also closely related to the outputs of the funded activities:

• Improvements in LA capabilities to develop strategic partnerships and deliver projects lead to alignment of local funding with town priorities, changes in community engagement, and changes in project delivery efficiency.
• Improvements in transport connectivity lead to changes in the use of transport options by residents and businesses, changes in journey times to town centres and places of employment, and changes in the affordability, sustainability, and accessibility of transport.
• Improvements in digital connectivity lead to changes in access to and quality of digital connectivity through 5G and broadband for residents and businesses.
• Activities targeted at urban regeneration lead to changes in access to mixed-use, leisure, and outdoor spaces; vacancy rates; land values; and visitor numbers and footfall in towns and high streets.
• Activities targeted at skills and enterprise improvements lead to changes in demand and supply of people trained for high-skilled work, changes in net migration amongst high-skilled workers, changes in the use of business development services, and broader changes in skills growth and innovation within local businesses.
• Activities and outputs from projects across the themes lead to changes in pride-in- and perception-of-place from changes in local pride and business dynamism.

These short-term outcomes create follow-on changes that require a longer period to be observed. The medium-term outcomes are a result of several short-term outcomes coming together:

• The experience of LAs managing these projects can lead to further improvements in local management capability, enabling further local growth.
• Improvements in transport and digital connectivity, urban regeneration, and skills can lead to employment changes, with more jobs and a higher share of high-quality jobs.
• These changes can also enable changes in private and public investment, driven in particular by improved perception-of-place and changes in potential return on investment.
• All of these short-term outcomes can also drive further business growth as a result of the expansion and increased longevity of existing businesses, changes in the diversity of the business community, and new and innovative business creation.

All medium-term outcomes contribute to long-term impacts. Changes in household income (due to higher employment rates and higher-quality jobs), local investments, productivity, and Gross Value Added (GVA), alongside increased business growth, will lead to local economic growth and a reduction in anti-social behaviour. Wider improvements in land use and the built environment in the town can also enable improvements in civic participation. More generally, changes in residents’ income and wellbeing, more equitable spread of benefits to lower-income areas, changes in social mobility, and changes in life satisfaction can lead to equitable improvements in income and wellbeing for residents.

There are a number of external barriers and enablers which also affect the realisation of these outcomes and impacts, which need to be considered when assessing the impact of TF. Enablers include other governmental and local funding used to undertake similar (or co-fund) projects and wider policies targeting local growth. Barriers include inflation, which can impact the affordability of the projects, local delivery partners (for example, supplier/consultancy contingencies), changes in regulatory policy frameworks, and the wider economic environment, such as changes in consumer needs, as seen during the COVID-19 pandemic.

4. Main challenges for the evaluation

The complexity of the Towns Fund (TF) presents several challenges for the evaluation. In general, place-based evaluations face challenges related to attribution, displacement, and leakages. The evaluation of the TF faces additional challenges due to the diversity of project completion timelines, project outcomes, and other contextual drivers. How best to address these challenges has been explored as part of assessing the feasibility of the TF evaluation and design.

As part of this process, guidance has been received from a range of experts on place-based evaluations. Most notably, the What Works Centre for Local Economic Growth and Professor Peter Tyler supported the design of an evaluation which overcomes as many challenges as possible. Multiple workshops with evaluation experts at Frontier Economics and the Department for Levelling Up, Housing and Communities (DLUHC) have also been held to test the thinking around how to approach the key evaluation challenges.

These challenges are discussed in more detail below.

4.1     Diversity in project completion timelines

The variety in project start and end dates poses a challenge for the evaluation. The current evaluation is anticipated to conclude by Q1 2026. While evaluation work may continue after the current evaluation period, at which point more projects will have been completed, not all of the projects funded by Town Deals (TD) and Future High Streets Fund (FHSF) will be completed in time to be evaluated within this initial evaluation. Since implementing an evaluation requires time to observe changes in the outcomes, undertake the analysis, and report the results, only projects that end well within the evaluation period can be included. More generally, some outcomes, such as changes to employment, may take longer than others to observe. For such outcomes, only projects which finished months or years prior to the evaluation end will be considered to ensure enough time had passed for changes in outcomes to be observed.

Analysis of reporting data shows that 34% of projects have an estimated end date after April 2025, with 5% of projects not having an end date estimate.^{[footnote 13]} The projects expected to conclude after April 2025 tend to be larger projects that may have wider and longer-term impacts. Any projects that end after the evaluation timeframe will not be included in the analysis. As a result, it may not be possible to analyse some outcomes and impacts as part of the programme-level evaluation due to a limited sample of relevant projects which conclude within the appropriate timeframe. The limited sample of projects with longer-term impacts, which are expected to emerge within the evaluation timeline, as well as the limited sample of larger projects more generally, is likely to limit the full understanding of the overall impact of the TF. However, in some cases, it may be possible to identify whether key long-term impacts are likely to be realised by assessing the intermediate outcomes that are expected to precede them, and the findings of this initial evaluation can also be used to inform future evaluation priorities.

4.2     The diversity of project outcomes  

The projects that the TF funds are diverse in their activities and outcomes. Local authorities (LAs) group their projects under 4 primary themes as part of their Monitoring and Evaluation (M&E) reporting requirements: urban regeneration, skills and enterprise, transport, and digital connectivity. An initial assessment of the project descriptions suggests that while the primary theme categorisation is helpful as a starting point, projects within each primary theme can have many different outcomes. For example, Norwich’s Town Investment Plan (TIP) includes 4 projects under the urban regeneration theme, but each differs in its outcomes. One project is developing an employment and commercial space aimed at impacting local employment by attracting new businesses, while another project is repurposing a Brownfield site into housing, intended to increase land values and improve residents’ perception-of-place.

In addition, projects can also lead to outcomes that could feasibly fit under different themes. For example, a project to construct a centre for young people would be classified under urban regeneration as the outcome would be a building, but this centre may lead to an increase in skills if it provides training to young people, meaning it could also fit under the skills and enterprise infrastructure category.

Given the diversity of projects and their outcomes, in order to undertake a robust impact evaluation, the projects will need to be re-categorised based on their outcomes and impacts instead of only relying on the themes that the projects were categorised into originally. A preliminary re-categorisation exercise has been undertaken as part of this feasibility assessment in order to assess the implications of this re-categorisation on sample sizes for the programme-level evaluation.

4.3     The geographical spread of projects within towns

Assessing the TF’s impact on a particular outcome requires a definition of geographical boundaries where the impact is expected to occur. The geographical boundary of each outcome may differ depending on the type of outcome and/or project. For example, activities that affect high street footfall could impact smaller geographical areas compared to activities that affect employment. As such, for each outcome of interest, the relevant geographical impact boundary must first be defined.

TF-funded projects are also diverse in their geographical spread within towns. If projects have the same outcomes and are situated within the same impact geographical boundary, their impact on the outcome will overlap. As such, consideration needs to be given to any overlap in the geographical boundaries of projects with the same expected outcomes in the evaluation design. This issue mainly relates to TD and less to FHSF, as projects in FHSF are usually closely located around the funded places, and the set of projects would overall have similar outcomes.

4.4     Data availability for outcomes at the required frequency and geographical granularity

The evaluation will depend on publicly available secondary data sources that record the outcomes of interest. Given the time it takes to detect changes in the outcomes of interest, it is important to have data sets which record outcomes (or proxies for outcomes) at a frequency that will allow us to observe changes and attribute a portion of that change to projects. With more infrequent recording of data, there is a greater risk that other material events may occur during the relevant period other than those directly related to TF projects. If other events do occur, it might not be possible to attribute the change to the TF. Data sources might also have lagged recording, which would further reduce the ability to use them for the evaluation. For example, in March every year, the Office of National Statistics (ONS) publishes the average employment data for the 4 quarters prior to the publication. As such, employment evaluation in February would be based on nearly one-year-lagged data.^{[footnote 14]}

Data are also required at the most appropriate geographical level. Given that the TD and the FHSF aim to improve towns and local growth, it would be important to identify changes in outcomes of interest at those geographic levels. Outcomes that are reported only for larger geographical areas, for example, at the LA level, may include several towns. This would make identifying impacts intended at the town level challenging as the data would capture both the impacted and non-impacted areas. Some outcomes, such as footfall, may also affect smaller geographical areas. For such outcomes, data at a more detailed level of geographical granularity will be needed.

In the assessments, data availability and whether they are published at the required frequency and geographical level of granularity will be considered. Where gaps will be identified, other options will be considered, such as primary data collection.

4.5     Substitution, leakage, and displacement issues

While TF projects are targeted at specific local areas, their impact can also affect other geographical areas outside the target areas. As the Green Book explains, evaluations of place-based interventions (that is, those with geographically defined objectives) should consider substitution, leakage, and displacement effects.

• Substitution occurs when the intervention substitutes an outcome instead of changing its quantity or quality. For example, an employment intervention may lead to workers switching between 2 similar productive sectors. In this case, while a change in employment in the individual sectors when considering them in isolation would be observed, the overall number of jobs or average productivity per employee would not have changed, and, therefore, the intervention would not have had a net employment or productivity effect.
• Leakages relate to impacts seen in neighbouring areas. For example, an intervention focused on skills and employment investment in a target area may lead to increased employment in neighbouring areas. Leakage is not necessarily a zero-sum game; while the leakage of employment effects into neighbouring areas may reduce employment effects in the target area, it may also lead to increases in neighbouring areas without reducing the magnitude of the effect in the target area.
• Displacement occurs when the impact observed in the target area is due to opposing changes happening in other areas. For example, if employment increases in town A because jobs have been lost in town B, employment is displaced from town B.

These factors are all related and should be considered in the context of the TF evaluation. Across all outcomes of interests, an assessment of whether there has been a net economic impact, any leakages, and whether these changes have occurred at the cost of other geographical areas is needed.

4.6     Other local growth funds

It is also necessary to consider other area-based interventions which might affect the geographical areas targeted by TF. This is important in order not to falsely attribute the entirety of the outcomes to TF when they may have been caused by a combination of interventions.

Other DLUHC funds, with similar objectives to TF, might also have been given to places which receive TF. At the moment, there are 6 DLUHC funds that have similar objectives. For example, TF, the UK Shared Prosperity Fund (UK SPF), the Community Ownership Fund, and the Levelling Up Fund are all aimed at increasing local growth and economic activity. There is also a wider set of funds and investments with similar objectives to TF, which other departments provide and manage. For example, the Department for Transport (DfT) has implemented the Bus Service Improvement Plans to improve local transport connectivity. In addition, areas which might be used as controls in the evaluation might have used other local funds for projects with similar outcomes to those funded by TF. Discretion at the local level about how to use some of these funds increases the likelihood that control areas might have used another fund to deliver similar projects to the TF.

Aside from funding projects with similar outcomes, in some cases, non-TF governmental and private investments co-fund the TF projects. For example, within Blackpool’s TD, one project that aims to create new sports facilities has secured funding amounting to £24.2 million.^{[footnote 15]} Their TD funding only contributes 27% of the total funding with the remainder coming from private funding through the Blackpool football club. In cases where multiple funding sources are secured, it might be challenging to attribute the observed impact to TF in particular. Recipient towns may also have received historical funding from funds such as the European Regional Development Fund, which might have had long-term impacts or impacted the baseline outcomes in the area. Overall, it is necessary to consider the impact of these other sources of funding in order to appropriately design the evaluation and, where possible, isolate the impact of the funding provided by TF.

The numerous funds provided to the same areas also mean that LAs might be asked to contribute to the evaluations of several funds. If multiple evaluations are undertaken at the same time, it might create pressure on LAs. The selection of places that will be reached out to for their input will need to be coordinated, to the extent possible, with other DLUHC evaluation teams.

4.7     COVID-19 and other contextual drivers

The impacts of COVID-19 are also relevant for the evaluation. According to Brien et al. (2022), COVID-19 had a profound impact on local economies and businesses, as well as on the way individuals use various spaces (public places, work, home, etc.). These impacts are relevant for the evaluation as they might have impacted places’ likelihood of receiving the funds, the level to which projects can generate the foreseen outcomes, and the baseline outcomes in recipient places. Towns’ resilience to COVID-19 and how much they were able to recover might also reveal common factors indicative of places struggling or being left behind.

Other contextual drivers, such as the increase in the cost of living and Brexit, might further impact the baseline and the outcomes which will be explored. The contextual drivers will have to be considered when exploring how TF projects are delivered and how the causal links might create the foreseen impacts under these circumstances.

5. Impact evaluation feasibility

An impact evaluation seeks to understand if an intervention had the intended (and any unintended) impacts and, if possible, assess the size of those impacts. It aims to assess if, how, and to what extent the fact that the intervention occurred (the factual) has led to the foreseen impacts compared to a hypothetical world where the intervention would not have occurred (the counterfactual).

The Magenta Book describes various methods and approaches that can be used for an impact evaluation. Impact evaluations can broadly be grouped into 3 overall categories:

• Theory-based impact evaluations test whether the causal links expected to bring about the change are supported by strong enough evidence to rule out alternative explanations and are concerned with both the extent of the change and why the change occurs.
• Quasi-experimental impact evaluations measure the impact of an intervention by estimating changes in treated and control groups, where these groups are constructed by controlling for relevant characteristics across groups analytically.
• Experimental impact evaluations also compare outcomes in treated and control groups to estimate the impact of an intervention, with these treated and control groups selected through randomisation in an experimental design.

The feasibility and appropriateness of each depend on the intervention and its complexity. Given the complexities of this place-based evaluation (see Section ‎4), Department for Levelling Up, Housing and Communities (DLUHC) has commissioned a feasibility study of 2 complementary impact evaluations:

1. Programme-level evaluation: aimed at understanding the Towns Fund (TF) impact as a whole by assessing the impact on outcomes (for example, level of employment) in recipient areas relative to a counterfactual world where the fund was not granted. In particular, DLUHC were interested in understanding if a quasi-experimental approach could be used.

2. Intervention-level evaluation: aimed at exploring the impact of different types of projects funded by TF. Intervention, in this case, refers to similar projects within the same town that are grouped together for evaluation purposes. This assessment is performed at a more granular level, allowing a more detailed assessment of the way in which the fund generates impact. DLUHC were interested in understanding if it would be feasible to deploy 20 case studies looking at the way impact is created through the funded activities.

5.1     Programme-level evaluation

5.1.1    Summary of feasibility assessment

A difference-in-difference (DiD)^{[footnote 16]} analysis for the programme-level evaluation is feasible for a subset of short- and medium-term outcomes that are expected to be realised and for which data would be available for analysis in the timeframe for the evaluation (that is, by Q1 2026). The proposed quasi-experimental approach will use the Propensity Score Matching (PSM)^{[footnote 17]} technique to identify control groups that did not receive TF funding but are similar to the funded group. Other techniques (for example, Regression Discontinuity Design [RDD]) were considered but, due to various technical considerations, were found to be less applicable in this case. The matching exercise and the analysis will be undertaken separately for the Town Deals (TD) and Future High Streets Fund (FHSF). The selection of towns/places for each fund was conducted using different techniques, which means that the identification of control units (that is, those that were not chosen) would need to be done separately for each fund and then analysed separately.

To ensure displacement issues are captured in the analysis, a comparison of the changes in outcomes in the impacted geographical area with changes in neighbouring areas that are around it (using concentric circles) will be conducted. For a quasi-experimental approach to be successful, there is a need to be reasonably confident that there are sufficient data to detect an effect where one exists and that the outcomes of interest are likely to have materialised sufficiently to be detected within the timeframe of the evaluation.

Since some projects will not be completed within the timeframe of the evaluation (Q1 2026), the programme-level evaluation is most likely to be partial. It is more likely to identify impacts that are more likely to materialise shortly after the conclusion of the projects for which data at the relevant frequency and geographical granularity level exist. Identification of longer-term impacts is less likely as not enough time will lapse from the completion of projects. Future delays to projects’ completion might limit the ability to observe changes with statistical significance, although meaningful sample sizes for a subset of short- and medium-term outcomes were observed.

5.1.2    The proposed approach for the programme-level evaluation

The proposed approach for the programme-level evaluation follows the 6 steps presented below:

1. identifying the treatment group

2. defining the geographical unit

3. identifying data sources

4. finding control groups

5. undertaking the econometric analyses

6. reporting

Each step is discussed below.

Step 1: Identification of the treatment group

The first step is to identify the group of treated units for which an impact is expected to be seen. Most evaluations would include all the units that received treatment, as the intervention usually leads to the same outcomes for all treated units. For example, if training is given to a group of individuals to increase their employability, the outcome, ‘employment within the following 6 months’, would be relevant for all treated units.

This evaluation seeks to understand the impact of nearly 840 TF-funded projects on their recipients.^{[footnote 18]} These projects are diverse, and even within the primary themes assigned to the projects by the local authority (LA) (urban regeneration, skills, and digital and physical connectivity), projects differ significantly in terms of their intended outcomes.^{[footnote 19]} In order to address this diversity, the programme-level evaluation will include several sub-evaluations of different outcomes that TF-funded projects are expected to affect. Each sub-evaluation will revolve around one outcome and should include projects that would affect that outcome. The results of each sub-evaluation will be combined as part of the value-for-money (VfM) assessment (discussed more below). The high diversity of project outcomes, even within a primary theme, makes it challenging to identify the projects that can be grouped under one outcome set for evaluation.

In the early stages of the evaluation, the projects will be re-categorised based on the outcomes they are expected to affect. To do so, the projects will be assessed by their description and the outcomes they are likely to create will be identified. There are currently 516 projects with estimated project end dates within the evaluation period.^{[footnote 20]} A representative sample of 15% of these projects has been re-categorised as part of the feasibility stage, as described in Section 5.1.3. The remainder will be re-categorised in the initial stage of the evaluation.

Step 2: Defining the geographical unit of the analysis

For both the TD and the FHSF, the town/place was initially identified as the relevant geographical unit for analysis, as that is the area the fund intends to affect. TD impacts are expected to be defined at the town level.^{[footnote 21]} Although the FHSF projects occur in a particular high street or town centre, the outcomes and impacts are expected to be wider.^{[footnote 22]}

Although an understanding of the impacts on the town/place level is desirable, in practice, the programme-level evaluation of certain outcomes will need to be conducted on a more granular level. First, some outcomes have a much narrower geographic impact boundary than a town/place. For example, footfall would most likely be impacted around the project/street; looking at footfall in a given town might not reveal the changes around the treated street, as it would also include areas outside the expected geographical impact boundary.

Second, given the difference in the geographical spread of projects across towns/places between TD and FHSF, there might be a need to decide the geographical level of impact separately for the 2 funds to account for outcomes overlapping. As such, the most appropriate geographical areas of evaluation for each outcome of interest will be assessed separately for TD and FHSF. This assessment of the appropriate geographical boundaries for each outcome will be primarily based on evidence from the academic and impact-evaluation literature, where possible. For example, Gibbons et al. (2021) analyses the impact on employment within a 0- to 5-kilometre boundary of the intervention location. The assessment will be complemented by input from stakeholder interviews gathered as part of the intervention-level evaluation, in particular where there is a lack of evidence from the existing literature.

Step 3: Identifying data sources

A quasi-experimental approach to evaluation at the programme level requires data to be:

• Available at a local level: many outcomes from TF projects are likely to occur within a relatively constrained local area (such as a town, output area, or Lower Layer Super Output Area [LSOA] level). To be reasonably confident of detecting impact, means data are needed at a geographical level that matches the likely area over which outcomes are expected to occur.
• Available nationally: data needs to cover both treatment and control units geographically.
• Tracked and updated regularly over time:data need to be available, at a minimum, before and after the completion of TF projects so that the formation of a baseline and observation of change over time for both treatment and control areas are possible. To be reasonably confident of detecting impact, the data need to be regularly updated so that the period the data refer to matches the likely timeframe for impact relatively closely.

Mainly secondary data sources will be used for the programme-level evaluation. Section 5.3 provides an overview of the assessment of secondary datasets’ availability for each evaluation theme.

In some limited cases existing secondary datasets may be supplemented through web-reading exercises. For example, natural language processing could be used to gather data from online forums and job posting sites in order to build new datasets on job postings and vacancies. This option is limited as it requires accessing information on various websites and recording the available data. Section 5.3 provides further details about this option.

While primary data collection through surveys can, in principle, gather required data at the right granularity level and frequency (including for baselining), it is not a viable source for the programme-level evaluation due to its high resource and cost requirements.

Step 4: Identifying the control groups

A quasi-experimental design (QED) relies on being able to identify a set of towns/places that are similar to the treated town/places to use as comparators. Based on the result of testing several techniques (including RDD and synthetic controls), matching techniques can be deployed to identify control groups, and this will be done separately for TD and FHSF.

The appropriate methodology for identifying a suitable control depends on how the treatment group was selected and what information is available for the identification of the control group. The 2 funds, TD and FHSF, have distinctly different selection processes and designs, which means that the selection of control units would also have to be done separately and differently for each. Given those differences, TD’s control group selection should mainly focus on finding towns with deprivation characteristics, such as income deprivation domain, productivity, and skills similar to the chosen towns. For the FHSF, the control selection would be based on factors such as deprivation rates, vacancy rates, and geographic location that are common to those streets deciding to participate in the FHSF competition and then being successful.

As part of assessing the best econometric approach for TD and FHSF, the feasibility of several techniques was reviewed and tested. The list below presents the econometric techniques that were tested and the reasons they were found not to be feasible:

• Randomised Control Trials: requires randomisation of treatment and control unit prior to treatment rollout. Neither TD nor FHSF were randomly allocated to places, and both were rolled out before the start of the feasibility assessment.
• Step-Wedge Randomised trial: requires treatment to be rolled out at different points in time on a randomised basis. In both TD and FHSF, the treatment timing is not randomly assigned, and differences in funding timing across places are due to process considerations. The differences in project end dates are relatively small (within 2 years), leading to a not large enough step-wedge for comparison of treatment and control before the control moves into the treatment group.
• Interrupted Time Series: compares outcomes of interest over time before and after an intervention while taking into account any underlying trends. Both TD and FHSF have projects ending across various points in time, which have impacts that might take time to realise and show trends.
• Regression Discontinuity Design: makes use of a treatment threshold (or cut-off) to define treatment and control groups. It is assumed that units just below the cut-off provide a good control group for those just above the cut-off. In the case of TD, the selection methodology for towns did not have any cut-off threshold. Although for FHSF, there was a scoring selection system, only 29 towns were not selected, making the sample size too small for econometric analysis.
• Instrumental Variable: uses a proxy variable that:
  • does not correlate with the characteristics of the treatment and control groups and
  • must influence participation in the treatment, a condition known as ‘relevance’

Given the complexity of TD and FHSF with multiple intervention themes and possible outcomes, finding instrumental variables that would satisfy the 2 criteria would be challenging

All of the techniques mentioned above were found unfeasible for various reasons, for example, the need for trial design before the start of the intervention for Randomised Control Trials (RCTs).

It was found that a PSM approach can be used to identify the control group for both TF sub-funds based on a series of initial tests for both TD and FHSF. For TD, PSM tests showed it would be feasible to find an appropriate control for almost all the towns that were chosen for TD across all categories.^{[footnote 23]} For FHSF, the preliminary results found that, even when using a limited and basic set of local characteristics, some predict the likelihood of being chosen to FHSF with statistical significance.

Tests in both cases are preliminary, and only after a full evaluation of each outcome will the true quality of the matched control groups materialise. There is also a need to include any further drivers that might impact selection into the funds. A baselining exercise will be undertaken for each outcome to assess any pre-intervention trends which would reveal any factors that might have impacted selection into the programme.

Step 5: Econometric analysis

The econometric analysis will compare observed changes (before and after the treatment, which is, in this case, the completion of the fund projects)^{[footnote 24]} in outcomes between the treated and non-treated towns (that is, treatment versus control). This approach is DiD. It allows the assessment of the extent to which the changes in outcomes seen were above what would have been seen without the funding over the same time in similar towns/places.

There are several considerations to be taken when specifying the DiD approach.

Displacement issues

Place-based interventions might also affect neighbouring areas to the impact boundary areas; however, since a DiD only compares the impacted areas to the control areas, it does not identify displacement effects. To address this issue, a comparison of the changes in the impacted areas to neighbouring areas that were beyond the geographical impact boundary for that outcome will be conducted.

For example, suppose through the DiD, an observation is made that footfall has increased in streets that received FHSF above the average increase in similar towns over the same time. This would be the result of the simple DiD. However, the observed increase might be because visitors started frequenting the treated streets instead of going to the high streets they used to visit before the project. This would be considered displacement, meaning that the original impact which was found does not create a real change but rather moves outcomes from one area to another. A positive unintended effect can also occur in neighbouring areas where a positive impact around the impact area will be observed – this is called spillover.

To analytically assess if displacement (or spillover) occurred and to what extent, outcomes within the impact boundary and neighbouring areas will need to be compared. The common approach is creating concentric circles around the area of impact. Comparing the outcomes between the impact area and the concentric circles around it reveals if there are significant changes in neighbouring areas (that is, if displacement/spillover occurred).^{[footnote 25]} Any displacement evidence will then be discussed together with the identified impact to understand the net effect that the fund had.

Other contextual drivers

The evaluation will seek to isolate the impact of the intervention from other potential drivers (such as other funds or projects). Contextual drivers, such as higher inflation and prolonged effects of recovery from COVID-19, may also impact the outcomes which will be assessed. These factors have to be accounted for, as much as possible, in the econometric design.

For the evaluation of each outcome, the current contextual drivers will be explored with the goal of controlling them, as much as possible. The ability to do so will depend on data availability with regard to these drivers. If data are not available, the aim is to address those through proxies or other design choices. Any drivers which cannot be addressed will be mentioned as limitations.

5.1.3    Limitations

Outcomes included in the programme-level evaluation

Short- and medium-term outcomes of projects, such as footfall, were concluded to be feasible to assess through the programme-level evaluation. Long-term impacts, such as deprivation levels and an increase in high-paying jobs, are not likely to be realised in time for the evaluation, as project completion dates are typically too close to the end of the evaluation period. As most of the projects will end just before the evaluation is undertaken (mid-2025), it is likely that long-term impacts will not have time to present any significant changes.^{[footnote 26] [footnote 27]}

As such, the programme-level evaluation might not be able to detect statistically significant changes in the outcomes. The evaluation will aim to identify short- and medium-term outcomes (please see Section 3.3 for possible outcomes, such as changes in transport usage and local skills) that are more likely to be affected shortly after the conclusion of the projects and, for which data, at the relevant frequency and geographical granularity level, are available. The analysis will focus on the outcomes for which it is more likely that enough time has passed for changes to materialise, where data collection gaps are smaller (allowing for a higher probability for changes to be detected), and those for which sample sizes are the largest (allowing for higher ability for statistical inference).

In some cases, it may be possible to combine evidence on identified short- and medium-term outcomes with a theory-based approach to infer potential long-term impacts. For example, long-term impacts, such as changes in local economic growth and productivity, may be preceded by changes in short- and medium-term outcomes, such as local skills, employment, and land-use changes. These links between short- to medium-term outcomes and long-term outcomes will be explored further as part of the theory-based intervention-level evaluation described in Section 5.2. The links in outcomes will also be explored through further literature review as part of the programme-level econometric analysis to draw upon the most up-to-date evidence available on how the short- and medium-term outcomes identified may contribute to the long-term impacts of interest.

However, any long-term inferences would need to be interpreted and caveated appropriately. In particular, evidence on how short- and medium-term outcomes impact long-term impacts from the wider literature is not sufficient to establish a causal link between short- and medium-term outcomes from the TF and potential longer-term impacts and would only provide an indication of the long-term impacts (or range of impacts) which might emerge. These potential links can be affected by other factors, such as displacement effects and leakage.

For example, an increase in short-term employment and skills may not translate into longer-term economic growth if there is skills leakage due to trained individuals relocating to other areas. These factors will need to be carefully considered when inferring long-term impacts. The feasibility of inferring long-term impacts is also contingent on identifying statistically significant short- and medium-term outcomes.

The granular geographic impact evaluation and the division of the econometric analysis into sub-evaluations may limit the ability to assess the overall impact of the fund on a place. Towns receiving funds were able to invest in their areas of greatest need. This means that the econometric work focusing on a small set of pre-selected outcomes and based on granular geographic data may not fully capture the overall impact of the fund. Rather, the approach provides a pragmatic way forward, allowing for an assessment of as many of the anticipated impacts of the fund as possible within the relevant timeframe while flagging limitations.

Sample sizes

A QED is based on statistical inference. Econometric analysis can detect impacts with a high enough degree of confidence if the changes are very big, even if based on a small sample size. It can also detect smaller changes, but this requires a large sample size to reach a high confidence level that the observed changes are not random. Non-statistically significant results of an analysis that is conducted on a small sample size can be due to changes not being large enough or because the sample was too small. It is not possible to determine with certainty which is the cause. In these instances, a suggestion will be made to redo the analysis in the future – if sample sizes can be increased. 

The assessment concluded that the sample size is currently sufficient for a QED analysis, but further delay in project completion dates might reduce the number of projects in scope. The maximum possible sample size for the treatment group would be the total of all treated towns and places.

For TD, that is 89 towns, and for FHSF, that is 58 places^{[footnote 28]}. However, those numbers are reduced when only towns and places with data on project end dates and projects finishing before the evaluation are considered. In the information that was available for the feasibility assessment,^{[footnote 29]} 2 towns receiving TD did not have information about the end dates of all their projects. Of the towns with data available on project end dates, all the projects in 11 TD towns will likely end after April 2025.^{[footnote 30]} This means that only 76 of 89 TD towns can potentially be included in the programme-level evaluation.

For FHSF, 2 places did not have information about end dates for all their projects. Additionally, for one place, all projects are likely to end after April 2025. This means that only 55 places can potentially be available for an evaluation analysis.

Given the diversity of the projects within towns and the themes into which they are categorised, the programme-level evaluation would include several sub-evaluations for projects grouped by common outcomes. That means that sample sizes for the analysis of some outcomes would be further reduced. Although projects will be grouped by outcomes in the evaluation, a preliminary test of sample sizes has been conducted. The test was based on (i) the project’s primary theme, which LAs assigned to each project, and (ii) a manual re-categorisation of outcomes for a random, representative sample of 15% of the TF projects in scope for this evaluation.

• Sample sizes based on intervention themes: looking at projects under the theme of ‘Regeneration’, for example, which has the highest number of projects under both TD and FHSF, 87 towns and 56 places have at least one project categorised into this theme. Of these, 2 towns in TD and 2 places in FHSF have missing data on end dates for all projects within the theme. In addition, 22 towns in TD and 4 places in FHSF all have projects under the ‘Regeneration’ theme, which are likely to end after April 2025. This means that only 63 towns and 50 places in the ‘Regeneration’ theme can potentially be available for an evaluation analysis.
• Sample sizes based on manual re-categorisation of outcomes: as part of the feasibility assessment, a preliminary manual re-categorisation of a random, representative sample of 15% of the TF projects in scope for this evaluation has been undertaken. Each individual project can be assigned multiple outcomes. For example, a given project may have outcomes related to both sustainable economic growth and employment and skills. Based on the preliminary re-categorisation, there appears to be a viable sample of projects affecting the outcomes of sustainable economic growth, employment and skills, pride-in- and perception-of-place, and physical connectivity. Over half of all manually re-categorised projects have outcomes related to sustainable economic growth, nearly half have outcomes related to pride-in- and/or perception-of-place, a quarter have outcomes related to employment and skills, and a quarter have outcomes related to physical connectivity. However, the sample of projects which will be completed within the evaluation timeline and are expected to directly affect digital connectivity is much more limited. Preliminary tests suggest that only around 1% of projects are expected to deliver this outcome. These results mean it is unlikely that outcomes related to digital connectivity will be possible to assess as part of the programme-level econometrics. Finally, a relatively small proportion of projects in the re-categorised sample (less than 10%) have outcomes directly related to wellbeing and social mobility, although, as these outcomes may be indirectly affected by a broader range of projects, it may still be possible to identify a large enough sample size to explore these outcomes, econometrically.

Further delays may reduce the number of towns and high streets which can be included in the programme-level evaluation even further. Overall, while both sample size tests suggest there is a large enough sample to assess most of the key outcomes of interest as part of a QED, it cannot be said with certainty that the sample sizes would be large enough to observe statistically significant effects for all outcomes of interest (in particular for digital connectivity).^{[footnote 31]}

However, it is important to note that a risk to sample size does not mean an attempt will not be made to perform the evaluation with a QED approach; it simply increases the chance that statistically significant outcomes will not be detected.

5.1.4    Risks and mitigations

There are several risks to the programme-level evaluation, which have been grouped into 2 high-level categories:

• Risks due to project delays: delays to project completion beyond the evaluation period would lead to reduced sample size for the econometric analysis (as these projects would have to be dropped), which might reduce the chance of finding statistically significant results. Should this risk materialise, a suggestion of the next steps which can be taken after the evaluation period for DLUHC will be made to assess if impacts are not identified due to sample size instead of lack of actual impact.
• Risks due to data availability: data gaps or lack of data at the right granularity and frequency might restrict the ability to evaluate certain outcomes. This risk will be mitigated by looking for other proxies for which data might be available. Also, an assessment will be made to conclude if primary data collection (mainly through web reading) will be possible to fill in the gaps (see section 5.3 for further discussion about data). This exercise will be conducted shortly after the end of the feasibility stage to ensure gaps are filled as early as possible to maximise the ability to have baseline information.

5.2  Intervention-level evaluation

Intervention, in this evaluation, refers to similar projects within the same town that are grouped together for evaluation purposes. In particular, interventions are a set of one or more projects from the same TD and/or FHSF that (i) are in the same town/place, (ii) are likely to have similar outcomes over a similar timeframe, and (iii) are geographically close enough to influence the same people/businesses. This approach is taken to address the attribution problems within local areas. This differs from how interventions are defined in other parallel evaluations, such as the UK Shared Prosperity Fund (UK SPF).^{[footnote 32]}

5.2.1    Summary of feasibility assessment

It was concluded that it is feasible to evaluate 20 TF interventions using a theory-based case study approach that employs Realist Evaluation and Qualitative Comparative Analysis (QCA) approaches to capture short- to medium-term outcomes. Links to longer-term impacts will be based on evidence from wider literature rather than measured directly. The proposed flexible theory-based approach will primarily draw on evidence from project-level reporting data, available secondary data sources, and stakeholder interviews.

In some cases (particularly for outcomes related to pride-in-place and local wellbeing), additional primary data will need to be collected. Covering 20 TF interventions should provide a sufficiently diverse sample to provide a range of learnings relevant to the evaluation questions. However, the limited sample of relatively large, long-running projects expected to be completed within the evaluation timeframe means that the overall conclusions from the intervention-level evaluation may not be representative of all project types.

5.2.2    Evaluation design considerations

Theory-based evaluation methods

The intervention-level evaluation has 2 key aims: (i) to understand intervention-level impacts and validate the Theory of Change (ToC), and (ii) to understand the relative importance of project factors and how they interact within local contexts to create observed changes in outcomes. To address the first aim, using a Realist Evaluation framework is proposed to evaluate the impacts of each individual intervention. The results will then be combined with a QCA to systematically compare the Realist Evaluation findings across interventions and explore the factors and contexts associated with successful (or unsuccessful) interventions. These approaches are flexible to different data sources and types, which is a key consideration in the evaluation of the TF due to the diversity of project types. These approaches are summarised below:

• Realist evaluation aims to understand ‘what works, for whom, and in what circumstances’ by gathering evidence on the hypothesised causal mechanisms detailed in the individual intervention ToC, according to the Magenta book. There are 4 steps to the Realist Evaluation that are repeated for each selected intervention:

1. write hypotheses for the expected outcomes and mechanisms

2. collect relevant primary and secondary data

3. run interviews and workshops to gather qualitative evidence

4. analyse the evidence against the written hypotheses

This process produces a set of conclusions for each intervention.

• QCA allows the comparison of intervention factors and contexts to determine their individual or combined contribution to the hypothesised outcomes. It uses a systematic, process-driven approach to identify the combination of factors necessary or sufficient to produce a certain result. It allows evaluators to produce comparative statements across different interventions and identify how (and why) interventions of different types, or in different places, may have delivered different outcomes.

The QCA analysis will be run after all interventions have gone through the Realist Evaluation process and produced a set of conclusions. The high-level process will:

1. compare all interventions and arrive at a conclusion whether each topic (for example, TF impact on footfall) and group should be divided into ‘agree’ or ‘disagree’

2. identify the interventions that have the same conclusions and explore whether there are similar characteristics among interventions that agree

3. produce comparative statements (for example., footfall increased in areas that had funding for town centre regeneration but not in areas that had funding for skills development)

In cases where the sample is too small to reliably detect a pattern, a more qualitative approach will be taken to compare findings from the Realist Evaluation across interventions without the use of QCA software. 

Another commonly used theory-based evaluation method is ‘contribution analysis’. It was concluded that this method is not suitable for the TF analysis due to the large degree of project and context variation, which could bias the influences found in contribution analysis. Additional methods considered but not recommended, given this context, include Bayesian Updating and Process Tracing.

A theory-based evaluation includes both qualitative and quantitative analysis. As a result, quantitative techniques such as quasi-experimental analysis can be used as part of the case studies, where appropriate and feasible, as part of the overall theory-based approach. Within the Realist Evaluation framework, available data sources and quantitative analysis techniques will be used in order to understand the intervention-level impacts. Quasi-experimental methods will be considered on a case-by-case basis. In particular, depending on the type of intervention and the data available at the intervention-level, it may be possible to compare the outcomes of individuals or businesses in the area treated to outcomes in areas where there has been no treatment. However, data limitations are likely to prevent this from being a core part of the intervention-level case studies.

5.2.3    Summary of evaluation approach

Defining an intervention

Initially, it was expected this evaluation would be undertaken by selecting a set of projects, understanding their contexts, and identifying mechanisms that account for observed changes in outcomes. However, it is common for multiple projects within a single TD or FHSF portfolio to exhibit similar expected outcomes, making it challenging to determine whether an observed change is attributed to Project A, Project B, or a combination (commonly known as an attribution problem).^{[footnote 33]}

Therefore, the approach has been adapted to strike a balance between understanding what works at a project level and recognising that some interventions may be interconnected (geographically and/or by theme). As such, an ‘intervention’ is defined as a set of projects (or just one project) from the same TD and/or FHSF that are grouped together based on 3 criteria.^{[footnote 34]} The projects are:

1. in the same town/place

2. likely to have similar outcomes (over a similar timeframe)

3. geographically close enough to influence the same people/businesses

The evaluation then takes place at the intervention level and includes the grouped projects from that town’s intervention. This approach increases the ability to evaluate the impacts of selected projects while addressing the attribution problem.

This approach is applied consistently to both the FHSF and TD projects. In practice, the scope of projects included within an intervention for TD and FHSF is likely to vary: 

• FHSF: most projects are designed to be interconnected and take place in a highly concentrated geographical area. As such, an ‘intervention’ for the FHSF is likely to cover the entire FHSF awarded for a given place.
• TD: projects within a town can be geographically dispersed and target different (and sometimes multiple) outcomes. Using the criteria above, an intervention might focus on one project in a given town, whilst a different intervention in a different town may cover several projects. This will depend on the specific project mix in the chosen town.

This difference in the number of projects which will be evaluated for each chosen town arises from the different geographic scopes of TD (whole towns) and FHSF (a specific high street).^{[footnote 35]}

Proposed evaluation approach

The proposed approach to the intervention-level evaluation is as follows.

• Step 1: select starting projects for case studies. Twenty projects will initially be selected, 10 from FHSF and 10 from TD, based on a set of criteria and definitions agreed upon with DLUHC.^{[footnote 36]} A timeline will be decided on for each specific case study, depending on the project’s expected completion date and when the effects are expected to be realised.^{[footnote 37]}
• Step 2: frame the intervention. A determination will be made whether additional projects should be included in the intervention definition using the process described above. Next, the intervention-specific ToC will be developed, the evaluation questions will be defined with input from key stakeholders, and what data are available to assess these questions will be confirmed.
• Step 3: primary and secondary data collection. Data will be collected from sources relevant to the evaluation questions. This could include (but is not limited to) (i) project-level reporting data, (ii) existing secondary data, (iii) qualitative information from stakeholder interviews and workshops, and (iv) primary data collection via targeted surveys for key indicators of interest.
• Step 4: intervention-level case study analysis based on a Realist Evaluation framework. Using the context and ToC defined during step 2, the data collected in step 3 will be analysed to (i) explore how the outcomes of interest have changed over time and (ii) identify the presence and relevance of the hypothesised mechanisms and their association with the outcomes of interest.
• Step 5: a comparison across interventions using the QCA. Once all 20 case studies are complete, the Realist Evaluation results (step 4) will be compared across interventions to identify common factors that drive success (or lack of) in the outcomes of interest.
• Step 6: reporting. The results will be interpreted in light of the original aims and within the context of TF, and the results, conclusions, and any caveats and limitations of the analysis will be reported.

Steps 2 through 4 are repeated for each of the 20 individual projects.

Data sources for the intervention-level evaluation

For each intervention, data will be drawn from a selection of sources, including:

• Project-level administrative data will provide context, such as whether a project was delayed, what other funding it received, output information (such as the number of students enrolled), and some short-term outcomes.
• Secondary data will provide quantitative outcome data, such as local employment rates.
• Surveys will provide a mixture of quantitative and qualitative insights, in particular, they will be used to cover gaps in the secondary data, such as perception-of-place, pride-in-place, and wellbeing.^{[footnote 38]}
• Stakeholder interviews will provide various qualitative insights from additional contextual information, such as LA capabilities and capacity.

5.2.4    Main considerations and challenges

Selecting a range of diverse interventions

The selection process has been developed to ensure an appropriate range of projects is included to maximise the learning from the evaluation. Initially, the 961 TD and FHSF projects are filtered according to the selection criteria listed below.^{[footnote 39]}This set of criteria follows a maximum variation sampling approach that ensures the final sample has as much variation as possible.^{[footnote 40]}

Through this diversity, it will be possible to explore the unique factors that lead to a given intervention’s success (or failure) while identifying any common patterns across the interventions. Pathfinder areas, which have more flexibility in how to use different sources of DLUHC funding across their projects, have also been excluded from the selection. This reduces the available sample of projects to 840.

Below is a list of the selection criteria by level of importance.

1. Type of project funding: select c.10 projects from the TD and c.10 from the FHSF (and at least one town with both).

2. Project progression: select projects which are projected to be completed (and expected to yield outcomes) before the end of the evaluation period.^{[footnote 41]}

3. Project type: select at least one project from each primary intervention theme (as defined by DLUHC).

4. Geographic location: select projects from different parts of England, including a mix of coastal and non-coastal locations.

5. Size of award: select projects with a range of funding offer sizes.^{[footnote 42]}

6. Size of town: select projects from across a range of town sizes (based on population).^{[footnote 43]}

It is necessary to exercise a degree of judgement when applying these criteria, as it may not be possible to cover the full range of each criterion. For example, covering each area of England may not be feasible in practice. However, ensuring a mix of projects across these criteria reduces the chance of inadvertently excluding a distinct group from the evaluation.

A preliminary selection of projects has been conducted as part of the feasibility assessment (based on July 2023 reporting data), and it has been found to be feasible to select 20 projects that meet the selection criteria.

Changes to projects

Given that the evaluation period spans more than 2 years, projects that were selected at the feasibility stage may no longer be feasible to evaluate when the time comes for the actual evaluation. This is because changes to the selected projects are likely to occur throughout the evaluation. These could be due to delayed project completion dates or projects merging/cancelling due to cost changes. To mitigate the risk of some selected projects becoming non-feasible to evaluate, 10 reserve projects have been selected as substitutes if some projects in the shortlist are no longer suitable to evaluate.^{[footnote 44]}

In case there is a need to resample from the reserve list of projects, this will be done according to the original criteria (in consultation with DLUHC). However, some selection criteria only have a limited number of projects which meet them. If projects from the reserve list are needed, it might not be possible to maintain a sample of projects for the case studies that meet all the selection criteria outlined in the section above. Any resampling will be communicated and agreed upon with DLUHC. Places which have projects in the reserve list will also be engaged with early on in the evaluation process to ensure they are aware that they might be asked for input in the evaluation. Baseline data on reserve projects will also be gathered, where appropriate, to ensure that they are feasible to include in the evaluation at a later stage if necessary.

Evaluation timeline

Identifying longer-term impacts driven by a wide range of factors is likely to be challenging in the intervention-level evaluation. The timeline of the evaluation means observation of changes in some outcomes might not be possible, as insufficient time will elapse following the completion of the project for changes to occur. Data granularity and frequency of collection and reporting may further restrict the ability to detect changes in some outcomes (further details about challenges in this evaluation are available in Section ‎5).^{[footnote 45]} Those issues are likely to occur for outcomes such as the overall level of employment in an area, changes in net business creation, and changes in deprivation.

As a result, the intervention-level evaluation will need to focus on more directly observable ‘leading’ indicators that are already known to be linked, via the existing evidence base and wider academic literature, to changes in the longer-term outcomes. For example, longer-term employment and economic outcomes are associated with leading indicators on metrics such as business confidence and number of individuals trained/upskilled. Longer-term outcomes can also be explored further in future evaluation work after the conclusion of the current evaluation timeframe – in Q1 2026.

5.2.5    Risks and mitigations

There are several risks to the intervention-level evaluation, which have been grouped into 3 high-level categories:

• Risks due to selected projects completing after the evaluation timelines: shortlisted project completion dates might be delayed. Severe delays might lead to projects being completed outside the evaluation timelines and thus outside the scope of the evaluation. This risk will be mitigated by having ‘reserve’ case studies. However, larger regeneration projects are likely to be most severely delayed, limiting their representation in the sample. As a result, the overall conclusions from the intervention-level evaluation may not be representative of larger, long-running projects, or all project types.
• Risks due to disengagement of LAs: low engagement from LAs will result in difficulties validating the ToC and accessing the necessary data and personnel for qualitative interviews. This risk will be mitigated by engaging with the relevant DLUHC teams and external contractors to avoid unnecessary overlaps, which could lead to the overburdening of LAs and reduced engagement. Preliminary discussions with the LAs will be held to explain the process and importance of the evaluation early on in the main stage of the evaluation. These preliminary discussions will also include a clarification that the impact of the TF is being evaluated, and not the performance of the LAs, in order to minimise the risk that LAs are reluctant to participate due to concerns over being compared unfavourably to other areas.
• Risks to the evaluation’s effectiveness: the robustness of the qualitative evaluation could be limited if there are data quality and access issues or insufficient time for effects to materialise. This risk has been mitigated by adopting a flexible theory-based evaluation approach, which can draw on a wide set of evidence depending on what is available and feasible to collect.

5.3     Impact evaluation questions and datasets for use in the evaluation

This section provides an overview of the key evaluation questions for the TF evaluation and the key data sources which can be used to inform the overall evidence base.

5.3.1    Evaluation questions

Based on the ToC and Logic model described in Section 3, the following evaluation themes and questions have been identified:

1. Sustainable economic growth: to what extent has TF led to long-term sustainable improvements in local economic growth (that is, continues to occur year after year)? Indications of this may include business turnover, local labour productivity, high street footfall, and business investment.

2. Employment and skills: to what extent has TF led to improvements in local employment opportunities? Indications of this may include changes in the number of jobs, unemployment levels, employment across different sectors, and the number of individuals in training or further education.

3. Pride-in- and perception-of-place: to what extent has TF led to improvements in pride-in-place and perception-of-place? Indications of this may include changes in town perception by the local residents, visitors, and businesses and changes in community engagement and participation in local events.

4. Local wellbeing and social mobility: to what extent has TF led to improvements in local wellbeing and social mobility? Indications of this may include changes in deprivation levels, local household incomes, self-reported local happiness indices, and self-reported life satisfaction levels.

5. Physical and digital connectivity: to what extent has TF led to improvements in physical and digital connectivity? Indications of this may include changes in the quantity of local physical infrastructure, journey times, use of pedestrian and cycle paths, and the proportion of residents/businesses with access to superfast broadband.

6. LA capabilities: to what extent has TF led to improvements in local strategic management capability? Indications of this may include self-reported improvements in the capacity of LAs to deliver interventions and changes in the speed at which major planning applications are decided. The main sources for this will be primary data collection through surveys (to try and collect quantitative measures of self-reported capabilities) and interviews (to assess more contextual reasons for reported changes in abilities of LA leads) to assess changes in the capabilities of their workforce as well as their ability to provide other data (for example, number of workers with specific qualifications).

7. Differences in outcomes across different types of projects: to what extent have the outcomes and impacts above differed across different types of projects?

8. Differences in outcomes across different types of places: to what extent have the outcomes and impacts above differed across different types of places?^{[footnote 46]}

Both the programme and the intervention-level impact evaluation will seek to address all the questions above. The ability to do so will depend on the availability of data sources for the possible indicators. Sources can include secondary data sources, administrative and performance data recorded by LAs and primary data sources.

5.3.2    Assessment of secondary datasets

For the TF evaluation in general and programme-level evaluation in particular, secondary data sources are considered in the first instance, as these are more cost-effective and readily available.

For a secondary data source to be applicable to a given evaluation question, it needs to have the relevant geographical granularity, have coverage of the treatment areas and control areas, and be updated frequently enough so changes can be seen before and after the intervention occurred. A dataset should ideally be (i) able to show an impact within the evaluation horizon and (ii) easily accessible to be feasibly used for an impact evaluation.

Overall, it has been found that secondary data on key sustainable economic growth and labour indicators are available at the frequency and granularity required for the impact evaluation, although observing changes in longer-term outcomes, like employment and labour productivity, during the timeframe for the programme-level evaluation may still be challenging.

Physical and digital connectivity indicators are also relatively well-covered by the existing public data, although some gaps exist for physical connectivity, which will need to be explored further during the impact evaluation. However, there are more significant gaps in data on pride-in-place, perception-of-place, and local wellbeing and social mobility. These gaps will likely need to be filled with additional primary data collection (including baseline data collection) as part of the intervention-level case studies where possible. In terms of LA capabilities, while some secondary data are available at the appropriate level, there will likely be a need to rely more heavily on qualitative stakeholder input, given the difficulty of assessing this quantitatively. 

A high-level summary of the secondary data sources available for each evaluation question is provided below.

• Sustainable economic growth: data availability is High. Data on business turnover, business creation, and labour productivity are available from various Office of National Statistics’ (ONS) sources (such as the Business Register and Employment Survey, the Annual Population Survey, and the Inter-Departmental Business Register). Data on high street footfall are available from a number of proprietary datasets  (such as Springboard and Ipsos). Similarly, data on property prices and land use are available at a granular level from the Land Registry, websites like Zoopla and Rightmove, and the ONS, amongst others.
• Employment and skills: data availability is High. Data on skills/qualifications, earnings, and employment are available from various ONS-held sources (including the Longitudinal Education Outcomes and the Annual Survey of Hours and Earnings).
• Pride-in- and perception-of-place: data availability is Low. While some information is collected from sources like the Community Life Survey, it is not available at the granularity required for the impact evaluation.
• Local wellbeing and social mobility: data availability is Low. Data on proxies of social mobility are available to a certain extent, with information on household income (available from sources like the Family Resources Survey and housing affordability) possible to proxy using property datasets such as Zoopla and Rightmove. However, gaps still remain for indicators like child poverty, which does not appear to be available at the appropriate granularity and frequency. Information on wellbeing is more limited – while some information is available from sources like the Community Life Survey and ONS data on health outcomes are generally not available at the granularity required.
• Physical and digital connectivity: data availability is Medium. Good quality data on Broadband access at the required granularity can be acquired from sources such as Ofcom, and overall data on digital connectivity appears to be available. For physical connectivity, while data on indicators like journey times and usage of different modes of transport are available to a certain extent, key sources like the Department for Transport’s (DfT) National Travel survey data are not available at the required level of granularity, and there is a need to explore if more granular data can be obtained.
• LA capabilities: data availability is Medium. Information on various LA performance metrics is available from LG Inform. However, given the potential challenges with proxying overall LA capability, this data will likely still need to be complemented by qualitative stakeholder input to be interpreted appropriately.

5.3.3    Project-level Monitoring and Evaluation (M&E) data

As part of mandatory project reporting requirements, TF recipients are required to report to DLUHC on project spending, co-funding, progress against timelines and budgets, the number of full-time equivalent (FTE) individuals employed to deliver the investment, FTE permanent jobs created and safeguarded, and year-on-year changes in footfall over time. Recipients also report on a range of voluntary indicators across various themes. Below is a summary of some of the voluntary indicators.

• business: number of enterprises receiving non-financial support
• culture: number of visitors at cultural events, spending volume and value at cultural events
• economy: business investment, business sentiment, number of new enterprises, commercial vacancy rates
• education: number of students enrolling in/completing higher and further education courses
• health and wellbeing: average life satisfaction, happiness, and anxiety
• place: satisfaction by residents with living in area, change in civic participation, change in crime rates
• regeneration: number of buildings upgraded to an Energy Performance Certificate (EPC) rating of C or above
• transport: pedestrian, cycle, and vehicle usage, satisfaction with local transport

As project-level M&E, data are only collected for treated towns (it is not possible to use them as part of the programme-level econometric evaluation) and are only usable as part of the intervention-level case studies. While only a limited sample of monitoring data was available at the time of this feasibility assessment, the understanding from the data received to date and discussions with the DLUHC team is that the completeness and quality of M&E data are mixed and will vary across the different recipients.

The quality of the project-level M&E data will, therefore, need to be assessed as monitoring data continues to be reported and incorporated into the intervention-level evaluations on a case-by-case basis. As a result, it is mainly expected to be a source of information on immediate project outputs and general context for the intervention-level case studies, as opposed to a key component of the quantitative evaluation. However, in some cases, it may serve as an additional source of quantitative data for the assessment of outcomes and impact in the case studies.

5.3.4    Primary and other data collection requirements

While there is a range of available secondary data sources and project-level M&E data, there are some notable gaps in relation to key evaluation questions. Primary data collection will be conducted and other data collection options will be explored to address these gaps.

Surveys of impacts on pride-in-place and local wellbeing

Gaps are particularly significant for data on pride-in-place and local wellbeing. While some sources for this information have been identified (Understanding Society, the Community Life Survey [CLS], and the Annual Population Survey), these datasets do not have the required graphical granularity for evaluation purposes.^{[footnote 47]} In addition, these sources are primarily focused on questions of civic participation and wellbeing, as opposed to perceptions of the built environment and pride-in-place, as will be needed.

This issue affects multiple ongoing impact evaluations, including the evaluation of the Levelling Up Fund (LUF) and the UK Shared Prosperity Fund (UK SPF). As a result, DLUHC is currently commissioning an expansion of the Community Life Survey (CLS) to make it available at the LA level and to include additional questions relating to pride-in-place, high street regeneration, and community outcomes. While this enhancement primarily provides additional information for the UK SPF evaluation, the expanded list of questions and the ability to use sub-regional breakdowns can also support the TF evaluation.

This enhancement will only provide results at an LA level on an annual basis, which may not be granular and frequent enough for either the programme level or the intervention-level impact evaluation. An LA may include several towns, which can be a mix of treated and not treated towns/places and include a mix of interventions across the different programmes. LA-level outcomes may be used to construct a baseline for the control areas, but this information is unlikely to be granular enough to identify impacts in treated areas.^{[footnote 48]} To obtain information from the evaluation period, it will be necessary to commission a survey for a selected number of treated towns at the more granular level (for example, LSOA), aligning survey questions with the CLS enhancement.

This approach would provide consistency and allow the results from this survey to be compared with LA-level outcomes from the CLS enhancement of areas that did not receive the TF. While the additional survey could be deployed more widely and include a non-treated group (which then might be used for the programme-level evaluation), the current assessment is that a wide survey would not be cost-effective. As a result, additional pride-in-place surveys will be deployed only in towns/places selected for the case study analysis, as opposed to all areas that receive TF funding. This will include baseline surveys in areas where the projects of interest have yet to be completed. This baselining will occur for the list of reserve case studies as well to ensure that these case studies can be robustly assessed in the future if they are included in the evaluation.

Even with the CLS enhancement and additional survey work at the town or LSOA level, it may be difficult to establish a counterfactual or baseline for areas where TF projects have already been completed or are significantly underway. Even with these survey enhancements, it is unlikely that in all case studies it will be possible to estimate the additionality of the TF to pride-in-place and local wellbeing outcomes robustly.

Surveys of impacts on business outcomes

The existing data on business outcomes in datasets like the Business Structure Database (BSD) may not be appropriate or available at the required geographic granularity to observe business outcomes in some intervention-level evaluation case studies. It may be necessary to commission targeted surveys of businesses either directly involved in a TF project or in an area affected by a TF project. Such a survey will be deployed in targeted cases where there is a significant gap in the evidence and not in all of the projects selected for the intervention-level evaluation (or the programme-level evaluation).

Scope for using web reading to create novel datasets

For the programme-level evaluation, in some limited cases, it may be possible to complement existing secondary datasets with web-reading exercises. Web reading can be used to gather data from sources like online forums and job-posting sites using natural language processing to build new datasets on key outcomes, such as job vacancies, high street footfall, and visitor numbers to public places.^{[footnote 49]} However, there are limitations to the data which can be gathered via this method due to Application Programming Interface (API) limits, which vary across websites, access costs, and verifiability of data due to real-time changes, and the resources required to collect and clean this data can be substantive.

As a result, a continued exploration of areas where novel datasets can be created using web-reading and machine-learning techniques will be conducted as part of the evaluation, and consideration will be given to any sources that can complement the primary and secondary sources described above (if that would be appropriate and proportionate). Frontier Economics will coordinate with the Spatial Data Unit at DLUHC to learn from the team’s experience and ensure compliance with data collection protocols and effective data collection methods.

5.3.5    Managing evaluation datasets

The datasets described in this section may be relevant for future evaluation of the TF (beyond the current contract period) as well as for evaluations of other DLUHC programmes. As a result, the relevant teams within DLUHC will continue to be consulted to ensure that any datasets created for this evaluation are stored and maintained in a manner that allows them to support future evaluation work where appropriate. This will also help to avoid inconsistencies in datasets and duplication of work across evaluations.

5.4     Evaluation timeline and implications for methodology

The current evaluation is expected to conclude by Q1 2026. The methodology described in this report has been chosen in order to maximise learnings within this timeframe and deliver as robust as possible evaluation within the time available for the impacts of the TF to emerge.

While this feasibility study has not directly assessed the feasibility of a longer-term evaluation (beyond the current evaluation timeline), the existing evaluation design could be extended post-Q1 2026 to further explore the impacts of the TF beyond the current evaluation funding period. In particular, future evaluation work could:

• Rerun the programme-level econometric analysis once more data are available and additional projects have been completed. The quasi-experimental approach, proposed as part of the current evaluation, could be rerun in the future, drawing on the additional data and a larger sample of completed projects available post-2026. This rerun would add additional robustness to the analysis, and longer-term impacts, which were excluded from this evaluation, could also be assessed. This is unlikely to require fundamental changes to the approach proposed in the programme-level evaluation, and the same core DiD model can be used in the future.
• Revisit the intervention-level case studies again in the future to explore longer-term impacts. The 20 case studies proposed as part of the intervention-level evaluation could be revisited after 2026 in order to assess the longer-term impacts of the interventions, which are unlikely to emerge within the current evaluation timeline. Revisiting these case studies would allow future evaluations to explore how persistent changes in short- and medium-term outcomes have been in the selected case studies and if these outcomes have continued to evolve over time.
• Select additional longer-term case studies. The current case study selection has been limited by the expected completion dates of projects and, as a result, does not include many of the largest, long-running projects. Future evaluators could undertake additional case studies covering these larger projects, which would allow for a more complete picture of the impact of the overall TF programme. Where possible, these case studies should be identified and selected prior to their expected completion date in order to allow future evaluators to collect baseline data.

Any future evaluation work should be conducted around 2028. This timeframe would allow for additional time-series data to become available and projects to be completed. Waiting longer for follow-up evaluation work would increase the risk that the broader policy and economic context changes materialise, limiting evaluators’ ability to attribute impacts to the TF.

6. Value-for-money (VfM) feasibility

A VfM evaluation aims to assess if the intervention was a good use of public resources. It complements the impact evaluation by helping understand if the benefits of the intervention outweigh the costs.

6.1     Summary of the feasibility assessment

The feasibility assessment concluded that a break-even and a cost-effectiveness analysis should be feasible for the funded projects and can draw on evidence from the intervention-level evaluation. However, a full programme-level VfM assessment is likely not possible because of the difficulties in monetising many of the potential benefits of the Towns Fund (TF) and the difficulties in detecting statistically significant project impacts within the evaluation timeframe.

It may be possible to draw conclusions about the likely aggregate impacts of the TF across the portfolio of case studies. However, the sample of case studies for the intervention-level analysis does not include any projects that will be completed outside of the current evaluation period and, therefore, does not include some of the largest projects funded. As a result, these conclusions will not constitute a pseudo-programme-level evaluation. In addition, both the break-even analysis and cost-effectiveness analysis would provide a view of VfM of the projects overall but not the TF on its own, as projects, in many cases, are co-funded through other sources. As such, the impacts cannot be attributed to TF alone, as full funding is required for the impacts to occur.

6.2     Proposed approach

A VfM approach compares benefits to the costs of the intervention. As such, the feasibility study assessed what would be possible in terms of quantifying costs and benefits.

Information on the total costs of TF-funded projects is available in terms of the total funding from all sources required for project delivery. However, some projects are co-funded through other sources, including the Department for Levelling Up, Housing and Communities (DLUHC) funds, other departments, and private investment sources. In these cases, the projects’ benefits would be realised due to the combination of these funding sources, making it difficult to assess the proportion of the full impact attributable to the TF. As such, in any feasible VfM approach, the costs would include the full funding of the evaluated projects, and the VfM would assess the benefits versus the costs for the project rather than the TF on its own.

6.2.1    Break-even analysis

A break-even analysis assesses the likelihood that the benefits, converted into monetary values, outweigh the costs. Given the costs of the intervention, it examines the minimum impact needed to have occurred as a result of the intervention so the monetary value of the benefits outweighs the costs.

This type of approach requires the benefits to be monetisable but can be deployed even if a statistically significant change in the programme-level impact evaluation cannot be identified. For example, a break-even analysis could draw on estimates of the monetary value of one job being created and information on actual project costs to estimate the number of jobs which would need to be created for the project to break even. In support of this analysis, evidence from the case studies, the wider literature, similar interventions, and qualitative input from experts can be drawn upon to assess if the required change is reasonable to have occurred or will occur in the future.

6.2.2    Cost-effectiveness analysis

A cost-effectiveness analysis looks to identify which intervention produced the highest impact per pound spent. It compares the size of the impact created across various interventions and compares the results to the cost of each intervention. However, the analysis does not translate the impact into monetary terms. For example, a cost-effectiveness analysis could compare the impact of various interventions on pride-in-place relative to their costs, measuring the impact of the change in pride-in-place as opposed to the monetary value of this change.

This approach can be particularly helpful when impacts are difficult to monetise. Since this analysis will be based on case studies,^{[footnote 50]} it will allow for a comparative analysis of cost-effectiveness across projects targeting the same outcomes to be produced in order to understand what types of projects might be more or less effective at achieving certain outcomes. However, due to sample size limitations, any estimates of average cost-effectiveness by type of project from this analysis will need to be interpreted with caution.

6.2.3    Cost-benefit analysis

The feasibility of a full cost-benefit analysis has been assessed, and it has been concluded that this approach is not likely to be feasible for the TF. This type of analysis requires both to find changes in outcomes with statistical significance and be able to convert them into monetary values. Several risks to the proposed programme-level evaluation have been identified, which make it likely that impacts may not be identified (even short/medium-term) with any statistical significance. Even if statistically significant changes are identified for some outcomes, converting those into monetary impacts may not be possible or proportionate.

For example, the TF’s impact on the perception-of-place and community pride is not directly monetisable. With those risks, it is likely that a full-cost benefit analysis will not be possible for the TF VfM evaluation. However, where the programme-level evaluation finds statistical significance for outcomes that are monetisable, it may be possible to undertake a partial cost-benefit analysis. This will be considered once programme-level results are available.

7. Process evaluation feasibility

7.1     Aims and objectives

Process evaluations aim to understand how interventions are delivered and whether any factors help or hinder the effectiveness of the interventions. The process evaluation will complement the impact evaluation. While impact evaluations can isolate the outcomes and impacts caused by intervention, they are less useful for explaining why the impacts occurred or if they might have occurred differently under other circumstances.

The process evaluation will aim to understand how the Towns Fund (TF) (including Town Deals [TD] and Future High Streets Funds [FHSF]) application and delivery processes have worked. Feedback will be given to the organisations responsible for delivering TF, allowing for the processes to be refined in real time. Insights will also be helpful for learning lessons for future local growth funding programmes. The objectives for the process evaluation are to understand:

• how did the 2 different applications and funding processes (TD and FHSF) work
• to what extent and how did the design of the TD and FHSF processes support and promote better strategy-building, capacity-building, and partnership formation at a local level
• what barriers impeded the realisation of key success factors, and how were these mitigated
• to what extent did the theory of change (ToC) for the intervention work as expected
• were there any unexpected outcomes
• how effectively were the different types of projects delivered in different areas
• how effective was the sharing of learning and best practices across projects and areas with DLUHC within the lifecycle of the programme
• what are the lessons for future local growth funding programmes and other programmes/funds

7.2     Process maps

The first step of a process evaluation is to produce process maps that summarise key steps from funding allocation to the delivery and implementation of interventions. Process maps are helpful to ensure that all relevant processes and steps are interrogated at the mainstage process evaluation. 

• Based on a document review of TF and interviews with DLHUC delivery leads and local authorities (LAs), 2 process maps were developed to capture the processes for TD and FHSF.

7.2.1    Process map for TD

The TD process, followed the following steps:

• The Department for Levelling Up, Housing and Communities (DLUHC) selects towns to participate in TD; once selected, capacity funding is released. The LA for each town then establishes its Town Deal Board (TDB) and prepares investment plans with engagement from the local community. LAs may receive guidance and support when preparing their investment plans from third-party consultants, the Towns Fund Development Partner (TFDP) consortium, and/or other government agencies.
• LAs submit their investment plans to DLUHC for approval. If the plan is approved, DLUHC will offer the LAs Heads of Terms. If the plan is not approved, DLUHC will provide the LAs with feedback and support through the Towns Hub, and the LAs will have one opportunity to resubmit their investment plan.
• The LAs offered Heads of Terms will then develop their projects and business cases, again with support and guidance offered by third-party consultants, the TFDP consortium and/or other government agencies. LAs then submit summaries of their projects and business cases to DLUHC, and DLUHC reviews them. If the proposal is not approved, the LAs leave the TD process. If the summary is approved, DLUHC releases the first payment of funding to the LA.
• If LAs are delivering new projects using the TD funding, they will engage delivery partners at this stage and start delivering their projects. Whilst in delivery, projects may be changed either via Project Adjustment Requests or directly (provided that project changes do not affect more than 30% of the project outcomes).
• The LAs can make arrangements for project continuity post-funding.
• For the purpose of monitoring during project delivery (and up to 3 years after the project ends), LAs must submit progress reports every  6 months. The TDB will monitor reporting and risks to project delivery. DLUHC will use LA reports to check if delivery is on track. If delivery is on track, DLUHC will use this to share best practice learnings. If delivery is not on track, DLUHC Area Leads will provide feedback and support. If delivery does not improve, DLUHC may withhold further funding as a last resort.

Under TD, towns were categorised into different cohorts, allowing towns that need more time for preliminary work (for example, setting up TDBs and generating project ideas) the opportunity to do so. While the general process was the same for each cohort, the deadlines (for example, for submission of investment plans) varied.

Feasibility interview participants estimated that the whole process (from programme launch to final approval of business cases) takes around one year.^{[footnote 51]}Delays were often associated with timings of ministerial announcements (for example, there was a 5-month delay between confirmation of the projects to be funded and the Heads of Terms offer being made).

7.2.2    Process map for FHSF

The FHSF process, followed the following steps:

• LAs submit an expression of interest to DLUHC to participate in the FHSF. DLUHC reviews the applications submitted by the LAs. If an application does not meet the criteria for funding and the local authority is not eligible to participate, the process ends for the LA.
• If an application meets the criteria for funding and the LA is eligible to participate, DLUHC will then decide if the LA should proceed to develop its business cases and project plans. If an LA is not invited to develop business cases and project plans, the process ends for them. LAs who are invited to develop business cases and project plans will receive capacity funding to support them in this, as well as feedback on draft copies of their business cases.
• Business cases and project plans are then submitted to DLUHC to decide if they pass for capital funding. If the proposals do not pass, the process ends for the LA. If the proposal passes for capital funding, DLUHC will release the first payment to the LA. Once funding is released, LAs will start delivering projects, having engaged delivery partners. Throughout project delivery, LAs may benefit from support from the High Streets Task Force, and projects may be changed via Project Adjustment Requests (or directly provided that project changes do not affect more than 30% of project outcomes).
• The LAs can make arrangements for project continuity post-funding.
• For the purpose of monitoring, during project delivery (and up to 3 years after the project ends), LAs must submit progress reports every  6 months. DLUHC will use LA reports to check if delivery is on track. If delivery is on track, DLUHC will use this to share best practice learnings. If delivery is not on track, DLUHC Area Leads will provide feedback and support. If delivery does not improve, DLUHC may withhold further funding as a last resort.

Feasibility interview participants revealed that a significant number of applicants initially failed at the business-case stage due to errors in the value-for-money (VfM) element. With agreement from ministers, a clarification exercise on evidencing VfM provided applicants with further information on the importance of benefit-cost ratios and the option to re-submit their business case. This resulted in a higher number of successful business cases, with 72 out of 101 places being successful. Fifteen of those received the full amount, and 57 received a smaller proportion of their funding request.

7.3     Evaluation framework

Findings from the feasibility study conclude that a process evaluation (that addresses the aims and objectives outlined in Section 7.1) is feasible. The process evaluation will evaluate the same 20 core projects selected for the intervention-level impact evaluation to triangulate and maximise learning. An evaluation framework has been developed to address the process evaluation’s aims and objectives outlined in Section 7.1. The evaluation framework for the process evaluation distils the TF into 4 themes to cover programme and intervention-level learnings. The 4 themes are:

• Management and governance of the TF: this theme focuses on overall governance and programme management, including risk management and financial accountability.
• Design and planning of the TF and associated projects: this theme focuses on the design and planning of the TF and associated projects, including risk management, financial accountability, and the design of the fund.
• Structure and delivery of the TF and associated projects: this theme focuses on the structure and implementation of the TF activities, with an emphasis on how the approach of TD and FHSF may help facilitate outcomes and impacts.
• Evaluation and monitoring of delivery and outcomes: this theme focuses on how delivery and outcomes are monitored and evaluated.

For each theme, relevant evaluation questions, success indicators, and data sources have been identified.

7.4     Methodology

The following data sources will be used to answer the process evaluation research questions.

7.4.1    In-depth interviews

In-depth interviews are qualitative in nature and, as such, are particularly useful in eliciting rich descriptions and explanations from an individual’s perspective. In-depth interviews will be conducted with LAs to explore how projects were designed and implemented (including experiences of applying for funding for FHSF) among successful and unsuccessful applicants. Interviews will also be conducted with consultants LAs may have used to prepare bids or business cases. Additionally, in-depth interviews will be conducted with project deliverers (appointed by LAs) to ascertain how projects were delivered on the ground, what aspects worked well, what difficulties they faced, and how these were overcome.

The interviews will be semi-structured, following a topic guide that sets out key areas to be covered.^{[footnote 52]} Topic guides allow for a consistent approach between interviewers while allowing the discussion to be participant-led and for any unexpected topics or points of interest to be explored.

A case and theme-based approach (‘framework’ analysis) will be used to analyse the in-depth interviews. It involves the evaluators

• familiarising themselves with the evidence
• developing a framework to organise emerging themes (where columns represent themes and rows represent individual participants)
• summarising the qualitative data according to the key themes and sub-themes and
• working through the summarised data to explore the full range of processes, experiences and views, and seeking similarities, differences and the reasons for them

7.4.2    Focus groups

Focus groups are useful in collecting information from a group of stakeholders on their attitudes, perceptions and experiences in a way that allows participants to compare, contrast, validate and (respectively) challenge each other’s contributions. They will help to triangulate findings and contrast the views of LAs and project deliverers with those of other stakeholders regarding design, delivery, monitoring, and evaluation.

Conducting focus groups with a range of stakeholder groups is proposed, including:

• DLUHC delivery leads (including Area Leads and Towns Hub): to delve into management, governance and monitoring processes.
• TDB^{[footnote 53]}: to understand experiences supporting LAs in developing project plans/business cases/investment plans and their experiences of the monitoring process.
• High Street Task Force^{[footnote 54]}: to understand their experiences of supporting LAs to develop project plans/business cases/investment plans and to deliver projects.
• TFDP consortium^{[footnote 55]}: to delve into their experiences and contributions to the process of developing project plans/business cases/investment plans/applications for funding and project delivery.
• Project beneficiaries: to understand their experiences of project delivery and complement the views of LAs and delivery partners. Work will be done with LAs to identify project beneficiaries on a project-by-project basis. These can include businesses or individuals receiving support, users of facilities, and local residents (within a defined local area or mile radius).

Each focus group is anticipated to consist of 6-10 participants. Separate focus groups will be conducted per project to ensure project-based nuances are captured. The focus groups will be structured around the process evaluation themes and questions described in section 7.3. 

Where possible, the focus groups will be conducted in person, but remote participation will also be facilitated.

Evidence from the focus groups will be analysed using the ‘framework’ approach to qualitative analysis described in 7.4.1. When using this approach for focus groups, the rows in the matrix represent individual focus groups.

7.4.3    Observations on site

Observation is particularly well suited for gathering rich and detailed insights into complex phenomena, particularly insights which are less tangible and may be missed in interviews and focus groups.

LAs that took part in the feasibility interviews expressed their preference for evaluators to be on-site to see project delivery first-hand and to meet the teams involved in project design and delivery. Observation days on site will help evaluators get a ‘feel’ for how projects are delivered in practice. They will provide data in their own right and provide site-specific prompts in interviews and focus groups.

Evaluators will use pro forma to record their observations and conversations on the ground. Photos may also be taken, depending on the level of consent required and obtained. These notes and photos will be used to triangulate findings from in-depth interviews and focus groups and will be thematically analysed using the ‘framework’ approach described in Section 7.4.1.

7.4.4    Surveys

Surveys are effective in collecting a wide variety of information, including responses to factual, attitudinal, behavioural, and preference questions.

Surveying project beneficiaries (including local residents, users of facilities, programme participants, etc.) will be useful. The plan is that LAs and delivery partners will help access beneficiaries (it was concluded, from the feasibility interviews with LAs, that they would be willing to do this).

A set of process survey questions will be developed to understand beneficiaries’ engagement with projects and experiences of delivery. It is recommended that the survey questions will be asked simultaneously with the impact evaluation survey questions to maximise efficiencies and minimise the burden on respondents.

Survey methodologies will be agreed upon once the project selection is complete. These could include online, telephone, and push-to-web approaches.

Responses from the survey will be analysed to generate descriptive statistics. This will describe how responses differ for different types of respondents (such as residents) across areas with different socio-economic profiles.

Evidence from the surveys with beneficiaries will be triangulated with the qualitative focus groups with beneficiaries. The aim is that the focus groups will help to further explain survey findings.

7.4.5    Monitoring data

Monitoring data from DLUHC will be used to support the process evaluation under theme 1: ‘Management and governance of the TF’.

Accounting data will help provide evidence of any hidden costs and how these were addressed to understand whether the fund met its budgetary expectations.

The accounting data will be examined alongside insights from the focus group with DLUHC delivery leads.

7.5     Risks and mitigations

Two key risks and mitigations for the process evaluation are highlighted below.

• Difficulties recruiting participants for interviews and focus groups: some LAs may be less willing to engage with the process evaluation due to timescales and competing priorities (such as project delivery) or due to concerns that they are being scrutinised by DLUHC, which could lead to difficulties in recruiting participants for interviews and focus groups. This will be mitigated by allowing sufficient time for recruitment and fieldwork, giving participants flexibility in terms of the mode of participation (in person, online and/or via phone), and re-assuring participants that confidentiality and anonymity will be maintained. Also, a reserve project list is available in case sufficient participation cannot be achieved for a specific project.
• Design of research instruments: poorly designed research instruments (such as topic guides and questionnaires) will lead to unreliable data and could mean that some insights may be missed. It might also be that the instruments cannot capture the complexity and intricacy of programme delivery. Research instruments had been agreed upon with DLUHC ahead of fieldwork to ensure they meet policy and research objectives and balance the need for complexity without overburdening participants/respondents to mitigate this.

8. Evaluation plan

The current evaluation period will conclude in Q1 2026 and will focus on projects that finish by Q2 2025. Due to the uncertainty around the timelines of the projects funded by the Towns Fund (TF), the precise evaluation plan and timeline are subject to change. As a result, this section only provides a high-level indication of the current evaluation plan.

In the next stage, a number of key steps will be undertaken to set up future evaluation work in Q4 of 2023. This will include:

• Refining the econometric method and identifying further data gaps that arise: continuity will be maintained to refine the econometric approach for the programme-level evaluation and compile a detailed list of any additional data gaps that arise as a result of this refinement – beyond those already identified. This will cover the full list of indicators, including longer-term indicators which may not be possible to assess during the current evaluation to support the feasibility of future evaluation work that will take place after 2026.
• Collecting baseline data for both the programme-level and intervention-level evaluation: where there are gaps in the existing data, baselines will be collected using techniques such as web-reading and primary surveys to ensure baseline data are available for projects which have not yet been completed.
• Undertaking geographical unit analysis for the programme-level evaluation: the geographical level at which data need to be gathered for each outcome will be assessed in detail alongside the collection of baseline data.
• Engage with LAs selected for case studies: LAs selected for the intervention-level case studies and process evaluations will be liaised with to ensure they are aware of (and willing to engage in) the evaluation.

Following the completion of this initial work, the evaluation will be undertaken. The timing of the evaluation activities will vary by type and are subject to changes due to future project delays.

• Programme-level: since projects need to be completed and programme-level impacts need to materialise, the programme-level analysis will primarily occur in late 2025.
• Intervention-level: the first 3 intervention-level case studies will be undertaken in Q1 2024 for projects that have already been completed. This will allow for the refinement of the evaluation technique and for initial findings to be relayed to key stakeholders. The remainder of the case studies will be spread throughout 2024 and 2025, depending on the project completion date. The earlier case studies will be revisited in 2025 to check for longer-term impacts.
• Process evaluation: the process evaluation will proceed alongside the intervention-level evaluation, as it will focus on the same set of interventions.
• Value-for-Money (VfM): the VfM evaluation will be primarily conducted in late 2025 after the remainder of the evaluation has been concluded. Where possible, interim VfM findings will be prepared alongside the intervention-level case studies.

The next interim report will include early emerging findings from the 3 initial case studies, mainly focusing on the process evaluation. A second interim report is likely to report some initial findings and insights that can be shared with key stakeholders. The programme-level and VfM evaluations will be reported in this final report, alongside reporting all the activities and findings from this evaluation.

9. List of key terms and acronyms

• CLS: Community Life Survey. A household survey for measuring the performance and development of social activities within communities in England through measures such as informal volunteering and levels of loneliness.
• BSD: Business Structure Database. Provides annual snapshots covering all VAT or PAYE-registered businesses compiled from the Inter-Departmental Business Register (IDBR).
• DiD: Difference-in-difference. This is a quasi-experimental technique that compares the changes in outcomes for the treated group that received the intervention to the changes in outcomes for the control groups that did not receive the intervention – after the introduction of the intervention.
• DfT: Department for Transport.
• DLUHC: Department for Levelling Up, Housing and Communities.
• EoI: Expression of Interest.
• Experimental impact evaluation: Experimental impact evaluations compare outcomes in treated and control groups to estimate the impact of an intervention. These treated and control groups are selected through randomisation in an experimental design.
• FHSF: Future High Streets Fund.
• GVA: Gross value added.
• LA: Local authority.
• LSOA: Lower Layer Super Output Area.
• LUF: Levelling Up Fund. One of the other DLUHC local growth funds. Similarly to the Towns Fund (TF), the LUF is a major levelling up programme.
• M&E: Monitoring and Evaluation.
• Pathfinder Town: Projects in Pathfinder Towns have more flexibility in how to use different sources of DLUHC and have been excluded from the set of projects in scope for this evaluation. For more information on the Pathfinder Pilot, see the Simplification Pathfinder Pilot: technical guidance.
• PSM: Propensity Score Matching. A statistical matching technique which is used to construct an artificial control group by matching each treated unit with a non-treated unit based on similar observable (and relevant) characteristics.
• ONS: Office for National Statistics.
• QCA: Qualitative Comparative Analysis. Allows the comparison of intervention factors and contexts to determine their individual or combined contribution to the hypothesised outcomes. It uses a systematic, process-driven approach to identify the combination of factors necessary or sufficient to produce a certain result.
• Quasi-experimental impact evaluation: Quasi-experimental impact evaluations measure the impact of an intervention by estimating changes in treated and control groups, where these groups are constructed by analytically controlling for relevant characteristics across groups.
• QED: Quasi-experimental design.
• Realist Evaluation: Realist Evaluation aims to understand ‘what works, for whom, and in what circumstances’ by gathering evidence on the hypothesised causal mechanisms detailed in the individual intervention ToC.
• TD: Town Deals.
• TDB: Town Deal Board.  Groups that are set up for each Town Deal project, made up of representatives from the private sector, local government, MPs, Local Enterprise Partnerships, local businesses and investors, community and voluntary sector representatives, and other relevant local organisations. TDBs are responsible for producing Town Investment Plans, including putting forward suitable projects which align with the objectives of the Towns Fund, and for overseeing compliance with the Heads of Terms Agreement with government.
• TF: Towns Fund.
• Theory-based impact evaluation: Theory-based impact evaluations test whether the causal links expected to bring about the change are supported by strong enough evidence to rule out alternative explanations and are concerned with the extent of the change and why the changes occur.
• TIP: Town Investment Plan.
• ToC: Theory of Change. Captures how the intervention is expected to work, setting out the steps involved in achieving the desired outcomes, the assumptions made, and wider contextual factors.
• Towns Hub: The central town’s team within the Cities and Local Growth Unit (CLGU) in The Ministry of Housing, Communities & Local Government (MHCLG). The Towns Hub provides towns with support for developing investment plans and project plans to ensure towns are able to submit high quality proposals for funding.
• VfM: Value-for-money. A VfM evaluation aims to assess if the intervention was a good use of public resources. It complements the impact evaluation by helping understand if the benefits of the intervention outweigh the costs.
• UKSPF: UK Shared Prosperity Fund. One of the other DLUHC local growth funds. Similarly to the TF, the UKSPF is a major levelling up programme.

References

Brien, P., Harari, D., Keep, M., & Ward, M. (2022). Economic impact of covid-19 lockdowns. The House of Commons Library.

Cities and Local Growth Unit, DLUHC and BEIS. (2021). Towns Fund monitoring and evaluation strategy. GOV.UK.

Cities and Local Growth Unit, DLUHC and BEIS. (2021). Towns Fund monitoring and evaluation strategy: executive summary. GOV.UK.

Collins, K. M. T., & Onwuegbuzie, A. J. (2007). A typology of mixed methods sampling designs in social science research. The Qualitative Report, 12(2), 281-316.

Department for Digital, Culture, Media & Sport. (2023). DCMS sectors economic estimates 2022: business demographics - source data change summary note. GOV.UK.

Department for Levelling Up, Housing & Communities. (2021). Town Deals: full list of 101 offers. GOV.UK.

Department for Levelling Up, Housing & Communities. (2023). Towns Fund website.

Department for Levelling Up, Housing & Communities. (2023). Simplification Pathfinder Pilot: technical guidance. GOV.UK.

Duan, N., Green, C. A., Hoagwood, K., Horwitz, S. M., Palinkas, L. A., & Wisdom, J. P. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Adm Policy Ment Health, 42(5), 533–544.

Gibbons, S., Overman, H., & Sarvimaki, M. (2021). The local economic impacts of regeneration projects: Evidence from UK’s single regeneration budget. Journal of Urban Economics, 122.

High Streets Task Force. (2023). About us: High Streets Task Force.

HM Treasury. (2020). Magenta book: Central Government guidance on evaluation. HM Treasury.

HM Treasury. (2023). The green book (2022). HM Treasury.

Ministry of Housing, Communities and Local Government. (2018). Future High Streets Fund: prospectus. Ministry of Housing, Communities and Local Government.

Ministry of Housing, Communities and Local Government. (2019). Future High Streets Fund: supplementary guidance for bidding authorities. Ministry of Housing, Communities and Local Government.

Ministry of Housing, Communities and Local Government. (2019). Towns Fund: prospectus. Ministry of Housing, Communities and Local Government.

Ministry of Housing, Communities and Local Government. (2020). Towns Fund: further guidance. Ministry of Housing, Communities and Local Government.

Patton, M. Q. (2014). Qualitative research & evaluation methods. Sage Publications.

---

1. Based on performance reporting collected in July 2023. ↩

2. The contracted period will come to an end in early 2026. For a project to be part of the evaluation, it must commence a reasonable time prior to the end of the contractual period to allow for analysis and reporting of the results. ↩

3. Randomised control trials and wedge control trials were found to be unfeasible since the allocations of both funds (that is, selection of treatment units) were done before the start of the evaluation, removing the ability of randomisation of treatment assessment. Interrupted time series were found to be unfeasible as projects are spread across a wide range of time periods. Regression Discontinuity Design was unfeasible for the TD as there was no scoring creation to allow for a cut-off, and although the FHSF used a scoring system for treatment selection, there were only 29 non-chosen places, making the sample too small for this approach. The instrumental Variable approach was found to be unfeasible as finding a variable that would not correlate with the characteristics of the treatment group (selection criteria) but influence participation in the funds will be challenging. Please see section ‎5.1.2 for further details. ↩

4. The TF M&E report mentions that the full evaluation might be a year after the completion of the funded projects. That said, longer-term impacts such as employment might take longer than a year to realise and be detected in data sets. ↩

5. For the purposes of the TF evaluation, an intervention refers to similar projects within the same town that are grouped together for evaluation (Ministry of Housing, Communities and Local Government, 2018). This approach is taken to address the attribution problem within local areas. This differs from how interventions are defined in other parallel evaluations, such as the evaluation of the UK Shared Prosperity Fund. ↩

6. The list of projects can be expanded to include a set of projects with similar outcomes that are close to each other geographically. This combined set of projects is considered an intervention, for the purpose of this evaluation. ↩

7. Please see Section 4.3 for further details about the updated ToC and logic model. The original M&E report (2021). ↩

8. For more detail on these issues and the impacts they have on communities, see the TF Monitoring and Evaluation (M&E) strategy (2021). ↩

9. In some cases, changes to this initial list of approved projects have been made. These changes might have occurred for various reasons, for example, to facilitate cost changes due to inflation. ↩

10. Authorities that receive any type of funding and run projects are expected to see an improvement in their management skills. That said, since TD also provided resources, training, and support to LAs specifically to improve those skills, this has been identified as a separate theme – particular to TD. ↩

11. Places were eligible to submit EoIs if their definition of a town centre or high street aligned with the definition in the FHSF calls for proposals. ↩

12. The TFDP consortium, consisting of six private sector organisations working as one team, each fulfilling distinct strategic functions and bringing market-leading expertise. ↩

13. From the July 2023 Monitoring and Evaluation (M&E) reporting data. ↩

14. Employment data are published in the Business Structure Database (BSD), and BSD data files are published annually. Each annual file includes the average employment of a firm in the four quarters before the end of March of the same year. For example, BSD 2023 would include firms’ average employment in the four quarters before the end of March 2023. ↩

15. DLUHC TF Monitoring and Evaluation data, July 2023 ↩

16. DiD is a quasi-experimental technique that compares the changes in outcomes for the treated group that received the intervention to the changes in outcomes for the control groups that did not receive the intervention, after the introduction of the intervention. ↩

17. PSM is a statistical matching technique used to construct an artificial control group by matching each treated unit with a non-treated unit, based on these units having similar observable (and relevant) characteristics. ↩

18. While there are nearly 1,000 TF-funded projects in total, projects in Pathfinder Towns, which have more flexibility in how to use different sources of DLUHC, have been excluded from the set of projects in scope for this evaluation. More information on the Pathfinder Pilot. ↩

19. TF also includes a small proportion of revenue funding, which might have a separate outcome. However, the funding is heavily weighted towards capital funding, which is the focus of this evaluation. ↩

20. Further changes to projects and project end dates might change the number of projects in scope. ↩

21. Each selected town developed a Town Investment Plan (TIP) that outlined ‘A vision for the town, complementing agreed or emerging local economic strategies’ (Ministry of Housing, Communities and Local Government, 2019, p. 22). ↩

22. FHSF was awarded to LAs with the aim of ‘helping local high streets evolve and adapt to these changes’ (Ministry of Housing, Communities and Local Government, 2018). One LA was also able to apply and receive funding for several high streets. The bids were assessed for funding based on the ‘geographical spread of impact and wider economic considerations’ (Ministry of Housing, Communities and Local Government, 2019, p.2) among other factors. ↩

23. The initial test showed that, from the potential list of treated towns, only a few of the most deprived towns did not have a relevant control group. ↩

24. TF projects are subject to mandatory performance review, reporting to DLUHC at regular intervals. This review includes expected and actual project completion dates. The current understanding of project completion dates is based on the July 2023 reporting data. ↩

25. For some outcomes, ‘concentric circles’ might refer to other geometrical shapes. In addition, for some impacts, ‘neighbouring circles’ might relate to circles situated further away or not directly around the impact boundary. For example, displacement of footfall might occur from another high street that is not directly neighbouring the impact area. ↩

26. The TF Monitoring and Evaluation (M&E) report (2021) mentions that the full evaluation might be a year after the completion of the funded projects. That said, longer-term impacts, such as employment, might take longer than a year to realise and be detected in datasets. ↩

27. For some outcomes, in which baseline data (that is, data from before the project was completed) will need to be collected, projects that ended before the start of the evaluation will also be outside the scope of the econometric exercise. ↩

28. Kirkby and Sutton have received funding jointly. This also excludes pathfinder towns. ↩

29. Analysis based on performance reporting from July 2023. ↩

30. April 2025 is taken as a cut-off date, given an impact evaluation is expected to be delivered by the beginning of 2026. This will provide enough time for projects to start delivering outputs, for data to be generated, and for outcomes to start being reflected in the data. This might vary based on outcomes. ↩

31. Sample size guidelines for casual research design using quantitative methods to detect moderate effect sizes with 0.80 statistical power at the 5% level of significance is 51. See Collins and Onwuegbuzie (2007) for more information. ↩

32. According to UK Shared Prosperity Fund: prospectus (2022), ‘intervention’ refers to groups of similar projects across the country which are grouped together for evaluation purposes. ↩

33. Based on a review of a sample of TIPs, Monitoring and Evaluation (M&E) plans, and M&E reports from December and July 2023. ↩

34. Further details of this intervention-level design are provided in Section 6.2.3. ↩

35. The number of projects within an intervention will be assessed on a case-by-case basis ahead of developing the intervention-specific ToC. ↩

36. This assessment will be done with input from DLUHC and local area leads. ↩

37. For case studies evaluated earlier, there may be the opportunity to repeat steps 3-5 (with a lighter touch) after ~1 year in order to evaluate the persistence or improvement of outcomes. ↩

38. Where possible, a baseline and end-line survey will be run in each location to understand how factors have changed over time. However, baseline survey collection will not be possible where projects have already been completed. ↩

39. The selection criteria were based on discussions and feedback from DLUHC and available data. ↩

40. According to Patton (2014) and Duan et al. (2015), maximum variation sampling is a type of purposeful sampling technique that is commonly used when the objective is to recognise patterns common to a group of diverse case studies. More generally, purposeful sampling methods are suitable when random sampling is not appropriate, such as when the goal of the evaluation is to gain a more comprehensive understanding of the intervention and only a few case studies are being selected. ↩

41. Projects which do not have an estimated completion date and those projected to be completed after the end of 2024 have been excluded to ensure that at least some project outcomes are observable within our evaluation timeline. Also, project completion dates will be staggered such that the evaluation of projects takes place throughout the evaluation period (rather than all right at the end). ↩

42. A lower bound of £1m of funding per project has been applied, as it may be challenging to detect an effect for very small projects below this threshold. Also, projects in which TF funding represents a very small proportion of overall funding have been excluded from the selection. ↩

43. Varying across town size (and therefore LA size) allows for some variation in LA’s capacity to deliver interventions and types of LA to be captured. ↩

44. Projects can become unsuitable due to project delays, overlaps with other evaluations, or issues with data or stakeholder evaluation. ↩

45. Please see Section ‎5 for further details about challenges in this evaluation. ↩

46. This will be addressed through analysis within the programme-level evaluation that looks at different types of places, as well as the QCA proposed as part of the intervention-level evaluation. ↩

47. The design of the survey does not allow for a statistically robust inference from the collected sample to the population level when analysing data at a sub-regional level. ↩

48. In order to use the LA-level survey data for the baseline, the extent to which the treated towns/places are similar to the local LA will need to be assessed for those outcomes. ↩

49. In some cases, web-reading data on these outcomes may already have been collected by DLUHC or other commercial companies. This will be explored further during the early stages of the evaluation to avoid duplication of data gathering, where possible. ↩

50. Given the small sample sizes that had been identified for the impact evaluation, it would not be possible to further stratify the programme-level evaluation of each outcome into sub-project classifications. ↩

51. As part of the feasibility stage for the process evaluation, six interviews were completed with DLUHC delivery leads and 11 interviews with LAs between June and August 2023. The feasibility interviews were conducted to provide evaluators with an in-depth understanding of the processes of TD and FHSF as well as insights into the most appropriate data collection methods for the process evaluation. ↩

52. Interviews will be conducted via video conferencing or in-person, depending on the participant’s preference. Interviews will be conducted by experienced researchers and are expected to be one-to-one or paired. With participants’ permission, interviews will be audio recorded to support in-depth analysis ↩

53. Town Deal Boards (TDBs) are made up of representatives from the private sector, local government, MPs, Local Enterprise Partnerships, local businesses and investors, community and voluntary sector representatives, and other relevant local organisations (for example, FE colleges or Clinical Commissioning Groups). TDBs are responsible for producing Town Investment Plans, including putting forward suitable projects which align with the objectives of the TF, and for overseeing compliance with the Heads of Terms Agreement with the government. More information is available in the Towns Fund guidance (2020). ↩

54. The High Streets Task Force is a group of expert organisations, led by the Institute for Place Management, that offers information, advice, training and knowledge to towns on how to improve their high streets. The High Streets Task Force was tasked with helping towns in developing business cases for FHSF. More information can be found on the High Streets Task Force website. ↩

55. The TFDP consortium is a group of 6 private sector organisations led by Arup. This consortium helped towns to produce Town Investment Plans and develop business cases for projects. More information can be found on the Towns Fund website. ↩