Corporate report

DfT evaluation strategy and programme 2022

Updated 30 June 2022

Foreword

The Department for Transport (DfT) has been building a strong programme of evaluation results since we published our first evaluation strategy in 2013. This update to our evaluation plans describes how we are using evaluation on 29 of our most significant current transport projects. We also share some recent evaluation findings and provide links to our growing library of evaluation reports.

We have timed this update to also meet the requirement of the Cabinet Office and HM Treasury’s Evaluation Task Force to publish an evaluation strategy outlining our department-wide approach to identifying and prioritising areas for evaluation activity.

We have taken this opportunity to refresh our 2013 monitoring and evaluation strategy to reflect our experience of delivering transport evaluations in the subsequent 9 years. This strategy forms the second section of this report.

We have welcomed the increased attention on evaluation within government since the formation of the Evaluation Task Force last year. We are committed to embedding evaluation in our programmes to derive proportionate learning about their effectiveness.

As our strategy explains, we are seeking to prioritise this so we can provide the optimal feedback to improve delivery, provide better appraisals and inform decisions about future policies.

Amanda Rowlatt, Chief Analyst

Introduction

Background

DfT uses monitoring^{[footnote 1]} and evaluation^{[footnote 2]} to collect evidence about the effectiveness of our interventions in achieving their intended outcomes.

This activity informs the development of ongoing interventions and the options appraisal and planning for new initiatives. For the remainder of this document, ‘evaluation’ is used to refer to both monitoring and evaluation.

Our initial monitoring and evaluation strategy was published in 2013. This guided the development of our evaluation programme in subsequent years.

An initial monitoring and evaluation programme was also published in 2013. Since then, further updates on the programme have been published in 2014, 2015, 2016, 2018 and 2021.

These documents have detailed progress of the principal evaluation projects of the central department (rather than the whole DfT family which includes its agencies and arm’s length bodies).

The Evaluation Task Force (ETF) is a joint Cabinet Office – HM Treasury unit providing specialist support to ensure evidence and evaluation sit at the heart of spending decisions. It was set up following the 2020 Spending Review.

In 2022, as part of settlements to the 2021 Spending Review, the ETF required departments to publish an evaluation strategy which outlines their approach to identifying and prioritising areas for evaluation activity.

The purpose of this document

This document provides a comprehensive update on evaluation activity at DfT.

It serves 3 main purposes:

• to refresh DfT’s evaluation strategy to reflect our experience in the 9 years since it was introduced, in compliance with the requirement of the ETF
• to provide a comprehensive update on progress with the DfT’s principal current evaluation projects
• to provide information about the findings of past evaluation projects and links to published reports

This information is provided for transparency, to share our progress with evaluation with our stakeholders in the transport sector and the public.

Evaluation strategy

Why evaluation matters for DfT

Evaluation is an important activity for any learning organisation which aims progressively to improve its performance. It allows for systematic learning from past and current activities – ‘what works, what doesn’t work’ and ‘why’. This provides greater accountability and a strong evidence base to guide future operational delivery, policy development and investment decisions.

Our evaluation strategy is intended to ensure that the DfT can achieve these benefits from evaluation of our activities.

In the 9 years since our first Evaluation Strategy was published in 2013, DfT has built its capability in evaluation and developed monitoring and evaluation workstreams for most of its principal activities. This has produced an expanded evidence base from past projects and a pipeline of evidence for current activities.

Another outcome of this work has been a greater understanding of how to plan and deliver an effective evaluation of transport projects. This understanding has been built into this updated evaluation strategy.

Some challenges of evaluating transport projects

There are some distinct challenges for transport projects which shape our evaluation strategy.

The principal challenge is long timeframes for the planning and implementation of transport infrastructure, which can be more than a decade for major projects. Over long timeframes, there is a risk that the analytical links between the appraisal, monitoring and evaluation stages of the ROAMEF Policy Development Cycle (Rationale, Objectives, Appraisal, Monitoring, Evaluation and Feedback), which is described in the HM Treasury Green Book, can be broken.

We need strong procedures to counter this risk and connect ex-ante with ex-post analysis. Our report on Strengthening the links between appraisal and evaluation identifies approaches which can help to achieve this.

A second challenge is that experimental evaluation approaches, such as randomised controlled trials, which are commonly used in policy evaluation, are not available for transport infrastructure investments because it is not feasible or desirable to allocate them in an experimental way.

We use counterfactual-based designs where these are feasible, for example where locations receiving transport investment are compared with matched locations in a quasi-experimental impact evaluation design. Where such approaches are not feasible, we may use theory-based impact evaluation approaches which examine the mechanisms by which programmes are expected to deliver their benefits^{[footnote 3]}.

A challenge for evaluations of local transport investments is that management of evaluation by funded local authorities can make it difficult to obtain evidence in a consistent way across schemes. We tackle this challenge by working with our local partners to establish common expectations for evaluation, for example through setting evaluation frameworks.

For several recent programmes of local investment, we have appointed a national contractor to support local authorities with their evaluation activities, driving for greater consistency in methods and synthesis of findings.

Research and development (R&D) programmes, which have become a growing priority of DfT in recent years, are often difficult to evaluate effectively as they comprise a large number of funded projects with diverse aims and approaches. We are working with our delivery partners for R&D programmes to address this challenge.

Given the diversity of transport projects, we have not attempted to institute a common approach to evaluation except within portfolios of projects which have common characteristics.

Our aim is to develop whichever approach will provide the best evidence for a project. This necessitates using a wide range of approaches and it is often not feasible to reach the higher evidence levels in quality classification for evaluation such as the Nesta Standards of Evidence and the Maryland Scientific Methods Scale.

However, we recognise that these classifications provide a helpful point of reference and will use the Nesta scale to classify our principal evaluations, reflecting the steer of the ETF.

Achievements of evaluation strategy and programme to date

Since the start of our evaluation programme in 2013, we have steadily increased our commitment to the evaluation of our activities. Over this period we have published more than 50 evaluation reports.

Achievements include:

• a series of evaluations of active and sustainable travel programmes that have produced evidence that has shaped the design of future programmes, for example, findings from a substantial evaluation of the Local Sustainable Transport Fund fed into the design of current local transport programmes such as the Transforming Cities Fund and City Region Sustainable Transport Settlements

• establishing a programme of rail evaluation for the first time, this initially covered large investments such as High Speed 1 (HS1), Thameslink and Crossrail and has since been extended to cover the wider portfolio and Rail Transformation

• developing some consistent evaluation approaches at a portfolio level, for example for several local transport programmes and the Rail Enhancements Portfolio

• addressing the long wait for evaluation evidence from current infrastructure investment by conducting retrospective evaluations of legacy projects, as with our current studies of the transformational impacts of transport investments and the economic performance impacts of road enhancements (both described in this document)

• delivering a multi-year evaluation of Roads Reform, the changes that have been made to the management and operation of the Strategic Road Network since 2014, which will be completed in 2022. This evaluation provides a reference point for the development and evaluation of new change programmes such as Rail Transformation and the creation of Active Travel England

• embedding evaluation in the delivery of new forms of programmes that will shape the future of transport from their inception, for example, the Future Transport Zones and the e-scooters pilots

• increasing the level of evaluation for decarbonisation programmes, for example developing both retrospective evaluations and a new evaluation framework for schemes managed by the Office for Zero Emission Vehicles

Objectives for the new strategy

Despite these achievements, we have further to go to ensure that proportionate evidence is collected for all important programmes. In reviewing our progress across our activities, we have identified some areas that merit additional focus in our refreshed evaluation strategy:

Inform our strategic priorities

DfT’s strategic priorities, growing and levelling up the economy, improving transport for the user and reducing environmental impacts, set a direction for our work and provide a lens for considering how individual projects contribute to overarching goals.

We have established analytical groups to support each of these priorities. These groups will consider evidence pipelines from our evaluations and identify new work to address evidence gaps.

Underpin local evaluation from the centre

For local transport investment programmes, we will introduce more national, theme-led, evaluation planning to encourage the use of more consistent methods, enabling better synthesis of evidence and more generalisable learnings about the priority questions for future investment decisions.

Collect better evidence about innovation

For innovation and R&D programmes, which have lacked evaluation hitherto, we will work with policy teams and delivery partners, such as Innovate UK, to introduce evaluation frameworks which can improve the collection of evidence about programmes’ outputs, outcomes and impacts.

Standardise methods within portfolios

In general, across our evaluations, we will try to develop standard evaluation approaches at the portfolio level. We will aim to consolidate this standardisation with written guidance, starting with the publication of an initial Transport Appraisal Guidance (TAG) module on evaluation in autumn 2022.

Adjust for the effects of coronavirus (COVID-19)

Noting the significant effects of the COVID-19 pandemic in changing transport behaviours, we will endeavour to understand these changes so that we can distinguish how they have altered the benefits of schemes compared to what was forecast at the appraisal stage.

Strengthen dissemination

We will increase our attention to dissemination of evaluation findings. Having prioritised the development of many new evaluations for programmes in the last few years, it is a natural next step to focus more on refining approaches for feeding back evaluation findings into policy and appraisal as more of these evaluations reach the findings stage.

Prioritisation

DfT has a substantial number of projects and we need to prioritise our evaluation activity carefully to achieve best value. Our ambition is not to conduct evaluation of all projects but to conduct proportionate evaluation, which matches evaluation activity to the need and opportunity for good quality evidence, so maximising its value for money.

In our initial evaluation strategy in 2013, we established 5 proportionality criteria to guide evaluation planning.

Reflecting on the application of these since then, we can simplify our focus to 3 broad criteria for prioritisation:

1. Scale of investment

We would seek to evaluate all our highest-value projects on the grounds of accountability for the use of public funds, unless the nature of their impacts is so well understood as to render it unnecessary to conduct evaluation to understand their benefits.

2. Strategic importance

We will give particular attention to the evaluation of interventions that are central to the direction of government policy, including interventions that are novel and innovative. Reflecting this concern, our evaluation programme is structured according to DfT’s strategic priorities.

3. Contribution to our evidence base

We will seek to prioritise evaluations which provide an opportunity to collect robust, policy-relevant evidence that can inform current project implementation and future investment decisions.

Effective prioritisation depends on close relationships between our evaluation specialists and the policy teams they support so that decisions about prioritisation can be taken in a timely way as interventions are planned.

By embedding most of our evaluation analysts within policy directorates, we facilitate these sorts of judgements.

Equally, maintaining a small central evaluation team facilitates effective prioritisation of evaluation planning across DfT’s activities.

Guiding principles

Our approach to evaluation follows HM Treasury’s Green Book: appraisal and evaluation in central government and its Magenta Book.

In accordance with these guidelines, we look to embed evaluation planning in new interventions from their inception. This can enable both formative learning, to improve design and implementation and summative learning, to understand outcomes. By starting early, we can establish the baseline data, performance metrics and points of comparison that will enable us to understand a project’s impact.

We have developed 10 guiding principles for the implementation of our evaluation projects.

These reflect both the aforementioned challenges of evaluating transport projects and the new objectives we have established in this strategy. They further reflect practical learnings from the delivery of evaluations of transport projects over the past 9 years.

1. Be evidence-driven and methods-neutral

We will develop whichever evaluation approaches will deliver the best-value evidence for specific projects, which is likely to mean varying the evaluation approach between different policy areas. While we will note where our evaluation methodologies sit on the Nesta Evidence Standards, this will be a subsidiary consideration to addressing the right evidence needs.

2. Use programme theory

We will encourage the use of theories of change and logic mapping in the early stages of intervention design and evaluation planning. In our experience, articulation of the programme theory is advantageous both for policy development and evidence planning.

3. Be relevant for policy

We will work closely with policy officials to ensure we understand key policy questions, shape our evaluation designs according to evidence needs and feed evaluation findings back into policy development.

4. Link with appraisal

We will continue to work with appraisal and modelling specialists to strengthen links with their work. This means designing evaluation approaches that will provide feedback on appraisal assumptions, including the provision of ex-post benefit-cost ratios wherever practicable. We will follow the recommendations of our 2016 Strengthening the links between appraisal and evaluation report^{[footnote 4]}.

5. Address strategic objectives

At the same time, the focus of evaluation analysis will not be constrained by appraisal analysis and the economic case^{[footnote 5]} for investment, but will also pay attention to the strategic case^{[footnote 6]}. By addressing strategic objectives, we can ensure evaluations collect evidence about all outcomes of importance to transport users and the UK economy, not only those that are monetised in appraisal. In having this broader focus, evaluations should also enable the identification of unforeseen benefits and disbenefits.

6. Be embedded in intervention delivery

We will embed evaluation in intervention delivery wherever possible. This means having early discussions about the evaluability of an intervention so that it is possible to consider adjusting its method of delivery to improve the quality of evaluation evidence and the learning, that can be collected. Embedding should also mean that budgets and resources for evaluation are built into the management case^{[footnote 7]} for delivery of an intervention.

7. Complement benefits management

In the context of major projects^{[footnote 8]}, our evaluation analysts will collaborate with benefits management^{[footnote 9]} specialists to complement and avoid duplicating their work. Benefits management can capture the more easily measured, or direct, benefits of projects, using monitoring methods. This can leave evaluation to focus on harder-to-measure and indirect, benefits, such as the impacts of projects on the wider transport system and the local economy.

8. Match evidence to decision points

In planning evaluations we will seek to schedule evidence outputs so that they can inform reviews at key delivery stages of projects, including decisions about continuation or roll-out, so far as this is feasible. We will also look to inform decisions about future projects which have similar characteristics. One aspect of this will be timing interim reports for our longer evaluations so that they can feed into spending reviews.

9. Seek formative evidence

Noting that collective summative evidence about the impact of transport projects will often entail a long wait, we will seek to exploit opportunities for evaluation to provide formative evidence about the implementation and early impacts of projects.

Process evaluation will be prioritised where projects are innovative in their objectives or modes of delivery and where the mechanisms for capturing lessons learned in project delivery are less well developed (to avoid duplication of effort).

10. Increase consistency of approach

While having regard for the particularities of individual projects, we will work towards greater consistency of measurement approaches across our evaluation programme. This will include increasing the consistency of approach between different modes and types of project. This will support the joining up of evidence across DfT’s areas of operation.

Evaluation delivery

Evaluation staffing

DfT’s analysts who deliver evaluations are organised in a ‘hub and spoke model’. There is a small central team, the Evaluation Centre of Excellence (CoE), linked to multiple ‘embedded’ analytical teams within policy directorates.

The Evaluation CoE is located in Analysis Directorate which is managed by the Chief Analyst. It implements the evaluation strategy and co-ordinates the evaluation programme. The CoE is responsible for promoting, developing and quality assuring evaluations, as well as providing training, guidance and other support to build evaluation capabilities among our analysts and policy officials. It also leads our liaison with the ETF.

Embedded evaluation analysts are located in multi-disciplinary analytical teams that are overseen by a network of senior analysts who manage portfolios of analytical projects. Embedded analysts work closely with policy officials on the development and delivery of evaluations, ensuring they are tailored to policy need and feed into the decision-making process at the right time.

Funding is typically drawn from policy budgets, which ensures the policy team have ownership of their evaluations, while project management is typically discharged by analysts due to their expertise.

Spending Review processes

During the last 2 Spending Reviews, the Cabinet Office, HM Treasury and the ETF have increased the focus on evaluation planning. Bids for funding have been required to state what evidence supported the expected impacts of projects and what evaluation would be conducted to show what impacts actually occur. These returns have been scrutinised by the ETF and requirements for evaluation of funded projects have been included in settlement conditions.

Evaluation plans for priority projects have been set out in published outcome delivery plans.

DfT has embraced these prompts from the centre of government, using them to help strengthen early evaluation planning. Evaluation analysts have helped develop and quality assure evaluation plans for programmes which are summarised in DfT’s outcome delivery plan.

Governance and assurance for evaluation

There is a system of governance arrangements for evaluations in DfT.

Evaluation projects in policy directorates are managed by embedded analytical teams that are overseen by a senior analyst. Oversight for the whole department is provided by the Evaluation CoE which provides periodic quality assurance checks and reports for management. The Evaluation CoE also compiles periodic published updates of this evaluation programme to share our progress with our stakeholders and the public. This work is overseen by the Chief Analyst.

As part of DfT’s research programme, evaluations are included in policy directorates’ annual evidence and research plans which are reviewed by ministers. Approved plans are summarised in the published Areas of Research Interest documents which, like this document, are used to communicate our priority evidence and research needs to external researchers, as well as funders, investors, industry and small and medium-sized enterprises.

An R&D Board, chaired by our Chief Scientific Adviser, provides oversight of DfT’s research programme, including evaluations. The highest-value evaluations provide periodic progress reports to this Board.

For major projects, evaluation planning is built into business case development, alongside benefits management planning, as part of the management case. Evaluation plans are reviewed by internal specialists as part of the business case assurance process.

During project delivery, assurance reports on progress with benefits management and evaluation are provided to our Investment Committee by Evaluation CoE and project delivery specialists.

Regulations

DfT is responsible for a wide range of statutory instruments for which ministers have an obligation to undertake post-legislative scrutiny^{[footnote 10]} and post-implementation reviews (PIRs)^{[footnote 11]}.

Post-implementation evaluation is critical in ensuring that our regulatory interventions are delivering their intended outcomes, identifying any unintended consequences and checking that our estimates of the costs or benefits are accurate.

Reviewing regulations also enables us to learn lessons from previous interventions and apply these to future regulations. DfT publicly commits to review many other regulatory policies, for example, in the notes published alongside legislation or in impact assessments.

We will continue to monitor review dates, undertake proportionate PIRs and consider evaluation when designing new regulatory policies, drawing on guidance from the Better Regulation Executive, the Magenta Book and Transport Appraisal Guidance.

Contracting

Most of our evaluation studies are delivered by external evaluation contractors and academic researchers who are commissioned through competitively-tendered frameworks or individual contracts.

Our default framework for commissioning large evaluation studies is the Crown Commercial Services Research & Insights Market Place (RM6126). We also use sector-specific frameworks for rail and roads projects where the need for sector knowledge may outweigh that for general evaluation expertise.

In recent years, we have developed framework ‘call-off’ contracts to enable us to quickly commission research studies, including some evaluations.

Our Evaluation CoE holds a contract which has been used to commission rapid scoping studies, small evaluations and evaluation training. Other call-off contracts are used by our local transport and rail research and evaluation teams.

To date we have worked with a moderately-sized supply chain of contractors who deliver our evaluation studies. To maintain and extend this supply chain, we are increasing our engagement with contractors and considering how to provide notification of future tendering opportunities.

We endeavour to provide adequate time for tender processes and good-quality feedback for contractors who bid for them, to help explain our needs and the basis for tender assessments.

Partner organisations

DfT works with a range of partners in developing this evaluation programme.

We maintain a close relationship with the ETF at the centre of government and regularly share evaluation plans for their comments.

Some evaluations are developed with other departments, such as those for the Levelling Up Fund (led by the Department for Levelling Up, Housing and Communities) and local authority plans to tackle NO2 concentrations (through the Joint Air Quality Unit with the Department for Environment, Food and Rural Affairs).

Several evaluations of local transport projects, such as for the local authority major schemes, the Transforming Cities Fund and the Future Transport Zones, have been developed with local authority partners.

A common arrangement is for DfT to establish a framework to guide local authorities’ evaluation activities, backed up by the use of a national evaluation contractor to coordinate, support and synthesise evidence from local evaluation activity.

Some evaluations have been developed with or through our agencies and public bodies including those for the UK Search and Rescue (Maritime and Coastguard Agency), High Speed 2 (HS2) and East West Rail.

Another external partner is the What Works Centre for Local Economic Growth, which DfT funds and which has advised us on some studies and provided helpful evaluation resources, notably its evidence review for transport.

Outside the programme, National Highways, the arm’s length body charged with operating, maintaining and improving England’s motorways and major A roads, maintains its series of Post Opening Project Evaluations for major schemes.

DfT maintains a strong interest in the evidence from these schemes which informs its road investment strategies.

In the rail sector, DfT works with a range of industry partners including Rail Delivery Group, Network Rail, Office for Road and Rail, Transport Focus and train operating companies in the development and delivery of evaluation projects.

Following the publication of the Williams-Shapps Plan for Rail, Great British Railways will be established as a new public body to own the rail infrastructure, receive the fare revenue, run and plan the network and set most fares and timetables.

DfT will be working closely with the Great British Railways Transition Team (GBRTT) and later Great British Railways (GBR) in developing the rail evaluation programme.

Publication

We will publish evaluation reports in accordance with the Government Social Research: publication protocol. We will continue to use updates like this to our evaluation programme to highlight recent evaluation findings and protocols.

For our largest evaluation projects, we will aim to publish evaluation frameworks and plans in advance of the evaluation being conducted, so these can be shared with stakeholders. This will typically occur when the evaluation design has been finalised, which may be after the evaluation has been commissioned and initial scoping has been completed.

Where it is feasible and proportionate to do, we will consider publishing datasets from evaluations so facilitate further independent research to inform policy.

Measuring and reporting on progress

We will publish a further update of our evaluation programme in 2024 to describe our further progress with the evaluations listed in the programme and highlight further evaluation findings and publication. This will also enable us to report on the delivery of our revised Evaluation Strategy, identifying how the guiding principles and plans set out in this document have been realised and setting out what further development is required to achieve proportionate evaluation of DfT’s activities.

The current programme

An overview of the programme

The evaluation programme contains 29 current projects for which evaluation is planned or underway.

The projects are grouped under DfT’s Strategic Priorities which are growing and levelling up the economy, improving transport for the user and reducing environmental impacts.

Growing and levelling up the economy

Improve connectivity across the UK and grow and level up the economy by enhancing the transport network, on time and on budget.

The projects are:

• Local Authority Major Schemes
• Levelling Up Fund
• City Region Sustainable Transport Settlements (CRSTS)
• Transforming Cities Fund
• Future Transport Zones
• Roads Reform
• Roads Investment Strategy 2 scoping study
• Economic performance impacts of road enhancements
• HS2
• HS1
• Crossrail/Elizabeth line
• Thameslink
• East West Rail
• Rail Enhancements Portfolio
• Transformational impacts of transport investments

Improving transport for the user

Ensuring that the transport system is safe, reliable, joined-up and inclusive, building passenger and supply chain confidence following COVID-19 and reflects evolving travel patterns and demand.

The projects are:

• National Bus Strategy
• Inclusive Transport Strategy
• local highways maintenance and renewals
• Safer Roads Fund
• Young and Novice Driver Interventions (Driver 2020)
• Rail Transformation
• Rail fares and ticketing
• UK Second Generation Search and Rescue Aviation
• Tackling loneliness with transport

Reducing environmental impacts

Tackle climate change and improve air quality, including by decarbonising transport.

The projects are:

• Office for Zero Emission Vehicles
• Zero Emission Road Freight (ZERF) Demonstrator programme
• Local authority plans to tackle NO2 concentrations
• Active travel
• E-scooters trial evaluation

A fourth strategic priority is increasing our global impact, which entails boosting our influence as a leading trading nation and seizing the opportunities presented by Brexit to strengthen ties with countries around the world. We are starting to develop monitoring and evaluation plans for this area and will report on these in our next programme update.

Growing and levelling up the economy

This first group of projects supports the strategic priority of improving connectivity across the UK and growing and levelling up the economy by enhancing the transport network, on time and on budget.

Local authority major schemes

Local authority major scheme promoters are required to undertake proportionate monitoring and evaluation to assess whether the scheme has delivered value for money and to ensure that lessons are learned for future investment decisions.

DfT reviews and signs off scheme promoters’ evaluation plans, reviews evaluation reports to ensure that they are fit for purpose and commissions meta-evaluations of completed scheme evaluations.

We have previously published meta-evaluations of schemes funded between 2006 and 2010 and of schemes funded between 2007 and 2012. The most recent meta-evaluation, which will include around 43 projects approved for construction between 2011 and 2016, will be published in 2022.

Based on the findings of this meta-evaluation and wider stakeholder engagement, we plan to update the 2012 Local Authority Major Scheme Monitoring and Evaluation Framework. 

Levelling Up Fund

The £4.8 billion Levelling Up Fund (LUF) invests in infrastructure that improves everyday life across the UK. It supports town centre and high street regeneration, local transport projects and cultural and heritage assets.

The fund is jointly managed by HM Treasury, the Department for Levelling Up, Housing and Communities (DLUHC) and DfT. In October 2021, 105 successful bids were awarded a total of £1.7 billion during the first round of LUF.

DfT is supporting DLUHC to develop the monitoring and evaluation programme for LUF, which also receives input from DCMS. In March 2022, DLUHC published the LUF monitoring and evaluation strategy. This includes a theory of change (which sets out how the programme is expected to achieve the anticipated outcomes and impacts) and monitoring and evaluation aims, objectives and plans.

Local areas are required to carry out project-level monitoring and evaluation as a condition of funding and DLUHC is collecting monitoring data from the fund’s recipients to inform the programme-level evaluation.

DLUHC and DfT are currently scoping a feasibility study to assess the options for carrying out robust impact and value-for-money evaluation of LUF. This is expected to report in 2022, after which more detailed evaluation plans and timescales for reporting will be drawn up.

City Region Sustainable Transport Settlements (CRSTS)

The CRSTS is a £5.7 billion 5-year funding settlement which was allocated to 7 Mayoral Combined Authorities (MCAs) in England in April 2022.

MCAs’ CRSTS proposals aim to improve the quality of local transport networks, with a particular focus on public transport.

As a condition of funding, MCAs are required to undertake their own evaluations of their CRSTS schemes. DfT has also commissioned programme-level process, impact and value-for-money evaluations. The process evaluation will involve case studies to explore different aspects of how the CRSTS has been designed and delivered, with the aim of learning lessons to inform the delivery of future multi-year integrated funding settlements.

The impact and value-for-money evaluation will entail the development of a national evaluation framework to enable the evaluator to collate and synthesise evidence on the overall impact of CRSTS. This will include identification of comparison groups to measure the programme’s impact and an assessment of its cost-effectiveness. Interim outputs from the evaluation will be published by autumn 2024, with final reporting anticipated to take place in 2027.

Transforming Cities Fund

This £2.45 billion fund aims to drive up productivity by improving connectivity within city regions, achieved by investment in infrastructure to improve public and sustainable transport links.

Encouraging an increase in journeys made by low-carbon, sustainable modes is a central objective of the programme, which funds a variety of interventions including new and refurbished railway stations, enhanced bus services and better provision for walking and cycling.

Local and regional authorities in receipt of funding are expected to carry out evaluations of TCF-funded schemes. A national evaluation contractor appointed by DfT is conducting an evaluation of the TCF programme as a whole, which will draw on local and national data.

Learning outputs will include a baseline report from 2022 to 2023, a final report in 2025 and in-depth case studies across the lifetime of the fund. The first case study evaluated the novel ‘co-development’ process by which DfT and funded areas developed business cases collaboratively and was published in 2021.

Future Transport Zones

Future Transport Zones is a £92 million programme of funding for local areas to trial new transport models that incorporate pre-existing modes with innovations.

Schemes to be trialled feature:

• mobility as a service
• mobility credits
• transport data projects
• demand responsive public transport
• e-cargo bikes
• drones

Funding began in 2019 with a pathfinder area, Transport for West Midlands and in 2020 funds were awarded for 3 further areas following a competitive bidding process:

• West of England Combined Authority
• Derby and Nottingham
• Portsmouth and Southampton (Solent)

The programme runs to 2024.

Each funded authority will conduct its own monitoring and evaluation of its schemes. In addition, DfT has commissioned a national evaluation to support local areas, aiming to build consistency across local evaluations, conduct supplemental evaluation case studies to generate cross-cutting learnings, and synthesise evaluation learnings across the programme.

The scope of this evaluation is currently being reviewed by DfT in order to establish how learnings can most optimally be obtained.

Roads Reform

Roads Reform refers to a series of changes that have been made to the management and operation of the strategic road network (SRN) since 2014. The reforms aim to create an SRN that gives road users the best quality of service and supports economic growth.

Alongside the reforms, the first Road Investment Strategy set out a long-term programme for the SRN, involving over £15 billion of investment during the first road period of 2015 to 2020.

DfT has established Highways Monitor and Watchdog functions at the Office of Rail and Road and Transport Focus respectively. These bodies are a continual source of monitoring and evaluation evidence on the reforms.

DfT has commissioned a theory-based evaluation of Roads Reform that draws on a range of data sources including qualitative interviews with internal and external stakeholders for the principal organisations, telephone surveys of businesses in the Highways England supply chain and a review of documentation.

Findings from the first 2 waves of the evaluation were published in 2019. A final wave of the evaluation is underway and will be published in 2022 or 2023.

Roads Investment Strategy 2 scoping study

The Road Investment Strategy for 2020 to 2025 (RIS2) commits the government to spend £24 billion on operating, maintaining and upgrading the Strategic Road Network (SRN). Some of this will be used to build new road capacity but more will be used to improve the quality and reduce the negative impacts of the existing SRN, so that every part of the country will benefit.

Components of RIS2 are evaluated through National Highways’ Post Opening Project Evaluation (POPE) series, whilst key performance indicators of the SRN are measured annually by the Office of Rail and Road.

National Highways is also scoping a POPE meta-analysis and once this is complete (which is expected to be in 2022) we will conduct a scoping exercise to assess what additional monitoring and evaluation work will be appropriate for RIS2.

Economic performance impacts of road enhancements

This is an analysis project designed to improve DfT’s evidence about the relationship between investment in the road network and local economic performance. It will supplement existing evaluation programmes which focus on the transport outcomes of schemes, such as Highways England’s Post Opening Project Evaluation series and local scheme promoters’ Local Major Scheme evaluations.

A scoping study, which was conducted and published in 2019, identified schemes that could be covered, assessed data requirements and explored impact evaluation methods.

A full study was commissioned in 2020 and is expected to report in 2022. This will utilise portfolio-level econometric analysis to examine potential firm-level, labour-market and development impacts and scheme-level case study research to explore the evidence for impacts in local area contexts.

HS2

Phase One of High Speed 2 (HS2), between London and Birmingham, has begun construction and is due to open between 2029 and 2033.

Phase Two has been split into 2 stages:

• Phase 2a linking the West Midlands and the north via Crewe
• Phase 2b completing the railway to Manchester (the Western Leg) and the East Midlands (the Eastern Leg)

Phase 2a achieved Royal Assent in February 2021. Phase 2b Western Leg deposited its Bill in January 2022.

HS2 Limited and DfT have developed a comprehensive benefits management programme to measure, monitor, assess and report on the extent to which HS2 is being delivered in line with government commitments and that the investment is achieving its intended objectives.

DfT is writing an evaluation framework that will set out how we will evaluate HS2 Phase One, which will be published in 2023. A similar evaluation plan and benefits management framework is being finalised for Phase 2a and Phase 2b Western Leg.

HS1

High Speed 1 (HS1) opened in 2007 and the first interim evaluation was conducted from 2013 to 2014.

DfT has commissioned a second evaluation to examine the longer-term impacts of HS1 and so strengthen our overall evidence base for major projects and support planning for other schemes.

The study is using timetable modelling and published statistics to evaluate transport impacts, econometric analysis to understand social and economic impacts and interviews with local government, businesses and passenger groups. The report will be published in 2022 or 2023.

Crossrail/Elizabeth line

The Crossrail programme is the project to deliver the Elizabeth line, a new railway linking Reading and Heathrow to Shenfield and Abbey Wood, running through tunnels under central London. The Elizabeth line opened in May 2022.

TfL and DfT jointly commissioned a baseline evaluation study for Crossrail, which is due to be published in the summer of 2022. The study collected comprehensive baseline data for future post-opening evaluation, as well as evaluating some pre-opening impacts of Crossrail associated with construction and property values.

It considered a wide range of areas that will be impacted by the new railway, from transport and construction to land, property, the local economy and regeneration.

A subsequent study, which will evaluate the post-opening impacts of the programme against the baseline, is now in development and is expected to report in 2024 to 2025.

Thameslink

The Thameslink programme was a £7 billion, 10-year programme of extensive enhancements that has transformed north-south travel across London.

It consisted of major infrastructure work including the rebuilding of London Bridge station, procurement of a fleet of new trains, signalling upgrades and reconfiguration of train franchise arrangements to increase capacity on the Thameslink network.

A baseline study collected comprehensive data for measuring the benefits of the programme and was published in 2018. Further work is underway internally to monitor interim outcomes of the programme. A 5 year post-opening evaluation to assess the longer-term outcomes and socio-economic impacts is planned to be commissioned and delivered in 2025.

East West Rail

East West Rail aims to put in place transport connections between Oxford and Cambridge in 3 connection stages. So far, funding has been committed for construction of Connection Stage One which will deliver services between Oxford and Bletchley and Milton Keynes by 2025.

East West Rail Company has been created to oversee the project.

East West Rail Company and DfT have developed a benefits management programme for Connection Stage One, Learning lessons from HS2. East West Rail Company are also developing an evaluation framework which will be published in 2023.

Rail Enhancements portfolio

Throughout Control Period 6 (2019 to 2024), DfT has committed substantial investments in new or improved infrastructure that enables service changes and other benefits to improve journeys for passengers and freight customers across the rail network.

These investments will enhance the capability of the railway, typically adding increased or new capacity, creating new connectivity or providing technical improvements to the way the railway runs.

DfT has a benefits management and evaluation framework which is used to measure, monitor and report on the benefits and outcomes realised for individual rail schemes within the enhancements portfolio.

This is supplemented by additional evaluation activity for major programmes and/or cross-cutting priority outcomes, to understand the outcomes and wider economic impacts of individual schemes in comparison with appropriate control routes or areas.

The first post-opening study of the Great Western route modernisation and linked outputs from the Intercity Express Programme is expected to be completed in 2022.

Further baseline and post-opening evaluation studies for major rail programmes are currently being developed, with priority being given to those schemes with the highest levels of investment, as well as those with cross-cutting outcomes or with the greatest potential for learning.

Transformational impacts of transport investments

DfT commissioned an analytical project to review how rail and road transport projects have achieved transformational change in the past. The study included a literature review, case studies of 15 schemes, and qualitative comparative analysis.

The research found that coordination is critical for transport to play a meaningful role in transforming a local area.

There are various aspects to that coordination:

• the DFT and local authority
• public and private sector
• transport providers and developers

There were also some interesting findings on the importance of leisure travel and the role it can play in levelling up. The findings will be published in 2022.

Improving transport for the user

This second group of projects supports the strategic priority of ensuring that the transport system is safe, reliable, joined-up and inclusive, building passenger and supply chain confidence following COVID-19, and reflecting evolving travel patterns and demand.

National Bus Strategy

The National Bus Strategy, published 2021, sets out the government’s vision to deliver better bus services for passengers across England and increase bus patronage.

Bus service improvement plans (BSIPs), which set out the local vision for bus service improvements across local transport authorities in England outside London, are a major part of the strategy. Indicative funding to support the implementation of initiatives set out in BSIPs was announced on 4 April 2022.

DfT is currently scoping the process, impact, and value-for-money evaluation of BSIP initiatives, as well as an evidence synthesis of this and other ongoing evaluation activity which sits under the National Bus Strategy, to build understanding of what works to improve bus services, where, why and how.

Ongoing evaluation activity includes:

Monitoring and process evaluation of the Rural Mobility Fund

This provides funding for pilots of demand responsive transport schemes in 17 areas across England. This includes a 2 wave case study approach and programme-level monitoring.

The evaluation aims to understand the potential for demand responsive transport to fill gaps in public transport provision in rural and suburban areas, and lessons learnt in mobilisation and implementation. Interim evaluation outputs are expected in 2022 and 2023, with a final report expected in 2024.

Monitoring and evaluation of the Zero Emission Bus Regional Areas scheme

This provides funding for zero-emission buses and related infrastructure across 17 local transport authorities areas across England.

The evaluation approach is currently being finalised but will include programme-level monitoring of outputs, outcomes, and costs, including data to inform analysis of carbon impacts, and process evaluation interviews with local transport authorities, bus operators, and other involved parties.

The impact will be identified through investigating predicted causal relationships in a programme-level logic model, and through comparisons to modelled or actual counterfactual areas and bus routes.

The evaluation aims to understand the impact on measures such as air quality and bus patronage, and value for money of the programme. Interim outputs are expected in 2022, 2023, and 2024, with a final report expected in 2026.

DfT also oversees local monitoring and evaluation activity for a fares pilot in Cornwall which offers reduced and simplified bus fares for travel within Cornwall, and an all-electric bus fares pilot in Coventry, which provides funding to replace all diesel buses with electric buses in the area.

Interim findings are expected annually between 2022 and 2025 to 2026 respectively, with final reports expected in 2026 to 2027 respectively.

Inclusive Transport Strategy

The Inclusive Transport Strategy (ITS) sets out the government’s ambitions for disabled people to have the same access to transport as everyone else and to be able to travel confidently, easily and without extra cost.

The approach to evaluation is set out in the framework for the monitoring and evaluation of the ITS.

The study includes primary research with disabled and non-disabled people, assistance dog users, transport operators and local licensing authorities. It will also monitor change in 23 metrics relating to disabled people’s travel through the annual publication of an Inclusivity Scorecard.

The baseline evaluation report was published in January 2022, along with the Year 1 Inclusivity Scorecard.

Updates to the scorecard will be published annually from 2022 to 2024. The final evaluation report is due for publication in 2024.

Local highways maintenance and renewals

The government announced at the Spending Review 2021 that funding levels between 2022 to 2023 and 2024 to 2025 for both local highways maintenance, such as resurfacing roads, maintaining bridges and street lighting, tackling potholes and the Integrated Transport Block, will be set at the same levels as 2021 to 2022.

Funding for highways maintenance has been provided through a variety of sources including the Highways Maintenance Block (HMB), the Potholes Fund, and also as part of the overall CRSTS allocations for 8 eligible MCAs.

DfT collects data on local highways maintenance, based on their condition and treatment collected from local highway authorities, and produces annual publications. Monitoring has also been supplemented by qualitative research with local authorities to gather contextual information about their asset management strategies and challenges and explore the potential for additional data collection and monitoring and evaluation.

Findings from this research will be published in 2022.

Safer Roads Fund

The Safer Roads Fund is a £100 million fund targeted at England’s 50 most dangerous roads (as assessed by the Road Safety Foundation). The Fund was announced in 2017 and funding awards were made between 2017 and 2021. Implementation will continue until 2023.

As this was the first fund of its kind (encouraging a targeted, proactive approach to road safety in line with the principles of the Safe System approach), a process evaluation was commissioned to understand what worked well and less well in the Fund’s administration and to what extent it encouraged local authorities to adopt safe system principles.

A report of the first phase of the evaluation, covering the announcement and application stage, was published in 2018. A second phase of research, covering implementation, is underway and will report in 2022. We are considering options for further evaluation to assess the impact of the Fund.

Young and Novice Driver Interventions (Driver 2020)

DfT is undertaking a £2 million research programme to identify which new training, education and technology-based interventions for learner and novice drivers are the most effective in improving road safety (reducing collision risk). Following an initial evidence review in August 2017, fieldwork for this project, known as Driver 2020, commenced in 2019.

The study is trialling 5 interventions. 3 of these focus on the learning period: an app-based intervention designed to increase the amount and breadth of pre-test on-road experience, an e-learning intervention to train hazard perception skill and an educational intervention.

The remaining 2 interventions focus on the period immediately following test pass: an intervention utilising a mobile phone app to provide feedback to drivers on their driving style and a web-based intervention designed to engage parents or mentors in managing post-test driving in specific risky situations (such as driving after dark, and while carrying peer-age passengers).

The study uses randomised controlled trials with over 12,300 participants whereby those exposed to the interventions are compared with control groups to understand whether they reduce the likelihood of a collision post-test qualification.

The trials are supported by qualitative research with participants and those delivering the trials to explore how they are working in practice and any unintended consequences. The study will conclude and report in 2023.

Rail Transformation

The Rail Transformation Programme has been set up following the publication of the Williams-Shapps Plan for Rail, to deliver the reforms set out in the white paper. The programme involves organisations from across the sector and will establish a new organisation, Great British Railways, as the single guiding mind to deliver meaningful change for passengers, freight customers, taxpayers and society.

DfT and the Great British Railway Transition Team (GBRTT) have developed a structured benefits management approach to measure, monitor and report on the delivery of the programme.

DfT and GBRTT are currently working to scope, agree and plan additional evaluation activity. The first stage of evaluation is likely to involve an evaluation scoping study to assess data requirements and explore impact evaluation methods.

Rail fares and ticketing 

As set out in the Williams-Shapps Plan for Rail, DfT is transforming fares, ticketing and retailing for the railways with the aim of ensuring that these meet the needs of both 21st century passengers and taxpayers. We are developing fares and ticketing improvements across a number of areas, including new fares trials on selected routes and the rollout of contactless pay-as-you-go (PAYG) ticketing to new stations. 

An evaluation of the single-leg pricing trial on 3 London North Eastern Railway routes is currently underway and is expected to report in 2022. This study takes a quasi-experimental approach in comparing trial flows with similar routes outside the trial and will use rail ticketing data and passenger surveys to explore impacts for passengers and revenue.

An evaluation of the flexible season ticket product for part-time commuters is also currently underway and expected to report in 2022. This study primarily uses theory-based impact evaluation methods, drawing on both surveys with part-time commuters and rail ticketing data, to understand passenger and revenue impacts.

Further project-level evaluations are currently being scoped and will be confirmed as other new fares and ticketing options are rolled out. 

UK second-generation search and rescue aviation

The Maritime and Coastguard Agency (MCA) UKSAR2G programme replaces the existing contract for HM Coastguard search and rescue aviation services.

The service is monitored through internal analysis and search and rescue helicopter statistics are published regularly. Aerial Surveillance (ASV) planes are operated under a separate contract to search for missing and vulnerable people, detect pollution on the water and support law enforcement bodies.

UKSAR2G will replace ASV as well as driving innovation and value for money when the contract is awarded later in 2022.

UKSAR2G is to start operation in 2024.

A monitoring and evaluation programme will be delivered in three phases, the first of its kind for MCA.

Phase 1, which is in progress until UKSAR2G goes live, comprises the development of a monitoring and evaluation framework and the collection of baseline measures.

Phase 2 will be a mid-service process evaluation to assess how UKSAR2G was implemented and what effects it is having.

Phase 3 will comprise interim and final impact evaluations and an ex-post assessment of value for money.

Tackling loneliness with transport

In October 2018, the government launched its loneliness strategy, which identified loneliness as a significant social concern.

DfT will deliver pilot transport schemes that aim to reduce loneliness. It is funded by HM Treasury’s Shared Outcomes Fund.

The pilots will implement and deliver a range of effective, targeted pilot transport schemes for those at risk of loneliness. The pilots will be measured and evaluated rigorously to develop the evidence base.

DfT will conduct an exploratory evaluation that aims to understand what, if any, contribution transport can make to alleviating loneliness, and which types of interventions are more or less successful at doing so. The evaluation will also explore the benefits of the pilots over and above reducing loneliness. It will deliver both impact, process and value for money evaluation. It will report in full in 2023.

Reducing environmental impacts

This third group of projects supports the strategic priority of tackling climate change and improving air quality, including by decarbonising transport.

Office for Zero Emission Vehicles (OZEV)

OZEV develops and delivers initiatives to stimulate uptake of ultra-low and zero-emission vehicles, thus helping reduce air pollution on our roads and contributing to economic growth.

Its grant schemes provide support for the take-up of plug-in electric vehicles and chargepoint infrastructure across the UK, as well as innovation in the design, manufacture and use of zero-emission vehicles.

In addition to regular stakeholder consultations and routine monitoring of project delivery metrics, OZEV has commissioned 2 strands of monitoring and evaluation activity to improve its evidence about the impacts of its schemes.

A retrospective evaluation study

This will evaluate 3 schemes within the overall grant portfolio of OZEV:

• the plug-in vehicle grant for cars and vans
• the Electric Vehicle Homecharge Scheme (EVHS)
• the Workplace Charging Scheme (WCS)

This will report in 2022.

An overarching monitoring and evaluation framework

This will provide guidance on the actions OZEV can take between now and 2030 to collect the necessary information and data to track progress in delivering its interventions and feed into independent evaluations at a later date. 

The plan provides a portfolio-level logic model and an overarching annual data collection plan, complemented by scheme-level equivalents. It covers various elements of the current portfolio including the Plug-in Vehicle Grant, the Electric Vehicle Home Charging Scheme, the Workplace Charging Scheme, the On-street Residential Chargepoint Scheme, the Local Electric Vehicle Infrastructure Scheme and building regulations to implement chargepoints and enabling infrastructure.

Plans to examine consumer experience of chargepoint regulations are also included. The framework is intended for internal use, with subsequent relevant evaluations to be published in due course.

Additionally, an evaluation of the Rapid Charging Fund is being scoped.

Collectively, these evaluations for OZEV will improve the provision of evidence to measure the progress of the programme’s activities as we move towards the UK’s 2030 target for reduced emissions.

Zero Emission Road Freight (ZERF) demonstrator programme

The ZERF demonstrator programme will create an evidence base on which technology (or technology mix) is best suited to decarbonise UK’s heaviest road freight vehicles (40 tonne to 44 tonne trucks).

Hydrogen fuel cell and battery electric competitions, managed by Innovate UK, are launching shortly.

The transition to zero-emission trucks will help improve air quality, will create greener jobs and will reduce UK’s reliance on imports of foreign oil.

Three strands of monitoring and evaluation are planned:

• Innovate UK, with support from an external supplier and DfT’s analysts, will implement robust monitoring as part of a new comprehensive approach which will start in 2022, as soon as the programme is launched, this will include a collection of consistent baselines as well as inflight and post-completion data which will feed into periodic independent evaluations

• technical evaluation carried out by consultants commissioned by Innovate UK will focus on technical data logging and cleansing as well as on infrastructure and on user research

• independent impact evaluation is intended to run throughout the operational period of technologies operating on roads (current estimate: 2025 to 2029), the appropriate methodology is currently being scoped with options including difference-in-difference and interrupted time series analyses

These evaluations will build on the early findings from a pilot run by Innovate UK in 2021 which supported design and feasibility options.

As part of that pilot, 20 battery-electric heavy goods vehicles were trialled by public sector organisations, such as the NHS and local authorities. This enabled learning to be gathered from field testing vehicles in a real-world, real-time logistics environment. This project, along with additional 6 successful feasibility studies, helped to lay the foundation for the demonstrations.

Local authority plans to tackle NO2 concentrations

Defra and DfT published in 2017 a UK plan for tackling roadside nitrogen dioxide concentrations and a Clean Air Zone Framework, which set out the principles for setting up clean air zones in England.

In 2018, Defra and DfT commissioned an evaluation of local authority plans to tackle nitrogen dioxide concentrations, including charging clean air zones.

The evaluation provides evidence to inform the implementation and maintenance of local plans and assess how effective they have been in meeting their aims. It comprises trend analysis of air quality and traffic data and case studies of the implementation of local plans and cross-cutting issues of interest.

The Annual Evaluation Report 2021 was published in May 2022.

The evaluation is expected to conclude in 2024.

Active travel

In May 2020, DfT announced £2 billion of new money for cycling and walking over the course of this parliament. This funding will be spent on a wide range of cycling and walking interventions including new and improved infrastructure (for example, segregated cycle lanes, school streets and low traffic neighbourhoods) and behavioural interventions such as cycle training and encouraging children to walk and cycle to school.

DfT is planning and undertaking a wide range of monitoring and evaluation activity to learn lessons on the delivery of active travel interventions, and to generate an understanding of the impact and value for money of different types of schemes.

This evidence will help determine what interventions are contributing towards DfT’s cycling and walking targets outlined in Gear Change, 2020.

Due to the complexity of active travel funding, a thematic, cross-cutting evaluation is being delivered, to understand what works for particular types of interventions, why and how. A range of different research methods are and will be used including surveys, interviews, focus groups, cycle counters, meta-analysis and more.

A range of more focused trials will be conducted to generate evidence for particular intervention types such as, for example, comparing the usage of new segregated cycle lanes before and after new or improved infrastructure is implemented.

Much of the evaluation activity is still in the scoping stages. Early findings from the Active Travel Fund tranche 2 will be published in 2022 and 2023.

E-scooters trial evaluation

Trials of rental e-scooters are being conducted in more than 30 local authorities between 2020 and 2022 to test their safe use and wider impacts.

DfT introduced the legal changes to allow trials, published guidance for local areas and rental operators and worked with local authorities to coordinate and support the development of their proposals. Local authorities worked with e-scooter operators to procure new services under the guidance.

Following this, DfT has worked with local authorities and operators to resolve issues as they have arisen. Trials have been extended to 30 November 2022.

DfT has taken learnings from the trials to date and issued updated guidance for the trial extension to ensure trials are as safe and well run as possible.

DfT commissioned a national evaluation of the e-scooter trials. This evaluation was designed to complement local monitoring and evaluation activity and collect robust evidence about the safety, benefits, public perceptions and wider impacts of e-scooters to inform future policy and to learn implementation lessons to guide roll out of future services.

Fieldwork for the evaluation concluded in December 2021 and findings will be reported in 2022.

Completed evaluation studies and publication list

This section provides a record of the DfT’s completed evaluation studies.

For 2018 and earlier periods, we have only provided links. Studies after 2018 contain a short summary of the evaluation findings.

2022

Evaluation of Local NO2 Plans - 2021 Annual Report

This annual report of the evaluation provided early findings from the first local authority plans to be implemented, with a focus on the clean air zone in Bath.

The report shows that air quality in Bath is showing signs of improvement that are likely to be driven by factors other than reduced traffic flows cause by COVID-19 travel restrictions. Analysis of survey data found that there has been a small-scale shift in business and resident travel behaviours.

Additional analysis of traffic data highlights a clear move towards the use of cleaner vehicles within the Clean Air Zone area, indicating that it has been successful in encouraging vehicle compliance.

This evaluation of local authority plans to tackle NO2 concentrations is described in futher detail.

First of a Kind Innovation Programme

The First of a Kind Innovation Programme evaluation was commissioned to review the effectiveness of the Rail Innovation Programme with a view to understanding how best to stimulate innovation within the rail industry in future. The study focused on the ‘First of a Kind’ (FOAK) competition rounds 1 to 3, launched between 2017 and 2019. Evidence was collected through an online survey, stakeholder interviews and review of documents.

The evaluation found that FOAK adds value and supports participants to overcome some traditional barriers to innovation within rail and help projects advance through Technology Readiness Levels (TRL). However, its aim to support high TRL projects into market quickly was not always realised as a majority of projects covered (8 out of 15) were not being delivered to market within 12 months.

It was noted as a point for future consideration that that the FOAK process was often leaving projects at a point whereby there was no natural process for them to reach market, with additional time and funding normally required. The evaluation also identified some indirect benefits of FOAK, including helping innovators to develop relationships with key rail stakeholders.

Temporary Traffic Regulation Order impact evaluation

The Temporary Traffic Regulation Order impact evaluation was commissioned to examine the effects of the Traffic Orders Procedure (Coronavirus Amendment) Regulations of May 2020 and network management duty guidance which were introduced to enable local authorities to make temporary Traffic Regulation Orders to deal with the effects of COVID-19 and enable social distancing.

The evaluation involved an online survey of local authorities and in-depth interviews with local authorities and groups representing road users. It explored experiences of the measures and the planned greater use of digital communication methods to announce them.

The evaluation provided evidence about reactions to emergency measures which included some concerns about the nature of measures and the reduced time for consultation. The increased use of digital communication methods was welcomed by both by authorities and groups representing road users. However, potential costs savings from digital methods did not fully materialise as many local authorities felt that they needed to continue with use of print advertising.

Inclusive transport strategy: evaluation baseline and technical reports

This first published output of the evaluation of the ITS provides a detailed baseline picture of disabled people’s experiences using the transport network, based on extensive primary research with disabled people, the wider travelling public, bus and train operators, and taxi and private hire vehicle local licensing authorities.

The research began at the start of the COVID-19 pandemic but focused on people’s perceptions and experience in the 12 months prior to March 2020.

The evidence shows that the transport network is working well for most people most of the time, and that experiences when travelling are generally positive. However, across many areas, disabled people report more negative experiences than non-disabled people.

This included not having confidence to travel, not having enough time to board or alight a bus or train and having difficulties accessing parts of the transport system, such as insufficient space for a scooter or wheelchair. Thus, the research identified a broad range of challenges to overcome before the ITS ambitions are fully realised.

2021

Cycle City Ambition Programme

The Cycle City Ambition Programme provided £191 million in capital grants to eight cities between 2013 and 2018. Its evaluation focussed on 14 schemes in the 8 cities, comprising between a quarter and two-thirds of the DfT grant. It used local cycling counts and surveys of cyclists and residents.

The evaluation found that In all eight cities, city-wide cycling levels increased during the course of the CCA programme, with some mode shift from cars. Cycling levels increased between 4% and 79% across the cities, with a mean value of 37%, and comparisons with control sites confirmed that the bulk of the increase was attributable to the investment.

Cycle mode share in the cities increased by between 0.2%-points and 5%-points, sometimes accompanied by a corresponding decrease in car mode share. Analysis of individual CCA schemes provided evidence that some of the city-wide increase in cycling was attributable to the CCA investment, and that improvements in cycle infrastructure made cycling appealing to a wider range of people.

Transforming Cities Fund: co-development case study

This process evaluation of the co-development process used to develop TCF business cases used structured discussions with officials from DfT and officers from participating local authorities.

It found that the pilot was successful overall. The principal benefits were the combining of the government’s strategic objectives with locally-identified solutions, the concentrating of resources into long-term, multi-modal programmes, and the enhancing of mutual understanding between DfT and local government.

The evaluation found the process would have been improved by greater clarification of roles and responsibilities from the start, delivery that was tailored to and more consistent for funded areas, and more effective process for reviewing and managing local areas’ progress.

A government response to the study was published.

This noted that ‘supported bidding’ may be a more accurate term than ‘co-development’ and set out how the findings would be applied when such an approach is used in the future, including indicating a ‘funding envelope’ to set and manage expectations – an approach that has already been implemented in the CRSTS bidding process.

Longer semi-trailers: annual report 2019

Evaluation of a trial of longer semi-trailer HGVs (LSTs) since 2012 has found that their operation instead of standard semi-trailers saved an average percentage distance of around 8%, which equated to 1 in every 12 journeys. 

Trial conditions included special driver training (required by the trial), as well as additional attention to driver selection and route selection (reported by operators, though not required by the trial). 

The trial monitored safety incidents, finding that under the conditions of the trial LSTs were operated safely, even when used on routes involving some minor roads.

2020

Increased speed limit for HGVs

This 3 year study monitored the effect of increasing speed limits for HGVs over 7.5 tonnes in England and Wales to 50mph (from 40mph) on single carriageways and 60mph (from 50mph) on dual carriageways in 2015.

The study verified that speeds appeared to have increased following the policy change by a moderate amount (by 1.6mph on single carriageways and 0.5mph on dual carriageways) and that these increases could be assumed to translate into economic benefits of £225.8 million over a 16 year period.

Accompanying assessments found no attributable change in noise, no evidence of a change in air quality and no evidence of a change in collisions due to the policy.

2019

Bikeability Impact study

This impact evaluation of the Bikeability cycling training programme compared changes in cycling behaviour for pupils in schools where the programme was delivered in year 5 with pupils in matched schools where the programme was delivered a year later, in year 6.

The study found positive, and statistically significant, impacts associated with participating in Bikeability training that were consistent with the aims of the programme including higher cycling rates, higher incidence of cycling on the road, higher rates of parents allowing their children to ride on roads and greater pupil knowledge of how to ride on roads safely.

Evaluation of Roads Reform (Wave 2)

This evaluation of the changes that have been made to the management and operation of the Strategic Road Network (SRN) since 2014 and it will conclude in 2022 or 2023.

The Wave 2 report found that, whilst it was too early to assess the long-term impacts of the programme, progress made so far broadly aligned with the original expectations for the reforms.

It found indications of short-term changes that are expected to lead to intended long-term effects, and identified examples of the reforms having contributed to changes in organisational priorities, capabilities, structures, processes, working practices and behaviours that would have otherwise been unlikely to occur.

Scoping study: evaluating economic performance impacts of road enhancement

This scoping study prepared for a potential evaluation of the local and national economic impacts of road network investments that has subsequently been commissioned.

In developing an evaluation framework, it identified a range of economic performance outcomes that could be assessed in an evaluation, pointed to a generally good availability of data, and recommended several suitable econometric approaches.

The study also put forward a rationale for case studies and specified the approach that might be adopted, both to demonstrate outcomes to a range of stakeholders and provide an understanding of the mechanisms that result in economic performance.

Total transport feasibility report and pilot review

Total transport is about finding ways of commissioning public sector funded transport so that passengers get a better service with less duplication of resources. This review considered the progress made and results achieved by total transport pilot schemes run by 36 local authorities in England in 2016 to 2017.

2018

• 20mph speed limits impact and process evaluation
• Safer Roads Fund Process Evaluation Phase 1 Report
• Behavioural insights interventions to encourage learner drivers to practise more
• Cycling Cities Ambition: baseline and interim report
• Economic impact of new and improved rail lines: case studies and method
• Street works permit schemes: evaluation of effectiveness
• Impact evaluation of National Speed Awareness Course
• Inclusive Transport Strategy: monitoring and evaluation framework
• Impact of the Local Sustainable Transport Fund: synthesis of evidence
• Local Major Schemes: meta-evaluation 2006 to 2010
• Local Sustainable Transport Fund: final meta-analysis
• LSTF evaluation: carbon impacts and congestion relief
• LSTF evaluation: impact of sustainable transport measures on town centres
• LSTF evaluation: strategic employment sites and business parks
• LSTF evaluation: visitor travel and sustainable transport
• Sustainable Travel Towns: evaluation of the longer-term impacts
• Thameslink Programme Evaluation: baseline report

2017

• Exploring the economic benefits of strategic roads
• An evaluation of low cost workplace-based interventions to encourage the use of sustainable transport
• Drug driving law evaluation report
• Impact of the Local Sustainable Transport Fund: summary report
• Local Sustainable Transport Fund annual report 2014 to 2015
• Local Sustainable Transport Fund: what works?
• Low Carbon Truck and Refuelling Infrastructure Trial: final report
• Review of interventions to increase the safety of young and novice drivers
• Search and rescue helicopter statistics

2016

• Cycling infrastructure: framework for evaluating economic and social impacts
• Evaluating fixed penalty notices for careless driving offences
• Increased speed limit for heavy goods vehicles over 7.5 tonnes: initial summary report
• Longer semi-trailer trial evaluation: annual report 2015
• Street works lane rental evaluation
• Strengthening the links between appraisal and evaluation

2015

• Better Bus Areas programme impact evaluation baseline data report
• Better Bus Areas programme process evaluation report
• HS1: first interim evaluation
• Local Sustainable Transport Fund annual report 2013 to 2014
• Local Sustainable Transport Fund: interim meta-analysis
• Longer semi-trailer trial evaluation: annual report 2014
• Low carbon truck trial: executive summary 2014

2014

• Local major schemes: meta-evaluation 2007-2012
• Longer semi-trailer trial evaluation: annual report 2013
• Revenue support freight grant schemes: review

1. Monitoring is a process for tracking progress in the delivery of an intervention by collecting data on its outputs. ↩

2. Evaluation is the systematic assessment of an intervention’s design, implementation and outcomes. ↩

3. See the Magenta Book Annex on evaluation methods for descriptions of different impact evaluation methods. ↩

4. The report’s main recommendations were: to implement appraisal handover packs as a method of passing forward appraisal information to the evaluator; to develop portfolio approaches for evaluation to enable meta-analysis to check on appraisal and modelling assumptions; and to align evaluation work with appraisal evidence needs. ↩

5. The purpose of the economic case is to identify the proposal that delivers best public value to society, including wider social and environmental effects. ↩

6. The strategic case is one of Five Case Model Methodology for the preparation of business cases. Its purpose is to make the case for change and to demonstrate how it provides strategic fit. For information about business cases, see HM Treasury’s Guide to developing the project business case. ↩

7. The purpose of the management case is to demonstrate that robust arrangements are in place for the delivery, monitoring and evaluation of the scheme, including feedback into the organisation’s strategic planning cycle ↩

8. The term ‘major projects’ refers to government’s most complex and strategically significant projects and programmes which comprise the Government Major Projects Portfolio (GMPP), overseen by the Infrastructure and Projects Authority. During the last reporting year, 2020-21, DfT had 25 major projects in the GMPP. Most of these are covered by the Evaluation Programme either individually or at the portfolio level (for example, Rail Enhancements Portfolio) or are covered by the National Highways Post Opening Project Evaluation series. ↩

9. Benefits management is the identification, quantification, analysis, planning, tracking, realisation and optimisation of benefits, that is the measurable changes that a project seeks to deliver. It is a project delivery process, mandated by the Infrastructure and Projects Authority, that seeks to ensure organisations realise the planned benefits from their investments. See the IPS’s Guide for effective benefits management in major projects, 2017. ↩

10. Post-legislative scrutiny refers to a broad form of review, the purpose of which is to address the effects of a legislation in terms of whether the intended policy objectives have been met by the legislation and, if so, how effectively. ↩

11. Post-implementation review (PIR) refers to the review of regulatory policy that complements the ex-ante appraisal contained in an impact assessment. It is a process to establish the effectiveness of a regulation after it has been implemented and operational for a period of time. It addresses the extent to which a regulation is delivering its intended effects, whether there have been any unintended effects, how well it is working and the reasons why. ↩