Policy paper

DSIT evaluation strategy

Published 11 February 2025

© Crown copyright 2025

Sarah Munby, Permanent Secretary of the Department for Science, Innovation and Technology.

Foreword from the Permanent Secretary

As Department for Science, Innovation and Technology (DSIT) Permanent Secretary, I am pleased to introduce our first evaluation strategy.

DSIT is at the heart of the government’s work to improve people’s lives through maximising the potential of science and technology. Our aim is to enhance the UK’s digital presence, support the science and tech sectors, and maximise the power of AI and other emerging technologies. In addition, by combining technology policy and digital delivery, DSIT is a key enabler and a delivery partner to every department as we work together to implement a mission-driven government.

Delivering impact in DSIT relies on the generation, and use, of high-quality evidence and evaluation. Building a strong evidence-base of what works, for whom, and why, is essential to enabling policymakers to make informed choices that can lead to better outcomes for the public.

This strategy reflects our commitment to embedding evaluation into every facet of our work, ensuring that our policies and programs are driven by evidence and continuous improvement. Our vision is for a modern, innovative government that leverages evaluation to drive progress and deliver tangible benefits to UK citizens.

We will achieve this vision through a comprehensive set of actions outlined in this strategy; we will maintain and strengthen our evidence base across DSIT and its partner organisations, establish clear governance structure that support rigorous evaluation, and ensure findings are effectively used to drive continuous improvement and impactful outcomes. 

I look forward to working with our stakeholders to implement this strategy and achieve our goals as a newly expanded department.

Sarah Munby
Permanent Secretary of the Department for Science, Innovation and Technology (DSIT)

Executive summary

This is the Department for Science, Innovation and Technology’s (DSIT) first evaluation strategy. It outlines an ambitious plan for delivering a demonstrable change in how DSIT plans, implements, and learns from evaluation activity in the department.

DSIT will make evaluation integral to our policy development and decision-making, drawing on cutting-edge methodological advances to continuously learn, adapt, and improve the department’s impact on the economy and UK citizens.

To achieve DSIT’s vision, we have outlined 3 core objectives that we seek to achieve alongside a clear plan of action for each of them:

1. Effective evaluation governance – establishing clear, accountable governance structures that support rigorous and ethical evaluation practices.
2. Robust and innovative methodologies – employing scientifically sound and innovative methodologies to ensure the reliability and validity of our evaluations.
3. Improved evaluation dissemination and use – ensuring that evaluation findings are effectively communicated and utilised to inform policy and practice, driving continuous improvement and impactful outcomes.

1. Introduction

DSIT was founded in February 2023 and focuses on improving people’s lives by maximising the potential of science and technology. We accelerate innovation, investment and productivity through world-class science, ensure that new and existing technologies are safely developed and deployed across the UK and drive forward a modern digital government for the benefit of its citizens. The department will achieve this by: 

• Driving economic growth: by maximising the impact of government action to drive national and regional growth through supporting R&D, technology, and innovation.
• Improving the performance and productivity of government: by maximising value for money for the taxpayer by applying AI and digital technologies to transform the citizen experience of public services, making it more effective, convenient, and productive.
• Empowering citizens: by maximising the control people have over their interactions with government by widening and deepening their digital skills, knowledge, and trust in technology.

DSIT is supported by 16 agencies and public bodies. While some of our Arm’s-Length Bodies (ALBs) have published their own evaluation strategies, we collaborate closely to share expertise and maximise the impact of our evaluations.

2. Evaluation in DSIT

To ensure high-quality outputs, DSIT is committed to following His Majesty’s Treasury’s (HMT) Magenta Book guidance on evaluation and HMT’s Green Book: guidance on appraisal and evaluation. To ensure appropriate resourcing of evaluations, evaluation budgets in DSIT are expected to be proportionate to the programme and the relevant evidence base, constituting circa 1 to 10% of programme budget with the primary rule for budget allocation being proportionality.^{[footnote 1]} We are also committed to registering all DSIT evaluations on the Evaluation Registry in line with official guidance.

DSIT analysts are actively working to continuously improve our methodological capability, test new approaches and share relevant expertise. We have already completed several projects that help to address common methodological challenges. Examples of our innovative evaluation projects include (see Annex A for more detail):

• Guidance on Research Development and Innovation (RDI) evaluations that describes common methodological challenges and effective strategies to overcome them.
• Guidance on evaluating the impact of Artificial Intelligence (AI) interventions that reflects additional opportunities and challenges for evaluation in the AI space.
• Testing new methods, such as rubric-based Value for Money (VfM) approaches and synthetic control groups.
• Developing robust theory-based evaluations, such as the Open Networks Programme (ONP) and AI Safety Institute evaluations.

While DSIT strives to deliver robust evaluation methodologies, there are 3 core challenges to delivering science, technology and innovation impact evaluations that we aim to overcome:

• Difficulty in establishing counterfactuals: Investment in science and technology is typically awarded to proposals that have the highest chance of delivering economic and societal benefits. Sample sizes are often small, which makes experimental and quasi-experimental evaluation approaches difficult to implement.
• Long-term nature of programmes and outcomes: Investment in science, technology and digital programmes often take place over several years and sometimes decades. It is also common that projects will require several years, or decades, for benefits to be fully realised. This poses challenges with ensuring effective long-term learning from evaluation evidence.
• Indirect links between programmes and outcomes: Outcomes from science, technology and digital programmes are indirect and spread widely across the economy, making it challenging to establish clear causal links. In addition, research findings often build on other discoveries, making it challenging to attribute breakthroughs to a single intervention or source of funding.

3. DSIT’s evaluation vision

DSIT will make evaluation integral to our policy development and decision-making, drawing on cutting-edge methodological advances to continuously learn, adapt, and improve the department’s impact on the economy and UK citizens.

We will cultivate a thriving evaluation culture at the heart of DSIT, where evaluation is integral to impactful decision-making. We will prioritise embedding high-quality, innovative and proportionate evaluation plans across our investments and policies. By doing so, we will build a comprehensive evidence base to inform policy design, development, and implementation, ensuring that our interventions deliver maximum impact, and that public funding is spent as effectively as possible.

We are committed to fostering transparency, accountability, and excellence in all our activities, focusing on pragmatic but impactful solutions to drive technological advancements that are safely developed for the benefit of UK citizens. By continuously learning from previous initiatives, we will accelerate innovation, investment and productivity through world-class science, research and innovation.

4. Achieving DSIT’s evaluation vision

Evaluations play a crucial role in DSIT continuously striving to deliver better outcomes for the public. This strategy aims to enable learning and innovation not only in evaluation design, but also by cultivating a policy culture where new approaches can be trialled, tested, learnt from and adapted to maximise the impact of our policies.

To ensure we achieve those ambitions, we outlined 3 key objectives for this strategy:

1. Effective evaluation governance: Establish clear, accountable governance structures that support rigorous and innovative evaluation practices to reduce bureaucracy and promote collaboration with Arms Length Body (ALBs).
2. Robust evaluation methodologies: Use and promote new and innovative approaches in our evaluations to become a trailblazer in evaluation methodology.
3. Impactful evaluation dissemination and use: Effectively communicate and utilise evaluation findings to inform policy and practice driving continuous improvement and impactful outcomes.

Objective 1: Effective evaluation governance

Why

Evaluation governance is the foundation for conducting robust evaluations and ensuring evaluation evidence is used in decision-making. To develop a thriving evaluation culture, there needs to be effective coordination and support of evaluation activity across the department while minimising the bureaucratic burden. Establishing clear, proportionate governance structures will ensure that we have the necessary supporting mechanisms in place to hold ourselves accountable to deliver robust and impactful evaluations, and evidence-based policymaking.

How

DSIT aims to establish and embed evaluation governance that ensures proportionate oversight of evaluation activities across the department, utilising existing processes and products and learning from other government departments’ experiences. We will work collaboratively with the Analytical Leadership Team (ALT), the Investment Committee, and other internal oversight boards to ensure our proposals meet the diverse needs of our stakeholders.^{[footnote 2]}

What

To achieve this objective, we have identified 3 key areas of focus:

Focus area 1.1: Governance and assurance structures

Oversight and governance of evaluation activity is crucial to ensure that evidence meets the strategic needs of the department. To facilitate this, the central analysis team takes a core role in assessing evidence needs, ensuring plans are implemented to address the needs and that the departmental governance is both supported by, and works for, that evidence generation.

To achieve this, we will:

• Strengthen the current evaluation governance structure by developing an efficient and effective way of tracking and supporting evaluations in DSIT. Proposals include an appointment of single evaluation leads in each analytical team, who will meet quarterly with the central evaluation lead to discuss progress and challenges. Insights gained from completed evaluations will be analysed and disseminated throughout the department to ensure continuous learning from our interventions across the policy cycle.

Focus area 1.2: Evaluation plans in business cases

Monitoring and evaluation (M&E) planning is an integral part of new policy and programme development. DSIT will foster a culture of curiosity, championed by the Director of Analysis, we will establish a culture of learning and developing effective feedback loops back into policy. Using existing governance structures, to reinforce that culture across DSIT.

To achieve this, we will:

• Review and update DSIT’s M&E Business Case templates and guidance by March 2025 to improve the guidance on the standards of M&E expected at each stage of the business case.
• As part of the monitoring and evaluation (M&E) Business Case review, introduce a requirement for all DSIT programmes going through the Investment Committee approval (total budget over £40 million) to have an evaluation steering group to improve external challenge and the quality of DSIT evaluations.

Focus area 1.3: Evaluation commissioning

This area seeks to strengthen DSIT’s evaluation commissioning processes to enable the relevant and timely delivery of high-quality evaluations. Robust procurement practices are a crucial element of delivering robust evaluations. DSIT’s research assurance process is designed to be proportional, balancing the need for robust quality control with the requirement to produce evidence in a timely manner so that it can effectively inform policy.

To achieve this, we will:

• Develop a guidance document setting out best practice in research and evaluation procurement.
• Introduce a requirement for high priority, high value or complex evaluations that have been commissioned out, to conduct a lessons-learnt exercise within 3 months post completion (report publication). This requirement will be introduced in early 2025.

Objective 2: Robust and innovative methodologies

Why

Given the methodological challenges in delivering evaluations, we aim to integrate the use of cutting-edge methods to enhance our evaluation capabilities as set out in Section 2 and Annex A. By finding new ways to deliver robust and innovative evaluations, we can have confidence in the quality of our evidence, helping us to build a reputation for evaluation excellence and maximise public benefits from our investment and policies.

How

DSIT’s analytical community is actively working to continuously improve our methodological capability, test new approaches and share relevant expertise. We will continue building on the existing expertise and address our methodological challenges head-on through ongoing training and targeted evaluation projects.

What

To achieve this objective, we have identified 2 key areas of focus:

Focus area 2.1: Evaluation capability

To ensure that robust and innovative evaluations are delivered, DSIT strives to strengthen evaluation capability across the department. Formal evaluation training for analysts and in-house support networks are an important part of wider work to improve evaluation capability across DSIT. To achieve this, we will:

• Provide in-house introductory training to all new starters in analysis roles to ensure all analysts understand key evaluation concepts and its importance.
• Deliver 2 externally commissioned evaluation training courses to a cohort of analysts from January 2025. The training courses will include:
  • an introduction to policy evaluation and
  • an intermediate course in policy evaluation.
    After completion, we will conduct a formal review process to assess ongoing needs for further training provision in the department.
• Develop a long-term evaluation training programme in 2025 to ensure DSIT continues to upskill their staff in evaluation methods. The provision will also assess the optimal combination of training to inform a longer-term recommendation in future financial years.

Focus area 2.2: Evaluation methodology

DSIT strives to continuously improve the quality of our evaluations, test new approaches and share relevant expertise across the Department and ALBs. We will maintain and build more strong networks across the academic community to learn from their expertise and ensure DSIT is an outward looking, open and curious in its analysis. Constantly learning and innovating in the analysis that we do. To achieve this, we will:

• Continue to support the evaluation community (see Annex B) and review its structure in 2025 to facilitate knowledge sharing and peer-to-peer support on evaluation projects.
• Continue to run, and promote, DSIT’s RDI VfM and Evaluation Network. This informal network hosts over 80 analysts across government to foster collaboration and encourage a consistent approach to VfM appraisal and evaluation.
• Review how we can utilise DSIT’s College of Experts as part of its scoping stage in 2025. The Chief Scientific Advisor’s team will explore how the launch of DSIT’s College of Experts can aid the Department in ensuring evaluation designs are best suited to challenging and often highly technical policies and programmes.
• Develop a DSIT metrics dictionary, a list of metrics commonly used in DSIT by mid-2025. This will improve the comparability of metrics, enable more robust synthesis of findings, and improve the efficiency of evaluation planning.

Objective 3: Improved dissemination and use

Why

The value of an evaluation does not only come from its robustness but also from its practical application and influence on decision-making. If we do not implement learning from evaluations, we will not fully understand the effectiveness and impact of our policies, leading to missed opportunities for improvement. By ensuring evaluation findings are effectively communicated and utilised by policymakers, we can maximise the impact and effectiveness of our policies, driving continuous improvement.

How

We will work collaboratively with analysts across the department to ensure evaluations are disseminated effectively. We will work with key internal and external stakeholders, ensuring that findings from process evaluations and early impact evaluations are used to continuously improve our investments and ‘course-correct’ in-flight. Working with the policy profession in DSIT, we will ensure policy colleagues are offered regular training on the importance and value of evaluations in policymaking.

What

To achieve this, we have identified 2 key areas of focus:

Focus area 3.1: Evaluation dissemination

A large proportion of an evaluation’s value comes through its use and application. Dissemination plans should be considered early in the evaluation design. When evaluations are commissioned to external research agencies, government researchers should clearly set out their expectations for outputs and their dissemination. To achieve this, we will:

• Develop evaluation dissemination guidance by mid-2025 which outlines minimum requirements and best practice for dissemination. This will ensure evaluation findings feed into decision making and analysts learn from tested methodologies
• Ensure timely publication of evaluation reports to foster transparency and accountability, in line with the Government Social Research (GSR) Publication Protocol on an ongoing basis.
• Ensure the new evaluation governance structure aids dissemination of evaluation findings and promotes their use in BCs and wider policymaking.
• Utilise evaluation steering groups to effectively disseminate findings with key internal and external stakeholders.

Focus area 3.2: Evaluation training for non-analysts

While responsibility for methodological robustness mainly rests on analysts in DSIT, policy officials are crucial in ensuring effective use of evaluation evidence in policymaking. Therefore, the central analysis team provide M&E training to new joiners in the DSIT policy profession and offer regular evaluation teach-ins to non-analysts. This training offer aims to build an appreciation and understanding of the value evaluations bring to policy and programme design, help to bolster learning from evidence, and maximise the impacts of DSIT’s interventions. To continue building knowledge of evaluation in the policy profession, we will:

• Continue to deliver the policy foundations training and regular lunch and learn sessions for non-analysts including, but not limited to, policy, service designers, delivery managers, and product managers.
• Review the success of the 2024/2025 evaluation training offer based on collected feedback by May 2025. This will ensure that the training remains relevant and useful to non-analysts and maximise its impact.

5. Ensuring the strategy is implemented

Operationalising this strategy is the responsibility of all analysts in DSIT, led by the ALT, and supported by the central analysis team. However, analysts alone cannot ensure evaluation evidence is used to inform decision making. Therefore, ongoing support and strong endorsement from DSIT’s entire senior leadership as well as the policy profession and other relevant professions (e.g., project managers) is crucial to achieving the 3 key objectives of this strategy. Key responsibilities for main stakeholder groups are outlined below:

DSIT senior leadership:

• Championing for routine use of evaluation evidence and learning in policy and intervention design.
• Advocating for close collaboration between analysts and policy to ensure evaluation is built-in early in the intervention design.
• Supporting effective evaluation delivery through ensuring appropriate resourcing, evaluation budgets, and communicating the importance of evaluation with internal and external stakeholders.
• Overseeing the implementation and adherence to the strategy within their respective areas.

Central analysis team:

• Providing central oversight of priority evaluations, offering advice and linking with other experts where appropriate.
• Ensuring evaluation governance processes are clearly communicated across the department.
• Regularly engaging with the Evaluation Task Force (ETF), cross-government Evaluation Group, and other government departments to identify and share best practice and ensure strategic alignment of evaluation activities in DSIT and beyond.
• Monitoring implementation and impact of this strategy and reporting any related risks to the Director of Analysis to ensure timely action.

 Analysts working on evaluations across DSIT: 

• Disseminating evaluation findings amongst relevant analysts and policy teams, ensuring the evidence base is continuously updated and influences policy design.
• Utilising and seeking out evaluation learning opportunities and sharing feedback and learning within teams and the evaluation community.
• Participating in DSIT evaluation community meetings and teach-ins both as audience and presenters, seeking to highlight innovative and robust approaches, helping to identify new speakers and methodological issues that need tackling as part of the Community.
• Flagging issues relating to evaluations and adherence to this strategy to senior leaders and the central analysis team openly and early. Engaging in the process for addressing those issues.

Non-analysts (including policy officials, project managers, delivery managers, service designers and product managers): 

• Developing an awareness of this strategy and understanding the importance of timely and well-considered evaluation and evaluation evidence.
• Building an awareness and knowledge of relevant evaluation findings and utilising them in policy and programme development.
• Ensuring engagement with analytical colleagues early in the policy and programme development process, helping to secure M&E budgets and highlighting any potential M&E risks as early as possible.

6. Conclusion

This evaluation strategy represents a significant step forward in embedding evaluation into the core of our policy development and decision-making processes. By establishing clear governance structures, employing innovative methodologies, and ensuring the impactful dissemination and use of evaluation findings, we are committed to maximising the impact of our investments. This Strategy not only underscores our dedication to evidence-based policymaking but also fosters a culture of continuous learning and accountability within the department. As we move forward, we will work closely with our partners and stakeholders to build a comprehensive evidence base that informs policy design and ensures the effective use of public funds. Through this collaborative effort, we aim to drive progress, deliver tangible benefits to UK citizens, and position DSIT as a leader in evaluation excellence.

Annex A: Examples of evaluation work in DSIT

The strategy builds on existing innovative and robust evaluation projects in DSIT. Examples of DSIT’s evaluation work are described below.

Evaluation guidance

Guidance on evaluation methodologies for RD&I

DSIT commissioned research on what methods work for evaluating the impact of public investments in Research, Development and Innovation (RD&I). This research report provides examples of innovative evaluation methodologies from across the world to measure the impact of RD&I interventions. Case studies and lessons learnt from evaluation strategies used in “real world” contexts demonstrate how to address commonly encountered challenges in measuring the impact of RD&I interventions, such as long timeframes in realising benefits and indirect impacts. The guidance has been shared across DSIT via DSIT’s evaluation community and with other government departments via the Cross-government RDI VfM and Evaluation Network to share best practice and maximise its impact.

Evaluation of AI projects guidance

The ETF, in collaboration with DSIT, has also developed guidance on the specific challenges associated with evaluating AI interventions. The guidance highlights the unique challenges of this emerging and fast-moving field alongside the opportunities AI interventions bring in terms of possible evaluation approaches. The guidance lays out the key principles of best practice for evaluating AI interventions and grounds these in 4 hypothetical case studies to illustrate worked examples of conducting evaluations in this area. This guidance will directly support DSIT’s approach to AI and other emerging technologies evaluations.

Testing new evaluation approaches

Assessing VfM in DSIT’s international research and innovation funds

The Newton Fund, Global Challenges Research Fund (GCRF) and the new International Science Partnerships Fund (ISPF) have used an innovative rubric-based approach to assessing VfM based on work developed by Oxford Policy Management. It uses explicit evaluative reasoning/criteria to provide a transparent means to make robust VfM judgements from qualitative and quantitative evidence, helping to overcome methodological challenges associated with a lack of a counterfactual. These assessment criteria are organised within the NAO’s accepted ‘4Es’ (Economy, Efficiency, Effectiveness, Equity) framework.

Synthetic counterfactuals proof-of-concept study

The DSIT Data Science Team, working with the DSIT Space Analysis team, are attempting to construct a proof-of-concept counterfactual for grants supporting businesses in the space sector.  This research involves utilising secondary data and quasi-experimental methods (such as Propensity Score Matching and Synthetic Control) to attempt to quantify whether and in what circumstances space-related grants improve business survival and growth. This is challenging for several reasons including time-lags, data availability, and small sample sizes, and the team are aiming to address these challenges by using innovative methods such as machine learning for propensity score estimation and linking a wide range of administrative data sources. Findings and recommendations will feed into evaluation plans.

Designing robust evaluations of complex programmes

The Open Networks Programme (ONP) programme evaluation

The ONP is a complex programme consisting of a diverse set of R&D related projects, testing facilities, and supporting initiatives, which aim to bring Open Radio Access Networks (Open RAN) technology to market more quickly and effectively. DSIT commissioned an Initial Evaluation of ONP in April 2023 (the first of 3 planned evaluations) to assess how well the programme was delivered and what outcomes have been or are on course to be achieved. It also included a light-touch economic evaluation. This study was conducted before the completion of ONP programmes, so that learnings can be acted upon, and short-term outcomes can be reported while the programme is in-flight.

AI Safety Institute (AISI) process and Impact evaluation

The AI Safety Institute is the first state-backed organisation dedicated to ensuring advanced AI is safe and beneficial. AISI’s mission is to equip governments with an empirical understanding of the safety of advanced AI systems. This involves conducting research and building infrastructure to test the safety of advanced AI and to measure its impacts on people and society. They are also working with the wider research community, AI developers and other governments to affect how AI is developed and to shape global policymaking on this issue.

DSIT has commissioned an initial impact and process evaluation of AISI, to deliver findings by early 2026. This will focus on how well the organisation has matured and the initial impact that AISI is having on global AI safety, e.g. through conducting advanced research, supporting a new scientific field, and getting international consensus on how best to respond to AI safety issues. This will be a complex evaluation as AI safety is a nascent field and as such, AISI is rapidly moving organisation.

The initial evaluation covers the first 2 years of AISI’s operations, so that learnings can be acted on while AISI is still developing.

Annex B: Evaluation community

In March 2024, DSIT set up the evaluation community with ALBs joining in May 2024. The evaluation community promotes the value of evaluation with the purpose of: 

• Facilitating peer support and cultivating a thriving evaluation culture by reducing silos and encouraging collaboration between evaluators from across our member organisations and teams. 

• Strengthening evaluation practice through regular teach-ins, discussions and deep dives that showcase best practice and offer an opportunity to discuss methodological challenges. So far in 2024, the Community have delivered multiple sessions on a range of subjects, including Randomised Control Trials (RCTs), Spending Review 25 evaluation requirements, evaluation synthesis, approaches to R&D evaluation and qualitative VfM evaluations. 

The evaluation community has a Senior Civil Service (SCS) sponsor and works closely with CARE to ensure alignment with the strategic evaluation needs of the department. Externally, the Community maintains a link with other evaluation groups such as UK Evaluation Society and the ETF to keep abreast of new and emerging evaluation methods.

1. Evaluation Task Force (ETF) – 2025 Spending Review evidence and evaluation guidance. For a high-cost programme, the proportion of spend allocated to evaluation would be expected to be lower. ↩

2. The Investment Committee is chaired by DSIT’s Permanent Secretary. Reviews any business cases for investments over £40 million. ↩