Policy paper

Evaluation strategy for Defra

Published 19 September 2023

Foreword

I am very pleased to introduce Defra’s first evaluation strategy, which represents a significant step in us becoming a more evidence and data-driven department.

During the last few years, Defra has established itself as an international department. Our mission is to restore and enhance the environment and nature for future generations, leaving it in a better state than we found it. Our portfolio now includes implementing the major acts of legislation on post EU reforms, like those on agriculture and environment. In doing so, we play a leading role in driving a nature-positive economy at home and abroad. Good monitoring and evaluation evidence will be essential in ensuring that we are on the right pathway to meet our ambitious priority outcomes.

As head of the government Policy Profession I am a keen supporter of testing, adapting and learning about what works when implementing policies. It is through embedding a culture of evaluative thinking into policy making that we will deliver better outcomes for our citizens and our country.

My sincere thanks to the teams involved across the department in producing the strategy.

Tamara Finkelstein

Permanent Secretary, Department for Environment, Food & Rural Affairs

Introduction

This evaluation strategy sets out Defra’s vision and overarching goals for evaluation from 2022 to 2025, aligned with the Spending Review 2021 settlement conditions. Our evaluation strategy includes:

• our evaluation approach
• how we will use evaluation findings in policy development and delivery
• our capability building activities

Our vision and overarching goals

This section summarises our vision and overarching goals for the strategy, and how it links with policy development and the delivery of our priority outcomes.  

The strategic role of evaluation in Defra 

Defra’s evaluation portfolio has grown over the last 5 years, reflecting the expansion in Defra’s areas of responsibility, including the repatriation of EU functions (including farming and fisheries) and our commitments as set out in the Environmental Improvement Plan and the Net Zero Growth Plan. Many of our policies involve cross-departmental collaboration, for example Levelling Up; and Official Development Assistance work with the Foreign, Commonwealth and Development Office.  

Our vision for evaluation consists of 3 pillars: 

1. To grow our capability through upskilling our analytical community, and policy and project delivery professionals.  

2. To embed robust and proportionate evaluation in policy design and portfolio delivery through use of evidenced theories of change and good evaluation design. 

3. To foster a culture of learning from evaluations, through regular monitoring and feedback from research and evaluation evidence. This will inform policy development and operational delivery; and ultimately have a real impact on policy making. 

 We discuss how we will achieve our vision in subsequent sections.

Defra’s purpose

Defra is responsible for improving and protecting the environment. We aim to grow a green economy and sustain thriving rural communities. We also support our world-leading food, farming and fishing industries. Our mission is to restore and enhance the environment for the next generation, leaving it in a better state than we found it.

Our priority outcomes

Our priority outcomes are to:

• improve the environment through cleaner air and water, minimised waste, and thriving plant and terrestrial and marine wildlife
• reduce greenhouse gas emissions and increase carbon storage in the agricultural, waste, peat and tree planting sectors to help deliver net zero
• reduce the likelihood and impact of flooding and coastal erosion on people, businesses, communities and the environment
• increase the sustainability, productivity and resilience of the agriculture, fishing, food and drink sectors, enhance biosecurity at the border and raise animal welfare standards

Our Outcome Delivery Plan outlines our priority outcome delivery strategies, milestones and measures of success, together with the underpinning strategic enabler activities. The theory of change and evaluation approach for each priority outcome will provide valuable evidence about progress towards achieving these outcomes over the next 3 years.

Our commitment to evidence-based policy-making

Defra is committed to evidence-based policy-making and using evidence to improve delivery. We continue drawing in expertise from inside and outside government to develop policy and to show accountability for decision-making. In relation to assurance and monitoring and evaluation, some examples include:

• Monitoring and evaluation of policies and programmes: Proportionate evaluation plans are included in our Outcome Delivery Plan and outline our evaluation approaches, as well as how we will use findings from these evaluations to inform delivery.
• Integrated Assurance and Approvals Strategy: this governs the assurance of projects and programmes, ensuring we receive independent internal assurance by subject matter experts at each gateway stage. In addition, the Internal Audit function provides assurance on projects and programmes as part of its risk management portfolio. It regularly reports to Defra’s Executive Committee and Audit and Risk Committee.
• External assurance: We work with the Infrastructure Projects Authority and HM Treasury to independently review delivery of all major projects, as well as with the Cabinet Office and HM Treasury Evaluation Task Force.
• Relationships with external experts:  Defra’s Science Advisory Council provides expert independent advice on science policy to the department. This includes its sub-committee the Social Science Expert Group, which includes evaluation expertise. Defra evaluation managers also work with their expert networks to support individual evaluations.
• Our Research and Innovation Interests publication: In this document we share our key areas of research interest to engage the external research community.

These activities provide a rounded and balanced review of performance as well as insights into how we can improve our policies. This is important for good decision making and assessment of value-for-money of our policies and programmes. Our evaluation approach is designed to meet the two main purposes for carrying out an evaluation, as described in the HM Treasury Magenta Book and Green Book: learning and accountability.

Our evaluation approach

Our evaluation approach includes a range of quantitative experimental and quasi-experimental methods, as well as theory-based approaches and meta-evaluations. These methods follow the established methodologies of the Magenta Book, as well as responding to the complex environment in which Defra sectors operate.

We will adopt a proportionate and pragmatic approach to evaluation that is tailored to the needs of each policy and programme. Defra’s sectors operate in complex settings, and work with complex environmental, social and economic systems, often simultaneously. Defra has commissioned bespoke tools (such as the Complexity Evaluation Framework and the Theory of Change toolkit) to help evaluation and policy colleagues when designing new policies and programmes in this context.

In addition to complexity, our evaluation approaches include an element of continuous learning from programmes, through ‘test and learn’ and developmental approaches. Alongside these approaches, we use rigorous impact evaluation methods, including value for money evaluations, for our flagship policies and programmes.

Use of evaluation findings

Our Outcome Delivery Plan sets out our intentions for how we will use findings from our evaluations to inform the delivery of our programmes, in line with the third pillar of our vision (foster a culture of learning). We are designing our evaluations to maximise the use of emerging findings at critical points of programme or policy delivery. In doing so, we are guided by the principles underpinning the ROAMEF cycle and HM Treasury guidance for project business cases (2018).

Defra publishes its research and evaluation reports on its Science Search portal.

Our capability-building activities

Growing our capability and fostering a culture of learning from evaluations are 2 of our vision’s pillars. The Central Analysis and Insight Directorate manages the evaluation community of practice. It provides specialist advice to teams through a range of activities. It:

• supports evaluation practitioners in Defra and its arms-length bodies to design, commission and manage evaluations
• shares good practice and promotes a culture of evaluative practice and thinking, by providing in-house training and seminars for analysts, scientists, policy professionals and project delivery colleagues
• produces bespoke evaluation tools, such as Defra’s Theory of Change toolkit, the Complexity Evaluation Framework, and the Rural Wellbeing toolkit

In addition, the evaluation community of practice disseminates learning from evaluations in Defra.

References

Defra Complexity Evaluation Framework. Published on Defra Science Search.

Defra Outcome Delivery Plan

Defra (2022): Theory of Change toolkit. Published on Defra Science Search.

HM Treasury (2018): Guide to developing the project business case (PDF) London. Crown Copyright.

Infrastructure and Projects Authority (2017): Guide for Effective Benefits Management in Major Projects. Key benefits management principles and activities for major projects

The Magenta Book

The Environmental Improvement Plan