Policy paper

DHSC: evaluation strategy

Published 30 June 2022

Applies to England

Introduction

This evaluation strategy sets out the Department of Health and Social Care’s (DHSC) vision for high-quality evaluation of its policies, programmes and projects. It reflects the department’s strong commitment to developing a robust evidence base for future decision making, continued learning and to achieve its core objectives.

Evaluation helps deliver better outcomes. Undertaking robust and proportionate evaluations of our policies helps us learn about what works well, for whom and why, what does not work and value for money so that good practice can be replicated in the future. See the Magenta book for more information. This builds a stronger evidence base for future policy and spending decisions.

This strategy summarises the core objectives of the department, outlines the importance and value of evaluation in our everyday business, and sets out our vision for evaluation within this context. It also describes how we will implement our vision; illustrating how this will be embedded and enhance our evaluation activity, culture, capacity and capability.

DHSC strategic objectives

DHSC supports its ministers in leading the nation’s health and care system. Our departmental vision is to enable everyone to live more independent, healthier lives for longer by supporting healthy behaviours, improving our health and care system and creating healthy environments.

This is achieved through our 5 departmental objectives:

• improve healthcare outcomes by providing high-quality, integrated and sustainable care at the right time in the right place, by tackling the electives backlog and by improving infrastructure and transforming technology
• protect the public’s health through the health and social care system’s response to coronavirus (COVID-19)
• improve healthcare outcomes through a well-supported workforce
• improve, protect and level up the nation’s health, including reducing health disparities
• improve social care outcomes through an affordable, high quality and sustainable adult social care system

Importance of evaluation in DHSC

The importance of good evaluation is recognised across government and within DHSC and our arm’s length bodies. As the Magenta evaluation guide book states, evaluation plays a key role in policy design, development and delivery. It is important to understand and learn from our work and to ensure our accountability.

This is furthered through our recognition and reinforcement on the importance of evidence, analysis and research at every stage of the policy process. In the United Kingdom and worldwide, there is a wealth of research and evidence constantly being collated across and within the medical, health and social care settings and systems. As a department, we continually make use of this valuable research and evidence from a wide range of sources to inform our work and our thinking.

Across the department, the Magenta book and the accompanying Green book are promoted and underpin our approach to evaluation and policy development.

Evaluation evidence in the department has, and will continue to, be used to achieve various goals.

To demonstrate the impacts of large-scale projects and policies to provide learning, transparency and accountability

For example, we are developing a major programme of monitoring and evaluation to identify the short and long-term impacts of Adult Social Care Charging Reform. This will include robust process and impact evaluation, and cover impacts on local authorities and providers implementing the reforms as well as the impacts on users and carers.

Evidence from the evaluation will support future policy development and implementation and will inform the Secretary of State’s 5-yearly review of changes to social care charging rules.

This review will have regard to the financial burden of the state, local authorities, adults with needs for care and support and trends in healthy life expectancy: the evaluation programme will be crucial in demonstrating these impacts.

To test our approaches

For example, the overall aim of the phase one early evaluation of the Children and Young People’s Mental Health Trailblazer programme is to examine the development, implementation and early progress of the trailblazer programme.

The focus of the early evaluation has been to understand how Mental Health Support Teams in the trailblazer sites are implementing the programme and what activities are being developed, along with the perceived benefits and outcomes.

It will explore how service delivery models and implementation strategies differ across trailblazer areas, highlighting the factors that are inhibiting or promoting progress towards programme goals and drawing out the practical implications of the findings for the development of the programme and the design of the longer term summative and potential economic evaluation. The interim report providing early findings was published in October 2021.

To explore the effect of our policies and projects on target populations and broader groups to support our work on addressing health disparities

For example, embedding employment advisers in Improving Access to Psychological Therapies (IAPT) was piloted to provide tailored employment support to people with depression and anxiety who are currently accessing free cognitive behavioural therapies through the NHS.

A commissioned evaluation including a qualitative process evaluation; a mixed methods client research report; an impact evaluation which used IAPT data linked with administrative data held by the Department for Work and Pensions, was used to better understand the effect of the policy and to support the further roll out of the pilot.

Broadly, the findings demonstrated that the intervention was well-received by clients, therapists and employability partners and filled important gaps in local provision, as well as providing insights into further refinements ahead of future roll outs.

To improve the delivery of policies, programmes and intervention activities

For example, as part of our work to enhance the nursing workforce, we are evaluating interventions which encompass the 50,000 nurses programme. The evaluation will look to understand how domestic recruitment, international recruitment and retention interventions have worked and how recruitment and retention of nurses can be improved.

This process evaluation will use a range of methods including literature reviews, surveys, qualitative data collection such as interviews and focus groups, case studies, as well as analysis of administrative data sets to understand if and how various interventions have supported the recruitment and retention of nurses. Findings (and interim findings) from the evaluation will be valuable to our understanding of the workforce and to support future policy development.

Using the evaluation of the Oliver McGowan Mandatory Learning Disability and Autism Training trial to inform how we deliver the training

Led by the National Development Team for inclusion (on behalf of the department, Health Education England, and Skills for Care) the evaluation considers a range of forms of training with different packages trialled by partners.

The training will provide health and social care staff with the skills and knowledge to understand the needs of people with learning disabilities and/or autism. The evaluation will explore options and recommend a standardised training package for effective rollout as mandatory training.

To enhance the evidence base for future policy development

For example, our evaluation of adult social care system reform will include detailed evaluation of individual policies as well as monitoring the success of the overall reform programme. Within our overarching evaluation framework, there will be a specific focus on generating learning about what works in order to inform future policy development.

Several of our proposals have been established explicitly to test new and innovative ways of delivering support, and findings from the evaluation of these proposals represent a significant opportunity to strengthen our evidence base for the future.

To refine impact and appraisal assessments and forecasts

For example, over recent years a range of legislation has been introduced to discourage young people from taking up smoking or vaping, to encourage existing smokers to quit and to protect others from the harmful effects of cigarette smoke. DHSC has conducted post-implementation reviews to assess how legislation, such as smoking and tobacco-related regulations, have been implemented and make necessary recommendations to positively impact health outcomes.

Examples of recent post-implementation reviews include the review for the Tobacco and Related Products Regulations and of the Standardised Packaging of Tobacco Products Regulations (2022).

Our approaches and plans for evaluation are regularly reviewed and updated as part of our departmental Outcome Delivery Plan (ODP). ODPs are published yearly and set out our 5 priority outcomes and our strategy for achieving them. For each of the priority outcome areas there is an associated outcome evaluation plan which outlines monitoring and evaluation activity (planned and in-progress) for significant projects and programmes which underpin the priority outcome.

Our vision for evaluation

Our vision is to create conditions that allow us to continue to grow and enhance the evaluation culture within the department. This approach has multiple benefits. It strengthens our accountability, enables us to learn and grow our evidence base and, ultimately, support the department to develop and implement effective policies and decisions to achieve the government’s objectives.

There are 3 key elements that support the delivery of this vision and continue to foster the positive evaluative culture:

1. extend coverage of evaluation throughout DHSC
2. enhance analytical and evaluative capability
3. continue to deliver high-quality, independent and transparent evaluations

Implementing our vision for evaluation

Extend coverage of high-quality evaluation throughout DHSC

As stated previously, well-developed high-quality evaluation is important for our learning and accountability. Building on the evaluative culture in the department, we will continue to embed this message via thorough processes for identifying and prioritising areas for evaluation activity across DHSC, and work to develop these further where necessary.

To support this, we currently have a range of work underway and planned, which include the activities below.

Impact assessment guidance and process

We have strengthened our departmental guidance on impact assessments to emphasise the planning of evaluation from the very start of policy development.

Best practice and guidance on evaluation

We continue to promote best practice regarding evaluation across the department as well as working closely with our arm’s length bodies. We will emphasise the importance of proportionate evaluation across all areas of the department. Guidance and best practice will also draw on themes of evaluative and analytical capability detailed in the below section.

Additional check points for evaluation

We will be working with colleagues across the department to further promote and embed evaluation within the policy cycle and at key checkpoints, including through business cases, commercial practices and our Research and Development (R&D) committee.

Enhanced analytical and evaluation capability

To further embed our evaluation vision, we are continuing to prioritise and enhance our analytical and evaluation capability offer and continue to support our positive culture for evaluation across the department. To achieve this we are engaged in the following activities.

Learning and development

Governmental guidance (including the Magenta book and Green book) is supplemented by an extensive internal learning and development offer for analysts and the wider department (including policy professionals). There is current work underway to refresh our learning and development offer on analytical and evaluation topics for analysts and the wider department. Already part of our core curriculum are the topics of evaluation, logic models/theory of change and impact assessments.

Furthermore, we will continue to promote shared learning across the department including through ‘what worked, what didn’t, and why’ seminars and presentations of findings and interim findings from commissioned research.

Working together

We will continue to embed close working relationships across the department, our arm’s length bodies and external stakeholders, to ensure efficiency, best practice ways of working and sharing of expertise. This collaborative working will be important for refining and building on our evaluation approach.

We will also be looking to further extend departmental capability through greater engagement and sharing learning with other government departments and the cross-government evaluation group, as well as with independent research institutes and academic organisations.

Continued delivery of high quality and independent evaluation

DHSC invests significantly in research and development via the National Institute for Health and Care Research (NIHR). Working in partnership with the NHS, universities, local government, other research funders, patients and the public, the NIHR fund will enable and deliver world-leading health and social care research that improves people’s health and wellbeing and promotes economic growth.

Broadly, the NIHR funds research by the following process:

• funding is received from DHSC
• NIHR work with experts including patients to identify illnesses, conditions and health and care topics that need more research
• NIHR receives research proposals from researchers
• independent experts review the proposals and provide feedback to researchers
• expert committees meet to decide which proposals should be recommended for funding
• committees award funding to the best research projects that are most likely to benefit the public
• research projects are approved
• research takes place, with interim findings being shared
• research is published, for example in scientific journals and policy briefings

As part of their programme, the NIHR invests over £33 million a year through a dedicated Policy Research Programme (PRP). The PRP forms one of the main external mechanisms for DHSC to deliver robust evidence to inform policy development and implementation, including evaluation of policies.  To access this resource the department holds a R&D committee to triage and prioritise research and evaluation requests from the department and our arm’s length bodies.

The department’s commissioning research and evaluation process works as follows:

• policy and analytical leads and teams identify research needs
• through working with DHSC Science, Research and Evidence Directorate (SRE) teams, research proposals are submitted to the Research and Development Committee (R&D)

This submission happens in 2 stages.

In stage 1, the proposals are submitted for the R&D committee to consider and feedback is provided.

In stage 2, proposals are resubmitted. The committee agrees which proposals should be considered for commissioning.

SRE then work with policy and analytical teams authors to take forward actions from the committee, including commissioning options via the NIHR PRP.

Alongside the open research project calls, the NIHR PRP also, currently, includes a set of 15 policy research units (PRUs) which are mainly university-based. All units have evaluation capacity and there is a dedicated Policy Innovation and Evaluation Research Unit. The independent nature of these units ensures academic rigour and transparency in the evaluations they provide.

The contracts for the current NIHR PRUs will end in December 2023 and work is already underway in preparation for the commissioning of a new set of units to start in January 2024. This early planning will ensure coverage of the priority areas for health and social care policy and bolster evaluation capacity.

To maximise the dissemination and impact of NIHR funded research, the NIHR has adopted an open access approach to publications. Open access makes published academic research freely, immediately and permanently available online for anyone to read, share and reuse. This maximises the societal, academic and economic impact of publicly funded research, and enhances the integrity and rigour of research through greater openness and transparency.

Through the NIHR Journals Library and NIHR Open Research, all findings, including negative findings, from all NIHR funded research are made open access.

Across our portfolio of evaluation work we will continue to provide transparent accountability through timely and appropriate reporting and publication, within the department, to the centre and Parliament, and to the public.

Furthermore, to ensure the department’s approach to evaluation is transparent we will continue to publish evaluation activity and a wide range of statistics and metrics as part of our ODP monitoring.

Conclusion

Evaluation and the evidence created is of great importance to the department. Our strategy outlines our vision for expanding high-quality evaluation of policies, programmes and projects across our policy areas. Through promoting continued delivery of high quality and independent evaluation, enhanced analytical and evaluation capability, and extended coverage of high-quality evaluation throughout DHSC, we can ensure our activity is useful, effective and maximising value by embedding learning.

We acknowledge that it will take time to implement our evaluation vision. We will use our evaluation strategy as a platform for engaging with and learning from our colleagues in the department, across government and with wider stakeholders.