Research and analysis

Evidence-based decision making framework used by the Regulatory Horizons Council

Published 1 December 2021

What is evidence-based decision making?

An evidence-based approach ‘helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation’ ^{[footnote 1]}. Rather than being driven by intuition, feeling or instinct, evidence-based decisions rely on the rigour of different types of data. While the quality of evidence that is used is of extremely high importance when producing recommendations for decision-makers ^{[footnote 2]}, it is also vital that the evidence-base encompasses a variety of different sources and that there is a triangulation of data.

The challenge

The Regulatory Horizons Council (RHC) is an independent expert committee tasked with identifying the implications of technological innovation and providing the government with impartial, expert advice on the regulatory reform required to support its rapid and safe introduction. To substantiate the recommendations, the RHC carries out processes to gather an evidence-base.

The work of the RHC is forward-looking, examining the regulatory opportunities or likely governance requirements resulting from new technologies. Inherent within the innovative nature of these new or developing technologies, is the likelihood for the overall data available to be limited, or more qualitative in nature, and therefore less definitive. This means that – unlike areas of work relating to long-standing industries/technologies – there may not always be an appropriate volume of data available to conduct rigorous analysis to feed into recommendations.

In some of its recent work, some stakeholders wanted to understand more about the approach to evidence gathering that underpins our recommendations. This document is, in part, a response to those stakeholder interactions and aims to make the decision-making process of the RHC more transparent.

The RHC intends to ensure that its decision-making process is based on collecting evidence that encompasses and triangulates a wide range of primary and secondary sources, and stakeholder engagement. Reliance on these types of evidence, ensures that the recommendations of the RHC are constructed fairly and hold validity. No single piece of evidence is enough for policy decisions to be made – these must be the result of a careful consideration of the best available evidence.

This document outlines the RHC framework for evidence-based decision-making. It will act as a living document; evolving and developing as the RHC engages with different areas of innovation, reflecting on its methods and approaches.

Methodology

The central outputs of the RHC are its ‘deep dive’ reports into specific areas of technological innovation and cross-cutting topics on regulation and innovation more generally, for example, our current consideration of ‘pro-innovation principles for regulation’. A deep dive is a structured problem-solving tool to provide insight into public policy problems and generate potential solutions. The concept is flexible to enable focused consideration of complex issues in a wide range of contexts and timeframes. When exploring an area of innovation, the RHC and its secretariat will employ a wide range of tools to gather evidence from a variety of different sources; ensuring that they are able to gather as many views as possible and make informed recommendations in its publications. Tools that the RHC uses or could use to gather evidence include:

• desk-based research
• interviews and meetings
• surveys
• focus groups
• roundtables and workshops
• futures techniques
• advisory panels
• working groups
• social media sourcing
• systems thinking

Evidence collected is scrutinised by policy and analytical colleagues within the RHC secretariat and members of the RHC. As outlined in the Magenta Book, ‘It is crucial that the assessment of data quality be transparent and assessed at the outset’ ^{[footnote 3]}. Primary or secondary data used as evidence will be considered against the dimensions of completeness, uniqueness, consistency, timeliness, validity, and accuracy ^{[footnote 4]}. The RHC will also consider where the data was published and who it was published by. It is, however, important to note that ‘Different data uses will need different combinations of these dimensions; there are no universal criteria for good quality data’ ^{[footnote 4]}.

The RHC uses a variety of primary and secondary techniques, where available, for collecting evidence and it also ensures that these interactions – as well as its interactions more broadly – are done with a range of stakeholders that are considered relevant sources of information to the area that is being explored. Throughout the project lifecycle for a deep dive or cross-cutting project, the RHC engages with a broad range of stakeholders, including regulators, businesses, academics, non-governmental organisations (NGOs), venture capitalists and trade associations to gather its evidence. This list is not meant to be exhaustive and illustrates examples of stakeholders we may engage. The RHC is aware of the kinds of bias that can be introduced in the evidence-gathering process, depending on the interests and values of different stakeholders. During its stakeholder engagement, the RHC aims to ensure that the outcome is not dominated by the perspectives of any one group.

The interactions with stakeholders aim to ensure that the RHC has the best evidence available and is involving concerned and interested parties in its processes. However, it is important to highlight that stakeholder engagements are a resource-intensive activity and reach a point of diminishing value once data saturation ^{[footnote 5]} has been achieved. Consequently, the RHC endeavours to not speak to every concerned group and instead, engage with a representative set of stakeholders that provides sufficient representation. As an independent expert committee, the RHC can often approach its evidence gathering from the position of an outside, independent, party and can thereby aim to encourage more frank, open testimony from the various interested stakeholders.

As common themes emerge from these engagements, the RHC complements these findings with other sources of evidence to help validate the insight that is being provided. This for example, could take the shape of an initial survey to gauge views from stakeholders and then conducting follow-up interviews to delve deeper into particular issues. Alternatively, interviews conducted by the RHC may give rise to common themes that are then explored further with supplementary desk-based research to verify any conclusions. Using multiple sources of evidence, both primary and secondary (where available), ensures that emergent themes can be verified, and findings can be fully appraised.

Additionally, from the beginning of a deep dive into a technological innovation, the RHC co-ordinates with, and builds on, the expertise of other government departments and specialist teams within the Department for Business, Energy and Industrial Strategy. Many of these departments and teams have existing evidence bases that can be used to learn about future regulatory opportunities and challenges for technological innovation. This sharing of information helps to substantiate the findings that the RHC gathers through its own data collection tools.

There are occasions when the RHC collects pieces of evidence that show conflicting findings. When this occurs, the RHC refers to the different sources of evidence in its reports, alongside an explanation as to why we have taken a particular stance in our recommendations.

This process of evidence-based decision making ensures that the recommendations of the RHC reflect a proper understanding of regulatory issues and are better equipped to contribute to the rapid and safe introduction of products, services, and business models.

If you have any questions about this document and would like to get in touch, please send an email to the following address: RegulatoryHorizonsCouncil@beis.gov.uk.

1. Davies, P.T. (2004). Is Evidence-Based Government Possible? ↩

2. HM Treasury (2020). The Green Book ↩

3. HM Treasury (2020). Magenta Book: Central Government guidance on evaluation ↩

4. GOV.UK (2021). What is data quality? ↩ ↩²

5. Data saturation is the point within qualitative research whereby the collection of new data produces little to no new changes in findings. ↩