Research and analysis

Regulators' Pioneer Fund round 3: interim evaluation report

Published 22 May 2025

Acknowledgements

We like to thank project staff for sharing their valuable insights into the study. We would like to also thank the team at the Technology and Innovative Regulation Directorate (TIR) supporting the evaluation for their helpful steer through the first year. Finally, thanks also goes out to NatCen colleagues Joy Mhonda, Joanne McLean, Joe Killick and Thomas Freegard for their input at key stages of this study.

Executive summary

This report presents interim evaluation insights for the third round of the Regulators’ Pioneer Fund (RPF 3 or the Fund). This includes learning from the delivery of the Fund and the projects, as well as the outcomes and impacts of the Fund so far. These insights were gathered from 47 pieces of qualitative data collection encounters, including interviews with project leads on the nine short-term projects and the Fund staff supporting them. Interviews were also conducted with project leads from the first round of the RPF (RPF 1), regulatory authorities that did not apply for RPF 3, as well as those that applied and were not selected.

The outcome and impact insights are framed by a Theory of Change (ToC) developed with the Technology and Innovative Regulation Directorate (TIR) and informed by learning from previous RPF rounds. The ToC outlined how the RPF is expected to work and its anticipated outcomes and impacts, which can be grouped into three categories:

(1) developing a culture of innovation with regulatory authorities involved in the programme;
(2) promoting wider shared learning and partnership work among stakeholders; and
(3) generating innovations – bringing in new, innovative products, services and processes that benefit consumers and businesses.

What can be learned about the process of applying for, setting up and delivering the fund and innovative projects?

In contextualising fund entry, set-up and delivery insights, it is important to note that the RPF design and operational processes have been iteratively shaped by the feedback and evaluation learning since the first round in 2018. This learning led to key changes, such as the RPF allowing for projects of different lengths and extending the application time period.

In addition, although project leads made several set-up and delivery suggestions, these should be viewed in relation to the wider context of governmental processes and resourcing within which the RPF operates. There are some components of the programme which cannot be changed due to certain constraints, such as financial year expenditure and payment to projects in arrears.

Applicants observed improvements in the administration of the RPF 3 application process from the previous rounds:

• The pre-application support given by TIR, such as virtual ‘Question and Answer’ sessions provided a better understanding of what the application process involved.

• The longer time period to complete the application allowed applicants to gain input from a broader range of stakeholders.

However, new and unselected applicants identified aspects of the application process that may benefit from review:

• New applicants would have valued their ideas being pre-assessed by the Fund to see whether their innovation was appropriate and sufficiently developed before submitting their application.

• Unselected applicants would have welcomed more feedback and transparency on how their application was scored to give them confidence to apply again and/or inform the future development of their innovation.

Drawing on learning from previous RPF rounds, the Fund had extended short term projects from six months to eight months and provided them with the option to apply for pre-funding to help with the set-up period. However, those new to the RPF still found the 4–5-week period between being awarded funding and starting delivery too narrow a timeframe to recruit staff and establish partnerships to support delivery. In contrast, those who had participated in previous rounds were able to draw from their experiences and undertook some of the preparatory work prior to being awarded funding.

It is worth noting that having a shorter time frame for projects also worked well for the previous RPF 2 round.

Project leads emphasised the importance of support provided by the Fund in enabling them to deliver within the 8-month period:

• They felt that the regular catchups between Fund staff and projects leads was particularly important in helping to identify and resolve delivery risks early.

• The flexibility of the Fund in addressing challenges, such as being open to projects reviewing and revising their project goals, helped to deliver their projects on time.

Timely delivery was also supported by the ability of projects to manage contractors. Examples of good project management practice included effective liaison practices, such as having a single point of contact between projects and contractors, especially when innovations involved several regulatory authorities. Where projects experienced challenges with contractors, they reported:

• Contractors not providing regular progress updates.
• Delivery of poor-quality work.
• Not delivering at all.

Did the RPF achieve it’s set outcomes and impacts?

Project leads highlighted that the RPF was the exclusive reason why observed outcomes were achieved. As well as making their projects possible, the Fund provided the resources, such as staff time, to prioritise their innovations and a low-risk environment to innovate. Key to this was the Fund’s flexibility in allowing projects to learn and fail as they deliver, rather than being too narrowly tied to stated project outcomes.

The observed outcomes reflected those anticipated by the ToC, particularly around developing a culture of change within regulatory authorities and engaging stakeholders, such as innovators and other regulators. RPF had changed culture in three ways:

• It encouraged the immediate project team or the wider organisation to see supporting innovation as a part of their role.
• It helped to develop an organisational infrastructure to support innovation work.
• It supported the development of new approaches to develop and test innovations.

The RPF led to closer working relationships between regulators and innovators:

• There is increased partnership work and collaboration between regulatory authorities which has improved their practices and facilitated innovation.
• There are stronger relationships between regulatory authorities and innovators due to a greater understanding of their roles and needs.

There is limited evidence of new innovative products, services, and processes. It is important to note that innovation takes time to develop, and it is likely that not enough time has passed for these outcomes to materialise for the RPF 3 projects. In addition, we only spoke to five RPF 1 projects about their sustainable impacts.

An outcome not originally anticipated by the ToC was how RPF projects improved the UK’s profile abroad. Project leads felt that their innovations helped investors and innovators to see the UK as a place where there is a clear, authoritative, and streamlined process to assess innovations. This was particularly the case where innovations introduced guiding principles and benchmarks for industry, which reduced investor uncertainty to develop and commercialise products.

Although it was too soon to understand RPF 3 project impacts, RPF 1 project leads indicated that they had achieved sustainable innovation impacts; in particular:

• The long-term legacy of the RPF in sustaining the culture of innovation within regulatory authorities.
• Promoting partnership work with innovators, regulators, and other stakeholders.

1. Introduction

This interim report provides insights largely from the evaluation of the nine short-term projects for the third round of the Regulators’ Pioneer Fund (RPF or the Fund). It presents insights on the implementation of the Fund and the projects it funds, and an evaluation of whether anticipated outcomes and impacts were achieved and the extent to which the Fund contributed to these.

Background to the RPF

Policy context

Since 2021, the UK has faced notable changes, challenges and opportunities. It has exited the EU and is recovering from a global pandemic which has impacted society and the economy, public services, research and private industry. Against this backdrop, the UK government has placed a strong emphasis on innovation as the path to social and economic prosperity.

The 2017 Industrial Strategy for the economy was introduced by the previous government and aimed to boost productivity and earning power in the UK. It was later transitioned into a Plan for Growth, which added progression in existing and emerging industries to the previous government’s priorities. Investing in innovation was at the heart of the Plan for Growth.

Innovation-friendly regulation can play a critical role in encouraging and translating R&D investment into real-world, marketable products and services that could, in the future, be scaled across domestic and international markets to the advantage of the UK economy. The 2019 report, Regulation for the Fourth Industrial Revolution, was commissioned by the previous government and set out a plan to maintain the UK’s world-leading regulatory system in this period of rapid technological change.

The Technology and Innovative Regulation Directorate (TIR), part of the Department for Science, Innovation and Technology (DSIT) has and continues to support the Department for Business and Trade (DBT) in their efforts to reform the Better Regulation Framework, ensuring that the UK has a regulatory system which is pro-innovation while keeping consumers safe.

In 2022, the previous government announced a Pro-innovation Regulation of Technologies Review led by the Government’s Chief Scientific Advisor to advise how the UK can better regulate emerging technologies, which has led to the publication of six reports making recommendations in critical technology sectors and cross-cutting barriers to innovation.

In 2023, the previous government published the UK Science and Technology Framework, setting out ten key actions to make the UK a Science and Technology superpower by 2030, which included encouraging a regulatory environment that supports innovation.

The RPF

Within this policy context, the RPF initiative was set up by the Better Regulation Executive (BRE) in the Department for Business, Energy and Industrial Strategy (BEIS), and is now being delivered by TIR within DSIT, to explore and promote novel and experimental regulatory practices that give businesses the confidence to invest, innovate and bring state-of-the-art products and services to the market. The intention of the RPF is to promote and create a UK regulatory environment that encourages business innovation and investment, whilst protecting citizens and the environment. It also aims to enable economic growth, boost value for consumers and foster a pro-innovation business culture.

There have so far been two rounds of the RPF. RPF 1 started in October 2018 and awarded up to £10 million to 14 regulator-led projects. RPF 2 was set up in 2021, allocating up to £3.7 million to 21 regulator and local authority (LA) led projects. The third round (RPF 3) comes at a time of great social and economic disruption caused by COVID-19, the war in Ukraine and the subsequent cost of living and energy crisis. As such, it is part of an effort to resolve major societal challenges and achieve the previous government’s priorities, such as boosting economic growth and achieving net zero. It allocated up to £12 million across 15 long-term (12-18 months) and 9 short-term (8 months) projects to regulators and LAs to help businesses bring innovative products and services to market, allocating up to £1 million per project.

It is important to note that the RPF programme design and operational processes have evolved since 2018 to improve delivery and maximise project outcomes in response to stakeholder feedback and evaluation insights. Amongst other changes, the RPF 3 programme was developed to allow for different project lengths (8 months and 12-18 months), to have the option of pre-funding for recruitment activities, and to provide a longer application window.

In addition, it is also important to note that the RPF operates within the context of wider governmental processes and resourcing, which means that some components of the programme could not be changed due to certain constraints, such as financial year expenditure and payment to projects in arrears.

Short-term projects

The nine short-term RPF 3 projects involved seven regulators and two LAs (referred to as regulatory authorities). The projects represented a diverse range of sectors, including transport, health, environment, data and information, energy, finance, legal, health and safety, food safety and housing.

Summary of short-term projects.

Care Quality Commission

Piloting ways to support innovative practices in health and social care.

Environment Agency

Digital simulation of Humber industrial cluster for faster deployment of decarbonisation technologies.

Greater London Authority

Roadmap for market based approaches to surface water management in urban areas.

Health and Safety Executive

Delivery of regulatory sandbox for technological innovations in the construction sector.

Information Commissioner’s Office

Developing and testing a multiagency advice service for digital innovators.

Medicines and Healthcare products Regulatory Agency

Developing principles for safe use of AI in clinical pathways to support decision making.

National Institute for Biological Standards and Control

Developing guidance for regulatory approach in microbiomes.

Office for Nuclear Regulation

Piloting a regulatory sandbox on AI in the nuclear sector.

Wakefield Council

Developing an interactive tool for precautionary allergen labelling risk analysis.

Projects can be grouped into three broad types of innovation: advice provision, proof of concept development and curation and dissemination of good practice.

• ‘Advice provision’ refers to projects that have developed guidance for both regulators and innovators that allow them to regulate or develop innovative products and services. These projects also helped develop or deliver advice for businesses to support them to navigate uncertainties around regulatory compliance and how different regulatory requirements interact;

• ‘Proof of concept’ projects were concerned with testing new innovative products or frameworks within their regulatory authority to understand whether they can be implemented in real-life; and

• Finally, projects that focused on ‘curating and disseminating good practice’ were involved in capturing and disseminating examples of high-quality innovation and sharing learning with their sector.

Projects used sandboxing, which involves building and testing innovation concepts and solutions in a controlled, simulated scenario without the risks associated with real-world experimentation. Whilst sandboxing has many applications, we refer to regulatory sandboxing in this report which is used to test and explore the application of regulatory innovative practices specifically. The sandboxing allows for the virtual testing of new technologies; or through workshops with industry experts to explore how new technologies will apply to regulatory operations.

Evaluation approach

Aims and objectives

In 2022, the National Centre for Social Research (NatCen) was commissioned by BEIS to carry out a three-year evaluation of the RPF 3 fund. The evaluation is now managed by DSIT following government departmental changes. The evaluation assesses the Fund’s success in supporting regulators and LAs to adopt new and experimental approaches to regulation, which can support businesses to innovate. The evaluation is guided by three research questions:

What impact has the £12 million Fund had to date? What impact has the RPF programme had, including RPF 1 and RPF 2, and the long-term outcomes of RPF 3 against the priorities of the government when the fund was launched such as place based innovation, reducing the cost of living and achieving net zero?

• What can we learn about the process of delivering the fund and innovative projects?
• What can government and wider stakeholders learn from this fund?

These research questions inform the four evaluation aims and objectives:

• Assessing whether RPF 3 has delivered against its intended goals – exploring the range of outcomes and impacts achieved, for whom and why. As well as assessing the impacts of RPF 3 projects, the evaluation also focused on the longer-term impacts achieved by projects funded in previous rounds;
• Eliciting and understanding the lessons learned from delivering innovative projects –building on lessons from the previous RPF evaluations to better understand what works well and the challenges in managing and delivering the Fund and its projects. The evaluation focused on lessons learned at three levels:
  • Fund entry: exploring the ways in which the RPF application process supports or is challenging for applicants;
  • Fund support: exploring how RPF is delivered by the TIR innovation team; and
  • Fund delivery and outcomes: mapping and understanding project delivery and outcomes at both baseline and at project completion.
• Disseminating key lessons across government and beyond – for example, lessons around what worked in supporting and accelerating innovation and learning from projects that have not delivered as intended; and
• Effectively monitor the projects to manage risks and maximise outcomes – risks and mitigation plans formed part of the monthly monitoring information shared by RPF 3 projects, which were used to systematically record anticipated and new risks, as well as track their resolutions.

RPF 3 Theory of Change

The research team built on the initial Theory of Change (ToC) developed by TIR and other stakeholders. The ToC was informed by documented learning from previous RPF rounds and scoping activities with Fund staff undertaken by the research team to refine it further. The ToC outlines how the RPF is expected to work across all rounds, covering the range of funded projects and how they are envisaged to lead to the expected results of the RPF (see Figure 2).

RPF’s ToC sets out expected impacts, to emerge after three rounds of funding. It groups the key short-and-medium-term programme impacts into three categories which were developed through the initial scoping work with Fund staff, and explain the different ways in which the RPF aims to contribute to change:

• Developing a culture of innovation within regulators and LAs involved in the programme;
• Promoting wider shared learning and partnership work among stakeholders; and
• Generating innovation - bringing in new, innovative products, services and processes to benefit consumers and businesses.

Across these categories, key RPF outcomes and impacts focus on developing the commitment and capacity of organisations to support innovation. They focus on achieving joined-up working across a range of stakeholders to support better understanding of innovation needs and share good practice, as well as to develop consumer and business confidence in the regulatory frameworks governing innovations.

A few minor revisions were made to the ToC to incorporate findings/outcomes from the 8-month projects which weren’t anticipated at the outset of the RPF 3 programme.

Revised RPF 3 theory of change

Contribution Analysis approach

A Contribution Analysis (CA) approach was selected to address the key impact questions, which focus on the Fund’s impact and how far it can be evidenced that the Fund contributed to these impacts relative to other potential factors. CA provides a framework for making causal inferences and understanding what the mechanisms for change are^{[footnote 1]}. Drawing on this approach, the Fund’s contribution can be assessed through testing the ToC against what has been observed and assessing the influence of other contextual factors, barriers, and enablers. Pathways of causation through the ToC are illustrated using contribution statements, allowing them to be tested against evidence.

The contribution statements for the RPF programme at this point are:

1. Outcome: RPF creates a stronger culture of innovation within regulatory authorities or parts of it. Regulatory authorities or specific teams within it have a proactive approach to tackling regulatory issues and greater organisational capacity to do so. Input: The RPF provides resource to identify gaps in regulatory guidance and develop innovative solutions in an experimental way, where typical funding models do not allow for this type of risk. Process: Projects demonstrate new ways of working and upskill staff to innovate, which leads to leadership and teams within the regulatory authority feeling more confident and committed to applying new methods of solving regulatory matters.  Organisation infrastructure and processes are established or expanded to account for these new commitments.

2. Outcome: RPF leads to a better understanding of the relationship between innovation and regulation, including how new technologies and systems impact their sector and how to address their regulatory needs. Input: RPF projects allow for new ways of working on regulatory issues and for the authority to research solutions to underexplored areas in their field. Process: Organisational knowledge is generated from the ‘learning-by-doing’ process during project activities, hiring new staff and engaging with other regulatory authorities throughout the project lifecycle as they share information about their experiences in these areas. These learnings are recorded by the regulatory authority so they can be applied into the future.

3. Outcome: The RPF encourages regulatory authorities to work together to drive regulatory and commercial innovation, including those abroad. These activities demonstrate the UK’s role as a thought-leader on certain topics at the global level. Input: RPF provides the opportunity and forum for regulatory authorities to work together to provide solutions to common problems. Process: Projects bring together people and organisations over the project lifecycle to share information, knowledge and ideas. These engagements can lead to further collaboration, including combining resources to work together on a solution and aligning practices across multiple organisations. Engagement with regulatory authorities from abroad allows UK regulators to influence and provide leadership on global regulatory policy by showcasing newly developed regulatory solutions and sharing their expertise around these.

4. Outcome: The RPF enables closer working relationships between regulators and innovators. Each has greater confidence in the other’s ability to work on developing safe innovations and trust that they will be supported in doing so, creating a more efficient regulatory process. Input: The projects funded by programme create an environment where both parties work together towards a common objective, increasing their exposure to each other and creating a more balanced and collaborative dynamic. Process: This way of working allows innovators and regulatory authorities to have more extensive and openness in their communication, enabling greater two-way understanding of each other’s needs and remit. This allows for consideration of requests earlier on in the innovation development process and for regulatory authorities to provide more timely and targeted support to innovators.

5. Outcome: The RPF programme creates an improved regulatory environment for innovation stakeholders: helping facilitate more innovation and economic activity, increase adoption and diffusion, and ensuring safety in UK markets. Input: RPF projects create a stronger culture of innovation within regulatory authorities and increasing collaboration between other regulatory authorities and innovators. Process: By ensuring greater coordination throughout the innovation process between regulatory authorities and innovators, new products, services and processes that RPF projects focus on are likely to face fewer barriers before going to market. This reduction in uncertainty helps two-fold by increasing investor confidence in financing and commercialising innovations, and helping increase end user confidence in adopting them knowing that they are safe. A more coordinated regulatory environment also reduces the costs for businesses to meet the requirements to operate, giving them more resources for enterprise activities.

At this interim stage, the report focuses on the ToC, examining the evidence to understand whether it captures the Funds outcomes and impacts in full. This will help identify the strengths and weaknesses of the current understanding of the RPF, informing where further data collection may be needed.

This initial focus on the ToC is an integral first step to the Contribution Analysis methodology, as it builds on our early understanding of how the Fund works, which forms the basis of the contribution statements. This evidence informed a review and refinement of the statements: clarifying the terminology used, expanding coverage of the statements across the different types of projects, reflecting additional insights from the data and expanding on the casual links that lead to outcomes and their sequencing. The preliminary results suggest that the statements now capture the causal pathways well.

Qualitative data collection

The qualitative data collection addressed both the process, outcome and impact evaluation questions and involved 38 data collection encounters in total, with a majority of these involving interviews with staff leading on the RPF 3 8-month projects (‘project leads’) and Fund staff. In addition, interviews were also conducted with staff from regulatory bodies that were not taking part in the RPF 3 to understand their reasons for not applying and barriers to Fund entry.  

The data collection encounters were grouped according to whether they focused on experiences of Fund entry, support, delivery and outcomes or the experience of previous RPF rounds. Table 1 below provides an overview of the qualitative fieldwork against these categories, which includes qualitative monitoring data that projects submitted monthly. A full methodological summary for each data collection encounter is provided in Appendix B.

Table 1: Data collection groups

Focus	Coverage	Participant groups	Approach	Number of data collections
Fund entry	Understanding how the RPF eligibility and application process may prevent some regulatory authorities from applying or succeeding in their applications, and what can be done to better facilitate access	Unselected applicants and non-applicants	Interviews	6
Fund support	Understanding fund staff experiences delivering and managing the Fund, including supporting projects	TIR strategic and operational staff	Interviews	1
Fund delivery and outcomes	Understanding project delivery experiences, outcomes, and the role of the RPF 3 and other explanations for achieving intended outcomes	All short-term RPF 3 project leads at both the start (‘baseline’) and end of projects (‘endline’)	Interviews	18
Fund delivery and outcomes	Understanding project delivery experiences, outcomes, and the role of the RPF 3 and other explanations for achieving intended outcomes	All short-term RPF 3 project completed monthly monitoring progress reports to provide an early understanding of outcomes, achievements, learning, as well as risk and mitigation measures	Monitoring data	8 reports for each project – collected monthly
Fund delivery and outcomes	Understanding project delivery experiences, outcomes, and the role of the RPF 3 and other explanations for achieving intended outcomes	All short-term RPF 3 project end of project reports highlighting their key activities, outcomes and lessons learned	End of project reports	9
Previous RPF stages	Understanding longer term and sustained impacts from projects involved in previous rounds of the RPF	RPF 1 projects	Interviews	5

Interpreting insights

A key strength of the study approach was that it was guided by a well-developed ToC and sought to test key contribution statements that are anchored to it, without the qualitative work being restricted to these. Furthermore, the approach provides a three-sixty view of the Fund delivery, outcomes, and impacts by including the views of a range of participants, including Fund staff, projects leads and previous projects.

Although the report includes evidence on long-term impacts from RPF 1 projects, this needs to be read with caution for three reasons. Firstly, although the research aimed to speak to a range of RPF 1 projects, there were limitations in the number of RPF 1 projects the study could speak to. Secondly, beneficiaries such as innovators and businesses are not represented in the qualitative data collections but will be included in the next stages of the study. Thirdly, the focus is on RPF 1 project impacts as the longer-term impacts for RPF 3 projects have not yet materialised, given that project lead endline interviews happened shortly after projects were completed.

Finally, as mentioned earlier, the interim insights should be seen within the context of how the RPF programme has evolved over time in response to stakeholder feedback and previous evaluation insights, as well as the wider governmental context and constraints it operates in.

Report overview

The report is structured around the process, outcome and impact learning from the evaluation:

• Chapter 2 and Chapter 3 focus on the process evaluation questions by exploring regulatory authorities’ experiences of applying for, setting up and delivering projects.
• Chapter 4, Chapter 5 and Chapter 6 focus on insights around the outcomes and impacts of the Fund, as well as the extent to which the Fund contributed to these.
• Chapter 7 brings together the key conclusions and recommendations from the report.

2. Fund entry and set-up

This chapter explores why regulatory authorities applied for RPF funding, their experience of the application process, and the set-up time between the award and commencement of the projects. This chapter draws on interviews with unselected and non-applicants, TIR strategic and operational staff, and both baseline and endline interviews with RPF3 projects. As will be discussed, having previous RPF experience made the application and set-up stages easier for project leads.

Application experiences

The application journey involved three stages: the pre-application process, application process, and application outcomes. These stages will be discussed further with both contextual information and key learnings.

Timeline for the RPF competition

Stage	When
Pre-application process
Application opened	21 July 2022
Application closed	29 September 2022
Applicants notified	November 2022
Projects start	January 2023

Pre-application process

There were three main motivations for why regulatory authorities applied for RPF3 funding: the uniqueness of the Fund, previous RPF experience and the short timeframe. Projects were at different stages in their innovation development, with some applying to the Fund to develop an idea and others to implement it. Projects at the early stages of development particularly welcomed the Fund as a unique opportunity to experiment and refine their ideas. As will be discussed in Chapter 5, project leads indicated that there was no comparable funding that provided resources to focus solely on innovation work.

Having RPF experience encouraged project leads to apply, particularly if their previous outcomes were successful. They felt comfortable with the application process as they were familiar with what it involved and were positive about their chances of being awarded again.

Project leads that had not previously applied for the RPF applied for this round because they had heard about successful RPF1 and RPF2 projects and the innovations these projects had delivered.

The focused 8-month delivery period also motivated project leads to apply. They saw the short timeframe as helpful in accelerating the development of innovations that were in the early stages of conceptualisation, providing focus and direction to their work. The restricted timeframe was also seen to optimise the development of certain types of innovations, such as sandboxes, as well as reduce the risk to the project of staff turnover and stakeholder fatigue (see Chapter 3).

In the months leading up to the application process, TIR provided various forms of pre-application support for applicants:

• Question and Answer (Q&A) sessions and Frequently Asked Questions (FAQs) – TIR ran two virtual sessions between August and September 2022 for any potential applicants, where TIR addressed any questions. They also circulated a FAQs document;
• Stakeholder events – there were several events with past RPF recipients and potential applicants before the RPF3 competition where they were able to discuss options for the new round;
• MS Teams chat – Microsoft Teams chat for potential applicants to collaborate with each other; and
• Regulators Innovation Network – the TIR used this as a way of engaging regulators about the RPF3 via a presentation and sharing information via the email list.

TIR also provided other forms of pre-application support such as responding to queries through email and telephone, offering an online networking session, and sharing information about the new round via the Regulators Forum contact list.

Project leads valued all the pre-application support as it allowed them to ask questions and strengthen their application. The Q&A sessions were particularly useful as they explained the application process and provided an opportunity for project leads to raise any questions to Fund staff. By giving applicants the opportunity to hear questions from other applicants, the sessions contributed to some project leads considering ideas and concepts they might not have otherwise.  

It’s helpful because partly you can dial in and you can listen to other people’s questions, too, so even if you haven’t got that far through your application process, it prompts some of the things to think about (Curating and Disseminating Good Practice project, RPF3 project lead).

Application process

Potential projects had to meet the Fund eligibility criteria before project leads could apply for funding. The RPF3 fund was looking to sponsor projects that would enable improvements in a particular sector, encourage partnerships and bring regulators together, and address one or more of the previous government’s priorities which included:

1. Helping the country move to a net zero economy;
2. Supporting place-based innovations; and
3. Reducing the cost of living and making an impact and difference to the everyday lives of people and businesses.

Project leads had from July to September to submit their full application.

RPF applicants’ experience of the application process was explored in interviews with unselected applicants and RPF3 project leads. Participants reflected on two aspects of the application process: timing and the application form. These will be discussed below, in turn.

Drawing on learning from previous RPF rounds, the Fund had extended the competition timeframe as far as possible within the context of government guidelines and financial processes.  However, project leads had mixed opinions about the length of time needed to complete the application process and the timing of the process, which ran from July to September 2022. Those with RPF experience reflected positively on the timeframe as it was longer than the last round. Positive experiences may have also been due to familiarity with the process and so needing less time to understand what it involved.

Other project leads found the timeframe shorter than ideal to get buy-in from key partners and stakeholders that needed to be involved in the application process. The timeframe was also perceived as insufficient for project leads to draw on financial expertise to fully understand the costs and logistics of each work package before submitting their application. This issue was exacerbated by the application period taking place during the summer months, where key staff were likely to be on annual leave. This made it challenging to progress the application as there were delays getting project details agreed and signed off.

The application was split into three sections:

• Application details: This included a project summary and description whilst highlighting the regulatory functions that make them eligible for funding.
• Application questions: The second part included 6 independently scored questions, each with a 300-word limit.
• Project financial information: Projects had to provide a monthly breakdown of anticipated costs for the length of the project.

Project leads discussed both the content and the format of the form, and how it could be improved further. Those with previous RPF experience were generally more confident about the application process and found the questions clear and straightforward in terms of knowing the rationale and information they needed to provide for their project.

I think it helped us having gone through the process recently before. The process hadn’t changed majorly. The key questions and the justifications and the case that we needed to make, we understood how that worked from the other project. The fact that we were confident was because we were successful with the other bid, we kind of knew what we needed to get across (Curating and Disseminating Good Practice project, RPF3 project lead).

However, first-time applicants found the language in the form too technical and had to ask for input from external experts in some cases. Although these applicants were not specific about which aspects they found technical, a project lead gave the example of having to consult with a company who were specialised in the aerospace field to help write the bid and answer any technical questions. Project leads suggested that the TIR provides more help, support, and reassurance for first-time applicants. One project lead suggested having mentors to provide one-to-one support for applicants, in addition to the group FAQ sessions run by the Fund:

It was a hard experience to do. I didn’t know if I was doing the right thing or giving the right information. I don’t know how I would have improved. I don’t know how I would have done things differently. If I had a mentor, I could have discussed it (Unselected applicant).

The project lead felt that a mentor would have supported them to navigate the application process, giving them a better understanding of the questions asked and what information was needed. However, this suggestion should be seen within the context of the resource and cost implications to the Fund of providing individual mentoring support and the opportunity already provided for projects to reach out to Fund staff via a dedicated RPF email.

Other suggestions from project leads included pitching or having a pre-assessment of ideas to understand whether they were on the right track prior to application. This would have made unselected applicants aware that they needed to further develop their ideas before they applied for the RPF, saving them time spent on the application process.

Project leads had mixed views towards the 300-word limit for each question. Those who were in favour of the word-limit found that it encouraged clear thinking, helping them to succinctly summarise the key delivery aspects, outcomes and aims of the projects.

The word count was good because it gave us space to say stuff, but also, we had to prioritise not going too, like, into detail, into the weeds of things. It was a good exercise to go through for ourselves about articulating the idea (Proof of Concept project, RPF3 project lead).

Despite this, some project leads found the word-count too constraining, particularly those on projects with complex innovations and activities. They felt that the restricted word-count limited their ability to fully explain their project and were consequently marked down because of this. For example, one project lead found it difficult to avoid jargon and felt they needed more words to explain a concept to those without their regulatory background.

Application outcomes

Following an assessment of submitted proposals, an awarding panel made the final decision of funding in November 2022. TIR notified unselected applicants explaining why their application was not selected including why it was out of scope if this was the case. The feedback included suggestions on what was good about their project and areas for improvement.

Those with RPF experience felt this was an improvement from previous rounds, where they had not received any feedback. They saw feedback as helpful for accountability purposes, in explaining why their application was not selected to their organisation. However, some unselected applicants, particularly those applying for the first time, felt that the feedback did not help them understand how to improve their innovation idea. These applicants would have liked further clarity about what their application was lacking and how to improve it to be selected in future rounds of funding.

…being able to then have an addendum to respond to some of that, or some of the points that maybe were not included or were lacking in the thing, just to get a bit more clarification around being able to secure that funding from that pot, if possible. How many applications? We didn’t know how many applications were made and how many were rejected, so where you kind of stand on a scoring system. That was a bit closed from what - not as transparent on that side (Unselected applicant).

As context, it should be noted that the TIR did make publicly available the list of projects selected, including a high level overview of the assessment process and information on the number of applications received. However, unselected applicants recommended having more information on the scoring system as well as knowing the other types of projects that were unselected so that they can compare their project to others. They also suggested providing feedback via discussions with Fund staff so that project leads can understand how they can improve and build on their innovation ideas.

Where projects were not awarded funding, unselected applicants said that the RPF application process was a learning process for themselves and their organisation. The application process helped them explore the potential of innovation and the benefits it could have. For example, the process made some regulatory authorities think through their innovation ideas in more detail than they would have otherwise and gave them an understanding of how to develop these further.

Unselected applicants varied in their intentions to take their innovation idea forward, demonstrating the importance of the RPF funding in promoting innovation work (further discussed in Chapter 5). Some unselected applicants were not able to continue with their innovation projects as they were unable to fund it themselves or secure funding externally and were not intending to apply for the next round of RPF funding. Others intended to improve their application and wait for the next round of RPF funding. Some projects intended to continue but either at a slower pace or by delivering their projects in chunks as funding becomes available.

Outcomes for unselected applicants

Set-up experiences

Selected projects then entered the set-up stage, which was the five-week period between being awarded RPF3 funding and the start of the projects in January 2023. This set-up stage typically involved recruitment, procurement and onboarding staff and stakeholders, so that projects could begin at the start of January. Project leads discussed their experiences at this stage in relation to the set-up timeframe and the pre-project engagement work they did.

Set-up timeframe

Drawing on learning from previous RPF rounds, the Fund had extended short term projects from six months to eight months, and also provided the option for projects to apply for pre-funding to help with the set-up period. However, some project leads found the four-to-five-week period an insufficient time to get everything set-up before January. This was exacerbated by this stage taking place over the Christmas period, when people typically take leave and hiring tends to be more difficult. Project leads spoke about the pressures of hiring people, expertise and consultants in time. They highlighted that they needed more time to form partnerships, where needed, and to recruit people into posts, which led to delays and sometimes compromised delivery (discussed in Chapter 3).

This set-up challenge led some project leads to reflect on whether they would have benefited from applying for the 12–18-month RPF project instead, as this would have given them a longer set-up leading time.

A recommendation from project leads was that future RPF funding rounds should build in more time for selected projects to procure consultative support and complete contractual elements of work, without it negatively impacting on the time available to complete the project as planned. However, this change may not be possible as RPF is working within government timeframes and the time needed by the Fund to process applications.

Pre-project engagement was helpful for set-up

Project leads who were able to effectively navigate the set-up timeframe tended to have undertaken some pre-project set up work. For example, project leads had provisional conversations with stakeholders and contractors before knowing they were awarded to ensure that if they were selected, it would be possible for them to deliver in the timeframe. This early communication was important for projects in having stakeholders onboard promptly for delivery.

Other project leads addressed the short set-up period by using existing contacts and contractors they already built a relationship with. For example, some used their call-off contracts and others set up group meetings with existing people in their organisation. Project leads said that using existing contractors or recruiting internally rather than sending out a ‘cold tender’ and going through procurement processes saved time.

Again, project leads with RPF experience tended to be aware that this upfront project work was essential to starting on time, as they had experienced challenges previously. However, they indicated that the Fund had communicated the need for projects to have resources in place prior to the start of delivery more effectively in this round than in previous ones.

I think, what a lot of projects do is they don’t start recruiting their - if they are recruiting externally, or if they need people to do the work, they don’t have them in place for day one. You have to have all of those commercial things ready to switch on immediately, and I think that was becoming more and more recognised. People were informed better of that this time around than the previous time, and that one, better than the first time (Proof of Concept project, RPF3 Project Lead).

3. Delivery experiences

This chapter explores four key delivery experiences discussed by Fund staff and RPF3 project leads in their endline interviews. These experiences focused on being able to deliver projects on time, working with contractors, aligning internal financial processes with Fund requirements and engaging stakeholders. Across these experiences, both Fund staff and project leads identified a combination of factors that supported or hindered intended delivery activities. These factors related to the Fund, nature of the innovation, project practices and the regulatory authorities’ internal organisational context.

Delivery to intention

Project delivery activities varied depending on the nature of the innovation. As noted in the ToC, these activities included generating ideas for innovation, evaluating and monitoring project activities, raising awareness of the project innovation, sharing project learning, and working with partners and engaging end-users. These activities were intended to lead to a range of project outputs including:

• Delivery on the innovations;
• Generating evidence and learning from projects;
• Raising awareness of the projects and their innovation after delivery;
• Disseminating end of project learning with other regulators and wider industry; and
• Collaboration with innovators and the production of regulatory guidelines.

Projects varied in whether they were able to deliver their activities as they originally intended (‘delivery to intention’). There were three variations to this:

Delivery to intention

Delivery was slightly compromised

The two types of compromised projects leads mentioned were having less inout from innovators into prject activities than anticipated or the scope of the project had to be narrowed down due to the project timeframe.

Delivery was slightly compromised	Delivered to intention	Delivered additional added-value activities
The two types of compromised projects leads mentioned were having less input from innovators into project activities than anticipated or the scope of the project had to be narrowed down due to the project timeframe	All activities were delivered to intention	For example, where they were able to deliver additional dissemination work involving other regulators or the sector

The recurrent view was that slight compromises in project activities did not significantly affect project outputs and, therefore, outcomes. This is because these delivery compromises were minor and projects had already completed most of the other intended activities. For example, a project lead said that they had gained the learning they needed from delivering most of their sandboxes, despite being unable to deliver one of them.

In some instances, delivery compromises helped innovations to streamline, thus enabling delivery and targeted learning. For example, a project had to limit the geographical scope of its modelling work because of insufficient resources and time to deliver what was originally intended. However, they felt that streamlining enabled them to deliver the innovation to time and, importantly, provide a more detailed proof of concept model than they would otherwise have been able to. 

It [limiting project scope] just enabled us…in the timescale we had probably created a better product than we would’ve done if we’d kept that wide scope. It would’ve been much more diluted and not had the kind of detailing, and I think by refining it we were just able to focus our attention and just go, ‘Yes, it’s not as big as we expected but this is the potential and you can build on it geographically and also in terms of different things that you look at, like…’ (Proof of Concept project, RPF3 project lead).

There were also project leads who were unsure whether delivery compromises had an impact on their innovations. This was because it was difficult to predict what the outputs would have looked like if all the activities had been delivered to plan. For example, a project looking to develop innovation advice through consulting businesses and innovators had been unable to secure input from as wide a range of businesses as they would have liked. Although this limited input did not impact on guidance development, the project lead was unsure whether input received was as representative as it could have been.

Factors affecting delivery to intention

Project leads discussed a combination of factors that affected whether activities were delivered to intention. These related to the Fund (particularly its support and conditions), the nature of their innovation, project practices and the regulatory authorities’ internal organisational context.

These factors were discussed by project leads across four key delivery experiences: completing projects to time, working relationships with contractors, engaging stakeholders and aligning internal financial processes with Fund requirements.

Delivery experiences

Delivering on time

An important funding condition for projects was to deliver their activities within the 8-month period. Project leads had mixed experiences of being able to deliver within this timeframe. They felt particularly comfortable delivering where they had been able to navigate the set-up process quickly (as discussed in Chapter 2). This enabled them to have key delivery processes, staff and partnerships in place to begin delivery as early as possible.

Project leads also reflected positively on the timeframe if it supported their innovation; as noted in Chapter 2, there were two aspects to this. Firstly, if they felt that the shorter period of 8-months benefited their specific type of innovation. For example, one view was that the sandbox approach of testing innovations benefited from a quick development cycle of 8-months, rather than 12–18-months of RPF project funding.

Actually, the nature of a sandbox is that it should be sharp and short. It’s supposed to be fail fast. It’s supposed to be testing it. That’s why we did the eight months. (Curating and Disseminating Good Practice project, RPF3 project lead).

Secondly, project leads also preferred a shorter time-period for projects if it helped to sharpen organisational and partnership focus on the innovation. This was particularly the case where they thought any longer than 8 months could have resulted in projects being affected by staff turnover or stakeholders losing interest or motivation in being involved.

Conversely, the delivery period was seen as too narrow for projects which experienced set-up delays which, in turn, led to delivery starting later and having less than 8 months to work on their innovations. Other reasons why the short delivery period was perceived to be an issue by project leads included: 

• Organisational changes, such as staff turnover and restructuring, leading to delays in being able to access internal expertise and staff time to input into the innovation;
• Each stage of innovation activities taking longer than anticipated; and
• The working practices of businesses, innovators, and other partners unable to accommodate a fast delivery pace required for an 8-month projects.

Where the 8-month period was an issue, project leads felt this compromised their delivery by limiting their project scope or the partnerships and expertise they could bring into their innovation.

The support provided by the Fund was key to helping address delivery time issues. There were three aspects to this: regular catchups between Fund staff and project leads; the willingness of the Fund to be flexible with project deadlines in exceptional circumstances; and the Fund monitoring process.

In terms of regular catchups, each project had a dedicated Fund staff member who caught-up with project leads monthly and was also available to field ad-hoc queries. Fund staff indicated that these catchups were typically 45 minutes in duration. Project leads generally found the support to be helpful and described Fund staff as being approachable and interested in their innovation work, which encouraged project leads to reach out for support.

In addition, project leads found the regular catchups helpful because the agenda was responsive to understanding and addressing the emerging issues faced by projects. Both project leads and Fund staff felt this allowed issues to be identified and resolved quickly. However, there were a few instances where project leads felt the catchups did not work as discussions were narrowly focused on project updates, rather than understanding the issues faced in delivering the innovation. One project lead reflected on their meetings not taking place at regular intervals. In these instances, project leads expected that TIR staff could have done more to connect them with other projects more consistently.   

We had monthly catchups with DSIT to kind of discuss risk and issues and all the funding and everything…Anything that came up as a blocker… it seemed quite straightforward and we generally got an answer fairly quickly so that we could then move on and carry on (Proof of Concept project, RPF3 Project lead).

I don’t think we really received any [Fund support] …We had monthly calls, but they were very much an update reporting call, and we had several instances of the person not attending…We just got on with it…The relationship was a funding transactional relationship. The support that there had been indications there might be at the beginning did not materialise (Curating and Disseminating Good Practice project, RPF3 project lead).

In this context, as mentioned in Chapter 1, it is important to note that Fund staff have iteratively developed and refined their support offer through reviewing project feedback at each successive RPF round and will continue to improve their support processes where required.

As part of this support, projects leads reflected favourably on the flexibility of the Fund in extending project timelines in exceptional circumstances, particularly where projects had experienced significant set-up delays.

Project leads also reflected positively on the RPF monitoring process. As noted in Chapter 1, each project was required to complete a short monthly monitoring form, updating the Fund on project progress, challenges, mitigation measures and learning. Project leads indicated that the form helped to keep their project on track by providing an opportunity to have regular conversations with project staff and external collaborators to understand delivery progress and challenges. Project leads also felt the form was easier to complete than in previous RPF rounds, although there was a view that some sections of the form could be further streamlined.

Finally, at a project level, strong coordination and leadership was reported by project leads as helping keep their innovation on track. For larger projects involving several regulators, project leads highlighted that project activities were kept to time by having effective ways of working together, such as project working groups bringing together expertise from each regulator, and having senior staff buy-in. This senior staff support ensured that projects had the staff and coordination needed to be delivered to time.

For smaller project teams, strong coordination involved having a clear schedule of activities or deliverables, and project staff having clear but sometimes interchangeable roles that helped to keep the project moving. Interchangeable roles were important as this reduced delivery delays as staff could work in parallel on activities and ensured that team absences did not lead to interruptions in delivery. 

Working with contractors

External contractors were critical for some projects in ensuring key activities were delivered to time and scope. Projects relied on contractors for a range of activities, including developing and testing innovations, undertaking research, as well as bringing together and consulting industry and innovation experts.

Working relationships with contractors therefore significantly affected whether projects activities were delivered to intention. Where relationships worked well, project leads felt they could trust contractors to deliver activities to time and scope for a range of reasons. These included contractors prioritising and being interested in the innovation, having a clear understanding of what needed to be delivered (signing a Memorandum of Understanding to confirm this further reassured project leads) and being able to communicate delivery progress effectively. Good working relationships were sometimes a result of regulators having worked with contractors before and therefore having a shared understanding of how they would work together effectively.

Where relationships were not working as well with contractors, project leads reported three challenges which had different levels of implication for project delivery. At the lowest level of impact, there was uncertainty around whether the contractor would deliver activities to time and scope. This uncertainty was due to contractors not regularly communicating progress to project leads. For example, a project lead described their contractor’s working and communication style as “laid back”, whilst another said they felt disorientated about the innovation’s progress because their contractor did not provide regular updates.

At a higher level of impact, project leads sometimes felt the quality of contractors’ work affected project deliverables. Contractors’ written outputs, such as innovation plans, were sometime not well thought through and therefore needed a lot more project staff time and input than anticipated. At a more significant level, a contractor tasked by a project to bring together innovation experts lacked the required industry connections to do this. As a result, this, may have affected the diversity of views that informed the innovation.

The greatest impact to delivery was when contractors did not deliver key project activities. This was the case when a contractor tasked with developing and testing an innovation failed to deliver because they lacked the organisational capacity as their firm was undergoing significant changes.

I think there were some issues [with the contractor] around the quality… there were two things. So one that springs to mind is not fully understanding what the need of the [project team] was. So these are senior decision-makers in regulators, and what their expectations would be on a presentation, and how material is presented to them. At the end of the day, you would present what decision, or what is being asked of me as a person on a board, and I think there was a lack of clarity on that, so there was a bit of work that I know [the project team] had to do. Second to that was, the analysis on some of the material that they were generating, it didn’t feel like the level of depth, perhaps, was as expected (Advice Provision project, RPF3 project lead).

Project practices to effectively manage contractors helped to mitigate some of these challenges. Practices included projects drawing on trusted contractors, project leads chasing contractors for updates, monitoring their work closely and, where regulators were working in partnership, having a single point of contact to liaise with the contractor to avoid confusion.

Engaging stakeholders

Projects worked with a range of stakeholders to help shape their innovation. Stakeholders included innovation experts, businesses and other regulators that would be affected by the innovation or guidelines being developed.

Having the right stakeholders involved from the outset was important in developing innovations and regulatory guidelines. Early involvement worked where projects had established relationships with relevant stakeholders before the projects had started or, where they were able to access them through key gatekeepers, such as external contractors. A key barrier to this was when projects lacked the time at the set-up stage to identify relevant relationships or gatekeepers, or where contractors did not have the required connections.

Engaging stakeholders to contribute meaningfully during delivery was also important in helping projects to shape their innovations. Project leads optimised input by ensuring stakeholders felt listened to and promoted open discussion at stakeholder events. For example, one project avoided any commercially sensitive discussions during workshops to inform their innovation. This allowed all participants to contribute openly without concerns of divulging sensitive information. 

A key barrier to effective engagement was the ability of stakeholders to take part and contribute to consultations and other events. A key logistical barrier to this was stakeholders having insufficient time, particularly if events were face-to-face and required a greater time commitment. Other barriers included key stakeholders, such as businesses, reluctance to shape a regulation as they did not want the associated responsibility of doing so.

…what I would usually find is that they [some external stakeholders] had nothing to say [on regulation around an innovation]. I find that generally when it comes to these kind of things, people tend to complain, but then when you give them the opportunity, they don’t have much to say. For me, it was very weird to discover that. I feel there is a fear of responsibility (Advice Provision project, RPF3 project lead).

Aligning internal financial processes with Fund requirements

As noted in the introduction, the RPF operated within wider governmental financial year constraints, which meant that the earliest the RPF3 competition could be launched was summer 2023. These constraints, coupled with evaluation insights from previous RPF rounds pointing to projects needing more time between being awarded funding and delivery, meant that short-term projects could begin delivery in January 2023 at the earliest.

Although projects were informed of this funding period in the competition brief, some projects experienced two challenges around the funding period not aligning neatly with their financial year (‘spend profile’), which tended to operate from April to March.

Firstly, project leads felt that the misalignment of financial spend profiles led to initial challenges in their organisation signing off key project activities that straddled different financial years. This is because regulatory bodies did not want to make any financial decisions that did not align with their financial calendar, leading to delays in projects starting up. This issue was compounded by the fact that RPF funding, as with most similar government grant funding, was provided in arrears monthly rather than in one go. This meant that regulatory authorities did not have the option to hold the entirety of their RPF funding separately in a large pot, which their internal processes would allow for the allocation of money across different financial years.    

It [difficulty] was moving some of the budget around between the different financial years. ‘You have to spend all that money that you’ve said for that period of time by then.’ That was done, and then we managed to negotiate that it moved… (Advice Provision project, RPF3 project lead).

Secondly, project leads had to forecast expenditure monthly, which they found challenging as their projects were delivered at a fast pace and the costs associated with them were not always “straight forward” fixed costs, such as staff costs. This led to concerns that if projects did not forecast correctly, they would not be able to carry money over between the March and April financial year end.

Because the forecasting had to then be within that month and we had to then go back to RPF to ask for permission to move some of the… surplus - which wasn’t surplus. But to make sure we didn’t lose anything between March and April. So, it just doesn’t align very nicely in terms of that flow and in terms of allowing for contingency and in terms of moving the forecasting around when it’s something that is hard to forecast for, this creative process. If it’s staff planning, if it’s capital, if it’s fixed finance, it’s much easier to forecast for. But when it’s a moving feast, that is really difficult, so that’s what I mean (Advice Provision project, RPF3 project lead).

This issue was addressed by the flexibility of the Fund, with Fund staff reviewing project requests to reprofile project spend where it had an impact on project delivery. In addition, internal project managers were able to support projects by tightly planning how the funding would be used over the 8-month period to keep project finances on track.

…we did actually manage to negotiate to carry a little bit of funding over. I can’t remember how much… I must say, RPF were supportive and were helpful through this. It wasn’t a huge barrier to us, but it was something that we had to be aware of (Demonstration project, RPF3 project lead).

… we [Fund staff] have had projects come to us to say ‘Look, because of delays, or whatever, or because things have moved on since we sent the application in, our spend profile doesn’t fit anymore. Are we able to change it for these reasons?…where possible - in fact I think in all cases - we have been able to approve that, so hopefully, that has helped (Fund staff interview).

4. Fund outcomes

This chapter describes the key outcomes resulting from the third round of the RPF. These insights are informed by the baseline and endline in-depth interviews with project leads. Project outcomes presented below reflect and nuance the three impact categories described in the ToC. These were: (i) developing a culture of innovation; (ii) promoting stakeholder engagement and collaboration; and (iii) introducing new products, services and processes.

Developing the culture of innovation

Projects funded by the RPF contributed to cultural change within regulatory authorities in three ways: authorities saw innovation as part of their role; projects helped to create infrastructure within regulatory authorities to support innovations; and contributed to new approaches towards developing and testing innovation. However, there are important caveats to consider which are discussed at the end of the sub-chapter.

Regulatory authorities saw innovation as part of their role

Regulatory authorities seeing innovation as part of their role refers to the shift in thinking of the relationship between innovation and regulation so that they can enable, rather than inhibit, each other.

The work carried out under the RPF projects has begun to challenge the view that regulatory authorities obstruct innovation and instead demonstrates how they can hold an important role in facilitating innovation. This happened in two ways: projects have shown the value of regulators proactively devising solutions to support the development of innovative technologies ahead of time; and secondly, by illustrating how regulation can increase confidence in using cutting-edge technologies, helping increase the uptake amongst intended users.

In terms of pre-empting technological development, regulators are funded to ensure innovations adhere to regulatory criteria. Project leads said that this often means regulatory authorities engage with innovations in a reactive way, responding to industry queries on a case-by-case basis. In contrast, RPF projects have demonstrated to both regulators and industry that regulation can be forward thinking, anticipating future innovation challenges and developing working solutions in advance. For example, in the ability of projects to proactively address the challenges of disruptive technologies.

The work under the RPF has also contributed to regulators re-considering their responsibilities in helping to facilitate innovations, particularly around improving their uptake. For example, the MHRA aimed to better understand the ‘black box’ process of AI decision making in clinical diagnostics, the value of which can then be communicated to doctors to increase their confidence and adoption of this technology. By understanding how technology works and is deployed, regulators can provide assurance to consumers and investors that newly developed technology is safe to implement.

We are all invested in making sure [the sector] is safe and secure. I think [the RPF] has given people the space to challenge where we’re not doing that as well as we could. I say we; I mean the industry and the [regulator]. I think that probably existed before, but was a little bit unstructured (Advice Provision project, RPF3 project lead).

Sometimes [our sector] is not brilliant at learning from what other industries are doing, and innovation allows a sort of safe space if you like to look at what [other industries] are doing in robotics, for example, and bringing that into what we do” (Advice Provision project, RPF3 project lead).

However, it is too early to determine how this shift in thinking translates into a sustained shift in organisational culture (addressed in Chapter 6, where the sustainability of impact is assessed).

Caveats

It is important to note that cultural change did not necessarily take place beyond the immediate RPF project team within organisations. However, project leads highlighted that teams within regulatory authorities had already begun to think about how to improve their approach to innovation (as demonstrated by applying for the fund), so the RPF work may be building upon this and accelerating other work being done around innovation. As one project lead stated: 

I don’t think the Regulators’ Pioneer Fund project had a massive impact on the culture. I don’t think there was an awareness within the entirety of all the staff […] on what that project was. So, we publicised it, but some people just won’t acknowledge it.

What I think [it] has had an impact is the other activities that have gone around the project like [examples given]. I think that’s slowly started to ingrain the aspect of what you can do in terms of innovation and considering things being more open-minded. Also, it’s in parallel to some other work that [the regulator] has been doing as well around behaviours and culture (Advice Provision project, RPF3 project lead).

Promoting stakeholder engagement and collaboration

Project leads highlighted that their projects had led to closer working relationships with other regulators, as well as innovators. Additionally, and unanticipated by the original programme ToC, there had been interest from international regulators and innovators, helping to raise the UK’s profile in developing regulatory solutions to new technologies. 

Improved partnership work between regulators

The RPF increased collaboration between regulatory authorities to improve practices to support innovation. The scale of partnership working, and collaboration ranged from disseminating project findings, to regulators working in partnership on projects. Improved collaboration also extended to other bodies involved with regulation, such as standards bodies and government departments.

In terms of partnership work on projects, regulatory authorities with a shared innovation interest sometimes worked together on delivering a RPF3 innovation. This led to the alignment of regulatory practices across authorities in different United Kingdom nations.  At a less involved level, project leads also reached out to other RPF projects (past and present) for advice and guidance. For example, they sought advice and guidance around the benefits and implementation of sandboxes.

Regarding partnership work with other bodies, there were examples where projects had received financial support from industry standards bodies for their RPF projects or the innovation work that followed on from these. Project leads commented on how this investment demonstrated the regulator’s credibility in the innovation space and with other organisational bodies involved with regulation.

The regulatory alignment mentioned earlier also extended to other countries – an outcome not anticipated by the original programme ToC. Engagement with international regulators on the pioneering work carried out by RPF projects sometimes helped to position these regulatory authorities and, consequently, the UK as thought leaders in this innovation space. For example, the outcomes of one of the projects resulted in the UK taking a leading role in collaboration with other countries to establish regulatory principles for a particular innovation. This also led to the revival of an old collaboration agreement with a North American regulatory body.

I think there was maybe more of a cohesiveness between regulators and [collaborators]. I think that was definitely enhanced; […] we had a shared purpose. It was a more cohesive way of working and collaborating, and even within that, there was a lot of really good information sharing and knowledge sharing across the group in the interactions, so you couldn’t not learn something when you were having those conversations. I think there was more softer learning and impact that will have gone back into each organisation around having engagement (Advice Provision project, RPF3 project lead).

Stronger relationships between regulators and innovators

The RPF led to stronger and more dynamic relationships between regulatory authorities and innovators as both groups worked closely to drive the RPF innovation, challenging the way they had worked historically. Working together on RPF project also helped to foster a greater collaborative style of working between groups, leading to a better understanding of the types of considerations each party had to make during the innovating process.

For example, a project lead spoke of how their consultation with industry left them with the impression that industry had a negative view of the regulator. The regulator had taken this on board and changed their communication strategy with industry, particularly at events, so that they were much more proactive about aligning expectations between themselves and innovators to facilitate better interactions. Another project lead spoke of how their work influenced firms to work together to incorporate technology earlier in the production process to elevate regulatory risks before they arose.

Introducing products, services and processes

Although there are some examples of how products, services and processes were being commercialised following on from RPF projects, the evidence for this was limited as there has not been enough time for this outcome to be realised, given that the projects have recently been completed. Instead, it is important to focus on whether the intermediary steps needed for this to happen are occurring.

New products, services and processes

There is evidence to suggest that for some projects, regulatory guidance and tools that have been developed and will go on to support the assessment of innovators’ queries will lead to more efficient procedures for approval. For example, tools for precautionary allergen labelling will help SMEs meet regulatory approval when developing new products, helping firms to comply with regulation whilst formulating their products. The publication of up-to-date guidance of what evidence is needed to approve microbiome healthcare products will allow innovators to understand what standards of evidence in their R&D process need to be met when submitting their products for regulatory approval.

However, whether these outputs have a significant effect on the number of innovations coming to UK markets is dependent on how well they will be disseminated by regulatory authorities (within and outside of the authority) and used by innovators. Evidence from the interviews suggest that there would be challenges for the regulatory tools to have sustainable impact if they are not embraced by their intended end users.

Increasing competitiveness of the UK as a place to invest

Some project leads suggested that their RPF work had led to improving the conditions for the UK to become a place to invest and do business because project outputs provided regulatory clarity, where there was uncertainty before. Project leads said that this was particularly the case in policy areas with complex and rapid innovation developments, where there is limited consensus among experts on how to best regulate innovations coming through. By creating guiding principles and benchmarks of evidence for industry to work towards when developing new products, RPF projects have helped to provide direction and thought leadership, thus increasing the confidence of investors in the UK regulatory framework.

Project leads suggested that the provision of these clear benchmarks and guiding principles improve the standing of the UK as a place to invest by reducing investor uncertainty to develop and commercialise products (see Figure 6 below).  This is because having a consistent regulatory framework for an innovation area would make investors feel secure and confident to invest in that area. This effect is amplified where the regulator is perceived to have robust standards and streamlined assessment processes that are internationally aligned, so that if products are able to pass their standards, then they be able to easily meet the standards of other countries.

How RPF is theorised to increase investor confidence

I personally think it’s massive for the UK, because it’s kind of a call for people to come here because we have the way to approach your product. [It means] people will realise that there is a streamlined process for products to be approved in the UK, and that it’s worthwhile coming to this market […] because [the Regulator] is so well regarded across the world, and because we’re part of international consortia, and once you’re approved with [us] you can actually get faster approval in other regulators. That means that we are a gateway to the market for so many people. That means that I do expect to see a lot more investment in the UK, and I know there are already discussions for manufacturing facilities (Advice Provision project, RPF3 project lead).

This outcome was not anticipated by the programme ToC, which focuses on consumer rather than producer confidence. At the time of reporting, it was too soon to know if RPF projects had streamlined regulatory approval or raised the profile of the UK and therefore led to increased investment. Nonetheless, multiple project leads indicated that their projects had raised the profile of their regulatory authority amongst other regulators and innovators both at home and abroad. Their work had demonstrated their ability to be forward thinking and proactive about addressing regulatory challenges posed by emerging technologies.

5. Understanding outcomes

This chapter discusses the factors that contributed to the project outcomes discussed in Chapter 4, drawing on endline interviews with RPF3 project leads. As will be discussed, RPF funding to develop and deliver innovations was seen by project leads to be the key reason why outcomes were achieved.

The importance of RPF funding

As noted in the ToC, the financial resources provided by the Fund was an important programme input and project leads from across all types of regulatory authorities highlighted that it was the only reason their project was able to achieve the outcomes discussed in Chapter 4.

The RPF funding supported outcomes by providing the financial resources that enabled regulatory authorities to prioritise innovations, the funding conditions that encouraged them to innovate and the exposure that allowed them to access expertise. These will be discussed in turn below.

Funding helped to prioritise innovations

The importance of the RPF funding should be seen within the constrained financial context in which regulatory authorities operate. Project leads described this financial situation as one in which they lacked the resources to work on activities outside of their day-to-day regulatory duties and were unaware of any external funding comparable to the RPF that they could access to resource innovation work.

I think it’s so valuable to have governmental [funding] …because it would be impossible to get money from anywhere else to do something like that. Even for research grants, it’s just so difficult to apply for something that fundamental, that type of work (Advice Provision project, RPF3 project lead).

Consequently, project leads were not confident that innovation work would have taken place without RPF funding. Mirroring the views of unselected RPF3 applicants discussed in Chapter 2, they said that innovation would not have happened without the RPF or would have been delayed significantly, between two to ten years, and delivered in a fragmented way. For example, projects being delivered gradually in compartmentalised chunks as pockets of funding became available. They said that this would have meant that regulators could not engage with sector innovations in a timely way, leaving them reacting to emerging innovations and compromising their advantage as thought-leaders in their sector and wider.

I couldn’t have done it, there was no way…without their [RPF] funding… Because … what I did was it is well beyond my responsibilities… I could not justify someone’s time on that from my team. We generally have no time whatsoever, we really struggle (Advice Provision project, RPF3 project lead).

I think we’d have continued doing some stakeholder work in this space, but … progress would have been much, much slower. The engagement would have been much, much more piecemeal. Yes, it would have been the work of years. Multiple years, rather than a really focused bit of work over eight months. There’s not an alternative explanation for the whole lot [project outcomes] (Curating and Disseminating Good Practice project, RPF3 project lead).

In this financial context, the RPF funding significantly contributed to project outcomes by enabling projects to prioritise innovations. It did so by supporting them to bring in dedicated resources for projects, including funding staff to coordinate, drive and deliver projects, as well as external expertise from contractors.

…without funding, the project wouldn’t have gone ahead…RPF funding has been fundamental for the [organisation] in terms of just enabling the organisation to be able to have the money and staff and the time to be able to explore these [innovation] options (Advice Provision project, RPF3 project lead).

In addition, RPF funding also facilitated joined-up working between regulators and other partners by providing a significant sized single pot of funding that incentivised collaboration. Project leads indicated that this helped to drive innovation in a more coherent way than if individual organisations were seeking out individual funding and working in parallel.

I don’t think there are many projects [outside of RPF funding] that bring the people that we brought together. Tech companies will work with industry, and maybe tech companies will work with a regulator, although less so. What we did was we brought everyone together… It’s a more productive way of doing things. Rather than doing these engagements separately and then coming together, let’s all get together and work on these challenges together (Curating and Disseminating Good Practice project, RPF3 project lead).

Funding conditions provided a low-risk environment to innovate

Project leads highlighted that the flexible funding conditions supported project outcomes by providing a low-risk environment, which encouraged them to be more exploratory in their project delivery.

The Fund provided a low-risk environment in two ways. Firstly, as discussed in Chapter 3, the RPF was open to projects reviewing and revising their project goals mid-delivery as they learned more about the innovation process. This enabled projects goals to be responsive to both innovation learning and delivery challenges as projects progressed, giving project leads the opportunity to adjust the type and scale of goals when needed without penalty. Secondly, project leads felt comfortable with taking risks and exploring innovations as the Fund did not penalise projects for not achieving all anticipated goals.

… it’s the one source of funding where you can actually do, you can explore things without the constraints that are sometimes placed on you. If it fails, it’s not a disaster, as long as you learn from it (Proof of Concept project, RPF3 project lead).

Providing exposure facilitated partnership development

As noted, the funding provided by the RPF was important in encouraging partnership development.

In addition, project leads felt that being known as awarded RPF funding helped them to develop partnerships and collaborations needed to realise innovation outcomes. This was because the Fund was regarded highly among regulatory authorities and the publicity it generated for projects helped to positively position the regulatory authorities delivering these as serious about innovation. For example, a project lead reported that they were able to work with another regulatory authority because of the publicity and “kudos” of being involved in the RPF.

… the Pioneer Funding did open those doors [to partner with another regulatory authority]…I think if it was just us saying…we’ve had this great idea…and we’d funded it ourselves, I don’t know, would there have been the same voice …I think it [RPF]  gives that kudos… (Advice Provision project, RPF3 project lead).

Alternative explanations and contributory factors

Project leads highlighted that innovation outcomes were exclusively a result of the RPF and that there were no alternative explanations, such as sector-related reasons. As discussed, this is particularly because the RPF funding helped them to prioritise innovations, easing the financial constraints in which they operated. The absence of alternative explanations may also be because the projects were short term and so they experienced limited changes in the delivery context which could have affected outcomes, such as sectorial and organisational changes. It will be interesting to observe whether alternative explanations emerge for the longer 12-18-month projects.

As noted in Chapter 2, there were several factors that supported project delivery and so helped projects realise key outputs, which then contributed to the achieved outcomes. These factors included regulatory authority organisational context, projects practices and the wider support provided by the Fund.

6. Long-term impacts

This chapter provides insights into the long-term impacts of the Fund, drawing on interviews with five RPF1 project leads. As will be discussed, these impacts were a continuation of the ToC outcomes discussed for the RPF3 projects. The impacts particularly illustrate RPF’s legacy in instilling a culture of innovation, shared learning about innovation practice and partnership work within regulatory authorities. An impact not anticipated by the original ToC was the ability of a project to significantly shape the wider regulatory purpose and remit of an organisation, and not just its approach to innovation.

RPF1 funding context

The first round of RPF funding was launched by the Better Regulation Executive (BRE) within BEIS in 2018). A competition to select projects was run by BRE and Innovate UK and took place between July and September 2018. Fourteen regulators were awarded, and their projects lasted between six and eighteen months. Five RPF1 project leads were interviewed to gain an in-depth understanding of the longevity of impacts since the funding ended, and what helped sustain them.

A development of an innovation culture and continued collaboration with other regulators and innovators were the key long-term impacts for RPF1 projects. To a lesser extent, RPF1 projects also led to innovation in services and products, but this should be seen in the context of the limited number of RPF1 projects interviewed. In addition, projects also acted as a ‘springboard’ mechanism in accelerating regulators’ thinking around innovation, inspiring them to do similar innovation activities.

What’s happened with it, I think, is - and I’ve used this phrase a few times; it’s been a springboard to everything we are doing now. So, everything we learned there we’ve then been able to take into - and this obviously leads into the outcomes - the work we do on developing UK regulations (RPF1 project lead).

The RPF1 impacts in relation to the three ToC impact categories will be discussed further in the sections to follow: developing the culture of innovation, promoting stakeholder engagement and collaboration, and introducing products, services and processes. In addition, there was one unanticipated impact from the original ToC relating to wider organisational change which has altered organisations’ whole remits rather than just innovation specifically.

Developing the culture of innovation

As discussed in Chapter 4, cultural change has three outcomes: changing attitudes so regulation is seen as part of their role, development of infrastructure to support regulation and new approaches to develop and test innovation. All three outcomes have been sustained since the RPF1 projects ended and have meant that regulatory authorities are more willing and better able to support innovation.

As previously noted, sometimes culture change did not happen across a given regulatory authority but within specific teams. However, the RPF programme helped to seed this focus on innovation, and it may take a longer time for culture change to be imbedded within an organisation.

Regulatory authorities continued to see innovation as part of their role

Regulatory authorities have continued to see innovation as part of their role. This is demonstrated where regulatory authorities have undergone a substantial shift in their long-term aims and strategies to better reflect the innovation landscape. For example, the use of innovation and technology has been incorporated into one regulator’s three-year strategy. Innovation is driving their agenda and has become one of their main priorities for the foreseeable future.

I think the short-term goals [RPF 1 project] were very much around us getting up and running, and proving the concept that there was a need for innovation. I think, absolutely, we nailed that within the period of the RPF bid. I think the long-term legacy is that innovation is now one of the key strategic board priorities (RPF1 project lead).

Regulatory authorities continued to benefit from changes to their infrastructure and innovation approaches

Regulatory authorities have continued to benefit from infrastructure developments to support innovation stimulated by their RPF1 projects. These included organisational changes, such as the hiring of staff with specific skills and expertise to support innovation which organisations did not previously have. There were wider organisational changes which promoted better joined-up working to support innovation. For example, where regulatory authorities brought staff and expertise together from different parts of the organisation and built innovation focused teams.

In addition to this, some regulatory authorities have continued to change their internal structure and become more innovation focused over time. For example, one project has now divided up their divisions and reorganised their regulatory group from a team of ten to nearly 200 people. These changes have meant that regulatory authorities are more confident and able to support innovation, because they have more people focused on this particular area.

our innovation and technology work has developed considerably and a lot of it could be credited to the success of RPF 1. We bought in resource, so we bought in innovation and tech resource specialists, etc. We’ve got a team now of about four, whereas there was only two of us before (RPF1 project lead).

Furthermore, the development of new methodologies and approaches during their RPF1 projects are still enabling regulators to foster and support innovation. For example, one methodology was developed to test new regulation and policy ideas through an innovative journey mapping approach. It helped the organisation to distinguish what needs changing in regulations for innovative technologies to be introduced. This tool has been invaluable and has been used several times since the project has ended and is now being used internationally (discussed further in the next section).

Promoting stakeholder engagement and collaboration

As discussed in Chapter 4, these outcomes focused on improved collaboration between regulators, and stronger relationships between regulators and innovators. There is evidence that both outcomes have been sustained after RPF1 projects were completed. In addition to this, there has been an increased understanding of how regulators can work together.

Relationships and learning between regulators have continued

RPF1 projects have helped regulators to have a better understanding of how to work together and have continued to collaborate to support innovation. For example, regulatory authorities have set up steering groups to keep everyone up to date with regulation developments. Others have created innovation hubs which have enabled them to continue working with other regulators in addition to their internal team.

It’s a quarterly meeting where we don’t stop anyone doing anything. It’s not like you have to have everything approved through that group, but it’s a way of ensuring we’re all up to date with the current what is going on, so we had one yesterday… Someone else gave an update on some seafarer training issues. It’s that cross-agency working that you would normally go (RPF1 project lead).

Regulatory authorities have also continued to share learning about innovation practices where they have set up long-term forums and have presented at events to share their learning and introduce other regulators to innovation.

As discussed previously, this impact was anticipated to take place nationally but there was also evidence that regulators have collaborated internationally because of increased interest and discussion. For example, one project developed a methodology for internal use which is now used in Sweden, described as the “most innovative piece” for policymakers. Regulatory authorities have cooperated and shared learning with regulators both in the UK and other countries.

We’ve had a lot of collaboration with the American regulator, with Canada, with Singapore. …We’ve also been to Australia. All the big regulators out there, we are collaborating now to see how they overcome the issues that are naturally there when you’re talking about [their sector]. Learning from each other and almost helping in a collaborative sense. So across the world, we’re helping to unlock some of these thorny regulatory issues (RPF1 project lead).  

Continued relationships between regulatory authorities and innovators

The positive RPF1 project outcomes and impacts have made regulators and innovators more open to working with one another. For regulators, RPF1 projects demonstrated the value of working with innovators and the role that that they and the innovators have in progressing innovations. For innovators, the projects have improved the credibility of regulators as organisations that are willing and able to support innovation.

This relationship between regulatory authorities and innovators is evidenced by their interaction. For example, one of the regulatory authorities is frequently contacted by innovators asking them to test their products and give guidance at industry events. Another regulatory authority has proactively increased their reach within the innovation community by running innovation events where they have brought together new start-up businesses and innovators to discuss certain topics.

I think one is that our profile as an organisation has improved enormously. For example, there’s an event tomorrow in London…where we’ve got a stand, which probably, a few years ago, perhaps before RPF 1, we probably wouldn’t have attended an event like that. The second is that we’re seen, to some extent, as an honest broker in some of these things, to facilitate conversations, to direct, particularly firms and solicitors if we can help them (RPF1 project lead).

Introducing products, services and processes

As discussed in Chapter 4, this impact category refers to the benefits to consumers and/or businesses from innovation as end users. There was limited evidence of the development of products and services benefitting consumers and business. However, this insight should again be considered within the context of only five RPF1 project interviews and the limited input from wider project stakeholders and beneficiaries.  It may also still be too early for innovative products and services to materialise, despite five years having passed since the RPF1 projects.

There was evidence of one project contributing to the development of innovative products and services. Following on from completion of the project, the regulatory authority ran a competition among innovators to progress innovation in their field. As a result of this competition, two charities awarded competition funding went on to develop a prototype of an innovative product they wanted to create.

The two that won were actually further along, they’d got a product that they wanted to create. The funding brought in additional resources in terms of developing it. I think, yes, I would say it was about turning half-formed ideas into fully formed ideas, or ideas that, particularly in the charitable sector, that required more funding just to get it, get more resource in. All of them started as something that was tangible but became more realisable by the end of the process (RPF1 project lead).

Wider organisational change

There was evidence that the RPF1 project helped to re-shape the general remit and direction of an organisation, in addition to how it approached supporting innovations. This impact was not anticipated by the original ToC and relates to structural and core change outside of the innovation landscape.  For example, a regulatory authority revised their remit to also focus on renewable energy because their RPF1 project provided proof of concept of how important and feasible it was for them to shift to this focus. The impact has highlighted that the RPF fund can have a long-lasting impact on the organisations involved and significantly change their direction.

Understanding long-term impacts

The factors influencing whether long term impacts were achieved were similar to those affecting whether RPF3 projects achieved their outcomes. These were the importance of RPF funding, organisational buy-in to the innovations and a shared-vision among stakeholders to drive the innovation.

As discussed in Chapter 5, the RPF funding provided both an opportunity to prioritise innovations and a low-risk environment to innovate. Without the funding, the regulators would not have the space to experiment and explore their innovation capability. The funding meant that they were able to prioritise time and resource for their projects, helping to shape a culture which supports innovation and stimulates partnership working.

Senior team and organisational buy-in were pivotal in encouraging regulatory authorities to see innovation as part of their role and releasing resources and staff time to dedicate to it. Senior staff viewed innovation work positively because the RPF1 projects provided proof of concept of the innovation itself and the ability of their organisation to engage with it. For example, one of the regulators’ chief executives supported the idea of innovation being mainstreamed and has made it part of their long-term vision.

The RPF1 projects have made regulators and innovators realise the importance of innovative activities. At a wider level, having a shared vision amongst stakeholders has been a key enabler for the long-term impacts. The shared vision and enthusiasm from industry has meant that everyone is working towards the same goal and has the same ideas about making innovation a priority. It has enabled greater relationships with both regulators and innovators and has meant that more stakeholders have been willing to work together.

7. Conclusion

This report presents interim insights into the lessons learned from delivering innovative projects and the extent RPF3 has delivered against its goals so far. This chapter summarises the main insights and the next steps for the evaluation.

Understanding lessons learned from delivering innovative projects

In contextualising fund entry, set-up and delivery insights, it is important to note that the RPF design and operational processes have been iteratively shaped by the feedback and evaluation learning since the first round in 2018. This learning led to key changes, such as the RPF allowing for projects of different lengths and extending the application time period.

In addition, although project leads made several set-up and delivery suggestions, these should be viewed in relation to wider context of governmental processes and resourcing within which the RPF operates. These processes sometimes constrain changes to some components of the programme, such as financial year expenditure and payment to projects in arrears.

Fund entry and set-up learning

A key motivation for regulators to apply for RPF funding was the absence of any alternative, sizeable and dedicated source of funding that could support projects to work on innovations. Regulatory authorities were motivated to apply if they had previous experience of applying for the fund. This was because they found the application process more accessible as they were familiar with it and, if they had been previously selected, they felt reasonably confident of being awarded RPF3 funding.

Although RPF3 project leads highlighted that the application process had noticeably improved since previous rounds, new applicants would have liked greater support to understand whether their innovation was on the right track prior to applying through, for example, an additional pre-assessment process to screen initial ideas. Unselected applicants would have liked more feedback and transparency regarding the scoring system to help them feel more confident applying again.

A key set up challenge for some projects was the short timeframe between the award and project start, which compromised delivery. A key enabler for set-up was having pre-project engagement which including using existing contacts or stakeholders or recruiting internally to save time. Those with RPF experience faced fewer challenges with the set-up stage as they were aware of the elements that were essential for delivery to run smoothly.

Delivery learning

Project leads clearly reiterated the need for the RPF to fund shorter-term projects. Funding for 8-month projects support innovations that benefit from a focused delivery period, such as sandboxes, and can help projects focus thinking, resources, and partnerships on these innovations.

Project leads highlighted the importance of Fund support and the regular catchups between Fund staff and project leads in helping them to deliver to time and scope. The interim insights underline the importance of these catchups focusing on identifying, understanding, and resolving emerging project challenges. In addition, the need for the Fund to be flexible in addressing key project challenges, such as navigating different spend profiles, was highlighted as important.

The interim insights also point to the importance of effective project management in helping innovations to be delivered to time and scope. In particular, it was important for projects to manage contractors effectively, as they were often key to delivery of some of the innovations. Examples of good contractor management practice included drawing on trusted contractors and effective liaison practices, such as having a single point of contact between projects and contractors in innovations involving a consortium of partners.

Understanding project outcomes and impacts

Achieved outcomes

The project outcomes reflected the three impact categories described by the ToC: developing a culture of innovation, stakeholder engagement and generating new innovations. Additionally, the outcomes were well reflected in the contribution statements, though further refinement of the statements was needed to reflect the complete story of the RPF’s impact.

Project leads, in particular, commented on the considerable progress they had made on developing a culture of innovation and engaging and collaborating with stakeholders through the work of the RPF. However, not enough time had passed to determine whether commercial innovative products and services had entered the market because of the Fund.

Although the reported outcomes were largely anticipated by the programme ToC, a set of outcomes around how the RPF improved the UK’s profile abroad was not originally considered by the ToC, which has now been revised to reflect this.

Understanding outcomes

Project leads felt that the RPF funding was the exclusive reason why outcomes were achieved. They felt the funding made it possible for projects to develop. As noted in previous chapters, project leads underlined how a dedicated innovation fund, such as the RPF, is important in providing the resources needed to prioritise innovations and the low-risk funding conditions to innovate. Key to these conditions was the RPF’s flexibility to support projects to learn as they deliver, as opposed to being too narrowly tied to achieving stated project outcomes.

The absence of alternative explanations may also be explained by the fact that projects were short term and therefore experienced limited sectorial or organisational changes.

The factors that supported project delivery contributed to outcomes by helping projects realise key outputs. These factors included regulatory authority organisational context, projects practices and the wider support provided by the Fund.

Understanding impacts

Although care must be taken in interpreting the impact insights as there were a limited number and range of projects involved, all RPF1 projects achieved some level of sustainable innovation impacts. These impacts represented a continuation and deepening of the three ToC impact categories. In particular, the long-term legacy of the RPF was in sustaining the culture of innovation within regulatory authorities and in promoting partnership work with innovators, regulators and other stakeholders.  

In addition to supporting a long-term innovation outlook, an unanticipated outcome was the way in which a specific project had changed the whole remit and direction of the regulatory authority to focus on renewables. It had done so because it provided a proof of concept on how that organisation could shift its direction.

Provisional changes to the ToC

As noted, the evidence largely supported the original ToC. However, it needed to be revised to accommodate two additional outcomes from the evidence. Firstly, the role of the RPF in facilitating collaboration and engagement was enlarged to also reflect how this takes place internationally. That is, how RPF projects lead to relationships with regulators from other countries and the impact this has on helping UK develop world-leading regulatory practices that are innovation friendly. 

Secondly, the ToC needed to reflect more explicitly the impact of the Fund in improving investor confidence into UK markets. Although the original ToC spoke to improving consumer confidence in products and services, this was not extended to the way in which it improved regulatory practices. In cutting-edge sectors where regulation is less well-defined, creating a more predictable governance over industry can lead to a better environment for capital investment.

Next steps

These interim insights provide early insights into the key delivery lessons and Fund outcomes and impacts and have enabled refinement of the ToC and its associated contribution statements. Following from this, detailed case study work with a select few 8-month projects to further unpack assumptions around how projects influence outcomes and impacts, and the main drivers of change and associated safeguards and risks will be conducted.

In the longer term, the ToC and the contribution statements will be further tested against the emerging insights from the qualitative work undertaken with the 12-18-month projects later this year and in 2025.

References

Befani, B., Mayne, J. 2014. “Process Tracing and Contribution Analysis: A Combined Approach to Generative Casual Inference for Impact Evaluation.” IDS Bulletin 45 (6): 17-36.

Department for Business and Trade; Department for Business, Energy and Industrial Strategy. 2018. Better Regulation Framework gov.uk.

Department for Business, Energy and Industrial Strategy. 2017. Industrial Strategy: building a Britain fit for the future HM Government.

Department for Business, Energy and Industrial Security. 2019. Regulation for the Fourth Industrial Revolution gov.uk.

Department for Business, Energy and Industrial Strategy. 2022. Regulators’ Pioneer Fund: competition brief for funding round 3.

Department for Science, Innovation and Technology, Department for Business, Energy and Industrial Strategy. 2021. UK Innovation Strategy: leading the future by creating it Crown Copyright.

HM Treasury. 2023. Pro-innovation Regulation of Technologies Review.

HM Treasury. 2021. Build Back Better: our plan for growth Crown Copyright.

Kantar (Public Division). 2021. “Evaluation of the Regulators’ Pioneer Fund (round 1) - Main Findings Report.”

Appendices

Appendix A: Project descriptions

Project	Description
Developing approaches to make AI algorithms more interpretable using AI as a medical device as an exemplar (MHRA)	A key concern for regulators and end users relating to AI algorithms that support clinical decision making is that they are not always transparent about the chain of reasoning resulting in a specific decision being recommended. This can affect trust in AI algorithms and their subsequent adoption by end users. While the preferred solution may be to use inherently transparent models (white box or glass box) models, there is a potential role for more complex black box models, provided any decisions recommended by the latter are explainable, allowing them to be appropriately interpreted by the end user in their context. Explainability is about understanding how the algorithm arrived at a decision: for example, what factors did the algorithm consider and what weight did it assign to them? Interpretability, on the other hand, refers to the degree to which a human user can understand how a decision made by an algorithm applies to their own context. For clinician end users, a basic degree of explainability would be required for understanding the internal logic of the algorithm so that they have confidence in either accepting the algorithm’s decision or overruling it. This project aims to produce a workable methodology to support regulation and clinicians for both transparent and complex models. This will improve the possibility of introducing complex AI safely into the clinical pathway
Enabling innovation – piloting a multi-agency advice service for digital innovators (ICO)	The UK digital sector contributed nearly £151 billion to the economy in 2019, with 1.7 million filled jobs in the sector in 2020. Regulatory clarity and certainty are crucial to its continued growth, giving innovators confidence to bring new ideas to market. Regulation of the digital sector is changing, with new legislation being considered across online safety, data protection, financial services, cybersecurity and competition in digital markets. Offering joined-up regulatory advice would reduce burdens and complexity for businesses navigating multiple regulatory remits when developing new products and processes. The Digital Regulation Co-operation Forum (DRCF) is a collaboration between the Information Commissioner’s Office (ICO), Ofcom, the Competition and Markets Authority (CMA) and the Financial Conduct Authority (FCA). It seeks to deliver greater co-operation and coherence between these regulators of digital businesses. The DRCF aims to help make the UK the most innovation-friendly jurisdiction in the world by making it easier for firms that operate across digital regulatory boundaries to do business. To advance this ambition, the DRCF will undertake exploratory research and pilot a multi-agency advice service for digital innovators needing joined up advice from multiple regulators. The service will be designed around the views, needs and working practices of innovators across the digital economy. Success will be measured by the accessibility of the pilot service and the impact of our advice on the businesses who use it. By August 2023, the project will deliver a report to the DRCF on whether and how to introduce a multi-agency advice service for innovators.
Using digital twin technology to enable low carbon industrial clusters (EA)	Our project will, for the first time, use world-leading digital twin modelling to help industry shape industrial clusters based on real-world environmental constraints –enabling rapid deployment to support energy security and sustainable economic growth. This is the first time that cutting edge digital twin technology has been used with environmental data to explore environmental limits and optimise decarbonisation technology in industrial clusters. The concept of ‘digital twins’ allows the creation of a digital representation of real-world places and systems. We can use a virtual counterpart of a real system to try different combinations of technology in places, providing invaluable early insight into environmental risks and informing and streamlining regulation. We will use digital twin technology to simulate the operation of multiple low carbon technologies in an industrial cluster. Using the outcomes from this project we’ll explore opportunities for industry to work innovatively to mitigate environmental impacts and avoid costly delays in deployment. This provides an entirely new capability to “see the future” and shape it –leading to faster deployment timescales and lower environmental risk. We will work with local and national Environment Agency specialists, Microsoft and their digital partners, industry representatives and other UK environmental regulators to achieve this. We will investigate how digital twin technology can be used as a positive tool to understand and plan to mitigate environmental limits as part of the overall ambition to facilitate the development of the first UK low carbon industrial cluster.
Pilot of a regulatory sandbox on artificial intelligence in the nuclear sector (ONR)	Regulators have an important role in minimising regulatory uncertainty around innovation and play a key role in enabling the adoption of innovative approaches and technologies. The Office for Nuclear Regulation’s (ONR) Innovation Hub is currently trialling a number of products for internal and external use to enable innovation in the nuclear sector where it is in the interest of society and is consistent with safety, security and environmental protection expectations. Working with the Environment Agency, this project will pilot a regulatory sandbox process to support adoption of innovative technologies and approaches in the nuclear sector. Regulatory sandboxing is when regulators and industry work together in a non-regulatory environment to explore how innovative proposals can progress to deployment. Artificial intelligence (AI) has been chosen as the pilot topic and the project will focus on two potential applications in the nuclear sector. The pilot sandbox will enable innovators in AI to access regulatory advice in a safe space to derisk future deployment, while maintaining independence from regulatory decision making. This would be the first application of a regulatory sandbox by nuclear regulators in the UK and the learning will be shared with key stakeholders and industry.
Capturing innovation to accelerate improvement in health and social care (CQC)	Meeting the increasing demand and challenges faced by health and social care to deliver positive outcomes for our population requires innovation, creativity, and a willingness to try something new. As the regulator of health and social care in England, the Care Quality Commission (CQC) is committed to championing and enabling innovation. And we want to accelerate our own learning to be an innovation-friendly regulator. Through this project, we will explore and pilot ways of developing the right regulatory environment for innovation to flourish. Over eight months, CQC will work with a group of innovators and partners within the health and care system. Our aim is to: capture examples of high-quality innovation and share this learning explore and articulate the role of an enabling and supportive regulator in an innovative health and care system pilot a suite of innovation-supporting activities and products to accelerate innovation and improvement based on research evidence about the ways CQC can impact improvement in health and social care. Learning from this project will make sure that innovation in England’s health and social care system is safe and supported. It will help make it easier for health and social care to design and adopt high quality innovation and ensure that regulation is seen as an enabler, not a barrier, to innovation.
Navigating SMEs through the complexities of ‘Precautionary Allergen Labelling Risk Analysis’ through the use of innovative technologies (Wakefield Council)	The aim of the project is to provide businesses, particularly SMEs with a free, interactive tool to assist them in the learning, understanding and application of precautionary allergen labelling (PAL) risk analysis. There are currently no existing support resources comparable to the one being produced. The tool will be a ‘first of its kind’, interactive interface, which will allow users to select from a series of diverse retail and hospitality business formats from a traditional coffee shop, through to a more complex restaurant providing a range of dishes that include ‘free-from’ allergen offerings. Once the user has selected an operational model with similarities aligned to their existing operating model / workplace, they will be able to navigate the outlet’s operational layout and characteristics and complete a series of hypothetical risk-based scenarios that will aid decision making in completing their PAL risk analysis. The scenarios have been designed to challenge the thinking of Food Business Operators (FBO) and their employees, to help them distinguish the difference between controlled and uncontrolled cross-contamination of allergens through good food hygiene and safety practices. The tool will be licensed to allow national regulators the permissions to host or signpost the tool through web integration, to promote a standardised approach and provide a national resource to aid businesses with effective PAL analysis and in turn detract from general or blanket statements, i.e., ‘may contain’ being used without appropriate and proportionate risk analysis, which can often devalue the use of PAL, and exclude and mislead consumers with food allergies.
Developing a market-based approach to enable significant SuDS investment through collaborative streetworks (Greater London Authority)	This project will develop a strategic and innovative market-based approach to facilitate the delivery of critical sustainable drainage systems (SuDS) at scale through the Infrastructure Coordination Service (ICS) collaborative streetworks programme. The Independent Review into 2021 London floods and Thames Water’s draft Drainage and Wastewater Management Plan (DWMP) have promoted their importance with Thames Water planning to increase delivery from a current rate of 20 hectares every 5 years to 7,000 hectares by 2050. A recent pilot project promoted by the ICS and delivered by Cadent Gas and Enfield Council identified an opportunity to enable 25%cost saving in the delivery of SuDS through the ICS collaborative streetworks programme, however blockers remain preventing the approach being scaled. This project will seek to exploit the opportunity identified by leveraging the learning from UK environmental markets to address identified blockers to investment and enable scaled collaborative delivery. The project will also identify any wider opportunities for application of market-based approaches to improve investment in critical flood resilience measures in London. We expect that this work will support delivery of significant benefits to London’s businesses and communities by removing blockers to investment, reducing SuDS delivery costs, increasing flood resilience, and supporting innovation and local economies. We would be happy to share the outputs of this study with other local authorities interested in this approach.
Enabling Innovation in Industrial SafetyTECH (HSE)	Great Britain’s health and safety (H&S) performance has seen significant improvements over the last 30 years. However, Health & Safety Executive (HSE) statistics show that each year over a million workers are injured or made ill by their work. This affects individuals, families, employers, government and wider society.  Technology is changing rapidly and should be embraced to keep the workplace a safer and healthier place for all. Industrial Safetytech (IS) is the term for innovative technologies, products and services that are redefining approaches to H&S management. It can play a crucial role in promoting compliance to H&S legislation, especially for Small to Medium Enterprises (SMEs). However, regulations established to protect the public interest can become barriers to innovation. Addressing this potential regulatory barrier requires regulators and innovators to collaborate, to promote and stimulate emerging technology. This project employs a regulatory sandbox approach enabling experimentation with technological innovations. It benefits both regulator and industry, providing clarity to innovators on regulatory expectations and confidence to industry to invest in and adopt Industrial Safetytech. HSE will lead this ground-breaking project in partnership with Safetytech Accelerator Ltd to generate practical evidence-based assessments of IS against regulatory frameworks in Construction. The project outcomes will support safe deployment of emerging technology providing confidence to large business and SMEs around IS adoption. This initiative will generate and harness new thinking, methods and technological approaches to improve regulatory delivery and performance, thus enabling innovative new products and services to come to market for the benefit of all.
Developing guidelines to accelerate innovation for microbiome therapeutics and diagnostics (NIBSC)	The research of the human body-associated microbes, the microbiome, can advance our understanding of human health and disease, providing new solutions for public health and allowing us to move towards personalised medicine. The complex and novel nature of microbiome therapeutics and diagnostics poses a challenge for both the regulators and companies, with the absence of specific guidelines delaying the approval process and discouraging companies from starting the process in the UK –they would commonly choose to first launch their products or business in the USA where microbiome activity is more prominent and more established. In order to encourage innovators to invest in the UK health care, the National Institute for Biological Standards and Control (NIBSC) aims to develop appropriate internal and external guidelines that will facilitate the regulatory process, bringing confidence to the microbiome field and ensuring the safety of our patients. To achieve this, the NIBSC will collaborate with the different MHRA departments, as well as with its extended national and international network of scientists, clinicians, companies and regulators, to ensure that it can capture the views and requirements of the field. NIBSC will further utilise this knowledge to deliver public communication pieces aiming to inform and educate scientists, clinicians and patients on this emerging field.

Appendix B: Methodological summary

Fund staff

A paired interview was completed with Fund staff managing the short-term projects. The participants were recruited directly through the TIR. The interview lasted for an hour and half and involved discussing the staff members’ role and responsibilities and how they have supported the projects at different stages. They were asked how they felt RPF projects have progressed and whether there were any emerging outcomes they have identified.

RPF3 Projects

Nine interviews, combining both single and paired interviews, were completed with all RPF3 strategic and operational project leads at the set-up stage of their projects. TIR provided NatCen with each regulatory authorities’ contact details. NatCen sent out emails alongside a privacy notice, and then scheduled the interviews based on project leads’ availability. The interviews lasted for approximately 70 minutes and focused on the participants reasons for applying to RPF, their experiences of the application process, the context behind their projects innovation and any outcomes they anticipate coming from the project.

An additional nine interviews were completed at the end of their projects and used the same recruitment approach, also compromising both single and paired interviews. The interviews, lasting 80 minutes, focused on the participants experiences of delivering the different stages of the project, what expected and unexpected outcomes they have achieved and how the impact of these outcomes could be sustained in the future.

Monitoring Data

Each regulatory authority completed monthly progress reports for the TIR guided by the ToC and contribution statements (Appendix C). A secure folder was set up so that the TIR could share the monthly reports to NatCen securely. The evaluation team then produced quarterly monitoring reports synthesising three months’ worth of progress reports, based on a subset of measurement indicators. The quarterly reports were produced in Microsoft PowerPoint and showed a high-level summary of project progress, widespread challenges, risks and barriers, and any key learning that emerged across the projects.

The evaluation also drew on the insights from the end of project reports that each RPF3 project was asked to complete after the 8-month period. These reports provided further insights on key project outcomes, delivery experiences, learning and outcomes. 

RPF1 Projects

Five interviews were completed with RPF1 project leads. The projects were selected purposively based on TIR’s guess of sustainable outcomes, and whether they have been involved in RPF1 and RPF2. TIR provided contact details to NatCen who then made contact and scheduled the interviews accordingly. These interviews lasted an hour, and involved discussing their relationships to the past project, any immediate and sustained project outcomes that they have witnessed, and what contributed to these.

Unselected/non applicants

Four interviews were completed with unselected applicants and one interview was completed with a non-applicant. Like all other qualitative data encounters, TIR provided the details for the regulators and local authorities, whom NatCen were able to make contact. These 45-minute-long interviews focused on the reasons for applying for RPF, their experiences of the application process, their views on the application outcome and any improvements that could be made to the process of applying for RPF.

Data analysis

All interviews were transcribed verbatim, and data summarised using NatCen’s qualitative Framework approach, conducted using NVivo 12. This approach facilitates data management and analysis by case and theme within an overall matrix. In this study the cases were participants who have taken part in the different stages of qualitative data collection and the themes were based on the aims and objectives of the research. This approach to data management allowed for a robust and systematic analysis grounded in the views and experiences of the interviewed participants. When analysing the data, special attention was made to the Theory of Change and contribution statement.

Appendix C: Monitoring Form Template

Available in the Evaluation of the Regulators’ Pioneer Fund (Round 3) – interim report (PDF version)

Glossary of key acronyms

Acronym/Term	Definition
AI	Artificial Intelligence
Baseline interviews	Start of project interviews with project leads for RPF 3 awarded projects
BEIS	Department of Business, Energy and Industrial Strategy
CQC	Care Quality Commission
Delivery to intention	Whether projects were able to deliver their activities as they originally intended in their proposal
DSIT	Department for Science, Innovation and Technology (DSIT)
EA	Environment Agency
Endline interviews	RPF 3 end of project interviews
HSE	Health and Safety Executive
ICO	Information Commissioner’s Office
Impact categories	The different types of RPF impacts in the Theory of Change
MHRA	Medicines and Healthcare products Regulatory Agency
NISBC	National Institute for Biological Standards and Control
ONR	Office for Nuclear Regulation
PoC	Proof of Concept
R&D	Research and Development
Regulatory authorities	Regulators and Local Authorities
Sandboxing	Building and testing innovation concepts and solutions in a controlled, simulated scenario without risk to operations. Within this report, we refer to regulatory sandboxes which is defined in Chapter 1.
SMEs	Small to medium sized enterprises (1-250 employees)
TIR	The Technology and Innovative Regulation Directorate (TIR), part of the Department for Science, Innovation and Technology (DSIT)
ToC	Theory of Change

1. Befani, B. & Mayne, J., 2014. Process Tracing and Contribution Analysis: A Combined Approach to Generative Causal Inference for Impact Evaluation. IDS Bulletin, 45(6), pp. 17-36. ↩