Guidance

Supplementary Guide: Guidance for conducting regulatory post implementation review (HTML)

Updated 2 July 2025

Glossary

Alternatives to regulation: alternative ways to achieve policy outcomes including industry self-regulation, co-regulation, information and education, economic instruments and improving the use of current regulation. More details](https://www.gov.uk/government/policies/reducing-the-impact-of-regulation-on-business/supporting-pages/using-alternatives-to-regulation)

Baseline: a reference level against which a proposed regulatory intervention is monitored. For a new measure, the baseline will typically be a defined period before enactment. This may be distinct from the counterfactual position.

Benefits management: an approach to project delivery which emphasises measuring project success not only by the delivery of outputs but also by the positive, measurable improvements the relevant measure has delivered for stakeholders (identified using monitoring techniques that often overlap with evaluation).

Better Regulation Executive (BRE): the directorate within the Department for Business, Energy and Industrial Strategy that leads the regulatory reform agenda across the Government.

Better Regulation Unit (BRU): the team in each department responsible for promoting the principles of good regulation and advising departmental policy makers

Cost-benefit analysis: form of economic analysis which involves identifying and where possible quantifying and monetising costs, benefits and risks of the proposed policy options.

Cost-effectiveness analysis: form of economic analysis which values the cost of implementing and delivering a policy and relates this amount to the total quantity of outcome generated to produce a “cost per unit of outcome” estimate.

Counterfactual: an assessment of what would have happened if a regulatory intervention had not been made or if it had taken a different form.

De minimis threshold: the EANDCB threshold applied in relation to each measure to determine whether or not it is subject to scrutiny by the RPC. Measures that have been certified by Departments or regulators as falling below the de minimis threshold (namely any measures that have an EANDCB of +/- £5 million, and are not novel, high-profile or contentious) are not subject to RPC scrutiny; all other measures are.

Departments: Government Departments and agencies.

Domestic (measures): a measure which is neither EU-derived nor based on an international obligation.

Equivalent annual net direct costs to business (EANDCB): a figure which represents an estimate of an intervention’s annual net direct costs to business in each year that the measure is in force.

Evaluation: a systematic process of understanding how a policy has been implemented, what effects it has had for whom, how and why.

Fit for purpose: a statement in a RPC opinion which indicates that the RPC believes that the analysis in the relevant RIA or PIR impact meets an acceptable standard, known as a ‘green’ rated opinion.

Impact: an effect that can be identified as resulting from the implementation, removal or modification of the measure.

Logic model: a tool that describes how a policy is expected to achieve its objectives. It does this by describing the causal pathways underlying the rationale for a policy, linking its inputs to its intended outcomes.

Measure: any primary or secondary legislation, regulation, statutory or regulatory guidance, or policy proposal to promulgate any of them.

Monitoring: a process for tracking progress in the delivery of a measure by collecting data on its outputs. Monitoring data is likely to make an important contribution to the evidence for a PIR.

Net present value (NPV): the sum of the future discounted benefits of an intervention less the sum of future discounted costs.

Net present social value (NPSV): The sum of the future discounted benefits of an intervention to society less the sum of future discounted costs to society.

Non-regulatory: measures which do not involve regulation, such as tax or spending measures.

Post-implementation Review (PIR): a process implemented to monitor and evaluate the effectiveness of a measure after it has been implemented and operational for a period of time. It addresses the extent to which a measure is achieving its intended effects, whether it has resulted in any unintended effects, how well it is working and the reasons why.

Reducing Regulation sub-Committee (RRC): a Cabinet sub-committee established to take strategic oversight of the delivery of the Government’s regulatory framework.

Regulatory Impact Assessment (RIA): an assessment tool to enable the Government to weigh and present the relevant evidence on the positive and negative effects of a measure. A RIA summarises the rationale for intervention, the different policy options and the impacts of the intervention, as well as quantifying expected costs and benefits.

Regulatory Policy Committee (RPC): an independent advisory non- departmental public body which scrutinises the quality of evidence and analysis supporting new measures and checks the estimates for the EANDCB of new measures to ensure decisions are made on the basis of robust analysis. The RPC is also the appointed independent verification body responsible for verifying the economic impact on business of measures in scope of the business impact target (BIT).

RPC opinion: a written opinion published by the RPC indicating its determination of whether or not a RIA or PIR is fit for purpose.

Small and Micro Business Assessment (SaMBA): an analysis undertaken to ensure that all new measures are designed and implemented so as to mitigate disproportionate burdens on small businesses (up to 50 employees) and micro- businesses (up to 10 employees).

Sunset and review clauses: provisions included in a measure to ensure that measures with a significant cost to business must face review (and where applicable, be subject to sunsetting).

Theory-based evaluation: an evaluation approach which involves articulating a theory about how the intervention is intended to achieve its impacts and testing this theory with evidence.

Unintended effects: a term used to describe any outcomes or impacts (whether positive or negative) of a measure which were not included in the original IA because they were not anticipated or intended, including any outcomes or impacts which arise due to changes in the external environment.

1. About the guide

1.1. Purpose of this guide

This guide provides analytical guidance for conducting a Post-Implementation Review (PIR) of a regulatory measure^{[footnote 1]}. It is designed to support policy officials, analysts, and others responsible for conducting PIRs by setting out a proportionate approach to PIR delivery. It should be read alongside guidance on producing PIRs of the Better Regulation Executive (BRE), including its principles of best practice of July 2018.

• Chapter 2 provides a step-by-step guide on Preparing for a PIR. Preparation includes assessing what level of evidence is proportionate, involving professional expertise, defining key PIR questions and deciding what approaches are proportionate.
• Chapter 3 provides additional advice on Methods for delivering a PIR including guidance on monitoring data collection, engaging with stakeholders, and evaluation options to support a more substantial PIR.
• Finally, Chapter 4 sets out How PIR evidence should be used to support decisions about next steps and provides some guidance on writing the PIR report.

While examples are provided, the guidelines are not prescriptive about which methods should be used or what level of resource is proportionate. Departments will need to decide these matters for themselves, based on their analytical and policy needs. When preparing for a PIR, Departments may find it helpful to refer to the PIR template available from BRE and the guidance of the Regulatory Policy Committee (RPC)^{[footnote 2]}.

This guide has been published as an annex to the Magenta Book guidance for evaluation. Many of the concepts explored in this guidance are covered in greater depth in the Magenta Book, and signposting is provided throughout.

1.2. What is a Post-Implementation Review?

A PIR is a process to assess the effectiveness of a measure after it has been implemented and operational for a period of time. PIRs are often triggered by a review clause within the measure. The timing of the review is usually three to five years after it comes into force and on a five-year cycle thereafter but a PIR can be conducted earlier if a Department decides to do so.

Conducting a PIR is important because it ensures there is a review of the evidence base for the measure and a check of whether the assumptions that were made when it was devised have held up. This is particularly important when the measure is expected to have high impact and when it is believed that it may have had unintended effects.

A PIR seeks to establish whether, and to what extent, the measure:

• has achieved its original objectives as set out in the original Regulatory Impact Assessment (RIA),
• has resulted in any unintended effects,
• has objectives which are still valid,
• is still required and remains the best option for achieving those objectives; and
• can be improved to reduce the burden on business and its overall costs.

The level of evidence needed to answer these questions will vary according to the existing evidence base, the scale and timing of expected impacts, and contextual factors (see section 2.2 Assessing what level of evidence is proportionate).

The output of the PIR will be a report which answers these questions and provides the analysis to support decisions about the next steps with a measure, which are:

• renewal;
• amendment;
• removal; or
• replacement.

The steps that follow completion of a PIR are detailed in the BRE’s guidance (see Better Regulation Framework and Producing PIRs Principles of best practice, both 2018).

Where measures are certified as having an EANDCB of more than the de minimis threshold of £5 million, the completed PIR should be sent to the RPC for formal quality assessment. Further guidance on how the RPC assesses PIRs can be found in its RPC case histories document.

Relationship between PIRs and RIAs

A RIA is a pre-regulatory process which seeks to estimate the effects of a measure, while a PIR is a post-regulatory process which seeks to measure the actual effects of a measure after it has been enacted.

Although a PIR report will normally be produced after 3-5 years of implementation, planning for it should begin much earlier, at the time the measure is initially developed and before the RIA is completed. This early planning will help ensure that the right baseline and post-implementation evidence is monitored and collected to be available to inform the PIR. Therefore, the RIA should give some consideration to the most proportionate way of delivering a PIR.

In some circumstances, a PIR will be used to inform the redesign of that measure, or the design of a new measure to replace it. In such circumstances, the PIR and RIA processes may both support the same decision and use some of the same evidence. The PIR needs to focus on assessing the impact of the existing measure, while the RIA assesses the design of the new measure.

Relationship between PIRs and evaluation

Evaluation is a systematic process of understanding how a policy has been implemented, what effects it has had, for whom and why. In Government, analysts who conduct evaluations may come from a range of professions including social and operational researchers, economists, and statisticians.

When planning for a PIR, it is important to be aware of any evaluation activities that are intended to develop an evidence base around the impacts of the measure or wider policy issue. Additionally, evaluations can be conducted at a policy-level rather than focusing on a specific measure, which furthers the understanding of impacts more broadly. Wherever practicable, a PIR should draw upon evidence generated by such evaluation work, so that duplication of effort is avoided.

Where a PIR draws on published evaluation findings, it will be sufficient for the PIR to present a short summary of the key evidence together with references to the relevant reports. Only if the evaluation report is not yet available should it be necessary to provide a detailed account of the relevant findings within the PIR.

2. Preparing for a PIR

This chapter explains what needs to be considered when preparing for a PIR. It provides guidance on:

• At what stage to prepare for a PIR;
• Assessing what level of evidence is proportionate;
• Who should be involved in completing a PIR;
• Defining the scope of the PIR;
• What to do if PIR preparation begins late.

2.1. At what stage to prepare for a PIR

A fit-for-purpose PIR depends upon appropriate data collection plans and systems being established at an early stage; i.e. when the RIA is drafted.^{[footnote 3]} The appropriate level of PIR planning for different stages of the process is set out below.

• By the time the RIA is finalised: It is good practice for the RIA to set out a plan for the PIR, including its objectives and expected methods (see BRE’s RIA template, Annex 1). Detailed planning should begin at this stage.
• After an RIA has been finalised, planning of the PIR should continue with setting up data collection systems that will inform it, including any baseline data.
• By the time the measure is enacted, the PIR strategy, data collection plans and budgets should be finalised, baseline data should have been collected, and systems for the collection of post-implementation monitoring data should be in place.
• After the measure comes into force, monitoring data should be collected and the PIR strategy and data collection plans should be implemented, with revisions as appropriate to take account of emerging issues.

2.2. Assessing what level of evidence is proportionate

For PIRs, as for other Government activity, the need for good quality evidence should be balanced against other priorities to ensure value for money for the taxpayer. This section provides guidance on factors to consider when assessing what level of evidence will be proportionate for a particular PIR.

Extent of the existing evidence base

Firstly, a PIR plan should consider how the PIR might draw on evidence generated by other planned regulatory review activities, such as:

• existing plans for monitoring of the measure’s impact;
• evaluation or other research activities relating to the relevant policy area;
• PIRs/statutory reviews for related measures;^{[footnote 4]}
• post-legislative scrutiny of primary legislation; a “stock review” of existing legislation;
• a review by the European Commission, either of a specific measure, or measures in a particular field.

PIRs should make full use of existing evidence to reduce duplication of effort.

Scale of the estimated impacts

After taking account of any existing evidence that is available, a judgment should be taken about the level of additional evidence that may need to be collected. The evidence collected should be proportionate to the scale of the measure and its expected impact, while also recognising other contextual factors. It may be helpful to refer to the RIA to see what level of analysis was then judged to be proportionate.

Where substantial impacts are expected to arise from measures, there may be a stronger case for substantial expenditure to ensure that evidence is captured robustly (so long as such expenditure will make an appreciable difference to the quality of the evidence collected and will not duplicate existing evidence collection activities). Equally, where impacts are anticipated to be low, a lighter touch review may be more proportionate.

Although judgments should be made on a case-by-case basis, where the equivalent annual net direct cost to business (EANDCB)^{[footnote 5]} exceeds £50m, a more substantial review is likely to be needed. Where net costs or benefits fall below £10m, and especially when below the de minimis threshold (+/- £5m), a light touch review is likely to be proportionate.

Timing of the estimated impacts

The anticipated timing of impacts is another important factor to consider when determining a proportionate approach. In some cases, sufficient time will have passed for a measure’s full impacts to have occurred by the time of the initial review of it. In other cases, a longer period will be required for impacts to be observed, so the PIR should focus on shorter-term outputs and outcomes which are expected to lead to the ultimate impacts in the long-term. In this way, the PIR can check whether or not the measure is operating as expected and moving in the right direction, even if it is too soon to capture its full impacts.

Where it is not possible to capture the impacts of a measure at an initial review point, this may justify a light-touch PIR, with plans being put in place for a more comprehensive review at a later date, possibly as part of a broader policy evaluation. The rationale for this approach should be clearly set out in the PIR.

Contextual factors

Due consideration should also be given to other contextual factors when considering the level of resourcing that is proportionate. Secondary considerations which may justify a stronger evidence base (and more substantial resourcing) include:

• whether the measure is high profile or contentious;
• whether the measure is particularly novel or risky or is based on untested assumptions;
• whether there is a lack of established data sources to provide evidence of its impact (so bespoke data collection will be required).

Secondary considerations which may justify a lighter touch review include:

• whether there is limited scope for Government to change its approach to the measure (e.g. where the need to comply with international requirements limits how far the Government can amend how the measure is implemented);
• whether the full impacts of the measure are expected to take a long time to emerge, particularly if this is likely to be after the review period.

Summary

Since there is considerable variation in the scale of expected impacts of measures, the levels of analysis and resources devoted to PIRs are expected to fall along a wide spectrum: from light-touch reviews to significant evaluation projects with bespoke data collection. Figure 1 provides an overview of the type of resourcing that might be deemed proportionate, after the Department has considered the scale of impacts, contextual factors, and the existing evidence base. Annex 1 of this guidance includes some case study examples of recent PIRs, including summaries of the decisions that were taken about a proportionate approach.

Further guidance about proportionality in the delivery of RIAs and PIRs is available from RPC.

Figure 1: Framework for assessing proportionate level of evidence and resourcing

Level of evidence needed	(First consider…) Scale of impacts:	(Then take account of…) Contextual factors that may influence what is proportionate:	(To decide…) What would be proportionate?
Low	Low impact, e.g.: Measures below de minimis (EANDCB less than +/- 5M) Or EANDCB less than +/- £10M Or NPV/NPSV less than +/- £10M	Examples include: Limited scope for Government to change approach to the measure. Long-term impacts, which occur after the review period (so don’t yet justify high evidence but may require a future review)	A light touch, low-resource piece of work which establishes whether or not the measure has broadly achieved its objectives and has met its success criteria; and whether or not there have been unintended effects. - Uses readily-available monitoring data and/or management information for evidence. - Collates evidence of the views and experiences of stakeholders in a time-efficient way. - Is completed relatively quickly. - Is conducted in-house.
Medium	Falls in- between the scenarios above and below.	Falls in-between the scenarios above and below.	If no existing evaluation evidence base is available for the PIR to summarise: a medium-resource piece of work which provides evidence on the outcomes, and ideally the impacts, of the intervention; as well as unintended effects. - Uses some bespoke monitoring and evaluation data if this is needed to fill key gaps in the evidence base. - Involves some formal consultation or interviewing with stakeholders to collect their views and experiences. - Provides estimates of the actual costs and benefits of the measure. - May involve some commissioned primary data collection to fill gaps.
High	High impact, for example with EANDCB, NPV, or NPSV in original RIA greater than +/- £50M	Examples include: Measure is high profile and will attract a high level of scrutiny. Measure involves a novel or risky approach, based on untested assumptions. Where there is a lack of existing data to show impact. Where high need for learnings to inform future policy making and RIAs.	If no existing evaluation evidence base is available for the PIR to summarise: a high-resource piece of work which provides robust findings on the outcomes, and ideally the impacts, of the intervention; as well as unintended effects. Likely to require substantial analytical support. - Uses bespoke monitoring and evaluation data collection to address evidence gaps. - Explains the experiences of stakeholders and how the measure achieves its effects. - Quantifies, monetises and re- estimates the actual costs and benefits of the measure for business and society, as far as possible. - Often involves commissioned work.

2.4. Who should be involved in completing a PIR

Planning of a PIR is often multi-disciplinary, to ensure that there is a proportionate review of the policy context, analytical approaches and practical considerations. Some of the principal groups and professions which may be involved, and the types of input they can provide, are summarised below.

• Policy professionals. As experts in the detail and context of specific measures, these individuals will be able to provide information about the specific details of the policy, the anticipated impacts and the environment that the measure is operating within. They should also be able to advise on any policy questions that may be informed by the evidence gathered in the PIR. A policy professional will usually be the Senior Responsible Officer for delivery of the PIR (while the Permanent Secretary will be responsible for the Department’s overall performance on PIRs). This person will often manage and write the PIR, with support from analysts.
• Regulators. It will be particularly important to seek input from staff in the regulatory body which enforces the measure, where applicable. Regulators can provide a practical perspective on the delivery and effectiveness of the measure, particularly compliance levels and enforcement mechanisms.
• Economists. It is likely that economists will have been involved in the drafting and production of the analysis that features in the IA. They will therefore be able to advise on how to measure impacts that were estimated in the RIA. In some instances, econometric analysis could be helpful in trying to isolate the impact of the regulatory change so that its significance can be better understood. An economist can advise on whether such techniques are appropriate or not.
• Social Researchers. Social researchers are experts in analysing the attitudinal and behavioural aspects of policies and are knowledgeable about a variety of quantitative and qualitative methods of data collection and analysis. They are skilled in developing evaluation strategies and in identifying gaps in the evidence base and providing advice about appropriate ways to fill these gaps.
• Other analysts. Other departmental analysts who may contribute to a PIR include statisticians, who may advise on the statistical robustness of data collection designs and evidence, and operational researchers, who may contribute mathematical modelling of the effects of measures.
• Scientists and Engineers. Advice from scientists and engineers may be required to assess to what extent claims about the effectiveness and efficiency of technology used under the policy are valid.
• Legal Advisers. As some PIRs are required to support legally mandated reviews it may be helpful to liaise with a legal adviser to ensure compliance with legal requirements or to assess interactions between the measure and other legal requirements. Legal advisers may also recommend changes to the legal texts enacting the measure.
• External stakeholders. It may also be helpful to consult external stakeholders about the scope, timing and delivery of the PIR. External stakeholders offer valuable insights regarding impacts and unintended consequences of measures, as well as potential alternative solutions.

2.4. Defining the scope of the PIR

Defining key PIR questions

The PIR provides evidence to assist with answering the questions of a regulatory review. The core questions that all PIRs should seek to address are set out in the PIR template available from BRE:

1. What were the policy objectives of the measure?

• What were the intended effects, and how far were these expected to have been delivered by the review date?

2. What evidence has informed the PIR?

3. To what extent have the policy objectives been achieved?

• What have been the actual costs and benefits of the measure and its effects on businesses, and how do these compare with what was forecast?

4. What were the original assumptions?

5. Were there any unintended effects?

6. Has the evidence identified any opportunities for reducing the burden on business?

A PIR should provide evidence to address these questions and inform the policy options for next steps. Where the RPC reviews a PIR, it will seek to establish that the Department has considered the evidence to these questions.

Figure 2 below sets out how evidence from a PIR (in the first box) can form part of the wider range of questions that a regulatory review might seek to address.

Figure 2: Regulatory review questions

1. To what extent is the existing measure working?

(PIR contributes evidence here)

• To what extent has the policy achieved its objectives?
• To what extent have the success criteria been met?
• Have there been any unintended effects?
• What have been the actual costs and benefits of the policy? How do these compare with the estimated costs and benefits?
• What have been the impacts on small and micro businesses? How do these compare with the estimated impacts?

2. Is government intervention still required?

• What would happen if you removed the measure (has the problem been solved either because of the policy, because the market has changed, or due to other factors)?
• Are the objectives of the measure still valid / relevant?

3. Is the existing form of government regulation still the most appropriate approach?

• What are the likely costs and benefits going forward?
• How likely are unintended effects in the future?
• How effective is the implementation / enforcement mechanism for the policy?
  • To what extent do compliance levels indicate that the enforcement mechanism chosen is inappropriate?
  • What are stakeholders’ views on implementation?
• To what extent would non-statutory measures achieve the same benefits?

4.  If this measure is still required what refinements could be made?

(What scope is there for simplification, improvements?)

Could refinements be made:

• To improve enforcement / compliance levels?
• To simplify implementation to reduce burden on business (including small and micro businesses) and reduce costs?
• To reduce the scope of the policy to take organisations out of the regulatory obligations?
• To otherwise improve implementation to enhance benefits?

5. If this measure is not required, but government intervention in some form is, what other regulation or alternatives to regulation would be appropriate?

Defining objectives and intended outcomes^{[footnote 6]}

One of the first steps to take when planning a PIR is to look for documentation which defines:

• The objectives of the measure;
• How the measure is expected to achieve its objectives; and
• The costs and benefits of the measures (particularly for businesses and CSOs).

The RIA will be the starting point for this information and will set out the assumptions which applied when the measure was planned and implemented. You should also look for subsequent reformulations of this information, noting their dates and any stated reasons for the change.

Collectively, this information will set out the assumptions about how the measure will achieve its objectives. One of the purposes of a PIR is to check the validity of these assumptions.

Where a high level of evidence and robustness has been deemed proportionate, this review of objectives could be enhanced by the production of a logic model. A logic model is a tool that describes how a policy is expected to achieve its objectives. It can be used to support both the RIA and PIR processes by:

• Setting out a measure’s objectives and success criteria in an accessible way;
• Supporting identification of baseline and post-implementation data collection requirements;
• Enabling hypotheses about unintended effects to be developed.

A logic model would typically set out:

• Context – What issue does the measure address and what is the policy environment in which it will operate?
• Inputs – What resources and activities must be committed for this measure to proceed?
• Outputs – What immediately happens following implementation of the measure?
• Outcomes – What results are expected from the measure in the short to medium-term?
• Impacts – What are the longer-term outcomes from the measure including wider outcomes for the economy and society?

These stages can be presented diagrammatically as a sequence, running from left to right, which moves through a sequence of ‘if … then’ reasoning (‘if this input is put in place then this output will follow’). Figure 3 provides a simple example of a logic model for a measure. A more developed logic model would seek to cover all a measure’s long-term impacts and the causal mechanisms that are intended to produce those impacts. A more advanced logic model might also provide detail on assumptions, particularly around the mechanisms through which each stage of the model leads to the next.

Figure 3: Example logic model for a measure (Dartford Crossing Charging Scheme 2013)

Context	Inputs	Outputs	Outcomes	Impacts
The issue that the measure addresses and the policy environment in which it will operate.	The measure and the resources and activities used to implement it.	What immediately follows from the measure, e.g. how the regulated activity proceeds.	Short and medium-term results of the measure.	Long-term outcomes from the measure including wider economic and social outcomes.
Road user charge collected at Dartford crossing since 2023 to manage demand; Department wishes to improve traffic flow.	Free Flow charging scheme (users pay online / by post / by phone rather than at payment booth) implemented and publicised.	Delivery of DartCharge payment platforms; removal of payment booths and barriers.	Road users switch to using DartCharge system; traffic flow at DartCharge improves; journey times decrease.	More efficient movement of people and goods; enhanced business productivity.

Unintended effects

The PIR plan should also take account of the need to capture any unintended effects of the measure. ‘Unintended effects’ is a term used to describe any outcomes or impacts (whether positive or negative) of a measure which were not included in the original RIA as they were not anticipated or intended, including any which arise due to changes in the external environment.

Some unintended effects from recent PIRs include:

• An EU-derived measure which restricted the use of certain hazardous substances in new equipment was found to have the unintended consequence of restricting the supply of second-hand equipment and spare parts.
• An EU-derived measure to develop interoperability on the EU rail system was found in some cases to have been applied in a way which restricted the choice of vehicle which could be commissioned for use on the UK network, potentially resulting in increased costs for operators.
• A measure which set minimum medical standards for driving with diabetes was found to have misclassified some drivers as having ‘severe’ conditions when they actually represented a low risk of danger to themselves and other road users.

Unintended effects on businesses or CSOs are of particular relevance when undertaking a regulatory review.

If proportionate, the PIR could also capture information about any divergences from the intended implementation of the measure. For example, it is standard practice to assume at the RIA stage that there will be 100% compliance with the measure, unless there is evidence to the contrary. At the PIR stage, it may be of use to test this assumption empirically, where it is feasible and proportionate to do so, and flag if compliance is low.

It is important for the PIR both to capture unintended effects and to assess the reasons why they have occurred, so far as this is possible and proportionate. This will likely require the PIR to draw on qualitative data (such as in-depth interviews, or written consultation responses) exploring the views and experiences of relevant stakeholders.

2.5. What to do if PIR preparation begins late

It is good practice for PIR planning to begin at the start of the policy process, when the RIA is drafted. A late start to PIR planning brings risks to the quality of evidence as opportunities for data collection may be missed. If your PIR preparation does begin late, begin by speaking to policy officials and analysts who are involved with the measure and the wider policy area to seek the following types of evidence:

• Objectives and intended outcomes: review published documentation, including the RIA, to identify these.
• Baseline data: check whether the RIA and any associated models provide baseline data. If not, check whether past monitoring data can be used to estimate a baseline position.
• Monitoring data: as well as baseline data, need to check whether there are any data sources providing ongoing monitoring after the implementation of the intervention which will allow post-implementation outcomes from the measure to be captured. (This may well overlap with data sources used for baseline data.)
• Stakeholder feedback: collect retrospective feedback from policy officials, regulators and key stakeholder groups about their experiences of the effects of the measure. Review available evidence from consultations or other channels of engagement with stakeholders. As well as checking whether objectives and intended outcomes were realised, probe for whether there were any unintended effects.

The main disadvantage of carrying the process out retrospectively is that you may miss evidence which would allow the baseline position and the building of impacts following enactment to be understood. Where lack of availability of data constrains the conclusions that can be drawn from the PIR, this should be explained in the PIR.

In situations like this, the RPC can offer advice and support, so early engagement is recommended.

3. Methods for delivering a PIR

This chapter explains some of the most common approaches for delivering a PIR. It will cover some of the methods that are likely to be proportionate for a light touch PIR, before going on to set out some of the approaches that may be more proportionate for PIRs where more substantial evaluation is required. Specifically, this section provides guidance on:

• monitoring data
• stakeholder engagement
• evaluation evidence

3.1. Monitoring data^{[footnote 7]}

Monitoring allows progress in the delivery of a policy to be tracked by collecting data on its outputs. Monitoring data is likely to make an important contribution to the evidence for a PIR. It can take a number of forms including performance measures, benefits management data and other administrative or management information.

Monitoring data can provide a relatively light touch evidence base for a PIR, which is particularly useful in cases where it is not proportionate to undertake substantial primary data collection or additional analysis. In some cases, monitoring data relevant to a measure may already be captured on a regular basis. In other circumstances, it will be necessary to plan ahead to ensure that the data that will be useful for a PIR is captured.

Planning for the collection of monitoring data and baseline measures

The key tasks when planning the collection of monitoring data are:

• To liaise with policy officials, regulators and analysts to agree key indicators that can be monitored to assess the extent to which the impacts in the RIA are being achieved
• To assess the extent to which monitoring systems can be developed to collect relevant, timely and reliable data
• To develop and agree a data collection plan, including timeframes for when the data will be collected and reviewed (recognising the likely need for baseline data), and by whom

By planning monitoring requirements at an early stage, it may be possible to build collection of monitoring data into policy, or statutory or regulatory instruments. Depending on the circumstances, data may be collected in a continuous process or in a series of points of time (e.g. annually).

As far as possible, monitoring data should be collected during a baseline period before implementation of the measure, as well as during the post- implementation period, so that trends in key measures can be understood.

Where third parties are involved in delivery of the policy, it is important to ensure that requirements for the collection and provision of relevant monitoring data are clearly defined and built into contracts.

In some cases, it will be advantageous to collect individual-level data to monitor the impacts of the measure on the people who will be affected by it. Appropriate data protection arrangements will need to be put in place at the outset to facilitate such uses of the data.

Post implementation

After monitoring plans have been established and baseline data has been collected, monitoring activities should continue after the policy has been implemented to measure whether the key indicators are changing in the way that was anticipated. It will usually be helpful to have regular monitoring in place so that time series data will be available to show change in the levels of outcomes over time. For some measures, it may take time for impacts to become clear and, conversely, the effects of the measure may not become apparent for some time. The measure’s RIA should provide an indication of when the full effects are expected to be realised.

It is best practice, particularly where a measure is particularly novel, controversial or risky, to monitor the impacts of the measure on an ongoing basis (prior to the preparation of the PIR) to collect evidence of any early issues and emerging concerns relating to the regulatory framework. As well as informing the subsequent PIR, such early evidence can be used to amend or revise the methods of implementation of the measure or the level of support provided to key stakeholder groups that are affected by it.

3.2. Stakeholder engagement

One of the limitations of monitoring information is that (for the most part) it will tend to focus on the impacts that were already anticipated to arise from the measure in the RIA. While monitoring can usefully measure key indicators, it may provide limited detail about why these changes have come about and may not be able to address questions about unintended effects. For this reason, it will often be helpful to supplement monitoring information with stakeholder engagement activities, in order to understand the views of those businesses or CSOs who are directly affected by the measure. Consulting with relevant stakeholders can also help to generate support and buy-in for your approach to the PIR, enhancing credibility when the findings are published. It may also be possible to gain access to other data sources that stakeholders have access to (for example management information).

If a high-evidence PIR has been deemed proportionate, stakeholder engagement would likely be delivered by specialist social research analysts or form part of a wider programme of evaluation. Evaluation evidence is covered in greater detail in Section 3.3, so the remainder of this section will focus primarily on light touch approaches to stakeholder engagement.

The key tasks when planning engagement with stakeholders are:

• To decide on the type of stakeholder engagement that will be most appropriate to meet the PIR’s evidence needs.
• To liaise with policy advisors, regulators and analysts to identify the most relevant stakeholders that should be considered for inclusion in the engagement exercise.
• To develop and agree the questions that stakeholders will be asked; which will depend upon the kind of stakeholder engagement that is judged most appropriate.

Types of stakeholder engagement

The type of stakeholder engagement that will be most appropriate will depend on the evidence needs of the PIR. For a light touch review, qualitative approaches are often a proportionate way of addressing common evidence gaps around unexpected outcomes and stakeholder perceptions of the measure.

Qualitative data from individual stakeholders could be collected through written consultations or qualitative interviews (conducted in person or via telephone). Where it is appropriate to bring together larger groups of stakeholders, workshops or focus groups could be used. Qualitative data can be analysed thematically to provide a rich, in-depth insight into questions that monitoring data alone cannot address. However, it is important to note that qualitative information cannot be used to measure the strength or prevalence of perceptions or impacts discussed, nor can it draw wider inferences about a population beyond those directly involved in the engagement.

In circumstances where a robust sample design can be achieved (further details are provided below), a questionnaire could be administered to measure indicators that are not available through monitoring data. It should be noted that for a light touch review, it can be challenging to obtain robust quantitative evidence from stakeholders within the constraints of time and budget.

Where possible and proportionate, bringing together a combination of both qualitative and quantitative evidence is often the best approach to help address some of each methods’ respective limitations. For example, providing measurement of key themes emerging from qualitative information with a survey; or exploring the possible reasons behind quantitative findings with an in- depth interview.

Identifying stakeholders for engagement

After the type of engagement that will best address the PIR’s evidence gaps has been decided, the next step is to identify the stakeholders who should be invited to participate. Policy advisors, regulators and analysts involved with the measure may all be able to indicate specific organisations and individuals who will be affected by the measures or are stakeholders more broadly. After an overview of the relevant stakeholders has been confirmed, a decision needs to be taken about the ‘sample’ of stakeholders who should be involved in any engagement.

When undertaking qualitative stakeholder engagement, it is most common to use a purposive sampling approach. This approach involves purposefully (rather than randomly) selecting participants, often with the intention of ensuring a broad representation that does not exclude any important sub-groups (e.g. ensuring that small and micro businesses are included; or that both supporters and critics of a controversial measure have the chance to offer their views). For a measure with a particularly narrow scope, it may instead be about selecting the individuals most likely to be directly affected, or with the greatest expertise in the area. When undertaking this type of sample design, it is important to clearly explain the rationale for selection in the final PIR report.

Where a quantitative approach is used, it is important to consider how to achieve a representative sample. Quantitative analysis is designed to draw inferences about the wider population, and a low response rate increases the risk of bias in the data (e.g., that certain sub-groups are under-represented). Particularly for light touch reviews, taking a quantitative approach may pose a risk if time and resources are too limited to enable a representative sample to be obtained. If the number of stakeholders is low, it may be appropriate to take a ‘census’ approach. This approach involves inviting and, as far as possible, securing responses from the entire group of stakeholders.

Whatever means of engagement is used, during analysis it will be important to recognise any key gaps within the group of participants who have been included. Further details are provided in Section 4.3 dealing with how to address limitations in the PIR when writing the report.

Designing stakeholder engagement questions

Whatever approach is taken, it will be important to design a consistent set of questions that allow for comparison between stakeholders, while clearly targeting key research questions and gaps in the existing evidence base. It is likely to be advisable to seek guidance from social researchers or others with experience of designing research questions at this stage.

One of the key principles is to ensure that questions do not lead participants towards certain conclusions or otherwise bias their answers. For qualitative engagement, questions should be open-ended and allow time to follow up on new points of information that cannot be anticipated when questions are developed (e.g. about unintended effects of the measure or the reasons why particular effects occurred). Although open-ended questions can be used in a quantitative questionnaire to capture these kinds of responses, careful thought should be given to the amount of resource available to process these responses. It may not be proportionate for a light touch PIR that would instead benefit from a smaller number of more in-depth qualitative engagements.

If a more resource-intensive approach to stakeholder engagement is to be used, it would be advisable to consult more in-depth guidance about appropriate question design.

After stakeholder engagement begins, make sure time is allowed for stakeholders to consider their views, especially if you are speaking to membership organisations who will need to gather the views of their members. Be clear about the purpose of the engagement, the nature of contribution requested, and timescales. Stakeholders may suffer from “consultation overload” and will need to take decisions about which engagement activities to prioritise.

3.3. Evaluation evidence^{[footnote 8]}

For measures where a more substantial PIR has been judged proportionate, it may be necessary to draw upon more resource-intensive evaluation evidence. This section describes some of the main types of evaluation which could be used for PIRs and highlights some issues to consider when deciding what is appropriate.

As set out in Section 2.2, in the first instance it will be important to understand the range of existing evaluation activity planned for the measure and the wider policy area. Where an evaluation programme is already in place, it may well be sufficient for a PIR to summarise this evidence. Measures that are high profile, novel, or contentious are more likely to have existing plans for an evaluation in place than those where a light touch review is more proportionate.

If a programme of evaluation activity is not already in place, and a light touch PIR has been judged to be insufficient, it will be important to involve analysts with evaluation expertise in the PIR process early (ideally at the planning stage) as they will be able to provide advice on which methods are the most appropriate for answering specific research questions.

The main types of evaluation that might inform a more substantial PIR are briefly summarised below. Magenta Book should be consulted for further details.

Impact evaluation

Impact evaluations address questions like:

• What outcomes (positive and negative) did the policy have?
• What impact did the policy have relative to other factors in generating those outcomes (over and above what would have happened without the policy?)
• Were there any unintended effects?

Robust impact evaluations will use analytical techniques to provide a measure of the extent to which changes in key outcomes were caused by the intervention. This approach relies on comparing what happens under the measure with estimates of a counterfactual position where the regulatory change did not occur.

However, because most measures are rolled out across an entire regulated population at once, in the regulatory context it can be particularly challenging, or infeasible, to identify a “non-regulated” comparison group which is directly comparable to the “regulated” treatment group. The Magenta Book contains guidance on the range of options that are available to deliver a robust evaluation in these circumstances (e.g., it may be more appropriate to consider a theory- based evaluation if there is sufficient analytical resource available).

Because of the complexity of these approaches, the most robust evaluation designs are likely to require dedicated analytical support and sufficient resource to deliver an evaluation programme. It is likely that a PIR report would summarise the findings of such an evaluation, referring to the detailed findings and methods contained in the evaluation report.

Process evaluation

Process evaluations address questions like:

• How was the policy delivered/implemented? Was this as intended?
• What aspects of the policy have worked more or less well, for whom and why?
• How is the policy achieving its impacts? How does this vary for different types of stakeholders or contexts?

Economic evaluation

Economic evaluations address questions like:

• What are the costs and benefits?
• Did the benefits of the policy justify the cost?

Two common methods are cost-effectiveness analysis and cost-benefit analysis.

The types of evaluation set out above are not mutually exclusive and it is often appropriate to use them in combination. For example:

• Economic evaluation is often based on the results of an impact evaluation, to ensure that robust estimates of changes to costs and benefits are used in the analysis.
• Process evaluation is often used alongside impact evaluations, to help understand why particular outcomes and impacts occur and how they are experienced by people.

4. How PIR evidence should be used

This chapter provides guidance on how to use the evidence from a PIR to support the analysis of options. It also discusses important considerations for writing up PIRs, and highlights the potential future uses of PIR evidence.

4.1. Possible options resulting from a PIR

For domestic measures, the owning Department should use the evidence provided by the PIR to support a decision about whether the instrument should be renewed, amended, removed, or replaced. Figure 4 outlines what these options mean.

Figure 4: Possible options resulting from a PIR

Option	Description
Renewal	Measure continues without amendment or, where a sunset clause applies, is renewed
Amendment	Measure remains but marginal changes are made to improve its objectives or implementation arrangements
Removal or expiry	Measure is removed (or expires) without replacement
Replacement	Measure is replaced or redesigned substantially

There is no hierarchy between these possible options and no presumption of which should apply, except in the case of sunset clauses (where the presumption is that the measure should expire unless a case is made for a different option).

4.2. Developing options

By addressing the main PIR questions (see Section 2.4), the PIR can provide evidence that supports the choice between the options set out in Figure 4.

Figure 5 sets out a guide to how the analysis in the PIR might inform the choice between the possible options. In addition to this evidence framework, decisions about options will need to reflect political and strategic choices.

Figure 5: Possible options resulting from a PIR and the circumstances that support them

Option (label)	Supporting circumstances / evidence
1. Measure should remain as is (renewal)	The policy is on course to achieve most or all of its objectives and key success criteria have been met, and: - costs[footnote 9] have been proportionate to benefits - government intervention is still required – (if intervention was withdrawn the problem would return) - compliance levels are sufficient to support achievement of objectives - alternatives that are less burdensome to business and/or overall have not been identified
2. Measure should remain, but implementatio n should be revised or improved (amendment)	The policy is achieving most or all of its objectives, and success criteria have been met, but amendments to the implementation process could help to: - achieve further benefits; - reduce costs or burdens on business and/or overall; - simplify the implementation processes; - improve compliance; - reduce unintended or negative effects. Considerations (for the PIR and any new RIA): - consider associated transitional costs - set out the revisions proposed, who will be consulted, what are the implications of the revisions, and what new guidance will be issued
3. Measure should be removed without replacement (removal or expiry)	Where one or both of the following applies: - The policy is not, or is no longer achieving most of its objectives or key success criteria - costs are disproportionate compared to benefits AND one or both of the following applies: - compliance levels are insufficient to support achievement of its objectives and are unlikely to be improved - Government intervention is no longer required (the original policy objectives are no longer relevant or it is clear that if the intervention was withdrawn the problem would not return)
4. Measure should be replaced or redesigned (replacement)	If the policy is not or is no longer achieving most of its objectives or key success criteria and/or: - costs are disproportionate compared to benefits - compliance levels are insufficient to support achievement of its objectives and are unlikely to be improved AND: Government intervention is still required to address the problem AND: The same or better performance could be achieved using a different measure or alternative to regulation, which: - costs less - creates less burden on business and/or overall - creates fewer negative impacts - increases benefits

In all the above instances, the PIR must clearly set out the evidence available about costs and benefits. It must compare this evidence with the estimated costs and benefits from the RIA and an explanation provided for any substantial differences(see PIR template).

The strength of the support for options that can be drawn from a PIR will be linked to the quality and comprehensiveness of the evidence on the PIR questions. Where a lighter touch approach has been used to assess impacts of a measure, appropriate caveats should be included in the report and more caution may be required in arriving at a recommendation.

4.3. Writing the PIR report

For transparency and consistency of reporting, it is recommended that a summary of the findings, conclusions and recommendations of a PIR should be set out in the PIR template available from BRE.

It is for Departments to decide on the best way to present the detailed findings. However, all PIRs should include a summary of the PIR methodology and a critical review of the coverage and quality of the evidence and data collection that has informed the PIR.

The design of a PIR will generally involve analytical decisions that it will be important to explain in the final report. Departments should provide a rationale for the approach they have taken for the PIR within the document. Where PIRs are reviewed by RPC, this will help the RPC to assess whether or not the analysis in a PIR is fit for purpose. Early engagement with the RPC on this matter can help ensure that expectations are aligned.

In particular, the report should state the rationale for:

• Proportionality of approach: The report should clearly justify the level of resourcing used to address the PIR questions. In doing so, they may find it helpful to refer to the guidance here on assessing what level of evidence is proportionate (in Section 2.2).
• Methods used: The report should explain why the selected methods were adopted, and potentially explain why alternative options were ruled out.
• Sample design: Particularly for light touch reviews where stakeholder engagement forms a substantial part of the evidence base, it will be important to set out how decisions were made about who would participate, and what selection criteria were used.

The PIR should note any limitations on the evidence collected. Some of the most common issues that are likely to need acknowledging are listed below:

• In the absence of an estimated counterfactual or comparison group, it may not be possible to attribute changes in outcomes to the influence of the measure with certainty. The report should acknowledge that external factors may have influenced these outcomes, identifying any such factors that are believed to be relevant.
• Where qualitative stakeholder engagement has been undertaken, care should be taken if making generalisations to a wider population. Qualitative research can provide a rich understanding about the experiences of those who participated, and this may highlight themes that apply more widely, but this evidence alone will be insufficient to measure the prevalence or strength of these perceptions. In particular, it would be inappropriate to quantify these findings (e.g. reporting percentages or counts of participants who held a certain view).
• Caution should also be taken about making inferences to a wider population when using quantitative stakeholder engagement that is unable to credibly claim a representative sample. Apart from where a census approach with a small group of stakeholders has been achieved, most light touch quantitative research will not be able to do so.
• Additionally, where any form of stakeholder engagement or consultation experienced non-participation or drop-outs, it will be important to acknowledge the potential role of selection bias (e.g. that those who did not participate may represent groups or attitudes not included in the sample of those who did respond).

The PIR should also record any significant lessons for the RIA process arising from the evidence, for example where costs or benefits had been found to be misestimated.

4.4. Clearing and publishing the PIR

After a PIR has been finalised and internal approval and peer review processes have been followed, the Department should follow BRE and RPC guidance on next steps in clearing and publishing the PIR. The BRE’s principles of best practice provides detailed guidance on these steps.

4.5. Future uses of PIR evidence

A good understanding of the costs and benefits of interventions and learning about implementation and delivery mechanisms provides a valuable evidence base for future policy making and resource allocation. Policymakers should be able to use individual PIRs, as well as syntheses of multiple PIRs, to learn and apply lessons about what has worked well and what worked less well in different contexts.

It is also important that findings from PIRs feed into the appraisal process. RIAs use a wide range of data and evidence but make assumptions where data or evidence is not available. PIRs can help to improve assumptions in future RIAs by filling gaps in the evidence base, with the result that future RIAs are stronger and give more accurate estimations of impacts. Referencing PIR and broader evaluation findings in RIAs is a good way of demonstrating that the findings of relevant PIRs or syntheses of PIR have been considered.

Departments should also learn from experience in completing PIRs to improve their planning and delivery of future PIRs. If PIR planning is done promptly and thoroughly, the PIR report is much more likely to provide a robust analysis of the measure’s impacts. But if PIR planning is not started well in advance, opportunities to use appropriate impact evaluation methods or to collect monitoring data for the measure can be missed. Departments should ensure their systems, processes and resources are focused on early planning.

Key references

BEIS Better Regulation Executive (2018) Producing post-implementation reviews principles of best practice

BEIS Better Regulation Executive (2018) Business regulation: producing post- implementation reviews

BEIS Better Regulation Executive (2018) Better Regulation Framework Guidance

Gov.uk. (2019). Regulatory Policy Committee Official Website

Regulatory  Policy   Committee   (2019).  RPC   case   histories - post-implementation reviews

Annex 1: PIR Case Studies^{[footnote 10]}

Case Study A

Measure: Channel Tunnel (Safety) (Amendment) Order 2013

The Channel Tunnel (Safety) Order (CTSO) 2007 established a common regulatory framework for railway safety in the Channel Tunnel and was amended by the Channel Tunnel (Safety) (Amendment) Order 2013 to reflect EU Directives introduced since 2007.

This measure had 4 policy objectives:

1. To bring the Channel Tunnel into line with provisions in place in the UK and France;
2. To achieve consistency of rail vehicle maintenance across the European Union;
3. To assist railway undertakings to better control risks and costs through an appropriate maintenance regime; and
4. To reduce duplication, costs and delays in safety authorisation.

Level of evidence: Low evidence

A low evidence approach was considered proportionate for the PIR, as the measure primarily affected government rather than business. Equally, the original RIA had forecast negligible costs and benefits for the measure because of the small number of operators affected and the minor nature of the changes.

Method used: Management information and light-touch qualitative stakeholder review

Because of the low evidence requirement for the PIR, one of the key sources informing this review was the available management information detailing the implementation of the changes. These covered the frequency of authorisations to railway undertakings, the general safety performance of the tunnel against common safety indicators and other records documenting the measure’s implementation.

In addition, further primary evidence was gathered through a light touch qualitative stakeholder engagement exercise. The five stakeholders directly affected by the measures were purposively prioritised for engagement in this review. Three of these provided responses using a written pro-forma to answer questions about the performance of the measures. While it would have been ideal for all five directly affected parties to respond, the low evidence requirement of the PIR meant that it was not deemed proportionate to continue pressing for responses after the deadline had passed.

PIR’s recommendation: Renewal

Overall, the available management information and the responses to the consultation showed that the measure’s immediate objectives had been achieved (e.g. the relevant safety authorisation processes had been developed and implemented). The stakeholder consultation was able to identify areas where the provisions could be implemented more effectively – but no alteration to the measure was required to achieve these.

Feedback from RPC: N/A

As this PIR fell below the de minimis threshold, it did not require review by RPC.

Case Study B

Measure: Biocidal Products and Chemicals (Appointment of Authorities and Enforcement) Measures 2013 (BPC)

The Biocidal Products and Chemicals (Appointment of Authorities and Enforcement) measures 2013 (BPC) is a Statutory Instrument providing supporting domestic legislation for the Biocidal Products measure (BPR), Classification, Labelling and Packaging for substances and mixtures measure (CLP) and Prior Informed Consent (PIC) Measure covering enforcement arrangements and the appointment of competent authorities (CAs) (for BPR and CLP) and designated national authorities (DNAs) (for PIC).

This aims of BPC were for the UK to meet the EU requirements set out in the Biocidal Products measure (EU) 528/2012 (BPR), simplify and streamline domestic arrangements and meet a recommendation in the Löfstedt review of Health and Safety Legislation to consolidate biocides sectoral legislation.

Level of evidence: Low evidence

The expectations in the impact assessment (RIA) were that the costs to business would be negligible, in terms of familiarisation and changes in fees. The full RIA subsequently estimated that there would be no net cost to business. This PIR was signed off on 18 June 2013 and published. No compelling evidence was found in this review to contradict the RIA and the PIR therefore fell under the de minimis threshold for automatic submission to the RPC.

Method used: Review of administrative data and consultation with stakeholders

There was sufficient documentary evidence available to demonstrate that the measures achieved the aims of meeting the requirements in BPR on enforcement, penalties, CAs/ DNAs and biocides cost-recovery mechanisms; meeting the Löftstedt recommendation to consolidate biocides legislation, simplifying and streamlining domestic legislation; making PIC enforcement more proportionate; making biocides fees transparent and predictable and to making biocides fees compliant with Managing Public Money (PDF, 1,544KB

To identify any unintended consequences and understand how the implementation of the measures could be improved, HSE also sought qualitative evidence from businesses affected by the measures, reviewed relevant data sources and consulted with enforcing authorities.

PIR’s recommendation: Renewal

The measure was assessed as having achieved its objectives. Although a small number of issues in enforcement were noted by HSE stakeholders, the small number of breaches identified are predominantly technical. Based on the risk gap and relative priority therefore, HSE did not propose changes in the BPC legislation beyond those essential to EU exit.

Feedback from RPC: N/A

As this PIR fell below the de minimis threshold, it did not require review by RPC.

Case Study C

Measure: A282 Trunk Road Dartford-Thurrock Crossing Charging Scheme Order 2013

The Dartford-Thurrock River Crossing is a key part of the strategic road network, forming a link in the M25 motorway between Essex and Kent. A road user charge has been collected at the crossing since 2003 as a way of managing the high demand for its use. The Dartford ‘free-flow’ charging scheme (Dart Charge) was introduced on 30 November 2014. Road users were now able to pay the charge either online, by phone, at a payzone retail outlet or in advance by post, rather than needing to stop and pay during the crossing.

The main objectives of the measure were to:

1. improve traffic flow and reduce congestion at the crossings, and;
2. continue collecting charges, maintaining cumulative cash flows estimated to 2039.

Level of evidence: High evidence

A high evidence PIR was considered proportionate, as the measure had been forecast to result in a net impact of £1,629m. Equally, the novel design of the scheme provided opportunities for learnings that could inform future scheme development.

Method used: Review of existing evaluation evidence and light-touch stakeholder review

The presence of existing monitoring and evaluation arrangements through Highways England’s Post-Opening Project Evaluation (POPE) programme provided a strong foundation for this PIR. The POPE evaluation provided before and after assessment of:

• journey time and journey time reliability
• forecast costs and benefits vs. outturn costs and benefits

To supplement the POPE report, the Department undertook primary stakeholder engagement. Because of the comprehensive nature of the POPE report, it was not proportionate to undertake an extensive representative survey. Instead the priority was to provide a channel through which interested parties could make their views known. Ultimately, 7 business users and 42 individuals responded to the surveys. The PIR also reviewed other evidence about stakeholder views, including a local petition which had received a Government response.

PIR’s recommendation: Renewal

The POPE report provided evidence showing that key shorter-term objectives had begun to be achieved as of April 2018 when the review was produced. Average journey times had improved in both directions. The average time savings were lower than initially forecast, but this was attributed in part to extended M25 roadworks during 2015. Net proceeds for the Dartford Crossing had increased since the introduction of Dart Charge, and so the scheme was considered to have been successful in maintaining cash flows.

Ultimately, the PIR recommended that the measure should be retained.

Feedback from RPC: Green rating

RPC provided positive feedback about the timely planning of the evaluation, with Highways England’s POPE report able to set an appropriate baseline. RPC suggested it might have been preferable to undertake a qualitative approach to stakeholder engagement, such as focus groups or ‘town hall’ meetings, given the scale of the measure. But RPC acknowledged that the PIR made good use of other sources of evidence as to local feeling to bolster understanding of stakeholders’ views.

Case Study D

Measure: The Reports on Payments to Governments Measures 2014

Natural resources often provide substantial income for developing countries, however governments of these countries can fail to appropriately handle the large payments they receive from extractive sector companies. A potential solution to this issue is increased transparency around payments made by extractive entities to governments. This measure was predicted to have two positive effects: improving citizens’ ability to hold their governments to account; and improving the ability of extractive entities to assess risks associated with extraction. This measure implemented Chapter 10 of the EU Accounting Directive, requiring all large extractive companies in the UK to report publicly payments made to governments in the countries where they undertake extractive and logging operations.

Level of evidence: High evidence

A high evidence PIR was considered proportionate, as the RIA forecast a cost to business for the first year of implementation of £19.7 million, and a total cost of £69.8 million over 10 years. Benefits were hard to accurately assess, but also predicted to be substantial.

Method used: Extensive stakeholder review

BEIS commissioned PricewaterhouseCoopers (PwC) to assess the cost borne by companies in scope, and the benefits that accrue to these companies, their respective investors, and CSOs with an interest in this measure. Responses were collected from companies, investors and CSOs via telephone interviews, an interactive PDF form, face to face interviews, and written submissions. Of the 91 companies that submitted reports to Companies House and/or the Financial Conduct Authority, 32 participated in the research; and 9 CSOs from the Publish What You Pay campaign participated in interviews; investors were largely unwilling to participate, which may reflect low awareness of this relatively new measure.

PIR’s recommendation: Renewal

The general view among companies and CSOs was that, given the timing of the measures, the full benefits were unlikely to be realised at the time of the research (one year after the first reports were published). However, there was an overall view from companies and CSOs that benefits to all stakeholders would accrue over the medium- to long-term, and CSOs provided examples in which the reports were being used to benefit governments and citizens.

It was concluded that the measure is on course to achieve its objectives, as key success criteria have been met in terms of greater levels of transparency, increased compliance levels, and avoidance of unnecessary costs to business. In the medium to long term, the benefits of the measure are likely to outweigh the costs imposed by them (estimated aggregate cost of compliance for all companies in scope is £52.5 million). Further company reporting, and experience of the requirements was considered necessary before any final conclusions of the effectiveness of this reporting regime could be drawn.

Feedback from RPC: Green rating

The RPC provided positive feedback stating that the PIR included sufficient evidence to support the retention of the measure at that time. The research was judged to have provided useful information on business costs. The PIR could be improved by being more measured in its positive assessment of the impacts at this early stage and by explaining when more substantial conclusions could be drawn.

Case Study E

Measure: Control of Asbestos Measures 2012 (CAR 2012)

The main objective of the Control of Asbestos Measures 2012 (CAR 2012) is to set out a framework for preventing exposure to asbestos from work activity or reducing exposure as far as is reasonably practicable. The core requirements of CAR 2012 include assessing the risks from asbestos; putting in place measures to prevent exposure and prevent the spread of asbestos; providing appropriate work equipment; and providing information, instruction and training to workers.

Level of evidence: High evidence

A high evidence review was considered appropriate given that the measures are considered to be high impact, high profile and were expected to attract a high level of scrutiny, particularly from stakeholder groups. Additionally, the Department had not produced a definitive RIA that captured all the costs together and the usefulness of the data in the different RIAs that were produced was limited. Monetisation of costs and benefits was therefore considered to be an important gap in the evidence base that the PIR should address.

Method used: Range of evaluation approaches and stakeholder engagement

Extensive stakeholder engagement was undertaken, combining different methods of data collection (e.g. workshops, focus groups, and an online survey) across a range of groups with different duties under the Measures. Additionally, there was a light-touch communications and engagement strategy to gather views from the wider asbestos community. Other elements of the review included analysis of HSE enforcement data, a questionnaire to labour Inspectorates in other EU member states and desk-based research to compare how the measures had been implemented in other EU countries. To assess impact of the measure, the Health and Safety Executive’s (HSE) mesothelioma projection model illustrated the potential scale of health benefits that overall regulatory changes (encompassing a range of specific control requirements) may have had up to now and may have in the future. HSE research on the Costs to Britain of Work Related Cancer was used to monetise the benefits to society of preventing cancer deaths.

PIR’s recommendation: Renewal

The collective evidence supporting the PIR showed that the measure was achieving its intended objectives and that those objectives remained valid. The stakeholder consultation identified recommendations for changes to administrative guidance and processes to provide greater clarity for duty holders and potentially reduce the burden on business. The recommendations included providing greater clarity around the distinction between licensable, non- licensable and notifiable work and alignment of frequencies for medical examinations for licensable and notifiable work.

Feedback from RPC: Green rating

The RPC provided positive feedback stating that this PIR was comprehensive and commensurate with the high impact and profile of the measure. It was noted that there were some areas where the PIR could be clearer or strengthened. The low response rate to some of the questionnaires was noted. The RPC suggested that sensitivity analysis on the appraisal period and discount rate assumptions would have strengthened the analysis.

1. This term is used throughout this guide to cover all rules or guidance with which failure to comply would result in the regulated entity or person coming into conflict with the law or being ineligible for continued funding, grants and other applied for schemes. ↩

2. Gov.uk.(2019) How the RPC works with departments and submissions [Accessed 8th November 2019] ↩

3. Section 2.5 sets out what to do if opportunities for early planning are missed and you have to start a PIR at the end of the regulation lifecycle. ↩

4. If multiple measures are related and the timing of their review obligations is coincidental, it may be most efficient and helpful to cover them in a single PIR. If you wish to do this, you should explore this option with the RPC at an early stage of planning. ↩

5. Or estimated business and societal impacts: net present value (NPV) / net present social value (NPSV) ↩

6.  Magenta Book 2020, Chapter 2.2 – Evaluation Scoping ↩

7.  Magenta Book 2020, Chapter 4.2. – Data Collection Methods ↩

8. Magenta Book, 2020, Chapter 3 – Evaluation Methods ↩

9. Costs should be interpreted broadly to include indirect and direct, monetisable and non-monetisable, and known and unintended costs and impacts. ↩

10. See also RPC case histories - post-implementation reviews, March 2019: https://www.gov.uk/government/publications/rpc-case-histories-post-implementation-reviews-march- 2019 ↩