Research and analysis

Plan for monitoring and evaluating the new pro-competition regime for digital markets

Published 29 November 2024

Introduction

1.1 This document sets out the government’s plans for monitoring and evaluating the new pro-competition regime for digital markets. It details the monitoring and evaluation (M&E) activities that will be undertaken by the Competition and Markets Authority (CMA) and the government.

The new pro-competition regime for digital markets 

1.2 The digital sector contributes a significant amount to the UK economy. In 2022 the sector contributed £158.3 billion to the UK economy in Gross Value Added (GVA),^{[footnote 1]} and accounted for approximately 1.9 million filled jobs^{[footnote 2]}.

1.3. However, there is compelling evidence that digital markets have become increasingly concentrated, with a small number of large, global tech companies becoming dominant. Whilst the size and presence of these firms is in itself not a concern, the concentration of entrenched market power amongst these firms can undermine effective competition, restrain growth and innovation, and harm consumers.

1.4 To address these concerns, the government has introduced a new pro-competition regime for digital markets as part of the Digital Markets, Competition and Consumers Act. Central to the new regime is the creation of the Digital Markets Unit (DMU) within the CMA. It will be given the regulatory powers required to proactively drive more dynamic markets and prevent anti-competitive and harmful practices that restrict innovation and growth.

1.5 The measures in the regime will be targeted at a small number of firms designated as having ‘Strategic Market Status’ (SMS) in respect of specific digital activities. The regime will have the following tools:

• Conduct requirements (CRs) – The DMU will be able to set tailored rules for each SMS firm. These will help manage the effects of market power and ensure markets are open to competition and innovation.
• Pro-competition interventions (PCIs) – The DMU will be able to implement targeted interventions to tackle the sources of SMS firms’ market power.
• Merger reporting – SMS firms will be required to file a report with the CMA before completing certain mergers that could harm UK competition and consumers.

Overview of this monitoring and evaluation plan

1.6 This M&E plan has been produced by the government. The CMA has also been involved in developing this planned approach to M&E, including its role in monitoring and evaluating the regime. Monitoring involves the collection of information, and evaluation involves an objective assessment of an intervention.

1.7 This document has drawn upon existing work on this subject, including the regime’s M&E framework document,^{[footnote 3]} Impact Assessment^{[footnote 4]} and relevant government guidance.^{[footnote 5]}

1.8 It details the objectives and planned activities across the three main types of evaluation: process, impact and value-for-money.

Purpose of this monitoring and evaluation plan

1.9 There are a number of reasons for putting in place plans for M&E prior to the implementation of the regime:

• To ensure that useful information is collected, providing a strong evidence base to assess whether the regime has achieved its objectives. M&E will also enable the government and the CMA to identify any lessons and apply these to current and future interventions.

• This plan will help ensure that roles and responsibilities for M&E activities across the CMA and government are clearly defined. For the government this includes understanding whether the regime is fit for purpose and meeting its objectives. For the CMA this includes monitoring the impact of the regime and ensuring it can be held accountable for its work.

• The government has committed to undertaking a Post-Implementation Review (PIR) of the regime, which will include an assessment of the effectiveness of the regime against its objectives. This usually takes place three to five years after the intervention comes into force and on a five-year cycle thereafter.^{[footnote 6]} This plan ensures that processes are in place to assist with undertaking the PIR, including information gathering.

• The CMA is committed to taking an evidence-based approach in operating the regime^{[footnote 6]}. This plan will help establish processes that assist with this.

Process monitoring and evaluation

Objectives and evaluation questions

2.1 Process M&E will consider how effectively the regime has been designed, implemented and delivered. It will seek to answer the following overarching evaluation questions:

• How effectively were the regime and its tools designed?
• Was the regime adequately resourced?
• Have key regime activities operated effectively and as planned?
• To what extent has the regime operated in line with its initial objectives?
• How has the context, including external factors, influenced delivery of the regime?
• What improvements could be made to the delivery of the regime?

Process monitoring and evaluation activities

Publication of output metrics – CMA Annual Report

2.2 The CMA plans to publish information about the output indicators of the regime in its Annual Report and Accounts, which are laid before Parliament. These will capture the work of the regime in each year and assist with assessing and monitoring the operation of the regime.

2.3 The CMA Annual Report and Accounts will include the following output indicators:

• Number of SMS investigations launched;
• Number and breakdown of active SMS designations (i.e. which firms, which activities, how long left);
• Number of CRs imposed;
• Number of PCIs imposed;
• Number of breach investigations undertaken (if any);
• Enforcement measures undertaken (if any);
• Commitments agreed (if any);
• Number and breakdown of stakeholder meetings;
• Number of reported mergers, and proportion that proceed to investigation;
• Number of fines for unreported mergers;
• Total Annual Gross Expenditure on the DMU (within the Resource Accounts);
• Levy charged (within the Trust Statement);
• Fines/penalties income imposed (within the Trust Statement).

Ongoing internal monitoring

2.4 The government and CMA will conduct ongoing internal analysis to monitor and assess the operation of the regime and assist with answering the overarching evaluation questions. This may include methods such as interviews, surveys, case studies, and document reviews.

2.5 The government and CMA will ensure, where appropriate, there is coordination and information sharing in relation to their respective internal monitoring activities and the findings from these.

Process evaluation in the Post-Implementation Review

2.6 The government has committed to conducting a PIR of the regime. This will include an assessment of how the regime is operating and seek to answer the overarching process evaluation questions.

2.7 It will draw upon the output metrics published in the CMA Annual Report and findings from ongoing internal monitoring.

Impact monitoring and evaluation

Objectives and evaluation questions

3.1 Impact M&E will consider what changes have occurred because of the regime. It will seek to answer the following overarching evaluation questions:

• To what extent has the regime produced or contributed to its intended outcomes?
• What particular aspects of the regime have contributed to these outcomes and impacts?
• To what extent has the regime resulted in unintended or negative outcomes?
• Have any factors other than the regime influenced outcomes?
• How have outcomes relating to the UK’s digital markets regime changed relative to those in other key jurisdictions?

Impact monitoring and evaluation activities

CMA case-specific outcome indicators

3.2 The CMA will take a case-specific approach when assessing the effectiveness of its competition requirements.^{[footnote 8]} To review effectiveness, it will refer to the original purpose and aim of the competition requirement(s) to understand whether the intended impact has been achieved. In some cases, the CMA may consult on outcome indicators when it imposes competition requirements or separately.

3.3 These outcome indicators will capture the case-specific changes that have resulted (or are expected to result) from competition requirements.

3.4 The case-by-case approach means that outcome indicators will be dependent on the specific designations and competition requirements. As a result, it is not possible to state what these indicators will be in advance of competition requirements being introduced. The CMA will also consider effectiveness in the round, recognising that there are likely to be various factors driving market dynamics and consumer behaviour.

3.5 Information sources the CMA may draw on include:

• Market intelligence;
• Market/financial data;
• Research such as consumer surveys;
• Submissions from SMS firms and from third parties including businesses;
• SMS firms’ compliance reports;
• Complaints and information from whistleblowers.

3.6 The CMA will review effectiveness of competition interventions:

• When undertaking a further SMS investigation to determine whether an SMS firm should be re-designated for a further 5 years and, if so, whether competition requirements should be adjusted.
• When undertaking reviews of pro-competition orders.
• Where the CMA identifies sufficient reason to do so through its ongoing monitoring, for instance in response to well-evidenced submissions from firms designated with SMS or other third parties regarding effectiveness.

3.7 The CMA will publish summary highlights of progress against outcome indicators in its Annual Report.

3.8 Outcomes of the regime may include:

• Pro-consumer or pro-competition changes in terms and conditions; – A greater range of services available;
• Improved protection from exploitation and unfair or misleading practices;
• A greater chance for businesses to compete with SMS firms, including by having appropriate access to data and functionality;
• A reduction in exploitation or anti-competitive behaviour experienced by businesses that rely on SMS firms.

Tracking of broader developments in UK digital markets – CMA Annual Report

3.9 Alongside the case-specific outcome indicators, the CMA will also keep track of broader developments in UK digital markets.

3.10 This will draw on a range of third-party data sources and track wider impacts such as levels of growth, innovation and investment.

3.11 These findings will be included in the CMA Annual Report.

Impact evaluation in the Post-Implementation Review

3.12 The PIR will include an assessment of the changes caused by the regime, and how these compare to intended outcomes and impacts.

3.13 It will draw upon the CMA’s case-specific outcome indicators and the CMA’s tracking of broader developments in UK digital markets (such as levels of growth, innovation and investment). The government may also conduct other activities to support evaluating the impact of the regime.

3.14 The government’s PIR of the regime will use impact evaluation methods described in the Magenta Book.^{[footnote 9]}

Value-for-money monitoring and evaluation

Objectives and evaluation questions

4.1 Value-for-money M&E will be used to assess whether the regime was/is a good use of resources. It will seek to answer the following overarching evaluation questions:

• Have the benefits of the regime outweighed the costs?
• Has this intervention been the most effective use of resources to achieve the intended outcomes?

Value-for-money monitoring and evaluation activities

Direct financial benefit to consumers – CMA Annual Report and Impact Assessment

4.2 The CMA publishes its Annual Report alongside its Impact Assessment. These documents will include an estimate of the direct financial benefit of the DMU’s actions for consumers.

4.3 This estimate is calculated based on expected impacts at the time decisions are made. It will be accompanied by details about the methodology used and the independent verification process.

Total Annual Gross Expenditure on the DMU – CMA Annual Report

4.4 A figure for the total annual gross expenditure on the DMU will be published in the CMA Annual Report.

Value-for-money evaluation in the Post-Implementation Review

4.5 The PIR will include an assessment of whether the regime has been a good use of resources.

4.6 It will draw upon the direct financial benefit to consumers and total annual gross expenditure figures published by the CMA.

4.7 It is also expected that the government will use other methods to assess costs and/or benefits that might not be quantified by the CMA, including costs to SMS firms.

1. DCMS and digital sector GVA 2022 (provisional). Published 15 February 2024. ↩

2. Economic Estimates: Employment in DCMS sectors and Digital Sector, January 2022 to December 2022. Published 13 July 2023. ↩

3. Monitoring and evaluating the new pro-competition regime for digital markets. Published 4 May 2023. ↩

4. Impact Assessment: A new pro-competition regime for digital markets. Published 25 April 2023. ↩

5. The Magenta Book – Central Government guidance on evaluation. Published March 2020. ↩

6. Magenta Book – Supplementary Guide: Guidance for Conducting Regulatory Post Implementation Reviews. Published 1 April 2020. ↩ ↩²

7. ‘Competition requirements’ are the requirements imposed on SMS firms under the Act. These are: conduct requirements; requirements imposed by interim enforcement orders; requirements imposed enforcement orders; requirements imposed by final offer orders; requirements imposed by pro-competition orders; and requirements to comply with commitments. ↩

8. Magenta Book. Central Government guidance on evaluation. Published March 2020. ↩