Shared Rural Network: Evaluation plan

Question 1

1. Executive summary

Accepted Answer

Building Digital UK (BDUK), an Executive Agency of the Department for Science, Innovation and Technology (DSIT) commissioned Ipsos UK and partners to undertake an evaluation of the Shared Rural Network (SRN) programme. This report sets out the recommended approach to the full evaluation of the SRN programme.

1.1 Introduction

Summary of the programme

At the time of writing (spring 2024), the SRN programme aims to improve mobile coverage in rural areas across the UK through a deal with the four mobile network operators (MNOs) - EE, Three, Virgin Media O2 (VMO2) and Vodafone - delivering 4G coverage to some of the hardest to reach parts of the country.

The Shared Rural Network programme objectives are to:

Deliver 4G mobile coverage from at least one MNO to 95% of the UK landmass (as of September 2019 66% of the UK landmass had 4G mobile coverage from all four operators, and 91% had coverage from at least one operator) by the end of programme, underpinned by legally binding coverage obligations. The programme aims to achieve this by the end of 2025.
Deliver additional 4G mobile coverage to at least 280,000 premises and to 16,000km of roads by the end of programme. The programme will lead to increases in coverage across all four nations, with the biggest coverage improvements in rural parts of the UK.

In order to achieve these, the SRN programme has been implemented in two main phases. In the first phase of the SRN the four MNOs will collectively invest in a shared network of new and existing phone masts. This investment is in areas of partial commercial coverage, referred to as partial not spots (PNS) - areas where there is currently coverage from at least one, but not all MNOs. The MNOs are obliged to complete this element of the programme by June 2024 to meet their licence obligations.

The second phase, running in parallel, will see the government invest to go even further to significantly reduce total not spots, which are areas where there is currently no coverage from any mobile operator. There are two projects running in phase 2 – the Extended Area Service (EAS) which is a critical part of the Emergency Service Network (ESN) and the Total Not Spots (TNS) projects.

Evaluation objectives

The evaluation aims are set out clearly in the Invitation To Tender, namely to:

Provide evidence that fulfils BDUK’s mandated requirements – Subsidy Control Act and Infrastructure Project Authority; wider government requirements – for example Public Accounts Committee and National Audit Office; and supports improvement in the delivery of the SRN and future policy development.
Test the department’s hypotheses about the value of mobile connectivity, establish the outcomes of the programme and improve the evidence base of the benefits of mobile connectivity.

Outcomes and impacts

The SRN programme is expected to have the following outcomes and impacts:

outcomes for the public / consumers and businesses, including improved mobile broadband reliability, speeds and coverage, which is expected to lead to improved consumer choice, access to and use of online services and changes in perceptions of Areas of Outstanding Natural Beauty
outcomes and impacts for MNOs, including increases in energy use, operational costs, market competition and overall market size and changes in each MNOs market share
longer term impacts, including improvements in health and well-being, improved digital inclusion, improved economic opportunities, impacts on traffic and congestion, changes in carbon emissions, increased productivity and employment and impacts on emergency service performance

Evaluation approach

The recommended evaluation approach includes a process, impact and economic evaluation. The process evaluation will collect evidence to answer key evaluation questions around the programme design, identification of intervention areas, engagement and site survey activities, planning applications, grant agreements, infrastructure build and management and governance. Information will be collected and analysed using programme management information, secondary data sources and stakeholder interviews.

To assess the impact of the programme on mobile connectivity, different approaches should be used for the TNS and the PNS and EAS.

For TNS, it is recommended that the areas the project provides coverage to are compared to a counterfactual case of total not spots that do not receive coverage through infrastructure funded by the TNS using a difference in difference technique to estimate the impact of the project.

For EAS and PNS, a before and after analysis (comparing outcomes to a baseline measure prior to the intervention taking place) or interrupted time series analysis is proposed as the most appropriate approach to estimate the impact of the projects on mobile broadband availability. This would explore if the provision of PNS and EAS infrastructure has led to a step change in coverage.

To assess the downstream economic impacts, a pipeline approach to developing a counterfactual case is proposed, which will utilise matching and difference in difference techniques to estimate the impact of the programme. This approach can also be utilised to explore the impact on house prices, subjective wellbeing (from secondary data sources) and if sufficient traffic and energy consumption data is available some environmental impacts. This analysis will explore the impact of the SRN as a whole, and should be feasible to implement for both the PNS and EAS programmes individually (subject to sufficient numbers of individuals and businesses being located in each programmes footprint), but is not feasible to utilise for the TNS.

To assess the social impacts of the programme, and some environmental impacts, a before and after approach will be utilised, exploring the change in outcome metrics between a time before enhanced mobile broadband connectivity and when enhanced mobile broadband connectivity has been introduced.

For all outcomes a contributions analysis approach is recommended to explore how and why outcomes have been achieved and how the SRN programme contributed towards this.

For the economic evaluation, it is recommended that a cost-benefit analysis approach is undertaken, comparing the lifetime costs of the overall SRN programme (and individual programmes) to their long-term economic, social and potentially environmental impacts. Alongside this (and in the absence of a cost-benefit analysis if impact analysis is infeasible), a mixed methods value for money assessment will be undertaken, following the National Audit Office’s 4Es approach.^{[footnote 1]}

Question 2

2. Introduction

Accepted Answer

Building Digital UK (BDUK), an Executive Agency within the Department for Science, Innovation and Technology (DSIT) commissioned Ipsos UK and partners to undertake an evaluation of the Shared Rural Network (SRN) programme. This report sets out the recommended approach to the full evaluation of the SRN programme.

2.1 Summary of the programme

At the time of writing (spring 2024), the SRN programme aims to improve mobile coverage in rural areas across the UK through a deal with the four mobile network operators (MNOs) - EE, Three, Virgin Media O2 (VMO2) and Vodafone - delivering 4G coverage to some of the hardest to reach parts of the country.

The Shared Rural Network programme objectives are to:

Deliver 4G mobile coverage from at least one MNO to 95% of the UK landmass (as of September 2019, 66% of the UK landmass had 4G mobile coverage from all four operators, and 91% had coverage from at least one operator) by the end of programme, underpinned by legally binding coverage obligations. The programme aims to achieve this by the end of 2025.
Deliver additional 4G mobile coverage to at least 280,000 premises and 16,000km of roads by the end of programme. The programme will lead to increases in coverage across all four nations, with the biggest coverage improvements in rural parts of the UK.

In order to achieve these, the SRN programme has been implemented in two main phases.  The phases of the programme are discussed in more detail below.

In the first phase of the SRN the four MNOs will collectively invest in a shared network of new and existing phone masts. This investment is in areas of partial commercial coverage, referred to as Partial Not Spots (PNS) - areas where there is currently coverage from at least one, but not all MNOs. This part of the programme is wholly for the MNOs to deliver, but where possible BDUK provides support, for example by endorsing planning applications. The MNOs are obliged to complete this element of the programme by June 2024 to meet their licence obligations, which is for each MNO to provide coverage to 88% of the UK landmass (together with individual nation targets for England, Wales, Scotland and Northern Ireland). Delivery of the PNS projects is the responsibility of the MNOs and there is no grant funding cost to the government.

The second phase, running in parallel, will see the government invest to go even further to significantly reduce total not spots, which are areas where there is currently no coverage from any mobile operator. There are two projects running in phase 2 – the Extended Area Service (EAS) and the Total Not Spots (TNS) projects. A proportion of the Government funding will come from BDUK grants to the MNOs for the TNS project, a separate amount from BDUK grants to the MNOs for the EAS project, and the remainder is funded by BDUK to the Home Office for the EAS project. For the TNS project, MNOs will provide coverage in Scotland only, as Scotland has the lowest current mobile broadband coverage of the four UK nations, whereas the EAS will provide coverage across rural parts of Scotland, England and Wales. The TNS project grant is being managed by BDUK and delivered by Digital Mobile Spectrum Limited (DMSL), a joint delivery vehicle of the four MNOs.

The SRN EAS project will upgrade masts being built as part of the Home Office’s Emergency Services Network (extending and allowing MNOs to utilise the masts for commercial deployment), which will allow all four MNOs to utilise the masts to provide 4G coverage if they choose to. This element of the SRN programme is to be completed by January 2027. The SRN EAS project is being managed by BDUK and being delivered by a combination of the Home Office and DMSL.

The TNS project will provide grants to support the construction of masts in areas which currently have no coverage to improve 4G coverage. The four MNOs will build this new infrastructure. In order to minimise the impact on the countryside and the environment, all four operators will share each new mast. Under the 2020 licence, operators are required to increase UK geographic coverage by at least 1% through TNS projects, and if they do not, they will be required to fund infrastructure to achieve this outcome. This element of the SRN programme is to be completed by January 2027 to meet licence obligations.

The Government funding provided through the SRN is grant funding, which is provided to DMSL. Commercial analysis indicated that a competitive process for the construction and operation of the SRN infrastructure would not be appropriate, and that a direct grant to DMSL was the only viable solution. In summary, DMSL was considered the only organisation which can guarantee delivery of mobile coverage to all customers, share commercially sensitive information in a competition law compliant manner and create a joint radio plan.

The grants awarded to DMSL have been structured in a way to protect the public purse. The grants only provide funding to areas where market failure (the lack of commercial provision of mobile broadband) persists. If any government funded sites become commercially viable for any MNO in the life of the project, ongoing government support to the relevant MNO for operational expenditure will cease and clawback mechanisms for a proportion of capital expenditure have been provided for (where a higher capacity threshold has been exceeded).

DMSL and key project participants use the grant funding to run procurements of subsidised contracts in a manner compliant with the Public Contracts Regulations 2015. This means that suppliers can bid to DMSL to deliver elements of the work to provide mobile broadband coverage.

The programme meets subsidy principles set out in Article 3.4 of the UK-EU Trade Cooperation Agreement.

2.2 Evaluation objectives

The evaluation aims are set out clearly in the Invitation To Tender, namely to:

Provide evidence that fulfils BDUK’s mandated requirements – Subsidy Control Act and Infrastructure Project Authority; wider Government requirements – for example Public Accounts Committee and National Audit Office; and supports improvement in the delivery of the SRN and future policy development.
Test the Department’s hypotheses about the value of mobile connectivity, establish the outcomes of the programme and improve the evidence base of the benefits of mobile connectivity.

2.3 Approach used

This report is based on the following research activity:

a review of programme management information, including the business case, and descriptions of the programme, grant agreements and others
a review of wider, contextual published information, such as news items, Connected Nations data, and planning applications
a literature review to identify the potential impacts of mobile broadband coverage
a review of potential data sources which could be used to support the evaluation
stakeholder interviews with five stakeholders involved in programme delivery at BDUK and DMSL

2.4 Structure of report

The remaining sections of this report are structured in the following way:

a presentation of the theory of change of the programme
a description of the processes used to deliver the programme
a description of the data sources which could be used to support an impact evaluation
a discussion of potential approaches to an evaluation framework
an outline of how the evaluation can be implemented

Question 3

3. Theory of change

Accepted Answer

This section presents the theory of change for the SRN programme, including the background and context for the evaluation. The theory of change will provide the overall analytical framework for the study.

3.1 Rationale for SRN interventions

Expanding mobile connectivity to a larger proportion of the UK landmass has been a longstanding priority of successive governments. It is a core part of modernising the UK’s digital infrastructure. There are significant proportions of the UK landmass which are not served by all four MNOs, or where residents are not served by any MNOs. There are potentially significant economic, social and consumer benefits to the UK of delivering mobile connectivity to these more rural areas, alongside potential equity issues if enhanced broadband coverage is not delivered, as more services are moved online – meaning those with no or poor internet access may be excluded from important services.

However, mobile broadband connectivity is provided by private companies. In these rural areas, the companies have explored providing connectivity, and concluded that it is not economically viable to build infrastructure in these areas. This could be due to difficulties in building in these areas, leading to an increase in the cost of construction, or a lack of demand for mobile broadband connectivity in the areas. In both cases, the cost of providing mobile broadband coverage has been assessed as being higher than the income that would be generated.

In order to ensure mobile broadband coverage is provided in these areas, government intervention was needed.

3.2 Inputs

There are multiple inputs into the SRN programme, including financial inputs, human capital inputs – namely the skills, knowledge and expertise of staff involved, regulatory agreements, and existing infrastructure. These are presented below.

Public financial input:^{[footnote 2]} Public funding across both EAS and TNS will be provided for the construction and operational maintenance of 4G infrastructure. The public funding covers a 20-year appraisal period, including both upfront capital costs over 6 years and ongoing operational and maintenance expenditure once sites are constructed. The 20-year period over which operational costs will be funded was agreed upon as it is identified as being the average lifespan of a mobile mast. (The financial inputs for the programme are taken from the original SRN business case, which was developed in 2020. These costs may change as the SRN is delivered.)

Commercial financial input:^{[footnote 3]} commercial financial input for PNS infrastructure is committed across all four MNOs. This investment is divided between the MNOs and will be spent on new and upgraded masts across the UK, with each MNO having individual targets of how much additional coverage they need to provide:

existing infrastructure: access to existing MNO infrastructure to upgrade coverage
EAS infrastructure: the Home Office is upgrading its existing Emergency Service Network (ESN) and constructing an additional 292 masts in rural areas. As many of these 292 sites as possible will be made available for the MNOs to deploy coverage as part of the SRN Extended Area Service (EAS) project. Although in some instances upgrades at mast sites might not be possible due to technical, financial or other reasons.
existing telecommunications expertise: the only four MNOs (EE, Three, VMO2 and Vodafone) that operate a physical nationwide network in the UK were tasked with delivering the infrastructure to reduce partial not-spots are the UK’s four major mobile network providers and possess a wealth of expertise and experience in delivering infrastructure.

BDUK contracting / management expertise: BDUK possess experience and expertise in the management and oversight of large-scale infrastructure contracts through the delivery of projects such as the Superfast Broadband programme, Local Full Fibre Network programme and Project Gigabit. By bringing the SRN programme and reporting within the BDUK portfolio, the deployment of fibre and mobile network infrastructure can be delivered more efficiently and effectively.

Information from Ofcom / MNO data on current coverage: Ofcom monitors existing coverage. MNOs provide details of their coverage to Ofcom, and Ofcom combines these submissions into a single dataset to identify areas that are covered by all MNOs, numerous MNOs, a single MNO or has no mobile broadband coverage. This is made available to the public through its broadband and mobile coverage checker, which enables users to check indoor/outdoor mobile availability for voice and data services across all major operators.^{[footnote 4]} Ofcom is tasked with measuring compliance with the mobile coverage obligations set out in the SRN programme agreement. MNOs will provide details of the coverage provided by its infrastructure to Ofcom, specifying which frequency bands are used to predict coverage.^{[footnote 5]} MNOs use this data to inform decisions about where to locate infrastructure build for the programme. Progress on increasing coverage will be reported in Ofcom’s Connected Nations report^{[footnote 6]} which is usually published 3 times per year.

Joint delivery vehicle: Digital Mobile Spectrum Limited (DMSL) is a joint venture of the four MNOs established in 2012 and is tasked with delivering the infrastructure required under the SRN programme. The entity provides reporting and tracking information to the four MNOs, BDUK and Ofcom and leads on all external communications and stakeholder engagement, working closely with the MNOs and their trade body, Mobile UK.^{[footnote 7]}

Agreements: Several commercial agreements have been established to guarantee deliverability of the programme. These include:

DSIT (including activities undertaken by DCMS prior to 2023) / DMSL grant agreement: A series of intensive workshops and all-day sessions involving DSIT, DMSL and MNOs took place between early January to mid-February 2020. These discussions were attended by lawyers and specialist advisors who debated the grant provision and agreed text to be contained in the grant agreement or negotiated issues for executive consideration and agreement. As part of the grant agreement negotiation process a list of eligible expenditure was also developed. This list was reviewed by separate technical experts on behalf of the government and MNOs to ensure that the minimum number of items are eligible without creating a prohibitive list that could compromise effective delivery. Subsequently, by mutual consent, there have been several amendments to the text of the grant agreement
Ofcom licence conditions: This licence obligates each MNO to provide 4G service to specific areas of the UK that, in combination, will achieve the target of 95% coverage. The geographic areas must cover a minimum number of premises and kilometres of road that the MNO does not currently provide 4G coverage to. The licence conditions also stipulates that the service is to be delivered in two stages, one lasting four years and the other lasting 6 years. Ofcom holds the legal power to fine MNOs if they fail to meet these licence conditions and has exercised such power in the past
Memorandum of Understanding (MoU) between DCMS and Home Office: A MoU exists between the two departments containing a formal agreement over how the SRN programme and Emergency Services Network (ESN) will align and the governance and management arrangements between DCMS and the Home Office. A key aspect of the MoU for the SRN programme stipulates that as many of the 292 EAS sites as possible will be made available for consumer 4G coverage. Other key elements of the MoU include funding requirements for upgrading EAS sites, delivery timetable and project plan, budget management and financial controls, and the Home Office resourcing strategy to ensure upgrades are feasible
Service Agreement between Home Office and MNOs: This services agreement outlines the responsibilities of the Home Office and each MNO in relation to the EAS element of the SRN programme. This agreement is required because MNOs will be reliant on the Home Office for the provision of certain services in order to provide coverage from EAS sites. These services include site access arrangements, the technical specification that sites will be upgraded to, operation and maintenance / servicing of the passive elements of sites

3.3 Activities

There are multiple activities which need to be delivered as part of the SRN programme. These activities are highlighted below:

Identification of infrastructure locations and MNO provider selection: The MNOs / DMSL are responsible for identifying potential locations for infrastructure build / enhancements which are required to achieve the programmes targets. This utilises their expertise and the Ofcom data on Partial and Total Not Spots. For EAS the site locations are chosen by the Home Office as part of the ESN project.

Site visits: MNOs will visit potential sites to undertake surveys, to establish if it is feasible for new infrastructure build / enhancements to be made at the site. The site survey will result in a report which will be used to inform decisions as to whether to proceed with infrastructure build / enhancements at the site.

Developing planning applications: For sites where build / enhancements go ahead, MNOs will submit planning applications to local authorities to obtain permission to proceed with the infrastructure project. At this stage MNOs will also apply to landowners for access and wayleaves to potential sites.

Secure sufficient power supply: MNOs will ensure that the site has sufficient power to support the new / enhanced infrastructure. This can involve consultations with and remedial work by the national grid / power companies, or for some sites the installation of a generator.

Building / enhancement of masts: The MNOs will construct new / enhance existing masts to support the delivery of mobile broadband connectivity.

Installation of transmission infrastructure: MNOs will install their transmission equipment to the site – this could involve the installation of up to four sets of transmission equipment if the mast is supporting all four MNOs.

Live masts: Once all the equipment is installed, the MNO will make the mast live and provide mobile broadband coverage to the surrounding area.

Operational maintenance: The MNOs are responsible for maintaining the operation of the masts over a twenty-year period.

3.4 Outputs

The key outputs that will be achieved by the programme are:

Planning applications submitted: Planning applications outlining proposed new 4G infrastructure will be submitted to local authorities across the UK.
Planning applications approved: Local authorities will approve a proportion of the planning applications submitted as part of the SRN.
New / enhanced broadband infrastructure: As a result of the activities, there will be new mobile broadband masts constructed, and existing broadband masts which will have been upgraded as part of the programmes. This is expected to lead to:
UK geographic coverage: Each of the four MNOs are obligated to achieve 88% of UK geographical coverage by June 2024 and 90% by January 2027 through shared infrastructure. This will deliver aggregate coverage (by at least one MNO) of 95% of the UK. Each of the four MNOs have individual targets for coverage per nation.
Roads and premises: The SRN programme will deliver coverage improvements to a minimum of 16,000km of main roads and at least an additional 280,000 premises. There is also expected to be further indirect improvements to coverage over time, including a boost to ‘in car’ coverage on approximately 45,000km of road and improved indoor coverage for approximately 1.2 million premises. Each MNO has an individual coverage obligation for roads and premises coverage which do not need to be met if the aggregate 16,000km and 280,000 premises is achieved.
Increased market competition: The involvement of the four major UK MNOs and minimum coverage obligations create a presence of multiple operators competing for customers within regions with limited coverage. The programme reduces the cost barrier to entry in rural areas by awarding public funding and removing the monopoly of power in some areas.

3.5 Outcomes

The outcomes of the SRN programme can be grouped into related benefits for the public/consumers, businesses and outcomes for MNOs. The project each outcome relates to is indicated at the end of each paragraph.

Outcomes for the public / consumers and businesses

The outcomes for the public, consumers and businesses that receive enhanced mobile broadband coverage are:

Improved reliability: The upgrade of existing infrastructure and increase in the number of masts and MNO coverage will contribute to improved reliability – with mobile coverage being less intermittent (reduced instances of dropping out), as the signal provided to areas will be stronger. Relevant for all projects.

Improved mobile broadband speeds: New and upgraded existing 4G infrastructure resulting in increased coverage will result in faster mobile broadband speed for consumers. Relevant for all projects.

Improved consumer choice: The delivery of increased geographical 4G coverage by the four MNOs in rural areas with limited, or no coverage at all, will increase consumer choice for local residents. Under normal conditions without intervention, rural areas with limited or no coverage are usually sparsely populated and therefore present fewer commercial incentives to install infrastructure to improve coverage. The delivery of the programme via DMSL also supports this outcome by creating shared responsibility across the four main MNOs. For more urban areas, or areas where there is good coverage from one but not all MNOs, consumers will be able to select their MNO based on factors other than availability. Relevant for all projects.

Improved access to online services: Residents in remote, rural areas will be able to access online services at home rather than being required to travel to other areas with better coverage (and have improved access whilst out and about). This will contribute to reducing the digital divide between rural and urban areas. Visitors of rural tourism hotspots will also gain access to improved online services as a result of the programme. This will be a particular benefit to rural locations which do not have, and are not expected to have, good quality fixed broadband infrastructure. However, it will also be important to access services that use two factor authentication (for example online banking) in rural areas with good fixed broadband coverage, as 2G and 3G services used to deliver text messages and mobile calls are switched off and 4G is used to deliver texts. Relevant for all projects.

Take-up of new services: Improved coverage will create new services for local consumers and the opportunity to gain greater value from their mobile network contract through the ability to use their entire mobile network allowance. Relevant for all projects.

Increased use of online services: New and improved access to online services may lead to greater uptake in rural areas. Residents will be able to benefit from a range of services previously unavailable to them, such as virtual GP appointments, online shopping, access to online government services, navigational services (both at home and whilst out and about). As above, this will be particularly relevant to areas with poor fixed broadband coverage and for services that require two factor authentication. Mainly relevant for TNS and EAS, partially relevant for PNS.

Change in perception of quality of Areas of Outstanding Natural Beauty (potential disbenefit): The construction of 4G infrastructure in rural Areas of Outstanding Natural Beauty is likely to impact the way these locations are viewed by local residents and visitors as masts may disrupt the perception of ‘natural beauty’. Relevant for TNS and EAS.

Outcomes for the MNOs

The outcomes from the programme for the MNOs are:

Energy usage for construction of masts (potential disbenefit): The construction of the new or enhanced infrastructure will also require significant power consumption, which could lead to an increase in carbon emissions. Relevant for all projects.

Increase in operational costs (potential disbenefit): As a result of the increased energy usage to operate the new and enhanced infrastructure, and other maintenance costs for the infrastructure, MNOs will face an increase in operational costs. Much of this cost will be covered by the grant agreements for TNS and EAS, but for PNS the MNO will have to pay for the increased operational costs. Relevant for all projects.

Increase in MNO competition: Due to the increase in mobile broadband coverage in partial not spot areas, MNOs which already had coverage in these areas will face increased competition from the remaining MNOs for customers that live or work in those areas – as customers are likely to select an MNO that has coverage in areas they spend a lot of time in. There may also be price competition implications. Relevant for PNS.

Increase in MNO market size: Where the EAS and TNS have provided coverage in areas that previously were Total Not Spots, there are likely to be new customers for mobile broadband services, as they were unable to access services previously (or they may increase their mobile broadband consumption as a result of having coverage in areas they spend a lot of time in). This will increase the size of the mobile broadband market. However, given the areas that the TNS and EAS operate in, it is likely to be a very marginal increase. Relevant for TNS and EAS.

Change in MNO market share: As a result of changes to the market size and changes to competition in some areas, there may be changes in the proportion of the market taken by each MNO following the SRN intervention. Relevant for all projects.

Increased energy usage by new masts (potential disbenefit): The new and enhanced infrastructure provided through the SRN will require power to operate. This means there will be an increase in energy consumption as a result of operating the enhanced infrastructure, which could lead to an increase in carbon emissions. Some of the power supply for masts, particularly for the TNS project, may utilise off-grid power solutions, for example generators. Relevant for all projects.

Infrastructure in place for 5G coverage in rural areas: Some of the infrastructure used to deliver the SRN could potentially support 5G rollout in rural areas in the future, reducing the marginal costs faced by MNOs to provide 5G in the future. This could influence their commercial decisions as to where to rollout 5G coverage. Relevant for all projects.

3.6 Impacts

Longer-term expected impacts of the SRN programme are centred on health and wellbeing impacts and economic impacts. It should be noted that the extent the programme can generate these impacts will be influenced by fixed broadband coverage, as many of the impacts can be achieved using fixed broadband solutions as well as mobile broadband connectivity. Due to the good quality publicly available data on fixed broadband coverage in the UK (Ofcom Connected Nations data), the evaluation can account for the effect of fixed broadband infrastructure.

Health & wellbeing impacts

Improved wellbeing: The series of impacts listed below, from increased inclusion, access to services and leisure, communication and so on, are expected to lead to changes in how people feel about their life and their level of self-reported well-being (or life satisfaction). These are expected to result from increased connectivity both in the home and out and about. However, there are some elements of improved connectivity that may also lead to a negative impact on well-being, such as increasing work hours, internet addiction and online bullying. However, it is anticipated that the positive well-being effects would outweigh the negative impacts for most individuals. Relevant for all projects.

Remote monitoring of health: Increasing and improving access to online services (both in and out of the home) will provide the opportunity for health and social care practitioners to monitor patient health remotely in a greater number of regions. Remote patient monitoring enables practitioners to gather real-time information on the physiological conditions of patients, leading to more efficient communication, a faster response to medical issues, increased patient confidence, the visualisation of health trends and greater patient knowledge of their medical condition.^{[footnote 8]} Mainly relevant for TNS and EAS, partially relevant for PNS.

Improved health outcomes: Improvements in the monitoring of health (both in and out of the home) can be expected to improve individuals’ health outcomes, for example reducing the severity of health conditions and improving individuals’ well-being. In addition to this, better monitoring of health can also lead to a reduced burden on the health service (GPs and hospital consultations), leading to a positive impact for the government. Mainly relevant for TNS and EAS, partially relevant for PNS.

Increased digital and financial inclusion: Residents will be able to access a greater number of financial services online (both in and out of the home), including fintech products and services. Relevant for all projects.

Improved access to public services (e-government): As an increasing number of public services are offered online (e.g. online NHS appointments, renewing a driver’s licence, filing a tax return), rural residents will be able to benefit from accessing services from home (and when out and about) rather than commuting to a nearby public sector institution or post office to mail forms. Mainly relevant for TNS and EAS, partially relevant for PNS.

Improved access to entertainment: Improved coverage and access to 4G within homes (and while out and about) will provide residents with the opportunity to access online entertainment in the form of streaming platforms (music, podcasts, television and film), eBooks and online gaming. This is likely to generate a greater demand for these services. Mainly relevant for TNS and EAS, partially relevant for PNS.

Improved access to remote education services: Improved 4G coverage will provide the opportunity for children and adults in higher education to access education remotely. This provides benefits to those who have accessibility requirements, enables continuity of education when disruption arises (e.g. pandemic, illness requiring home studying) and provides the opportunity for students to study further at home. Mainly relevant for TNS and EAS, partially relevant for PNS.

Improved household opportunities: Improved coverage, reliability and 4G speed enables residents in remote areas to access a greater pool of jobs that offer opportunities to work from home. This can provide benefits to wellbeing for those unable to commute over long distances (e.g. due to health, accessibility and/or financial restrictions) and economic benefits by enabling access to a wider range of jobs. Relevant for all projects.

Improved online communication: Residents in rural locations will be able to communicate with others online without the need to travel to a location with better coverage or utilising mobile broadband coverage from multiple MNOs (for example having two mobile phone agreements with different MNOs). Relevant for all projects.

Improved safety/security (e.g. remote monitoring, real-time comms): Residents and tourists in areas where they previously could not access mobile broadband will have more opportunities to improve their personal safety. The use of apps that allow real time tracking, and being constantly able to contact others will improve safety, particularly for individuals who are alone. Further to this, individuals will be able to access home security equipment while they are out of the house (for example doorbells or security cameras), improving the safety of their home. Relevant for all projects.

Change in traffic / road congestion / reduced need to travel: Greater access to opportunities for online work from home may result in a reduction in the need for rural residents to commute to work. As a result, there may be fewer cars on the road during peak travel times (early morning and early evening) and therefore a reduction in road traffic and congestion. The increase in the ability to utilise online facilities (such as online shopping) should decrease the need for travel by car in areas where mobile broadband has been delivered. These outcomes could contribute to a decrease in carbon emissions. However, there could also be an increase in traffic if tourist destinations become more popular as a result of enhanced connectivity, with more visitors driving to the area. Relevant for all projects.

Change in carbon emissions (uncertain direction): The potential reduction in the need to commute to work and fewer cars on the road producing emissions may lead to a reduction in carbon emissions. However, the increase in energy to construct and operate new masts will increase carbon emissions, meaning it is uncertain whether the programme will have a positive or negative effect on carbon emissions. Relevant for all projects.

Changes in air quality: Related to the two impacts above, if there are fewer cars on the road producing emissions and harmful pollutants will improve the air quality in areas where rural residents are driving to work – however an increase in visitor numbers driving to the area will decrease air quality. Relevant for all projects.

Increased access to and notification of emergency services: Through EAS, an additional 292 masts in remote areas will enable fast, safe and secure voice, video and data which will provide first responders with immediate access to life-saving data, images and information in emergency situations.^{[footnote 9]} For more, consult the Emergency Services Network overview. The ability to make contact with the emergency services is a benefit of the Emergency Services Network. However, there are potentially further benefits to the emergency services from the enhanced mobile broadband coverage provided, such as being able to access wider health and social care systems and allowing the individual to access key data which may support the emergency services (for example online health data, contacting next of kin), or reduced congestion on the roads making the journey for emergency service vehicles more straightforward. There is an additional benefit to the public and non-emergency services, such as roadside assistance, as the public will also be able to notify these services faster as they are more likely to be in an area with reliable and fast coverage. Relevant to TNS and EAS.

Benefits to emergency services: Related to the impact above, EAS will result in emergency services being able to access a wider range of information through the ability to use new technologies in remote regions, which will help to support improved outcomes for the emergency services (for example, health outcomes, arrests made). Relevant to TNS and EAS.

Economic impacts

Improved business opportunities / increased entrepreneurship: Residents of remote and rural areas will gain the opportunity to collaborate with other businesses online and/or set up a new business or expand their existing business online. Additionally, in rural areas, increased mobile broadband coverage may be able to signpost more customers to businesses as tourists or commuters are more aware of where the businesses are located (for example food service businesses receiving more passing trade). Relevant for all projects.

Improved access to internet of things and improved industrial autonomy: Business will have improved access to the Internet of Things which can support their business to work more efficiently. For small businesses this could include accessing heating controls when they are outside the workplace to manage energy costs, or accessing the business servers when employees are away from the workplace (for example accessing files when visiting clients or suppliers, uploading records after completing a site visit). It could also include some automated processes in the workplace where previously connectivity was too poor to support this. Mainly relevant for TNS and EAS, partially relevant for PNS.

Upskilling of workforce: Through the improved access to learning and education (described above) and as businesses change their processes, the workforce will need to upskill to be able to deliver any new ways of working (for example automated farming processes or manufacturing automation). This could be delivered through on the job training or potentially online training utilising the enhanced connectivity in very rural areas. Mainly relevant for TNS and EAS, partially relevant for PNS.

Improved access to workforce (remote working): By improving connectivity and the ability for households in the upgraded areas to work from home (by improving connectivity and resilience – the ability to continue to work from home if the fixed line broadband fails) means that businesses around the country will now have access to staff that can work flexibly from home. This could increase the match between skills and opportunities at a national level, and contribute to reducing skills shortage vacancies. Relevant for TNS and EAS.

Improved tourism facilities: Particularly in rural areas covered by the EAS and TNS, where some tourist attractions previously had very poor broadband coverage, the increase in mobile connectivity will allow the tourist attraction to utilise new services (for example payment mechanisms, interactive experiences, marketing). These new services will improve the experience of tourists visiting the attraction, and in the longer term potentially increase the number of visitors and income for the tourist attraction, allowing them to make further investments in the attraction. Relevant to TNS and EAS.

Increase in experience and tourist numbers: The improved facilities described above are likely to improve the experience of tourists that visit attractions, and make them more likely to recommend them to friends and family or leave positive online reviews. This will in turn lead to an increase in tourists visiting the attraction and local area. Relevant for all projects.

Increased GDP, productivity and job creation: The improved industrial autonomy, ability to work from home, new business opportunities and entrepreneurship would be expected to lead to improvements in productivity and output in the local area, and potentially increase the number of local jobs (for example if tourist attractions become more popular as a result of enhanced connectivity). Relevant for all projects.

House price impact: The improvements in connectivity and employment opportunities is likely to lead improvements in well-being, which contribute to places becoming more desirable to live in. This could lead to population shifts and an increased demand for housing, which will increase house prices in the area. Relevant for all projects.

Regional economic effects: The improved productivity and increased opportunity to work from home in these more rural areas is likely to lead to improved economic performance which aimed to contribute to the previous government’s Levelling Up agenda, and to the new Government’s local economic growth strategies.^{[footnote 10]} These strategies aim to reduce geographic inequality within the UK through boosting productivity, jobs and community pride, which will lead to an improvement in living standards and well-being in all areas of the UK. Relevant for all projects.

Other impacts

More efficient public service provision: As more households will have access to decent broadband connections (particularly in very rural areas), and more individuals have access to mobile broadband out of the house, public service providers can be more confident in rolling out digital services without reducing access to services. The digital services are seen as key by public service providers to supporting the delivery of services to a larger population without reducing the quality of the services provided. The rolling out of digital services allows public service employees to be redeployed to more productive occupations. Relevant for all projects.

Reduced journey times (e.g. real-traffic information): The improved mobile broadband access will allow more people to utilise travel planning applications, and get updates from the applications more frequently. This will support them in their journey planning, avoid getting lost and avoid existing traffic and congestion. This will lead to a reduction in congestion, improve journey times and therefore carbon emissions. Relevant for all projects.

3.7 Key risks and contextual factors

There are several risks which could impact upon the SRN programme’s ability to achieve its intended outcomes and impacts. These are:

Potential merger between MNOs: At the time of writing, two MNOs are in the process of merging. This is being assessed by the Competition and Markets Authority (CMA). If the merger proceeds, this may affect the SRN programme’s ability to achieve its targets and the effect on consumers. It is unclear at the time of writing what recommendations the CMA will make around the merger and the impact it will have on the SRN programme.

Planning applications: MNOs have to submit planning applications to local authorities to obtain permission to build / enhance infrastructure. However, at the point the application is submitted, the MNOs and the programme lose control of the process. Local authorities could reject applications which MNOs expected to be approved. If local authorities reject planning applications, it may become challenging for MNOs to deliver all the infrastructure upgrades planned by the programme, which would diminish the outcomes the programme is able to achieve.

Delays and cost increases: As with all infrastructure projects, there are risks associated with delays in the delivery of the programme. These could have been caused by the effects of Covid in the early stages of the PNS project, or be caused by a lack of capacity in the supply chain, a lack of materials or other external factors. However, delays in the delivery of infrastructure projects are usually accompanied by increased costs. As the public funding for the programme is fixed, the increased costs would be borne by the MNOs – however, there is a risk that the increase in costs is too great and some of the planned upgrades do not happen, which would diminish the outcomes and impacts of the programme.

Consumers do not utilise new services: There is a risk that even if the mobile broadband coverage is provided, consumers do not change their existing behaviour, or do not change it sufficiently, to lead to the expected impacts of the programme. For example, individuals may have got used to living without broadband, or without mobile broadband, and have set patterns of behaviour that are hard to alter (for example not turning on mobile broadband on their device and only using fixed line broadband). They may also become loyal to a particular MNO or mobile provider, and not take up newer or cheaper offers. If individuals do not alter their behaviour, then the programme will struggle to achieve its intended outcomes and impacts.

Employers roll back on flexible working arrangements: Since the Covid pandemic, many employers have supported a more flexible approach to working, with more people able to work remotely. However, employers can alter this approach at any time, and require workers to be more workplace based (for example the Government amending civil service work practices). If employers decide to move back towards a less flexible working environment, some of the benefits of the SRN programme will not be realised.

Population changes impact upon wellbeing: Changes in connectivity could lead to the upgraded areas becoming more desirable to live in or visit, and lead to population changes. This could have a negative impact upon individuals wellbeing if the area is busier, for example if it is harder to get appointments at the GP practice or if schools are oversubscribed.

Changes in fixed broadband rollout plans limit impacts: The SRN programme is happening at the same time as the government is supporting the rollout of gigabit capable fixed broadband to rural areas of the UK. Changes in fixed broadband connectivity could occur in the same locations as the SRN is operating. Many of the key impacts (for example working from home, introduction of new working practices, accessing online services) could be achieved using fixed broadband connections only. It will be important to consider this as part of the evaluation, as changes in fixed broadband connectivity will impact upon many of the same outcomes as the SRN. Additionally, in the areas where SRN delivers mobile broadband connectivity, if there are good fixed broadband options available to consumers, the impact of the SRN programme may be limited.

Indoor vs. Outdoor benefits: Many of the outcomes and impacts above relate to both benefits from accessing mobile broadband in the home as well as out of the home. However, the SRN programme targets providing coverage to 95% of the landmass, and does not target indoor coverage. Therefore, indoor coverage may be weaker than outdoor coverage, potentially limiting some of the outcomes and impacts that relate to indoor use.

Technological developments: The way in which technologies evolve may affect the impact that 4G coverage can have – for example if the connection speeds that apps and technologies require to run (for example resolutions of videos, gaming requirements) may mean 4G services do not offer the impacts intended.

Unused infrastructure: For the TNS programme in particular, there is a risk that there is infrastructure which becomes commercially unviable and unused in the longer-term (as infrastructure no longer has to be used at the end of its lease for the programme, 20 years). Therefore, the unused infrastructure will generate none of the benefits outlined in the theory of change after 20 years, and may generate disbenefits to local communities. Any infrastructure that is unused for 4G may be repurposed for alternative uses once the lease has expired. However, this risk is out of scope for the current evaluation.

3.8 Logic model

The figures below present the logic models for the social outcomes and impacts, business outcomes and impacts, environmental and MNO outcomes and impacts. These demonstrate the outcomes and impacts of the programme in a static environment (excluding external factors).

Logic model 2: economic impacts

Logic model 3: environmental, emergency services and MNO impacts

Question 4

4. Description of processes used

Accepted Answer

This section describes the processes that are being used to deliver the SRN programme. This section draws on information from a review of documentation and interviews with five key stakeholders, and is accurate at the time of writing (spring 2024).

4.1 Delivery model of SRN

The overall SRN programme is split into three individual programmes – the industry-funded Partial Not Spot (PNS) project, and the HMG-funded Total Not Spot (TNS) and Extended Area Service (EAS) projects. Each of these projects is delivered in a slightly different manner, and is therefore described separately below.

The programmes objectives are underpinned by Ofcom licence obligations. This required MNOs to commit to increase their individual 4G coverage to 88% of the UK landmass by 30 June 2024, and to 90% by 31 January 2027.^{[footnote 11]} The 88% obligation is delivered through the PNS project, with a further (minimum of) 1% of coverage to come from the TNS programme and the remaining1% to come from the EAS project. This licence was agreed in 2020 and involved Ofcom and the four MNOs.

Following the agreement of the coverage targets, the MNOs submitted to Ofcom data which described their current coverage across the UK. Ofcom then combined this data to identify pixels which were total and Partial Not Spots. This information was relayed to MNOs to demonstrate the scope against which their progress for the SRN would be measured against.

Partial Not Spot Project

The PNS project aimed to provide enhanced coverage to areas that have mobile broadband coverage from at least one MNO, but not all four. The PNS project is underpinned by regulatory requirements, set by Ofcom, which influence where MNOs build new infrastructure. The lack of public investment means that the PNS project operates in a different manner to the TNS and EAS projects.

For the PNS project, the MNOs have used the data from Ofcom, and their own data of their existing coverage to identify areas that were Partial Not Spots but were not included in their own current coverage. From these areas, the MNOs selected sites for infrastructure build or enhancements which would increase their coverage of the UK land mass. MNOs would then make informed business decisions of the most appropriate places to locate new or enhanced infrastructure whilst maximising the coverage of the UK land mass. As is typical for commercial rollouts, the MNOs identified a greater number of sites than required, in anticipation of delivery difficulties such as technical or access issues or changes in prioritisation.

Once the potential sites were identified by MNOs they began engaging with landowners over access to the potential sites for a site survey. For this the landowner needed to grant access for engineers and equipment to enter the site to assess the technical and commercial (cost) viability of constructing infrastructure on the site. Once permission was granted, the site survey was undertaken, exploring the space and topology of the site, the distance any infrastructure could cover from the site, the power supply and the feasibility of undertaking construction work at the site (for example access). Following the site survey the MNOs produced a report which outlined the feasibility of the site. The MNO then assessed the information from the site survey and decided whether to proceed with the acquisition, design and build process for the site.

For sites where the MNO decided to proceed, the next step was to arrange for long term access to the site for construction and to operate the site. This involved making a lease agreement with the landlord (which could initially be a Heads of Terms to enable work to start quickly), and ensuring wayleaves were in place for the site. Alongside arranging access, the MNO submitted a planning application to the local authority to obtain permission for the construction of the infrastructure. The local authority then considered the planning application and enabled the public to submit responses to the application, before deciding whether to accept the application or reject it. If the application was rejected, the MNO considered the reasons for the rejection and decided whether to reapply with an amended build plan, drop the application, to move on to another site, or potentially appeal the decision.

Once the lease agreement, wayleaves and planning permissions were in place, the MNO moved onto building the infrastructure. This involved purchasing and delivering the required equipment for the build to the site, building or enhancing towers, ordering transmission to the site, installing transmission equipment, and ordering and securing sufficient power to run the infrastructure (either from the grid or utilising generators when grid delivery is delayed). Following the completion of the construction, provision of power, and installation and commissioning of radio equipment, the mast became live, providing mobile broadband coverage. The MNO is then responsible for maintaining the mobile network infrastructure.

Total Not Spot Project

The TNS project aims to deliver 4G mobile broadband coverage to the areas identified by Ofcom data to have no mobile broadband coverage. The delivery of the infrastructure is funded through a grant from BDUK, and as a result of the public funding element there are some slightly different processes used.

The delivery body for the TNS project is DMSL – a joint venture of the four MNOs which manages delivery of the EAS and TNS projects. Although DMSL had originally been set up to resolve issues in the radio spectrum in the early 2000s, it was repurposed as the organisation through which all four MNOs would deliver the TNS and EAS projects. DMSL’s Board includes representatives from all four MNOs.

DMSL used existing coverage data from Ofcom and licence obligations to identify target areas, and potential sites which could be used to enhance mobile broadband coverage. The identification process follows similar considerations to those detailed for the PNS, but with DMSL taking the primary lead instead of each individual MNO.

Following the identification and collective agreement (by DMSL and the MNOs) of potential sites, DMSL discussed with the MNOs about which MNO would lead on each site. In order to identify which organisation should lead on a site, the locations of each MNOs existing infrastructure is considered by DMSL, with those with existing infrastructure closer to the site the preferred option, but with DMSL aiming for an even split of sites between the four MNOs. DMSL would then finalise which MNO is expected to lead on each site for TNS.

The selected MNO would then follow a similar approach to that described above, of engaging with landowners, undertaking a site survey and making an assessment of the sites feasibility for infrastructure build.

Following the MNO completing the infrastructure build, the other three MNOs would then access the site and install their own equipment. Once all the active radio equipment has been installed, the MNO with responsibility for the site will ensure that the mast will become live, providing mobile broadband coverage. The MNO with responsibility for the site is then responsible for operating and maintaining the mobile broadband infrastructure during its lifetime.

The grant payment to the MNOs is provided via DMSL when individual sites provably (with evidence provided) progress through agreed milestones as set out in the Grant Agreement. The first payment is made when the site has been acquired and is ready to proceed with build activity; the second payment is made after the building of the main infrastructure; the third payment when all of the transmission equipment is installed; and BDUK will pay the final tranche of the grant to the MNOs once the site is live and providing mobile broadband coverage. Each year, the MNOs will make a claim for operational expenditure for the site.

Extended Area Service Project

The Home Office has contracted EE to supplement its existing mobile broadband coverage by building 292 masts for the Emergency Services Network. These masts will give the emergency services coverage in some of the most remote and rural areas of Great Britain. These sites are known as the Extended Area Service (EAS). BDUK has provided funding to the Home Office to upgrade sites for SRN use by upgrading the masts’ physical infrastructure, transmission backhaul and power (where applicable), provided they pass BDUK assessments regarding value for money and technical feasibility. The sites will then be handed over the MNOs and activated for commercial use with the MNOs installing their radio transmission kits. The sites have been selected for the EAS by the Home Office to fulfil the needs of the emergency services, and the planning permission and infrastructure build has been completed by EE prior to the BDUK funding intervention. Though approval will need to be sought depending on lease arrangements for upgrades.

Once the sites are handed over by the Home Office to the MNOs (excluding EE) they will decide which sites to install radio transmission equipment on. This decision will be based around whether they have other infrastructure in the area which provides mobile broadband coverage to a similar population (or they were planning to build infrastructure nearby through the PNS project), and therefore putting their equipment on the EAS infrastructure would offer poor value for money. The MNOs would also undertake a site visit to inform their decision, to ensure that the site is suitable for their transmission equipment.

Once the MNOs have decided whether they will utilise a site, DMSL would allocate an ‘Active Host’ to each site – the MNO that is responsible for ensuring that the site will be live. The Active Host would be responsible for agreeing Service Level Agreements with the Home Office and agreeing access to the site, ensuring that all planning permissions are in place for any amendments that need to be made to the EE installed masts and to enhance the existing infrastructure and install their transmission equipment. The Active Host has 12 months from their initial allocation to complete these tasks. Following the completion of the Active Hosts installation work, the site will then be handed over to the remaining MNOs to install their transmission equipment.

Once all of the MNOs that have agreed to utilise the site have installed their transmission equipment, the Active Host will ensure that the mast will become live, providing mobile broadband coverage.

4.2 Management and Governance

The management and governance of the three projects differs, and a summary of the approaches used are presented below:

For PNS, the management and governance of the project sits with the MNOs. As there is no public funding for the project, the infrastructure build can be managed in a similar way to commercial build, with the MNOs reporting internally on progress to ensure that they will achieve their stated objectives. The MNOs do not need to provide any information to DMSL or to BDUK about the sites that they are building or the level of coverage achieved. The MNOs do have to report to Ofcom about the pixels that are covered by their infrastructure and how this has changed since 2021, in order to demonstrate they are meeting their commitments and licencing obligations.

For TNS, the management and governance of the project is largely overseen by DMSL. For each assigned site, the MNO will manage the day-to-day delivery of the project. However, each MNO will report into DMSL with their build plans, expected delivery timelines and any issues that they are having with delivery. This reporting takes place on a monthly basis. The progress is reported into the DMSL delivery board, which then reports on a monthly basis to the SRN Oversight Board, which then meets quarterly. The SRN Oversight Board has representatives from Ofcom, BDUK and the MNOs, as well as regular attendance by the project’s Independent Assessor, Real Wireless. DMSL, the MNOs and Home Office report into the SRN Oversight Board to keep them informed of progress and to escalate any performance issues to.

For EAS, the procurements are run by the Home Office and EE, with BDUK providing funding to the Home Office to upgrade and allow MNOs to utilise masts. Therefore, the reporting on progress made is primarily provided by the Home Office, with delivery updates from the MNOs once sites have been handed over to them. Governance includes a fortnightly Operational Delivery Forum, bi-monthly Strategic Delivery Forum (SDF) (both chaired by the Home Office). These feed into the monthly SRN EAS Project Board (chaired by BDUK) and progress is reported into the SRN Oversight Board and BDUK SRN Programme Board.

Question 5

5. Measurement of outcomes

Accepted Answer

This section sets out a review of the available data sources of relevance to an evaluation of the SRN programme. Sources have been reviewed with a view to establishing the extent to which they could be used to explore the outcomes and impacts defined in the theory of change.

5.1 Overview

The evaluation will involve the collection of the following data:

management Information providing information on progress in delivery of the Programme and to support an assessment of the impact of the SRN on beneficiaries by identifying areas that the programme has operated in
primary research (such as depth interviews, group interviews) with stakeholders involved in the delivery of the Programme and with MNOs and DMSL to obtain strategic views on the impact of the Programme and to support the interpretation of results
case studies to explore the impacts of the programme in specific local areas, including public service provision and to support further analysis of local economic and social outcomes
a quantitative survey to collect information about households and businesses in areas benefitting from subsidised coverage at two time periods (this will be a single survey covering premises in areas benefitting from coverage, which can include households and businesses)
secondary data to explore the connectivity, economic, social and environmental impacts of the programme
innovative data sources, such as social media data and web scraping exercises, to support analysis of connectivity and social outcomes

The table below presents a summary of how the data collection plan relates to the analytical work packages for the evaluation.

Table: data collection summary table

x = relevant work package

Work package	Management Information	Stakeholder consultations	Local case studies	Secondary data sources	Quantitative surveys	Innovative data sources
Availability of mobile broadband and number of MNOs	x	x	x	x		x
Economic impacts	x	x	x	x	x	x
Social and public value impacts			x	x	x	x
Environmental impacts	x	x		x	x
Public service impacts			x	x
Direct impact on beneficiaries	x	x
Consumer outcomes		x	x		x

5.2 Management data

BDUK held data

The evaluation will make use of information collected by the SRN programme. However, due to the nature of the programme, it is anticipated that there will be a limited amount of programme level management information and internal documentation. It is anticipated that the research team will need access to the following information sources:

List of infrastructure build sites: For the TNS and EAS projects, it is understood that BDUK possesses a list of sites which will be built / upgraded to provide enhanced mobile broadband coverage. The evaluation team will need access to a list of these sites (postcode or pixel data), and the dates at which these sites become live for mobile broadband use. This information can be used to support the identification of treatment and comparator areas.

Grant agreements and cost information: BDUK provide grants to MNOs via DMSL for the delivery of TNS and EAS sites. The evaluation team will require details of the grant payments made, including the year of payments. This information will be needed to support an economic evaluation.

Change requests / descoped sites: An initial list of potential sites for the TNS project has been developed, but it is likely that not all sites will be required / will be funded for the project. This is because some sites may prove to be too expensive to use, or have issues with landowners / planning permission, or coverage targets may be met by utilising fewer sites. Where an initial site has not been utilised, the evaluation team would require data on the descoped sites and the reasons the site was not used.

Some of this information may also be held by DMSL and MNOs, as well as BDUK. Therefore, there are multiple routes to accessing this data, and the evaluation team may use alternative routes if these prove more efficient than accessing the data from BDUK (for example due to complications with data sharing agreements).

DMSL / MNO held data

In addition to the BDUK held data, there is information which is held by DMSL or the MNOs which would be beneficial for the evaluation team to have access to. In order to access this information agreements will need to be reached with the MNOs. This data includes:

List of infrastructure build sites: For the PNS, BDUK holds no information about the sites of infrastructure build. The evaluation team would like access to a list of these sites (postcode or pixel data), and the dates at which these sites become live for mobile broadband use. This information can be used to support the identification of treatment and comparator areas.
Total cost information: For the PNS, no cost information is collected by BDUK. It would be useful to the evaluation team to have information about the cost of the infrastructure build for the PNS project (the capital expenditure). Similarly, for the TNS and EAS, BDUK have information about the grant values provided to DMSL and MNO, but not the total cost of the projects. It would be useful for the evaluation team to be able to access this information.

Secondary data sources

There are a wide range of secondary and administrative datasets that could be utilised in the evaluation of the SRN programme. Many of these were used in the previous evaluations of the broadband infrastructure, however scoping research has identified some additional data sources which will also be used for the evaluation. The data sources are presented in the table below and linked to the outcomes and impacts described in section 2.

Table: secondary data sources to be used in the SRN evaluation - connectivity and broadband market data

Data source	Description and access	Key data points	Use in the evaluation
Ofcom Connected Nations	Ofcom Connected Nations data is produced annually detailing the current state of connectivity in the UK. It is publicly available through the Ofcom website. The dataset provides information about fixed line coverage at a postcode level. It also produces data about mobile broadband coverage at a parliamentary constituency level, which includes coverage in the home, outdoors and along major road routes.	Postcode data on Gigabit capable coverage. Ward level data on mobile broadband coverage.	This dataset will underpin the analysis of how the programme has provided additional coverage and will support the process evaluation workstreams. The fixed line data will be important to use to explore the differences in impacts in areas with good fixed broadband and those without. The mobile broadband datasets will be used to identify potential areas to focus case studies in and to support the impact evaluation
Ofcom publicly available Not spot data	Data on a postcode level for areas which do not receive mobile broadband coverage. This shows postcode areas in which at least one property is estimated to be unable to obtain reliable indoor mobile signal from any MNO	Postcode data on total not spots	This dataset could be used to identify Total Not Spots, and those which receive coverage through the programme and those that do not. However, this is not the data which was used to underpin the SRN programme. This data relates to information provided in 2019, therefore they may not accurately match Not Spots for the programme. If Ofcom and DMSL are not able to provide the Not Spot data used to underpin the programme, then this data could be used for the evaluation of the TNS and some of EAS.
Ofcom Not Spot data used to underpin SRN programme	MNOs provided a pixel dataset in 2021 to Ofcom that described their coverage. Ofcom combined this data to show each UK pixel as covered either by zero, 1-3 or 4 MNOs on the SRN’s 4G outdoor definition. This is similar to the Connected Nations data, except this is the baseline for the programme. MNOs will report this data again to Ofcom to demonstrate increases in coverage.	Pixel data on total and partial not spots (postcodes covered by under 4 MNOs)	This dataset could be used to identify Partial and Total Not Spots, and those which receive coverage through the programme and those that do not. However, at this stage it is not certain that this dataset will be made available to the evaluation team.
Ofcom Connected Nations	Ofcom Connected Nations data is produced annually detailing the current state of connectivity in the UK. It is publicly available through the Ofcom website. The dataset provides information about fixed line coverage at a postcode level. It also produces data about mobile broadband coverage at a parliamentary constituency level, which includes coverage in the home, outdoors and along major road routes.	Postcode data on Gigabit capable coverage. Ward level data on mobile broadband coverage.	This dataset will underpin the analysis of how the programme has provided additional coverage and will support the process evaluation workstreams. The fixed line data will be important to use to explore the differences in impacts in areas with good fixed broadband and those without. The mobile broadband datasets will be used to identify potential areas to focus case studies in and to support the impact evaluation.
Ofcom publicly available Not spot data	Data on a postcode level for areas which do not receive mobile broadband coverage. This shows postcode areas in which at least one property is estimated to be unable to obtain reliable indoor mobile signal from any MNO	Postcode data on total not spots	This dataset could be used to identify Total Not Spots, and those which receive coverage through the programme and those that do not. However, this is not the data which was used to underpin the SRN programme. This data relates to information provided in 2019, therefore they may not accurately match Not Spots for the programme. If Ofcom and DMSL are not able to provide the Not Spot data used to underpin the programme, then this data could be used for the evaluation of the TNS and some of EAS.
Ofcom Not Spot data used to underpin SRN programme	MNOs provided a pixel dataset in 2021 to Ofcom that described their coverage. Ofcom combined this data to show each UK pixel as covered either by zero, 1-3 or 4 MNOs on the SRN’s 4G outdoor definition. This is similar to the Connected Nations data, except this is the baseline for the programme. MNOs will report this data again to Ofcom to demonstrate increases in coverage.	Pixel data on total and partial not spots (postcodes covered by under 4 MNOs)	This dataset could be used to identify Partial and Total Not Spots, and those which receive coverage through the programme and those that do not. However, at this stage it is not certain that this dataset will be made available to the evaluation team.

Table: secondary data sources to be used in the SRN evaluation - data usage and satisfaction

Data source	Description and access	Key data points	Use in the evaluation
Ofcom Telecoms market data	Data on mobile broadband subscriptions, data service revenue and volume for the UK. This data is provided to Ofcom by MNOs.	Average monthly data use, spend on mobile telecoms, Mobile Broadband subscriptions.	This data is available at a national level, and could be used at this level to assess the market impact of the SRN. For more detailed geographic analysis, for example looking at the local authority areas the SRN has operated in, further exploration will be needed from Ofcom.
Ofcom customer service data	Data on the level of satisfaction customers report for their mobile provider, including satisfaction with reception or signal strength.	Customer satisfaction with provider.	This data is available at a national level (England, Wales, Scotland and Northern Ireland), and could be used at this level to assess the market impact of the SRN. For more detailed geographic analysis, for example looking at the local authority areas the SRN has operated in, further exploration will be needed from Ofcom.
Mast data	Data on coverage and signal strength by MNO	Strength of signal available	This dataset is available for the UK, and geographies can be drilled down. However, the data is proprietary. Data is compiled by planning application analysis and users, which means the data is not guaranteed to be complete.

Table: secondary data sources to be used in the SRN evaluation - economic outcomes data

Data source	Description and access	Key data points	Use in the evaluation
Business Structure Database	This provides vital data for enterprises and local units on key variables such as employment, turnover, Standard Industrial Classification and output area. Data is provided from 1997.	Turnover and employment for businesses by output area.	Local units / businesses in an output areas which have been subject to SRN interventions can be collated and turnover per worker can be calculated to assess the changes in labour productivity.
Annual Business Survey	The Annual Business Survey is a large-scale survey of enterprises in the UK, which includes a financial enquiry aiming to establish levels of capital investment, turnover, GVA, employment, and other financial measures. The survey is mandatory for all large firms (250 employees or more) and as a consequence, provides a potentially useful longitudinal panel dataset that could be exploited.	GVA, factor cost data per business by output area.	The availability of the ABS is subject to some lags. Sample sizes are likely to cause issues in the use of this dataset, as there will be a limited number of businesses with 250+ employees within SRN treatment areas.
Claimant Count	NOMIS provides longitudinal data on the number of claimants (combining Jobseekers Allowance and Universal Credit claimants) at a small area level (LSOA)	Out of work benefits claimant’s by LSOA.	This can be used to explore the effects of the programme in reducing unemployment and exposure to long-term problems of unemployment as well as benefits dependency.
Annual Survey of Hours & Earnings	The Annual Survey of Hours and Earnings can be accessed at individual level through ONS SRS to analyse real hourly wage changes. Data is available from 1997, and covers earnings, hours worked, age, gender, industry and occupation.	Wages, hours worked by output area.	This is a possible alternative to measuring productivity changes, although the relationship between wage growth and productivity growth is tenuous and can be influenced by exogenous variables separate from productivity growth.

Data source	Description and access	Key data points	Use in the evaluation
Annual Population Survey	The Annual Population Survey can be accesses at individual level through ONS SRS. It includes data points on subjective well-being, as well as age, employment status and occupation.	Subjective well-being by output area.	This dataset can be used to undertake further secondary data analysis to estimate the impact of the programme on subjective well-being.
Understanding Society Survey	The Understanding Society survey is a UK household longitudinal study which surveys each adult member of a nationally representative sample about subjects such as health, work education, income, family and social life, all of which would serve to understand the general well-being of the UK population.	Subjective well-being by LSOA	This dataset can be used to undertake further secondary data analysis to estimate the impact of the programme on subjective well-being.
Land Registry / Registers of Scotland	Data on median house prices are publicly available through the ONS available as house price statistics for small areas in England and Wales, including individual data on house sales. This dataset is updated quarterly and details the mean or median price paid for houses by MSOA in that quarter. The dataset also includes variables for the type of property. Similar data is available for Scotland through the Registers of Scotland.	Mean and median house prices by MSOA.	This dataset could be used to explore property market effects of the SRN as a proxy measure for changes in subjective well-being. The one drawback of this is that the geographic granularity (MSOA level) has the potential to dilute the extent to which effects might be visible should effects be very localised (within a small number of postcodes from an intervention site). This dataset only covers England and Wales, however the registers of Scotland data provide comparable data for Scotland.
Census data	Census data provides a variety of useful contextual data about the makeup of local areas – for example population sizes, age profiles, demographic information, travel to work data.	Population size and demographic indicators; journey to work times	The census is undertaken every 10 years, and the 2021 baseline will act as a baseline measure for this evaluation. 2031 data could be used to explore population changes in treated areas – however, this would be outside the scope of the existing evaluation.

Table: secondary data sources to be used in the SRN evaluation - public sector outcomes data

Data source	Description and access	Key data points	Use in the evaluation
Patient Online management information programme data	This tracks individual GP practices and its total number of online patients registered to use an online Patient Recorded Access Service. Starting in 2015-16, data is collected on an annual basis.	E-health activity by GP practice and postcode.	This data can be matched to programme delivery areas to explore any changes in e-health activity in areas that have been upgraded.
GP patient survey	The GP patient survey is a postal survey providing practice level data on the views of GP patients on their experiences of primary care and will be used to refresh estimates of the impact of the programme on the efficiency of public service delivery.	Patient satisfaction, use of online services by GP practice and postcode	This data can be matched to programme delivery areas to explore any changes in patient views or usage in areas that have been upgraded.

Table: secondary data sources to be used in the SRN evaluation – Tourism

Data source	Description and access	Key data points	Use in the evaluation
Visitor Attractions data	VisitBritain collects data on visitor numbers to specific attractions. If these attractions fall in an area benefitting from the programme visitor numbers could be impacted.	Number of visitors to attractions	This dataset could be used to explore the effects of the SRN on visits to tourist attractions.
Accommodation occupancy and short-term rental data	VisitBritain also collects occupancy and short-term rental data. This data could potentially be disaggregated to smaller geographic areas.	Occupancy rates of short-term rentals	This dataset could be used to explore the effects of the SRN on occupancy in rental homes in areas where the programme has operated.

Table: secondary data sources to be used in the SRN evaluation – Environmental

Data source	Description and access	Key data points	Use in the evaluation
DfT road traffic statistics: Basemap countpoint data^{[footnote 12]}	DfT road traffic data provides street-level traffic data for road-links on the motorway, ‘A’ road and minor road network in Great Britain up to 2023	DfT publish manual and automatic vehicle counts for major and minor roads. Local road network traffic counts can be obtained independently from each local authority.	Traffic count data could be analysed to estimate the causal effect of the programme in terms of the number of journeys made in the rollout areas compared to control group areas. However, this relies on the treatment areas having sufficient traffic count points to develop a sufficiently large sample for analysis, which would need to be verified once the programme areas have been provided. This data can be applied to DfT TAG values for air pollution and Greenhouse Gas emission reductions associated with these journey savings.
TrafficMaster congestion Data^{[footnote 13]}	TrafficMaster data is licensed by the DfT and provided free to local authorities by DfT.	Data on bi-directional traffic speeds and congestion levels on relevant and adjacent streets.	This dataset could potentially be used to assess the impact of the programme on travel speeds and therefore journey times (for example from using navigation apps or faster recoveries of accidents / broken down vehicles). However, this relies on the treatment areas having sufficient traffic count points to develop a sufficiently large sample for analysis, which would need to be verified once the programme areas have been provided.

5.3 Primary data collection

This section presents the proposed approach to collecting data through primary research methods. This approach has been designed to complement and fill gaps from the secondary data sources listed above.

Household survey

A survey of premises that are covered by the enhanced coverage provided by the SRN will be undertaken. This will include residents and businesses that are covered, with the majority of responses expected to be from households. The survey will take place over two points in time – a baseline survey, undertaken before the enhanced coverage is provided (in mid-2024), and a follow-up survey, taking place at least six months after the enhanced coverage has been provided. The timing of the follow-up survey for each programme may differ, as the deployment times for the enhanced infrastructure differs. The survey will be conducted using a cross-sectional approach, meaning the same areas will be targeted at baseline and follow-up, but not necessarily the same individuals.

Sample selection will be an important consideration, particularly if BDUK wish, for example, to achieve minimum response levels for the PNS, TNS and EAS projects. It is anticipated that an average of 20 households and 28-29 individuals would participate per 100 addresses sampled. Within households with up to two adults, the survey will select all adults to participate; for households with three or more adults, it would allow two adults to participate. With this approach shows we can expect 1.4 adults to complete the survey within each participating household. With these estimates, the survey would aim to achieve approximately 1,400 responses from a sample of 5,000 addresses. The sample will include households and businesses being covered by two of the three projects (PNS and EAS) due to an inability to access data on the delivery sites of TNS, with the majority being those covered by the PNS project. For TNS, a baseline survey could be implemented at a later date to collect the same information, depending on the size of the population covered.

Sampled individuals must have a motive for taking part in a study, and these motives need to outweigh their reasons for not taking part. Motivations to respond will differ across the sample. Typical motivations for taking part in a survey are:

altruism (e.g. the importance of the study for the government)
self-interest (e.g. wishing to influence policy)
survey-specific factors (e.g. an interest in the survey topic, seeing how the survey works)

These reasons for taking part will be promoted in the initial communications but will not persuade everyone. In general, incentivising survey participation increases response rates, and are commonly used in ‘push-to-web’ surveys where they are especially effective in the absence of an interviewer. Incentives improve response rates, but with diminishing returns as the value of the incentive increases. The precise impact on a survey’s response rates is difficult to estimate in the absence of experimental evidence. The survey design has costed for the use of incentives - a £10 shopping voucher per completed questionnaire.

The survey contact strategy would be as follows:

invitation letters: to all sampled addresses inviting up to two resident adults to participate
reminder letter 1: 1-2 weeks later to non-responding/partially responding households
reminder letter 2: 1-2 weeks after reminder 1 to all non-responding/partially responding households. Non-responding/partial households will receive two or one postal questionnaires respectively
final reminder (postcard in envelope): 1-2 weeks after reminder to non-responding/partially responding households

Offering a postal questionnaire option is also essential for obtaining a representative sample. It is important the digitally excluded (a relatively low proportion of adults) can participate but also those more likely to respond to a paper questionnaire

The survey would run for eight weeks in each wave, both before and after infrastructure has been implemented. The questions included in the survey will be consistent across all project areas, and include:

demographic information
the fixed broadband connection they have in their home
how the participants use the internet at home and outside the home
their levels of health and wellbeing
their levels of satisfaction with their mobile broadband
economic activity and working from home arrangements
travel times and frequency / mode of travel
recontact question
questions exploring willingness to pay for mobile broadband (in the baseline survey only)

Local case studies

A series of case studies will be used to collect evidence of the impact of the SRN programme (and mobile broadband deployment more generally) in local areas. It is proposed that six case studies are undertaken, utilising two areas for the PNS, three areas for the EAS and one area for the TNS project. Selecting local areas for the PNS project may prove challenging given the potentially dispersed nature of the intervention, but it should still be possible to identify a sufficient population that have been impacted by the change in coverage. The case studies will be used to provide more details of how enhanced mobile broadband can support local communities. In particular, the local case studies will focus on:

How enhanced connectivity has supported local businesses: For example, whether enhanced mobile broadband helps divert customers to the business (particularly for tourism and food service businesses), how connectivity has supported their customers, operations, output and employment.
How enhanced connectivity has supported the local authority to implement digital strategies and deliver public services: Have local authorities been able to implement strategic changes as a result of the connectivity provided by the programme and deliver their local services. This could include how mobile connectivity supports mobile working and working from home.
How has tourism altered as a result of enhanced connectivity: Whether there has been an increase in the number of visitors, whether businesses have altered how they market themselves and so on.
How enhanced connectivity affects how people feel about where they live: How local residents feel about the area they live in, how enhanced connectivity alters their perception of the local area.

The case studies will explore how mobile broadband has supported local communities and can be used to explore differences in the experiences of areas that have improved mobile broadband infrastructure and those that have not yet received enhanced coverage.

The research team will work with BDUK to identify areas which would be suitable for local case studies. For TNS, these are likely to be in rural Scotland, but some of the PNS and EAS case studies may be in less rural areas.

The research team would initially make contact with the local authority selected and hold an initial workshop with the local authority to identify the types of topics the case study should focus on, and the types of stakeholders who should be contacted for the research and to ask for their support in identifying how to make contact with the stakeholders. The research team would then develop a short, two-page plan for the case study, and share this with the local authority and BDUK. We would recommend researchers go in person to the local authority area to conduct research face-to-face with stakeholders. Each case study will comprise a series of six depth interviews with the following groups of stakeholders (not all types of stakeholders would be interviewed for all case studies):

Local authorities: To understand its approach to providing public services has altered due to enhanced connectivity, how enhanced connectivity has supported the development and delivery of local digital strategies, how connectivity has impacted on the local economy and perceptions of the area. There may be challenges in engaging with the local authority for the TNS case study for local opposition and political reasons. The research team will work with BDUK and other relevant organisations to try to secure engagement.

Local public service providers: To explore similar issues around the delivery of public services, and any challenges faced due to users of the service not having access to gigabit capable broadband connections.

Local businesses or business groups: These could be the local chamber of commerce or sector bodies, alongside businesses, to explore how connectivity in the local area has supported their business.

Local interest or community groups: These could be local community organisations or organisations with an interest in a specific area or community asset (for example groups with an interest in geographic parks or a museum, theatre), to explore how enhanced mobile connectivity impacts upon the community / area / asset they represent.

It is anticipated that interviews will last for 30 minutes. Interviews will be offered via Microsoft Teams but would ideally be undertaken face to face in the local area, to allow researchers to visit the area and generate a better understanding of the locality and how connectivity will support the area.

Stakeholder consultations

In order to understand the outcomes achieved by the SRN programme, and to identify some of the more qualitative outcomes, it will be essential to undertake primary research with a range of stakeholders. This will be to support the assessment of impacts on programme beneficiaries (how the contracts have affected the contract holders), an understanding of the nature of the mobile broadband market at the time that the programme is being delivered, for example challenges faced by the market or opportunities, to get a better understanding of the additionality of the intervention. The stakeholder consultations will also be essential in collecting information about some of the wider outcomes of the programme, such as tourism outcomes, effects on areas of natural beauty, potential environmental impacts of the programme (particularly related to the construction and running of mobile broadband masts).

Interviews with these types of organisations will help to contextualise the findings from the analysis of management information and secondary data sources. The evaluation has budgeted for 40 stakeholder interviews. It is recommended that these interviews are distributed in the following way:

12 interviews with MNO staff (to explore coverage, technical and environmental outcomes) – assumed to be three interviews for each of the MNOs
10 interviews with wider stakeholder groups. These are to include stakeholder groups with an interest in the environmental effects of mobile broadband in rural areas (potentially Campaign to Protect Rural England or CPRE, and the John Muir Trust), business groups in the areas affected (for example local Chambers of Commerce), tourist organisations (such as VisitBritain and the Holiday Homes Association), and consumer organisations (for example Which)
eight interviews with key stakeholders from DMSL, BDUK and Ofcom, exploring how the Programme has been delivered and the impacts it has had on the mobile broadband market
10 interviews with local authorities in areas that the programme has operated in (in addition to the local case studies), to explore some of the local impacts and barriers to the programme achieving its objectives

It should be noted that it is known that some of the wider stakeholders and local authorities are opposed to the SRN programme, in particular new infrastructure build in very rural areas. However, it is important that the evaluation collects evidence from these organisations and analyses this objectively. Prior to selecting stakeholder groups to participate, the evaluation team will explore public statements from potential organisations to determine their likely views and to ensure the sample does not include only organisations with a single view of the SRN programme.

The table below provides a summary of the topics the stakeholder consultations should cover.

Table: qualitative interviews: coverage of topics

x = Qualitative interviews conducted

Topic	MNO	DMSL	Wider stakeholder groups	BDUK	Ofcom	Local Authority
Impact of the Programme on MNOs	x			x
Market conditions	x	x		x
Planning applications	x		x			x
Delivery of infrastructure build	x	x	x	x	x	x
Supply chain capacity	x	x
Pricing	x	x	x		x
Environmental impacts	x	x	x			x
Business and tourism impacts			x			x

It is anticipated that interviews will have a duration of 60 minutes and will be undertaken using Microsoft Teams. Interviews will be recorded and transcribed with the consent of the respondent. The sample for these qualitative interviews will be agreed with BDUK. Interviews will be coded and analysed thematically using the NVivo qualitative analysis software package, and where conflicting responses are provided, these will be compared to objective data points (from management or secondary data) to establish the reliability of the statements.

5.4 Innovative data collection

In addition to the more traditional forms of data collection described above, the evaluation will also utilise alternative, innovative approaches to data collection. These are described in more detail below.

Data scraping planning applications

To supplement (or to use in the absence of) management data around the location of infrastructure build, and to identify some of the challenges the Programme has had with planning permission, the evaluation will identify all planning applications for MNOs. To do this, we will identify all planning applications submitted to local authorities and collect information about their status (for example approved, declined). As there are many local authorities and applications, this will be undertaken using web scraping approaches, and the information from the planning applications pulled into a single file to analyse.

One of the key elements of this approach is to identify suitable terms to search to identify mobile broadband infrastructure build. It is anticipated that terms such as ‘4G’, ‘Antenna’, ‘telecommunications infrastructure’, ‘transmitters’, ‘M-Range’, ‘pole’, and ‘transmission dish’. These will be searched for using an ‘or’ Boolean search.

Following the collection of the data, the information will be examined to check that the planning application relates to an application for 4G mobile broadband, and irrelevant data collection will be removed.

One of the outcomes highlighted in section 2 was around tourists’ views of areas of natural beauty. One way that this information could be captured is through social media posts about particular areas of natural beauty. Ipsos have an in-house AI social listening technology provider, Synthesio, which can extract and analyse conversations on social media (including X, formerly known as Twitter, Instagram). This tool could be used to capture a number of outcomes of the SRN programme, including:

views of tourist attractions and areas where coverage has been provided
views on areas of natural beauty where mobile broadband infrastructure has been constructed

The tool can extract data from time periods before the SRN programme provided mobile broadband coverage / infrastructure in an area, and after. The research team would use tags for specific locations or tourist attractions on social media.

The tool will also collect data on tourist attractions and beauty spots where as there was existing mobile broadband coverage prior to the SRN. Areas where there is still no mobile broadband coverage after the SRN programme, to provide a comparator for the SRN locations.

Synthesio will identify those who posted, reposted, tagged, and/or commented about the tourist attractions / areas and what was posted, and will assign a scale of positive and negative sentiment to the post. The analysis will select a period of time (for example a three-month window) for both the before and after infrastructure build. The analysis will include the number of positive and negative posts, and the strength of the post to inform the analysis.

It should be noted that there are potential issues with the representativeness of individuals who are posting on social media, and those posting about specific topics. However, the analysis should give some indications about the direction of travel of views of attractions and areas.

Further data sources

The evaluation team will also explore the potential to collect further location information which may highlight use of mobile broadband apps as they become more widely available. One example is Strava data, which may help to locate when and where people are undertaking activities, and activities around new infrastructure build could be examined.

Question 6

6. Evaluation framework

Accepted Answer

This section presents the approach to a process evaluation of the SRN programme. It sets out the key evaluation questions and how evidence will be collected to analyse the effectiveness of the SRN programme.

6.1 Summary of recommended evaluation approach

The recommended evaluation approach includes a process, impact and economic evaluation. The process evaluation will collect evidence to answer key evaluation questions around the programme design, identification of intervention areas, engagement and site survey activities, planning applications, grant agreements, infrastructure build and management and governance. Information will be collected and analysed using programme management information, secondary data sources and stakeholder interviews.

To assess the impact of the programme on mobile connectivity, different approaches should be used for the TNS and the PNS and EAS.

For TNS, it is recommended that the areas the project provides coverage to is compared to a counterfactual case of total not spots that do not receive coverage through infrastructure funded by the TNS using a difference in difference technique to estimate the impact the project.

For EAS and PNS, a before and after analysis or interrupted time series analysis is proposed as the most appropriate approach to estimate the impact of the projects on mobile broadband availability. This would explore if the provision of PNS and EAS infrastructure has led to a step change in coverage.

To assess the downstream economic impacts, a pipeline approach to developing a counterfactual case is proposed, which will utilise matching and difference in difference techniques to estimate the impact the programme. Alongside this, a pipeline approach can be used to measure the economic impacts of the EAS and PNS individually, subject to sufficient numbers of businesses being located within the areas the programmes have provided enhanced coverage to. For an assessment of the downstream economic impacts of TNS, a mixed methods approach will be utilised, due to the small numbers of businesses that are expected to be located within the TNS footprint.

To assess the social impacts of the programme, and some environmental impacts, a before and after approach will be utilised, exploring the change in outcome metrics between a time when there was no enhanced mobile broadband connectivity and when enhanced mobile broadband connectivity has been introduced.

For all outcomes a contributions analysis approach is recommended to explore how and why outcomes have been achieved and how the SRN programme contributed towards this.

For the economic evaluation, it is recommended that a cost-benefit analysis approach is undertaken, comparing the lifetime costs of the programme to the long-term economic, social and potentially environmental impacts of the programme. Alongside this, a value for money (VfM) assessment will be undertaken, which can be undertaken separately for the three programmes and for the SRN as a whole.

6.2 Process evaluation framework

The processes used to deliver the SRN programme are set out in section 3. The table below presents the process evaluation questions which can be used to assess how effectively the programme is being delivered, where information could be collected from to assess the key process evaluation questions in order to identify areas where processes could be improved.

Table: process evaluation framework

x = Evidence available

Process	Evaluation questions	Management Information	Secondary data	Home Office interviews	Ofcom interviews	DMSL interviews	BDUK staff interviews	MNO interviews
Programme design	How were the targets for mobile broadband coverage set?	x	x		x	x	x	x
Programme design	Were all relevant stakeholder groups involved in setting the coverage targets?				x		x	x
Programme design	Were all potential funding options (grant funding) considered?				x	x	x	x
Programme design	How appropriate was the evidence used to shape the final Programme design?	x			x	x	x	x
Programme design	Were all options for the delivery vehicle for TNS and EAS (DMSL) considered?			x	x	x	x

Process	Evaluation questions	Metrics	Management Information	Secondary data	Home Office interviews	Ofcom interviews	DMSL interviews	BDUK staff interviews	MNO interviews	Local authority interviews
Identification of intervention areas	How accurate was the data used to identify partial and Total Not Spots in 2021?	Number of Partial and Total Not Spots	x	x		x			x	x
Identification of intervention areas	To what extent was all relevant evidence used to identify potential sites for PNS to maximise coverage across the four MNOs?								x
Identification of intervention areas	Were all relevant stakeholders/organisations involved in the process to identify relevant sites?				x		x	x	x	x
Identification of intervention areas	How effectively did the PNS, TNS and EAS projects work together to avoid duplication of areas?				x		x		x
Identification of intervention areas	How effectively did the four MNOs work together to identify potential sites for the TNS?						x	x	x
Identification of intervention areas	Were all relevant factors considered when identifying areas for TNS sites, including impacts on MNOs?						x	x	x
Identification of intervention areas	How could the identification of areas process have been improved?				x	x	x	x	x	x

Process	Evaluation questions	Home Office interviews	DMSL interviews	BDUK staff interviews	MNO interviews
Engagement with landowners	How effective were the processes used to engage with landowners about access to potential sites?	x	x		x
Engagement with landowners	What were the key challenges in engaging with landowners to arrange potential access to identified sites?	x	x		x
Engagement with landowners	To what extent did support from central Government aid this process?		x	x	x

Process	Evaluation questions	Metrics	DMSL interviews	MNO interviews
Site survey activity	What were the main challenges in undertaking site surveys and how were these overcome?		x	x
Site survey activity	To what extent were all relevant staff available to undertake a site survey in a timely manner?			x
Site survey activity	Were site surveys completed in a timely manner to support delivery?	Duration between site identification and site visit	x	x
Site survey activity	How appropriate were the levels of resources required to undertake a site visit?	Staff and equipment cost required for site visit	x	x
Site survey activity	Were appropriate thresholds used to assess site survey information and inform decisions on whether to proceed with the site?		x	x

Process	Evaluation questions	Metrics	Secondary data	Home Office interviews	DMSL interviews	BDUK staff interviews	MNO interviews	Local authority interviews
Planning applications	What were the main challenges in submitting planning applications to local authorities and how were these overcome?		x	x	x	x	x	x
Planning applications	What were the main challenges with securing permanent access to sites (and wayleaves) for the construction of infrastructure and how were these overcome?		x	x	x	x	x	x
Planning applications	How effective was the support provided by DMSL and central Government in submitting planning applications and securing access to sites?				x	x	x
Planning applications	To what extent were planning applications and access to sites resolved in a timely manner to support the construction of infrastructure?	Duration of time between site identification and planning approval	x		x	x	x	x
Planning applications	To what extent was the feedback on rejected planning applications sufficient to inform decisions on resubmission of applications?				x	x	x	x
Planning applications	How did the submission of planning applications and securing access to sites for the SRN differ to business-as-usual approaches?						x	x

Process	Evaluation questions	Metrics	Management Information	Home Office interviews	DMSL interviews	BDUK staff interviews	MNO interviews
Grant agreements	Were all relevant stakeholders involved in the design of the grant agreements?				x	x
Grant agreements	Were all relevant factors considered when deciding the grant amount for each infrastructure build?				x	x	x
Grant agreements	To what extent did the wording of the grant agreement ensure the Programme aims were met?					x
Grant agreements	How effective were the processes to query a grant agreement if the infrastructure build encountered difficulties?					x	x
Grant agreements	What were the main challenges with the grant agreement and how were these overcome?			x	x	x	x
Grant agreements	To what extent were the value of the grants sufficient to cover the additional cost of construction and operating costs?	Value of grants Cost of construction Operating costs	x		x	x	x
Grant agreements	Were all relevant stakeholders involved in the design of the grant agreements?				x	x
Grant agreements	Were all relevant factors considered when deciding the grant amount for each infrastructure build?				x	x	x
Grant agreements	To what extent did the wording of the grant agreement ensure the Programme aims were met?					x
Grant agreements	How effective were the processes to query a grant agreement if the infrastructure build encountered difficulties?					x	x
Grant agreements	What were the main challenges with the grant agreement and how were these overcome?			x	x	x	x
Grant agreements	To what extent were the value of the grants sufficient to cover the additional cost of construction and operating costs?	Value of grants Cost of construction Operating costs	x		x	x	x

Process	Evaluation questions	Metrics	Home Office interviews	Ofcom interviews	DMSL interviews	BDUK staff interviews	MNO interviews
Infrastructure build	Did MNOs have sufficient capacity to delivery build at the scale anticipated? Did the Programme have any adverse effects on parallel programmes of investment?				X		X
Infrastructure build	How effectively were risks and managed?		x		x	x	x
Infrastructure build	How effectively did the MNOs engage with other organisations required to support the infrastructure construction (e.g. national grid)?						x
Infrastructure build	Were infrastructure builds completed in line with time expectations?	Number of builds completed in line with expectations	x	x	x	x	x
Infrastructure build	How effectively did MNOs work together to ensure sites had the required equipment to support all MNOs in a timely manner?		x		x	x	x
Infrastructure build	What challenges did MNOs face in delivering the build and how did DMSL/BDUK support overcoming the challenges?		x		x	x	x

Process	Evaluation questions	Home Office interviews	DMSL interviews	BDUK staff interviews	MNO interviews	Local authority interviews
Management and Governance	How effectively did DMSL and BDUK support the delivery of the SRN?	x	x	x	x	x
Management and Governance	To what extent did the project management approach ensure the project was delivered on time, to budget, and in line with the project aims?		x	x
Management and Governance	Were requests to make changes to build sites responded to a timely manner?		x	x
Management and Governance	How effective were the data sharing processes between BDUK, DMSL and MNOs?		x	x	x
Management and Governance	Were there differences in project delivery by Programme?		x	x	x

6.3 Impact evaluation framework

The key outcomes of the SRN programme have been outlined in section 2, and the data which will be used to measure these outcomes is set out in section 4. In this section, the approach to using this data to assess the impact of the SRN Programme is set out.

Impact evaluation questions

The impact evaluation of the SRN Programme has the following aims:

provide a robust assessment of the extent to which the outcomes of interest (as defined in the theory of change) would have occurred without BDUK’s investments in the SRN Programme
understand the key procedural and contextual factors that determine the degree of success achieved to support on-going learning and aid improvements to the development and implementation of broadband coverage

The evaluation team have developed some key impact evaluation questions, which are set out below:

What are the benefits and disbenefits of mobile broadband connectivity to households, businesses and public service providers?
What are the differences in impacts between moving from no mobile broadband coverage to total coverage and moving from partial to total coverage (including the relative change from one, two or three MNOs to four MNOs)?
What would be the implications of not having mobile broadband coverage?
Do the impacts vary by type of business, type of household or area?
Through what mechanisms does access to mobile broadband influence changes in behaviour among households and businesses?
How are the impacts expected to vary over time? Which impacts persist over time, and are there lagged impacts?
How has the programme impacted upon the MNOs delivering mobile broadband infrastructure? Has it had a material impact on market positions?

Alongside these, there are two subsidy control impact evaluation questions, which are covered by the questions that are set out above. The subsidy control questions are:

What are the expected outcomes of the SRN programme? Did the programme achieve its expected outcomes?
What changes individuals’ / organisations’ behaviour for these outcomes to come about?

Impact evaluation methodology

This section sets out the considerations and potential approaches for the impact evaluation design, before setting out the recommended approach at the end of the section.

Key considerations

There are several key issues that need to be addressed in the design of the impact evaluation:

Counterfactual: The provision of mobile broadband networks (in the absence of the programme) depends on the investment decisions made by private suppliers. There is a possibility that expansions in 4G coverage resulting from the programme would happen in the absence of the SRN or happened at a slower pace. However, indications from examining data and the qualitative research undertaken so far is that there would have been little rollout of 4G infrastructure in the absence of the programme, as MNOs had already provided mobile broadband coverage to nearly all areas it was commercially viable for them to do so, and had switched to providing rollout of 5G services in more urban areas.

Despite the qualitative information pointing towards no commercial rollout in the absence of the programme, a credible evaluation will still need to explore the counterfactual case in detail. To provide a credible quantitative assessment of what would have been achieved without the programme, it is necessary to compare areas affected by the programme to a comparison group of areas that did not benefit or have not yet benefitted but are otherwise equivalent to those that did. The selection of this comparison group involves several complications. Areas which are selected to receive coverage through the SRN may systematically differ from other Partial or Total Not Spots that are not included in the programme. For example, areas selected for support in the PNS programme are likely to be more commercially viable, or less costly to build infrastructure in, than other partial not spot areas (for example having a higher population of potential customers); or the TNS landmass covered by the programme may have more premises within range of the infrastructure build than other total not spot areas.

Programme: There are three individual projects which are being delivered through the SRN – the PNS, TNS and EAS. There are significant differences between the areas being targeted by the different projects, how the projects are funded and operate and when infrastructure build will occur. These differences were determined by the existing level of mobile broadband coverage by the MNOs and the location of the Emergency Services Network (managed by the Home Office) and the potential benefits that they would generate. This may mean there are significant differences between the areas covered by each of the projects (such as population size, number of businesses, rural urban, levels of fixed broadband coverage). As most of the delivery of PNS is occurring before the delivery of TNS and EAS, and these areas are likely to be significantly different, this may present some challenges for the selection of a counterfactual area.

Scale of impact: The impact that the SRN programme has is likely to vary depending on the relative change in mobile broadband coverage in an area. For example, a change from no mobile broadband coverage to having four MNOs providing coverage is likely to differ significantly from the impact of moving from three to four MNOs providing coverage. Therefore, it will be important for the impact evaluation to consider the starting position of treatment areas in terms of MNO coverage, and not just whether the area has received enhanced mobile broadband coverage.

Fixed line infrastructure: Alongside mobile broadband businesses and households have access to fixed line broadband (and potentially fixed wireless or satellite solutions). Significant differences in the coverage of fixed line solutions (for example the technologies and speeds households and businesses are able to access) and costs may lead to differences in the impacts associated with the provision of mobile broadband coverage – therefore the existing and future coverage of fixed broadband infrastructure will need to be considered when designing a counterfactual case.

Timescales: Some of the key outcomes and impacts of the SRN are dependent on take-up and usage of mobile broadband networks. Some of these impacts may develop beyond the current timescale for the evaluation, and therefore will not be captured by any impact analysis. Therefore, it is important to note that the impacts measured will be the impacts as of the year of the evaluation. The impacts which are measured using secondary data sources could be revisited in future years to explore if the outcomes and impacts have further evolved over time.

Assessment of potential for developing a counterfactual case

The options available to identify a counterfactual for the evaluation of the impact on mobile broadband coverage are presented below. The options are dependent on having access to data about where SRN infrastructure is built/enhanced, and when infrastructure goes live. The discussion below includes considerations about the robustness of the counterfactual case. As the programme has already begun delivering, and the nature of the intervention, a randomised control trial is not feasible for this evaluation. The research team have identified two potential approaches to developing a counterfactual case, which are presented below.

Option A1: Partial not spots and total not spots not covered by the SRN

The SRN Programme aims to bring coverage of mobile broadband for each MNO to 90% of the UK landmass, which is expected to mean that 95% of the landmass has some form of mobile broadband coverage. However, the programme will still leave some locations which are partial and Total Not Spots.

The evaluation team have not yet been able to analyse exactly where the PNS, TNS and EAS infrastructure build is planned to be or has been delivered, or the Partial and Total Not Spots which are not going to receive mobile broadband coverage. Therefore, in theory, these non-delivered to areas could be used as a counterfactual case for the SRN programme.

In practice, it may be possible to use this approach for the TNS project. As the grant funding is awarded to MNOs to build infrastructure in these total not spot areas, the expected commercial rollout in these areas could be expected to be similar to the total not spot areas that do not receive grant funding for coverage.^{[footnote 14]} However, this will need to be explored in more detail once the research team can access the location data.

To ensure that there is comparability between the grant funded and unfunded areas, a matching approach would need to be undertaken (see analytical approaches below). This would ensure that analysed non grant funded areas shared similar characteristics to those included grant funded infrastructure build plans. The matching approach would need to consider factors such as:

supply side factors: for example, distance from other mobile broadband infrastructure, distance from fixed line broadband infrastructure, average download speeds.
demand side factors: for example, premise and population density, average household income, employment, number of visitor attractions in the area.

However, this approach will not be feasible for the PNS and EAS projects. For PNS, because of the universal obligation on MNOs to increase their mobile broadband coverage, and because it is privately funded, there will be no commercial rollout of mobile broadband infrastructure which falls outside the project remit. All infrastructure build undertaken by MNOs since 2021 that is in Partial Not Spot areas that is not grant funded will fall under the PNS. Similarly for EAS, any commercial build brought forward in that time would fall under the PNS project – therefore there is no unfunded comparator group.

Option A.2: pipeline methods

The SRN is being delivered in a phased manner, with the PNS delivery expected to be completed in 2024, and EAS and TNS expected to deliver coverage in later years to 2026. Therefore, in theory, a pipeline approach could be used, comparing the coverage achieved in PNS areas with coverage delivered in areas that the TNS and EAS are expected to deliver to in the future. However, this approach again is unfeasible. Firstly, the MNOs know which areas are going to receive grant funding from the EAS and TNS, so in any event would be unlikely to commit private funding to these areas. Additionally, the areas the PNS is delivering to are significantly different to the areas TNS will provide coverage to, so these areas would not be a suitable counterfactual case.

Option B.1: pipeline design

A pipeline design could utilise the staggered rollout of the SRN programme to estimate the downstream economic, social and environmental impacts of the programme. Due to the small areas and populations covered by the TNS project, assessing the downstream impacts of the SRN intervention will be more appropriate than attempting to estimate the impacts for each project individually. However, it should be feasible to explore the impacts of PNS and EAS projects individually, subject to a sufficient number of beneficiaries living or working in the footprint of each intervention.^{[footnote 15]}

The basis of the pipeline design for the downstream impacts is that areas that have received BDUK investment first are likely to experience the impacts of those investments first. The approach has the following features:

Feasibility: This approach is only feasible with longitudinal panel data (i.e. annual observations of the same individuals, firms, or areas) to exploit staggering in the rollout of the programme. This does not present an issue for analysis focusing on the secondary data sources identified in section 4.

Selection bias: As mentioned above, the strength of the results are partly dependent on an assumption that there are no systematic differences between areas receiving investment earlier and later that are also correlated with the outcomes of interest. For example, if the areas receiving mobile broadband coverage earlier are areas experiencing higher productivity growth in the absence of the programme, then this will overstate the impact of the programme. This assumption would need to be examined prior to the analysis taking place, as PNS areas differ in terms of population and mobile broadband coverage from the other areas covered by the SRN programme. However, this is less of an issue when exploring the impact of the PNS and EAS programmes individually, although the order of PNS rollout may have been influenced by commercial viability and populations served, so this will still need to be investigated.

Nature of results: This approach identifies the impact of broadband infrastructure but does not account for whether the infrastructure would have come forward without the programme. As such, results would need to be combined with estimates of the likelihood infrastructure would not have come forward in the absence of the programme (from the analysis on coverage described above) to provide an estimate of its net effects.

Defining a ‘treatment’ variable at an area level: For analyses at an area level, an appropriate measure of the treatment variable will need to be defined, as the number of premises receiving enhanced mobile broadband coverage is likely to vary substantially. Assuming that the scale of the impact will be proportional to the number of premises enabled with mobile broadband, it may be appropriate to reflect this in the modelling (by defining the treatment variable as the number of premises enabled).

Option B2: synthetic control group

An alternative approach to forming a counterfactual group for the SRN would be to form a synthetic control group. This approach uses historical data to construct a ‘synthetic clone’ of a group receiving a particular intervention, in this case businesses which are in areas which have received enhanced mobile broadband coverage. The synthetic control group would need to be created from businesses that were in locations which were Partial or Total Not Spots in 2021. However, this approach would need to create matched businesses using mobile broadband, fixed broadband and business performance outcomes, as well as other local factors, which could limit the pool of potential businesses to be included in the control group.

This approach would only be feasible for the economic outcomes, and would not support any assessment of environmental or social outcomes. This is because the social and environmental impacts will largely be drawn from primary research, rather than utilising secondary data sources – therefore there will not be an opportunity to utilise synthetic control groups.

Analytical approaches

The potential quantitative approaches which could be used to measure the impact of the SRN programme are discussed below.

Before and after comparison (Level 2)

A before and after comparison methodology involves analysing the outcomes of interest over time for the treatment group. It does not involve the selection of a counterfactual group. This includes time periods both prior to and after the introduction of programme. It can include simply observing the changes over time or including control variables in a regression analysis (interrupted time series). However, a before and after methodology can struggle to robustly identify the impact of a programme it there are important external factors also likely to influence the outcomes of interest.

In the context of the SRN programme, a basic before and after methodology should be possible for all the outcomes of the programme, and it should be feasible to use for the individual assessments for each individual programme. Given the difficulties in forming a counterfactual case for the coverage outcomes for the PNS and EAS, it would be the preferred approach to assessing connectivity outcomes for these projects. However, this approach will need to control fully for external factors which could affect the connectivity outcomes, such as MNO behaviour and external factors (such as supply chain capacity) which could influence the outcome.

This approach would also be used for the analysis of household outcomes from the survey of households. The analysis would compare the outcome measures prior to infrastructure providing enhanced coverage and the outcomes measures post infrastructure build.

However, this approach is more challenging to implement for the downstream economic outcomes, due to the number of factors which could influence the outcome measures. Therefore, for these outcomes it is proposed that a matching and difference in difference approach is used to support the pipeline counterfactual design.

Matching approaches (level 3)

A matching approach involves refining a counterfactual group (made up of areas which are yet to benefit from the programme) to ensure it has comparable pre-intervention characteristics to the areas that have benefitted from the SRN Programme. This can be done by estimating the probability that each postcode / LSOA in the treatment and comparator group were proximate to one another, based on their initial characteristics. This probability is then used to match postcodes in the treatment group to similar postcodes in the comparator group (an approach known as property score matching). Areas that cannot be matched are excluded from the analysis. The outcomes achieved by the refined treatment and comparator groups are then compared to estimate the impact of the intervention.

The benefits of a matching approach are that it helps to reduce any selection bias between the treatment and comparator group. If all factors which could influence the selection of actors for the programme can be observed and measured, then matching methods have the potential to provide estimates of impact that are unbiased. The table below presents an initial assessment of the characteristics the research team would look to include in a matching approach, and the characteristics which it would be beneficial to include but may be difficult to access data for.

Matching approaches only provide robust results to the degree that it is possible to observe all factors that determine both the likelihood a particular postcode benefits from funding through the programme and the outcomes of interest. Clearly, this cannot be assured in this case as there are likely to be a number of important aspects that are unobservable (such as expectations of future local economic growth). While matching is likely to be helpful in refining samples for analysis, the application of further techniques will be needed to increase the robustness of results to unobserved confounders as described in the following section.

Difference in Difference / Fixed effects (level 3)

The availability of longitudinal data from the administrative and national survey data sources allows for the creation of a longitudinal panel covering areas that the SRN programme has delivered to and those it expects to deliver to in the future. Longitudinal panel techniques (such as fixed effects models or difference-in-differences) extend a standard regression model so as to control for unobserved variables. Where particular characteristics remain invariant across time or invariant across observation units (or groups of observation units) the effect of the characteristic on the outcome can be estimated. This approach can be used with a matched sample (as described above), where data for the comparator case is included only for the matched sample.

Using the annual data that tends to be available from administrative datasets, it will be feasible allow for a broader range of fixed effects such as unobserved time, sector or location specific shocks. However, this methodology would not be able to account for changes in the unobserved characteristics of individual firms or households included in the analysis. This has the potential to reduce the robustness of the findings, as changes in the outcomes could be due to the unobserved characteristics (such as outsourcing or changes in local consumer preferences) rather than the SRN Programme.

It is feasible to use difference in differences and fixed effects approaches in the evaluation of the SRN programme for economic outcomes which draw on secondary data sources. These approaches can also be combined with the matching approaches described above, to improve the comparability of the treatment and comparator groups.

Theory-based approaches

In addition to the quantitative analytical approaches described above, the impact evaluation should also utilise theory-based approaches, to support an assessment of how and why outcomes have been achieved. One approach which would be appropriate to use for the evaluation of the SRN is a contributions analysis approach.

A contributions analysis approach is an analytical framework which aims to identify how the expected (or unexpected) causal pathways have contributed to the achievement of an outcome or impact in a rigorous manner. This analysis is heavily linked to the theory of change for an intervention or set of interventions.

The research team will develop a contributions, or performance story, for outcomes expected to be achieved in local areas with enhanced mobile broadband coverage. This contribution story will be linked to the theory of change. The research team will then collect qualitative and quantitative data to test the performance story. During the data collection process, the research team will reflect on the performance story and make amendments to this based on the evidence collected, and then collect further evidence to test the revised performance story. This analytical approach seeks to identify the extent to which the SRN has contributed to the achievement of the intended outcomes.

Impact evaluation approach

The discussion above sets out the potential evaluation options for an impact evaluation of the SRN programme. There are some significant challenges to undertaking an evaluation of the SRN, which are discussed above, but the recommended approach for the impact analysis is:

To assess the impact of the programme on mobile connectivity, different approaches should be used for the TNS and the PNS and EAS
For TNS, it is proposed that the areas the project provides coverage to is compared to a counterfactual case of Total Not Spots that do not receive coverage through infrastructure funded by the TNS. It is anticipated that this will lead to an assessment that the coverage provided has a very high additionality rate, but this will test that assumption. The analytical approach would utilise matching and difference in difference techniques to estimate the impact the project has had on mobile broadband coverage, utilising data provided by Ofcom.
For EAS and PNS, a before and after analysis or interrupted time series analysis is proposed as the most appropriate approach to estimate the impact of the projects on mobile broadband availability. This would be done by examining time series data of partial and total not spots from Ofcom’s Connected Nations dataset, and exploring if the provision of PNS and EAS infrastructure has led to a step change in coverage. This analysis can be undertaken at the parliamentary ward level if more granular level data is not available, or at a pixel level if data is available, and utilise information about the locations and dates of infrastructure builds through the projects. This approach is recommended as no approach to identifying a counterfactual case could be formed, as all upgrades during this period can be attributed to the PNS project.
To assess the downstream economic impacts, a pipeline approach to developing a counterfactual case is proposed, which will utilise matching and difference in difference techniques to estimate the impact the programme. This approach can also be utilised to explore the impact on house prices, subjective wellbeing (from secondary data sources) and if sufficient traffic and energy consumption data is available some environmental impacts. This analysis will explore the impact of the SRN as a whole, and should be feasible to implement for both the PNS and EAS programmes individually (subject to sufficient numbers of individuals and businesses being located in each programmes footprint), but is not feasible to utilise for the TNS, therefore the assessment of the impact of TNS will solely utilise a contributions analysis approach.
To assess the social impacts of the programme, and some environmental impacts, a before and after approach will be utilised, exploring the change in outcome metrics between a time when there was no enhanced mobile broadband connectivity and when enhanced mobile broadband connectivity has been introduced.

For all outcomes, including those not mentioned above (such as MNO outcomes), a contributions analysis approach is recommended to explore how and why outcomes have been achieved and how the SRN programme contributed towards this.

6.4 Economic evaluation

The economic evaluation of the programme, providing an assessment of its lifetime costs and benefits and its value for money (the ratio of public sector costs to the benefits involved), is linked to the type of impact evaluation approach taken, and will be the final analytical task. If during the course of the evaluation it is feasible to quantitatively assess the impact of the programme on social, economic and environmental outcomes, a cost-benefit analysis will be feasible. If not, the analysis will focus on a mixed methods Value for Money (VfM) approach.

Cost-benefit analysis

The cost-benefit analysis will consider costs and benefits to date and over a period extending to 2041 (a twenty-year time horizon from the launch of the programme in 2021). This analysis will also be undertaken for the PNS and EAS Programmes individually, following the same approach as for the SRN as a whole. However, it will not be possible to undertake this analysis for the TNS on its own, due to the small number of beneficiaries living and working in the areas the TNS operates in. Therefore, the economic evaluation of the TNS programme will utilise a mixed methods approach to assessing VfM (see below).

The analysis will include:

Costs: Estimates of the present value of expected costs over the lifetime of SRN programme will be taken from monitoring information from BDUK and the MNOs. If the MNOs cannot provide private cost information, modelling assumptions can be used to estimate the total cost of the programme. This information will provide underlying estimates of the public and private costs associated with the programme.

Benefits: Estimates of the benefits of the programme will be derived from the results of the econometric analysis and are expected to cover:

Productivity gains derived from improved efficiency of businesses: These will be developed from estimates of the effects of mobile broadband on the turnover per worker of firms that do not change their location (to avoid the risk of overstating the net impact of the programme by including effects driven by the relocation of firms, which would involve offsetting effects for other areas). This will also need to allow for any decay in these impacts over time.

Labour supply impacts driven by (a) reductions in long-term unemployment and (b) any increases in economic activity rates. These impacts will be monetised in line with approaches adopted in the appraisal of other forms of infrastructure investment projects:

Social benefits: These will be estimated either by using estimates of changes in subjective wellbeing or will use (as was used in the Superfast Broadband Programme evaluations) a hedonic pricing methodology (assuming the social value of the programme was capitalised into house prices).

Environmental impacts: The CBA will also be augmented with estimates of the environmental costs and benefits of the programme if these can be estimated in a robust manner.

Additionality: All estimates will need to be adjusted in light of the estimated additionality of mobile broadband connectivity brought forward through the Programme, which is estimated to be high.

Mixed methods value for money

Either alongside or in the absence of a CBA, it is recommended a value for money assessment, guided by the National Audit Office’s (NAO) 4Es framework,^{[footnote 16]} is undertaken. This approach focuses on the economy, efficiency, effectiveness and equity of the programmes. This is summarised in the figure below.

NAO 4E framework

Ipsos’ adaptation of NAO 4E diagram^{[footnote 17]}

The VfM assessment would be mixed methods in nature, utilising the outcome metrics described in Section 4 and data collected in qualitative interviews. In addition to this data, the evaluation team will undertake a review of existing policy and evaluation evidence of mobile broadband provision, so that the evidence collected for this evaluation can be compared to other types of mobile broadband provision (international evidence), to inform the VfM assessment.

An example of how the evaluation will assess VfM is presented in the table below. This highlights the evaluation question, the criteria for assessment and key evidence to be included.

Table 5.2 - VfM assessment

Evaluation question	Criteria for assessment	Evidence for assessment
To what extent did the funding option used provide value for money to the government?	Appropriateness of funding models used Number of funding options considered Public funding provided	Documents evidencing decision process Evidence of costs and benefits Feasibility of different approaches Qualitative interviews with delivery stakeholders and suppliers
How far has the funding provided achieved the stated outputs of the programme?	Number of new masts constructed / upgraded Area covered by enhanced coverage Number of premises covered Public funding provided Total funding needed to provide enhanced coverage	Number of outputs achieved Funding required to deliver outputs Qualitative interviews with delivery stakeholders and suppliers, local case studies, survey of residents
How far has the SRN achieved the outcomes set out in the Theory of Change?	Number of additional outcomes achieved Funding used to deliver outcomes Other factors that have driven outcomes	Number of outcomes achieved Funding required to deliver outcomes Qualitative interviews with delivery stakeholders and suppliers, local case studies, survey of residents

Assessment of offering value for money:

High – achieving more than the desired outputs / outcomes given the level of investment.

Adequate – achieving the desired outputs / outcomes given the level of investment.

Low – Not achieving the desired outputs / outcomes given the level of investment.

Question 7

7. Implementation

Accepted Answer

This section sets out how the SRN programme evaluation should be implemented. It draws on the information provided in the preceding sections of the report. The section includes the focus of the evaluation, the timelines for each evaluation stage and the key risks for the evaluation.

7.1 Delivery timeline

It is recommended that the evaluation takes place in three main stages. These would be:

A baselining exercise: This should take place in 2024 and 2025, using data from secondary sources for 2021. This will provide a description of the economic, environmental and social situation in the areas where the SRN is being delivered prior to the delivery of any subsidised build. This can then be used to compare changes in outcomes to in subsequent evaluation activity.

An interim evaluation for each project: This activity should take place while the SRN programme is still delivering coverage, or just after completion. There will be a separate interim report for each of the PNS, EAS and TNS projects, focussing on the individual project. This means for PNS the evaluation activity will need to take place in 2024, whereas for TNS and EAS this activity can be expected to take place in 2025 and 2026. Each interim evaluation will include:

a process evaluation for the project, exploring how the project has been implemented and key lessons learned from delivery
an assessment of the impact the project has had on mobile broadband coverage, including additional landmass coverage and the number of providers with coverage in the project areas
an early assessment of how the project has impacted on the mobile broadband market. Findings from the local case studies exploring the effects that enhanced local coverage has had on the area

A long-term impact evaluation: This evaluation should take place towards the end of the programme, allowing time for contracts to be completed, take-up to be realised and impacts to be generated. This would explore the impact of the programme on coverage, economic, social and environmental impacts, and market analysis for the Subsidy control evaluation. It will also include recommendations for the future evaluation of the SRN. This is expected to be completed in 2028.

The proposed timetable for evaluation activity is presented in the table below.

Proposed timeline for the evaluation of the SRN Programme – Milestones 2,3 and 4

Baseline indicators	Nov – Dec 2024
Baseline wellbeing survey	Jul – Dec 2024
Baseline wellbeing survey report	Dec 2024

Proposed timeline for the evaluation of the SRN Programme – Milestones 5,6,7 and 8

Qualitative stakeholder interviews	Jan – Mar 2025
Case study interviews	Jan – Jul 2026
Economic and environmental analysis	Jan – Mar 2025 Oct – Dec 2026 Oct – Dec 2027
Market and financial analysis	Oct – Dec 2026 Oct – Dec 2027
Follow-up wellbeing survey	Jul – Dec 2025
Cost benefit analysis and value for money analysis	Dec 2027 – Jan 2028
PNS process and early impact report	Jun – Aug 2025
EAS process and early impact report	Dec 2026 – Feb 2027
TNS process and early impact report	Jul – Sep 2027

7.2 Risk management

An updated risk register for the evaluation of the SRN has been developed for internal use. This highlights the key risks to the evaluation and actions which can be taken to mitigate these risk

Question 8

8. Our standards and accreditations

Accepted Answer

Ipsos’ standards and accreditations provide our clients with the peace of mind that they can always depend on us to deliver reliable, sustainable findings. Our focus on quality and continuous improvement means we have embedded a ‘right first time’ approach throughout our organisation.

For more information:

3 Thomas More Square London E1W 1YW

t: +44 (0)20 3059 5000 www.ipsos.com/en-uk http://twitter.com/IpsosUK

About Ipsos Public Affairs

Ipsos Public Affairs works closely with national governments, local public services and the not-for-profit sector. Its c.200 research staff focus on public service and policy issues. Each has expertise in a particular part of the public sector, ensuring we have a detailed understanding of specific sectors and policy challenges. Combined with our methods and communications expertise, this helps ensure that our research makes a difference for decision makers and communities.

Cookies on GOV.UK