UK Space Agency Evaluation Strategy

Question 1

1. Foreword

Accepted Answer

Within DSIT, the UK Space Agency is responsible for unlocking the powerful and exciting benefits of space for UK citizens. Our priority outcomes are driving economic growth and national security, but we also support bringing further benefits to UK citizens from UK space services and data, an innovative economy and international relationships. Evaluation must be at the heart of our work if we are to succeed. We are committed to undertaking relevant, high-quality evaluation which helps us learn how to build a better future and deliver the best outcomes for UK citizens.

In this Evaluation Strategy, we outline our vision for evaluating our impact and the way we support the UK space sector. We recognise the inherent methodological challenges in the area and are eager to find solutions: for example, the long lags often experienced between our interventions and the full realisation of their benefits.

In the interests of transparency and the spirit of learning we publish details of our evaluations through the ETF Registry - a single location for all HMG evaluation, with the aim of making evaluations easier to find, more accessible and more impactful.

Finally, we invite anyone with a mutual interest in the UK Space Sector to work with us – we want to share our learning and learn from others, particularly those in adjacent sectors who are keen to understand how space can benefit them.

Rebecca Evernden
Director, UK Space Agency

Question 2

2. Introduction

Accepted Answer

2.1 About us

The UK Space Agency is part of the Department for Science, Innovation and Technology (DSIT). We work closely with other departments and research organisations which have equities in space. The UK Space Agency coheres space policy, strategy and delivery across the whole of government, driving innovation and delivering world-class space science programmes in partnership with the UK space sector and international partners. The UK space sector generates an annual income of £18.6 billion and employs 55,000 people across the country. More generally, space and satellite services are estimated to support wider industrial activities worth £454 billion to the economy, or 18% of GDP.

Our intended audience for this Strategy is anyone interested in UK space sector, and how we measure the impact of government intervention in it. This includes, but is not limited to:

those working in the UK space sector, and related areas (eg, UK Research and Innovation)
policy makers
international space agencies/organisations
evaluators, who might bid for future work

2.2 Why evaluation matters

We believe evaluation is critical to understanding our impact, deciding how to allocate our resources and deliver value for money for the taxpayer. The evidence base for understanding which space interventions work, why, and how is continually developing. Space is a rapidly growing and evolving sector with new technologies, new capabilities and new markets, such as Active Debris Removal (ADR) and In-space Servicing, Assembly and Manufacturing (ISAM). Research, development and innovation (RDI), which is central to the space sector and a key part of the UK Space Agency’s investment portfolio, is challenging to evaluate. There are necessarily long timeframes (sometimes decades) before the benefits and impacts of this work can be realised.

In recent years we have substantially increased our commitment to evaluation but recognise that there is more that we can and should do to build the evidence base. This Evaluation Strategy, now updated to reflect the UK Space Agency’s merger with DSIT, outlines how we will achieve that aim.

2.3 Collaboration

We recognise that others working across the UK government and the space sector, in the UK and internationally, face similar challenges. We outline how we would like to work together with others later in this report (section 3.6 to 4.2). We welcome challenge and want to share our learning.

2.4 Our commitment

We intend to publish full updates of this strategy in line with spending review cycles. In the interim, we will continue to publish annual progress updates on high-level findings of our portfolio of evaluation and update the Evaluation Task Force (ETF) Evaluation Registry with details of our published evaluations.

The rest of this document outlines our vision and objectives for evaluation, before detailing how we will achieve our ambitions.

Question 3

3. Vision and objectives

Accepted Answer

3.1 Our vision for evaluation

Our vision is for evaluation findings to meaningfully inform programme design and spending decisions, primarily for space but also for others working in related areas, like RDI. We want our portfolio of evaluation to help inform strategic thinking across DSIT, including for fiscal events.

To achieve this, it is vital that we have access to a strong body of evidence about what works so that we maximise the impact and value for money of our interventions. Evaluation of our programmes forms a large part of this body of evidence.

In line with the Magenta Book and the Green Book, the UK Space Agency seeks to produce as robust a body of evidence as possible through:

embedding evaluation planning in new interventions from their inception to maximise the potential for meaningful evaluation
disseminating and using findings in future programme development (both formative and summative) so that we can make the best decisions about where and how to deliver, closing the feedback loop of the policy-making cycle

We have identified four broad types of challenge to doing space sector evaluation, as outlined in the box below.

3.2 Time lag between programme delivery and impacts being realised

Mid-Technology Readiness Level

The UK Space Agency funds a lot of mid-level Technology Readiness Level (TRL) projects (for example 4-6). As such our evaluation activity typically fails to explore what happens when products are brought to market, when impacts are most pronounced. While this varies by the type of product or technology, impacts are sometimes not realised 10-20 years after initial funding. For example, instruments for science missions can take decades to return data.

RDI

In a similar vein, the UK Space Agency invests in RDI, which is challenging to evaluate: RDI can lead to new technologies, innovations, solutions and spillovers including outside the space sector, but this often occurs over a long period of time and there is uncertainty about when benefits will be realised.

Catalysing investment

Measuring the impact of how the UK Space Agency programmes catalyse investment is also difficult due to long time lags between making investments and observing the impact of new technologies. Attribution and additionality are challenges here too: we seek to answer questions such as: What impacts can we attribute to UK Space Agency activity? or: What is the added value of UK Space Agency interventions?

UK government and ESA (European Space Agency) spending cycles

Government spending cycles, determined by spending reviews, are much shorter than the time it takes for impacts to be achieved. It is difficult to make long-term commitments to funding evaluation beyond government spending cycles, which poses a challenge to understanding the full extent of impact.

The UK Space Agency spends a high proportion of its overall budget through its ESA contributions. However, ESA spending cycles do not generally align with UK government spending cycles. This makes it difficult to ensure that the findings are available to inform decisions at the right time.

3.3 Identifying a counterfactual

UK Space Agency programmes often involve small sample sizes (for example, small numbers of recipients for grant programmes). Small sample sizes make it difficult to detect impact using a counterfactual as we lack statistical power.

Small sample sizes are partly due to the diversity of the sector, and the fact that it is made up of wide-ranging types of firms, organisations and individuals, which also makes it difficult to identify a counterfactual due to small pools for comparator groups.

Our interventions are often applied to complex and emergent systems, which make identification of a meaningful counterfactual challenging. For example, a programme designed to unlock the benefits of space for business – this is an emerging area, with multiple actors, creating a complex environment.

3.4 Data challenges

There is a dearth of relevant administrative/other data in the space sector that we can use for counterfactual analysis, for example:

it is usually not possible to form counterfactuals from space sector survey data (e.g. through synthetic control groups or counterfactuals) because they are not representative of the sector, nor granular enough for this purpose
it is difficult to recognise space companies in Office for National Statistics (ONS) data; some space companies may have diversified into space, but do not present as such in ONS data if the parent company is categorised differently (e.g. as aerospace); there is not a distinct Standard Industrial Classification (SIC) code for space
some organisations might fit our definition of the space sector but do not perceive themselves as space organisations and so it is difficult to collect data on them

3.5 Difficulty quantifying outcomes and complexity

Nature of outcomes

While some UK Space Agency programmes seek to achieve outcomes which are easier to measure (e.g. Launch), others target outcomes which are less tangible and therefore more difficult to measure.

For example, the International Bilateral Fund (IBF) programme seeks to increase the UK’s credibility in becoming a more meaningful actor on the world stage but this is hard to quantify. This makes it difficult to monetise benefits, presenting challenges to value for money evaluations and cost-benefit analysis.

Complexity

It is sometimes difficult to identify the impact and additionality of UK Space Agency programmes and projects. The UK Space Agency can be one of many investors and supporters and it can be difficult to trace our investment into the resulting value of a product, service or other activity.

The UK Space Agency might provide some early funding or an intervention very early on in the bigger picture. For example, an initiative might be designed to encourage a young person to take up a STEM subject but it would be challenging for us to follow up with this young person decades later to find out the impact on their career and, even if we could, to disentangle what role our intervention played relative to other factors.

3.6 Objectives

We will deliver our vision, and address the challenges, through five objectives:

Long-term evaluation: Commit to evaluation beyond government spending cycles, through robust evaluation governance, to ensure meaningful learning.
Standard of evidence: Improve the standard of evidence through embedding evaluation in programme design, increasing the likelihood of more experimental or quasi-experimental designs, or meaningful theory-based evaluation where this is not possible.^{[footnote 1]}
Partnership working: Collaborate with colleagues across government and beyond to ensure that we are learning and share our knowledge. We want to facilitate knowledge sharing between experts and evaluators.
Commercial approach: Clearly communicate our requirements with the market and understand what makes for attractive opportunities so that we attract the best bids when commissioning evaluations.
Dissemination and learning: Publish our evaluation findings, communicate them effectively, learn from them and demonstrate accountability for how we spend public money. In turn, increase the credibility of the UK Space Agency and better satisfy the requirements of the National Audit Office (NAO) and HM Treasury (HMT), amongst others we are accountable to.

The next section outlines how we will overcome the challenges, deliver our objectives and achieve our vision.

Question 4

4. How we will deliver

Accepted Answer

4.1 Integration with wider DSIT evaluation strategy

Subsequent updates to this Strategy will be developed in close collaboration with other colleagues in DSIT. In particular, we will ensure that we continue to align closely with the DSIT Evaluation Strategy when it is refreshed in future.

4.2 Culture of celebrating evaluation

Creating a culture where evaluation is prioritised and celebrated will be central to achieving our vision. To help embed and promote the right culture we intend to:

deliver space sector-specific evaluation training for staff
run evaluation surgeries for programme teams, run by members of the evidence team.
Establish an evaluation community, within and beyond the UK Space Agency, of those interested in understanding and advancing evaluation of the space sector
leverage our new position within DSIT and will work closely with its existing analytical function and its wider community of evaluation experts

4.3 Governance and processes

We need to get the right governance and processes in place to implement a robust portfolio of evaluation. We will continue to embed these processes within the UK Space Agency, aligning closely with DSIT to maximise their effectiveness.

Evaluation governance – panel of experts

We will continue to operate our panel of evaluation experts, set up when the UK Space Agency was an arm’s-length body (ALB), which will meet up to twice a year. Members include evaluation experts from across and outside government. This panel will review our portfolio of evaluations, providing scrutiny of our performance against our objectives, highlighting areas of improvement and make recommendations. The UK Space Agency Analysis Team will provide the secretariat function and report into senior UK Space Agency colleagues.

Business cases and project management

Business cases in government provide the evidence base in support of a spending proposal. They facilitate transparency, approval, post-evaluation, the accountability of public funds and optimisation of public expenditure. We will be designing evaluations at an overarching level, rather than of individual programmes or projects.

We will continue to use our comprehensive project management and evaluation tools and processes, as well as DSIT central analysis oversight processes to promote a consistent and efficient approach across evaluations and throughout the lifecycle of evaluation, from programme inception and evaluation design, through to evaluation learning.

Working in this way will help us to streamline our resources and help ensure we are able to provide comprehensive coverage.

Clarity of roles and engagement points

We will ensure that roles and responsibilities of those working on evaluation at the UK Space Agency are clearly delineated, to make sure we are getting the most from our people. This includes, but is not limited to, the UK Space Agency’s Analysis Team, programme and policy teams, and DSIT central analysis and commercial teams.

Quality assurance

Continuing to embed robust quality assurance processes for the full evaluation cycle, from design and inception through to analysis and reporting will help us to improve the quality and consistency of our evaluations. Amongst other measures, this includes clear sign-off processes and the use of peer reviewers.

Procurement

To attract the best bids, we will maximise the overall contract value of tenders through commissioning contracts at an overarching level (as discussed above). We will emphasise our requirement for evaluation expertise above all else.

Publication

We publish annual progress updates on high-level findings of our portfolio of evaluation and we plan to continue these updates. We will publish evaluations on GOV.UK and upload details of our evaluation portfolio on the ETF Evaluation Registry (a central registry of all government evaluation).

4.4 Partnership working

We recognise that there are others across government who are working in related fields and experience similar challenges. We want to both learn from others and contribute to this community.

We will work within DSIT’s wider evaluation community to tackle methodological challenges in evaluations, disseminate findings to enhance evidence-based programme design and decision making.

A list of all partners that we have agreed to work with includes:

Department for Business and Trade (DBT)
Department for Energy Security and Net Zero (DESNZ)
Department for Transport (DfT)
Innovate UK
international space agencies
Met Office
UK Research and Innovation (UKRI)

This list is not exhaustive – we intend to work with many more organisations. If you think we should be speaking to you about space sector evaluation or have feedback about our evaluations, please contact UKSA.evaluation@dsit.gov.uk.

4.5 Proportionality

We intend to conduct higher-level, more overarching evaluations. Within areas we will be proportionate in how we allocate resources. We will be guided by budget, strategic priority/profile (as summarised below), with the most resources being invested in Level 3 programmes. Pilots, which could inform the direction and shape of future programmes, (whether they continue, on what scale) will also be a factor in prioritisation.

Table showing risk and uncertainty against budget and profile

Risk and uncertainty	Budget and profile (low)	Budget and profile (medium)	Budget and profile (high)
Low	Level 1	Level 2	Level 2
Medium	Level 2	Level 2	Level 3
High	Level 2	Level 3	Level 3

Table notes:

Description of levels:

Level 1: Light-touch, could be internal evaluation, emphasis on monitoring
Level 2: Consider commissioning externally, with appropriate budget allocation especially if helps overall evaluation of the relevant UK Space Agency programme area
Level 3: Comprehensive, externally commissioned evaluation with appropriate budget

Indicative budget thresholds:

Low: <£5 million
Medium: £5 million to £10 million
High: £10 million and over

Budget and profile level descriptions:

High: Large programme with significant budget, and/or high profile with media and public interest, and potentially high impact
Medium: Medium-sized programme with moderate budget, and/or some media and public interest, expected to have a sizeable impact
Low: Small budget and/or limited public or media interest, with relatively low impact

Risk and uncertainty descriptions:

High: Complex programme design, and/or significant risk and uncertainty around programme outcomes
Medium: Programme not especially complex or risky, but some uncertainty around outcomes
Low: Straightforward, low-risk programme with low uncertainty around the outcomes

4.6 How we will address the specific space evaluation challenges we have identified

Time lag between programme delivery and impacts being realised

We will continue to work with UK Space Agency colleagues leading work on benefits management to develop a robust approach to long-term monitoring of programmes, beyond the end of funding. Monitoring is key for projects where impacts take a long to be realised. This will enable both long-term evaluation but also give reassurance to funders and other stakeholders that the project is on track to deliver envisaged benefits.

In addition, we will use proxy indicators where this is possible and appropriate. Please see Annex A for more on the relationship between benefits management, realisation and evaluation.

Identifying a counterfactual

We will invest in data where it is lacking (administrative data, and survey/economic data). We will also invest in scoping and feasibility studies to explore the variety of ways to identify a counterfactual.

Through our project management systems and processes we will influence programme design earlier, to maximise the opportunity to identify counterfactuals (for example, through delaying the start of programmes for cohorts to create a comparison group).

We are committed to upskilling those with responsibility for designing programmes in different types of evaluation design to maximise the evaluability of programmes.

Data challenges

Working with colleagues across government, we will develop a long-term and strategic approach to improve the quality of data through streamlining our surveys and making them more representative of the sector, and making them more targeted, with the granularity required to support evaluation of our programmes.

Difficulty to quantifying outcomes and complexity

Where counterfactual analysis is simply not possible or appropriate we will focus on doing robust theory-based evaluations, with qualitative approaches to impact evaluation. Through qualitative impact evaluations we can explore the contribution that our programmes have made to intended outcomes, especially in the short-to-medium term. Programmatically we will also start to more routinely consider the appropriateness of longer-term re-contact to explore whether benefits and impacts are achieved much later.

We will invest in process, impact and value for money evaluations, recognising that the strongest programme evaluations combine all three. We will conduct research on how to monetise benefits, for meaningful cost-benefit analysis and value for money evaluations.

Question 5

5. Appendices

Accepted Answer

5.1 Annex A

Interaction with monitoring, benefits management and realisation

We have separated monitoring from evaluation for this strategy. We recognise that good evaluation is underpinned by good monitoring. Monitoring will be informed by our Benefits Framework; a catalogue of benefits that we will use across UK Space Agency programmes to promote consistency and facilitate comparison between programmes. Some additional monitoring for the purposes of evaluation may be required which we will review on a project-by-project basis.

We consider benefits management and realisation to be complementary to, but distinct from, evaluation. Benefits management focusses on direct, measurable benefits. Its purpose is to ensure benefits are being realised, and to change approach if not. It is rooted in project management.

Evaluation, on the other hand, is broader; it seeks to understand ‘what works, for whom, in what circumstances and why?’. Some benefits management/realisation work will be helpful for evaluation and evaluators should be able to make use of it.

Benefits management/realisation work will not provide everything an evaluator needs; specifically, it will not provide insights evaluators need on outcomes and impacts that are less tangible and harder to define and measure. Here there might be additional monitoring requirements.

Given the potential for overlap the relevant teams will work together, including with wider DSIT monitoring and benefits stakeholders, to avoid duplication.

We will use our project management processes, business case development and quality assurance processes (discussed in section 3.6 to 4.2) to help us deliver this. ↩

Cookies on GOV.UK