Research and analysis

DCMS R&D Science & Analysis Programme Phase 1 evaluation, final report

Published 12 October 2026

Key highlights

Phase 1 of the R&D Science and Analysis Programme launched in late-2022 to generate long-term strategic evidence. The programme aimed to complement responsive research, and lead to a ‘step change’ in how DCMS approaches its evidence base.

Phase 1 funded 22 research projects over a three-year delivery period, spanning DCMS and DSIT’s Digital Technologies and Telecoms directorate (following Machinery of Government changes in February 2023). Expenditure was £7.7m, 92% of allocated budget, with efficiencies realised through the use of competitive bidding processes.

Delivered through the DCMS CSA Office, the programme’s governance arrangements were adequate for ensuring accountability. However, the Programme’s Board could have been leveraged more fully in decision making, strategic direction, and raising internal engagement.

The programme was well managed, including in navigating the Machinery of Government changes. However, management activity was significantly restricted by resource availability, impacting on awareness raising, dissemination and monitoring. This represented a high level of risk to delivery which should be addressed in Phase 2.

A higher number of smaller projects were funded than planned. This expanded the breadth of topics covered, but also reflected a reluctance of officials to identify and lead larger and higher-risk projects. In practice, some of the Phase 1 projects may have happened anyway. Greater clarity on the strategic focus and the intended risk profile of the project portfolio should be considered in Phase 2. In parallel, a more consistent approach to project identification and selection is needed; this varied in Phase 1, which contributed to uncertainty over the programme’s purpose and scope within DCMS and DSIT.

The programme delivered a substantial volume of research outputs, including written deliverables and toolkits, and supported dissemination activities. However, dissemination both internally and externally has been more modest in scope and inconsistent in delivery than it should have been, with adverse implications for outcomes and the exploitation of the evidence.

The programme has led to intended benefits in the departments’ capability to undertake R&D activity effectively and efficiently including via upskilling of analysts, improved capability/confidence in advanced and novel research techniques, and a better alignment of external research and government needs. However, these outcomes were largely limited to those directly involved in research projects, and have not been diffused more widely.

The programme itself represented a more proactive and strategic approach to evidence, and has generated robust evidence to be drawn on as policy priorities demand. There are examples of the evidence being used to inform policy discussions. However, the impact at this stage on decision-making has been influenced by delays in the completion and publication of outputs, and the scope and scale of dissemination. The programme’s value at this point is primarily in enabling DCMS and DSIT to ‘hold’ rather than ‘use’ the evidence. This aligns with a focus on strategic (not responsive) research, given the associated timescales to impact.

Overall, Phase 1 has provided a good grounding on the path to achieving a ‘step change’ in how DCMS approaches its evidence base. Key recommendations for Phase 2 include: clarifying the underlying theory of change and strategic objectives; ensuring a greater focus on processes and activities for raising awareness and disseminating learning; and improving monitoring and data collection processes.

Executive summary

The Department for Culture, Media and Sport (DCMS) R&D Science and Analysis Programme (the programme) was launched in late-2022 with the aim of delivering a ‘step change’ in how DCMS approaches its evidence base. The programme was intended to complement existing research arrangements, providing an opportunity to fund research on longer term, cross-cutting and more speculative research questions.

Machinery of Government changes in February 2023 resulted in a change of scope for DCMS, and the creation of the Department for Science, Innovation & Technology (DSIT). As a consequence of this, programme funding was split roughly equally between DCMS and DSIT. The programme continued to be managed by the Chief Scientific Adviser’s Office in DCMS.

Phase 1 of the programme funded 22 research projects over its delivery period between the winter of 2022 and spring of 2025. Projects were funded as fixed price pieces of work, via two different commercial arrangements: tendering (13 projects) and Memorandums of Understanding (9 projects). Total expenditure in Phase 1 was £7.66m, of which the majority related to the research projects, with less than 5% spent on salaries and overheads.

SQW was commissioned to undertake an evaluation of Phase 1 of the programme, reflecting upon the findings of Phase 1 to inform delivery of Phase 2 (running from 2025 to 2030). This comprised two separate but linked strands of evaluation:

• process evaluation assessing the effectiveness of delivery and implementation of the programme, with a focus on what has worked well or less well

• impact evaluation assessing the extent to which the programme has delivered benefits.

The evaluation adopted a theory-based approach, applying the principles of contribution analysis. The analysis was informed by mixed-methods evidence from primary and secondary research, which was triangulated, synthesised and assessed against the programme theory of change developed by DCMS. The evaluation approach followed Magenta Book guidance. The key findings from the evaluation are presented below.

The programme’s governance arrangements in Phase 1 were adequate for ensuring accountability, and for realising the programme’s objectives. However, for Phase 2 there are opportunities to better articulate the programme’s decision-making processes and the remit of the Programme Governance Board, and to more fully leverage the Board to provide strategic direction. There may also be an opportunity to leverage more fully the DCMS College of Experts to support programme delivery and impact.

The programme was well managed. However, the scope of programme management activity was significantly restricted by resource availability. The model adopted in Phase 1 was highly reliant on a single Portfolio Programme Manager, representing a high level of risk to delivery.

The programme was reasonably well integrated within DCMS, although awareness and understanding of it varied across the Department. Dissemination of the activity was limited as a result of the resource availability, which has had implications for the potential impacts of the programme, both internally within the Departments and externally with wider stakeholders. Within DSIT, the programme appears to have been valued, but it has not been fully integrated into the wider programme of research (noting that joint delivery was driven by circumstance, and that DSIT will not be involved in Phase 2).

The most effective approaches to identifying areas of research focus combined both ‘top-down’ and ‘bottom-up’ engagement, whilst ensuring alignment with the programme’s rationale. In practice, the approach to prioritising research projects varied each year. This reflected learning from ongoing programme delivery, but also meant that the processes were not consistent across the years which may have led to a lack of clarity on the rationale and purpose of the programme across the Departments.

The programme funded a higher number of smaller projects than originally intended. Positively, this expanded the breadth of research areas covered. However, there were some concerns about reluctance to identify and lead on larger and higher-risk projects, which is related to an observation that a small number of projects may have been delivered anyway.

Experiences with project delivery varied across the portfolio but this generally worked well, with positive feedback on the role of both the external research teams (leading delivery) and the government teams (providing inputs). Lessons on ‘what works’ are context-specific, but common themes of success included: clarity in responsibilities and ownership; alignment of expectations between government and delivery organisations; consistency of inputs from government teams; and regular communications. In projects where these were not evident, there were typically some adverse effects, including on the timeliness or quality of outputs.

The programme has enabled the production of a substantial volume of research, including written deliverables and toolkits, and supported dissemination activities. The quality and coverage of outputs generally aligned with government expectations, although there were challenges experienced in some cases, e.g. due to a lack of clarity around expectations and sub-optimal communication between the delivery organisation and government teams. There was a missed opportunity in relation to dissemination of project learning and findings, which has been more modest in scope and inconsistent in delivery than it should have been, partly due to limited resourcing.

Two main routes to impact were apparent, in terms of outcomes associated with:

• Generating evidence i.e. capability to undertake R&D activity effectively and efficiently: There is evidence of benefits relating to reinforcing processes of knowledge spillovers, upskilling of analysts, and improved capability/confidence. The programme has also led to a better alignment of external research and government needs. Outcomes related to knowledge, skills and confidence/capability were largely limited to those directly involved in projects, with limited evidence that these benefits have been diffused more widely across the participating departments.

• Holding and using evidence i.e. capability to exploit and apply the R&D: The programme represented a more proactive and strategic approach to evidence within DCMS, and has generated robust evidence which can be drawn on as policy priorities emerge. The evaluation identified limited evidence that the research at this stage has directly informed or improved decision making in strategic policy areas, although there are some examples of specific contributions to policy discussions from across the programme portfolio. For this route to impact, the programme’s value to date has therefore been primarily in relation to enabling DCMS and DSIT to ‘hold’ rather than ‘use’ the evidence. This is consistent with a focus on strategic rather than policy-specific research, and the associated timescales to impact.

Phase 1 of the programme has also generated some benefits beyond its stated objectives. These include: reputational benefits for DCMS and its status as an R&D active department; new and stronger links with the research community; and wider impacts outside DCMS through the application of research findings.

The programme has played an important role in realising the above outcomes. This is underpinned by reasonably high ‘activity additionality’ (i.e. most projects would not have happened otherwise or would have been impacted in terms of quality, scope or timing) which is linked to a high level of ‘outcome additionality’ (i.e. the benefits would not have happened). However, a small number of projects may have been funded through other mechanisms.

External factors have influenced the relevance and applicability of the research, and this will continue to influence impact going forward, notably in relation to informing policy decisions. Delays in publication of outputs have influenced impact to date, recognising in particular the time-sensitive nature of some areas of research. This is linked to management resource.

Overall, Phase 1 of the programme has provided a good grounding for progress towards achieving a ‘step change’ in how DCMS approaches its evidence base. Phase 2 should leverage and build on this progress, informed by recommendations which include actions related to:

• Ensuring an appropriate level of resource is allocated for programme management
• Reviewing the remit, role and composition of the Programme Governance Board, and considering opportunities to leverage the expertise of the College of Experts
• Reviewing and updating the programme theory of change including its aim and objectives, and developing a strategic portfolio proposition statement to inform funding decisions
• Introducing a more consistent, structured and transparent approach to project selection
• Taking steps to mitigate key risks around project delivery
• Ensuring a greater focus on processes and activities for raising awareness of the programme and disseminating the learning
• Improving processes to ensure robust monitoring and data collection.

1. Introduction

1.1 The programme

The Department for Culture, Media and Sport (DCMS) R&D Science and Analysis Programme (the programme) was launched in late-2022 with the aim of delivering a ‘step change’ in how DCMS approaches its evidence base. The programme was intended to complement existing research arrangements, providing an opportunity to fund research on longer term, cross-cutting and more speculative research questions. The programme had a distinct focus on generating long-term strategic evidence, ‘what works’ evidence, and evidence using advanced and novel research techniques.

The programme had four objectives:

• Improve decision making in strategic policy areas by delivering a robust evidence base
• Promote savings and efficiencies through internal capability building, and enhanced value from external contractors
• Facilitate knowledge transfers and spillover effects between government and subject matter experts by involving them in early stages of evidence-based development
• Initiate a department-wide culture change, through building the department’s familiarity and confidence with new methods and innovative experimentation.

Phase 1 of the programme funded 22 research projects over its delivery period between the winter of 2022 and spring of 2025. The programme launched before the Machinery of Government changes in February 2023 which resulted in a change of scope for DCMS, and the creation of the Department for Science, Innovation & Technology (DSIT). As a consequence of this, funding was split roughly equally between DCMS and DSIT. The programme continued to be managed by the Chief Scientific Adviser’s Office in DCMS.

1.2 The evaluation

In line with DCMS’s commitment to monitoring and evaluation ^{[footnote 1]}, SQW was commissioned to undertake an evaluation of Phase 1 of the programme. This evaluation was intended to reflect upon the findings of Phase 1 (2022-2025) and inform delivery of Phase 2 (running from 2025 to 2030). This comprised two separate but linked strands of evaluation: process evaluation (assessing the effectiveness of delivery and implementation of the programme, with a focus on what has worked well or less well) and impact evaluation (assessing the extent to which the programme has delivered the desired benefits). The research questions are included in Annex A.

The evaluation adopted a theory-based approach as the overarching framework, applying the principles of contribution analysis^{[footnote 2]} (see Annex A for further detail). This followed Magenta Book guidance, taking into account that the programme is complicated and operates in a complex landscape. The evaluation methods are listed in Table 1-1, and mapped against the two evaluation strands. Fieldwork was undertaken in two phases (the approach to sampling is explained in Annex A). This evidence was then triangulated, synthesised and assessed against the programme theory of change (see Figure 1-1).

Table 1-1: Evaluation methods

	Process	Impact
Interviews with governance/programme management stakeholders (x14)	x	x
Interviews with ‘high involvement’ government teams (x11)[footnote 3]	x	x
Interviews with ‘lower involvement’ government teams (x7)[footnote 4]		x
Survey of delivery organisations (x15)	x	x
Follow-up interviews with delivery organisations (x4)	x
Impact tracing case studies[footnote 5] (x5)		x
Analysis of secondary data on outputs		x

Source: SQW

1.3 Structure of the report

The remainder of this report is structured as follows: Section 2 outlines key findings for the process evaluation and Section 3 for the impact evaluation; and Section 4 presents the conclusions and recommendations. There are two supporting annexes, providing further detail on the evaluation approach (Annex A) and the projects funded by the programme (Annex B).

Figure 1-1: Theory of change for the R&D Science and Analysis Programme

(A flow chart of the programme theory of change.)

Source: DCMS

2. Process evaluation

Key Findings

• The programme’s governance arrangements in Phase 1 were adequate overall for ensuring accountability, and for realising the programme’s objectives. However, for Phase 2 there are opportunities to better articulate the programme’s decision-making processes and the remit of the Programme Governance Board, and to more fully leverage both the Governance Board and the independent College of Experts.
• The programme was well managed but the scope of programme management activity was significantly restricted by resource availability. The model adopted in Phase 1 was highly reliant on the Portfolio Programme Manager, and represented a high level of risk to programme progress and delivery.
• The programme appears to have been well integrated within DCMS, although awareness and understanding of it varied across the Department. Within DSIT, the programme appears to have been valued, but it has not been integrated into the Department’s wider programme of research activities (noting that joint delivery was driven by circumstance at the time, and that DSIT will not be involved in Phase 2).
• The approach to prioritising research projects varied each year. The most effective approaches to identifying areas of research focus combined both ‘top-down’ and ‘bottom-up’ engagement, whilst ensuring alignment with the programme’s rationale.
• The programme funded a higher number of smaller projects than originally intended. This was perceived to be primarily due to risk aversion in project selection, with some reluctance to identify and lead on larger projects which were perceived to be riskier.
• The use of two commercial arrangements, through Memorandums of Understanding and tendering, was appropriate and provided helpful flexibility, with each having merit in different contexts.
• Experiences with project delivery were generally positive. There were some examples where project delivery worked less well. Lessons on ‘what works’ are context-specific, but common themes include: clarity in responsibilities and ownership; alignment of expectations between government and delivery organisations; interest and consistency of inputs from government teams; and regular communications.

This section outlines process evaluation findings, focusing on the effectiveness of programme governance and delivery, including the selection of research projects and commercial arrangements. The analysis is informed by: (i) a review of programme documentation and monitoring data; and (ii) feedback from governance and programme management stakeholders, DCMS/DSIT delivery teams, and the external delivery organisations leading on projects. In each sub-section, a summary box explains the relevant context, followed by evidence from the evaluation (with summary assessments in bold).

2.1 Programme governance and delivery

2.1.1 Programme governance

Context for programme governance

The programme was launched in late-2022 as a vehicle to fund long-term, strategic and experimental research on key topics within DCMS sectors. In February 2023, Machinery of Government (MoG) changes resulted in a change of scope for DCMS and the creation of the Department for Science, Innovation and Technology (DSIT), a brand new Department with initially limited infrastructure to deliver against its research priorities. Responsibility over the digital and cyber security policy and evidence portfolios was transferred from DCMS to DSIT. Following the MoG changes, R&D Programme funding was split roughly equally between the two Departments to achieve mutually beneficial outcomes (with both involved in selecting projects). The programme was expected to fund research covering all DCMS areas and DSIT’s Digital Technologies and Telecoms directorate (ex-DCMS policy areas).

The entire programme, including the DSIT element of the funding, was delivered through the Chief Scientific Adviser’s (CSA) Office in DCMS^{[footnote 6]}, led by a dedicated Portfolio Programme Manager. There were two Senior Responsible Owners (SROs): the DCMS Chief Scientific Advisor and DCMS Director of Analysis.

Oversight was provided by a Programme Governance Board involving representatives from DCMS, DSIT and UKRI. The purpose of the Board was to provide advice and challenge, alongside reviewing milestones. Board meetings were held monthly for the first 18 months of delivery, thereafter shifting to quarterly meetings.

Funding decisions were reviewed and signed off by the two SROs, with inputs from the programme management team. The Governance Board was kept updated on the topics of interest with their views taken into account in project selection by the SROs; however the Governance Board did not have decision-making power on project selection. Project-level procurement decisions were made by the SROs and programme management team.

The programme’s governance arrangements were adequate overall for ensuring accountability, and appropriate in seeking to support the programme in realising its intended objectives. There was value in the programme being led by the CSA Office; this positioning brought credibility to the programme, and ensured the programme could take a strategic view across the Department in relation to research priorities and opportunities. The two SROs provided effective strategic leadership for the programme.

It is recognised that the involvement of both DCMS and DSIT in the programme, and therefore its governance arrangements, was driven by circumstance (i.e. MoG changes), rather than because this was the optimal arrangement for programme delivery (with the initial expectation that the programme would be DCMS-specific). However, feedback from government stakeholders indicates that this relationship worked well in practice. Those involved in governance were clear on the rationale and objectives of the programme, which was not seen to have been impacted materially by the MoG changes (note there were some concerns about the impact on wider communications/dissemination beyond the Governance Board which is explained in the ‘delivery of research projects’ sub-section, and the MoG changes had more of an impact on programme management which is covered in the next sub-section).

The Programme Governance Board operated effectively as a source of advice to facilitate effective programme delivery, and as a mechanism for reporting and accountability in delivery progress. Feedback indicates that the Board was well-managed, and successful in retaining engagement of the Board members from across DCMS, DSIT and UKRI. Regular meetings provided an effective mechanism to keep the Board updated on live projects across the entire portfolio, and a good forum for discussing any challenges and sharing potential opportunities to facilitate, improve and/or address challenges in project delivery. It was highlighted that there was good and consistent attendance at the Board meetings, and the Board was successful in achieving a “collaborative culture” in its activities and discussions. The make-up of the Board was considered by most (relevant) consultees to have provided sufficient balance in terms of organisations as well as roles. However, two stakeholders suggested that it was perhaps too ‘top-heavy’ and it would have been helpful to include a wider range of perspectives at different levels (e.g. more Deputy Directors and policy leads and senior analysts). For these consultees, this would have helped to ensure that “different voices at different levels” are involved in governance, which may have a positive impact on buy-in and engagement with the programme.

There are opportunities to strengthen governance arrangements for future phases. First, the remit and role of the Board regarding the programme’s decision-making processes could have been more clearly articulated to support delivery, strategic alignment and accountability. There was some lack of clarity around who was driving decisions and the role of the Board in this process, and the reasons why decisions were made including on project selection. For example, the Board was given opportunities to provide feedback on shortlisted projects, but they did not have decision-making power when it came to funding decisions.

We would have some useful discussions and then there would be decisions made elsewhere. We wouldn’t necessarily understand the rationale.

– Governance stakeholder

Second, and linked to this, the Governance Board could arguably have been leveraged more fully to provide ‘strategic direction’ to programme activity. Feedback indicates that its role was primarily to provide a forum for discussions about progress and potential opportunities i.e. acting as a ‘sounding board’. However, its role appears more limited in terms of actively driving programme delivery, identifying and progressing mechanisms to catalyse and maximise impact, and informing decisions to ensure strategic alignment and contribution. Given the seniority of Board members, this was a missed opportunity for the programme in Phase 1.

It’s been a good forum for sharing opportunities […] but in terms of formal decision making they seem to have had less of a role than they could have. I can’t remember very many tangible decision points.

– Governance stakeholder

Recommendation 1: Review the remit, role and composition of the Programme Governance Board and ensure that this is clearly articulated and communicated to those involved in the programme. This is an opportune moment for reconsidering the structure and balance within the Board, given that some changes are already anticipated with DSIT not involved in Phase 2. The review should consider whether the Programme Governance Board could be leveraged more fully to provide strategic direction to programme activity, and support awareness raising and dissemination.

Third, the DCMS College of Experts was set up in 2021 to provide independent expert advice to the Department, acting as a source of scientific and technical input. The College of Experts was renewed in 2025. Feedback from stakeholders indicated that there is a potential opportunity for this group of independent experts to be integrated into the programme’s governance arrangements to further strengthen the programme’s accountability.

Recommendation 2: Consider opportunities to leverage the expertise of the College of Experts to support programme delivery and impact. Potential roles may include: the provision of strategic advice to the Board and DCMS on the scope/prioritisation of research topics and project selection; expert review of deliverables; and support with the dissemination of findings. Any potential role for the College of Experts should take into account capacity and potential conflicts of interest for this group, and their relative contribution to other departmental priorities and activities.

2.1.2 Programme management

Context for programme management

The programme was managed by a dedicated Portfolio Programme Manager, responsible for scoping the projects, engaging with (potential) suppliers, managing the relationships with partners (e.g. UKRI), coordinating project monitoring and reporting to the Board, and ensuring continual alignment to departmental priorities. The Programme Manager was also responsible for project-level management, including representing DCMS on relevant project steering groups, and the practical management and oversight of all funded projects. The Portfolio Programme Manager received support from the CSA’s team and the DCMS Analysis function.

The programme was delivered with less management resource than originally planned. The Analysis Directorate’s budget settlement for 2022/23 included funding for five posts, but only one (for a Portfolio Programme Manager) was sought and subsequently approved in the Business Case.

The programme spent 92% of its Phase 1 budget ( £7.66m of the £8.34m). Most of this expenditure related to the research projects, with less than 5% spent on salaries and overheads. There was some underspend all three years, though this progressively reduced over time (see Table 2-1). The underspend was driven by the competitive bidding process, with some successful bidders delivering below the allocated budget, providing efficiencies for the programme in the delivery of research objectives.

Table 2-1: Programme expenditure

	2022/23	2023/24	2024/25	Overall
Budget	£1,667,481	£3,334,962	£3,334,962	£8,337,405
Actual spend (% of budget)	£1,447,406 (87%)	£2,985,004 (90%)	£3,227,093 (97%)	£7,659,503 (92%)

Source: DCMS

The programme is considered to have been well managed. The Portfolio Programme Manager was regarded as “instrumental in driving things”, bringing the right skills and expertise to the programme. The Programme Manager’s previous experience in academia was seen as especially important.

However, the scope of programme management activity was restricted by resource availability, and the model adopted in Phase 1 represented a high level of risk to programme progress and delivery. There was agreement that the programme did not have adequate resourcing: the programme management team comprised a single member of staff, instead of the five originally anticipated. The Portfolio Programme Manager was therefore overcommitted which restricted programme delivery, for example in levels of engagement and dissemination activity within and outside the Department. Additional resource may have helped to raise awareness of the programme more widely and faster (through both scoping activities and dissemination of outputs), ultimately helping to embed the programme and achieve its aim to bring about culture change. It is also noted that the impact of MoG change was significant, with considerable management time spent on navigating the changing context. The MoG change was considered to have been managed well overall, although some consultees (from both delivery organisation and government side) commented on the disruption. This was seen to have resulted in a perception of sub-optimal arrangements and less streamlined processes, owing to the involvement of a higher number of government stakeholders across two departments.

Progress in delivery was therefore highly dependent on the effectiveness of the appointed Programme Manager, both at programme- and project-level. In practice, no major issues were experienced associated with limited capacity regarding day-to-day activities, and the programme also benefited from continuity in the appointed Programme Manager. However, the strong reliance on the inputs of a single individual, for programme delivery and project-level management, represented a high level of risk and a ‘single point of failure’ should the Programme Portfolio Manager become unavailable. This was not the optimal arrangement, particularly given the breadth, complexity and novelty of the programme for DCMS. This should be a key consideration for Phase 2.

Recommendation 3: Ensure an appropriate level of resource is allocated for programme management, indicatively 2.5-3 FTE. The resource should reflect the scale of the programme and pro-actively consider and mitigate risks around staff turnover, capacity and continuity in project-level management and provider engagement.

The programme appears to have been well integrated within the wider DCMS research and analytical landscape, although consultation evidence indicates that awareness and understanding of its rationale/objectives varied across the Department. Within DSIT, feedback suggested that the programme was valued by those involved directly. However, there was no evidence that it had been integrated into the Department’s wider programme of research activities; although it is noted that DSIT will not be involved in Phase 2.

2.1.3 Selection of research projects and commercial arrangements

Context for selecting research projects The programme funded 22 research projects (see Annex B), identified in three ways:

• In Year 1, five projects were selected from the UKRI (specifically EPSRC and AHRC) portfolio of recently funded and well-reviewed research. These were aligned to cross-cutting evidence priorities (internally and externally).
• In Year 2, nine projects were selected via a four stage process. At Stage 1, a review was conducted of the Departmental Areas of Research Interests (ARIs)^{[footnote 7]}, cross-cutting evidence priorities, and the relevant parts of the UKRI portfolio, identifying areas of strength and gaps in long-term research and funding (resulting in over 100 research questions). At Stage 2, the identified questions were discussed with the relevant policy and analysis teams to determine appetite and feasibility for new research (resulting in a shortlist of c. 35 questions). At Stage 3, the questions were reviewed, prioritised and signed off by the R&D Governance Board and SROs. At Stage 4, the signed-off questions were taken back to the policy and analysis teams to refine and turn into concrete research scopes.
• In Year 3, eight projects were selected in a process that was driven by an overarching research priority setting exercise, conducted by the DCMS Central Analysis Team and approved by the DCMS Executive Board. In DSIT, project topics were recommended by the DSIT Analysis Directorate and approved by the DSIT Director of Analysis.

Projects were funded as fixed price pieces of work, via two procurement routes: tendering (13 projects) and Memorandums of Understanding (9 projects)^{[footnote 8]}. The tendering route utilised a Government Commercial Agency framework for research and insights services.^{[footnote 9]}

Approaches to prioritising research projects & the projects funded

The approach to prioritising research projects varied in each delivery year, in part owing to external circumstances and timing imperatives. This demonstrated positively a process of effective adaptation in response to learning from ongoing delivery. For example, it was recognised that the approach taken in Year 1 (i.e. top-up awards of existing UKRI grants) was largely driven by a pressure to “get money out the door”, and led to very limited interest and engagement from government teams in the subsequent project delivery stage. In Year 2, the selection process incorporated extensive engagement with the relevant policy and analysis teams (i.e. a bottom-up approach), in addition to reviewing priorities at the Departmental level (i.e. a top-down approach). In terms of the top-down approaches, it is noted that there were some shifts in government policy prioritisation which the programme had to respond to in order to seek to retain relevance to the prevailing policy agenda.

Consultation feedback suggests that the most effective approaches to identifying areas of research focus combined both ‘top-down’ and ‘bottom-up’ engagement (as in Years 2 and 3) to ensure both strategic alignment and buy-in from colleagues, whilst being mindful of the programme’s original rationale. However, challenges impacted on the effectiveness of the prioritisation process in Phase 1:

• Low levels of awareness and engagement with the programme across DCMS and DSIT, due to the approach taken in Year 1, which involved limited ‘bottom-up’ inputs to project selection.
• Whilst representing effective adaptation, the changes in approach to project selection within the programme appear to have led to a lack of clarity and understanding of the programme’s rationale across the Departments, in some cases resulting in selection of research projects that are (or are very close to) ‘business-as-usual’ research activity.
• Incorporating ‘bigger picture thinking’ on strategic research priorities and technology and societal trends has been undertaken to some extent, and appears to have improved over the three years of delivery. However, this requires ongoing review of other research activities within DCMS and across government to inform commissioning decisions, with resource implications. Further, this can be impacted by rapid changes in policy priorities and agendas.

Clarity on the core rationale and objectives of the programme (i.e. why it is needed and what it is looking to achieve), and the anticipated and resulting profile of the project portfolio that flows from this, is important in this context. This clarity will help to establish a shared understanding of what the programme is for and, in turn, help to frame and inform project development and selection, and ensure that strategic and funding decisions align with the underpinning case for intervention and programme purpose.

Further, the evaluation indicates there was varied understanding on the overarching programme aim to deliver a ‘step change’ in how DCMS approaches its evidence base, and what this meant in practice. Interpretations identified in the evaluation included generating a step-change in internal capability to commission and communicate research; the type of research funded and the level of risk appetite within the Department; how research is delivered with the external knowledge base; and internal Departmental management and co-ordination of research. These are not mutually exclusive, however, they are different. Importantly, the concept of the ‘step change’ was not formally defined at the outset of the programme. This should be considered in Phase 2.

Recommendation 4: Review and update the programme theory of change, focusing on clarity of rationale, aim and objectives, and pathways to impact (including timing and influencing factors). The updated theory of change should inform strategic planning around project identification and selection, and monitoring and evaluation. In reviewing and updating the theory of change, DCMS should consider the ongoing relevance and applicability of the overarching aim to deliver a ‘step change’ in how DCMS approaches its evidence base. If this aim is considered to be appropriate for Phase 2, this should be defined to provide clarity of purpose and intent.

The programme funded a larger number of smaller projects than originally intended^{[footnote 10]}. Feedback from programme stakeholders suggests this was primarily due to risk aversion in project identification and selection, with some reluctance from individuals developing project ideas to identify and lead on larger projects which were perceived to be riskier. This had mixed implications. Positively, the number of projects supported reflected strong demand, and helped to ensure that a wider range of research priorities and themes were covered than would have been possible with a more limited scope. However, there was general agreement from consultees that a programme portfolio with ‘fewer but larger’ projects would have been preferable both (i) to reduce the management and administrative time associated with a large number of projects and (ii) ensure alignment with the programme’s rationale of tackling longer term, cross-cutting and more speculative research questions, and its overarching aim to deliver a ‘step change’ in how DCMS approaches its evidence base.

In practice, the portfolio was perceived to be mixed in terms of alignment with the programme’s objectives. With different approaches to identifying research priorities, projects have: aligned ‘top-down’ and ‘bottom-up’ priorities; considered areas of interest that had previously remained unexplored for a lack of suitable funding mechanisms (e.g. measuring the legacy and impact of major events, and capturing engagement numbers at unticketed events); and addressed emerging technology areas (e.g. AI, metaverse). However, consultees identified some concerns that the purpose of the programme was not fully understood at all levels, leading to a small number of commissions that are closer to business-as-usual. Feedback from delivery organisations seemed to support this, with three project leads responding to the survey considering that their research was not novel or experimental (see Figure 2-1).

Recommendation 5: Building on the updated theory of change, develop a ‘strategic portfolio proposition’ statement for the programme. This statement would provide clarity on the intended composition and balance within the portfolio (and key considerations informing this) to inform project identification, prioritisation and selection. This should include consideration of: * risk tolerance thresholds for larger projects, with explicit reference to the risk of projects being funded which lead to inconclusive findings or are outpaced by technological developments and/or the political agenda * barriers to and incentives for ensuring higher risk projects are identified and funded, including practical implications (e.g. the length/coverage of scoping activity for longer-term/higher-risk projects, government team expectations and requirements).

Recommendation 6: Introduce a more consistent, structured and transparent approach to project selection across the programme period, aligned to the ‘strategic portfolio proposition statement’ (as recommended above).

This should continue to involve prioritisation of research topics via a combination of ‘top down’ and ‘bottom up’ approaches, alongside: * formally defined work flows i.e. articulation of a consistent process for project selection including key stages, progress and decision-points * eligibility criteria i.e. what is and is not considered to be in scope for the programme, including a definition of in-scope R&D activity, research/topic area coverage * minimum requirements i.e. any essential elements or characteristics that need to be included in all projects (e.g. specified research questions, clarity on alignment/relevance to departmental research priorities, use of novel or innovative methods or data-sets) * clear spheres of responsibility within government, recognising different levels of input and responsibility (notably the Portfolio Programme Manager, SROs and Programme Board).

Changes to the project selection processes during the programme (i.e. between years) should be minimised as far as practical, to provide continuity and certainty in programme management, accountability and communications.

In relation to the terms of the commissioned research, feedback from delivery organisations indicates a high level of satisfaction with the available budget and clarity of expectations (see Figure 2-1). However, some projects would have benefited from longer delivery timeframes from the perspective of delivery organisations. This aligns with feedback from government teams about some projects that were, in hindsight, considered to have been overly ambitious reflecting the scope and anticipated timescale.

Figure 2-1: Delivery organisation feedback on project terms

(A graph showing delivery organisation feedback on project terms.)

Source: Delivery organisation survey (n=15)

Commercial arrangement

The use of two commercial arrangements, through MoUs and tendering, was appropriate in principle for achieving the programme’s objectives. The MoU route was initially used “by necessity” in Year 1 to enable faster deployment of funds, effectively ‘topping up’ five existing UKRI grants (with the programme funding additional elements that would generate actionable research for DCMS/DSIT). This mechanism was then used again in Year 2 for extensions to two of the Year 1 projects.^{[footnote 11]} In Year 3, funding was awarded to two new projects through the MoU route, one with UKRI and another with the Natural History Museum. All other projects were delivered via competitive research framework commissioning process.

The two different routes to commissioning provided flexibility, with feedback suggesting that each has merit in different contexts. The advantages of one route are also often the disadvantages of the other, implying that the choice over the most appropriate arrangement depends on the specific project. The main merits of each mechanism are as follows (see Table 2-2 for a full list of advantages/disadvantages):

• The MoU route involves more flexible agreements, with scope to adjust objectives and plans as projects evolve (and therefore the potential for unanticipated benefits) but less control over outputs. For the tendering route, terms of reference are more clearly defined, offering less flexibility but greater control over outputs.
• The MoU route allows Government to leverage partner organisations’ expertise in identifying ideas/opportunities, but this means less exposure to a variety of ideas, compared to the tendering route which invites bids from a range of organisations.
• The MoU route enables better access to academic partners, who may be less likely to engage with tendering and/or are not registered on procurement frameworks. However, MoUs do not offer the potential for cost efficiencies (in project budgets) through a competitive bidding process, which is a key advantage of tendering
• The MoU route enables faster deployment and procurement efficiencies, compared to the standard tendering process. However, in practice there were initial challenges in setting up the MoU paperwork (though ‘lessons learned’ were noted).

Feedback on the advantages and disadvantages of each arrangement, identified through the evidence gathered in the evaluation, is summarised in Table 2-2. When considering the MoU route, it is important to distinguish between the advantages/disadvantages that are inherent to the MoU as a mechanism and those that appear to be more specific to working with UKRI (with the latter denoted in italics). For example, work with the Natural History Museum demonstrates that an MoU can be used effectively to initiate and co-develop projects with partners - which is very different to the UKRI ‘top-up’ approach.

Table 2-2: Advantages and disadvantages of different commissioning routes

Route	Advantages	Disadvantages
MoU	Faster deployment via an existing funding mechanism* Access to academic partners who may be less likely to engage with tendering Leveraging partner organisations’ expertise in identifying opportunities Flexibility in delivery, which may bring some unanticipated outputs/benefits Opportunity to work with experts who would not engage with tendering due to lack of capacity or experience	Initial administrative challenges for getting the MoUs in place Less exposure to different ways of thinking/approaches Less control of the process and outputs Risk of lower engagement by government teams Reputational risk from perception of ‘picking winners’ Restricted by UKRI funding restrictions, and the research groups being funded
Tendering	Exposure to different bids (and ways of thinking/approaches) Efficiencies through competitive bidding process More control of the process and outputs Provides structure for large projects	Restricted to organisations approved onto the tendering framework used Tendering process is more resource intensive Risk of favouring the ‘big players’ with ability to tender Less chance of unanticipated outputs/benefits

Source: SQW based on feedback from stakeholder consultations. *faster deployment may not have been achieved in practice, due to difficulties setting up the UKRI MoUs.

Recommendation 7: Continue to offer two different routes to commissioning (through MoU and tendering) to provide flexibility, reflecting on the most appropriate route on a project-by-project basis (informed by trade-offs around timescales, flexibility/control, etc). Where the MoU route is chosen in projects that have been identified by partners, the programme management team should consider ways to ensure good engagement and buy-in from government teams.

2.2 Delivery of research projects

Context for project delivery

The funded projects were led by research teams at the delivery organisations commissioned to deliver the research, which included academic institutions, research organisations, and consultancies. Projects were delivered by a mix of single organisations and consortia.

Government teams supported the delivery organisations by providing various inputs (e.g. advising on research scope/design, reviewing and providing feedback on deliverables), with most projects managed by mixed teams of at least one analyst, one policy adviser, and the Portfolio Programme Manager (constituting a project-level Steering Group).

The scope and shape of the government team was bespoke for each project, including the number of individuals involved, the intensity and nature of this involvement, and the mix of analytical and policy officials. This reflected the scope, subject matter, and research approach of each project, and different interest, engagement and capacity of analytical and policy officials.

The Portfolio Programme Manager provided weekly reports to the two SROs, who had the option to join project-level Steering Groups if necessary.

Feedback from both delivery organisations and government teams suggests that experiences with project delivery were generally positive in terms of the skills/expertise of those involved and the processes adopted. However, this did vary by project, and there were examples where project delivery worked less well. This variation reflects in part heterogeneity within the programme portfolio in terms of the types of projects funded, commissioning routes, and people involved. Although lessons on ‘what works’ in project delivery are context-specific, feedback suggests some common themes. These lessons were informed by learning across the programme portfolio, including both projects where delivery had worked well (i.e. these factors were evident) as well as projects where things did not work so well and opportunities for improvement were identified (i.e. at least some of these factors were not evident). Specifically, the common themes for successful project delivery related to:

• clear responsibilities and ownership
• alignment of expectations at the outset (for timings, deliverables, etc)
• buy-in and interest from government teams
• consistency of inputs from government teams
• regular communications (including interim outputs/checkpoints).

More detailed feedback is provided in the sub-sections below, on the role of both the research teams leading project delivery and the government teams.

2.2.1 Role of research teams leading project delivery

Feedback from government consultees suggests that research teams from delivery organisations leading project delivery generally had the right skills and expertise. Many were consortium bids, drawing on expertise across consultancy and academia. There are examples of projects that required a degree of flexibility which the delivery organisations were able to deliver. However, some government consultees reflected on whether the mix of skills in the team appointed was, in hindsight, sub-optimal; these comments related to three projects (out of the 22). In two cases, feedback suggested that a consultancy-led team would have been preferable. This reflected considerations of the style, tone and accessibility of outputs developed from an academic perspective, and the pace of delivery, that did not align with expectations. In one case, it was suggested that a consortium of academics would have been more appropriate; the explanation for this preference in hindsight was not provided.

Similarly, overall feedback from government consultees on the research processes, communication and co-ordination by research teams from delivery organisations is positive. Interim touch points worked well to ensure projects were progressing as expected. Government consultees noted the importance of delivery organisations building in sufficient time for client review of interim outputs (especially if any decisions are required).

The consistency of inputs of research teams across the lifespan of the projects was generally seen as appropriate by government teams. However, in several cases where final deliverables went through several rounds of feedback there was a perception that “enthusiasm and appetite to engage waned” amongst delivery organisations. Further, in some projects, communication challenges were reported to have adversely impacted delivery, with key issues including:

• a lack of agreement (e.g. in terms of scope or methodology) or clarity (e.g. expectations for timings or deliverables) between the research team and the government teams
• requests for government teams to review deliverables at short notice and in parallel (instead of a more staggered approach), resulting in bottlenecks
• final deliverables that were of lower quality than anticipated, requiring several rounds of iteration, which may have been avoided by building in opportunities for government teams to review interim deliverables.

Across a portfolio of 22 research projects, explicitly focused on undertaking longer-term and novel research, it is not unexpected that some issues were experienced. However, the issues identified can helpfully inform thinking on project set-up and implementation in Phase 2.

2.2.2 Role of government teams

Delivery organisations indicated generally high satisfaction with the skills/expertise of government teams involved in the projects, the processes in place, and the level of involvement. Feedback on the analytical skills and expertise of government teams was particularly positive (see Figure 2-2). This aligns with feedback from government consultees who generally considered the projects to have been well-aligned with their own background and experience.

Figure 2-2: Delivery organisation feedback on inputs from government teams

(A graph showing delivery organisation feedback on inputs from government teams.)

Source: Survey of delivery organisations (n=15)

Delivery organisations reflected that government teams generally had the right level of involvement across the different stages (planning, delivering and quality assuring the research). Only a handful reported too little involvement (five respondents^{[footnote 12]}) or too much (one respondent) across at least one of the three stages. Feedback on what worked well, and less well, is outlined below.

Table 2-3: Delivery organisation feedback on the level of involvement by government teams

	Planning	Delivering	Quality assuring
About the right level of involvement	13	13	12
Too little involvement	2	2	2
Too much involvement	-	-	1

Source: Survey of delivery organisations (n=15)

In the majority of the portfolio where the involvement of government teams was considered to have added value, survey responses highlighted examples of various support throughout project delivery (in line with the above), from defining the research questions and research design, to ongoing steer and feedback on deliverables. Two clear areas of added value were identified, with government teams complementing the skill set of the delivery organisations: helping coordination with other government stakeholders (e.g. by collating feedback); and bringing an understanding of the wider context for policymaking (e.g. in terms of research design being aligned with DCMS/DSIT needs, how the findings could be communicated for greater impact). Feedback from government teams indicates that they valued opportunities to steer the projects and feed into conversations about project progress and findings. More generally, stakeholders from both sides emphasised the importance of effective collaboration, underpinned by open and consistent communication.

We had a strong and valuable working relationship with the programme, with timely and thorough communication and guidance throughout the project.

– Delivery organisation

[Government teams] brought that strategic view of where the research was going to go and what it might do in future.

– Governance stakeholder

In the smaller proportion of the portfolio where government teams were considered to have added less value, there were some common challenges relating to:

• Limited engagement from government teams, due to a lack of capacity and/or interest, which affected adversely the level and quality of inputs

The government team added value, but they were significantly under-resourced given the size, scale and complexity of the project. There did not seem to be sufficient input - and at times, interest - from the government teams.

– Delivery organisation

• Staff turnover at DCMS/DSIT, including due to MoG changes, which particularly affected the consistency of inputs in longer-term projects (though it was noted that the Portfolio Programme Manager had a considerable role in filling those gaps)

There was a lot of turnover on [DCMS/DSIT] side with multiple policy leads over the course of the project. This resulted in a loss of institutional memory.

– Delivery organisation

• Ambiguity about expectations and responsibilities/ownership, both between the delivery organisations and government teams, or across the latter, which led to reported sub-optimal inputs from government teams

[…] we’d be having meetings with the research team and it felt like there was an important point to be made, but it wasn’t clear which of us owned that point.

– Policy adviser

Recommendation 8: Take further steps to mitigate key risks around project delivery, informed by the evaluation evidence on ‘what works’. This should build on programme management to date, with a focus on ensuring that the following aspects are consistently experienced across the portfolio:

• Alignment of expectations: project plans should clearly establish the responsibilities/ownership, timings and requirements for deliverables (including scope, tone, intended audiences, etc).
• Project progress and relevance considerations in the context of technological developments and policy priorities: appropriate interim milestones and reflection points should be incorporated into project plans, particularly for longer-term projects.
• Consistency of inputs from government teams: addressing barriers relating to both interest (ensuring that the relevance of the research and/or value in developing new skills/knowledge is clearly tested, defined and articulated at the outset in discussion with team members) and capacity (assigning a ‘lead’ contact for each project with responsibility for DCMS-side project management, involving more colleagues as contingency, with clearly established processes for handover; this would also help to free up programme management resource).

3. Impact evaluation

Key findings

• The programme has produced a substantial volume of research, including written deliverables and toolkits, supported by dissemination activities. These outputs generally aligned with expectations, but some quality issues were reported. In Phase 2, monitoring processes for consistently recording the outputs could be improved.

• Based on the programme theory of change, two key routes to impact are evident, through: (1) generating evidence (i.e. DCMS’s capability to undertake R&D activity); and (2) holding and using evidence that has been generated (i.e. the capability to exploit and apply the R&D).

  • In relation to Route 1, there is evidence that the programme contributed to knowledge spillovers, upskilling, improved capability/confidence, and better alignment of research to DCMS’s needs. These benefits were largely limited to those directly involved in projects, representing a missed opportunity in wider benefits diffusion. There is very limited evidence of savings/efficiencies or a reduced need to outsource research.
  • In relation to Route 2, the programme represents a more proactive and strategic approach to evidence within DCMS, and has generated a breadth of robust evidence. There are specific examples of the use of evidence, however, overall there is limited evidence of improved decision making in strategic policy areas to date, and therefore no evidence of savings and efficiencies as a result of better policy. The value to date has been primarily from ‘holding’ rather than ‘using’ evidence.
  • The programme has also generated outcomes beyond its stated objectives, such as reputational benefits for DCMS; new/stronger links with the research community; and wider impacts outside DCMS through application of the research.
• The programme has played an important role in realising these outcomes. This is underpinned by reasonably high ‘activity additionality’ (i.e. project activity would not have happened otherwise) which is directly linked to a high level of ‘outcome additionality’ (i.e. the benefits would not have happened otherwise). However, a small number of projects may have gone ahead anyway, using other funding mechanisms.
• Key programme aspects that contributed positively to outcome realisation included consistency in delivery and buy-in across levels. More widely, the enabling role of technological advancements, political agenda/interest, and a cross-government culture change were highlighted as important factors influencing the realisation of outcomes.

This section outlines evidence relating to the impact evaluation, including project outputs and outcomes/impacts. For the latter, two reinforcing ‘contribution stories’ are presented. The analysis is informed by: (i) a review of programme documentation and monitoring data; and (ii) feedback from governance and programme management stakeholders, DCMS/DSIT delivery teams, and the external delivery organisations leading on projects.

3.1 Project outputs

The projects funded through the programme have produced a substantial volume of outputs, covering a range of research topics. All 22 funded projects generated written deliverables, most had organised/attended events (16 projects) to disseminate findings, and a smaller number had delivered toolkits (6 projects). By April 2026, written deliverables for 18 projects had been published. For the six projects with deliverables published by DCMS^{[footnote 13]} since November 2024, data on views indicate a reasonable level of engagement with over 7,700 unique page views^{[footnote 14]} for the primary reports and over 9,200 unique page views for other published resources associated with these projects. These figures appear to indicate a reasonable level of engagement with the findings of the research projects supported by the programme. However it is noted that: (i) it is difficult to contextualise these given the lack of an appropriate benchmark; and (ii) page views are not the sole indicator of engagement given that project deliverables may have been circulated via other channels (particularly as some resources were intended primarily for internal audiences).

Table 3-1: Outputs across the portfolio

Type of output	Number of projects
Written deliverables (reports, policy briefings, journal articles, datasets)	22
Toolkits (software/code, models/spreadsheets, workflows, checklists)	6
Events (workshops, policy labs, conferences, training days)	16

Source: SQW based on DCMS data

Evidence from the case studies: deliverables

Case studies highlight the range and scope of deliverables associated with the programme, both across the project portfolio, and within individual projects. For example, the “Measuring the Legacy and Impact of Major Events” project produced two outputs: a practitioner-focused evaluation toolkit, and a set of applied case studies. The toolkit is a structured framework for evaluating major events, including templates and guidance. The case studies operationalise this framework by applying the methods to four events. The two phases of the “Shaping the Metaverse” produced a range of publications of different types. Phase 1 involved the publication of a detailed report aimed primarily at government officials working in relevant policy areas, but also with navigation for members of the public and researchers. Phase 2 then involved the publication of eleven research papers. Both stages of reporting were published on a dedicated page about the project on the delivery organisation’s website; the Phase 1 report had c. 2,800 downloads by April 2026, and some of the Phase 2 papers have been submitted for publication to academic journals.

The evaluation found that the programme has not consistently collected and reported on monitoring data on outputs. There are some gaps and lack of detail in monitoring data (e.g. in terms of attendance at events, whether the events were internal- or external-facing) and inconsistencies (e.g. in terms of whether project inception meetings constitute ‘events’). This should be improved in Phase 2 to ensure that there is a clear understanding of what has been delivered. At a minimum, the programme management team should keep a record of the number of deliverables expected and delivered (as a ‘live’ document, including publication status) with supporting data regularly reviewed and updated (e.g. page views). There is also an opportunity to collect more detailed data to support future evaluation activity (e.g. purpose, audiences, number of attendees at events; publication and reach of academic articles, policy briefings, etc).

Recommendation 9: Improve processes to ensure robust data collection, monitoring and evaluation.

The evaluation of Phase 1 of the programme highlighted a particular gap in consistently recording project outputs (across the three categories, as per theory of change). This should be addressed as a priority to support future evaluation of the programme.

DCMS should also consider developing a programme monitoring and evaluation framework, to ensure there is a structured plan for assessing performance against the intended outcomes and impacts (aligned with the updated theory of change). This should inform ongoing evidence collection of outcomes, e.g. by asking for feedback at project completion from Government teams and delivery organisations, and establishing time-points for follow-up primary and secondary research (where appropriate). As part of this framework, the programme team should also put in place a proportionate mechanism to track ongoing evidence on the use of the Phase 1 research findings by the Department.

Qualitative evidence from government stakeholders confirmed that the projects had produced evidence across the three categories set out in the Business Case: long-term strategic evidence; ‘what works’ evidence; and evidence using advanced and novel research techniques.

The written outputs and toolkits were generally in line with government expectations. However, in some cases government teams reported quality issues, for example that analysis was too descriptive or too technical, or there was less detail than expected. This led to delays in project completion and publication. Quality of outputs appears to have been affected by the key challenges for project delivery outlined in Section 2, including a lack of clarity around expectations and sub-optimal communication between the delivery organisation and government teams. It was highlighted that for broad dissemination it is important for written deliverables to achieve the right balance between accessibility and a sufficient level of technical detail (i.e. catering to different types of audiences), which has not always been achieved. In that context, government stakeholders who had attended any dissemination workshops provided positive feedback, with these sessions being perceived as informative and helpful for generating engagement with the research.

Dissemination workshop quality was very good, with colleagues asking questions on how to take [the findings] forward for policy. There was a buzz following the workshop.

– Analyst

As noted in Section 3 in relation to process, it is expected that across 22 research projects some challenges will be experienced in delivery in relation to quality. This is particularly relevant for projects adopting novel and/or advanced research techniques. Further, it is important that quality control of deliverables is maintained, with strong evidence from the evaluation that the programme team took action to seek to address concerns in advance of project completion and sign-off. This is a positive finding.

3.2 Contribution story

We have developed two reinforcing contribution stories that assess the programme’s contribution to outcomes associated with two key routes to impact:

• outcomes associated with generating evidence (i.e. the Department’s capability to undertake R&D activity effectively and efficiently)
• outcomes associated with holding and using evidence that has been generated (i.e. the Department’s capability to exploit and apply the R&D).

Progress across these two themes has been taken for the evaluation as the routes to achieving a ‘step-change in how DCMS approaches its evidence base’ (the programme’s overarching objective), which also encompass the programme’s four objectives. The key outcomes from the theory of change associated with each contribution story are listed in Table 3-2.

Table 3-2: Mapping outcomes across the two key routes to impact

Route to impact	Associated outcomes from the theory of change
1. Generating evidence	Facilitate knowledge spillover effects between government and external experts Build internal capability and confidence with advanced/experimental methods Upskilling of researchers and government colleagues Reduced need to outsource evidence work due to greater internal capability Better alignment of external research and government needs Savings and efficiencies (e.g. as a result of new/improved skills)
2. Holding and using evidence	Evidence that is based on robust research and can be drawn on Improved decision making in strategic policy areas by delivering a robust evidence base, especially in the longer term and strategic evidence areas not covered by existing research arrangements A more proactive and strategic approach to evidence Savings and efficiencies (e.g. savings as a result of better policy decisions, which is more indirect/longer term)

Source: SQW

The sub-sections that follow present findings in relation to the two routes to impact, based on evidence in relation to the following questions:

• Is there a reasoned theory of change? Have activities been implemented as set out in the theory of change?
• Is there evidence that the expected outcomes have occurred (or are expected to)?
• Was it the programme, rather than other influencing factors, that made the difference to these outcomes (or the decisive difference)?
• What are the factors of programme design that have delivered this contribution?

In relation to each of the questions, a summary assessment is provided (in bold), followed by an overview of the underpinning evidence which informed this.

3.2.1 Route to impact 1: Generating evidence

Is there a reasoned theory of change? Have activities been implemented as set out in the theory of change?

A reasoned theory of change is evident, which remained appropriate and relevant throughout delivery, but awareness and understanding of the programme’s purpose varied. There is broad consensus that the programme provided a route to generating evidence that would not have been possible via other funding mechanisms, in line with its rationale. Delivery organisations reported ‘openness to novel methodologies’ and ‘level of research freedom’ as the main factors setting this programme apart from other government research activities of which they had experience and knowledge. However, feedback indicates lower awareness of the programme’s rationale among policy advisers and analysts involved in the projects (relative to governance/delivery stakeholders), with some concerns expressed by consultees for the evaluation that the purpose of the programme was not fully understood more widely among colleagues across DCMS.

Activities were delivered largely as anticipated, but some aspects worked differently in practice. Projects were led by delivery organisations, with inputs from government teams which were generally considered valuable. Across the portfolio, project outputs included written deliverables, toolkits and events. However, in three respects the scope of activities as delivered was different from the original theory of change:

• The scope of programme management was significantly restricted by resource availability (see Section 2.1.2).
• The programme funded a larger number of small projects than originally intended, and there were some concerns that a small number of projects may have been closer to ‘business as usual’ research than ‘fundamental’ research (as referenced in the theory of change; see Section 2.1.3).
• In some projects, inputs of government teams were perceived to be limited or of lower value (see Section 2.2.2).

Is there evidence that the expected outcomes have occurred (or are expected to)?

Of the six outcomes relevant to this route to impact, there is evidence of four outcomes having been achieved (see Table 3-3). The underpinning evidence is presented below.

Table 3-3: Evidence on outcomes

Relevant outcomes from the theory of change	Assessment
Facilitate knowledge spillover effects between government and external experts	Achieved
Better alignment of external research and government needs	Achieved
Build internal capability and confidence with advanced/experimental methods	Achieved
Upskilling of researchers and government colleagues	Achieved
Reduced need to outsource evidence work due to greater internal capability	Not achieved
Savings and efficiencies	Not achieved

Source: SQW

Outcome: Facilitate knowledge spillover effects between government and external experts

There is evidence the programme has led to knowledge spillovers between some government colleagues and external experts. In projects where there was successful engagement between government teams and delivery organisations, knowledge spillover benefits were reported for both sides. Spillovers are inherently challenging to evidence, reflecting that they often relate to informal engagement and discussions throughout project delivery, and are not captured in real-time. However, government consultees referred to access to ‘top experts’ as a key benefit of the programme, enabling ongoing discussions and engagement with relevant experts in the topic areas. There were examples cited in the evaluation of knowledge transfer with government teams developing technical knowledge, and delivery organisations developing a better understanding of the policymaking context. These spillovers are particularly important in the context of the programme connecting experts, within government and externally, who “wouldn’t normally have come across each other”.

We were able to get a different view on the policy area that we’re working in, by working with people who are at the cutting edge of AI research in the UK. They thought about it from our perspective and we thought about it from their perspective. It was really helpful to have that opportunity, and then to be able to integrate that into our policy thinking.

– Policy Adviser

It provides an opportunity for the policy side to work closely with the research community and say, this is how I’d like this information presented; this is how the information will be useful to us. This experience will help researchers understand what is useful in practice. There have definitely been some spillover impacts from that.

– Governance stakeholder

However, there is an apparent missed opportunity in facilitating wider spillover effects through broader and more effective dissemination of learning. Feedback indicates that knowledge spillovers were largely limited to the individuals directly involved in the projects, within government and delivery organisations. This is not unexpected. However, these benefits have not been diffused as widely across the research community through dissemination and knowledge sharing activities as may be expected reflecting the breadth and scale of research supported by the programme.

This said, it is noted that capturing these effects is challenging. There are examples of activity which has engaged wider groups where knowledge spillovers may have been realised, with the programme providing a platform for engagement across different groups and communities of interest.

Evidence from the case studies: providing a platform for wider spillovers

Phase 1 of the “Shaping the Metaverse” project involved extensive engagement including a workshop with 30-40 DCMS policy analysts and advisors, and five follow-on evidence gathering workshops across the UK. The over 100 attendees across the workshops self-identified as having research interests in, or professional experience of, metaverse related technologies or policy areas, and represented NGOs, local government and UK government. The project concluded with a two-day conference at Queen’s University Belfast for international experts which was considered a helpful opportunity to share the findings and invite feedback.

Outcome: Better alignment of external research and government needs

There is good evidence of the programme leading to better alignment of external research and government needs. There are two key routes to this benefit. First, through the programme’s initial ‘horizon scanning’ and commissioning against an identified government research need. Second, and related to the previous outcome, through knowledge spillovers between government teams and the delivery organisations which has led to a mutual understanding of the requirements and capabilities of both parties. In projects involving academic partners, consultees referred to the experience helping to “break down language barriers” between government and academia. The majority of delivery organisations explicitly referred to an improved understanding of government research needs (9 of the 15 survey respondents) as a result of their participation in the programme. There is also potential for any upskilling benefits (see further below) to further strengthen the alignment of external research and government needs over the longer-term.

Relatedly, better alignment between DCMS and its key partners was also reported. Examples include the programme’s approach to commissioning via UKRI being replicated by another department, and DCMS working with a research council on future research planning to avoid duplication of efforts.

Evidence from the case studies: aligning research with government needs

The “Measuring the Legacy and Impact of Major Events” project represents a strong example of aligning research with government needs. Although legacy claims are frequently used to justify public investment in such events, evaluations typically focus on short-term outcomes and rarely assess impacts against a credible counterfactual. The project sought to develop long-term evidence and capability, rather than immediate policy demands or evidence requirements related to specific event opportunities. The aim was to provide practical guidance for evaluation practitioners to improve methodological rigour in future assessments of impact. The research is expected to deliver future savings and efficiencies for DCMS by reducing duplication and the need to develop new approaches when commissioning future evaluation work. The toolkit and case studies were published in early-2026. It is noted that publication of project outputs was significantly delayed, and dissemination plans are now underway. This includes an internal seminar which was attended by 50+ Government policy and analyst colleagues.

Outcome: Build internal capability and confidence with advanced/experimental methods

There is evidence of the programme helping to build internal capability and confidence with methods within government, but this was dependent on the extent to which government teams were engaged with the research. There are several examples of knowledge development in emerging technology areas (e.g. AI) and with innovative methods (e.g. long-term evaluation, tracking engagement at unticketed events via social media and other sources). Further, it was highlighted that given the aims of the programme, the value of the research can sometimes be in relation to identifying gaps in knowledge and understanding (rather than generating knowledge per se) which can inform future research activities. As one government stakeholder put it, the programme has a role in “turning unknown unknowns into known unknowns”. A potential downstream benefit of the capability building relates to confidence with future commissioning of research because “if you’ve never seen these methods before, you are less likely to commission work using them.” In a small number of projects, such benefits had not been experienced due to limited involvement of government teams.

However, the capability and confidence building is largely limited to the teams directly involved in project delivery and not transferred across the Departments. Feedback indicates that there were “no structured ways to transfer the learning” from those directly involved to the other 100+ analysts across DCMS (and DSIT). This represents a missed opportunity in terms of maximising the potential benefits from the programme, but also in terms of mitigating against the risk of staff leaving (and taking the knowledge with them).

Outcome: Upskilling of researchers and government colleagues

There is evidence of the programme helping to upskill the researchers at delivery organisations. Feedback from delivery organisations indicated a range of benefits, including: enhanced research/technical capability or capacity (12 of the 15 survey respondents); improved staff skills/expertise (11); and increased organisational profile or credibility (9). Qualitative feedback referred to: the programme being a useful ‘springboard’ for early career researchers; improved project management capabilities; diversification of research portfolio; and improved reputation among key stakeholders.

This research enhanced our organisational reputation through creation of a new research output with value for the culture sector and policymakers.

– Delivery organisation

The project allowed us to expand our programme reach and subject areas. It also diversified our research portfolio.

– Delivery organisation

There is also some evidence of the programme upskilling government colleagues directly involved in the funded projects, directly linked to the evidence reported above in relation to spillover effects and capability/confidence building. However, this has not been transferred more widely across the Departments, and there was some evidence from case study research that the coverage of outputs could have been designed to better support skills development. The underpinning evidence aligns with the outcome above (building internal capability and confidence).

Evidence from the case studies: upskilling of government

There is evidence that the “AI Skills for Life and Work” project led to upskilling of DSIT analysts involved. The project’s work on AI skills definitions - with the identification of three types of AI workers - has provided a foundation for further job vacancy analysis. The project has also increased interest in more complex research methods: the project adopted a Delphi Study which was seen as an effective method for researching novel/rapidly evolving technologies, and future demand modelling. However, the lack of a technical annex accompanying the report on future demand has hindered analysts from building upon this work more fully. There was also evidence of upskilling of DCMS colleagues from the “Future of TV Distribution” project. This project involved advanced modelling techniques to consider audience behaviour, demographics, technology and infrastructure constraints to 2040. Benefits to DCMS analysts arose from exposure to long-term modelling approaches, including contributing to discussions around underlying modelling assumptions.

Outcome: Reduced need to outsource evidence work due to greater internal capability

There is very limited evidence of the programme leading to a reduced need to outsource evidence work due to greater internal capability. Where there is evidence, this lacks consensus and is project- or context-specific. Some governance consultees believed that, with changes and concerted action, such as around a greater focus on internal dissemination, there is potential for this outcome to become apparent in Phase 2 (and also for Phase 1 projects where outputs are newly published or yet to be published). There is also a potential link to the outcomes above: as skills and confidence develop, more tasks can be undertaken internally and the need to outsource evidence is reduced. However, it was highlighted that this outcome is unlikely to be fully achieved because of the inherent value and importance of DCMS having access to external, independent evidence.

Outcome: Savings and efficiencies

There is no evidence of the programme leading to tangible savings and efficiencies to date, which is not unexpected at this stage. This aligns with the evidence on reduced need to outsource evidence work due to greater internal capability (presented above) which would be a key route to savings and efficiencies, but has been limited to date and is unlikely to be fully achieved in the future. It was, however, suggested that DCMS has become a “more informed customer of research” which may lead to cost savings over the longer-term (e.g. via more effective procurement or quality control of deliverables).

Other outcomes

Beyond the logic model outcomes, feedback indicates that the programme has led to two other outcomes for DCMS associated with generating evidence:

• Enhanced perception of DCMS as an R&D-active department: It was considered that having an R&D programme has given DCMS a ‘seat at the table’ in relevant cross-government discussions, and helped to ‘make the case’ internally for the importance of R&D. The programme was described as instrumental for embedding the CSA Office within DCMS by demonstrating its valuable role in relation to meeting the research needs of the Department. This has coincided with wider efforts to strengthen the department’s evidence base and research culture, including the launch of a new DCMS Monitoring and Evaluation Strategy and a renewed College of Experts (both in 2025).
• New or strengthened links with the science and research community: Feedback from government stakeholders referred to a benefit in relation to networks and partnerships, potentially acting as “a gateway to other things”. Several projects reported ongoing communications with the delivery partner (following project completion), and in one case this had led to further collaboration activity. This was confirmed by feedback from the delivery organisations, with the majority referring to a strengthened relationship with government (13 of the 15 survey respondents), and new or strengthened relationships with other research organisations (9 respondents).

Was it the programme, rather than other influencing factors, that made the difference to these outcomes (or the decisive difference)?

The programme has played an important role in realising the above outcomes, and in some cases was seen as critical compared to any other contributory factors. This is underpinned by the reasonably high ‘activity additionality’ associated with the programme, i.e. its role in funding research that would not have happened otherwise. The majority of the portfolio is unlikely to have been prioritised/funded otherwise (at least not within the same timeframe), and therefore the benefits would not have occurred (i.e. high ‘outcome additionality’). However, as explained in Section 2, it is possible that some of the projects may have been funded anyway, particularly those closest to current policy priorities.

This said, the in-depth evidence from the case studies also suggests that where projects had been funded, the scale, quality or scope of the research may have been impacted. For example:

• For the “Future of TV distribution” project, some strands of the research would likely have been progressed, but elements including audience modelling would not have been feasible, and a lower quality supplier (with associated reduced cost) would have been used.
• The “AI Skills for Life and Work” would not have happened in its current form without the programme, due to its holistic and longer-term approach; only smaller elements (e.g. surveys or labour market analysis) would likely have been delivered.
• For the “Natural History Museum AI Pilots”, most of the AI pilots would not have been progressed; one of the five pilots may have happened (linked to a wider programme of activity), but this would have been delayed and DCMS is unlikely to have been sighted on this work, limiting the potential for the activity to lead to knowledge spillover benefits.

Other factors that have contributed to the outcomes (in addition to the programme) relate primarily to those influencing buy-in and interest in the specific research topics supported by the programme. Notably (i) technological advancements in relevant areas (e.g. AI); (ii) the level of government priorities and political interest (e.g. loot boxes); and (iii) wider societal interest in these areas. It was also noted that there has been a wider culture change across government towards a more strategic approach to evidence. There are possible causal linkages between these factors, e.g. AI tools being made publicly available has led to increased interest from the public, which in turn has made this area more of a priority for policymakers. This said, around half of the consultees (particularly analysts and policy advisers) were not able to identify other contributing factors, which supports the evidence of the programme making a strong relative contribution.

What are the factors of programme design that have delivered this contribution?

The key aspects of the programme that have contributed positively related to consistency in programme delivery (with strong oversight and steer from the Portfolio Programme Manager), and buy-in at both senior levels (to enable commissioning decisions) and from the analysts and policy advisers directly involved in projects (with personal interest in project contributing to success). Other factors positively influencing project delivery include close working relationships between project teams and delivery organisations, effective communications, and alignment of expectations (for full findings on what worked well, see Section 2.2).

The main hindering factor was the limited management resource allocated to the programme which limited awareness raising and dissemination activities. The MoG change in 2023 impacted on set up and early delivery (e.g. in terms of who should be leading on projects related to the ‘digital’ policy area) which led to some delays. Other factors were project-specific, e.g. where delivery was impacted by turnover of staff, misalignment of expectations, limited inputs from government teams, or short delivery timescales (for full findings on what worked less well, see Section 2.2).

3.2.2 Route to impact 2: Holding and using evidence

Is there a reasoned theory of change? Have activities been implemented as set out in the theory of change?

A reasoned theory of change is evident but there is lack of clarity about some of the pathways to impact, which warrants a review of the logic model in Phase 2 of the programme (linking to a recommendation earlier in the report; see Section 2.1.3). Activities were delivered largely as anticipated, but there was limited dissemination of project outputs which has restricted the use of evidence produced through the programme. This route to impact can reasonably be expected to take longer to evidence compared to Route 1, reflecting the programme’s focus on longer-term and more speculative research questions, the varying ways in which research can inform policy development processes. Outcomes associated with holding and using the evidence also only emerge following project completion and (where relevant) publication and dissemination; they are inherently ‘later’ in the pathway to impact than those associated with generating the research.

The programme theory of change incorporates a valid expectation that research can lead to a more strategic use of evidence and influence decision making, but how this would be realised was not clearly articulated. Evaluation evidence indicates that there are various assumptions underpinning this route to impact which did not hold in Phase 1 (e.g. there is sufficient dissemination) and, on reflection, the outcomes as defined in the theory of change relating to policy influence may be more appropriate classified as impacts (i.e. reflecting greater uncertainty and longer time to impact). This should be considered in the proposed updated theory of change for Phase 2, with implications for monitoring and evaluation.

Relatedly, feedback indicated that a key challenge for the programme was achieving an appropriate balance between addressing gaps in evidence on long-term structural issues, thereby providing a body of evidence where the value is that the evidence is “there if we need it”, and generating actionable evidence where the value is that the evidence can be or has been “used now”. Both of these are valid in principle and align with the programme’s rationale, but there could have been greater clarity on what the programme was seeking to achieve (which informs the intended/preferred balance between the two routes) in order to help frame expectations around impacts and contribution.

Is there evidence that the expected outcomes have occurred (or are expected to)?

Of the four outcomes relevant to this route to impact, there is evidence of three outcomes having been fully or partially achieved (see Table 3-4). The underpinning evidence is presented below.

Table 3-4: Evidence on outcomes

Relevant outcomes from the theory of change	Assessment
Evidence that is based on robust research and can be drawn on	Achieved
A more proactive and strategic approach to evidence	Achieved
Improved decision making in strategic policy areas by delivering a robust evidence base, especially in the longer term and strategic evidence areas not covered by existing research arrangements	Partially achieved
Savings and efficiencies	Not achieved

Outcome: Evidence that is based on robust research and can be drawn on

There is strong evidence from the evaluation that the programme has led to evidence that is based on robust research and can be drawn on. This outcome is associated with DCMS (and DSIT) now holding the evidence that has been generated through the programme. This constitutes an important outcome, as this evidence is now available and accessible, with the potential to be used to inform policy discussions in the future, as necessary and appropriate.

The projects funded by the programme have produced a large volume of evidence across a breadth of research topics, which has been presented in different formats (written reports, toolkits, events/webinars). The research topics largely align with DCMS/DSIT priorities, and so the findings from this evidence should be relevant to colleagues across the Departments. Reflecting the speculative and experimental nature of the programme, in some cases the evidence was considered to have helped to “crystallise the evidence gaps” identified by previous research, or have provided further nuance instead of “black or white answers”. There was some expectation that new evidence (including in relation to novel methods) could be drawn on in the future.

The project took a long time, its findings are broad and disparate. This is a really good thing, because it encourages cross-department working, it breaks down siloes, it looks under the hood of a phenomenon rather than just seeing it at face value. It’s much more about the underlying mechanisms and drivers.

– Analyst

Evidence from the case studies: robust evidence that can be drawn on

The “AI Skills for Life and Work” project has provided robust research in an area which is high-profile and topical, but where evidence is limited. Where it exists, there are often concerns about impartiality and unclear methodologies. The project addressed both concerns, enhancing confidence in its use. DSIT analysts have used the findings to consider (alongside other sources) which groups/sectors are most lacking in AI skills, to potentially inform future skills interventions. The evidence also provides a baseline for AI skills, which can then be tracked in the future. Further, DSIT and other government stakeholders have drawn on the evidence for a range of uses. For example, it was used to support the business case for a pilot AI Upskilling Fund, to demonstrate how DSIT have met their requirement to understand the AI Skills Gap (as set out in the AI Opportunities Action Plan), and is cited in Skills England’s 2025 Skills Need assessment of priority sectors.

Outcome: A more proactive and strategic approach to evidence

There is strong evidence from the evaluation that the programme represents a more proactive and strategic approach to evidence. The route to this benefit is primarily through the existence of the programme which has enabled DCMS to be more proactive and strategic in its approach to generating evidence. Although a wider emphasis across government for a more proactive approach to evidence development and application was reported by consultees, this programme was seen as important for DCMS, particularly because it offers a cross-department mechanism which enables a greater degree of coordination compared to directorate-level budgets (see also Route to impact 1). At the project level, some consultees reported that the programme represented a ‘blueprint’ on researching emerging technology areas and novel methods, offering a model to encourage future similar research within government.

If you think about the really big strategic questions […] those are things that will affect DCMS over the longer term, but it doesn’t really have anything it can do about it in the short to medium term. It’s just on a risk register somewhere. What the R&D programme gives you is an actual lever to go and do something to understand these issues.

– Governance stakeholder

However, at this stage there is no evidence that the programme has contributed to a more proactive and strategic approach to the use of evidence. This likely reflects in part time-paths to impact and the effects of the delays in project completion and publication as discussed previously; this has mitigated the potential for the evidence developed through the programme to be used in a strategic and co-ordinated way to this point. Further, for some consultees, projects in Year 1 were not as strategic as they could have been. This was owing to the way these early projects were selected which was driven by time pressures, with insufficient time for comprehensive horizon-scanning (see Section 2.1.3).

Outcome: Improved decision making in strategic policy areas by delivering a robust evidence base

There is some evidence the programme has helped to improve decision making in strategic policy areas, but the impact has been limited. Some of the funded projects were referred to as having informed or influenced discussions about policy, been used in business cases, and cited in policy submissions to senior government stakeholders. There is also an example of a project which had influenced internal budgets within partner organisations. The impact on decision making is particularly evident for projects which were focused on research topics that were related to short-term policy priorities (e.g. loot boxes, AI as noted above), and therefore more directly applicable. The “Future of TV Distribution” project also demonstrates that when the timing of the research aligns with the policy agenda, the programme has made a contribution in supporting policy making.

Evidence from the case studies: informing decision making

The “Future of TV distribution” project analysed the shift in TV distribution and consumption from digital terrestrial TV (DTT) to internet-delivered services (IPTV), with projections on how UK households will receive television through to 2040 without a policy intervention. It sought to address the need for robust, forward-looking evidence to inform future policy decisions in this area. The research was conducted against the backdrop of increasing policy attention on TV distribution, alongside related work such as DCMS’s 2022 broadcasting review (Up Next) and Ofcom’s 2023-24 Future of TV Distribution assessment, which was also commissioned by DCMS. DCMS used the research to inform discussions on the future of TV distribution. Following publication of the project report, the government convened a stakeholder forum and commissioned further research informed directly by its recommendations.

For the remainder of the portfolio, there is no evidence of a policy impact to date, but there is potential for benefits in the future. This finding partly reflects the nature of the programme (generating research for longer-term priorities) and the associated timescales to impact, with “a long lead time in terms of influencing policy.” However, there is also an example of a project which was topical at the time (the metaverse) but where the policy agenda has since moved on, and so the evidence has not been used. This is not unexpected, and is directly related to the programme’s role in funding longer-term, speculative research which implies a degree of risk in whether the topics may be aligned with future policy agenda (or otherwise).

Outcome: Savings and efficiencies

There is no evidence of the programme leading to savings and efficiencies as a result of better policy decisions linked to the programme’s research. A small number of government stakeholders referred to the potential for savings and efficiencies in the future, through application of the research findings (in the context of outputs making specific recommendations for efficiencies), learning from project delivery (e.g. setting up contracts and designing the project plan in an efficient way), and in some case the use of toolkits and other materials developed through the programme. For example, the “Measuring the Legacy and Impact of Major Events” is expected to support DCMS in the commissioning of future evaluations in this area, and the toolkit is expected to deliver future savings and efficiencies by providing detailed guidance that reduces duplication and removes the need to develop new approaches when commissioning future evaluation work.

There is no evidence of savings and efficiencies having occurred to date, which is not unexpected given time-paths to impacts, but is also directly linked to the limited evidence of improved decision making as a result of the programme’s outputs. It is not clear to what extent it is reasonable to expect savings and efficiencies as a result of better policy decisions, informed by the evidence produced as part of this programme. Further, the likely contribution of the programme to this outcome is likely to be very challenging to observe in practice given the wide range of other factors that will influence costs/service delivery.

Other outcomes

Beyond the logic model outcomes, there is some evidence of outputs leading to wider impact outside DCMS. Many of the reports were published, and some project teams had undertaken dissemination activities to share findings, including workshops and webinars. In relation to wider dissemination, delivery organisations reported: discussing the findings with wider stakeholders (12 of the 15 survey respondents), other research organisations (10) or other government departments/agencies (7); publishing related media outputs such as blogs or articles (9); and delivering formal presentations (6). There are a small number of examples of other public sector bodies using the research, although there was limited detail provided on this. Delivery organisations also referred to the projects helping to shape their own future research priorities (8 of the 15 survey respondents).

The case studies also provide evidence of how the projects may be impacting positively other organisations, which reflects the diversity of activity supported by the Programme in Phase 1.

Evidence from the case studies: other outcomes

The “Natural History Museum AI Pilots” project involved developing and testing new AI-driven solutions in the museums sector, rather than research for the purpose of evidence generation and policy making; it therefore represented a distinctive project within the Phase 1 portfolio. Led by NHM, and working with other museums, the project identified five AI tools that could in principle assist the museums in their research, management of collections, or public engagement. All five tools were successfully progressed to proof-of-concept stage. Three of the tools have been subsequently implemented in practice by the museums involved in the projects. They have variously led to time-saving benefits, improved and/or accelerated digitisation of collections, and improved collection compliance with accessibility standards. There is also continuing interest in developing the AI tools created by the pilots from law enforcement agencies, international researchers, and businesses working on digitisation in the museums sector. Further, one of the pilots is being adapted and used in UKRI’s Distributed System of Scientific Collections UK (DiSSCo UK) project to support its work to digitise the UK’s natural science collections. If implemented, it is expected to generate significant time and cost saving benefits on digitisation of these collections when compared to a manual process.

Was it the programme, rather than other influencing factors, that made the difference to these outcomes (or the decisive difference)?

The programme has played an important role in realising the above outcomes, and in some cases was seen as critical compared to any other contributory factors. Evidence on this aligns with Route 1, with the programme’s contribution underpinned by a reasonably high ‘activity additionality’ which in turn led to a high level of ‘outcome additionality’ (although it is possible that the projects closest to policy priorities may have been funded anyway). An additional point supporting the importance of the programme is that even if some projects would have been funded via other means, the added value of the programme is the ability to shape and steer projects in a way that ensures the usability of outputs for DCMS’s purposes (linked to the outcome relating to better alignment within Route 1).

Similar to Route 1, other factors that have contributed to the outcomes (in addition to the programme) include those influencing buy-in and interest in the research, within DCMS and more widely: technological advancements in relevant areas; government priorities and political interest; and wider societal interest. This has led to ‘validation’ for the programme in terms of its role in building the evidence base, but is also important because the use of evidence depends on whether it is aligned with the wider agenda/priorities, and wider efforts to encourage a cross-government culture change in relation to evidence-based decisions.

What are the factors of programme design that have delivered this contribution?

The key aspects of the programme that have contributed positively relate to consistency in programme delivery and buy-in across the DCMS/DSIT (similar to Route 1). Buy-in was particularly important for the evidence being used - as one government stakeholder put it: “Everyone’s too busy. If it’s not something they can see a benefit from for them, they won’t engage with it.” Other factors positively influencing project delivery included close working relationships between project teams and delivery organisations, effective communications, and alignment of expectations (for full findings on what worked well, see Section 2.2). In particular, close working relationships between government teams and delivery organisations helped to align projects to government priorities (which had a positive influence on buy-in) and alignment of expectations helped to make sure the outputs were valuable and actionable (which meant the evidence was more likely to be used).

There are two main hindering factors:

• In some projects, there were considerable delays between project completion and publication. These were considered to have resulted from publishing restrictions due to change of government, as well as the time required to ensure compliance with accessibility requirements. These delays meant a longer gap between the evidence being produced and it being ready for use, with a risk that technology/policy agenda had moved on in the meantime, which in some cases meant lower interest and applicability of findings to policy discussions.

Final publication of the research was a long while after […] so I felt at the point of publication no one was really interested.

– Delivery organisation

• Dissemination activities, within the Departments and more broadly, were limited. This represents a missed opportunity because evidence can only be used by those aware of it, and is linked to the finding that the value of the programme to date has been primarily from DCMS (and DSIT) ‘holding’ rather than ‘using’ the evidence. Related to the points about buy-in above, it was highlighted that engagement with the evidence is more likely if the findings are pitched at the right level and, where possible, tailored to different audiences (e.g. targeting specific individuals who may be interested, and identifying key findings that would be of particular relevance to them - see also the recommendation that follows).

Who is aware of the output and what do they need it for? What are the next steps for helping to continue the conversation? That is the only thing not effective to me - [the project outputs] just went into a black hole.

– Delivery organisation

Recommendation 10: Ensure a greater focus on processes and activities for raising awareness of the programme and disseminating the learning, both of research deliverables, and the skills and expertise developed by those directly involved at programme- and project levels.

• At the programme level, the management team should scale-up outreach and engagement activities to raise awareness of the programme across DCMS, and provide clarity on programme purpose/scope. Other opportunities linked to more effective dissemination should also be considered actively. These might include: ‘easy wins’ (e.g. making clearer references to the programme when talking about the findings), and more ‘strategic’ activities (e.g. building an internal database/tool to help navigate the research that has been commissioned and produced, leveraging the networks of the College of Experts).
• At the project level, DCMS should ensure that each project: (i) has a dedicated and ringfenced minimum budget threshold for dissemination activities, both external and associated with knowledge exchange within the Department; (ii) has a detailed dissemination plan in place; and (iii) that this activity is delivered in full to the level and scope expected. DCMS should also ensure that they track delivery against the dissemination plans for all projects.
• In delivering against this, DCMS should consider developing a programme-level framework that captures the timing, progress in delivery and coordination of planned dissemination activities. This framework could also be used to inform monitoring, and the identification of opportunities to maximise the impact of dissemination, including both cross-project and links to external relevant activities.

• The project level dissemination plans should include the following:

  • responsibility for dissemination, e.g. a lead contact from government teams, with inputs from Programme Governance Board and College of Experts
  • key considerations for dissemination, e.g. format and accessibility of deliverables, timings (and mitigation of any risks which may delay publication)
  • timing of dissemination, e.g. initial thinking on potential relevance to DCMS teams (and wider stakeholders) specified at the outset, to be reviewed at set points during projects delivery, with dissemination activities after project close
  • specific activities for internal dissemination, e.g. events/workshops to share findings and other learning (DCMS-wide or targeting specific teams), direct approaches
  • specific activities for external dissemination, e.g. presentations and conferences and other forums, think pieces, blog posts, etc.

4. Conclusions and recommendations

4.1 Evaluation conclusions

The DCMS R&D Science and Analysis Programme was introduced in late-2022 in response to a perceived gap in the Department’s ability to fund long-term strategic evidence. The programme was intended to complement existing research arrangements, providing an opportunity to increase departmental confidence with innovative research methods while addressing evidence gaps in strategic evidence, particularly in nascent policy areas or topics, thus reducing risks associated with over-reliance on responsive research on immediate policy priorities and agendas.

Soon after the launch of the three-year Phase 1 of the programme, MoG changes in early-2023 led to a change in the policy responsibilities of DCMS and the creation of DSIT, a brand new department with no established analysis function. This resulted in Phase 1 being split with DSIT, with both subsequently involved in selecting projects and analysts and policy officials from both involved in research project oversight. This was an arrangement primarily driven by necessity, and represented a challenge for the programme. Considerable management resource was spent on navigating the implications of MoG changes for the programme, which was managed well overall. However, this inevitably had implications for other aspects of delivery given limited programme management resource, which itself presented a key risk and limitation to delivery.

Against this backdrop, the programme continued to work towards embedding itself within the two Departments as a new funding mechanism. The programme evolved in response to learning from ongoing delivery. For example, changes were made to project selection processes in the second and third year in order to enhance buy-in across government teams. This demonstrated positively a process of effective adaptation and improvement, but it also meant that the processes were not consistent. This may have led to a lack of clarity amongst policy and analysis teams of the programme’s rationale across the Departments.

Programme delivery was anchored by its governance, which helped to ensure accountability. There was value in the programme being led by the CSA Office, bringing credibility and ensuring a strategic view across DCMS. The Programme Governance Board provided effective oversight. However, in Phase 2 there are opportunities to more clearly articulate decision-making processes and the remit of the Board, and to more fully leverage the Board to provide strategic direction, and potentially the DCMS College of Experts. It is also important for those involved in governance to consider opportunities for further programme promotion to ensure a shared understanding of the programme across the Department, both in terms of its purpose and the evidence that has been generated.

Phase 1 of the programme consisted of the funding of 22 research projects. The approach to identifying research areas varied in each year, with the most effective approaches combining both ‘top-down’ engagement (to ensure strategic alignment and senior buy-in) and ‘bottom-up’ engagement (to ensure buy-in at operational levels). Projects were funded via two different commercial arrangements, MoUs and tendering, which provided flexibility. The appropriate route depends on the context of the project, and this should be considered in Phase 2 funding decisions.

The programme funded a higher number of smaller projects than originally intended. Whilst this expanded the breadth of research areas covered, there are some concerns this was perceived to be due to reluctance by officials to lead on larger and higher-risk projects, which is directly related to an observation that a small number of the projects funded may have been funded anyway via other mechanisms. In Phase 2, the programme team should be clearer on the intended balance within the portfolio, guided by a strategic approach which considers risk tolerance thresholds. 4.7 The delivery model for each project was bespoke, including in terms of the number of individuals involved, the intensity and nature of this involvement, and the mix of analytical and policy officials. This reflected the scope, subject matter, and research approach of each project, and also different levels of interest, engagement and capacity of analytical and policy officials.

Across this diversity, the evaluation evidence indicates that project delivery generally worked well, with positive feedback on the role of both the external research teams (leading delivery) and the government teams (providing inputs). Although lessons on ‘what works’ are context-specific, common themes of key success factors included: clarity in responsibilities; alignment of expectations; consistency of inputs from government teams; and regular communications. In projects where these were not evident, there were typically some adverse effects on delivery including the timeliness and/or quality of outputs.

The funded projects have generated a breadth of evidence across a range of topics, in different formats, and for different audiences. This is a key success for the programme, aligned with its core objectives. Two main routes to impact were apparent, in terms of outcomes associated with:

• Generating evidence i.e. capability to undertake R&D activity effectively and efficiently: There is evidence of benefits relating to reinforcing processes of knowledge spillovers, upskilling of analysts (for example in advanced research techniques), and improved capability/confidence. The programme has also led to a better alignment of external research and government needs. Outcomes related to knowledge, skills and confidence/capability were largely limited to those directly involved in projects, with limited evidence that these benefits have been diffused more widely across the participating departments. It is not unexpected that outcomes are more evident for those more closely involved, however, this does represent a missed opportunity for greater dissemination of knowledge and capability. Further, at this stage, there is very limited evidence of the programme leading to savings/efficiencies or a reduced need to outsource research. Whilst there is potential for these outcomes in the longer term (via capability building), there will still be value in DCMS commissioning external, independent research.
• Holding and using evidence i.e. capability to exploit and apply the R&D outputs: The programme represented a more proactive and strategic approach to evidence within DCMS, and has generated robust evidence which can be drawn on. This is a valuable resource that can be used to respond to policy priorities as they emerge. The evaluation identified limited evidence that the research at this stage has directly informed or improved decision making in strategic policy areas, although there are examples of specific contributions to policy discussions from across the project portfolio, which is encouraging. For this route to impact the programme’s value to date has therefore been primarily in relation to enabling DCMS and DSIT to ‘hold’ rather than ‘use’ the evidence, which reflects the focus of the programme on strategic rather than policy-specific research, and the associated timescales to impact.

These findings have two associated considerations. First, they highlight the importance of effective dissemination, as evidence can only be used by those aware of it, and the imperative to ensure mechanisms are in place to raise awareness of the evidence base generated; this should be a priority going forward. Second, they illustrate the risk of wider developments and agendas outpacing projects, which can limit the value of the evidence generated. Notably, in Phase 1 in some cases, research was less relevant at the point of completion than launch owing to changes in the external environment. This is an inherent risk for strategic and longer-term research, including related to technology change, which needs to be embraced and managed in programme direction and delivery. Leveraging fully learning in relation to methods and techniques, and identifying potential connections and applicability to adjacent areas/themes is also important where agendas may have moved on; this should be a consideration for Phase 2.

Positively, Phase 1 of the programme has also generated benefits beyond its stated objectives: reputational benefits for DCMS and its status as an R&D active department; new and stronger links with the research community; and wider impacts outside DCMS through the application of research findings.

The programme has played an important role in realising these outcomes compared to any other contributory factors, underpinned by the reasonably high activity additionality, i.e. the programme is filling a funding gap. However, a small number of projects may have been funded anyway. This indicates a trade-off for the programme to balance: evidence of the research being used is stronger for projects linked to shorter-term priorities, but for these projects additionality is lower. In Phase 2, it will be important to reflect on and more clearly set out what the programme is looking to achieve in this context - as reflected in the theory of change - and how it can therefore ensure additionality across the portfolio through project selection and delivery aligned strongly to the underpinning rationale and objectives.

Further, it will be important that DCMS continues to seek to capture evidence on the use and application of the research from Phase 1 over the longer-term to provide fuller understanding of its contribution. This reflects both the timing of the completion and publication of some of the projects - with some delays in publication experienced - and the role of the programme in generating strategic and long-term evidence that can be drawn on as priorities and policy agendas emerge. The full contribution of Phase 1 may not be evident for some time. 4.14 Phase 1 of the programme has provided a good grounding for progress towards achieving a ‘step change’ in how DCMS approaches its evidence base. Phase 2 should leverage and build on the progress made to date, informed by the list of recommendations provided in the next sub-section.

4.2 Recommendations for Phase 2 of the programme

The bullets below presents ten recommendations for Phase 2 of the programme, informed by the evaluation findings. These recommendations have been introduced individually in the relevant sections throughout the report, and are listed here in the order that these are mentioned in Sections 2 and 3 of the report (not in the order of priority). We understand that some of these recommendations may reflect changes that are already being considered or have been made for Phase 2.

Recommendations for future delivery

1. Review the remit, role and composition of the Programme Governance Board and ensure that this is clearly articulated and communicated to those involved in the programme. This is an opportune moment for reconsidering the structure and balance within the Board, given that some changes are already anticipated with DSIT not involved in Phase 2. The review should consider whether the Programme Governance Board could be leveraged more fully for decision making, and to provide strategic direction, and support awareness raising and dissemination.

2. Consider opportunities to leverage the expertise of the College of Experts to support programme delivery and impact. Potential roles may include: the provision of strategic advice to the Board and DCMS on the scope/prioritisation of research topics and project selection; expert review of deliverables; and support with the dissemination of findings. Any potential role for the College of Experts should take into account capacity and potential conflicts of interest for this group, and their relative contribution to other departmental priorities and activities.

3. Ensure an appropriate level of resource is allocated for programme management, indicatively 2.5-3 FTE. The resource should reflect the scale of the programme and pro-actively consider and mitigate risks around staff turnover, capacity and continuity in project-level management and provider engagement.

4. Review and update the programme theory of change, focusing on clarity of rationale, aim and objectives, and pathways to impact (including timing and influencing factors). The updated theory of change should inform strategic planning around project identification and selection, and monitoring and evaluation.
   
   In reviewing and updating the theory of change, DCMS should consider the ongoing relevance and applicability of the overarching aim to deliver a ‘step change’ in how DCMS approaches its evidence base. If this aim is considered to be appropriate for Phase 2, this should be defined to provide clarity of purpose and intent.

5. Building on the updated theory of change, develop a ‘strategic portfolio proposition’ statement for the programme, providing clarity on the intended composition and balance within the portfolio (and key considerations informing this) to inform project identification, prioritisation and selection.
   
   This should include consideration of:

   • Risk tolerance thresholds for larger projects, with explicit reference to the risk of projects being funded which lead to inconclusive findings or are outpaced by technological developments and/or changes in the policy agenda

   • Barriers to and incentives for ensuring higher risk projects are identified and funded, including practical implications (e.g. the length/coverage of scoping activity for longer-term/higher-risk projects, government team expectations and requirements).

6. Introduce a more consistent, structured and transparent approach to project selection across the programme period, aligned to the ‘strategic portfolio proposition statement’.
   
   This should continue to involve prioritisation of research topics via a combination of ‘top down’ and ‘bottom up’ approaches, alongside:

   • Formally defined work flows i.e. articulation of a consistent process for project selection including key stages, progress and decision-points

   • Eligibility criteria i.e. what is and is not considered to be in scope for the programme, including a definition of in-scope R&D activity, research/topic area coverage

   • Minimum requirements i.e. any essential elements or characteristics that need to be included in all projects (e.g. specified research questions, clarity on alignment/relevance to departmental research priorities, use of novel or innovative methods or data-sets)

   • Clear spheres of responsibility within government, recognising different levels of input and responsibility (notably the Portfolio Programme Manager, SROs and Programme Board).

   • Changes to the project selection processes during the programme (i.e. between years) should be minimised as far as practical, to provide continuity and certainty in programme management, accountability and communications.

7. Continue to offer the two different routes to commissioning (through MoU and tendering) to provide flexibility, reflecting on the most appropriate route on a project-by-project basis (informed by trade-offs around timescales, flexibility/control, etc). Where the MoU route is chosen in projects that have been identified by partners, the programme management team should consider ways to ensure good engagement and buy-in from government teams.

8. Take further steps to mitigate key risks around project delivery, informed by the evaluation evidence on ‘what works. This should build on programme management to date, with a focus on ensuring that the following aspects are consistently experienced across the portfolio:

   • Alignment of expectations: project plans should clearly establish the responsibilities/ ownership, timings and requirements for deliverables (including scope, tone, intended audiences, etc).

   • Project progress and relevance considerations in the context of technological developments and policy priorities: appropriate interim milestones and reflection points should be incorporated into project plans, particularly for longer-term projects.

   • Consistency of inputs from government teams: addressing barriers relating to both interest (ensuring that the relevance of the research and/or value in developing new skills/knowledge is clearly tested, defined and articulated at the outset in discussion with team members) and capacity (assigning a ‘lead’ contact for each project with responsibility for DCMS-side project management, involving more colleagues as contingency, with clearly established processes for handover; this would also help to free up programme management resource).

9. Improve processes to ensure robust data collection, monitoring and evaluation.
   
   The evaluation of Phase 1 of the programme highlighted a particular gap in consistently recording project outputs (across the three categories from theory of change). This should be addressed as a priority to support future evaluation of the programme.
   
   DCMS should also consider developing a programme monitoring and evaluation framework, to ensure there is a structured plan for assessing performance against the intended outcomes and impacts (aligned with the updated theory of change). This should inform ongoing evidence collection of outcomes, e.g. by asking for feedback at project completion from Government teams and delivery organisations, and establishing time-points for follow-up primary and secondary research (where appropriate). As part of this framework, the programme team should also put in place a proportionate mechanism to track ongoing evidence on the use of the Phase 1 research findings by the Department.

10. Ensure a greater focus on processes and activities for raising awareness of the programme and disseminating the learning, both of research deliverables, and the skills and expertise developed by those directly involved at programme- and project levels.

• At the programme level, the management team should scale-up outreach and engagement activities to raise awareness of the programme across DCMS, and provide clarity on programme purpose/scope. Other opportunities linked to more effective dissemination should also be considered actively. These might include: ‘easy wins’ (e.g. making clearer references to the programme when talking about the findings), and more ‘strategic’ activities (e.g. building an internal database/tool to help navigate the research that has been commissioned and produced, leveraging the networks of the College of Experts).

• At the project level, DCMS should ensure that each project: (i) has a dedicated and ringfenced minimum budget threshold for dissemination activities, both external and associated with knowledge exchange within the Department; (ii) has a detailed dissemination plan in place; and (iii) that this activity is delivered in full to the level and scope expected. DCMS should also ensure that they track delivery against the dissemination plans for all projects.

• In delivering against this, DCMS should consider developing a programme-level framework that captures the timing, progress in delivery and coordination of planned dissemination activities. This framework could also be used to inform monitoring, and the identification of opportunities to maximise the impact of dissemination, including both cross-project and links to external relevant activities.

• The project-level dissemination plans should include the following:

  • responsibility for dissemination, e.g. a lead contact from government teams, with inputs from Programme Governance Board and College of Experts

  • key considerations for dissemination, e.g. format, accessibility and timings of deliverables (and mitigation of any risks which may delay publication)

  • timing of dissemination, e.g. initial thinking on potential relevance to DCMS teams (and wider stakeholders) specified at the outset, to be reviewed at set points during projects delivery, with dissemination activities after project close

  • specific activities for internal dissemination, e.g. events/workshops to share findings and other learning (DCMS-wide or targeting specific teams), direct approaches

  • specific activities for external dissemination, e.g. presentations and conferences and other forums, think pieces, blog posts, etc.

Annex A: Further detail on evaluation approach

Research questions

The research questions for the evaluation are listed in Table A-1 (process evaluation) and Table A-2 (impact evaluation).

Table A-1: Research questions for process evaluation

No.	Research Question	Sub-questions (if relevant)
1	How effective were the processes implemented by suppliers/research teams to deliver the research projects?	Did research teams have the right skills and expertise? Did research teams have the right processes in place, e.g. Quality Assurance, and were these followed sufficiently? Did research teams communicate sufficiently amongst themselves and with the client? Were they well-coordinated? Did research teams input consistently across the lifespan of the projects they were involved in?
2	How effective were the processes implemented by Government teams to manage the delivery of the research projects?	Did Government teams have the right skills and expertise? Did Government teams have the right processes in place (e.g. peer review of outputs), and were these followed sufficiently? Did Government teams input consistently across the lifespan of the projects they were involved in?
3	How effective was the overarching governance and approach to the programme?	Were the selected research projects (and research questions) the best ones to choose in order to achieve the objectives of the programme? Were the approaches to prioritising research projects effective and which was the most effective? What were the relative advantages and disadvantages of working with UKRI vs through Invitations to Tender? Did the programme have adequate resourcing? Did those involved in the management and delivery of the programme have the right skills and expertise? Was the programme sufficiently integrated into the wider DCMS and DSIT ecosystem? Was the programme sufficiently accountable to the rest of the department? Were the programme’s accountability arrangements adequate for realising the programme’s objectives? Were existing Commercial arrangements adequate for achieving the programme’s objectives? E.g., were Invitations to Tender through existing frameworks and collaboration with UKRI the best routes forward for the programme?
4	What recommendations might be made for the second phase of the programme in light of the process evaluation?

Source: SQW based on DCMS requirements

Table A-2: Research questions for impact evaluation

No.	Research Question	Sub-questions (if relevant)
1	To what extent has the first phase of the programme delivered its intended outcomes?	Key outcomes of interest include: (i) an overall step change in how DCMS approaches its evidence base; (ii) improved decision making in strategic policy areas by delivering a robust evidence base; (iii) savings and efficiencies through internal capability building, and enhanced value from external contractors; (iv) knowledge transfers and spillover effects between Government and subject matter experts by involving them in early stages of evidence-based development; (v) initiating a department-wide culture change, through building the department’s familiarity and confidence with new methods and innovative experimentation.
2	How do the outcomes from the programme compare to outcomes from other routes for commissioning and delivering evidence for Government, e.g. did the programme ultimately offer anything demonstratively distinct from other modes of evidence development? If the programme did not exist, is it likely that the evidence outcomes would have been achieved anyway?
3	Has the programme achieved any benefits beyond its stated objectives?
4	What recommendations might be made for the second phase of the programme in light of the impact evaluation?

Source: SQW based on DCMS requirements

Thematic analysis

The evaluation used thematic analysis for assessing both the effectiveness of delivery and implementation of the programme (i.e. the process evaluation) and the impact of the programme (i.e. the impact evaluation). This approach was used for systematically analysing qualitative data across the two waves of primary research, enabling triangulation of feedback from governance stakeholders, government teams and delivery organisations. Specialist software, MaxQDA, was used to identify, code and collate the themes within the data. A coding framework was developed for analysis, aligned to the research questions, which was tested and further refined during analysis. This was used consistently across different sources of data to enable cross-read. The coding framework was applied to interview notes, with segments of interviews coded to the relevant codes in the framework. The coded segments were then exported to Microsoft Excel for analysis and reporting.

Contribution analysis

Contribution analysis is a theory-based evaluation approach that involves gathering evidence to understand the likelihood that an intervention has contributed to an outcome observed (or not), especially its importance in generating outcomes relative to other factors. It uses a logical, step-by-step process to explore how the intervention’s contribution would have come about (developing a ‘contribution story’) and uses a broad range of evidence to test this. Contribution analysis can also draw out learning on how and why a programme is generating change across its activities and engaged groups. It is particularly useful for interventions where (a) it is not possible to establish an experimental design to test cause and effect, and (b) it is known that outcomes have likely been achieved through multiple factors.

Contribution analysis is guided by the following four principles:

• Considering the relative role of an intervention, and how it works alongside other activities
• Considering explicitly other explanatory factors that may explain outcomes
• Seeking an evidenced, systematic and logical line of reasoning on the extent to which an intervention has contributed to outcomes
• Using multiple sources of evidence, and triangulation across these to strengthen findings.

The evaluation used contribution analysis as the overarching framework for assessing the impact of the programme (i.e. the impact evaluation). Two main routes to impact were identified, informed by the programme theory of change and evidence from primary research. For each route to impact, a ‘contribution story’ was developed by assessing evidence against the following questions:

• Is there a reasoned theory of change? Have activities been implemented as set out in the theory of change?
• Is there evidence that the expected outcomes have occurred (or are expected to)?
• Was it the programme, rather than other influencing factors, that made the difference to these outcomes (or the decisive difference)?
• What are the factors of programme design that have delivered this contribution?

For each outcome from the programme theory of change, a summary assessment was provided. These assessments were informed by the nature and strength of the underpinning evidence, ranging from ‘achieved’ and ‘partially achieved’ to ‘not achieved’.

Our approach to contribution analysis was iterative, with evidence triangulated across two ‘waves’ of research:

• An initial round of evidence collection, comprising consultations and an online survey, informed the development of draft contribution stories which were discussed with the client group at DCMS. The first wave of research also helped to identify some key themes and issues which should be explored further.
• A second wave of evidence collection was then completed, including further consultations and case studies. The draft contribution stories were reflected on in light of the new evidence, leading to the development of the final contribution stories.
• The final contribution stories were tested in a challenge workshop with senior staff at DCMS. This provided feedback on the findings, with comments reflected in the final report.

Fieldwork sampling

The sampling for fieldwork was undertaken in two phases. DCMS provided a longlist of contacts involved in the governance and management of the programme, and in the delivery of the funded projects (at DCMS/DSIT and research organisations). From this list, we identified and agreed with DCMS a set of priority contacts, based on their level of involvement and coverage across projects. These individuals were contacted for the first phase of fieldwork.

Sampling for the second phase of fieldwork was informed by the emerging findings from the first phase, in terms of:

• Identifying opportunities to fill any outstanding evidence gaps
• Including any additional stakeholders suggested by the individuals consulted
• Informing the selection of impact tracing case studies which intentionally focused on the projects with emerging evidence of outcomes being achieved, in order to illustrate the routes to impact.

Annex B: Projects funded by the programme

Table B-1 provides the key details on the 22 funded projects.

Table B-1: Details of funded projects

No.	Project name	Year	Lead department	Commissioning route	Project description
1	AI Skills Phase 1	2022/23	DSIT	MoU (UKRI)	This project utilised interdisciplinary methodologies - including systematic literature reviews, international policy comparisons, and multi-stakeholder future-visioning workshops - to provide evidence-based recommendations for AI regulation, skills development, and governance in the UK.
2	Responsible AI Phase 1	2022/23	DSIT	MoU (UKRI)	A two-part programme comprising a multi-work package study by the University of Edinburgh and the Ada Lovelace Institute, which employed Policy Fellows to produce research on AI risks and regulation lessons from other sectors, and a Policy Lab and skills workstream by the Trustworthy Autonomous Systems Hub, both commissioned to inform the UK’s AI Strategy and Risk Framework.
3	Shaping the Metaverse	2022/23	DCMS & DSIT	MoU (UKRI)	A six-month multidisciplinary Horizon Scan commissioned by DCMS and DSIT that used a combination of regional and international workshops, focus groups, and interviews with UK and international academic experts to explore the future implications of metaverse technologies and inform UK policy.
4	ICT Carbon Emissions	2022/23	DCMS & DSIT	MoU (UKRI)	This project examined UK ICT’s carbon footprint and limitations of accurate assessments, conducted a set of stakeholder interviews to better understand drivers of growth in ICT and how its footprint is currently understood by those in video streaming and ICT infrastructure related industries, and developed a test case model to analyse ICT embedded in the context of physical sporting events.
5	Future Observatory Policy Fellowships	2022/23	DCMS	MoU (UKRI)	This project employed six Ecosystem Fellowships to map design ecosystems across sectors like fashion, digital, and architecture to understand sustainability best practices and inform government policy.
6	Responsible AI Phase 2	2023/24	DSIT	MoU (UKRI)	The project delivered a set of reports and events designed as ‘deep dives’ into applications of GAI/foundation model and AGI risks. These included, in particular, questions about the societal risks arising from the use of GAI in the journalistic context, different approaches to the evaluation of foundation models, and comparisons between AI and regulation in other high-tech sectors.
7	Shaping the Metaverse Phase 2	2023/24	DSIT	MoU (UKRI)	A deep dive into three areas of interest identified in the first phase project: (i) metaverse standardisation and interoperability, (ii) analysis of the privacy and security implication of metaverse technologies, and (iii) a state-of-the-art on Distributed Ledger Technology use cases and their likely impact
8	Measuring the Legacy and Impact of Major Events	2023/24	DCMS	Tendering	This project used four past event case studies (including London 2012 and the UK City of Culture Programme) to test robust counterfactual evaluation methods like Difference-in-Difference and Propensity Score Matching for developing better guidance on assessing the long-term socio-economic legacy of major events in the UK.
9	Capturing Engagement Numbers with Unticketed Events	2023/24	DCMS	Tendering	This project developed and tested multi-source data-driven methodologies - including mobile app, activity, social media, and aerial photography data - over two strands of research to improve the accuracy of attendance and participation estimates for unticketed cultural and sporting events.
10	AI Skills for Life and Work	2023/24	DSIT	Tendering	This project sought to identify the essential AI skills needed currently and in the future for working and living with AI technology, using a multi-faceted methodology that included an expert Delphi study, a rapid evidence review, quantitative skills forecasting, and a general public survey.
11	AI-Assisted vs human-only evidence review	2023/24	DSIT	Tendering	A comparative trial conducted in early 2024 by the Behavioural Insights Team (BIT) for DCMS and DSIT, which used ‘AI-assisted’ (leveraging tools like ChatGPT 4, Claude 2, Elicit, and Consensus) and ‘human-only’ methods to produce two rapid evidence reviews on the topic “How technology diffusion impacts UK growth and productivity” to evaluate AI’s impact on review quality and speed.
12	Digital Sector Emissions	2023/24	DSIT	Tendering	This project intended to quantify greenhouse gas emissions from digital technologies globally, project their likely trajectories to 2030, and evaluate the effectiveness of international policies and Big Tech’s voluntary initiatives to inform UK policy on achieving Net Zero.
13	Future of TV Distribution	2023/24	DCMS	Tendering	This project examined how television consumption trends are likely to evolve in the UK up to 2040 by drawing on a methodology that included academic papers, industry studies, audience data, and new interviews to analyse audience, technology, market, and net zero impacts.
14	Local Civil Society Infrastructure	2023/24	DCMS	Tendering	A multi-phased research programme using a mixed-methods approach, including a literature review, secondary data analysis, surveys, and in-depth qualitative case studies, to build an evidence base on the impacts and effective models for strengthening local civil society infrastructure in England.
15	Barriers and Enablers of Tech Adoption	2024/25	DSIT	Tendering	This project was a qualitative, iterative study that explored the drivers of business adoption decisions through a literature review and over 80 interviews with businesses and industry representatives.
16	Economic Modelling of Cyber Systemic Incidents	2024/25	DSIT	Tendering	This project utilised a multi-faceted methodology - including a systematic literature review, impact mapping, and bespoke economic modelling - to quantify the potential economic harm of systemic cyber attacks on the UK’s gas and rail networks.
17	Economic Modelling of Intellectual Property and Knowledge Assets Theft as a result of Cyber Attacks	2024/25	DSIT	Tendering	The project utilised a multi-step methodology - including a systematic literature review, econometric analysis of UK firm-level data (using Advisen and FAME datasets), and qualitative case studies - to estimate the economic impact of intellectual property and knowledge assets theft resulting from cyber attacks on UK businesses and the broader economy.
18	Economic Modelling Of Impact on Consumers of Cyber Attacks	2024/25	DSIT	Tendering	This project utilised a mixed-methods approach - including a systematic literature review, impact mapping, and a bespoke consumer survey - to develop methodologies for quantifying the economic harm to UK consumers resulting from a loss of access to essential goods and services across five key sectors following a cyber attack.
19	Economic Modelling of Sector Specific Costings of Cyber Attacks	2024/25	DSIT	Tendering	This project utilised a multi-faceted methodology - including a systematic literature review, analysis of existing databases (e.g., the Cyber Security Breaches Survey and KPMG’s Industry Insights Database), and expert workshops - to develop an economic model that quantifies the sector-specific costs of cyber attacks for UK businesses by firm size and attack type.
20	Understanding Player Experiences of Loot Box Protections	2024/25	DCMS	Tendering	This project assessed the effectiveness of the 2023 industry-led protections for loot boxes in videogames, using large scale surveys of children and parents, and a number of qualitative interviews.
21	Culture Connect Data Observatory	2024/25	DCMS	MoU (UKRI)	A pilot for an open-access data observatory that brings together cultural data, e.g. funding data, cultural participation data, from the Newcastle region.
22	Natural History Museum AI Pilots	2024/25	DCMS	MoU (Natural History Museum)	This project involved running five six-month pilot research projects in collaboration with partner institutions to test the application of various AI/machine learning models (including Large Language Models, Natural Language Processing, and computer vision) to drive productivity and develop sector-wide AI capability in the museums and galleries sector.

Source: DCMS

Table B-2 shows the total and average value of projects by year the projects were awarded.

Table B-2: Project value per year

Year	Total project value	Average project value
2022/23	£1,432,000	£286,400
2023/24	£4,320,044	£480,005
2024/25	£1,644,542	£205,568

Source: SQW based on DCMS data

Figure B-1 shows the value of projects across the portfolio.

Figure B-1: Distribution of project value across portfolio

(A graph of the distribution of project value across portfolio.)

Source: SQW based on DCMS data

Annex C: Delivery organisation survey responses

The tables below provide a breakdown of responses to the delivery organisation survey. Questions with open responses have been excluded.

Table C-1: How was your research project(s) commissioned?

Commissioning route	Number of responses
Tendering	9
MoU	5
Don’t know	1

Source: SQW analysis of delivery organisation survey (n=15)

Table C-2: Before being commissioned to do this project(s), had your organisation previously delivered any other research funded or supported by DCMS or DSIT (including predecessor departments)?

	Number of responses
Yes	8
No	3
Don’t know	4

Source: SQW analysis of delivery organisation survey (n=15)

Table C-3: Thinking about the level of involvement of the Government team(s) at the following stages of project delivery, would you say this was…?

	Planning the research	Delivering the research	Quality assuring the research
Too much involvement	0	0	1
About the right level of involvement	13	13	12
Too little involvement	2	2	2

Source: SQW analysis of delivery organisation survey (n=15)

Table C-4: To what extent do you agree or disagree with the following statements?

	Strongly agree	Agree	Neither agree nor disagree	Disagree	Strongly disagree
The research was novel and experimental	4	8	0	3	0
The budget for delivering the research was sufficient	2	12	0	1	0
The timescale for delivering the research was sufficient	0	8	2	4	1
Expectations for the project(s) were clearly communicated	3	9	2	1	0
Government team(s) had the right analytical skills and expertise for the project(s)	6	6	3	0	0
Government team(s) ensured appropriate policy advice and guidance to inform the project(s)	3	7	2	3	0
Government team(s) had the right processes in place to manage their inputs (e.g. peer review of outputs)	0	10	3	2	0
Government team(s) provided timely feedback on draft outputs / deliverables	2	10	2	1	0
Government team(s) provided comprehensive and appropriate feedback on draft outputs / deliverables	3	9	1	2	0

Source: SQW analysis of delivery organisation survey (n=15)

Table C-5: What impact has delivering this research project(s) had on your organisation to date (or is expected to in the future)? Please select all that apply.

Impact achieved	Yes	No
Improved staff skills or expertise	11	4
Enhanced research/technical capability or capacity	12	3
Increased organisational profile or credibility	9	6
Strengthened relationship with Government	13	2
Improved understanding of government research needs and priorities	9	6
Generated new or strengthened relationships with other research organisations	9	6
Shaped future research priorities	8	7

Source: SQW analysis of delivery organisation survey (n=15)

Table C-6: Following the completion of the research project(s), have you done any of the following? Please select all that apply.

Action taken following research project completion	Yes	No
Discussed the findings with other Government departments/agencies	7	8
Discussed the findings with other research organisations (including academia and private sector)	10	5
Discussed the findings with wider stakeholders (e.g. businesses, voluntary and community organisations)	12	3
Delivered formal presentations (e.g. at conferences, webinars, workshops)	6	9
Published related media outputs (e.g. blogs, articles)	9	6

Source: SQW analysis of delivery organisation survey (n = 15)

---

1. DCMS Monitoring and Evaluation Strategy 2026 to 2031 ↩

2. Contribution analysis is a theory-based evaluation approach that involves gathering evidence to understand the likelihood that an intervention has contributed to an outcome observed (or not), especially its importance in generating outcomes relative to other factors. It uses a logical, step-by-step process to explore how the intervention’s contribution would have come about (developing a ‘contribution story’) and uses a broad range of evidence to test this. ↩

3. The distinction between ‘high’ and ‘lower’ involvement was identified by the programme team. ↩

4. Including three external advisers supporting government teams. ↩

5. Including document review, interviews with delivery organisations, Government teams and any wider stakeholders (separate from those listed above). ↩

6. The role of the CSA Office is to deliver informed scientific and technical advice and challenge, increase access to expertise and promote stronger links with the science and research community. The team ensures the delivery of the department’s vision for science and its core objectives (that research and evidence is embedded into the culture at DCMS, and that the department is well supported in its policy and decision making through enhanced internal capability and external engagement). ↩

7. The ARIs outline the key research questions and needs facing the department, with the aim of aligning scientific and research evidence with policy development and decision-making. ↩

8. With the Arts and Humanities Research Council (AHRC), the Engineering and Physical Sciences Research Council (EPSRC) and the Natural History Museum (NHM). ↩

9. RM6126 Research & Insights. ↩

10. The Business Case expected 3-5 projects in Year 1 and 6-10 in Years 2 to 3, vs 22 projects funded. ↩

11. The project teams were invited to bid for an extension using a bidding template. These bids were reviewed by Government (including the SROs) and feedback was provided. Bids were refined in response to feedback, leading to a positive funding decision. ↩

12. The individual respondents reported ‘too little involvement’ in either one stage (four respondents) or two stages (one respondent), which varied with no clear pattern. No respondents indicated this across all three stages. ↩

13. The remainder were published by DSIT or the delivery organisations. ↩

14. Time series data generally indicate a large spike in views close to publication, with much lower (but relatively consistent) levels of engagement thereafter. Correlation factor between views and time since publication is low at 0.35, which is likely to be affected by the limited number of observations. ↩