Research and analysis

Evaluation of the Cyber Security Academic Startup Accelerator

Published 4 August 2023

1. Executive summary

The Cyber Security Academic Start-up Accelerator (CyberASAP) programme is funded by the Department for Digital, Culture, Media and Sport (DCMS) and delivered by the Knowledge Transfer Network (KTN) ^{[footnote 1]}. It supports the commercialisation of UK cyber security research and helps academic researchers to turn ideas into fully rolled-out commercial projects by developing the academics’ entrepreneurial skills. It has two phases as set out in Figure 1:

Figure 1: CyberASAP overview

This evaluation report relates to the CyberASAP programme which was initially piloted in 2017/18 and has been delivered over five years to 2021/22, with one cohort of researchers each year. ^{[footnote 2]}

Conclusions and recommendations against each of the core evaluation questions are outlined below.

1.1 Process evaluation

Was the programme delivered as intended?

The programme has evolved each year and delivery has taken place as intended in the Memorandum of Understanding (MoU) between DCMS and Innovate UK.

What worked well, or less well, for whom and why?

There are several examples of elements of CyberASAP that worked well, including:

• the application process as participants indicated they were provided with sufficient information prior to applying for CyberASAP and felt the process was straightforward
• a delivery structure which included a part-time and stage gated process ^{[footnote 3]} combined with a digital and hybrid model to support participants with a range of abilities and from a variety of locations
• the provision of support for value proposition, market validation and proof of concept development
• the provision of opportunities for participants to meet and engage with key industry figures and entrepreneurs which resulted in the development of tangible business relationships
• improved entrepreneurial and business management skills
• that those who did not proceed to phase 1(b) or phase 2 still benefited from the programme as it led them to re-think or pivot their idea to a more viable proposition
• the development of a strong ‘community’ who keep in touch and support each other
• that participating teams have successfully attracted investment from a variety of sources, including private business and public sector funds
• 89% (n=47 of 53 survey respondents) feeling their commercial awareness had improved following participation
• the geographical diversity of participants, as shown by 64% (n=35) of the 55 survey respondents being located outside of London and the South-east

There were also several examples of areas for development. This included:

• the need for more time between DCMS funding being confirmed, publishing the call for programme applications and commencing delivery to promote the programme with key target groups
• the need for greater support from university technology transfer officers and flexibility with internal university policies ^{[footnote 4]}
• the need for increased linking of the various cyber programmes available

Recommendation 1: we recommend that decisions to fund CyberASAP are communicated at least 9 months in advance (rather than the current timeframe of c. 4 months) to allow sufficient time for programme promotion with potential entrepreneurs from underrepresented groups.

Recommendation 2: we recommend that CyberASAP engages with university Technology Transfer Offices (TTOs) to identify potential candidates for the programme.

What can be learned from the delivery methods used? / Were there any unexpected or unintended issues in the delivery of the intervention? / How did external factors influence the delivery and functioning of the programme?

The majority of survey respondents (98%, n=54) were satisfied or very satisfied with the programme structure.

However, COVID-19 limited face-to-face interaction and qualitative feedback suggested this was detrimental to relationship building between participants and with investors. Other factors that impacted delivery across all years of the programme included that:

• academics involved in teaching had limited time to develop their idea / business
• a lack of engagement from the university TTOs restricted awareness and engagement with the programme by academics, and in some instances meant any issues for the university were only raised at phase 2

How can the existing programme be improved to become more effective?

The existing programme could be improved by:

• issuing a thematic call for project proposals focused on market failures in the cyber security sector (in addition to the current format for an open call). This call would be designed to complement other DCMS-funded programmes and external funding calls while also aligning with the needs of industry
• moving delivery of the programme to less busy times of the academic calendar which would improve the programme for participating teams as they are less likely to have teaching responsibilities
• confirming funding earlier in the year or providing funding for more than one year at a time
• improving monitoring and reporting through the development of SMART ^{[footnote 5]} targets for all outcome and impact measures in the Theory of Change (ToC)

Recommendation 3: we recommend that SMART outcomes are further developed for CyberASAP and there is detailed reporting on these. KTN to develop based on the CyberASAP ToC metrics and agree with DCMS. Evidence should be collected by KTN for progress against each using both published data and primary research with participants. For example, those noted in the DCMS MoU with KTN for Year 6 and including:

|Outcome measures| |Measure 1: number of new cyber security start-ups / spinouts (company registrations) attributed to CyberASAP
Evidence source: published datasets| |Measure 2: number of viable products or services trialled and attributed to CyberASAP
Evidence source: participant feedback via surveys or interviews| |Measure 3: increased confidence and skills to deliver cyber security start-ups / spinouts
Evidence source: participant feedback via surveys or interviews| |Measure 4: increased business survival rates for cyber start-ups / spinouts supported by CyberASAP
Evidence source: published datasets| |Measure 5: increased investment in cyber security start-ups / spinouts supported by CyberASAP
Evidence source: published datasets and participant feedback via surveys or interviews

1.2 Impact evaluation

The programme has delivered business impacts. To date 108 projects have been supported by CyberASAP ^{[footnote 6]} and of these 12 were categorised as spinouts; 2 were acquired by other firms and 5 developed a patent.

The DCMS funding has leveraged £12,382,895 in additional investment (for example corporate / private investment, acquisition, VC investment, angel investment, and seed funding / equity fundraising).

Programme participants were able to meet with organisations, however it will take time to understand the impacts from these meetings and any relationships.

It has also been successful in developing entrepreneurial skills and confidence, including for participants who did not progress to the proof-of-concept stage however completed part (a) or all of phase 1.

To what extent has the programme been successful in commercialising academic research / accelerated the process to commercialisation?

Overall, 26% (n=12 of 47 tracked participants) have spun out companies successfully ^{[footnote 7]}, some of whom provided their project outputs as an open-source product and through published articles. In addition, survey respondents reported an increased capability to commercialise their research following participation in the programme as:

• prior to participating in the programme 11% (n= 6) of 54 respondents ranked their capability to commercialise research between 8 and 10
• following participation 66% (n= 35) of 53 respondents ranked their capability to commercialise research between 8 and 10 (10 meaning very strong) ^{[footnote 8]}

Nevertheless, participants still found it difficult to commercialise their research after the programme had finished and felt that further support is required, either from their university and / or by having further support CyberASAP. Specific examples included ensuring that universities and TTO staff maintain commitment to commercialisation throughout and are as committed to the commercialisation of the Intellectual Property (IP) as the investors. Based on the contribution analysis conducted CyberASAP has had:

• strong contribution to project teams increasing their successful trials and to their business knowledge and skills
• some contribution to the development of new cyber security start-ups or spinouts and short-term investment in these
• negligible contribution to the improvement of participating teams’ ability to enter other incubators and increase input of industry into academia

While 77% (n=41) of the 53 respondents who answered the question had not yet received investment following the programme, some of those will only be at the stage of seeking investment. Therefore, the above contribution analysis is only part of the answer. It will be important to continue tracking progress of the projects after the programme ends to measure longer term impacts.

Recommendation 4: we recommend that the outcomes and impacts from the programme continue to be tracked. It should be a requirement within the delivery partner contract to provide evidence of the capability and business outcomes being achieved for participants at one, two and three years after completing the programme to provide evidence of the longer-term benefits.

We recommend DCMS set up a monitoring template that covers all the outcome and impact measures expected from the programme. This should be completed by the delivery partner for participants when they finish the programme and then at 6, 12 and 24 months after completion.

To what extent does the evidence suggest future funding would be more effective if targeted differently (at less well funded universities, for example)?

The premise of the current CyberASAP model is to attract and identify the most promising commercial opportunities from different parts of the UK academic research base and to make funding opportunities accessible to universities from all regions as well as those outside the Academic Centres of Excellence in Cyber Security Research (ACE-CSR) and Russell Group.

The project has been successful in achieving this as 82 of the 108 (76%) participating universities from non-Russell Group institutions and 84 (77%) outside of London.

Due to the programme both reaching several non-Russell Group universities and there being a lack of notable difference in outcomes between Russell and non-Russell Group universities (as discussed below), a focus on the best commercial opportunities should remain.

To what extent does the success of the programme differ between different cohorts, types of firm/idea, university?

There is no notable difference in outcomes between cohorts, projects or between universities, including Russell Group and non-Russell Group institutions (with one exception). The only notable variations were:

• Cohort 5 – survey respondents from this cohort reported fewer intended outcomes as identified from the ToC being achieved to date. This is likely due to their more recent completion of the programme compared to preceding cohorts
• Cohort 1 – survey respondents from this cohort reported a greater number of spinouts and receiving further funding. This could be due to (1) having completed the programme c. 4 years previously and (2) the design of the programme in Year 1 meant there was an obligation to spinout/form a company

In addition, there is a similar proportion of the 108 Russell Group and non-Russell Group university projects that are in development, licensed and acquired. ^{[footnote 9]} However, a higher difference in the amount of spin-outs achieved was identified between the two groups. KTN feedback suggests this is likely to be the result of Russell Group universities typically being better funded and possessing mature and well-established commercialisation capacities. ^{[footnote 10]} This is shown by the proportion of Russell and non-Russell group universities that have:

• reached development stage (Russell Group: 42%; Non-Russell Group 40%)
• spun out (Russell Group: 33%; Non-Russell Group 23%)
• been licensed (Russell Group: 8%; Non-Russell Group 3%)
• been acquired (Russell Group: 8%; Non-Russell Group 3%)

Do participants join other cyber growth programmes after completing CyberASAP?

Evidence obtained from stakeholder interviews indicated that most of the CyberASAP participants who fully completed all phases of the programme did not go on to other cyber growth programmes. Those who did not progress through all stages of CyberASAP tended to either (a) re-apply for CyberASAP; (b) progress to the London Office for Cybersecurity Advancement (LORCA); (c) progress to the Cyber Runway programmes, or (d) access other sources of funding / development programmes. This was in part due to CyberASAP helping them to better understand the schemes available, suggesting the programme plays an important role in the wider cyber growth and innovation ecosystem.

What are the additional or unintended benefits of the programme?

Delivery partner feedback suggests the programme has helped to change the way universities approach commercialisation as some are now more willing to be pragmatic about how much equity they will receive. For example, Royal Holloway University applied a policy which assigned IP to the start-up company in exchange for an under 10% share to make the deal more attractive to investors. ^{[footnote 11]} KTN have also suggested the programme is starting to see repeat participation from universities who are beginning to change their approach in this area.

2. Introduction, terms of reference and methodology

2.1 Introduction

RSM Consulting LLP were commissioned by the Department for Digital, Culture, Media and Sport (DCMS) ^{[footnote 12]} to undertake independent evaluations of the CyberASAP, Cyber Runway and UKC3 programmes. The evaluations will help DCMS to understand the impact of these programmes and the findings will be used to inform the development of future interventions.

The CyberASAP programme supports the commercialisation of UK research into cyber security and helps academic researchers to turn ideas into fully rolled-out commercial projects by developing the academics’ entrepreneurial skills. In doing this, it recognises the barriers that academics face when commercialising research, including the lack of dedicated time available to research the market and to validate potential products.

This evaluation report relates to the CyberASAP programme which was initially piloted in 2017/18 and has been delivered over five years to 2021/22, with one cohort of researchers each year. ^{[footnote 13]}

The evaluation incorporates delivery and performance across all years to date.

2.3 Methodology

The evaluation methodology was agreed with DCMS and includes the following stages:

Scoping phase

(1) Project initiation meeting: the project commenced with a project initiation meeting involving the evaluation team and DCMS to: (1) review and agree the evaluation methodology and timetable; (2) discuss access to relevant information and (3) finalise arrangements for project management and progress updates.

(2) Desk research and analysis: a review of the strategic and delivery context for the programme and mapping was conducted to identify other sources of funding available to support the commercialisation of cyber security academic research.

(3) Review of programme documentation setting out rationale for funding measures: review of the programme business case; MoU’s between DCMS and Innovate UK; Key Performance Indicators (KPIs) agreed between DCMS, Innovate UK and KTN; and previous research / theories of change relating to the programme to identify the rationale for the intervention and the outputs and impacts expected from it.

(4) Development of ToC: an online workshop was facilitated with DCMS staff involved in the business case for funding and the design and management of the programme to test and refine the draft ToC and associated metrics. The final ToC (see Appendix B – Theory of Change) was used to inform the research tools that were developed, specifically the participant survey and guides for the participant, delivery partner and case study interviews.
(5) Evaluation plans for each programme: an evaluation plan was developed detailing the design and approach being taken to address the evaluation questions. This was informed by the ToC and outlined how each of the ToC metrics would be measured.

Data collection

(1) Analysis of programme monitoring information / impact information and published data: to inform the assessment of programme performance against its core KPIs (as per the ITT) and those in the agreed ToC.

(2) Surveys and consultations: this involved:

• a survey of CyberASAP participants with 55 respondents (response rate = 71% ^{[footnote 14]}) (see Appendix C – CyberASAP Participant Survey)

• interviews with the delivery partner (KTN)

Survey methodology: the survey was designed by RSM UK Consulting in collaboration with DCMS to collect evidence against the key evaluation questions and ToC metrics. As it was not possible for participant details to be shared with RSM UK Consulting without consent, an online survey link was distributed via KTN, with subsequent reminders by email and targeted telephone follow-up to ensure a representative sample across regions and cohorts.

Note: where ‘n=’ is used during survey analysis, it is referring to the number of respondents responding in a certain way / to a specific answer choice, rather than the entire respondent base. Where applicable, a base number has been provided in figure titles or in text to provide more general information on total number of respondents. This base number will occasionally vary from the overall survey participant number of 55 depending on relevance of the question and if respondents choose not to answer.

Participants survey profile:

Table 1: CyberASAP participants respondents by region (base number = 55)

Region	Sample of CyberASAP survey respondents - frequency	% of survey respondents
Scotland	2	4%
Wales	2	4%
Northern Ireland	0	0%
North-west England	5	9%
North-east England	2	4%
Yorkshire and the Humber	2	4%
West Midlands	5	9%
East Midlands	2	4%
East of England	7	12%
South-west England	8	14%
South-east England	10	18%
London	10	18%

Table 2: CyberASAP participants respondents by cohort (base number = 55)

Cohort	Sample of CyberASAP survey respondents - frequency	% of survey respondents
Year 1	7	13%
Year 2	15	27%
Year 3	12	22%
Year 4	11	20%
Year 5	10	18%

(3) Counterfactual: nine interviews were completed with those participants successful in applying to phase 1 of CyberASAP but did not proceed to phase 1(b) or phase 2.

(4) Case studies: four in-depth case studies were developed to provide qualitative insight into the benefits of participating in the programme. These were selected to provide a representative sample across regions, cohorts, and stage of idea development and are shown in Table 3.

Table 3: Case Studies

Project	Region (England, Wales, Northern Ireland, Scotland)	Cohort (Year)	Stage of development
Cydon	England (University of Wolverhampton)	Year 2	New company established
PriSAT	Scotland (University of Glasgow)	Year 3	In development
Secure development	England (Lancaster University)	Year 4	In development
WalletFind	England (University of Salford / Bristol / Manchester Metropolitan)	Year 5	In development

Analysis and reporting

• analysis of monitoring data for each programme by sector, geography, university, and cohort: to assess performance against targets and to identify variations between cohorts, universities, and cyber security sectors
• contribution analysis: this focused on why the results have occurred and the role played by CyberASAP; it involved 2 key steps:

(1) based on the ToC, 6 contribution statements were developed describing the outcomes CyberASAP intends to achieve and how

(2) based on the data collected in the previous stages the strength of evidence was assessed against each contribution statement, as well as evidence of any other factors that have contributed

The strength of evidence was determined by reviewing:

• clarity
• frequency of a particular theme
• diversity of the stakeholders who provide the evidence (across cohorts)
• emphasis placed on the evidence by stakeholders
• examples of how the stakeholders know about the evidence they report
• availability of the evidence across primary, monitoring and Management Information (MI) data

High strength of evidence includes:

• evidence that is articulated clearly and frequently by different stakeholders without the need for probing
• where the survey findings show high rates of “strongly agree / disagree” and similar responses across different measures

(3) The contribution of CyberASAP to the expected results as described by each contribution statement was assessed as strong, some, or negligible, with:

• strong contribution meaning that CyberASAP’s activities contributed substantially to the observed results
• some contribution meaning that CyberASAP’s activities contributed to the observed results but not substantially
• negligible contribution meaning that CyberASAP had no or minimal contribution to the result

Reporting: included a progress presentation, interim and final reports, a final presentation, and a closing workshop with DCMS which will act as a learning event.

Limitations

Counterfactual: it was not feasible or appropriate to contact those who were unsuccessful in their application to the CyberASAP programme to form a counterfactual group as their characteristics were too dissimilar to complete a robust regression discontinuity analysis - based on information provided by KTN – due to:

• their project being out of scope
• too little information being provided to make an assessment
• some of the application questions not being answered
• finances being incorrect within the application (for example, applied for too much)

The absence of a robust counterfactual means that caution should be applied to over interpreting the results of the impact evaluation, in that it is not possible to rule out the possibility that some of these impacts will have occurred under the counterfactual.

Therefore, it was agreed with DCMS that a qualitative counterfactual approach would be applied by completing interviews with participants who did not complete all phases of the programme in order to identify what they did instead and if / how the programme impacted on this.

3. Rationale and programme overview

This section details the strategy and delivery context, and the rationale for the CyberASAP programme, as well as providing an overview of other programmes in this space.

3.1 Review of the strategic and delivery context

3.1.1 Policy

The CyberASAP programme was expected to contribute, or has the potential to contribute, to several key national strategies, as set out below.

Table 4: Strategic context

Strategy	How the CyberASAP programme is expected to contribute
National Cyber Security Strategy (2016 - 2021) - supports the creation of a growing, innovative and thriving cyber security sector in the UK National Cyber Strategy 2022 - focuses on strengthening the UK Cyber Ecosystem	The CyberASAP programme was expected to contribute to this by supporting the commercialisation of innovation in academia and providing training and mentoring to academics that will help to ‘create a cyber ecosystem in which cyber start-ups proliferate, get the investment and support they need to win business around the world [and], to provide a pipeline of innovation that channels ideas between the private sector, government and academia’ While the CyberASAP programme was designed before the National Cyber Strategy 2022 was developed, it contributes to the objective of fostering and sustaining sovereign and allied advantage in the security of cyberspace-critical technologies through ensuring the UK becomes “more successful at translating research into innovation and new companies in the areas of technology most vital to our cyber power.”
DCMS UK Digital Strategy (2022)	The CyberASAP programme could contribute to one of the key actions within this strategy to ‘generate ideas and IP’ as it supports the successful development and commercialisation of new ideas. In addition, the Digital Strategy includes a focus on spreading digital prosperity and levelling up, and CyberASAP contributes to this by: • attracting and identifying the most promising commercial opportunities from different parts of the UK academic research base • making funding opportunities accessible to universities from all regions and to those outside the ACE-CSR and Russell Group

3.1.2 The UK cyber security sector growth and innovation space

The UK cyber security sector is growing rapidly as outlined in the UK Cyber Security Sectoral Analysis published in 2022. This is shown by:

• approximately 52,700 full-time equivalents working in the cyber security sector (a 13% increase from the previous year, of which 64% work in large firms with over 250 employees)
• an estimated revenue of £10.1 billion (an increase of 14% compared to the 2021 publication)

The UK has a reputation as a global leader in cyber security research, with 19 ACE-CSR, four Engineering and Physical Sciences Research Council – National Cyber Security Centre (EPSRC-NCSC) Research Institutes, four Centres for Doctoral Training, the Centre for Security Information Technologies (CSIT) and the PETRAS National Centre of Excellence in Cyber Security of Internet of Things (IoT). he 2022 UK Cyber Security Sectoral Analysis also notes that investment in cyber security firms has increased, with over £1.4 billion being raised in 2021 across 108 deals. In addition, the sector is playing a critical role in responding to emerging cyber threats and challenges, and the rapid proliferation of connectable products.

However, research in 2020 found that long-term investments in other nations, especially the USA, France and Germany, are leading to the development of large clusters of research excellence. This can pose a threat to maintaining the UK’s position as a leading nation for research and innovation in cyber security, given a potential brain drain from the UK. It suggests a need for the UK to further invest in cyber security research in various forms, including clusters of research excellence in cyber security; doctoral research funding to train future research and development (R&D) leaders in cyber security; and national research facilities.

The UK Innovation Strategy highlights that UK universities have become more effective at attracting investment and bringing ideas to market in recent years and of the top ten universities ranked by levels of funding raised by spinouts, the UK has five. However, this trend needs to be expanded beyond a small group of research-intensive UK universities, ensuring that technology transfer skills and expertise of a broader range of universities are enhanced to make the sector more accessible for investors.

The challenges within the sector and the innovation landscape include:

• investment being skewed to larger companies - start-ups and Small and Medium-sized Enterprises (SMEs) receive considerably less investment than larger, more established firms and the 2022 Sectoral Analysis found that 85% of the total 2021 investments targeted large-medium firms
• a regional divide in the cyber security sector - for example, 33% of job postings are in London, with only 2% in Wales and according to the 2022 Cyber Sectoral Analysis, 53% of Cyber Security firms are registered in either London or the South-east, with only 2% of the firms in Northern Ireland
• the UK no longer having access to EU cyber security funding - this would have amounted to approximately €1.65 billion (available across the entirety of Europe) over the period 2021 to 2027 ^{[footnote 15]}
• difficulty commercialising academic research into cyber security - as it faces barriers related to access to funding, ability to dedicate time to market research and validation, and the need to balance teaching, research and commercial activity ^{[footnote 16]}
• the sector not being diverse - the cyber security skills in the UK labour market report for 2022 highlights that 22% of the cyber security workforce is female (compared to 30% across the digital sectors as a whole), but ethnic minorities make up 25% of the cyber security sector workforce, which is higher than 12% in the overall workforce, and 15% in the digital workforce, that ethnic minorities represent ^{[footnote 17]}

3.1.3 Mapping of other programmes

CyberASAP is part of a wider ecosystem of cyber security growth and innovation programmes across different stages of the ‘innovation pathway’.

Figure 2: Cyber security growth and innovation programmes

CyberASAP is the first ‘stage’ in the innovation pathway focused on pre-seed and proof of concept ideas. It supports the commercialisation of UK cyber security research into fully rolled-out commercial projects. It is complemented and followed by programmes that support companies at different stages of the business lifecycle to:

• Launch - to support the establishment of new companies in the sector by helping to transform early-stage cyber security ideas into workable proposals and potential new businesses (programme: Cyber Runway Launch)
• Grow - to support existing SMEs in the early and growth stages of the life cycle to improve the survival rate of early-stage cyber businesses (programmes: NCSC for Strategy, Cyber Runway Grow and LORCA)
• Scale - to support cyber security scale-ups to address barriers to growth nationally and internationally (programmes: LORCA and Cyber Runway Scale)

These interventions are also complemented by other government and private sector initiatives with a cyber security element, illustrated in the following table.

Table 5: Mapping of other programmes

Project name	Target group	Funders and funding	Expected outcomes
Knowledge Transfer Partnership (KTP)	A KTP is a partnership open to a UK-based business or not-for-profit organisation and university, college or research and technology organisation.	Partnerships are part funded by grant, however businesses also have to contribute to the cost of the project. A typical KTP project delivers a support package valued at £80,000 - £100,000 per year. [footnote 18]	KTPs connect forward thinking businesses with knowledge bases to deliver business led innovation projects. The academic will work with the business to develop the project with academic input, while also recruiting a suitable graduate to the business to support with the project.
UK Research and Innovation (UKRI) commercialising quantum technologies challenge	Companies and projects involving products/technologies based on advances in quantum science.	UKRI is investing £153 million, supported by £205 million from industry. [footnote 19]	This competition aims to develop new products and technologies based on advances in quantum science. The challenge will comprise four areas to support the UK quantum industry. This includes: product and service innovations, industry-led technology development projects, supply chain feasibility projects, and an investment accelerator.
UK-Singapore Collaborative R&D	UK businesses looking to collaborate on R&D projects	Innovate UK will invest up to £3 million in partnership with Enterprise Singapore, with each individual project able to apply for a maximum grant of up to £350,000 (funding applies from September 2022 - August 2025). [footnote 20]	This competition aims to fund business led collaborative R&D projects focused on industrial research, with cyber security specifically identified as a sector that they would particularly welcome applications from. Proposals must include at least one partner from the UK and one partner from Singapore and demonstrate a disruptive innovative idea leading to new products / processes / services.
Industrial Strategy Challenge Fund (ISCF) digital security by design - software ecosystem development	UK registered organisations	UKRI investing up to £8 million in R&D projects (each project can request a total grant from £200,000 - £1.4 million, with funding applying from April 2022 – December 2024). [footnote 21]	The competition aimed to fund a range of projects that work to enrich and expand the Digital Security by Design (DSbD) software ecosystem prior to the availability of commercial hardware. The funding is from the ISCF.

There are several programmes available to support innovation within the UK’s wider cyber security ecosystem. However, there are no other initiatives focused primarily on the pre-seed, concept stage as, while Cyber Runway Launch aims to support the establishment of new companies in the sector, CyberASAP’s main focus is on addressing the challenges faced by academics in the commercialisation of research.

3.2 Programme overview and funding

3.2.1 History / background to the programme

The CyberASAP programme was initially set up as a pilot in 2017 based on research by KTN into barriers for the commercialisation of research in cyber security. The programme’s development is outlined in the following table.

Table 6: CyberASAP development – summary

Year	Summary
Year 1 (2017)	The initial pilot was an Innovate UK funded programme, delivered by the SetSquared partnership. Year 1 was developed initially incorporating the Innovation to Commercialisation of University Research (ICURe) model [footnote 22] as the source of training. KTN provided additional business training to support projects in the development of a Minimum Viable Product (MVP) [footnote 23] and delivered a showcase / demo day at the end of the programme in October 2017.
Year 2 (Programme Design, 2017 and delivery 2018-19)	Working with the leads from DCMS and Innovate UK; KTN led a review and workshopped a new programme based on the feedback from teams and insights gained by DCMS and Innovate UK through the Year 1 programme. This new programme split the activities into commercial proposition development stages, with an external selection panel comprised of industry experts assessing the projects before they moved to the next stage. In addition, a range of personal development skills were introduced as appropriate for each stage of the programme. The Year 2 programme removed the obligation to spinout / form a company at the development grant stage as it was identified that the projects required more support and further development.
Year 3-5 (Delivered over each financial year 2019-2022)	The programme was further developed based on cohort, alumni and industry feedback. In Year 5 the programme was opened to participants of the Security of Digital Technology at the Periphery (SDTaP) programme. Grant funding for SDTaP was from an Innovate UK budget (UKRI Strategic Partnerships Fund) and programme running costs were shared with DCMS. The programme content was the same for the DCMS and SDTaP projects. [footnote 24]

Source: Information provided by KTN to RSM UK Consulting (February 2022)

The evaluation incorporates delivery and performance across all years to date.

3.2.2 Programme delivery

The programme has consisted of two phases based on a similar structure used in the ICURe programme, the proof-of-concept phase was added after the pilot year as it was felt this was missing in the original ICURe model. This included the:

• development of value proposition and market validation by academics - phase 1(a) and phase 1(b)
• development and demonstration of a proof of concept by the most promising projects (phase 2)

A two-stage selection process is used to both (1) ensure there is a wide selection of ideas and universities involved and not only those ‘most likely to succeed’ and (2) for industry experts to filter out those that do not have a robust idea / concept that is viable to take forward to phase 2.

Figure 3: CyberASAP overview

3.2.3 Key outcomes / impacts expected

The initial ICURe pilot programme had a simple set of KPIs. This included:

• supporting the creation of at least 5 trading entities or licensing agreements
• recruiting 6-7 possible concept demonstrator teams for phase 2 funding and progress to demo day
• delivering a programme of monitoring and support for the selected teams
• creating and delivering a programme of business planning, development and mentoring

From Year 2 these were developed further as KTN introduced a logic model to measure the social and economic impacts of the programme based on Year 2 activities and programme design. Key programme objectives included (based on the 2021/22 delivery year):

Figure 4: CyberASAP Year 5 (2021/22) objectives

4. CyberASAP process evaluation

This section details the CyberASAP programme governance structure; key stakeholders; the application process; how the programme is delivered; and reporting requirements. It focuses on assessing whether the programme was delivered as intended and what could be improved in delivery.

4.1 Governance structure

In Year 1 (2017) the delivery partners were:

• Year 1 (Phase 1) SetSquared (as part of the ICURe programme) - focused on delivery of the support programme & management, and funding to university teams
• Year 1 (Phase 2) SetSquared (as part of the ICURe programme) - focused on management & payment of funding to university teams
• Year 1 (Phase 2) KTN - focused on delivery of the business support programme

The governance structure for the CyberASAP programme in Years 2 – 5 is outlined below:

Figure 5: CyberASAP governance structure (Years 2- 5; 2018 – 2022)

Roles and responsibilities

The following table provides further detail on the roles and responsibilities of each organisation.

Table 7: Roles and responsibilities

Organisation	Role / Responsibilities
DCMS	DCMS manages: • funding – provides funding to Innovate UK for administrative costs; KTN costs and delivery costs • attendance at events in an observatory capacity
Innovate UK	Innovate UK manages: • the funding of the delivery partner, KTN, who are delivering the complete programme to the academic teams • the funding of payments & monitoring of the academic teams via their universities through a standard Innovate UK grant-funded competition Specifically: • project management - in accordance with UKRI-Innovate UK’s standard operational procedures, and its corporate policies - review progress of the programme at regular intervals with DCMS - meet quarterly with DCMS - ensure that programme management includes sufficient focus on the evaluation of benefit forecasts, with appropriate teams and resource consulted to deliver an effective evaluation strategy as part of the management of the programme • contracting – with KTN to deliver the CyberASAP programme • reporting – providing DCMS with financial and monitoring reports
KTN	Innovate UK contract KTN to deliver the programme content. They are responsible for devising and delivering the programme via: • bootcamps • training • tools • mentoring • peer to peer learning • an industry showcase (demo day) • KTN have delivered the programme from Year 2, taking over the role from the SetSquared partnership after the first two phases of Year 1 were completed as part of the ICURe programme

Source: DCMS and Innovate UK MoU (2021)

4.2 Programme delivery

4.2.1 Key phases / stages

Design ^{[footnote 25]} - KTN discusses proposed changes and evolution of the programme year-on-year in collaboration with DCMS and Innovate UK before submitting a formal proposal to Innovate UK and DCMS in December / January containing a plan for the next programme.

Delivery - in all years there has been a single call for applications / competition and intake per year. ^{[footnote 26]} Further information on delivery includes that:

• in Year 2, the competition opened in January and the programme commenced in February, running until the following January
• for Years 3-5 the programme has been delivered within a single financial year; broadly the competition opens in February / closes early March and delivery is from April until the following February, with minor adjustment to dates as needed for operational reasons
• each year the delivery and completion of the programme has taken place as planned (there has not been any slippage/rescheduling of planned dates)

4.2.2 Communications and promotion

KTN communicated and promoted the programme in several ways, including via:

• a social media strategy
• Twitter and LinkedIn accounts
• the in-house KTP advisors who support the KTP programme (a KTP connects universities and UK businesses meaning the KTP advisors cover all UK universities)
• a dedicated communications lead / KTN communications team
• a dedicated events lead who worked with the communications lead and the core delivery team to ensure that all activities were promoted and supported (this included the promotion of competition and wider events)

KTN also maintains a year-round general ‘expression of interest’ in the programme which is used to promote the competition when it opens. Between Year 3-5 of the programme, KTN also introduced weekly online (Zoom) drop-ins for participants and alumni, as well as an alumni newsletter highlighting opportunities for grants, investment, event participation and new programmes. They also introduced an alumni tracking programme to facilitate and encourage alumni engagement.

While the programme is promoted in several ways, delivery partner feedback highlighted this is hindered by the lack of certainty regarding funding for the next year or programme. It is not known until approximately November or December each year if the programme will be funded for delivery the following April, leaving a short time frame from publishing the call to receiving responses and assessment. It was suggested that if it were possible to market the programme earlier (for example, from June in the preceding year) there could be a greater focus on addressing any regional, diversity or gender target by engaging with organisations in these areas, exemplified by those supporting women in cyber.

CyberASAP participant findings:

Awareness of the CyberASAP programme

Awareness of the programme from participants was mainly through their university (31%, n=17 were first made aware of the programme in this way). Both other researchers and KTN were the next most common sources of information (24%, n=13). This highlights a clear emphasis on these three information streams for raising awareness of the programme. No respondents indicated they became aware of the programme via print / social media, suggesting these approaches may have been less worthwhile.

Figure 6: How participants became aware of CyberASAP (base number=55)

4.2.3 Application process

Eligibility

The most recent CyberASAP programme eligibility criteria includes the following:

Figure 7: CyberASAP eligibility criteria (2021/22)

All individuals based in a UK academic institution are eligible, including but not limited to early career researchers and senior academic researchers. KTN have noted that the criteria were intentionally wide-ranging, reflecting the nature of cyber security.

Changes which have been made to the eligibility criteria over the duration of the programme have focused on having the support of the TTO within the university, specifically: ^{[footnote 27]}

• in Year 3 it was emphasised that applicants had to have the support of their university, as there were issues in year 2 with projects having applied without informing their university
• for Year 4 onwards it was a requirement that the TTO, or equivalent, is named on the application form to try to ensure that they are involved
• for Year 5 onwards KTN are trying to directly engage with the TTOs rather than relying on the academic project leader to do this, including holding briefing events for them

Process

The application process for Phase 1 was via a standard Innovate UK grant-funded competition. Teams were assigned a monitoring officer to support the project management, reporting and finances. This approach has been consistent from Year 2 onwards. In Year 5 the programme piloted a challenge-led approach ^{[footnote 28]} and incorporated SDTaP university projects. The application process for the SDTaP university projects was almost identical to that for other teams, with the SDTaP projects indicating if they were applying for the SDTaP cohort or the open cohort. Following this phase:

• projects were assessed by external industry-led panels and only projects that passed proceeded to phase 1(b)
• applications to Phase 2 were via a closed Innovate UK competition focused on the development of a proof of concept; feedback is provided by Innovate UK to those who are unsuccessful at the end of Phase 1(a) and 1(b). A summary of the initial application process is illustrated in figure 8

Figure 8: CyberASAP application process – phase 1

CyberASAP participant findings:

Application processes

Participant survey responses suggest they were provided with sufficient information prior to applying for CyberASAP, specifically:

• 60% (n=33) of the 55 survey respondents strongly agreed and 29% (n=16) agreed that the right amount of information was provided
• 96% (n=53) of the 55 survey respondents felt the application process was straightforward

However qualitative feedback highlights information they would have liked to have been provided with prior to participation, with the most common theme being an increase in information provided around programme structure:

A bit more detail on how the programme is structured would have been useful.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

A bit more advice on the role of the three major stages - value proposition / market validation / commercialisation.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

4.2.4 Support provided

Each phase commences with an online 1–2-day bootcamp, followed by intermediate training and other sessions, and ends with a formal assessment panel for which the teams submit reports and / or presentations in advance. The exception is Phase 2 which ends with a demo day where the teams pitch and demonstrate their proof of concept to an invited audience of investors and industrialists. In Year 4 and 5, in line with the online delivery of this event, the teams prepared pitch videos and demonstrator videos for their proof of concept. As a condition of the grant award, project teams are expected to attend the scheduled activities.

Figure 9: CyberASAP - activities overview

In Year 4 provisions were included by KTN, supplementary to contract due to a surplus budget as a result of fewer events being held due to COVID-19. This included:

• weekly online drop-ins throughout the programme (April 2020 – February 2021)
• R&D Tax Credits Webinar
• proof of concept demonstrator videos for each project – fully edited
• pitch videos for each project – support for recording and creation from media contractor

CyberASAP participant findings:

Most participants stated they felt support was available for value proposition development (77%, n=37), market validation (71%, n=34) and proof of concept development (52%, n=25).

Figure 10: Participants perceived availability of support throughout the programme (base number = 48)

However, Table 8 shows fewer respondents were satisfied with the support provided for proof-of-concept development, specifically:

• 60% (n=29) of 48 respondents were very satisfied and 35% (n=17) were satisfied with support for value proposition development
• 54% (n=26) of 48 respondents were very satisfied and 33% (n=16) were satisfied with support for market validation
• 33% (n=15) of 45 respondents were very satisfied and 36% (n=16) were satisfied with support for proof of concept development

In addition, the majority of participants (98%, n=54) were satisfied or very satisfied with the overall structure of the programme (the provision of support for the three aspects listed above).

Table 8: Participant satisfaction in areas of the programme where support was provided (base number = 48) ^{[footnote 29]}

Answer choice	% Very satisfied	% Satisfied	% Neither satisfied nor dissatisfied	% Dissatisfied	% Very dissatisfied	% Not sure N/A
Training for value proposition development	60% (n=29)	35% (n=17)	0%	0%	0%	4% (n=2)
Materials for value proposition development	56% (n=27)	38% (n=18)	2% (n=1)	0%	0%	4% (n=2)
Support for value proposition development	60% (n=29)	35% (n=17)	0%	0%	0%	4% (n=2)
Training for market validation	60% (n=29)	33% (n=16)	2% (n=1)	0%	0%	4% (n=2)
Materials for market validation	56% (n=27)	33% (n=16)	6% (n=3)	0%	0%	4% (n=2)
Support for market validation	54% (n=26)	33% (n=16)	4% (n=2)	2% (n=1)	0%	6% (n=3)
Grant funding (n=46)	37% (n=17)	41% (n=19)	9% (n=4)	2% (n=1)	0%	11% (n=5)
2-day bootcamps (n=47)	51% (n=24)	38% (n=18)	4% (n=2)	0%	0%	6% (n=3)
Selection panel assessments (including Q&A and feedback)	50% (n=24)	35% (n= 17)	10% (n=5)	0%	0%	4% (n= 2)
Mentoring sessions	46% (n=22)	44% (n=21)	6% (n=3)	0%	0%	4% (n= 2)
Industry showcase demo day	50% (n=24)	35% (n= 17)	13% (n=6)	0%	0%	2% (n= 1)
Peer-to-peer learning (n=47)	43% (n=20)	32% (n=15)	15% (n=7)	0%	0%	11% (n=5)
Legal training (including IP training) (n=46)	28% (n=13)	35% (n=16)	26% (n=12)	0%	0%	11% (n=5)
Training for proof of concept development (n=47)	36% (n=17)	40% (n=19)	13% (n=6)	0%	0%	11% (n=5)
Materials for proof of concept development (n=46)	30% (n=14)	33% (n=15)	22% (n=10)	0%	0%	15% (n=7)
Support for proof of concept development (n=45)	33% (n=15)	36% (n=16)	16% (n=7)	0%	0%	16% (n=7)

4.2.5 Data and reporting

Innovate UK provide DCMS with monitoring reports each quarter. Each report includes an overall rating for the programme, progress on deliverables, recommendations for the quarter and any other actions that may need to be taken. It also provides a view on the extent to which benefit forecasts are being realised and highlights actions to resolve any issues preventing the realisation of identified benefits.

Data collected to date by KTN for in-programme delivery has mainly focused on key output statistics such as application numbers and universities engaged, among others, and on confirming that projects have participated in the planned activities. In addition to the in-programme delivery reporting, KTN has tracked CyberASAP programme graduates. This has included the development of an impact and insights report as well as several 1–2-page case studies on key projects. A CyberASAP website (microsite on the KTN website) has also been developed to showcase graduated projects. From Year 5 DCMS introduced a monthly logframe report which includes:

• a cover sheet with a brief narrative overview of progress
• a financial report
• KPI and milestone progress
• a risk register and guidance on the risk register

KTN and Innovate UK populate the report, which is then reviewed by the DCMS Programme Manager on a monthly basis with a follow-up discussion. DCMS feedback suggests the logframe reports provide all the information required to effectively oversee delivery of the programme, including risks and progress against the KPIs agreed in the MoU and no action has been required to address underperformance.

The logframe format was introduced to standardise financial and performance data being collected across several DCMS programmes and to align with onward reporting requirements to Cabinet Office and DCMS central finance. Areas for development include:

• the development of specified targets for all KPIs (the contract between DCMS and Innovate UK does not specify targets and instead notes a number of qualitative milestone objectives which the logframe report indicates as either completed, delayed or not complete)
• an explanation of why any delays occur

The monitoring approach could be strengthened by:

• formulating SMART targets for each KPI
• providing narrative information about progress against the milestones
• providing targets for milestones – this would be especially valuable where delays occur so that these can be understood

It may be useful to consider if key outputs or outcomes from the programme could be incorporated in regular reporting through the logframe, including:

• the number of proof-of-concept demonstrators
• the number of market validated value propositions
• new licensed cyber security technologies
• the number of cyber trademarks or patents

These are core ToC outputs which would improve reporting on milestones for each of the CyberASAP phases, without being onerous for programme teams to provide the relevant information.

In addition, there is currently no requirement for alumni engagement and follow-up. While KTN have completed work in this area as a result of underspend in 2020/21, consideration should be given to ongoing and regular follow-up as the alumni group continues to grow in order to demonstrate longer term impacts, for example at 6 months, 1- and 2-years post completion.

CyberASAP participant findings:

Reporting processes

The majority of CyberASAP participant survey respondents highlighted they were satisfied or very satisfied with the frequency of reporting; amount of information they needed to provide; and clarity of reporting requirements.

Table 9: Participant’s assessment of the reporting arrangements on CyberASAP (base number= 54)

Answer choice	% Very satisfied	% Satisfied	% Neither satisfied nor dissatisfied	% Dissatisfied	% Very dissatisfied	% Not sure N/A
Frequency of reporting	31% (n= 17)	43% (n= 23)	13% (n= 7)	9% (n= 5)	2% (n= 1)	2% (n= 1)
Amount of information to provide	31% (n= 17)	39% (n= 21)	19% (n= 10)	7% (n= 4)	2% (n= 1)	2% (n= 1)
Clarity of reporting requirements	37% (n= 20)	41% (n= 22)	11% (n= 6)	6% (n= 3)	2% (n= 1)	4% (n= 2)

This is also reflected in the qualitative feedback provided:

Something that massively helped was that the reports were entirely in line with what we would need to compile to get the company progressed anyway - it was not “busy” work but rather the genuine and progressive output from the project which was reported. I found checking in with the MO was also useful as a focal point for making sure I had everything together.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

The reporting seemed just about right, not too onerous. Having someone to speak to (even if it was just a short meeting that was needed) was very useful.

– CyberASAP survey respondent (Cohort 5 – 2021/22)

4.2.6 Factors impacting on programme delivery

The impact of external factors on programme delivery

Views on the impact of COVID-19 on programme delivery were mixed with 49% (n=26) of the 53 respondents indicating COVID-19 had not impacted delivery at all, while 43% (n=23) suggested it had a moderate to large impact. However, qualitative feedback suggests the impact of COVID-19 in reducing face-to-face interactions was detrimental to relationship building:

There’s less interaction among participants and also with the organisers of the programme.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

The main impact of COVID-19 was not being able to meet with the CyberASAP team and the other teams in the competition

– CyberASAP survey respondent (Cohort 4 – 2020/21)

The entire programme was run remotely. This affected the ability of the cohort to build stronger cohesion and limited networking possibilities. Although the CyberASAP team has done an amazing job at keeping activities captivating and agreeable and have been available as much as needed for the cohort.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

This was also reflected in feedback from participants who did not progress through the full programme who referred to the benefits of informal and constructive discussions with other participants and stakeholders between seminar discussions. However, they also noted that teaching while developing their idea / business was one of the most notable external factors hindering progression as CyberASAP was run alongside academic requirements.

What worked well in programme delivery

Qualitative feedback from participants highlighted a key benefit of the programme was its ability to provide opportunities for participants to meet and engage with key industry figures and entrepreneurs, which has led to tangible business relationships:

The meet the entrepreneur days - excellent insight and feedback from the experts, it was a unique and informative experience.

– CyberASAP survey respondent (Cohort 3 – 2019/20)

The people involved in the programme: not only the organisers, but also the people invited to give talks and presentations.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

Participants also valued the skills development training provided, with survey respondents noting that the programme had developed their entrepreneurial (85%, n=46 of the 54 respondents) and business management (56%, n=30 of the 54 respondents) skills. This is also illustrated in the qualitative feedback received on how the programme had impacted on their development:

Skills development - especially market validation, value proposition, and presentation in a commercialisation context.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

Training on managing a sales funnel was superb.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

What worked less well in programme delivery

Feedback from participant survey respondents suggests they would like greater support from their university TTOs with Figure 11 showing that 37% (n=20) felt they only received a small level of support and 24% (n=13) feeling TTOs had not helped at all. Qualitative feedback from several participants highlights that their universities did not have a technology transfer team or equivalent to support them.

Figure 11: Assessment of support provided by universities and their TTOs (base number= 54)

At the time we didn’t have a proper tech transfer officer…

– CyberASAP survey respondent (Cohort 3 – 2019/20)

At the time the university did not have a dedicated technology transfer officer, however they supported [me in] exiting the university to run the business.

– CyberASAP survey respondent (Cohort 1 – 2017/18)

In relation to issues participants encountered during programme delivery, the most common was internal university policies. This, combined with the lack of positive impact regarding university TTOs, highlights the need for improvement in this area.

Figure 12: Issues encountered during participation in CyberASAP (base number= 54) ^{[footnote 30]}

This was also reflected in delivery partner feedback which indicated TTOs within the universities had not sufficiently engaged with the programme, meaning any issues were only identified at a later stage during Phase 2 (in relation to IP, for example). In addition, it was suggested that technology transfer officers would be an efficient way to raise awareness of the programme in all universities across the UK (Russell and non-Russell group) rather than the delivery partner contacting academics directly.

While only 20% (n=11) of the 54 respondents felt there were gaps in delivery, key themes highlighted by those who provided further detail included a lack of preparation for life after CyberASAP, and specifically the key issues facing newly formed companies. For example qualitative feedback includes:

Further training on the development and management of a start-up company could be a great add-on.

– CyberASAP survey respondent (Cohort 5 – 2021/22)

More insight into a company formation process would be useful.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

Information on how different share types work in a new company.

– CyberASAP survey respondent (Cohort 3 – 2019/20)

What improvements could be made to programme delivery

In relation to how the programme could be improved, 16% (n=8) of the 50 free-text responses referred to an increased emphasis on funding for participants and 55% (n=29) of 53 respondents suggested that grant funding was insufficient to achieve the maximum benefits for their projects. Qualitative feedback on potential improvements included:

The grant for phase 2 was insufficient (less than advertised).

– CyberASAP survey respondent (Cohort 3 – 2019/20)

Increased funding to extend the scope of the proof-of-concept outcomes.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

Follow up funding after proof of concept.

CyberASAP survey respondent (Cohort 2 – 2018/19)

Extra financial support to get printing and business planning.

CyberASAP survey respondent (Cohort 4 – 2020/21)

4.3 Summary of key findings

The programme has been delivered effectively with positive feedback and high levels of satisfaction from participating academics on the application process and support provided at each stage for their value proposition, market validation and proof of concept development. Participants valued the opportunity to meet and engage with key industry figures and entrepreneurs and highlighted the training undertaken had led to personal development in key areas for commercialising research, including entrepreneurial and business management.

While COVID-19 impacted on programme delivery, this was mainly in relation to relationship building and not the quality of training / support provided. However, key issues which participants highlighted as a barrier to their participation and progression included commitments to teaching while developing their idea / business, the lack of support from their TTOs and internal university policies. Each year the delivery partner has tried to increase the engagement of the technology transfer officers with universities and while this has developed to an extent, participant feedback suggests there is still a need to get greater university ‘buy-in’ to the programme.

5. CyberASAP impact evaluation - performance

This section provides an overview of the performance of CyberASAP against its KPIs, delivery objectives, and outputs as specified in the programme ToC. The sources used are based on monitoring information which KTN and Innovate UK submit to DCMS, including logframe reports. ^{[footnote 31]}

5.1 Participants

The following diagram provides an overview of participant numbers for each phase of the programme in each year (no targets were set for each phase):

Figure 13: Participants for each phase of CyberASAP 2017 - 2022 ^{[footnote 32] [footnote 33]}

Of the 108 participating universities for which data is available ^{[footnote 34]} 82 (76%) are from non-Russell Group institutions and 84 (77%) are from outside of London. The following table outlines the proportion of non-Russell Group and non-London based participating universities over the 5 years of the programme. This shows that while there has been some fluctuation between years, non-Russell Group and non-London based participants have always represented the majority in each year, peaking at 87% and 92% respectively.

Table 10: Proportion of non-Russell Group and non-London based participating universities (base number= 108)

Cohort	Proportion of non – Russell Group participating universities	Proportion of non – London based participating universities
Year 1	61%	92%
Year 2	81%	77%
Year 3	87%	80%
Year 4	68%	71%
Year 5	73%	73%

5.2 Performance against KPIs

This section assesses performance against the delivery partner contracted KPIs based on data submitted in the logframes to DCMS (note: in years 2017 – 2020 no formal monitoring KPIs were set).

Table 11: Performance against KPIs

Indicator / Target	2017	2018	2019	2020	2021
1 Teams progressing through programme phases [footnote 35]	Target: N/A Actual: 12	Target: N/A Actual: 26	Target: N/A Actual: 26	Target: N/A Actual: 28	Target: 20 Actual: 21
2 Running total investment attracted by CyberASAP projects / companies (£) [footnote 36]	N/A	N/A	Target: N/A Actual: Year 1-3, over £3.2 million (cumulative total as of 2019) [footnote 37]	Target: N/A Actual: Year 1-4, over £7.5 million (cumulative total as of 2020) [footnote 38]	Target: N/A Actual: Year 1-5, £17.2 million (cumulative total as of March 2022) [footnote 39]
3 CyberASAP Alumni Companies registered (running total) [footnote 40]	Target: N/A Actual: 9	Target: N/A Actual: 9	Target: N/A Actual: 14	Target: N/A Actual: 20	Target: N/A Actual: 22 (as of February 2022)
4 CyberASAP articles published [footnote 41]	N/A	N/A	N/A	N/A	Target: 12 (for whole year) Actual: 33 (as of March 2022)

Source: CyberASAP Programme logframe (March 2022, Year 1-4 Annual Reports, Year 5 final logframe, KTN data provided to RSM UK Consulting LLP.

The latest logframe report (March 2022) and Year 1 to 4 Annual reports indicate that CyberASAP has met its KPIs. The logframe also tracks whether delivery milestones as per the MoU between DCMS and Innovate UK are on track as planned. As of March 2022:

• most delivery milestones for CyberASAP were met
• the only delivery milestone that was not completed as planned was ‘CyberASAP Year six programme confirmation’, i.e. confirmation of whether the programme would continue into its sixth year; this milestone was planned for November 2021 and completed one month later in December 2021
• milestones relating to the Year 6 programme were delayed (specifically: competition promotion, publishing, opening and closing) as DCMS were waiting for the outcome of the Year 3 spending review before proceeding, however are now complete

The logframe does not specify targets for the running total (£) attracted by CyberASAP projects / companies or CyberASAP alumni companies registered.

Feedback from delivery partners suggests while the monitoring and ToC KPIs are helpful to work towards, it is not clear if the KPIs are ambitious or stretching enough as there is no baseline evidence to measure against.

5.3 Performance against the Theory of Change

5.3.1 Activities and outputs

The activities and outputs conducted and achieved over Years 1 to 4 ^{[footnote 42]} based on KTN annual reports include:

Table 12: Activities and outputs

Year	Activities	Outputs
Years 1-4	• selection panels and application assessments • bootcamps and webinars • concept demonstrators / showcase events • events such as: Meet the Entrepreneurs Day, Information Security Europe Show • training such as: presentation skills, sales and commercialisation, meet the investor	• 123 applications, 92 projects • 47 Demonstrator events • 20 companies formed • over £7.5 million in investment secured over Years 1-4, including by private businesses, from EU grants, and Innovate UK • over Years 1-4, at least 3 projects joined HutZero. Other projects joined: IoTWales, BetaBen, Cyber101 (at least 5 projects)
In addition to the above:
1	• business plan development	• demonstrator showcase events for each project in Phase 2 (7 – 6 from original Year 1 cohort, and 1 that joined from ICURe)
2	• pre-announcement registration of interest phase to promote programme	• market validation phase participation by 17 projects • 1 project shortlisted for funding from the Mayor of London’s Civic Innovation Challenge
3	• pre-announcement registration of interest phase to promote programme • Twitter activity and publication of articles • alumni conference	• market validation phase participation by 20 projects • Years 1 to 3 total: three patents registered and three companies who have provided their project outputs in open source format • one project received £200,000 in spinout support from Scottish Enterprise, and another secured £50,000 in funding from Innovate UK to develop a mobile app
4	• pre-announcement registration of interest phase to promote programme • Twitter activity and publication of articles • alumni conference • 1:1 Persona Development Calls	• 20 market validation projects

Across the five years participating teams have successfully attracted investment from a variety of sources, including private business and public sector funds (such as EU grants or Scottish Enterprise support). Other participants have spun out companies successfully, some of whom provided their project outputs as an open-source product and through published articles.

5.3.2 Outcomes

Entry into other incubator / accelerator programmes

The majority of 53 survey respondents (79%, n=42) had not participated in other programmes aimed at supporting the commercialisation of research into cyber security since completing CyberASAP. Of those who had, most (67%, n=6 of the 9 who provided detail on the programmes accessed ^{[footnote 43]}) noted they had accessed other external accelerator programmes.

In addition, most of the participants interviewed who did not progress through the full programme continued onto other programmes, applied for other funding, or decided to reapply to CyberASAP. The other schemes reportedly accessed included:

• ICURe run by SetSquared Partnership
• Scottish Funding Council
• Enterprise fellowship
• LORCA
• Cyber Runway
• Defence and Security Accelerator
• Innovate UK funding
• CyLon
• RCUK (research funding EPSRC)
• university accelerator programmes

Increased number of new cyber security start-ups / spinouts (company registrations)

The following two tables detail the current status and outcomes of all CyberASAP participants from Cohorts 1 – 4 based on data provided by KTN (Cohort 5 data was not yet available). This shows a similar proportional performance between Russell Group and non-Russell Group universities, however a higher number of Russell Group university projects are in development, spun out, licensed and acquired. As shown in Table 14, non-Russell Group projects constitute the majority of those that have raised investment and participated to other accelerator schemes.

Table 13: Current status of participating companies from both Russell Group and non-Russell Group universities (Year 1 – 4)

Status	Number from Russell Group universities (base number=12)	% of total Russell Group universities in the programme (i.e. what % of the Russell group universities spun out or in development	Number from non-Russell Group universities (base number=35)	% of total non-Russell Group universities in the programme (i.e. what % of the Non-Russell group universities spun out or in development)
In development [footnote 44]	5	42%	13	37%
Licensed [footnote 45]	1	8%	1	3%
Spun Out [footnote 46]	4	33%	8	23%
Acquired [footnote 47]	1	8%	1	3%
Unsure [footnote 48]	0	0%	6	17%
Dormant [footnote 49]	0	0%	1	3%
Abandoned [footnote 50]	1	8%	5	14%

Source: CyberASAP Alumni Status Tracking (March 2022) provided to RSM UK by KTN

Table 14: Current outcome of participating companies from both Russell Group and non-Russell Group universities (Year 1 – 4)

Outcomes	Number from Russell Group universities (base number in brackets)	% of total Russell Group universities in the programme (i.e. what % of the Russell group universities raised investment)	Number from non-Russell Group universities (base number in brackets)	% of total non-Russell Group universities in the programme (i.e. what % of the Non-Russell group universities raised investment)
Raised investment [footnote 51]	8 (n= 30)	27%	22 (n= 30)	73%
Patent granted [footnote 52]	2 (n= 4)	50%	2 (n=4)	50%
Other accelerators scheme accessed	7 (n = 17)	41%	10 (n= 17)	59%

Source: CyberASAP Alumni Status Tracking (March 2022) provided to RSM UK Consulting by KTN

There were mixed findings in relation to the programme’s ability to support an increase in company registrations/spinouts with 49% (n=23) of 47 participant survey respondents who answered this question stating their company’s spinout is fully, or partly, attributable to CyberASAP while 51% (n=24) did not feel that CyberASAP was attributable to this process.

However this may be partly due to insufficient time post-CyberASAP to deliver in this area as qualitative feedback includes:

Much of the above is in progress. We only finished CyberASAP in February 2022. We are in the process of moving invention reports to patent applications. We are applying for funding to help with the spinout process.

– CyberASAP survey respondent (Cohort 5 – 2021/22)

[COVID-19] meant all my efforts had to be turned to online teaching which meant I have had no time to commit to developing the business further. I am only now just starting to look at picking this up again.

– CyberASAP survey respondent (Cohort 3 – 2019/20)

Table 15: Has CyberASAP been impactful in supporting participants’ business development process? (base number=51) ^{[footnote 53]}

Answer Choice	Yes	Yes, partly	No
I have spun out a company (n=47)	36% (n=17)	13% (n=6)	51% (n=24)
I have licensed IP from my research (n=46)	17% (n=8)	17% (n=8)	65% (n=30)
My product or service has been acquired by another firm (n=47)	9% (n=4)	4% (n=2)	87% (n=41)
My company has been acquired by another firm (n=46)	4% (n=2)	2% (n=1)	93% (n=43)
I have secured further funding (n=48)	25% (n=12)	19% (n=9)	56% (n=27)
I have employed more people (n=47)	21% (n=10)	4% (n=2)	74% (n=35)
My company is more productive (n=46)	22% (n=10)	11% (n=5)	67% (n=31)
My company’s turnover has risen (n=44)	11% (n=5)	5% (n=2)	84% (n=37)
I have made my technology open source (n=46)	15% (n=7)	4% (n=2)	80% (n=37)

Case Study – Cydon: commercialising academic research

About Cydon

Cydon is an academic start-up company. Their product is a decentralised data management platform which allows regulation of data sharing across organisational boundaries and supply chains. ^{[footnote 54]} They applied to the CyberASAP programme through the University of Wolverhampton and were part of the second cohort of the programme. They have now filed a patent in both the UK and USA.

Probably as academics, we all think of things from the academic point of view and knowledge prior to the programme was restricted as we didn’t know how to take it to the commercial side/business side.

When they began the CyberASAP programme, Cydon was just a project. The academics hoped that the programme would enable them to spinout and knew that receiving commercialisation support was the best way to achieve that.

While one or two other support programmes existed, the entry requirements prohibited them from qualifying, in relation to IP and government clearance. They found CyberASAP to be more inclusive and easier for them to “tap into training and resources.”

Support received from CyberASAP

The academics describe the support they received from CyberASAP as “endless” as it covered a comprehensive range of topics in several different formats. They recall a variety of useful training sessions around topics such as market analysis, sales and IP. There were handouts provided and slides that contained lots of commercial content vital for developing a business from an initial research idea.

As well as the benefit of funding, the academics felt that the most useful part was receiving training that helped academics think with a commercial mindset. They said that lots of support was provided around this during sessions at various times throughout the year.

Attending commercial events such as Infosec and presenting in front of investor panels gave them a platform to increase awareness of their academic research and the potential business that could be developed around it.

We had to create an elevator pitch during programme and commercial planning which was extremely useful.

People from the programme have been described as “very contactable” as the academics felt they were thoroughly supported with follow up questions and issues. The academics said that the CyberASAP team were very responsive especially around business modelling and IP. They found additional support on these aspects to be of great value.

CyberASAP also gave them the opportunity to tap directly into business angels and venture capitalists. This engagement was ‘eye opening’ for the Cydon team and helped them understand where the value lies from a customer standpoint rather than a technical academic’s perspective. Cydon pivoted in phase three of the programme (Note: newer renditions of the programme only have two phases). Working with external entrepreneurs made the academics realise that their target market had minimal potential. While the underlying technology didn’t change, they changed how it was used so that the product could be applied across multiple sectors and tap into larger markets.

CyberASAP helped us understand the business/commercial model around the product and understand market segments and helped us pivot our idea so we can tap into different markets.

The academics felt that the reporting process was tedious and required too much information but say that this has since been improved. They know this as one of the academics has participated in five projects through the CyberASAP programme and can see changes being made over time. While the reporting burden has lessened, they still feel that it’s on the high side. During their time on the programme developing Cydon, they felt that producing a long report each month took vital time away from carrying out business development having to report on a monthly basis.

How CyberASAP could be improved

One issue that persists with the CyberASAP programme is the timeframe. When the Cydon team were involved with the programme there were three phases, beginning with two short phases lasting a few months each. However these phases were too short to allow for the recruitment of researchers who could have supported business development. Unless a team of researchers already exists for a project, the work is left entirely to academics which poses a challenge as they must prioritise teaching and departmental responsibilities over commercialising their research. The Cydon team feel that CyberASAP would find it more beneficial to fund fewer programmes for longer periods. They justify this saying that this would add value to the programme as the spinouts would reach higher maturity. They feel that this is still an issue with the programme. Although there are now two phases in the programme, the academics feel that:

… having a short phase in the beginning [is] a hindrance because when doing [the] market validation phase, you need help setting up focus groups etc.

A member of the Cydon team felt that there was a gap in the content covered by the programme. Providing information on how different universities set up spinouts and IP agreements would have been useful to guide them with how to approach universities. This is around best practice sharing between TTOs and how they are setting up their business.

Another point made by the academics was related to ‘next steps’ post programme participation. There is a belief that the support needs to be extended beyond the demo day to help them reach the next level and be signposted to follow up funding. The academics feel that the programme had brought them close, but not quite to the “finish line”.

CyberASAP is brilliant where it brings us to a point where we are able to do proof of concept/MVP and then just drops off a cliff.

One minor point made by Cydon was that a little more time and resource could be dedicated to business modelling for start-ups to provide a little more detail on this. The academics feel that post participation there needed to be a little more support in shifting away from the hand holding during the programme. One of the team members has fed-back this to the CyberASAP team and an alumni group has been formed in subsequent iterations of the programme.

Impact and benefits of support received

The Cydon team felt support received for the commercialisation of their academic research was most beneficial for them. Another important aspect was gaining access to contacts within the industry. Overall, they have spun out the current company (Cydon) and were granted a patent as a result of participating in CyberASAP. Therefore, the programme enabled them to achieve their initial objective of commercialising their idea.

It was suggested most medium-terms objectives have been fulfilled however ongoing negotiations mean some of these have now become long term, for example selling products and generating turnover. The sudden stop in monetary support and the COVID-19 pandemic made it difficult for the academics to devote time to the project, especially as the project was put on pause during that period.

Lessons learnt from CyberASAP and wider support accessed

One lesson learnt was that the normal scientific approach may not be the way forward when it comes to commercialising a product. The academics now feel more knowledgeable on the vocabulary needed to sell innovation to an audience. They feel that CyberASAP is an “an excellent programme and this is why we kept coming back.”

In the past, Cydon had accessed other programmes that were different to CyberASAP and had never joined again as they didn’t see much value in their participation.

Without CyberASAP, they would have applied to UK grants and searched for support from TTOs. The team wouldn’t have set up Cydon as a company for another three to four years at least without participation in the programme and don’t think they would have had the opportunity to file a patent for the innovation. They have gained access to networks and received publicity that they would not have found elsewhere.

CyberASAP is brilliant because it gives you the right type of information you need and takes you from idea to proof of concept with a business model.

The academics are willing to participate in another Cyber Growth and Innovation programme as they feel it is the next level of support and resource that a new start-up might need, especially one that hasn’t yet received investment. They strongly feel that “nothing else covers the same level of training and grant support” as CyberASAP. They give thanks to the CyberASAP team who were both helpful and passionate. CyberASAP and the team has resulted in Cydon feeling that they have “massively benefitted from this programme.”

Increase in new viable products or services trialled

A number of the 53 participant survey respondents who answered the question on CyberASAP outcomes (45%, n=24) stated their ability to trial a new product or service was a key outcome. In addition, almost all survey respondents felt that the programme supported their development of a value proposition, market validation, and proof of concept.

Table 16: An assessment of the product/service development support provided by CyberASAP (base number= 53)

CyberASAP supported:	Yes	No	Not applicable
A value proposition [footnote 55]	100% (n=52)	0%	0%
Market validation	96% (n=51)	2% (n=1)	2% (n=1)
Proof of concept	92% (n=49)	0%	8% (n=4)

Increased industry input to help shape and validate market relevant technologies, products and services from academia

Of the 54 participant survey respondents answering the question on industry member support, almost all (94%, n=51) felt the support provided by industry members was helpful or very helpful. This is also supported by qualitative feedback which highlights how industry engagement as part of the programme has supported product and company development:

During the programme I met a company which eventually led to our first bit of revenue.

– CyberASAP survey respondent (Cohort 1 – 2017/18)

[The programme was helpful in] establishing useful contacts and new partnerships.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

Case Study – PriSAT: receiving external validation

About PriSAT

This project was involved with the 2019 cohort of the CyberASAP programme. Its goal was to help organisations navigate a range of issues around data protection regulations and aligning business processes with compliance processes, bridging a potential skills gap for academics who wanted to step into entrepreneurship.

Context and challenges faced

Before participating in the CyberASAP programme, the research idea faced some logistical, organisational and cultural challenges. The academic, who now works at a different UK university, faced a challenge with commercialisation due to the previous university’s cultural stance on spinout companies. Academics were expected to be highly involved in primary activities, teaching, research and publishing papers. Therefore, it was challenging to receive recognition from the knowledge transfer office.

Entrepreneurship and research commercialisation had no support structure, so even if you engaged in it, there was no acknowledgement or structure around it, couldn’t dedicate time to it.

The academic involved in this project disagreed with the universities approach and felt that this was a limiting factor. They wanted to put their research into practice and find a practical solution to cyber issues faced in society. The academic has a great interest in expanding the reach of their research beyond peers and hopes to have a wider impact. This was their main motivation for participating in the CyberASAP programme and drawing down the skills needed for entrepreneurship.

The support accessed

The academic feels that CyberASAP has been hugely beneficial in bridging the gaps in their knowledge about spinouts. Training about pitches and engaging with investors provided particularly valuable information which they could apply to their project. The academic also felt that being shown how to use a business model and the IP law training was useful.

Alongside the training courses, having the opportunity to take the project to market was a positive experience. Talking to potential customers and gathering their opinions on the product has been hugely insightful for product development. They feel that this aspect of CyberASAP exposed them to insights that they would not have accessed outside of this environment. They highly praised the support structure of the programme and stated that demonstrating work in front of others pushed them further than other programmes have in the past.

Loved that you were in front of a review panel and defending a piece of work, it was a big challenge, but the preparation process was good for me.

There were parts of the curriculum that they struggled to relate to, but the academic feels that in hindsight these were still relevant. While CyberASAP had a lot of information about the process of using innovation canvas and tools for bringing ideas to fruition, they feel that this could be stretched further. Ideally this could be addressed by integrating training in writing a business plan in the final stage of the programme.

Impact and benefits of support received

Due to the programme, the academic reported becoming more knowledgeable in a variety of business matters while exposure to entrepreneurship has led to an increased understanding of the impact that their research can have.

CyberASAP gave me remit to transfer academic research into real practice.

CyberASAP gave the academic a platform to receive recognition outside of their “usual” circle. ^{[footnote 56]} For example, they spoke to a number of investors which they feel has “opened doors.” The academic feels that, post participation, they have a larger voice within their university.

Without CyberASAP it [spinning out the project] would have just been a pipe dream.

The academic notes that the programme was vital for providing external validation and that it provides academics with a new opportunity. They feel academics do not usually meet the requirements of incubation programmes and are appreciative of CyberASAP’s recognition of this group. Some incubators have age limits attached to the eligibility criteria such as ICURe meaning experienced academics are overlooked on occasion with programmes focusing on post-doctoral students.

Wider supports used beyond the programme

The interviewee also participated in another programme after CyberASAP as a way to reinforce their knowledge. More gaps emerged as the project was put into practice and it was a good opportunity to strengthen the learnings from CyberASAP. The academic highlights that involvement in this programme increased their network of like-minded people. The duration of each programme varied with CyberASAP running over a notably longer time frame. In a way the academic alluded to this programme as being a refresher course which reinforced the CyberASAP teachings and was subsequently put into practice more efficiently than without participation.

The academic described the other programme’s approach to networking, nothing there is a network that sits in one-to-one meetings with them. This differs from the CyberASAP approach where they had to source these contacts more independently and had to mention their involvement in CyberASAP to encourage people to talk to them.

In CyberASAP I had to go out myself and knock-on different investors doors, CyberASAP didn’t bring [the] network to me…

They are interested in participating in other Cyber Growth and Innovation programmes, particularly one by NCSC supported by DCMS and innovate UK. They describe this programme as the “next stage of CyberASAP”. However, the academic feels that they need to choose between it or academia. They recognise the benefits of participating in these programmes but are unsure whether it is worth leaving their employment. At the moment, the academic is more tempted to stay within academia.

Self-reported changes in business knowledge / skills

A majority of participant survey respondents reported that the CyberASAP programme positively impacted their entrepreneurial skills (85%, n=46) and knowledge of setting-up a business (69%, n=37). The programme also impacted on several other areas for 50% or more of respondents, as illustrated in Figure 14.

Figure 14: Areas of impact on the CyberASAP programme (base number = 54)

Case Study – Developer Security Essentials: identifying the right clients

About Developer Security Essentials

Developer Security Essentials is a workshop-based product that helps software development teams improve their software cyber security and privacy by helping software developers to identify the cyber security risks associated with their products and to develop security-related solutions on their own. The team, based at Lancaster University and UCL, has extensive prior experience in the field of cyber security. Through years of both practical and academic engagement, the researchers had found that developers were often unaware of many of the security risks associated with their applications or systems. The researchers also learned that the product managers who control the development teams’ work often do not see the importance of addressing identified security risks.

To address this problem, the team developed a series of workshops to help developers identify the risks present in their product, to find out how to address those risks, and to help product managers assess whether and when to do the work required.

Context and challenges faced

Developer Security Essentials’ team suggested they were already confident in their product, which the team had ‘tried and tested’ with a number of clients in the private and the public sectors. However, the idea of running workshops to solve cyber security-related issues was difficult to sell:

The workshop idea was there, we had run them with some government and private companies… It worked in different ways and all the ways had benefits, but it was hard to sell.

By participating in CyberASAP, the team wanted to identify how best to reach more companies and organisations with the workshops; specifically, to identify which organisations might benefit most and how they could best reach them. This was driven less by a desire for financial return, and more by a wish to gain more users for the workshop. The team aimed to consider how these clients could conduct the workshops independently or with the support of other, trained, consultants.

Impacts of CyberASAP on Developer Security Essentials

CyberASAP has led to several impacts for Developer Security Essentials. The most important for the team has been that a couple of large companies have started to use the workshops. Through CyberASAP, the team also identified that consultancy providers are potential clients. However, these consultants require ‘train the trainer’ training, which the Developer Security Essentials team has developed. To reach even more organisations, training and workshop material was developed and is now available for download through the securedevelopment.org website.

While the team felt they did not gain any new knowledge about how to run a company as they already had extensive experience setting up and running their own businesses prior to their academic career, CyberASAP helped them use their skills and knowledge better:

We didn’t learn anything new about running companies but we learned how to do things the right way. Things like marketing, we had been doing wrong – we learned to use the Business Model Canvas correctly and how to pitch for instance.

How CyberASAP helped Developer Security Essentials

Specific training courses were noted as being the most valuable aspect of CyberASAP. Training on sales pipelines and running sales meetings were highlighted, as the team “didn’t know how to do that before.” One specific training session that the team stressed was conducting a business model canvas, which helped them to focus on potential clients and competitors.

The training courses on how to run a sales pipeline and run a sales meeting were superb.

The competitiveness of the programme was highlighted as beneficial by the Developer Security Essentials team. By requiring participants to present their findings and work completed, as part of progressing through the market validation and proof of concept phases, the programme focused them on applying skills, both new and old, within a compressed timeframe. This approach:

Brought the skills to the new level. I have done programmes in the past but nothing close to what CyberASAP taught me.

What could be improved

Other than COVID-19 related issues, Developer Security Essentials did not have suggestions for improving the programme. One aspect that could be considered in future, given the trend toward hybrid working, is to find a better way to support the development of online communities of participating teams. Potentially, the best way to support this would be to consult with experts in the field of online community building. Going forward, conducting some content face-to-face, while maintaining other aspects of the programme online, was seen as a good proposition.

Other programmes

Other than obtaining related research grant funding, the Developer Security Essentials team has not participated in further support programmes. Other support they have received was not specific to the Developer Security Essentials workshops. In the absence of CyberASAP, the team would have continued to run the workshops and would have sought grants to further develop them. However, they suggested that the focus of CyberASAP on marketing the product itself was a key distinguisher when compared to grant programmes:

That [grant funding] can work but it takes one into specific directions, the great thing about CyberASAP was that it focused on an output.

In fact, the makers of Developer Security Essentials said that they would participate in CyberASAP again. Explaining this, they said:

Because the great thing was not that it taught you a lot (though it did), but that it taught you things in the context of having a particular problem to solve. A great structure for starting a new spinout.

Increased confidence, aspiration, and resilience to form and run businesses (such as using any business skills developed)

Participant feedback suggests the programme provided a mixed impact in these areas with 56% (n=30) feeling it increased their aspirations and confidence to run a business respectively. Positive feedback included:

I’ve met possible collaborators, industry contacts and learned that even if this idea didn’t go forward others might.

– CyberASAP survey respondent (Cohort 3 – 2019/20)

I am still very much active in my research work, but I found myself often exploring the commercial potential of my research work.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

I now look at and appreciate the need for marketing in a very new way.

– CyberASAP survey respondent (Cohort 5 – 2021/22)

I realised / learned how to capitalise on the small things.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

However, more negative views were also expressed on the lack of ability for participants to properly commercialise the concepts developed during the programme, for example:

I think the key in taking this further is the support for individuals in negotiating with the University in relation to IP and shareholding. Alongside this support for the academics in developing skill sets required to assist forcing their products into a crowded market. Without this support academics with great ideas go back to what is a comfortable environment and it can be treated as a further research project when the solutions, some fantastic, can benefit UK plc.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

If possible, I wish to see the programme, not stopping at the POC [proof of concept], but also, to us through a viable commercialisation route and let us then work on the route instead of us relying on the university / tech transfer office to explore the commercialisation route.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

In addition, participants interviewed who did not progress through the full programme indicated that their experience in CyberASAP helped them to further develop their proposition, with one participant noting the panel of experts providing feedback were “not ambitious enough” in valuing their product at £2 billion, suggesting the product could service a wider £24 billion market. This provided the participant’s university with greater confidence in the academic’s research being commercialised, resulting in the university supporting them to apply for further programmes (ICURe). The participant suggested “this may not have been possible without CyberASAP”. Furthermore, some of those who did not progress their initial proposition pivoted project idea due to the feedback and learnings received through the programme, which also increased their confidence in producing a business plan for commercialisation. For example, one participant stated that CyberASAP played a major role in ‘taking the leap’ into entrepreneurship as it meant they were better prepared for discussions with external stakeholders such as investors. The participant suggested the learnings were “sat there in a tool bag”, making the transition away from academia a smoother process a few years after participation in the programme.

Case Study – WalletFind: refining the market fit

About WalletFind

WalletFind is a product that detects software and hardware cryptocurrency wallets on laptops, desktop computers and other electronic devices. The team behind WalletFind consists of academics at the Universities of Salford and Bristol and Manchester Metropolitan University. In addition to the researchers, a project officer, developers and a UI designer worked on the project. The idea for WalletFind arose from discussions with a student who had a background in law enforcement and policing, who said there is a lack of comprehensive tools to investigate cryptocurrency wallets on seized devices. Based on this, the student and the supervisory team started to develop the tool and later applied to CyberASAP for support to improve it.

Context and challenges faced

Following discussions with students, the researchers behind WalletFind developed a tool that would be able to detect the presence of cryptocurrency wallets on electronic devices. The team was aware that there was a potential market gap for such a tool however wanted to explore this in more depth. It was not pure research work, but rather a technical solution to an issue identified through team members’ own experience:

I had a research idea that I could not take forward prior to CyberASAP as it was not pure research work but more a technical solution. When I became aware of CyberASAP, I found it would help me take this work forward and therefore I applied.

Participation in CyberASAP was driven by the need to confirm whether there was a market for WalletFind, and if so, who the potential customers could be. The most notable challenge faced by the team was the lack of commercial skills concerning business case development, market validation, pricing and how to licence products. WalletFind’s motivation in joining CyberASAP was:

To get support to develop the product further and get something ready to go to market.

Impacts of CyberASAP on WalletFind

Through CyberASAP, the WalletFind team was able to conduct market research using surveys and interviews. This confirmed that, beyond police forces, customers could include individuals who had lost access keys to their wallets and e-discovery clients. In addition, this market validation work showed that WalletFind could help companies’ and other organisations’ IT departments to detect unusual activities of employees on company devices.

Not only was the team able to validate the potential market for the tool, it was also able to further develop the product through the proof of concept phase:

Developing a product - so we got it done: this means that the simple tool with one function at the proposal stage is now a proper product with a user interface which can be shown to investors.

Finally, through meeting the KTN team and other academic teams, the network of WalletFind researchers / developers expanded. Discussions with this network as well as the training and support provided by KTN through CyberASAP developed the team’s confidence in their business skills and ability to start a business.

How CyberASAP helped WalletFind

WalletFind accessed all the support provided by CyberASAP: from training in workshops and seminars, through to networking and financial support. The team benefited most from the in-person events that took place in the second phase of CyberASAP (the proof of concept phase, for example). While acknowledging that COVID-19 meant that events in the first phase of the programme were limited to virtual delivery methods, the WalletFind team said that they were more engaged and simulated at face-to-face training sessions. During the second phase, the team refined how they presented their product to potential investors:

In the second phase we did presentation work, it helped us formulate the way we want to present our product and meet a lot of people from different universities with different backgrounds.

In the full day bootcamps that formed part of the CyberASAP programme, the team was able to hear from investors and the business community, rather than only from other researchers. This exposure helped to develop their presentation skills, but also led to stronger understanding of what is important to potential investors. One of the angel investors that the team met during CyberASAP subsequently approached the participants to explore additional collaborations.

What could be improved

While the bootcamp, full-day training sessions were positively received due to the knowledge and skills developed, they were also intense and challenging to complete. Due to COVID-19, full-day training had to take place virtually. This limited face-to-face interaction:

You see people behind the screen, completely different than when you see people there.

In addition, having to listen to presentations and in turn deliver presentations virtually was perceived as “intense and challenging”. To reduce this pressure, the WalletFind team suggested that virtual bootcamps could be split over multiple days.

WalletFind felt that the IP training, in particular, could have been improved: in the team’s view, the presentation was long, but not specific enough. To improve specificity, the team suggested that use cases should be included in similar IP training sessions.

In addition, it was suggested that the budget provided to each team could have been higher. Academics, especially during the first teaching semester during which the programme took place, are managing multiple responsibilities. Therefore, a combination of more funding to pay for additional programme days and a different delivery window within the academic year should be considered.

WalletFind would access other programmes that are relevant

The WalletFind team has not participated in other, similar programmes but was open to finding out more about available options. In the absence of CyberASAP, they stressed that the idea would have remained a research topic. It would not have developed into a fully-fledged proof of concept. To further develop the product and understanding of IP and licensing processes, the WalletFind team was open to participating in similar programmes again.

Enhanced business viability and short-term investment

In order to properly understand how the programme influenced the level of short-term investment into participating companies, it is important to separate the different forms of funding/investment provided for them. This section includes an assessment of three distinct areas, including:

(1) investment into participating companies

(2) government grant funding

(3) in-kind investment

Investment into participating companies

Most of the 53 survey respondents (77% n=41) have not received investment following the completion of the programme. The different levels of investment received by the 23% (n=12) who did are shown in Figure 15, highlighting that only a small number of participants received funding under £101,000. The most common investment level was over £500,000.

Figure 15: Levels of investment into participating companies (base number= 12)

Government grant funding

85% (n=45 of 53 respondents) stated they had not received any grant funding, mirroring the trend identified in the previous assessment of investment figures (investment into participating companies). The most common form of further funding access was from Innovate UK. Overall, 88% (n=7) of the 8 respondents receiving the funding reported to have been given over £100,000, with only 13% (n=1) selecting a different level of funding (£75,000-£100,000). ^{[footnote 57]}

In-kind investment

92% (n=49 of 53 respondents) stated they had not received any in-kind investment, with qualitative feedback indicating that investment came in various forms including:

Credit for use of Amazon Cloud (provided as part of Cyber101 program).

– CyberASAP survey respondent (Cohort 3 – 2019/20)

Grant for market research and demonstration equipment.

– CyberASAP survey respondent (Cohort 1 – 2017/18)

Time from business partners.

– CyberASAP survey respondent (Cohort 3 – 2019/20)

Increased / improved engagement with UKRI

The key areas of engagement between the participants and UKRI were via KTN and grant funding provided by Innovate UK. Participant survey respondents provided positive feedback on their experience with KTN, with 94% (n=51) of the 54 stating their support was helpful or very helpful. Qualitative feedback included:

Consistency in expert KTN staff supporting.

– CyberASAP survey respondent (Cohort 2 – 2018/19)

KTN trainers were very good.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

KTN worked hard to build a community, which I am still meeting and getting to know.

– CyberASAP survey respondent (Cohort 4 – 2020/21)

Assessment of grant funding sources found that funding most commonly came from Innovate UK. This suggests engagement with UKRI continued beyond the CyberASAP programme.

5.3.3 Impacts

Productive companies with turnover / revenue

Most participant survey respondents suggested that turnover and productivity increases were not attributable to CyberASAP. Specifically, only 16% (n=7) of 44 respondents felt that any turnover increase was either fully or partly attributable to the programme. While this figure is higher for productivity increase (33%, n=15 of 46 respondents), it remains in the minority.

Wider commercialisation of academic research and ideas - drawing technical expertise & capabilities out of universities

89% (n=47 of 53 respondents) noted that their commercial awareness had improved as a result of the programme. When asked to rank their capability from 1 (very weak) to 10 (very strong) 66% (n=35) of the 53 respondents selected a number between 8 and 10 and only 4% (n=2) selected a number lower than 6.

Greater diversity and inclusivity across the cyber sector

Analysis of the CyberASAP participant survey respondent profile shows a weighting towards both male participants (84%, n=46) and those not from an ethnic minority background (62%, n=34). However, 29% of the 55 respondents (n=16) are from an ethnic minority background (all others did not disclose this information), which is higher than both the cyber sector (25%) and UK workforce (12%) figures. ^{[footnote 58]}

Though the relevance of this demographic data is limited by the fact that it only includes information provided by survey respondents, the programme is unlikely to contribute in a meaningful way to wider industry demographic trends around diversity and inclusivity if it is unable to develop a diverse ecosystem itself.

Analysis of other ToC outcome areas

Participant survey feedback suggests that it is too early to have evidence of performance against other ToC outcome areas such as:

• greater acceleration, incubation and growth in the UK cyber sector
• new technologies, products and services adopted in the UK and internationally
• greater early years survival rates

5.4 Additionality

Some of the participants interviewed who did not progress through the full programme reported they would not have decided to pursue a spinout company had they not participated in the programme. For example, one participant highlighted that the programme showed them how to commercialise research ideas and how to approach investors which seemed daunting beforehand. In addition, other participants suggested the programme built their “interest in entrepreneurship” and “pushed me to think that I want to be in business” while another participant highlighted that:

We were not successful for phase 2 of the CyberASAP. We pivoted in our idea and went for it. Which was the best decision we made, and now we have a running start up with seven full-time staff members.

– CyberASAP survey respondent (Cohort 3 – 2019/20)

5.5 Contribution analysis

Based on the ToC outcomes and impacts, the following contribution statements describe the results to which CyberASAP is expected to contribute. This includes the programme:

• creating new cyber security start-ups or spinouts
• helping project teams increase their successful trials of new products or services
• improving the business knowledge and skills of participating teams, along with their confidence to run businesses
• improving the business viability of spin-outs and the short-term investment they receive
• leading to participants entering other incubator programmes
• increasing input into the work of academia to help shape and validate technologies

The analysis also considers evidence of other factors that contribute to the observed result, with:

• strong contribution indicating that CyberASAP has achieved substantial results with few or no other contributing factors
• some contribution indicating that CyberASAP has achieved some, but no substantial results with evidence that other contribution factors are at play
• negligible contribution indicating that CyberASAP has not or not yet achieved any or only very limited results or that the results are effects of other contributing factors

In addition, the contribution analysis assesses the strength of evidence underpinning the contribution findings as follows, as shown by the fact that:

• strong evidence is evidence from multiple sources that is clear with limited need for interpretation or prompting
• weak evidence indicates that participants and others discussed the result but that there is either only limited evidence for it materialising or that the evidence needed interpretation or prompts
• negligible evidence indicates that there is no or very little evidence for the result

Using evidence gathered through interviews, surveys and case studies, and reviewing documentation provided by DCMS, the evaluation has found the following evidence of CyberASAP’s contribution to each of these results:

Table 17: Contribution analysis

Contribution statement	Evidence and other contributing factors	Strength of CyberASAP’s contribution to the result	Strength of evidence
CyberASAP support leads to more new cyber security start-ups or spinouts.	CyberASAP survey participants presented a mixed perspective on the programme’s role in their spinout. Of the 47 respondents who answered the question on whether their spinout could be attributed to CyberASAP, 49% (n=23) felt it either could be fully, or partly, attributed. The other 51% (n=24), however, felt it was not attributable at all. However, amongst those participants who did not progress to the final stages of the programme, many interviewees suggested CyberASAP as a contributing factor to spinning out a company and in some cases leaving academia. There was no evidence of other contributing factors.	Some contribution: about half the participant survey responses suggested CyberASAP had a role in their successful spinouts. Among those who did not progress through all programme stages, many suggested that CyberASAP contributed to their eventual spinout. It should be noted that a spinout was not a requirement of participating teams.	Strong evidence, which is based on survey responses and interviews.
CyberASAP helps project teams increase their successful trials of new products or services.	Evidence from CyberASAP participant survey analysis suggests the programme has been relatively successful in contributing to new products and services being trialled, with 45% (n=24) of 53 survey respondents selecting new products or services as one of the key outcomes from their participation in the programme. Respondents felt the programme helped to develop the basic skills needed to create these new products and services (value proposition, market validation and proof of concept) with 100% of the 52 respondents [footnote 59] reportedly developing a value proposition as a result of CyberASAP, 96% (n=51 of 53) developing market validation, and 92% (n=49 of 53), developing a proof of concept. There was no evidence of other contributing factors.	Strong contribution: CyberASAP has resulted in the majority of survey respondents developing value propositions, market validations and proof of concepts. In turn, almost half developed new products or services. This suggests that the programme has provided the core knowledge and skills base required for product and service development and trialling.	Strong evidence, based on survey responses from participants and KTN data.
CyberASAP improves the business knowledge and skills of participating teams, along with their confidence to run businesses.	Analysis of CyberASAP participant survey results suggests strong performance in this area, with 85% (n=46) of 54 respondents feeling that entrepreneurial skills, and 56% (n=30) feeling that business management skills, had been improved by participation on the programme. This, combined with 89% (n=47 of 53 respondents) feeling that commercial awareness was impacted by the programme, further highlights the strength of programme performance in this area of business knowledge and skill development. In addition, 66% (n=35) of 53 respondents ranked their capability to commercialise research following CyberASAP between 8 and 10 (10 meaning very strong), and only 4% (n=2) of respondents ranking their capability lower than a 6. Those teams who did not progress beyond the first stage of the programme also highlighted the programme’s impact on their business acumen. There was no evidence of other contributing factors and case studies support the survey findings.	Strong contribution: evidence suggests the programme developed entrepreneurial skills even for those who did not progress through to final stages. It also developed skills and confidence needed to form and run businesses.	Strong evidence, based on interviews, case studies, and survey responses.
CyberASAP improves the business viability of spinouts and the short-term investment they receive	The CyberASAP participant survey analysis found the majority of the 53 respondents (77%, n=41) did not receive investment following completion of the programme. Of the 12 respondents who did receive investment, the most commonly selected amount was over £500,000 (selected by 33%, n=4). In addition, 85% (n=45) of 53 respondents noted they had not accessed government grant funding while 92% (n=49) had not received any in-kind investment. Case studies also support this finding.	Some contribution: there is evidence from KTN data, surveys and interviews that programme participants have received additional short-term investment. However it is not clear how strongly the programme contributed to this, compared to other factors. Most participants did not report receiving investment after the programme or participating in other programmes.	Strong evidence, which is based on KTN data, survey responses and case studies.
CyberASAP leads to participants entering other incubator programmes	Analysis of the CyberASAP participant survey results shows that the majority of the 53 respondents who answered this question (79%, n=42) have not participated in programmes aimed at supporting the commercialisation of research into cyber security since completing CyberASAP. Of those who had, most (67%, n=6 of the 9 who provided detail on the programmes accessed) noted they had accessed other external accelerator programmes and 33% (n=3) selected ‘other’. Free-text responses expanding on this selection did not have any repeated answers, but cyber101, CyLon and HutZero were all mentioned. Most participants who did not progress to the final stages continued into other programmes, applied for other funding, or decided to reapply to CyberASAP. Delivery partners felt there was a need for more formal cross-pollination across cyber programmes.	Negligible contribution: CyberASAP participants who completed the programme tended not to go on into other programmes. However, those who did not progress through all stages said they re-applied for CyberASAP or sought funding elsewhere.	Strong evidence, based on survey responses and interviews.
Industry increases its input into work of academia to help shape and validate technologies	A majority of the 54 CyberASAP survey respondents felt that support offered by industry members was very helpful (59%, n=32) or helpful (35%, n=19). However, there was limited detail on how industry input helped to shape and validate technologies in the interviews or case studies.	Negligible contribution: while participants valued the input of industry experts to the programme, there was little evidence of systematic shifts in terms of industry engagement with academia on a regular basis.	Strong evidence, based on survey responses, case studies and interviews.

5.6 Summary of key findings

In total, 113 projects have successfully applied to the programme and c. £17.2 million in additional investment has been raised by participants to date. ^{[footnote 60]} Participants have highlighted a number of benefits as a result of their participation including their ability to trial a new product or service, engagement and development of relationships with industry members, development of their entrepreneurial skills, and increased knowledge of how to start a business. However, while participants noted that the programme had resulted in improved commercial awareness, only a minority felt any reported increase in turnover and productivity were attributable to CyberASAP. Alumni tracking by the delivery partner has already commenced and this should be continued to evidence any longer-term impacts for participating companies.

Based on contribution analysis conducted for CyberASAP, programme support has had:

• strong contribution to two areas: project teams increasing their successful trials and to their business knowledge and skills
• some contribution to new cyber security start-ups or spinouts and short-term investment in these
• negligible contribution to participating teams entering other incubators and increasing the input of industry into academia

The programme has experienced underspends against the initial budget however this was anticipated by DCMS based on previous experience of working with academic teams.

6. Conclusions and recommendations

This section outlines conclusions based on the evidence collected against the evaluation questions and recommendations to inform future programmes.

6.1 Process evaluation

Was the programme delivered as intended?

The programme has evolved each year and delivery has taken place as intended in the Memorandum of Understanding (MoU) between DCMS and Innovate UK.

What worked well, or less well, for whom and why?

• the application process as participants indicated they were provided with sufficient information prior to applying for CyberASAP and felt the process was straightforward
• a delivery structure which included a part-time and stage gated process ^{[footnote 61]} combined with a digital and hybrid model to support participants with a range of abilities and from a variety of locations
• the provision of support for value proposition, market validation and proof of concept development
• the provision of opportunities for participants to meet and engage with key industry figures and entrepreneurs which resulted in the development of tangible business relationships
• improved entrepreneurial and business management skills
• that those who did not proceed to phase 1(b) or phase 2 still benefited from the programme as it led them to re-think or pivot their idea to a more viable proposition
• the development of a strong ‘community’ who keep in touch and support each other
• that participating teams have successfully attracted investment from a variety of sources, including private business and public sector funds
• 89% (n=47 of 53 survey respondents) feeling their commercial awareness had improved following participation
• the geographical diversity of participants, as shown by 64% (n=35) of the 55 survey respondents being located outside of London and the South-east

There were also several examples of areas for development. This included:

• the need for more time between DCMS funding being confirmed, publishing the call for programme applications and commencing delivery to promote the programme with key target groups
• the need for greater support from university technology transfer officers and flexibility with internal university policies ^{[footnote 62]}
• the need for increased linking of the various cyber programmes available

Recommendation 1: we recommend that decisions to fund CyberASAP are communicated at least 9 months in advance (rather than the current timeframe of c. 4 months) to allow sufficient time for programme promotion with potential entrepreneurs from underrepresented groups.

Recommendation 2: we recommend that CyberASAP engages with university Technology Transfer Offices (TTOs) to identify potential candidates for the programme.

What can be learned from the delivery methods used? / Were there any unexpected or unintended issues in the delivery of the intervention? / How did external factors influence the delivery and functioning of the programme?

The majority of survey respondents (98%, n=54) were satisfied or very satisfied with the programme structure.

However, COVID-19 limited face-to-face interaction and qualitative feedback suggested this was detrimental to relationship building between participants and with investors. Other factors that impacted delivery across all years of the programme included that:

• academics involved in teaching had limited time to develop their idea / business
• a lack of engagement from the university TTOs restricted awareness and engagement with the programme by academics, and in some instances meant any issues for the university were only raised at phase 2

How can the existing programme be improved to become more effective?

The existing programme could be improved by:

• issuing a thematic call for project proposals focused on market failures in the cyber security sector (in addition to the current format for an open call). This call would be designed to complement other DCMS-funded programmes and external funding calls while also aligning with the needs of industry
• moving delivery of the programme to less busy times of the academic calendar which would improve the programme for participating teams as they are less likely to have teaching responsibilities
• confirming funding earlier in the year or providing funding for more than one year at a time
• improving monitoring and reporting through the development of SMART ^{[footnote 63]} targets for all outcome and impact measures in the Theory of Change (ToC)

Recommendation 3: we recommend that SMART outcomes are further developed for CyberASAP and there is detailed reporting on these. KTN to develop based on the CyberASAP ToC metrics and agree with DCMS. Evidence should be collected by KTN for progress against each using both published data and primary research with participants. For example, those noted in the DCMS MoU with KTN for Year 6 and including:

|Outcome measures| |Measure 1: number of new cyber security start-ups / spinouts (company registrations) attributed to CyberASAP
Evidence source: published datasets| |Measure 2: number of viable products or services trialled and attributed to CyberASAP
Evidence source: participant feedback via surveys or interviews| |Measure 3: increased confidence and skills to deliver cyber security start-ups / spinouts
Evidence source: participant feedback via surveys or interviews| |Measure 4: increased business survival rates for cyber start-ups / spinouts supported by CyberASAP
Evidence source: published datasets| |Measure 5: increased investment in cyber security start-ups / spinouts supported by CyberASAP
Evidence source: published datasets and participant feedback via surveys or interviews

6.2 Impact evaluation

The programme has delivered business impacts. To date 108 projects have been supported by CyberASAP ^{[footnote 64]} and of these 12 were categorised as spinouts; 2 were acquired by other firms and 5 developed a patent.

The DCMS funding has leveraged £12,382,895 in additional investment (for example corporate / private investment, acquisition, VC investment, angel investment, and seed funding / equity fundraising).

Programme participants were able to meet with organisations, however it will take time to understand the impacts from these meetings and any relationships.

It has also been successful in developing entrepreneurial skills and confidence, including for participants who did not progress to the proof-of-concept stage however completed part (a) or all of phase 1.

To what extent has the programme been successful in commercialising academic research / accelerated the process to commercialisation?

Overall, 26% (n=12 of 47 tracked participants) have spun out companies successfully ^{[footnote 65]}, some of whom provided their project outputs as an open-source product and through published articles. In addition, survey respondents reported an increased capability to commercialise their research following participation in the programme as:

• prior to participating in the programme 11% (n= 6) of 54 respondents ranked their capability to commercialise research between 8 and 10
• following participation 66% (n= 35) of 53 respondents ranked their capability to commercialise research between 8 and 10 (10 meaning very strong) ^{[footnote 66]}

Nevertheless, participants still found it difficult to commercialise their research after the programme had finished and felt that further support is required, either from their university and / or by having further support CyberASAP. Specific examples included ensuring that universities and TTO staff maintain commitment to commercialisation throughout and are as committed to the commercialisation of the Intellectual Property (IP) as the investors. Based on the contribution analysis conducted CyberASAP has had:

• strong contribution to project teams increasing their successful trials and to their business knowledge and skills
• some contribution to the development of new cyber security start-ups or spinouts and short-term investment in these
• negligible contribution to the improvement of participating teams’ ability to enter other incubators and increase input of industry into academia

While 77% (n=41) of the 53 respondents who answered the question had not yet received investment following the programme, some of those will only be at the stage of seeking investment. Therefore, the above contribution analysis is only part of the answer. It will be important to continue tracking progress of the projects after the programme ends to measure longer term impacts.

Recommendation 4: we recommend that the outcomes and impacts from the programme continue to be tracked. It should be a requirement within the delivery partner contract to provide evidence of the capability and business outcomes being achieved for participants at one, two and three years after completing the programme to provide evidence of the longer-term benefits.

We recommend DCMS set up a monitoring template that covers all the outcome and impact measures expected from the programme. This should be completed by the delivery partner for participants when they finish the programme and then at 6, 12 and 24 months after completion.

To what extent does the evidence suggest future funding would be more effective if targeted differently (at less well funded universities, for example)?

The premise of the current CyberASAP model is to attract and identify the most promising commercial opportunities from different parts of the UK academic research base and to make funding opportunities accessible to universities from all regions as well as those outside the Academic Centres of Excellence in Cyber Security Research (ACE-CSR) and Russell Group.

The project has been successful in achieving this as 82 of the 108 (76%) participating universities from non-Russell Group institutions and 84 (77%) outside of London.

Due to the programme both reaching several non-Russell Group universities and there being a lack of notable difference in outcomes between Russell and non-Russell Group universities (as discussed below), a focus on the best commercial opportunities should remain.

To what extent does the success of the programme differ between different cohorts, types of firm/idea, university?

There is no notable difference in outcomes between cohorts, projects or between universities, including Russell Group and non-Russell Group institutions (with one exception). The only notable variations were:

• Cohort 5 – survey respondents from this cohort reported fewer intended outcomes as identified from the ToC being achieved to date. This is likely due to their more recent completion of the programme compared to preceding cohorts
• Cohort 1 – survey respondents from this cohort reported a greater number of spinouts and receiving further funding. This could be due to (1) having completed the programme c. 4 years previously and (2) the design of the programme in Year 1 meant there was an obligation to spinout/form a company

In addition, there is a similar proportion of the 108 Russell Group and non-Russell Group university projects that are in development, licensed and acquired. ^{[footnote 67]} However, a higher difference in the amount of spin-outs achieved was identified between the two groups. KTN feedback suggests this is likely to be the result of Russell Group universities typically being better funded and possessing mature and well-established commercialisation capacities. ^{[footnote 68]} This is shown by the proportion of Russell and non-Russell group universities that have:

• reached development stage (Russell Group: 42%; Non-Russell Group 40%)
• spun out (Russell Group: 33%; Non-Russell Group 23%)
• been licensed (Russell Group: 8%; Non-Russell Group 3%)
• been acquired (Russell Group: 8%; Non-Russell Group 3%)

Do participants join other cyber growth programmes after completing CyberASAP?

Evidence obtained from stakeholder interviews indicated that most of the CyberASAP participants who fully completed all phases of the programme did not go on to other cyber growth programmes. Those who did not progress through all stages of CyberASAP tended to either (a) re-apply for CyberASAP; (b) progress to the London Office for Cybersecurity Advancement (LORCA); (c) progress to the Cyber Runway programmes, or (d) access other sources of funding / development programmes. This was in part due to CyberASAP helping them to better understand the schemes available, suggesting the programme plays an important role in the wider cyber growth and innovation ecosystem.

What are the additional or unintended benefits of the programme?

Delivery partner feedback suggests the programme has helped to change the way universities approach commercialisation as some are now more willing to be pragmatic about how much equity they will receive. For example, Royal Holloway University applied a policy which assigned IP to the start-up company in exchange for an under 10% share to make the deal more attractive to investors. ^{[footnote 69]} KTN have also suggested the programme is starting to see repeat participation from universities who are beginning to change their approach in this area.

Acronyms and glossary

Accelerator	Provides early-stage companies that already have a minimum viable product with the education, resources and mentorship needed to promote what might otherwise be several slow years of growth into a few short months.
ACE-CSR	Academic Centres of Excellence in Cyber Security Research
BEIS	Department for Business, Energy and Industrial Strategy
CSIT	Centre for Secure Information Technologies
CyberASAP	Cyber Security Academic Start-up Accelerator Programme: a pre-seed accelerator programme to support cyber security innovation and commercialisation
Cyber Runway	A programme of cyber accelerators and bootcamps with three workstreams: ‘Launch’, developing proposals and establishing new businesses; ‘Grow’, providing business skills to help start-ups survive and grow, and ‘Scale’, building skills and networks to address barriers to growth nationally and abroad.
CyLon	Early stage investment company
DCMS	Department for Digital, Culture Media and Sport
DSbD	Digital Security by Design
EPSRC	Engineering and Physical Sciences Research Council
HutZero	A bootcamp aimed at transforming early-stage ideas into viable propositions and potential new businesses
ICURe	Innovation to Commercialisation of University Research
ISCF	Industrial Strategy Challenge Fund
Innovate UK	United Kingdom’s innovation agency, which provides money and support to organisations to make new products and services.
IoT	Internet of Things
IP	Intellectual Property
KPI	Key Performance Indicator
KTN	Knowledge Transfer Network
KTP	Knowledge Transfer Partnership
LORCA	London Office for Rapid Cybersecurity Advancement
MoU	Memorandum of Understanding
MVP	Minimum Viable Product
NCSC	The National Cyber Security Centre
PETRAS	Privacy, Ethics, Trust, Reliability, Acceptability, and Security
R&D	Research and Development
RCUK	Research Councils UK
SDTaP	Security of Digital Technology at the Periphery
SMART	Specific, measurable, achievable, relevant, and time-bound
SME	Small and Medium-sized Enterprise
ToC	Theory of Change
TTO	Technology Transfer Office
UKC3	UK Cyber Cluster Collaboration
UKRI	UK Research and Innovation

Appendix A - evaluation questions

Evaluation questions

Process evaluation questions: • was the programme delivered as intended? • what worked well, or less well, for whom and why? • were there any unexpected or unintended issues in the delivery of the intervention? • what can be learned from the delivery methods used? • how did external factors influence the delivery and functioning of the programme? • how can the existing programme be improved to become more effective?

Impact evaluation questions: To what extent: • has the programme been successful in commercialising academic research • has the programme accelerated the process to commercialisation • does evidence suggest future funding would be more effective if targeted differently (at less well funded universities, for example) • does the success of the programme differ between different cohorts, types of firm/idea, university • do participants join other cyber growth programmes after completing CyberASAP? • what are the additional or unintended benefits of the programme (improved commercial awareness, better inter-university collaboration, improved commercial knowledge of university knowledge exchange teams, private sector investment, patents, licences, open-source software)?

Appendix B - theory of change

9.1 Introduction

A ToC explains how activities undertaken by a programme could contribute to outcomes and impacts. It includes linkages between:

• inputs: resources required to deliver the programme
• activities: what is delivered
• outputs: what the recipient receives from the resources or intervention (direct benefits)
• outcomes: long term results of activities and outputs achieved; and
• impacts: wider economic and social outcomes

9.2 CyberASAP theory of change

The initial ToC was informed by the following complementary approaches:

• review of programme documentation outlining the aims, objectives and anticipated outputs, outcomes and impacts for the programme
• review of existing CyberASAP ToCs;
• an internal workshop with DCMS stakeholders including those responsible for the design and delivery of the programme

The ToC is outlined on the following page.

Figure 16: CyberASAP ToC

Appendix C - CyberASAP participant survey

Background information

Q1. Who was the project leader (or leaders) for the project? Please provide the name(s).

Q2. What was the name of the research project you applied to CyberASAP for commercialisation support?

Q3. What was your role on the project?

Q4. What was your university or research institute affiliation at the time of application?

Q5. What is your gender?

☐ Female

☐ Male

☐ Other:

☐ Prefer not to say

Q6. Are you from an ethnic minority background?

☐ Yes

☐ No

☐ Prefer not to say

Q7. On a scale of 1 to 10, how strong was your capability to commercialise research into cyber security before your participation in CyberASAP? SCALE 1 (lowest) to 10 (highest).

Q8. CyberASAP consists of the following phases - Phase 1a: Value Proposition; Phase 1b: Market Validation; Phase 2: Proof of concept. The next questions are about your experience of the phases you went through before joining CyberASAP. What was your previous experience of developing Value Propositions before your participation in CyberASAP?

☐ Outstanding

☐ Very strong

☐ Competent

☐ Underdeveloped

☐ Not developed

☐ Don’t know / unsure

Q9. What was your previous experience of Market Validation?

☐ Outstanding

☐ Very strong

☐ Competent

☐ Underdeveloped

☐ Not developed

☐ Don’t know / unsure

Q10. What was your previous experience of developing proof of concept?

☐ Outstanding

☐ Very strong

☐ Competent

☐ Underdeveloped

☐ Not developed

☐ Don’t know / unsure

Q11. How many researchers are working/have worked with you on the commercialisation of your project through CyberASAP?

☐ 0

☐ 1

☐ 2

☐ 3

☐ 4

☐ 5 or more

☐ Prefer not to say

How many full-time equivalent researchers are/were on the project, including yourself?

Application and reporting experience

Q12. How were you first made aware of CyberASAP?

☐ Direct contact by DCMS

☐ Government website

☐ Knowledge Transfer Network (KTN)

☐ Your university

☐ Other researchers

☐ Industry contact

☐ Social media

☐ Print media

☐ Unsure

☐ Other:

Q13. Please provide some detail about why you decided to apply for CyberASAP:

Q14. To what extent do you agree or disagree that you had the right information to inform your decision about whether to apply for CyberASAP?

☐ Strongly agree

☐ Agree

☐ Neither agree nor disagree

☐ Disagree

☐ Strongly disagree

Q15. What other information about CyberASAP would have been useful to have prior to applying?

Q16. Was the application process straightforward to complete overall?

☐ Yes

☐ No

If not: why not?

Q17. How satisfied or dissatisfied were you with the following aspects of the application process?

	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied	Not sure N/A
How the programme purpose was described
How the programme content was described
Amount of information you had to submit
Eligibility criteria
Support provided during the application process
Communication from KTN/ Innovate UK during the application process
Communication about the application outcome

Q18. What impact did the following have on the application process?

The following made it…	Very easy to apply	Easy to apply	Neither easy nor difficult to apply	Difficult to apply	Very difficult to apply	Not sure N/A
Guidance on application process
Clarity of the application form
Collating the information required to complete the application
Time to complete the application
Clarity of the eligibility criteria
Support from University
Other

If other, please detail any other factors which made it difficult to apply to the scheme:

Q19. As part of your participation in CyberASAP you were required to provide regular reports. How satisfied are/were you with the reporting arrangements on CyberASAP?

	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied	Not sure N/A
Frequency of reporting
Amount of information to provide
Clarity of requirements

Q20. Do you have any other comments on applying for CyberASAP and the reporting requirements?

CyberASAP programme structure

Q21. The CyberASAP programme provided assistance with value proposition development, market validation of that value proposition, and proof of concept development. How satisfied or dissatisfied were you with this overall structure?

☐ Very satisfied

☐ Satisfied

☐ Neither satisfied nor dissatisfied

☐ Dissatisfied

☐ Very dissatisfied

Q22. How satisfied were you with the phases of CyberASAP that you took part in?

	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied	Not sure N/A
Value Proposition Development
Market Validation of the value proposition
Proof of Concept Development

Q23. CyberASAP participants receive funding to take part in the programme. How satisfied or dissatisfied are you that the amount of time and resources spent on the programme was matched by benefits to your project?

	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied	Not sure N/A
Time
Money
People

Quality of support provided

Q24. Have you participated in phase 2 of CyberASAP, the development and demonstration of a proof of concept?

☐ Yes

☐ No

Q25. What support was available to you through the CyberASAP programme? Tick all that apply. [Multiple choices possible] IF Q24=No

☐ Training for value proposition development

☐ Materials for value proposition development

☐ Support for value proposition development

☐ Training for market validation

☐ Materials for market validation

☐ Support for market validation

☐ Grant funding

☐ 2-day bootcamps

☐ Selection panel assessments (including Q&A and feedback)

☐ Mentoring sessions

☐ Legal training (including IP training)

☐ Peer to peer learning

☐ All of the above

Q26. How satisfied or dissatisfied were you with the following aspects of the CyberASAP programme? IF Q24=No

	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied	Not sure N/A
Training for value proposition development
Materials for value proposition development
Support for value proposition development
Training for market validation
Materials for market validation
Support for market validation
Grant funding
2-day bootcamps
Selection panel assessments (including Q&A and feedback)
Mentoring sessions
Legal training (including IP training)
Peer to peer learning

Q27. What support was available to you through the CyberASAP programme? Tick all that apply. [Multiple choices possible] If Q24=Yes

☐ Training for value proposition development

☐ Materials for value proposition development

☐ Support for value proposition development

☐ Training for market validation

☐ Materials for market validation

☐ Support for market validation

☐ Grant funding

☐ 2-day bootcamps

☐ Selection panel assessments (including Q&A and feedback)

☐ Mentoring sessions

☐ Legal training (including IP training)

☐ Peer to peer learning

☐ All of the above

Q28. How satisfied or dissatisfied were you with the following aspects of the CyberASAP programme? IF Q24=Yes

	Very satisfied	Satisfied	Neither satisfied nor dissatisfied	Dissatisfied	Very dissatisfied	Not sure N/A
Training for value proposition development
Materials for value proposition development
Support for value proposition development
Training for market validation
Materials for market validation
Support for market validation
Grant funding
2-day bootcamps
Selection panel assessments (including Q&A and feedback)
Mentoring sessions
Industry showcase demo day
Peer to peer learning
Legal training (including IP training)
Training for proof of concept development
Materials for proof of concept development
Support for proof of concept development

External factors

Q29. To what extent has Covid-19 affected your experience of CyberASAP?

☐ To a large extent

☐ To a moderate extent

☐ To a small extent

☐ Not at all

☐ Don’t know / unsure

Q30. If it has had an effect, please describe how Covid-19 has affected your experience of CyberASAP either positively or negatively?

Q31. To what extent did support from the university’s Tech Transfer Office (or equivalent) help you?

☐ To a large extent

☐ To a moderate extent

☐ To a small extent

☐ Not at all

☐ Don’t know / unsure

Please provide detail on how the support from the Tech Transfer Office (or equivalent) helped you.

Q32. Did you encounter any of the following issues which affected programme delivery, during your participation in CyberASAP? Tick all that apply. [Multiple responses possible]

☐ Recruitment difficulties

☐ Challenges onboarding people into the team

☐ Internal University policies

☐ Prefer not to say

☐ Other

☐ None

If other, please provide details of what these factors were:

Q33. If relevant, what was the effect of these factors on programme delivery?

Q34. How would you describe the commercialisation knowledge of KTN staff who supported you?

☐ Very helpful

☐ Helpful

☐ Neutral

☐ Not helpful

☐ Very unhelpful

If not helpful, please explain why:

Q35. How would you describe the knowledge of industry specialists, entrepreneurs and investors who supported you?

☐ Very helpful

☐ Helpful

☐ Neutral

☐ Not helpful

☐ Very unhelpful

If not helpful, please explain why:

Q36. In your view, were there gaps in the programme content?

☐ No

☐ Yes

Q37. If yes, please briefly describe the main gaps:

What worked well and areas for development

Q38. In your view, what aspects of the CyberASAP programme worked best and why?

Q39. What improvements are needed?

Q40. What surprised you about the CyberASAP programme?

Impact

Q41. What were the reasons for your participation in CyberASAP? Tick all that apply. [Multiple choices possible]

☐ Need to develop my entrepreneurial skills

☐ Need to develop my business management skills

☐ Not enough funding available to turn my research into a product or business

☐ Not enough time to dedicate to developing my research into a product or business

☐ Not enough time to conduct market research about the viability of my product

☐ Need to develop my knowledge of the set-up processes involved in starting a business

☐ Need to develop my knowledge of the legal processes involved in starting a business

☐ Don’t have the business networks or contacts I need to attract investment

☐ Not enough support for commercialisation from my University

☐ Needed to build/confirm my aspirations to form and run a business

☐ Needed confidence in my skills to form and run a business

☐ Needed to build resilience in dealing with challenges in forming and running a business

☐ Other

Q42. If other, please briefly describe the reasons:

Q43. On a scale of 1 to 10, how strong would you say your capability to commercialise research into cyber security is now that you have participated in CyberASAP? SCALE 1 (lowest) to 10 (highest).

Q44. On which of the following has your participation in CyberASAP had an impact? Tick all that apply. [Multiple choices possible]

☐ My entrepreneurial skills

☐ My business management skills

☐ Amount of funding available to turn my research into a product or business

☐ Time I could dedicate to developing my research into a product or business

☐ Time I could dedicate to conduct market research about the viability of my product

☐ My knowledge of the set-up processes involved in starting a business

☐ My knowledge of the legal processes involved in starting a business

☐ My business networks or contacts I need to attract investment

☐ My aspirations to form and run a business

☐ My confidence in my skills to form and run a business

☐ My resilience in dealing with challenges in forming and running a business

☐ Other

Q45. If other, please briefly describe the reasons:

Q46. Which of these would you say are the three most important impacts for you? [Up to three selections possible]

☐ My entrepreneurial skills

☐ My business management skills

☐ Amount of funding available to turn my research into a product or business

☐ Time I could dedicate to developing my research into a product or business

☐ Time I could dedicate to conduct market research about the viability of my product

☐ My knowledge of the set-up processes involved in starting a business

☐ My knowledge of the legal processes involved in starting a business

☐ My business networks or contacts I need to attract investment

☐ My aspirations to form and run a business

☐ My confidence in my skills to form and run a business

☐ My resilience in dealing with challenges in forming and running a business

☐ Other

Q47. If other, please briefly describe the reasons:

Q48. Have there been any unintended benefits to participating in CyberASAP?

☐ No

☐ Yes

If yes, please give details.

Q49. Have there been any unintended negative consequences to participating in CyberASAP?

☐ No

☐ Yes

If yes, please give details.

Q50. Before CyberASAP, did you participate in other programmes aimed at supporting commercialisation of research into cyber security?

☐ No

☐ Yes

Q51. What were the other programmes? Tick all that apply. [Multiple choices possible] IF Q50=Yes

☐ ICURe

☐ University accelerator programme

☐ External accelerator programme

☐ Cyber Bootcamps, intense, short training courses aimed at improving cyber-related skills

☐ Other

If other, please name these programmes:

Q52. Have you participated in other programmes aimed at supporting the commercialisation of research into cyber security since you completed CyberASAP?

☐ No

☐ Yes

Q53. What are the other programmes? Tick all that apply. [Multiple choices possible] IF Q52=Yes

☐ ICURe

☐ University accelerator programme

☐ External accelerator programme

☐ Cyber Bootcamps, intense, short training courses aimed at improving cyber-related skills

☐ Other

If other, please name these programmes:

Q54. How would you say CyberASAP compares with other programmes in which you have participated? IF Q52=Yes

☐ CyberASAP is much better

☐ CyberASAP is better

☐ CyberASAP is neither better nor worse

☐ CyberASAP is worse

☐ CyberASAP is much worse

Q55. Please briefly explain what the main differences are between CyberASAP and the other programmes in which you have participated? IF Q52=Yes

Q56. As a result of CyberASAP, did you develop:

	Yes	No	Not applicable
A Value Proposition
Market validation
Proof of concept

Q57. Please tick if you can attribute any of the of the following to your participation in CyberASAP:

Answer Choice	Yes	Yes, partly	No
I have spun out a company
I have licensed IP from my research
My product or service has been acquired by another firm
My company has been acquired by another firm
I have secured further funding
I have employed more people
My company is more productive
My company’s turnover has risen
I have made my technology open source

Please explain your answers

Q58. Have you received investment for your product or service?

☐ No

☐ Yes

Q59. How much investment have you received? IF Q58 = Yes

☐ Up to £25k

☐ £26k to £50k

☐ £51k to £100k

☐ £101k to £250k

☐ £251k to £500k

☐ Over £500k

☐ None

☐ Prefer not to say

Q60. Approximately, how much of this funding would you attribute to your participation in CyberASAP? IF Q58=Yes Please give your best estimate if at all possible. If not sure, please type “unsure” in the comment box below:

SCALE FROM 0% TO 100%

Q61. Have you received other government grants for your service or product since completing CyberASAP?

☐ No

☐ Yes

Q62. Please specify the grant programme(s) from which you have received a government grant. If Q61=Yes

Q63. How much grant funding have you received? If Q61=Yes

☐ Up to £25k

☐ £25k to £50k

☐ £50k to £75k

☐ £75k to £100k

☐ Over £100k

☐ Prefer not to say

Q64. Have you received in-kind investment for your product or service?

☐ No

☐ Yes

Q65. Please specify from who you have received this in-kind investment.

Q66. Please specify what form this in-kind investment has taken and the value of the in-kind investment.

Q67. What other outcomes have resulted from your participation in CyberASAP? Tick all that apply. [Multiple choices possible]

☐ My commercial awareness has improved

☐ I have experienced better inter-university collaboration

☐ I have trialled a new viable product or service

☐ I have secured a patent for my product or service

☐ I have secured a trademark for my product or service

☐ I have developed a licencing model for my product or service

☐ I have developed open source software

☐ I have registered a business

☐ My business continues to operate

☐ I feel that my business is more likely to survive its early years of operation

☐ My understanding of the university’s tech transfer process has improved

☐ I or my company have been nominated for an award

☐ I or my company have won an award

☐ Other:

Q68. Was the grant funding you received from CyberASAP sufficient to allow you to achieve maximum benefits for your project?

☐ No

☐ Yes

If no, please explain why not:

Final questions

Q69. Do you have any other suggestions on how CyberASAP could be improved in the future?

Q70. Do you have any other final thoughts to share on CyberASAP?

Appendix D - participating universities and projects

Cohort	Region	University	Project
Cohort 1 (2017/18)	Wales	University of South Wales	Awen Collective
Cohort 1 (2017/18)	Northern Ireland	Queen’s University Belfast	IoT security – central and distributed protection (Phase 1 only)
Cohort 1 (2017/18)	North-west England	Liverpool John Moore University	Cyber D3sign
Cohort 1 (2017/18)	North-west England	Liverpool John Moore University	Cypher
Cohort 1 (2017/18)	North-west England	Liverpool Hope University	Secret Sharing for Cloud and Internet of Things (Phase 1 only)
Cohort 1 (2017/18)	East Midlands	University of Northampton	Commercialising the Concept of a Corporate Cyber-Immune System (Phase 1 only)
Cohort 1 (2017/18)	East of England	Anglia Ruskin University	Cyberbot Security
Cohort 1 (2017/18)	East of England	University of Cambridge	Cambridge Authentication
Cohort 1 (2017/18)	South-west England	University of Bristol	KETS Quantum Security
Cohort 1 (2017/18)	South-west England	Bournemouth University	Commercialisation of CAIRIS (Computer Aided Integration of Requirements and Information Security) (Phase 1 only)
Cohort 1 (2017/18)	South-east England	University of Surrey	Password+ (Phase 1 only)
Cohort 1 (2017/18)	South-east England	University of Southampton	Provenance for Digital Fabrication (3D Printing) (Phase 1 only)
Cohort 1 (2017/18)	London	Imperial College	Graphics Fuzz
Cohort 2 (2018/19)	Scotland	Glasgow Caledonian University	Mobile App Profiling For Malicious Activity Detection (Phase 1 only)
Cohort 2 (2018/19)	Wales	Cardiff University	Smoke Detector for IoT Security (originally Cyber security Monitoring for the Smart Home) (Phase 2 only)
Cohort 2 (2018/19)	North-west England	Liverpool John Moore University	Nanosec
Cohort 2 (2018/19)	North-west England	Manchester Metropolitan University	Zero Trust Security (Phase 2 only)
Cohort 2 (2018/19)	North-east England	Newcastle University	Verifiable electronic voting for real-world elections (Phase 2 only)
Cohort 2 (2018/19)	Yorkshire and the Humber	University of Huddersfield	Event Horizon
Cohort 2 (2018/19)	Yorkshire and the Humber	University of Bradford	Automated Pen-testing System (Phase 1 only)
Cohort 2 (2018/19)	West Midlands	Universities of Warwick & Newcastle University	SEEV
Cohort 2 (2018/19)	West Midlands	University of Wolverhampton	Cydon
Cohort 2 (2018/19)	West Midlands	Birmingham City University	Secure Flow Self-Healing Networks (Phase 1 only)
Cohort 2 (2018/19)	East Midlands	De Montfort University	ACTIVE
Cohort 2 (2018/19)	East Midlands	De Montfort University	Benefit and Cost of Cloud Computing Security (Phase 1 only)
Cohort 2 (2018/19)	East of England	University of Hertfordshire	CrypTier (TAME)
Cohort 2 (2018/19)	East of England	Anglia Ruskin University	Risk2Click An Intelligent Plugin to Stop Phishing and Threats related to the User’s Privacy in real-time (Phase 1 only)
Cohort 2 (2018/19)	South-west England	Bournemouth University	HuaHana Initiative
Cohort 2 (2018/19)	South-west England	Bournemouth University	BU-CERT Threat Intelligence activities (WIDECERT) (Phase 1 only)
Cohort 2 (2018/19)	South-west England	University of Gloucestershire	Strong random number generation from body physiological signals for secure communication between implants inside the body and the gateway outside the body (Phase 2 only)
Cohort 2 (2018/19)	South-east England	University of Brighton	Privacy Labs
Cohort 2 (2018/19)	South-east England	University of Oxford	SMART (Security Monitoring and Administration Residential Toolkit)
Cohort 2 (2018/19)	South-east England	University of Oxford	Quantum Random Number Generator Web Api (Phase 1 only)
Cohort 2 (2018/19)	London	City, University of London	City Defend Cyber Security Solutions Limited
Cohort 2 (2018/19)	London	City, University of London	Raven Science
Cohort 2 (2018/19)	London	City, University of London	Privacy preserving searching in the encrypted Cloud (Phase 2 only)
Cohort 2 (2018/19)	London	London Metropolitan University	AirID
Cohort 2 (2018/19)	London	University of West London	Cymond
Cohort 2 (2018/19)	London	Kingston University	Non-identifiable Non-reversible Sharing (Phase 1 only)
Cohort 3 (2019/20)	Scotland	University of Glasgow	PriSAT
Cohort 3 (2019/20)	Scotland	Glasgow Caledonian University	Vulnerability Prioritisation (ACTIVE Non-graduate) [footnote 70]
Cohort 3 (2019/20)	North-west England	University of Liverpool	Colour image encryption using Morris-Lecar neuron model based hardware (ACTIVE Non-graduate)
Cohort 3 (2019/20)	North-east England	Teesside University	Play2Secure: An AI enabled personalised cyberspace awareness game for young people (ACTIVE Non-graduate)
Cohort 3 (2019/20)	North-east England	Teesside University	Risk2IoT@Home (ACTIVE Non-graduate)
Cohort 3 (2019/20)	West Midlands	Coventry University	GuardKeeper
Cohort 3 (2019/20)	West Midlands	University of Wolverhampton	Onlyn Shield
Cohort 3 (2019/20)	West Midlands	Birmingham City University	AngelPass (ACTIVE Non-graduate)
Cohort 3 (2019/20)	East Midlands	De Montfort University	INSURE
Cohort 3 (2019/20)	East Midlands	De Montfort University	Simulated Critical Infrastructure Protection Scenarios (SCIPS) (ACTIVE Non-graduate)
Cohort 3 (2019/20)	East of England	University of Essex	SensiTech - Secure Blockchain Alert System (ACTIVE Non-graduate)
Cohort 3 (2019/20)	South-west England	Bournemouth University	CTLR / Draconia
Cohort 3 (2019/20)	South-west England	Bournemouth University	The Privacy Tool (Pitch)
Cohort 3 (2019/20)	South-west England	Bournemouth University	TAPCHA
Cohort 3 (2019/20)	South-west England	Bournemouth University	Actionable Cyber Intelligence Platform (ACTIVE Non-graduate)
Cohort 3 (2019/20)	South-west England	University of Gloucestershire	BioGenerate
Cohort 3 (2019/20)	South-west England	University of Plymouth	CRAB (ACTIVE Non-graduate)
Cohort 3 (2019/20)	South-east England	Royal Holloway	BLEMAP
Cohort 3 (2019/20)	South-east England	Royal Holloway	Prinesec
Cohort 3 (2019/20)	South-east England	Royal Holloway	Seclea (ACTIVE Non-graduate)
Cohort 3 (2019/20)	South-east England	University of Kent	Verifiable Credentials
Cohort 3 (2019/20)	South-east England	University of Kent	Proof of Claim (PoC): A human-in-the-loop AI based solution for semantic information integrity (ACTIVE Non-graduate)
Cohort 3 (2019/20)	South-east England	University of Oxford	PhishAR
Cohort 3 (2019/20)	South-east England	University of Southampton	Spyderisk
Cohort 3 (2019/20)	South-east England	University of Brighton	Cyberscape Training (ACTIVE Non-graduate)
Cohort 3 (2019/20)	South-east England	University of Winchester	Biothentity Multi-Modal Continuous Transparent Biometric Authentication for Mobile Devices (ACTIVE Non-graduate)
Cohort 3 (2019/20)	London	Goldsmiths, University of London	FACT 360 - “Transactional analysis for insider threat detection” (ACTIVE Non-graduate)
Cohort 3 (2019/20)	London	London Metropolitan University	SYCAMORE: Vulnerability Analysis and Risk Assessment of Cyber Security Policies (ACTIVE Non-graduate)
Cohort 3 (2019/20)	London	City, University of London	A triage playbook for assessing data harm for breach notification during initial personal data incident response. (ACTIVE Non-graduate)
Cohort 3 (2019/20)	N/A	Open University	CyberGaTE: A Gamified Intelligent Environment for Training and Assessing Cyber Security Behaviour (ACTIVE Non-graduate)
Cohort 4 (2020/21)	Scotland	Edinburgh Napier University	Memcrypt
Cohort 4 (2020/21)	Scotland	University of Strathclyde	Lupovis
Cohort 4 (2020/21)	Scotland	Glasgow Caledonian University	Vulnerability Prioritisation Tool (Phase 1A only)
Cohort 4 (2020/21)	Wales	Cardiff University	CyberGrid / CyCIS (Phase 1B only)
Cohort 4 (2020/21)	North-west England	Lancaster University	Secure Development
Cohort 4 (2020/21)	North-west England	University of Liverpool	Flexible querying of encrypted graph databases (Phase 1B only)
Cohort 4 (2020/21)	Yorkshire and the Humber	University of Leeds	ABBA IOT
Cohort 4 (2020/21)	Yorkshire and the Humber	University of Leeds	Developing a Quantum-inspired Key Distribution System (Phase 1B only)
Cohort 4 (2020/21)	Yorkshire and the Humber	University of Huddersfield	SocialSPI (Phase 1A only)
Cohort 4 (2020/21)	West Midlands	University of Wolverhampton	CyberMIND
Cohort 4 (2020/21)	East Midlands	University of Nottingham	AI-Driven Secured Connected and Autonomous Vehicles (AI-DRIVEs) (Phase 1B only)
Cohort 4 (2020/21)	South-west England	University of Essex	Sensei Chain
Cohort 4 (2020/21)	South-west England	University of Essex	Hash Identity
Cohort 4 (2020/21)	South-west England	Bournemouth University	Authentibility
Cohort 4 (2020/21)	South-west England	University of Bristol	Surface RF
Cohort 4 (2020/21)	South-west England	University of the West of England	Strengthening Cyber Science Knowledge & Teaching from non-STEM perspectives for school pupils - CyberDucate (PI) (Phase 1A only)
Cohort 4 (2020/21)	South-west England	University of the West of England	Blockchain-enabled devices mutual authentication in zero-trust Internet of Things (BazIoT) (Phase 1A only)
Cohort 4 (2020/21)	South-west England	University of Plymouth	MaCRA (Riskocity)
Cohort 4 (2020/21)	South-east England	Middlesex University	SALMAC
Cohort 4 (2020/21)	South-east England	University of Southampton	CyberHelper
Cohort 4 (2020/21)	London	City, University of London	Triage playbook (Phase 1A only)
Cohort 4 (2020/21)	London	Imperial College London	Shoji
Cohort 4 (2020/21)	London	Imperial College London	WhatML
Cohort 4 (2020/21)	London	University of East London	Cyber Insurability Toolkit (Phase 1A only)
Cohort 4 (2020/21)	London	London Metropolitan University	Institutional Trust, Secure and Scalable Sharing of Clinical Data (Phase 1A only)
Cohort 4 (2020/21)	London	London Metropolitan University	Enhancing Network Intrusion Protection Systems (Phase 1A only)
Cohort 4 (2020/21)	London	UCL	Service Resilience Capability Assessment Tool (SR-CAT) (Phase 1B only)
Cohort 4 (2020/21)	London	London Southbank University	Secure VNF Service Mgmt in 5G (Phase 1B only)
Cohort 5 (2021/22)	Scotland	University of Strathclyde	CAUDATA (Phase 1A only)
Cohort 5 (2021/22)	North-west England	Edge Hill University	CyberSignature
Cohort 5 (2021/22)	North-west England	University of Salford	Privextractor (Phase 1A only)
Cohort 5 (2021/22)	North-west England	University of Salford & Manchester Metropolitan University (& University of Bristol)	WalletFind
Cohort 5 (2021/22)	North-east England	Teesside University	Secure-IoMT4Health (Phase 1A only))
Cohort 5 (2021/22)	North-east England	Teesside University	HackEscape (Phase 1B only)
Cohort 5 (2021/22)	East Midlands	De Montfort University	CyberGames (Phase 1A only)
Cohort 5 (2021/22)	East Midlands	De Montfort University	System of Automatic and Configurable web cRawlEr Detection (SACRED) (Phase 1A only)
Cohort 5 (2021/22)	West Midlands	University of Wolverhampton	FedCam
Cohort 5 (2021/22)	West Midlands	University of Wolverhampton	Tymlo / TrustMe
Cohort 5 (2021/22)	West Midlands	Aston University	Synthetic Cryptocurrency Fraud Simulator (Phase 1A only)
Cohort 5 (2021/22)	East of England	University of Essex	UltraNetAI
Cohort 5 (2021/22)	East of England	University of Hertfordshire	OSIRT
Cohort 5 (2021/22)	South-east England	Middlesex University	MLighter
Cohort 5 (2021/22)	South-east England	Royal Holloway	Tensor Crypt
Cohort 5 (2021/22)	South-east England	University of Kent	ViBS (Voting in Blockchain Systems) (Phase 1B only)
Cohort 5 (2021/22)	South-west England	University of Gloucestershire	CloudIntell (Phase 1A only)
Cohort 5 (2021/22)	London	Imperial College London	Royal-Imperial Black Box
Cohort 5 (2021/22)	London	UCL	TAIMAS (SDTaP)
Cohort 5 (2021/22)	London	University of West London	Forbidden Pi: TPM 2.0 Secure Computing Extension (Phase 1A only)
Cohort 5 (2021/22)	London	University of Westminster	Consumer smart devices security solution (Phase 1B only)
Cohort 5 (2021/22)	London	University of East London	An Intelligence Cyber Security Incident Handling (i-CSIH) (Phase 1A only)

1. Innovate UK KTN - referred to as KTN in this report. ↩

2. Note – this evaluation covers the period up to, and including, the delivery of Year 5 of the programme. Year 6 was delivered while the report was being written and is due to complete in March 2023. Year 6 did not form part of the assessment. ↩

3. The process of dividing a project / programme into distinct stages or phases separated by decision points known as ‘gates’ ↩

4. Examples of university policies provided to RSM UK Consulting by KTN in August 2022 include: policies on formation of spinout companies; policies that require the university to retain a sizeable amount of equity in a spinout (which would then hamper any future investment funding); and if approval policies were slow to progress spinouts this meant the momentum built during participation in CyberASAP was lost ↩

5. Specific, measurable, achievable, relevant, and time-bound ↩

6. Data provided by KTN to RSM UK Consulting (May 2022). Total number of participants = 97 ↩

7. CyberASAP Alumni Status Tracking (March 2022) provided to RSM UK Consulting by KTN ↩

8. The scale from which this data was taken ranged from 0 - 10 ↩

9. Based on the current status of all CyberASAP participants from Cohorts 1 – 4 from data provided by KTN to RSM UK Consulting, Cohort 5 data was not yet available ↩

10. Information provided by KTN staff (March 2023) ↩

11. KTN (2023) CyberASAP Case Studies: Insights from CyberASAP Alumni [available online] [accessed 23/03/2023] ↩

12. In February 2023, the parts of UK government responsible for cyber security policy moved to the new Department for Science, Innovation and Technology ↩

13. Note – this evaluation covers the period up to, and including, the delivery of Year 5 of the programme. Year 6 was delivered while the report was being written and is due to complete in March 2023. Year 6 did not form part of the assessment. ↩

14. Based on a sample of 77 unique projects ↩

15. UK in a Changing Europe (2020): Cybersecurity after Brexit [Available online] [Accessed 14 January 2022] and European Council (2020): Digital Europe programme. [Available online] [Accessed 14 January 2022]. ↩

16. Ipsos MORI (2020): Evaluation of ICURe. Final Report. [Available online]. See also, Douglas Morgan et al (2013): Crossing the ‘Valley of Death’: Transitioning Cybersecurity Research into Practice. IEEE 11:2; and CyberASAP (2021): Impact and Insights Report. [Available online] [Both accessed 14 January 2022]. ↩

17. Ipsos MORI (2022): Cyber security skills in the UK labour market 2022. [Available online] [Accessed 26 July 2022]: and Diversity UK [Accessed 14 January 2022]. ↩

18. Knowledge Transfer Partnership Guidance (2022) [Available online] [accessed Dec 2022]. ↩

19. Commercialising quantum technologies challenge (2022) [available online] [accessed Dec 2022) ↩

20. https://apply-for-innovation-funding.service.gov.uk/competition/1085/overview ↩

21. https://apply-for-innovation-funding.service.gov.uk/competition/1020/overview#eligibility ↩

22. ICURe trains, funds, and supports teams led by university early-career researchers (ECRs) to determine whether there is a market for products or services that utilise their research, science, or technology. It is designed using lean start-up methodology and Strategyzer tools ↩

23. A minimum viable product, or MVP, is a product with enough features to attract early-adopter customers and validate a product idea early in the product development cycle ↩

24. DCMS implemented this change to re-direct funding towards another commercialisation pilot programme. By creating an SDTaP cohort running alongside the open cyber cohort, the delivery costs could be reduced as the content was mostly identical. ↩

25. Information provided by KTN to RSM UK Consulting (March 2022) ↩

26. Information provided by KTN to RSM UK Consulting (March 2022) ↩

27. Information provided by KTN to RSM UK Consulting (March 2022) ↩

28. The challenge led innovation approach starts with the challenge owner identifying a problem rather than the solution – in this case focused on those from the SDTaP programme ↩

29. Note- a revised base number is included alongside specific answer choices that were skipped by members of the overall base number of 48. Some rows total 99% / 101% due to rounding ↩

30. Note- respondents were able to select multiple answers therefore percentages will total more than 100% ↩

31. The logframe reports are documents in Excel format containing information on:
    - a cover sheet with a brief narrative overview of progress
    - a financial report
    - KPI and milestone progress
    - a risk register and guidance on the risk register ↩

32. Please note that RSM UK Consulting has been provided with contradicting information around participation in Year 1 Phase 1 from KTN, with both 12 and 13 being provided as the number of participating companies ↩

33. 6 of the 7 participants from Year 1 Phase 2 were from Phase 1, with the 7th joining externally ↩

34. In Year 1 (the pilot of what then became CyberASAP). KTN was not involved in Phase 1 and only supported delivery of Phase 2. As a result KTN did not receive complete details of all projects and there are 5 projects for which there is no detail on participating universities and therefore Russell Group status. ↩

35. I.e., moving from phase 1A to 1B to 2. ↩

36. I.e., post-programme funding raised by participating projects / companies. ↩

37. Year 3 annual report ↩

38. Year 4 annual report ↩

39. Year 5 final logframe ↩

40. I.e., projects that register as a company after participation. After Year 1 CyberASAP has not required projects to register as a company as a condition for being on the programme. In Year 1, the projects that proceeded to Phase 2 had a requirement to form a company due to State Aid legislation. CyberASAP was thereafter designed to allow a broader range of projects and routes to commercialisation. For example, under ICURe, licensing as a route to progress was not a funded option (information provided by KTN to RSM UK Consulting – August 2022) ↩

41. I.e., articles which published and posted on the CyberASAP website (sub-pages on the KTN website), on the CyberASAP LinkedIn pages and on Twitter (@CyberASAP) ↩

42. No KTN annual report was available for Year 5 ↩

43. 2 of the 11 respondents who selected that they had participated post-CyberASAP did not provide specific information on what programme / type of programme it was ↩

44. Focused on developmental activities (e.g. market research and product development) ↩

45. Licensing deals have been agreed ↩

46. Further development of IP through companies created in association with the relevant researcher and university / research institute ↩

47. Company has been purchased ↩

48. Current status is unknown ↩

49. Not currently carrying out business activities ↩

50. Ownership has been purposely surrendered ↩

51. An injection of funding in the form of investment has been received ↩

52. A patent is a legal right to an invention given to a person or entity without interference from others who wish to replicate, use, or sell it ↩

53. Note – base number is 51 despite none of the individual answers receiving that many responses as all respondents who responded to at least one of the answer choices were counted. Only 4 of the total 55 respondents skipped the question entirely (resulting in a base number of 51) ↩

54. www.cydon.co.uk/ ↩

55. Note - 1 of the 53 respondents skipped this option ↩

56. Other academics within the cyber profession. ↩

57. Note - equates to 101% with rounding up (87.5% and 12.5%) ↩

58. Ipsos MORI [2022]: Cyber security skills in the UK labour market 2022. [Available online] [Accessed 26 July 2022] and Diversity UK [Accessed 14 January 2022]. ↩

59. 1 respondent skipped the value proposition question ↩

60. CyberASAP Programme logframe (March 2022), Year 1-4 Annual Reports. ↩

61. The process of dividing a project / programme into distinct stages or phases separated by decision points known as ‘gates’ ↩

62. Examples of university policies provided to RSM UK Consulting by KTN in August 2022 include: policies on formation of spinout companies; policies that require the university to retain a sizeable amount of equity in a spinout (which would then hamper any future investment funding); and if approval policies were slow to progress spinouts this meant the momentum built during participation in CyberASAP was lost ↩

63. Specific, measurable, achievable, relevant, and time-bound ↩

64. Data provided by KTN to RSM UK Consulting (May 2022). Total number of participants = 97 ↩

65. CyberASAP Alumni Status Tracking (March 2022) provided to RSM UK by KTN ↩

66. Note- the scale from which this data was taken ranged from 0 - 10 ↩

67. Based on the current status of all CyberASAP participants from cohorts 1 – 4 from data provided by KTN to RSM UK Consulting, cohort 5 data was not yet available ↩

68. Information provided by KTN staff (March 2023) ↩

69. KTN (2023) CyberASAP Case Studies: Insights from CyberASAP Alumni [available online] [accessed 23/03/2023] ↩

70. Refers to university projects which did not complete all phases of CyberASAP however have indicated they are still actively working on their project (information provided by KTN in March 2023) ↩