GovTech Catalyst options for Phase 2 of Northern Ireland Audit Office challenge - 19 November 2019
Published 23 September 2020
GovTech Challenge Owner: Northern Ireland Audit Office (NIAO)
GovTech Catalyst Challenge: Developing a data enabled public sector audit approach
Decision: Feedback to be provided by email to [name and role redacted] the GovTech Catalyst by Friday 29 November 2019. Final decision to be taken at Steering Board conference call on Tuesday 3 December 2019 at 12:30.
Summary
The decision to progress to Phase 2 for GovTech Challenges is based on the application of five tests to the findings of Phase 1. Having applied these tests to the Northern Ireland Audit Office (NIAO) challenge on developing a data enabled public sector audit approach four of these tests have been met or partially met, and one test has not been met.
As a result of this assessment, the GovTech Catalyst team has concluded that this challenge does not require technology innovation to solve it, and that there is procurable technology available to NIAO through the organisation’s existing procurement process that could solve this challenge. This conclusion has been challenged by the NIAO. This is the second challenge that has not met/partially met all of the five tests, and is the first challenge where there has not been consensus between the GovTech Catalyst team and a Challenge Owner on the decision not to progress to Phase 2.
It is important to note that the GovTech Catalyst team’s recommendation is based solely on the 5 tests - in all other regards, the NIAO has been an an exemplar Challenge Owner in the delivery of this GovTech Challenge.
As we have not been able to reach agreement with the Challenge Owner, and given some of the handling sensitivities, we are asking the GovTech Catalyst Steering Board to decide on four potential options (details of which are set out in the main paper) .
- Option 1: Conclude the challenge at Phase 1 and do not progress to Phase 2 (Recommended).
- Option 2: Proceed to Phase 2 focusing on the scope as published for Phase 1.
- Option 3: Proceed to Phase 2 focusing only on financial audit.
- Option 4: Proceed to Phase 2 focusing only on value for money analytics.
Background
The Challenge Owner for this competition was the NIAO working in collaboration with Audit Scotland (AS) and the Wales Audit Office (WAO). The NIAO is responsible for the audit of public sector bodies and undertakes financial and value for money audits.
The objective of this challenge was to develop a data enabled public sector audit approach using analytics techniques and machine learning. The competition scope was to develop a holistic data enabled audit approach capable of capturing, extracting, cleansing, formatting, interpreting, analysing and presenting data from a range of sources.
In Phase 1, the NIAO asked suppliers to address at least one of two problem areas: financial audit analytics or value for money analytics.
Financial audit analytics involves analysing and audit testing whole financial transaction populations and identifying high risk transactions, anomalies and potential irregularities for further human investigation, possibly using machine learning. A solution could include the use of visualisations and automation. It could also explore the possibility of comparing data over time with other organisations and publically available data to identify trends and anomalies.
Value for money analytics involves analysing management information and publicly available data to provide evidence to support findings and recommendations in a wide variety of thematic reports.
Govtech Catalyst phase 2 progression criteria
Test | Description | Met/partially met/not met |
---|---|---|
Test 1 | Phase 1 should have further refined understanding of a current service or policy delivery problem | Met |
Test 2 | Phase 1 should have demonstrated a clearly defined user need. | Partially met |
Test 3 | The Phase 1 feasibility studies should have demonstrated that the challenge is the right size to result in a commercially sustainable product following a further development budget of £500,000. | Partially met |
Test 4 | There should be evidence that other organisations beyond the current challenge owner will benefit from solving this challenge. | Met |
Test 5 | The challenge must require technology innovation to solve it and this innovation must not be procureable through the organisation’s existing procurement process. | Not met |
Below is a detailed overview of the assessment for Test 5. In summary, for this test to be ‘met’ there would needed to have been demonstrable proof that innovative technology, rather than well-understood techniques, would be required to develop a solution. Phase 1 would also have had to demonstrate that a solution could not be developed by a traditional procurement route or using existing in-market products to achieve a solution.
Financial Audit Analytics
The GovTech Catalyst team assessed that the financial audit aspect of the challenge demonstrated that audit workflow could be digitised, resulting in savings of time and potentially an increase in accuracy. Also, the desired outputs are relatively clear e.g in the reconciliation of accounts to zero.
The key findings of the assessment include:
- Workflow automation approaches tried were fairly standard and could be procured through normal channels
- The challenge owner resisted adopting standard data science environments such as R, but the GovTech Catalyst team did not believe that this merited the creation of bespoke analysis environments — any analysis environment with sufficient flexibility was likely to have much of the same complexity as existing data science tools.
- There was evidence that financial audit analytics was a feasible area in which to apply workflow automation. However, there were no areas in which non-mechanical analytic techniques were applied. The more mechanical approaches tried did show promise of benefit in supporting the process of reconciliation and evaluation of public accounts, but were not trialed extensively with end users, and non-ledger accounts were not included in the Phase 1 analysis.
- Most elements of the solutions trialled to date could be bought using standard DOS or GCloud framework call-off procurements
- The potential process innovation of changing from random sampling to algorithmically-directed sampling was interesting but not fully explored, and it was assessed that this approach could not be proved to be free of bias.
Value for money
In the value for money aspect, the particularly interesting element from an innovation standpoint was the opportunity to use sophisticated semantic language models to map and understand the relative value of investments into public services across localities and public bodies.
The approaches to the value for money stream of work were the most technologically innovative part of the challenge, as they relied on advanced machine learning algorithms applied to natural language parsing models. One impressive approach to machine summarisation and ranking of existing sources of research and official statistics in the area of diabetes mortality was identified.
The key findings of the assessment include:
- It was assessed that the area of most innovation opportunity was in correlating public spending to real-world outcomes reported in validated research. However, the sources of data for public body spending had not been matched to the machine ranking of efficacy research, and it did not seem that this data could be obtained in future.
- There was limited evidence to show the increased efficiency of this approach. It was not demonstrated through the user engagement and research undertaken by suppliers that the workflow for these reports would be more streamlined or of higher quality. Phase 1 showed general enthusiasm for the concept and results obtained, but did not validate how these research results would be integrated into a VFM workflow.
Challenge owner consultation feedback
The GovTech Catalyst team has consulted with NIAO on this paper. Below is the feedback on the findings for Test 5:
- NIAO concludes that the original problem can only be solved through technology innovation as there is not currently any technology available on the market that meets the challenge criteria. This has been tested by NIAO and the other public audit agencies in the UK including NAO, Audit Scotland and Wales Audit office. NAO ran a ‘proof of concept’ using MindBridge technology and did not proceed to procurement on the basis of functionality. NAO found that there were issues with it handling public sector data since the system was designed using commercial business data.
- NIAO has highlighted that public audit agencies across the UK have also tried to develop their own data analytic techniques using existing software such as R. Although some progress has been made it is very piecemeal and will not provide the holistic data enabled audit approach that they were hoping to get from the GovTech catalyst.
Throughout Phase 1 there has been differing perspectives between the GovTech Catalyst team and NIAO on the availability of data.
The GovTech Catalyst team saw high potential in this project using the unique central viewpoint of the public auditor to correlate and compare both policy results and BAU spending across multiple public authorities using transparent machine learning techniques. However, it transpires that public accounts data cannot be used in this way because the permission does not exist. NIAO made available the data sets to suppliers that they indicated that they would at the supplier briefing event: accounts from two public bodies, amounting to millions of transactions.
From the inception onwards the GovTech Catalyst team has worked on the basis that cross-body correlation in both streams of work would be feasible only with public accounts from a much larger sample, including public bodies with comparable public duties in different geographical regions.
The GovTech Catalyst team accepts that it was never the plan of NIAO to make this cross-region correlation data available to suppliers and that it was not possible to develop these sources mid-project. However, Phase 1 could therefore not demonstrate that it was possible to develop machine learning models to correlate comparable accounts or to match public spend to statistical outcomes.
NIAO has highlighted that a very significant amount of data has been supplied and huge amounts of data transactions will be provided in Phase 2. They do not feel that providing exponentially more datasets would improve the chances of success of this project.
The GovTech Catalyst team has taken this mis-match in expectations on data availability into the service design process, and will attempt to mitigate this risk earlier in future rounds.
Summary and options appraisal
When considering which option to recommend, the GovTech Catalyst Steering Board should note:
- NIAO has been an exemplar Challenge Owner in the delivery of this GovTech Challenge. This has been demonstrated through four of the five GovTech Catalyst assessment criteria tests being met or partially met, and the high level of commitment from inception to end of Phase 1.
- NIAO do not agree with the assessment for Test 5 and this is the first GovTech challenge where there has not been consensus between the GovTech Catalyst team and a Challenge Owner on the decision to progress to Phase 2.
- Agreeing to progress to Phase 2 (options 2,3 and 4) will involve a labour intensive process of developing invitation to tender (ITT) documents by NIAO, and responding to the ITT by suppliers. These options do however provide an opportunity to review and assess the responses from the market to this challenge. There is no obligation to proceed with Phase 2 if no suppliers are identified as meeting the assessment criteria for Phase 2.
- Based on applying the five tests to the NIAO GovTech Challenge the GovTech Catalyst team has concluded that this challenge does not require technology innovation to solve it, and that there is procurable technology available to NIAO through the organisation’s existing procurement process that could solve this challenge. As such, this challenge has not met Test 5 of the GovTech Catalyst assessment criteria and the Govtech Catalyst team recommends Option 1: conclude the challenge at Phase 1.
Option 1: Conclude the challenge at Phase 1
Advantages:
- The GovTech Catalyst does not risk continuing to invest in solutions that are already available on the market.
- The GovTech Catalyst does not risk continuing to invest in solutions that do not develop technology innovation.
Disadvantages:
- NIAO will have difficulty in the near-term of having access to the funding (and developing the capabilities and data needed) to make their financial audit workflow more efficient, as such there is an opportunity cost.
- There is a political implication to ending a project after Phase 1, and the GovTech Catalyst team would carefully mitigate any suggestion that the NIAO or their suppliers have in any way failed to deliver satisfactorily, which has not been the case. The delivery of this GovTech Challenge has been excellent.
Option 2: Proceed to Phase 2 focusing on the scope as published for Phase 1, focusing on both areas of the original scope: financial audit and value for money.
Advantages:
- For the financial audit aspect of the challenge the workflow issues seem to be feasible for digitisation, and there is an existing process and team which suppliers could work to understand and improve. Also, the understanding of desired outputs is relatively clear (for example, reconciliation of accounts to zero), making impact easy to evaluate. Four of the five suppliers focused on this part of the challenge therefore a higher likelihood of funding quality applications.
- The value for money aspect of the challenge was identified as the most innovative part of the challenge from a technology point of view, relying on advanced machine learning language parsing models.
Disadvantages:
- The solutions for the financial audit aspect of the challenge do not rely on any particular technology or process innovation, and therefore do not fall within the scope of the problems the GovTech Catalyst fund is designed to address.
- Workflow automation approaches could already be procured through normal channels. It was assessed that elements of the solutions trialled to date could be bought now, indicating less need for GovTech Catalyst investment. Digital project skills on the client side within NIAO remain limited, so it is not clear whether NIAO can fully support the development of custom workflow software.
Option 3: Proceed to Phase 2 focusing only on financial audit
Advantages:
- The workflow issues seem to be feasible for digitisation, and there is an existing process and team which suppliers could work to understand and improve could be used.
- The understanding of desired outputs is relatively clear (for example, reconciliation of accounts to zero), making impact easy to evaluate. Four of the five suppliers focused on this part of the challenge therefore a higher likelihood of funding quality applications.
Disadvantages:
- The solutions do not rely on technology or process innovation, and therefore do not fall within the scope of the problems the GovTech Catalyst fund is designed to address.
- Workflow automation approaches could already be procured through normal channels.
- Digital project skills on the client side within NIAO remain limited, so it is not clear whether NIAO can fully support the development of custom workflow software. It was assessed that elements of the solutions trialed to date could be bought now, indicating less need for GovTech Catalyst investment.
Option 4: Proceed to Phase 2 focusing only on value for money analytics
Advantages:
- This is the most innovative part of the challenge from a technology point of view, relying on advanced machine learning language parsing models.
Disadvantages:
- It remains unclear as to whether suppliers would have the capability and desire to pivot. Certainly some companies involved in Phase 1 will have no desire in this quite different problem. There was limited evidence to show the increased efficiency of this approach. The data available does not seem to be sufficiently shareable to support cross-geographical mapping of VFM elements.