Public Sector Fraud Authority: Fraud Risk Assessment Accelerator
Tool to generate initial drafts of Fraud Risk Assessments.
Tier 1 Information
1 - Name
Fraud Risk Assessment Accelerator
2 - Description
An application that allows users to use Generative Pre-trained Transformer (GPT) in order to generate draft Fraud Risk Assessments (FRA)s by reference to a defined set of source documents, via Streamlit web application, hosted in Cloud Based Analytical System (CBAS), to allow both Public Sector Fraud Authority (PSFA) teams and external teams it collaborates with to generate draft FRAs using Large Language Models (LLMs).
3 - Website URL
N/A
4 - Contact email
fra-accelerator@cabinetoffice.gov.uk
Tier 2 - Owner and Responsibility
1.1 - Organisation or department
Cabinet Office - Public Sector Fraud Authority (PSFA)
1.2 - Team
Data Science
1.3 - Senior responsible owner
Product Manager
1.4 - External supplier involvement
Yes
1.4.1 - External supplier
Deloitte LLP
1.4.2 - Companies House Number
OC303675
1.4.3 - External supplier role
Deloitte LLP is engaged by Cabinet Office to supply consultants to support the Public Sector Fraud Authority with the delivery of a range of projects. Consultants deployed under this contract work as part of the PSFA Data science and analytics team to develop the application and perform associated business analysis and evaluation tasks.
1.4.4 - Procurement procedure type
Deloitte is engaged by Cabinet Office to provide the wider consultancy services described above through a contract procured under the pre-established MCF3 framework.
1.4.5 - Data access terms
Deloitte has no access to the data or the application. Deloitte’s staff deployed to the project are issued with Cabinet Office laptops and logins and use these exclusively to perform their work on the engagement.
Tier 2 - Description and Rationale
2.1 - Detailed description
Large Language Models (LLMs) are a subset of artificial intelligence models that generate a range of content, including text, imagery, audio, and synthetic data in response to user inputs. LLMs are pre-trained on a wide range of such content to predict (and then produce) the best response to the user’s input (for example, to answer questions or to provide summaries of information according to user instructions). The tool uses the LLM GPT-4.1, developed by OpenAI.
Our project involves the analysis of existing Fraud Risk Assessments (FRAs). FRAs are documents created to evaluate the fraud risk in schemes (e.g. grant schemes) that government departments plan to implement. FRAs contain an outline of assessments and recommendations in mitigating risks of fraud in government schemes. We also use existing proposed government grant scheme briefs. These contain details of eligibility and other details of a proposed grant scheme. We will also use guidance on creating FRAs. These are standards and guidance on producing FRAs within current standards.
We use GPT-4.1, via prompting, to analyse this data to generate draft FRA content. The draft FRA content will be reviewed by users in order to assist them with preparation of draft FRAs.
We have developed a web application using the python package Streamlit, hosted in CBAS, that allows users (e.g. fraud domain experts within government completing FRAs) to upload input material (such as guidance documents and government grant scheme briefs) and run the model to create a draft FRA. This will mean that a wider range of teams have the ability to prepare draft FRAs using multiple guidance documents.
2.2 - Scope
The purpose of the tool is to allow users to use GPT in order to generate draft FRAs by reference to a defined set of source documents. Qualified fraud risk assessors will use the application to develop FRAs, with the application’s outputs being reviewed, amended and / or enriched by experts in order to produce a final FRA.
The tool is currently in a Private Beta Phase. This means that the tool is available to a select few groups so that we can implement improvements. Following this we would be able to scale the tool for wider use.
2.3 - Benefit
This will accelerate preparation of FRAs by automating early stage tasks to assemble risk typologies.
2.4 - Previous process
Decisions around the tool have been informed by a multidisciplinary team consisting of Fraud Subject Matter Experts (SMEs), Governance, and Data Scientists from Public Sector Authority (PSFA). This was also supported by conversations with Microsoft on knowledge exchange on Azure.
Initially these groups contributed to a Discovery Phase where we experimented with a proof of concept to understand the capability of an LLM to generate FRAs. The Data Scientists worked on the technical development of the tool. Fraud SMEs provided feedback on the tool’s outputs to evaluate the tool’s effectiveness and the Data Scientists implemented improvements based on this feedback. Governance specialists advised the team on legal, ethics and data privacy considerations, including the preperation of a DPIA. They have continued to contribute these inputs for readying the proof of concept for Private Beta.
2.5 - Alternatives considered
The non-algorithmic methods is a Fraud Risk Assessor carrying out the following process:
- Kick-Off: Initial working sessions in order to understand the scope of the scheme.
- Futher Reading: Fraud SME conducting research based on internal documentation and additional relevant information provided by government departments.
- Drafting: Drafting inherent risks, respective controls, and residual risks.
Tier 2 - Decision making Process
3.1 - Process integration
The output of the tool is intended to be used by humans qualified to produce Fraud Risk Assessments (FRAs), i.e., Fraud Subject Matter Experts (SMEs)
The Fraud Risk Assessment Accelerator is intended to be integrated in the following process:
- Kick-Off: Initial working sessions in order to understand the scope of the scheme.
- Futher Reading: Fraud SME conducting research based on internal documentation and additional relevant information provided by government departments.
- Drafting: Drafting inherent risks, respective controls, and residual risks.
The tool supports these processes by ingesting relevant documentation and generating an initial draft of inherent risks in the form of Actor, Action, and Outcome (described below).
3.2 - Provided information
The tool will generate an initial Fraud Risk Assessment (FRA) draft in the form of an Excel Sheet, specifically containing:
Actor: Type of entity who commits the fraud. Entity refers to an archetype of an individual or organisation. Action: How the actor commits the fraud. Outcome: What happens as a result of the fraud, including actor benefit and effects on the public sector.
The output would not be a final submission. It should be used to inform/undergo review by a Fraud Subject Matter Expert (SME).
3.3 - Frequency and scale of usage
The number of users will increase over time. We plan to add additional users in phases during Private Beta, scaling the tool to perform up to 1000 FRAs a year once in Public Beta.
3.4 - Human decisions and review
Once the initial FRA is generated, a qualified Fraud Risk Assesor will evaluate the relevance, specification, and quality of the Fraud Risk Lines.
3.5 - Required training
Users will interact with the model via a user interface. This interface will state terms of use governing what types of document can be uploaded. This includes not uploading personal information. Users will be familiar with what a Fraud Risk Assessment is, given their roles. Users do not modify the algorithm but rather upload source documents, and review the tool’s outputs.
Public Sector Fraud Authority (PSFA) will also user guidance and give demos of the tool as it onboards users.
3.6 - Appeals and review
Not applicable - this is not available to the general public.
Tier 2 - Tool Specification
4.1.1 - System architecture
There are six main components of the architecture: Azure Blob Storage (Data Storage): Stores any project related documentation and its produced output. Data is hosted in the Azure Blob storage. Project related data consists of anything that will be ingested into the application e.g. Fraud Risk Assessment (FRA) Templates, Government Scheme information and Guidance documents used to support the creation of FRA. They are all stored using the folder structure: Templates, Schemes and Guides. With the outputs from the LLM stored in a separate container using the folder structure: Output, processed_guides, processed_schemes.
Azure AI Search: This is an information retrieval system that is highly compatible with generative AI search applications. The Index provides different types of search engines to retrieve the inputted documents, vectorize the documents (turn text into mathematical representations) and provide the necessary indexing.
Azure AI Search Index (Vector Database): Stores all the project related documentation as vector representations, to enable easier searching. Each index has a suffix identifying the respective documents that reside within it e.g. ‘-guides’, ‘-schemes’.
Open AI (LLM): Azure hosted version of the OpenAI framework. The integration provides a seamless connection between the Azure tech stack and leveraging the capabilities of advanced artificial intelligence. The application focussed on using GPT-4.1 to produce its results.
LangChain (OpenAI framework): Provides pre-built modules to be able to simplify the creation of LLMs within applications. Coupled with the chat.completions endpoint, LangChain is woven into the code to execute code in a manageable and readable way.
Azure OpenAI: Provides pre-built Large Language Models which can be called upon in application.
Azure WebApp: Platform service provided by Azure which builds, deploys, and scales web applications. In this case, our UI built with streamlit will be able to be accessible via a URL domain in a web browser. Users will need either CBAS credentials, or to be granted access, in order to interact with the application.
Our application calls on these services through their respective clients. The Azure Blob Storage Client allows us to download data so that it can be embedded and uploaded to vectors via Azure AI Search. With the vectors initialised we are then able to deploy an LLM model using Azure OpenAI (supported by LangChain) and create a system template to prompt the LLM. Streamlit is used to configure the application into a user-interface which is then deployed to Azure WebApp.
4.1.2 - Phase
Beta/Pilot
4.1.3 - Maintenance
Fraud Subject Matter Experts will be reviewing the output of the tool to ensure that it is working as intended.
Throughout Private Beta Deployment Public Sector Fraud Authority (PSFA) will review outputs and feedback from user testing group. During Public Beta PSFA will give permitted users the chance to periodically check in with the team around their experience.
We will also ensure that the application is maintained through code refinement, debugging, error handling, and other coding infrastructure considerations.
Maintenance of all Azure instances will be maintained by Cloud Based Analytics System.
4.1.4 - Models
GPT-4.1
Azure Language PII detection module.
Tier 2 - Model Specification
4.2.1 - Model name
GPT
4.2.2 - Model version
4.1
4.2.3 - Model task
Text Generation: Generates human-quality text in response to prompts. Image/PDF Understanding: processes and understands image/PDF contents to summarise and answer questions in response to prompts based on the image.
4.2.4 - Model input
The model input is a system prompt. This is a plain English set of instructions, questions, or other texts inputted by a human.
Additional unstructured data, such as longform documents, are also inputted as metadata for the model to use more relevant context in its answers.
4.2.5 - Model output
The model generates text based responses to prompts or inputted images. This text is distinct from its input.
This text then populates an Excel template as a final output.
4.2.6 - Model architecture
Large Language Model. Further architecture specific to GPT 4.1 is not available due to it not being released by OpenAI.
4.2.7 - Model performance
To date, only human review of output quality has been undertaken by Fraud Subject Matter Experts (SMEs). This feedback, which is quantitative and qualitative, is categorised into feedback categories.
We plan to carry out further user-feedback and incorporate Large Language Model (LLM) metrics such as using LLM-as-a-judge or other Natrual Language Processing metrics (i.e., BLEAU, BERT).
4.2.8 - Datasets
FRAs and respective schemes: Bounce Back Loans Nursery Milk Scheme 16-19 Bursary fund Youth Investment Fund Cladding Safety Scheme Internal FRAs used for training
Guidance Documents: These consist of government frameworks on creating Fraud Risk Assessments, as well as typlogies and definitions of fraud.
4.2.9 - Dataset purposes
The datasets have been used for RAG (Retrieval Augmented Generation), which provides an additional knowledge for a Large Language Model.
Tier 2 - Data Specification
4.3.1 - Source data name
FRAs, Schemes, Guidance Documents
4.3.2 - Data modality
Text
4.3.3 - Data description
FRAs: Fraud Risk Assessments Schemes: Government grant outlines in which are assessed through the FRA Guidance Documents: Extra context which are relevant for building FRAs
4.3.4 - Data quantities
FRAs: 12 Schemes: 6 Guidance: 9
4.3.5 - Sensitive attributes
Raw datasets contained:
- Government staff names
- Author names
Some schemes submitted may not be available to the public yet. Users will not be able to access other users’ data, as it is instance-based as opposed to kept long-term in the system. Furthermore, the tool is hosted in Cloud Based Analytics System and is ring-fenced in this server.
4.3.6 - Data completeness and representativeness
The dataset is unstructured/in long format which acts as an external knowledge source. It does not contain rows, and does not refer to a target population.
4.3.7 - Source data URL
N/A
4.3.8 - Data collection
These were collected from relevant fraud SMEs and open-source.
4.3.9 - Data cleaning
The data was scanned for personal data and these were removed.
4.3.10 - Data sharing agreements
N/A
4.3.11 - Data access and storage
The PSFA data science and analytics team will have access to back-end architecture. This includes coding repos in Azure DevOps, and input/output data stored in Azure Storage containers. This is hosted in a CBAS environment.
The user will be able to download generated FRAs through the user interface. Users will only be able to interact with documents that they upload. The user interface is an additional script using the Streamlit application, it is also hosted in CBAS. The user will be able to interact with the interface outside of CBAS.
Tier 2 - Risks, Mitigations and Impact Assessments
5.1 - Impact assessment
While the tool is not intended to process personal data, a Data Privacy Impact Assessment was completed to address the risk of individuals’ names (such as authors) appearing in uploaded documents and to consider the risks associated with the use of Generative AI models more generally (such as the unpredictable return of personal data in the model’s outputs). The DPIA was approved by PSFA’s SRO on 4 March 2025 and no material residual risks were identified.
5.2 - Risks and mitigations
The project uses the LLM GTP-4.1, as described in the earlier sections. Associated risks are:
-
Personal data in output/inputs: This has been mitigated by having no personal data be involved in inputs or prompting, and monitoring outputs.
-
Hallucinations: This is mitigated by establishing accountability and responsibility through human-in-the-loop feedback. We have had independent Fraud Subject Matter Experts evaluate generated Fraud Risk Assessments to implement improvements.
-
Fairness, Bias, and Discrimination: This is mitigated due to having no personal data/reference to specific persons. All Actors are defined in terms of archetypes. All outputs will be monitored for edge cases by the Public Sector Fraud Authority team.
-
Sensitive data: This has been mitigated by hosting the application on Cloud Based Analytics System as input data is sensitive.