DfT: EV Chargepoint Grant Photos Fraud Tool
The tool uses a deep learning image recognition model and Google's Vision API to detect potentially fraudulent or erroneous photos (photos of duplicate installations or photos downloaded from the web) that are submitted as evidence to EV chargepoint grant schemes
1. Summary
1 - Name
EV Chargepoint Grant Photos Fraud Tool
2 - Description
Why: the tool is being used to detect and prevent fraudulent/erroneous photos being submitted as evidence in multiple EV chargepoint grant schemes.
How: the pipeline has two sections. Photos submitted to the schemes are checked for two risks of error/fraud, one per section. These checks are 1) looking for images of duplicate installations across the schemes and 2) looking for images of chargepoints downloaded from the web
3 - Website URL
N/A
4 - Contact email
Tier 2 - Owner and Responsibility
1.1 - Organisation or department
Department for Transport (DfT) and Office for Zero Emission Vehicles (OZEV)
1.2 - Team
Data Science Team
1.3 - Senior responsible owner
Head of Data Science.
1.4 - Third party involvement
No
1.4.1 - Third party
N/A
1.4.2 - Companies House Number
N/A
1.4.3 - Third party role
N/A
1.4.4 - Procurement procedure type
N/A
1.4.5 - Third party data access terms
N/A
Tier 2 - Description and Rationale
2.1 - Detailed description
The purpose of the tool is to detect and prevent two kinds of fraud/error as described above in our EV chargepoint grant schemes.
The tool is built and maintained by DfTc/OZEV and is used in supoorting grants processing and payments workflows owned by DVLA & Counterfraud.
It is designed to be used to check every single usable photo in the grant schemes. (i.e. excluding images of documents, chargepoint labels etc. that appear in the database).
The pipeline consists of two sections checking for two separate types of potentially fraudulent/erroneous photo submission:
-
Duplicate installations – installers submitting images of one installation on multiple grant applications.
-
Downloaded images – installers submitting chargepoint images obtained from the web on a false application. These are checked for using Google’s Vision API which detects if an image appears on the web.
How are these checked?
-
We look for duplicate installations using a deep learning image recognition model to extract image representations and we calculate a similarity score for each pair of images in the database. Image pairs with a similarity score above a threshold are labelled as “similar” and returned to colleagues in the DVLA (grant administrators) to follow up with (i.e. determine if the images are of the same installation and if so whether there is a legitimate reason for them to appear on separate grant applications)
-
We look for images downloaded from the web using Google’s Vision API, which returns the URLs of webpages where the images appear. These are then returned to the DVLA following the same process described for duplicate installations above.
2.2 - Benefits
Moderate financial savings (~10s of £1,000s) from companies paying back falsely approved grant funding or not receiving it in the first place.
Very large time savings compared to manual labour cost of running the same level of detailed checks (far in excess of 100 years would be required to run the same checks manually).
These fraud checks could not be done at the same level of detail without this tool.
2.3 - Previous process
Earlier approaches to fraud risk identification relied on manual reviews and subjective assessments, which were resource intensive and less efficient.
2.4 - Alternatives considered
Non-algorithmic approaches not feasible given extent of manual checking that would be required.
Other algorithmic approaches considered during prototyping phase, current approach proved the most effective.
Tier 2 - Tool Specification
4.1.1 - System architecture
GCP Diagram can be found here: https://drive.google.com/file/d/12RkhLoNCJ5Vgr95Qtqr9GDKcM0m8ASTr/view?usp=sharing
Additional description: For web checks: Photos are loaded from the Google Cloud Storage bucket and processed by Google’s Vision API. Outputs are URLs of webpages where images appear.
For the duplicate checks: Photos are loaded from the GCS bucket and processed by the deep learning image recognition model to output image representations (arrays) organised by grant scheme and application number.
Image reps are then compared to output a similarity. Outputs are GCS links to pairs of images with high similarity scores.
4.1.2 - System-level input
Unstructured data: images
4.1.3 - System-level output
Unstructured data: text (urls) and images Numerical data: similarity scores for duplicate images
4.1.4 - Maintenance
Weekly review, monthly or more frequent maintenance as needed
4.1.5 - Models
Convolutional neural network model for extracting deep image feature representations
Google Vision API: Cloud-based machine learning models for image analysis and computer vision processing
Tier 2 - Deployment Context
3.1 - Integration into broader operational process
Currently, the tool batch-processes a backlog of images. Therefore, it does not change the decision making process as to whether a grant is initially approved or not.
Instead, the tool runs retrospective scans of images and flags any outliers as potentially problematic. These flagged images are fed back to the DVLA grant administrators for further checks. It is up to them to determine based on the images flagged by the tool whether or not there is an issue - it is important to note the tool will only flag images, it does not otherwise influence decision making.
For the duplicate checks, the tool provides the cloud storage links of images flagged as similar along with a similarity score to the users
For images flagged by the web checks, the tool provides 1 or more URLs where the image was detected online to the users
We are in the process of work to implement the tool in “real time” - this would mean that colleagues in the DVLA would see if images have been proactively flagged as potentially problematic before approving grants. This is so that the process is able to minimize the number of erroneously disbursed grants.
3.2 - Human review
A human still always makes the final decision - grant administrators look at the submitted evidence and make a decision on the grant outcome.
The images flagged by the tool are always checked by a human. The tool has no capacity to make decisions on its own. A human has to look at the images flagged as similar and make their own ascertation as to whether they are of the same installation or not. They are told that images may well be different.
3.3 - Frequency and scale of usage
Currently we have used the tool several times on large backlogs of images. In the future we plan to use it on each image that is submitted to the grant schemes which will be in the 10s or 100s per month.
However it is worth reiterating the tool makes no decisions - it provides information and the decisions are still made by DVLA colleagues. Citizens will not interact directly with the tool.
3.4 - Required training
Grant administrators are given guidance in the form of a brief Manual of Procedure that explains how to check the results from the tool.
3.5 - Appeals and review
The same mechanisms of appeal will apply as currently, as the decision making process remains largely the same. The only difference is grant administrators are making the decision with more information.
The appeals process is as follows:
- If DVLA decline a claim and the installer wishes to appeal, the installer would contact DVLA in the first instance.
- If the appeal relates to an issue DVLA is unable to deal with, DVLA would escalate it to OZEV’s grant team
- The grant team would review the information the installer has provided, request full details and supporting evidence as required and make a determination.
If an applicant is still unhappy with the decision, the usual routes of judicial review would apply.
Tier 2 - Model Specification
4.2.1. - Model name
EXEMPT: Following consideration of the public interest, detailed information about the model architecture and implementation has been withheld, as its disclosure could enable the tool to be manipulated or intentionally gamed. A high-level description has been provided in accordance with the ATRS Mandatory Scope and Exemptions Policy.
4.2.2 - Model version
same as above
4.2.3 - Model task
same as above
4.2.4 - Model input
same as above
4.2.5 - Model output
same as above
4.2.6 - Model architecture
same as above
4.2.7 - Model performance
same as above
4.2.8 - Datasets and their purposes
same as above
2.4.3. Development Data
4.3.1 - Development data description
4.3.2 - Data modality
4.3.3 - Data quantities
4.3.4 - Sensitive attributes
4.3.5 - Data completeness and representativeness
4.3.6 - Data cleaning
4.3.7 - Data collection
4.3.8 - Data access and storage
4.3.9 - Data sharing agreements
Tier 2 - Operational Data Specification
4.4.1 - Data sources
Images submitted as evidence in EV chargepoint grant applications
4.4.2 - Sensitive attributes
Installers are instructed not to include any personal data (e.g. photos of themselves, certain documents) in their image submissions, however some may get submitted to the dataset.
We have a filtering process which automatically removes most of this (e.g. forms incorrectly submitted as images). Developers/users of the tool are also instructed to destroy any personal data that they may come across.
All data is stored in line with GDPR, and there is a Memorandum of Understanding regarding data sharing, processing, storage and privacy between OZEV and the DVLA.
4.4.3 - Data processing methods
N/A
4.4.4 - Data access and storage
Currently the only operational data that is stored is the inputs (the images data described elsewhere) and the outputs (image representations in the form of numerical arrays and similarity scores for images).
Input data is personal and so the requirements stated in the OZEV Personal Information charter apply, we store these for 3 years.
The people who can access this data belong to 3 groups: DfT (including Data Science, Counter Fraud and OZEV, all with direct need and Cloud teams with a technical need for Cloud architecture requirements), the DVLA (Grant processing teams) and Capgemini, the contractor responsible for the data pipeline whereby grant evidence is submitted by installers on a webform, sent to DfT cloud storage and information is extracted to send to the MS Dynamics dashboard the DVLA use.
4.4.5 - Data sharing agreements
Data sharing between DfT (OZEV) and the DVLA is covered by OZEV’s Personal information charter (link below) along with a memorandum of understanding setting out specific requirements for use.
Tier 2 - Risks, Mitigations and Impact Assessments
5.1 - Impact assessments
DPIA completed 2022. Algorithmic IA to be completed. Key findings: Few risks, all low impact. Main risks and mitigation strategies from these IAs are: Risk that information could be lost, released or shared inappropriately. Exchanges between PSFA (Public Sector Fraud Authority) and DfT protectively marked as OFFSEN, therefore limiting ability to share outside .gov accounts without warning to check recipients.
Risk of personal data being sent to DfT in transfers from DVLA - tool already in place to filter and destroy this if it should happen.
Risk that image that is not a duplicate/gleaned from web being classed as potential fraudulent supporting image. Mitigated by manual checks by DVLA staff of all potentially fraudulent images.
Regarding monitoring and evaluation, we will continue to check the number of true (e.g. a duplicate pair) and false positives (e.g. images flagged as similar by the tool but classed as different by manual checks) being output by the tool. This happens on an ongoing basis. In addition, we supplement this with information from grant processors - when they spot other irregularities in the image data (without using the tool) we can then check the tool is picking these up or take action to improve the tool. We have already made successful modifications with this process.
5.2 - Risks and mitigations
Low risk. Main risk would be an incorrect assignment of a duplicate label. Risk significantly mitigated as decisions are not automated, all photos are reviewed by a human once the algorithm has flagged potential duplicates. In addition, the decisions from that point are also manual, as in a reviewer cannot simply click “accept” on a pair of images (which would make it easier to mistakenly label) they have to manually check and follow the process to confirm and take action. All reviewers are given guidance on the outputs of the tool and know that in many cases images may be flagged as similar but can be seen to be of different locations, so they are aware that they actively need to consider themselves any images presented as duplicates.