Standard

Food Standards Agency: Food Hygiene Rating Scheme – AI

Published 6 July 2022

name	tier and category	description	entry (please enter all the required information in this column)
Name	Tier 1 - Overview	Colloquial name used to identify the algorithmic tool.	Food Hygiene Rating Scheme (FHRS) – AI
Description	Tier 1 - Overview	Give a basic overview of the purpose of the algorithmic tool. Explain how you’re using the algorithmic tool, including: * how your tool works * how your tool is incorporated into your decision making process Explain why you’re using the algorithmic tool, including: * what problem you’re aiming to solve using the tool, and how it’s solving the problem * your justification or rationale for using the tool * how people can find out more about the tool or ask a question - including offline options and a contact email address of the responsible organisation, team or contact person	1. How the tool works: 1.1 The tool is made up of a machine learning model, integrated into a web application that can be accessed by Stakeholders within the Local Authorities (LAs) in England, Wales and Northern Ireland. Use of the system is voluntary and aims to provide a standard methodology for prioritising inspections of food businesses based on their predicted food hygiene rating. 1.2 A machine learning framework called LightGBM was used to develop the FHRS AI model. This model was trained on data from three sources: internal Food Standards Agency (FSA) FHRS data, publicly available Census data from the 2011 census and open data from HERE API. 1.3 Using this data, the model is trained to predict the food hygiene rating of an establishment awaiting its first inspection, as well as predicting whether the establishment is compliant or not. 1.4 Access to the web application is only granted to stakeholders within Local Authorities who have agreed to the terms and conditions of use. Users will have access to information on establishments within their remit, for example, users from the Ealing LA will only be able to access information about establishments within the Ealing area. 1.5 This information is presented to the user in three ways: * as a table showing the establishment, predicted compliance and the predicted rating * as an interactive map, again showing the establishment, predicted compliance, predicted rating, and finally the type of establishment * as a CSV file download 2.How it is incorporated into decision making process: 2.1 This will be understood in more detail post alpha testing phase. A high-level summary of how we expect this to be incorporated into the decision-making process can be found in section 3.1 3. Problem we’re aiming to solve: 3.1 A significantly greater number of registrations of new businesses have been received by local authorities compared with pre-pandemic levels, and more businesses have changed their functions, which may require a new inspection to take place. 3.2 As a result, the number of businesses classified as ‘Awaiting Inspection’ on the Food Hygiene Rating Scheme website has increased steadily since the beginning of the pandemic. This has been the key driver behind the development of the FHRS AI use case. 3.3 More broadly, it is necessary to investigate whether this new technology can help LAs to manage their resources and increase the impact of their regulatory activity. 4. Justification/Rationale: 4.1 The objective is to help local authorities become more efficient in managing the hygiene inspection workload in the post-pandemic environment of constrained resources and rapidly evolving business models. 4.2 Not using Machine Learning may reduce the ability of local authorities to make use of data and digital to tackle their backlogs or to operate more efficiently. In the long-term this unrealised benefit may threaten the integrity of the food hygiene rating scheme or reduce consumer protection because a backlog may remain, or local authority resources may be used less efficiently.
URL of the website	Tier 1 - Overview	If available, provide the URL reference to a page with further information about the algorithmic tool and its use. This facilitates users searching more in-depth information about the practical use or technical details. This could, for instance, be a local government page, a link to a GitHub repository or a departmental landing page with additional information.	The link below provides details of the Food Standards Agency’s Food Hygiene Rating Scheme (FHRS). This scheme sets out to help consumers choose where to eat out or shop for food by providing each establishment with a food hygiene rating between 0 and 5. The link does not cover the use of the FHRS AI tool currently, rather it should be read to familiarise yourself with the broader FHRS scheme that will be supported by the tool. Food Hygiene Rating Scheme - Food Standards Agency
Contact email	Tier 1 - Overview	Provide the email address of the organisation, team or contact person for this entry.	strategic.surveillance@foodgov.onmicrosoft.com
1.1 Organisation/ department	Tier 2 - Owner and responsibility	Provide the full name of the organisation, department or public sector body that carries responsibility for use of the algorithmic tool. For example, ‘Department for Transport’.	Food Standards Agency (FSA)
1.2 Team	Tier 2 - Owner and responsibility	Provide the full name of the team that carries responsibility for use of the algorithmic tool.	Regulatory Compliance Division (RCD)
1.3 Senior responsible owner	Tier 2 - Owner and responsibility	Provide the role title of the senior responsible owner for the algorithmic tool.	Julie Pierce (FSA Director of Openness, Data, Digital, Wales and Science)
1.4 Supplier or developer of the algorithmic tool	Tier 2 - Owner and responsibility	Provide the name of any external organisation or person that has been contracted to develop the whole or parts of or the algorithmic tool.	Cognizant Worldwide Limited
1.5 External supplier identifier	Tier 2 - Owner and responsibility	If available, provide the Companies House number of the external organisation that has been contracted to develop the whole or parts of or the algorithmic tool. You can get a company’s Companies House number by finding company information or using the Companies House API	07195160
1.6 External supplier role	Tier 2 - Owner and responsibility	Give a short description of the role the external supplier assumed with regards to the development of the algorithmic tool.	Cognizant, in collaboration with the FSA’s Strategic Surveillance Team and Regulatory Compliance Division, and numerous Local Authorities were involved in the creation of the FHRS AI system.. This involved: * working with business and technical subject matter experts (SMEs) within the FSA to understand and define the use-case scope, approach and expected benefits and risks * collating and preparing relevant data sources to be used for predictive modelling based on SME feedback and ensuring data is of an acceptable quality * collaborating with SMEs to assess valuable approaches and datasets * developing and testing a minimum viable product through iterative feedback sessions with the business owner and relevant SMEs * presenting the minimum viable product (MVP), proposed next steps and roadmap for MVP deployment * liaising with local authorities to test the model against human intellect and analyse the results * facilitate future testing initiatives with local authorities As well as a development role, Cognizant have provided Responsible AI consulting and delivery services, including the application of a parallel Responsible AI sprint to assess risk and impact, enable model explainability and assess fairness, using a variety of artefacts, processes and tools.
1.7 Terms of access to data for external supplier	Tier 2 - Owner and responsibility	Details the terms of access to (government) data applied to the external supplier.	Cognizant employees (and other subsequent data service providers) have access to secure FSA devices with their own login credentials. Access to any internal data is governed by contractual agreements between the FSA and the data service providers. The data is stored in a secure government cloud and access is managed by the FSA security team. In this particular case, the majority of data used to enable the system is from 2011 UK Census data and from the HERE API. This is data that can be publicly accessed by anyone free of charge. Information about an establishment’s Food Hygiene ratings are also readily available online.
2.1 Scope	Tier 2 - Description	Describe the purpose of the tool in terms of what it’s been designed for and what it’s not been designed for. This can include a list of potential purposes that the tool was not designed to fulfil but which could constitute possible common misconceptions in the future.	The system has been designed for: * Supporting local authorities by taking a data-driven approach to prioritising their inspection regime. The FHRS rating and compliance prediction from the model, along with the knowledge of the subject matter experts engaging with the system, should be used together to deploy LA resources more effectively to establishments who show traits of being at a higher risk of non-compliance, and therefore those which may have a lower food hygiene rating. The system has not been designed for: * Replacing the current FHRS scheme and inspection regime. This means that food hygiene rating predicted by the model should not be used in place of the rating provided via a formal inspection in any situation * Using the predicted food hygiene rating in isolation to prioritise the inspection regime. This process should be a collaborative interface between the system and human decision maker to avoid any automated decision making
2.2 Benefit	Tier 2 - Description	Describe the key benefits that the algorithmic tool is expected to deliver, and an expanded justification on why the tool is being used.	The primary benefit the tool is expected to deliver is increased efficiency for local authorities in prioritising and inspecting food businesses, especially those within the ‘awaiting inspection’ category which are still awaiting their first inspection since registration. Currently, this process is manual, labour intensive and inconsistent across Local Authorities. Therefore, the use of a robust, data driven approach may allow for a more efficient utilisation of limited resources, by prioritising inspections of those businesses that are more likely to be non-compliant.
2.3 Alternatives considered	Tier 2 - Description	Provide, where applicable, a list of non-algorithmic alternatives considered, or a description of how the decision process was conducted previously.	No non-algorithmic alternatives were considered for this use case. In terms of how the decision process is currently conducted, each local authority has a different process for triaging food establishments based on historical information and information inherent to the establishment, such as type of food, whether it is a care facility etc. This information would then be used to prioritise the inspection regime. Due to the variations across all LA’s, it would be impractical to document how each one conducts the triage process in detail but in general the following factors are looked at as part of the decision-making process. * Catering to Priority Groups * Handling Process and Food Type * Food Operator History * Premise Structure * Reach and Distribution
2.4 Type of model	Tier 2 - Description	Indicate which types of methods or models the algorithm is using. For example, expert system, deep neural network and so on.	The algorithm being used as part of the FHRS AI use case at the FSA is the Light Gradient Boosting Machine (LightGBM) which is a framework for Machine Learning. The LightGBM is based on decision tree algorithms and used for ranking, classification and other machine learning tasks. As part of the use case, the model is used for classifying businesses based on the identified features as being compliant or not and further used for predicting a food hygiene rating (0-5).
2.5 Frequency of usage	Tier 2 - Description	Provide information on how regularly the algorithmic tool is being used. For example the number of decisions made per month, the number of citizens interacting with the tool, and so on.	Since the use case is in a MVP (Minimum Viable Product) stage, it hasn’t been released to actual end users as yet and hence the frequency of usage is something that cannot be determined at this point in time (FEB’2022). The Alpha pilot is currently planned for beginning in April 2022, where in the end users (at the participating Local Authorities) will have access to the FHRS AI service for use in their day-to-day workings. This section will be updated depending on the outcomes of the Alpha Pilot where we will be looking at the number of users accessing the service and time spent amongst other aspects.
2.6 Phase	Tier 2 - Description	Describe the phase in which of the following stages or phases the tool is currently situated: - idea - design - development - production - retired This field includes date and time stamps of creation and any updates.	Currently, the use case has been developed as a Minimum Viable Product. It is now entering a testing phase where it will be shared with a limited set of Local Authorities for system testing and feedback with multiple rounds planned. This phase can be described to be a ‘Minimum Viable Product (MVP) testing’ phase. If the FSA decides to operationalise the service, further iterations of enhancement and testing will be conducted.
2.7 Maintenance	Tier 2 - Description	Give details on the maintenance schedule and frequency of any reviews. For example, specific details on when and how a person reviews or checks the automated decision.	Since the use case is in a MVP (Minimum Viable Product) stage, it has not been released to actual end users as yet and hence the maintenance schedule is something that cannot be determined at this point in time (June 2022). The Alpha pilot started at the beginning of April 2022, wherein the end users (the participating Local Authorities) have access to the FHRS AI service for use in their day-to-day workings. This section will be updated depending on the outcomes of the Alpha Pilot through which we aim to capture feedback around the following: * Time saved by usage of service * Usefulness of service in prioritising inspections * Suitability for use as part of day to day work in the future * User Interface design * Requirement for a bi-lingual interface, especially for the Welsh LAs * Feedback on onboarding process and guidance document * Qualitative feedback on end to end process
2.8 System architecture	Tier 2 - Description	If available, provide the URL reference to documentation about the system architecture. For example, a link to a GitHub repository image or additional documentation about the system architecture.	This diagram below captures the high-level design for the key details of the service.
3.1 Process integration	Tier 2 - Oversight	Explain how the algorithmic tool is integrated into the decision-making process and what influence the algorithmic tool has on the decision-making process. Give a more detailed and extensive description of the wider decision-making process into which the algorithmic tool is embedded.	Utilising the service, the Environmental Health Officers (EHOs) are provided with the AI predictions, which are supplemented with their knowledge about the businesses in the area, to prioritise inspections and update their inspection plan. Please find a graphical representation of the technology aided inspection process, as envisioned for the future. Once the service is released to Local Authorities for use as part of the Alpha pilot, we will have a better understanding of usage and how it fits into the LAs’ decision making processes. Note: the system has been designed to provide decision support to Local Authorities. FSA has advised Local Authorities to never use this system in place of the current inspection regime or use it in isolation without further supporting information.
3.2 Provided information	Tier 2 - Oversight	Describe how much and what information the algorithmic tool provides to the decision maker.	The FHRS AI service is designed to share the compliance predictions for Food Establishments which are awaiting inspection. Through different consumption options available through the service (a list of predictions, a geographic prediction map or a downloadable predictions file) the end users get access to the following information * Local Authority Code and Name the food business falls under * Business Name, Type and Operating Post Code * Compliance Prediction along with the predicted FHRS rating (0-5) * Probability of a food establishment being compliant The above information is only available to end users for the Local Authorities that they belong to based on the data security rules applied.
3.3 Human decision	Tier 2 - Oversight	Describe the decisions that people take in the overall process, including human review options.	The role of the FHRS AI service within the LA decision making process for prioritising business inspections has been illustrated in 3.1. It is expected that there will be possibly multiple review points, with users being able to access only the predictions of their own Local Authority.
3.4 Required training	Tier 2 - Oversight	Describe the required training those deploying or using the algorithmic tool must undertake, if applicable; For example, the person responsible for the management of the tool had to complete data science training.	While there is no training provided or a pre-requisite for using the service, a user guide has been created for users of the system. This will be a single document which the end users can reference to understand how the service needs to be accessed and the different features available along with providing guidance in the form of FAQs on the recommended usage.
3.5 Appeals and review	Tier 2 - Oversight	Provide details on the mechanisms that are in place for review or appeal of the decision available to the general public.	The FHRS AI service is a voluntary service being made available for Local Authorities and is foreseen being used as a data driven approach to support the inspection process by helping them prioritise inspections of businesses awaiting inspection. Since there will be no change to the current inspection process by introducing the model, the existing appeal and review mechanisms will remain in place. Although the model is used for prioritisation purposes, it should not impact how the establishment is assessed during the inspection and therefore any challenges to a food hygiene rating would be made using the existing FHRS appeal mechanism.
4.1 Source data name	Tier 2 - Information on data	If applicable, provide the name of the datasets used.	The different internal and external sources that are processed as part of the FHRS AI model are 1. FHRS Data (FSA Internal) 2. Census Data (Open) 3. HERE API Data (Open)
4.2 Source data	Tier 2 - Information on data	Gives an overview of the data used to train and run the algorithmic tool. It will also specify whether data is used for training, testing, or operating. It should include which categories of data - for example ‘age’ or ‘address’ - which were used to train the model and which are used as input data for making a prediction.	The FHRS AI dataset contains approximately 120 data descriptors. While all of these cannot be detailed, the broad groups that these belong to can be represented as follows: * Business Category and Type (Internal Data) * Business Name and address (Internal Data) * Local Authority Code and Average food hygiene rating (Internal Data) * Demographics of Area (External Data) * Business Size (External Data) * Socio-Economic and Census Details of Area (External Data) * Business Operating Hours (External Data)
4.3 Source data URL	Tier 2 - Information on data	If available, provide a URL to the dataset.	The dataset utilises data from sources both internal and external to the FSA. No URL is available for this dataset as it is not publicly available.
4.4 Data collection	Tier 2 - Information on data	Gives information on the data collection process, including the original purpose of data collection.	The majority of data used to create the dataset comes from the Food Hygiene Rating Scheme collected by the different Local Authorities across England, Wales and Northern Ireland. The Local Authorities are responsible for conducting inspections of food establishments. Along with the FHRS system, the Census data prepared by the Office of National Statistics was used along with geographic data extracted from the HERE API. The FHRS system database has a history of hygiene inspections along with information about all the food businesses in the country. Since the data is collected and updated by the Local Authorities, this is considered as a true record of the core data about a food business. The Census data is maintained by the ONS and is assumed to require no additional verification/validation. The HERE API has not undergone a formal vetting process.
4.5 Data sharing agreements	Tier 2 - Information on data	Provides further information on data sharing agreements in place.	There were no data sharing agreements required for the development of this system.
4.6 Data access and storage	Tier 2 - Information on data	Provide details on who has or will have access to this data, how long it’s stored, under what circumstances and by whom.	The dataset will be accessed by data scientists at the FSA and be used for generating predictions to identify likely non-compliant food businesses which can inform Local Authorities’ inspection planning. The Local Authorities will not have access to the raw data and will only handle the predictions being generated from the model. The details regarding governance are considered sensitive and are not being shared outside of the Agency for now with the details still to be finalised as an outcome of the Alpha pilot.
5.1 Impact assessment name	Tier 2 - Risk mitigation and impact assessment	Provide the name and a short overview of the impact assessment conducted.	The different impact assessments conducted during the development of the use case were 1. Responsible AI Risk Assessment 2. Stakeholder Impact Assessment 3. Privacy Impact Assessment
5.2 Impact assessment description	Tier 2 - Risk mitigation and impact assessment	Give a description of the impact assessments conducted.	Please find below a description of the impact assessments mentioned as part of section 5.1 * Responsible AI Risk Assessment - Identify potential risks related to the use case, data and technology used to enable the use-case. Identification of these risks naturally leads to the consideration and documentation of potential mitigation techniques. This is conducted iteratively throughout the development of the use case, ensuring risks are continuously identified, assessed and managed. * Stakeholder Impact Assessment - Build confidence in the way you’ve designed and deployed your system by bringing to light unseen risks that can threaten to affect individuals and the public good. Use this assessment to demonstrate forethought and due diligence to the wider public that you and the project team have collaborated to evaluate the social impact and sustainability of your AI project. * Privacy Impact Assessment - Structured process to identify and minimise data protection risks of a project. They are a legal requirement under the GDPR when processing data that may lead to high risks to individuals. Generally, it is good practice to complete a privacy impact assessment when using personal data or data you think may constitute personal data.
5.3 Impact assessment date	Tier 2 - Risk mitigation and impact assessment	Provide the date in which the impact assessment was conducted.	The assessments are conducted in an iterative process throughout the design, development and delivery of the use case. While the assessments were carried out at an interval of every 10-12 weeks, the exact dates can be provided if absolutely required.
5.4 Impact assessment link	Tier 2 - Risk mitigation and impact assessment	If available, provide a link to the impact assessment.	Due to the confidential nature of the information contained in these assessments, this information cannot currently be shared outside of the agency.
5.5 Risk name	Tier 2 - Risk mitigation and impact assessment	Provide an overview of the common risks for the algorithmic tool.	The foreseen risks associated with the use of the FHRS AI service include the following 1. Use of Personal Information 2. Model Bias 3. Inspection Bias 4. Decision Automation Bias / Technological Halo Effect
5.6 Risk description	Tier 2 - Risk mitigation and impact assessment	Give a description of the risks identified.	The risks listed in 5.5 are further detailed out below 1. A number of businesses are registered at private home addresses. This information is required to be managed and used securely as it may constitute personal information. 2. Potential bias from the model (e.g. consistently scoring establishments of a certain type much lower, less accurate predictions) 3. Potential bias from inspectors seeing predicted food hygiene ratings and whether the system has classified the establishment as compliant or not. This may have an impact on how the organisation is perceived before receiving a full inspection. 4. With the use of AI/ML there is a chance of decision automation bias or automation distrust bias occurring. Essentially, this refers to a user being over or under reliant on the system leading to a degradation of human-reasoning.
5.7 Risk mitigation	Tier 2 - Risk mitigation and impact assessment	Provide an overview of how the risks have been mitigated.	The risks listed in 5.5 and further elaborated in 5.6 have been carefully looked at and mitigated through 1. While the private addresses are not included as part of the Alpha version of the service, a Privacy Impact Assessment has been conducted and signed off by the FSA’s Knowledge, Information Management and Security (KIMS) team 2. Integration of explainability and fairness related tooling during exploration and model development. These tools will also be integrated and monitored post-alpha testing to detect and mitigate potential biases from the system once fully operational 3. Continuously reflect, act and justify sessions with business and technical subject matter experts throughout the delivery of the project, along with the use of the three impact assessments outlined earlier to identify, assess and manage project risks 4. Development of usage guidance for local authorities specifically outlining how the service is expected to be used. This document also clearly states how the service should not be used, for example, the model outcome must not be the only indicator used when prioritising businesses for inspection.