MoJ: AICE Chatbot "Nova"
FAQ Chatbot for MOJ/HMPPS Resourcing.
1. Summary
1 - Name
AICE Chatbot - “Nova”
2 - Description
The tool is an FAQ Chatbot for Resourcing (Recruitment) across roles for MOJ & HMPPS. The chatbot uses a bank of curated responses to frequently asked questions with free text question entry by the user. The tool uses Natural Language Understanding (NLU) to identify & categorise the input of the user in order to match it to the most relavent response.
3 - Website URL
https://help.nicecxone.com/content/aiassistantsandbots/botbuilder/cxonebotbuilder.htm - NiCE Bot Builder Setup & Technical Documentation
4 - Contact email
serviceimprovement@resourcing.soprasteria.co.uk; matt.garen@resourcing.soprasteria.co.uk
Tier 2 - Owner and Responsibility
1.1 - Organisation or department
Ministry of Justice
1.2 - Team
Resourcing Customer Hub
1.3 - Senior responsible owner
Head of Resourcing Customer Hubs
1.4 - Third party involvement
Yes
1.4.1 - Third party
NiCE Systems LTD, Route101 LTD
1.4.2 - Companies House Number
Nice Systems LTD- 03403044 Route101 LTD - 08325675
1.4.3 - Third party role
Both NiCE & Route101 were contracted to develop the tool in conjunction with TBS Resourcing. Backend configuration completed by both third parties. Training & Guidance were provided and building of the chat bot was completed internally by TBS Resourcing. NiCE provide cloud hosting for the Chatbot itself, and Route101 provide cloud hosting for the Chat widget deployed on TBS Resourcing websites.
1.4.4 - Procurement procedure type
NICE went through a competitive selection process and are a preferred supplier utilised for a range of SaaS capabilities. We have a Master Partner Agreement (MPA) in place but will also contract discretely depending on specific requirements. Route101 are a sub-contractor via NICE, as the implementation partner.
1.4.5 - Third party data access terms
NiCE are the SaaS supplier of the platform where the chatbot is hosted. Access by third party is restricted to employees assigned to our account with the relavent permissions and security controls. Route101 access limited to scope of implementation, with employees assigned to account with relevent permissions and security controls.
Tier 2 - Description and Rationale
2.1 - Detailed description
The tool is an FAQ Chatbot to support applicants or potential applicants to roles within the MOJ/HMPPS. The chatbot is designed to be conversational, allowing users to enter via free text any question or query. Responses provided by the chatbot are managed by the Resourcing Customer Hub Subject Matter Experts, and are made of up concise summaries of publicly accessible information. Responses are selected and matched to the input via Natural Language Understanding (NLU). The aim is to provide a self service option to customers to find quick answers to frequently asked questions about the recruitment process.
2.2 - Benefits
Enabled customers to get quick responses to frequently asked questions, without reading through articles/web pages to find an answer. This provides greater access to information for applicants/potential applicants for roles across the MOJ & HMPPS. This tool is designed to be quick and easy to access, removing the need to make a phone call/send an email to an Advisor within the Resourcing Customer Hub.
2.3 - Previous process
A chat bot tool did previously exist, containing much of the same FAQ information. The tool allowed users to select through information via button clicks to find an answer, but was not conversational (no direct question could be asked via free text)
2.4 - Alternatives considered
Considerations were made to use AI generated responses to FAQ’s via an LLM, to allow for a higher response rate and more tailored answers to questions. This would have enhanced the benefits, however the trade off was ensuring that information provided in responses was always appropriate and accurate. Given an LLM must always provide a response to the best of its ability, much more robust guardrails would be needed to ensure responses were always accurate and appropriate.
Tier 2 - Deployment Context
3.1 - Integration into broader operational process
The chatbot is deployed via chat widget onto both informational microsites, and onto the candidate portal. It is accessible publicly by anybody. The chat bot must be used initially by applicants/potential applicants before the user can be transferred to a human advisor via live chat. If prompted to, the tool will handover the user to the live chat within Resourcing Customer Hub opening hours. The tool does not make any decision, take any action relating to an individual users account/application, or have any influence on the process followed by human advisors. The tool exists to provide responses to frequently asked questions quickly and efficiently. The responses provided are generic information relating to the MOJ/HMPPS recruitment process, not specific to an individual. Information input by the user is processed against an NLU model to match the query to the most appropriate response. Conversation data is used by Customer Hub SMEs to manually input new utterences and intents into the model. The model is then trained based on the inputs provided by the SME’s, not directly using conversation data.
3.2 - Human review
The Customer Hub SMEs will review individual conversations, intent data, misunderstood messages data and conversation journeys to assess performance of the chatbot. Additions/changes to the content of responses is directly managed by the SMEs, as well as additions/changes to utterences and intents built into the model. Training and deployment of the language model is controlled by Hub SMEs.
3.3 - Frequency and scale of usage
The chatbot is available for usage 24/7, and handles on average 400-600 conversations per day (14,000-15,000 per month) This is a mix of one time and repeat users.
3.4 - Required training
No training is required for users/customers of the tool. Internal training is required for any individual prior to accessing any configuration element of the tool.
3.5 - Appeals and review
N/A no decisions.
Tier 2 - Tool Specification
4.1.1 - System architecture
The chatbot forms part of the wider CXone SaaS offering, and is hosted on AWS Clusters in the UK. Key technical features and configuration examples are viable in the below documentation. Bot Builder Documentation - https://help.nicecxone.com/PoC.htm#cshid=CXONEBOTBUILDER The chatbot is deployed to websites via an Iframe chat widget through Google Tag Manager.
4.1.2 - System-level input
General recruitment process queries e.g. Length of process, Interview preperation, Application guidance
4.1.3 - System-level output
Pre-defined and managed responses. General recruitment process information e.g. Application guidance, Interview Preparation guidance.
4.1.4 - Maintenance
Hub SME’s review chatbot content and conversations weekly, with scheduled development and deployment windows for inputting data into the model, and training/deploying the model to production.
4.1.5 - Models
Bot Builder uses Natural Language Understanding (NLU) to understand what contacts say and make accurate predictions about what they mean.
Tier 2 - Model Specification
4.2.1. - Model name
Self Hosted
4.2.2 - Model version
N/A
4.2.3 - Model task
N/A
4.2.4 - Model input
N/A
4.2.5 - Model output
N/A
4.2.6 - Model architecture
N/A
4.2.7 - Model performance
N/A
4.2.8 - Datasets and their purposes
N/A
2.4.3. Development Data
4.3.1 - Development data description
Conversation data through user input to the chatbot. Conversation data is manually reviewed by Hub SMEs to then manually make additions/changes to the model. Conversation data is not directly used in training of the model.
4.3.2 - Data modality
Text
4.3.3 - Data quantities
N/A (Datasets only used where gap between model performance and expected result is identified)
4.3.4 - Sensitive attributes
The chatbot does not require input of any sensitive data. An administrative control is in place, where the chatbot informs the user that no personal data should be entered. Given the input is free text, the user may still enter sensitive information of their own accord, however this data is not used to train the model, and is not accessible by any individual without the relevant permissions and security controls.
4.3.5 - Data completeness and representativeness
N/A
4.3.6 - Data cleaning
Not performed on data set
4.3.7 - Data collection
Purpose of collecting data input by user is to provide responses to recruitment queries, and to manually review and train model.
4.3.8 - Data access and storage
Access is only available to individuals with relevant permissions and security controls. Data is stored for 13 months as per wider data retention policy and automatically deleted. Responsibility for data storage is with NiCE Systems as the third party SaaS provider. Responsibility for data retention schedule sits with TBS Resourcing directly.
4.3.9 - Data sharing agreements
N/A
Tier 2 - Operational Data Specification
4.4.1 - Data sources
Applicants/Potential applicants, user input
4.4.2 - Sensitive attributes
N/A
4.4.3 - Data processing methods
N/A
4.4.4 - Data access and storage
Access is only available to individuals with relevant permissions and security controls. Data is stored for 13 months as per wider data retention policy and automatically deleted. Responsibility for data storage is with NiCE Systems as the third party SaaS provider. Responsibility for data retention schedule sits with TBS Resourcing directly.
4.4.5 - Data sharing agreements
N/A
Tier 2 - Risks, Mitigations and Impact Assessments
5.1 - Impact assessments
DPIA completed prior to deployment of tool, but not publicly accessible. Risks were found to be in appetite after mitigation via additional administrative control, and personel’s access via permissions and security controls.
5.2 - Risks and mitigations
The chatbot does not require input of any sensitive data. An administrative control is in place, where the chatbot informs the user that no personal data should be entered. Given the input is free text, the user may still enter sensitive information of their own accord, however this data is not used to train the model, and is not accessible by any individual without the relevant permissions and security controls.