NHS 111 online live assessment
111 online advises users about what to do and where to go when they have an urgent medical concern.
Digital Service Standard assessment report
NHS 111 online
| From: | Department |
| Assessment date: | 17/09/19 |
| Stage: | Live |
| Result: | Met |
| Service provider: | Department of Health and Social Care (DHSC) |
Previous assessment reports
Service description
111 online advises users about what to do and where to go when they have an urgent medical concern. It allows users with medical concerns to complete a health assessment online by answering a series of questions about their condition, based on a clinically-approved set of algorithms. Anyone in England can visit 111.nhs.uk, answer questions about their symptoms, and be directed to self-care advice or appropriate medical services. If needed, the user is invited to book a call with an urgent care service, like an out-of-hours GP or an emergency dentist. The service is free-to-use and available nationwide 24/7. NHS 111 online is integrated with all NHS 111 providers and out-of-hours GPs in England, meaning these urgent care services can accept patients directly from 111.nhs.uk.
Service users
Users of the service are anyone in England with an urgent medical concern who is unsure about what to do or where to go. The service is likely to have multiple millions of users every year.
1. Understand user needs
Decision
The service met point 1 of the Standard.
What the team has done well
The panel was impressed that:
- the service team has done considerable user research to try and understand its large and complex user base
- uncovered an array of reasons that users have cited for using this service, and the service’s role in the larger end-to-end journey
- the service team has worked iteratively: they have made design decisions based on their research findings, for example making a large number of changes to content as a direct result of insights gleaned through usability testing
- usability testing was conducted with users with a range of accessibility needs. A number of users with hearing difficulties were included in the research, and these users were consistently able to use the service without issue (although there were issues with onward referrals to phone based services). Research was also carried out with low digital-capability users who were, by and large, also able to use the service
- the service is gathering contextual feedback from users throughout the journey
What the team needs to explore
After going live, the team needs to:
- conduct more research with users with accessibility needs. While efforts were made to test with users with a wide range of accessibility needs - particularly deaf users – the number of these users was still relatively low and the service team would benefit from more testing with these user groups
- do more research aimed specifically at uncovering potential issues linked to the ‘edge cases’ they encountered, for example research highlighted that some users used several assistive technologies in unison, whereas the majority of testing focused on users using only one type of assistive technology
- include more systematic research with the healthcare providers who will also be using this service, such as frontline staff, and build on the work with out-of-hours providers and stakeholders in A&E and 999. While some research, largely informal, has been conducted with these healthcare providers, the main focus of research has been on members of the public using this service
- continue to improve non-binary gender identity options
2. Do ongoing user research
Decision
The service met point 2 of the Standard.
What the team has done well
The panel was impressed that:
- the service team has a detailed research roadmap ready to implement once the service has moved into the live phase
- this plan includes further work with users with low digital skills
- the whole service team is invested in the research process and considerate of the user experience
- research sessions are observed by numerous service team members and analysis sessions are run with the whole team
- in situ research is favoured over research in a usability lab, as the service team is keen to see how their service is used in context
- the team is running user forums to get feedback on how the service is impacting on the local healthcare system, which is being well received
- the team is sharing research and has been feeding into other areas of the NHS including Pathways, shaping the future of triaging across the NHS
What the team needs to explore
After going live, the team needs to:
- continue iterating the service based on high quality user research
- ensure the research roadmap is followed
- follow plans to conduct more research into how to make service names meaningful to both members of the public and users within the healthcare system
3. Have a multidisciplinary team
Decision
The service met point 3 of the Standard.
What the team has done well
The panel was impressed that:
- the service team is well-resourced with NHS Digital staff across the full range of expected delivery roles
- there is a low reliance on contractors
- the team includes two full-time user researchers and all staff are expected and encouraged to take part in user research activities
- the service team works closely with the implementation team and uses this relationship as an important feedback mechanism
- a delivery manager has been recruited for the implementation team
- there are clinicians on the team, giving it access to necessary expertise and building its credibility within the organisation
What the team needs to explore
Before moving into live, the team needs to:
- ensure there is an appropriate handover period for the service owner role, and that the new owner is a full time employee who can build on the current service owner’s excellent work of negotiating the complex governance landscape in the NHS
- ensure that there are plans for maintaining a stable service in the move to run/maintain, including strong management and executive support of any organisational change
- ensure procedures are in place to give the team access to the infrastructure lead for updates/emergency work on the service
4. Use agile methods
Decision
The service met point 4 of the Standard.
What the team has done well
The panel was impressed that:
- the service team has clearly iterated the structure and content of the service in response to user feedback since the last assessment (for example the positioning of a map feature and the presentation of instructions around calling 999)
- the team is using appropriate tools (including Slack, Trello, Jira, Skype) to overcome challenges of being based in different parts of the country
- there is a cadence to sprints and ceremonies that allows work to be delivered at pace
- the team has a clear narrative around the service and understands its history, including previous services assessments (for example “since the last assessment we’ve changed 169 questions based on specific user feedback”)
- the team has documented different options it has explored and discarded in user testing
- the team is empowered to make content changes and release them in a matter of days
- the team currently appears well-prepared to push larger changes through the complex NHS governance processes and boards
What the team needs to explore
After going live, the team needs to:
- continue to proactively work to streamline approvals processes that could slow down delivery of changes
5. Iterate and improve frequently
Decision
The service met point 5 of the Standard.
What the team has done well
The panel was impressed that:
- the team has done the hard work to make the symptom search work for users
- although constraints with the Pathways component mean the team can’t make content changes that impact on the flow of the triage, the team has reworded them based on user research, approved by clinicians
- the team is empowered to make content changes and release them in a matter of days
- the team has been working on 20 to 30 questions authored per sprint, making content clearer by breaking down questions and providing information on why questions are being asked
- the team has been using new online platforms to help collaborate remotely and in person on research and design throughout the process
- the content designers have heavily focussed on improving question comprehension, iterating between 20 and 30 questions per sprint which adds up to over 1000 so far
- they also demonstrated how they’ve made it easier for users to give accurate answers by explaining why questions are being asked. It was interesting to see future designs that could include images or audio
- the team effectively prioritises insights from user research, for example making choices around language that prioritises comprehension over sensibility
What the team needs to explore
After going live, the team needs to:
- continue to follow its summary research plan or to reprioritise its activities based on user insights
6. Evaluate tools and systems
Decision
The service met point 6 of the Standard.
What the team has done well
The panel was impressed that:
- the service team has responded to feedback by improving the system on multiple occasions, including in previous assessments and after penetration testing
- the service team could show clear improvement by re-evaluating the tool choice for address lookup by switching to a commercial tool when they realised their existing tool was not robust enough for the full service
- the service team has adopted a good microservice based architecture which allows them to iterate or improve specific component at a time, though this is not consistent across all components
- the service team was willing to shift from a largely IaaS to a largely PaaS infrastructure after deciding that the original configuration of the service (which relied on a number of Windows servers running Docker) was making management and deployment of the service more difficult
What the team needs to explore
After going live, the team needs to:
- continue to decouple version-dependent components in the system so that individual components can be modified or upgraded without affecting the entire service
- ensure that the practice of continuous improvement continues post-live, including regularly considering and evaluating new tools or infrastructure which could potentially improve the service and running small discoveries to test them
7. Understand security and privacy issues
Decision
The service met point 7 of the Standard.
What the team has done well
The panel was impressed that:
- the team clearly understands the privacy issues facing the service
- although the triaging functionality does not store user data, extra work has been done to mitigate risks, for example discouraging users from leaving personal information in feedback comments
- the team has made a clear effort to ensure the service is stateless and no data is stored until the ITK referral service is called, at which point data is stored in an encrypted database
- the service team has made a conscious decision to store their security keys offline from the cloud service to ensure complete separation from their cloud provider
- the service team has made sure that all service logging is anonymised (for example age and gender only)
- private keys that could be used by the service team for maintenance are stored in a secure vault
What the team needs to explore
After going live, the team needs to:
- check that their process for debugging the SQL database (by using a private key from a vault) is properly logged and audited so that no individual on the service team could access the potentially sensitive data without approval from a third party, and ensure there is appropriate segregation of duty between development and ops users
8. Make all new source code open
Decision
The service met point 8 of the Standard.
What the team has done well
The panel was impressed that:
- much of the code written by the service team has been open sourced or written in the open, with a few exceptions such as the licensed Pathways data
What the team needs to explore
After going live, the team needs to:
- continue to work with shared components (such as the DOS service) to encourage them to work in the open and open source their code to follow the Service Standard
9. Use open standards and common platforms
Decision
The service met point 9 of the Standard.
What the team has done well
The panel was impressed that:
- the service uses a large number of common or shared components, both from within and outside the NHS
- the service has worked carefully with their shared components to reuse as much as possible, and only built new code when required
- the service team is planning to adapt the code base so that it can be used as a generic clinical decision support API for potential future innovation
- the way the service team has shared components they have written with other teams, such as sharing the feedback API with NHS UK to help them get started early in their user journey
What the team needs to explore
After going live, the team needs to:
- consider if it’s possible to standardise or open the code and infrastructure that enables network connectivity with N3 so that other services can use it
10. Test the end-to-end service
Decision
The service met point 10 of the Standard.
What the team has done well
The panel was impressed that:
- the team is regularly researching the end-to-end journey and is working with NHS communications teams
- the team is working with stakeholders across NHS services to improve the service
- the service team has a comprehensive architecture monitoring and logging stack, with Nagios checking every endpoint regularly and performing basic checks such as pinging services to make sure they are available
- the service team effectively performs resiliency and failover tests during every code push due to their active-active configuration
- has used Akamai to front both the N3 connection and the primary service to ensure they can test, rate limit, secure and fully understand how traffic flows across the service
What the team needs to explore
After going live, the team needs to:
- continue their work to move into the NHSD central Splunk instance so that they can use the centrally provided operations and security desks
11. Make a plan for being offline
Decision
The service met point 11 of the Standard.
What the team has done well
The panel was impressed that:
- the team has a clear understanding of how users would be affected if the service were unavailable
- the team has a clear strategy for dealing with outages, primarily using the telephone service as an alternative channel
- plans have been made to switch off the IVR message in the event of an outage
- the team has configured the service in an active-active state ensuring a quicker return to service in case of an outage and a reduced requirement for DR testing
- technical, clinical and operational teams are on-call 24/7
What the team needs to explore
After going live, the team needs to:
- continue to review communications plans in the event of the service being unavailable, including via GP surgeries
12: Make sure users succeed first time
Decision
The service met point 12 of the Standard.
What the team has done well
The panel was impressed that:
- the team has been running exit surveys with users with access needs and have conducted follow up interviews with many of these users to better understand how to improve the end-to-end service
- there are many other points of entry, which the service sits alongside. Users can access services over the phone, through pharmacies and GP practices for example
What the team needs to explore
After going live, the team needs to:
- reach out to the cross government user centred design community to understand the work that has been done around webchat
13. Make the user experience consistent with GOV.UK
Decision
The service met point 13 of the Standard.
What the team has done well
The panel was impressed that:
- the team feed into the NHS service manual and NHS design patterns in a collaborative way through Slack, at meetups and design crits or through Github
What the team needs to explore
After going live, the team needs to:
- work more closely with different teams to ensure a linked-up and consistent user journey eg. The NHS app
14. Encourage everyone to use the digital service
Decision
The service met point 14 of the Standard.
What the team has done well
The panel was impressed that:
- the service has seen steady progress in uptake of the digital service, up to 11.9% of the total 111 usage across phone and online
- the team has gathered evidence around strategies for increasing digital take-up in particular regions (for example the locally-led information campaign in Manchester led to usage increasing to over 20% of total 111 usage across the phone and online)
- the team understands that the service exists in a complex ecosystem of NHS services, including phone and in-person channels, and has taken a pragmatic approach to complementing these existing routes
What the team needs to explore
After going live, the team needs to:
- develop and more clearly articulate its plans for moving users to the digital service, including yearly targets for increasing digital take-up for the next 5 years
15. Collect performance data
Decision
The service met point 15 of the Standard.
What the team has done well
The panel was impressed that:
- the team has gathered together a range of data which measures user behaviour, user feedback and performance against success criteria and KPIs
- data from different sources is used in combination to build a clear picture of performance
- this data is effectively used to identify areas of the service to improve and to inform the team and stakeholders of the performance of the service
- data analysis of user interactions with the service is carried out by the Data Analyst to identify issues and assess the impacts of changes made to the service
- the data analyst is embedded within the team and is fully involved in the service development process and works closely with key members of the team to effect iterative changes to the service
- the team has developed a data portal that enables stakeholders to self-serve the performance data for the service
- the team ensures that the data collected is held securely and without the unnecessary collection of PII
What the team needs to explore
While in live, the team needs to:
- investigate indicators of user difficulties in using the service, i.e. validation errors and multiple answers to the same question by the same user
16. Identify performance indicators
Decision
The service met point 16 of the Standard.
What the team has done well
The panel was impressed that:
- the team has clearly thought through the key user need, policy and business objectives for the service and have developed KPIs and corresponding metrics to measure performance against them
- the KPIs identified and measured go well beyond the four mandatory KPIs
- the team has a clear understanding of the success measures and KPIs for the service
- KPIs are clearly documented and the team is taking steps to gather additional performance data to fill any gaps in KPI measurement
17. Report performance data on the Performance Platform
Decision
The service met point 17 of the Standard.
What the team has done well
The panel was impressed that:
- the team has a page developed and completed on the Performance Platform for the service’s performance data and is waiting for GDS to provide access to upload data to it
What the team needs to explore
While in Live, the team needs to:
- upload performance data to the platform
18. Test with the minister
Decision
The service met point 18 of the Standard.
What the team has done well
The panel was impressed that:
- although the team has not tested the service with a minister, it has a plan to provide an update to the CEO of NHSX
What the team needs to explore
Before going live, the team needs to:
- seek approval from a relevant minister
Next Steps
This service can move into a live phase, following the recommendations outlined in the report.
The team should repeat the development phases (discovery, alpha, beta and live) for smaller pieces of work as the service continues running, following spend control processes as necessary.
Published 31 October 2019