111 online (Get urgent help with 111 online) beta reassessment
111 Online is a patient-facing digital service that helps users access the right care for their health needs.
Service Standard assessment report
111 online (Get urgent help with 111 online)
| From: | Central Digital and Data Office |
| Assessment date: | 30 January 2018 |
| Stage: | Beta |
| Result: | Met |
| Service provider: | Department of Health |
The service met the Standard because:
- the service team has a strong understanding of their users and their needs, and an ongoing user research plan to ensure the digital service meets those needs
- the team is working well together, is clearly skilled and competent, and has commendable working practices
- the team has taken on board and actioned both the mandatory and non-mandatory items raised in the last service assessment
The assessment panel has concluded 111 Online has shown sufficient progress and evidence of meeting the Digital Service Standard since the last private beta assessment and the service should now proceed to a public beta.
About the service
Description
111 Online is a patient-facing digital service that helps users access the right care for their health needs. 111 Online allows users with medical concerns to complete a health assessment online by answering a series of questions about their condition. They are then triaged based on their symptoms using a clinically-approved and safe set of algorithms, and then connected to the local services most appropriate for their effective treatment.
Service users
The users of this service are:
- members of the public seeking healthcare for themselves or others
- healthcare professionals (clinicians) in urgent care services
Although not direct users, critical stakeholders also include commissioners and providers of NHS 111 services.
Detail
After considering all the information presented at the private beta reassessment, the panel is very pleased to conclude that 111 Online is ready to move into public beta. The 111 Online team presented a clear and coherent vision of the service. The panel was particularly pleased to see the way the whole team worked together to answer questions, talk about user research, and discuss things they had learned during the private beta.
It is clear a lot of work has gone into the private beta, including addressing the recommendations from the last assessment. It is a significant jump from private to public beta and the team themselves acknowledged there is still a lot they need to do. The service owner is aware of the challenges facing the service going into public beta and has proactively put in place a plan for moving into public beta and a potential national rollout.
User needs
Overall the team showed a strong commitment to understanding their users and their needs, and to working together to build a service that works well for its users.
Since the last beta assessment the team has broadened their user research, including more contextual research at dental practices, minor injuries units, local service centres and libraries. This has given the team a good understanding of the range of potential users of the service, including those with disabilities and those who might not be able to use a digital service, or may have other reasons not too.
The team has documented what they have learned in a set of good empathy maps that describes different types of potential users in terms of their different circumstances and behaviours. The team has also created clear user journey maps layered with user needs and with descriptions of what users are thinking, feeling and doing at different points in their journey.
The team has done some research with people with disabilities. During public beta the team must do more contextual research with people with disabilities to understand problems they may have answering questions or acting on their disposition. For example, can people with a visual impairment answer questions that ask about the appearance of a symptom, or can people with a cognitive impairment answer questions that require some judgement. Another example is, can people with communication or mobility impairments follow the steps given in their disposition. In the context of 111 Online, a “disposition” is the assessment of what care you need and the steps you should take based on the information you provide to the 111 Online service.
The NHS Pathways behind the service includes too many questions to test their comprehension individually, and too many possible routes to test their comprehension in all contexts. During public beta the team needs to have a clearer strategy for testing the comprehension of questions within those constraints.
During public beta the team should also focus more research on users’ reactions to their disposition. Their acceptance and trust of it, their satisfaction with it, and their motivation and likelihood to act on it. The team should also explore how notifications about their disposition from this service and from follow-up services affect users’ behaviour.
Team
The team is extremely passionate about the project and from the assessment it is clear that the team work incredibly well together.
The team is split into two product teams and the relationship between the product teams including the division of work is a good one. The gaps identified in the team structure since the last beta assessment have been addressed. However, there is currently one delivery manager in post who is spread across both product teams and other responsibilities. The panel recommends a second delivery manager must be recruited to the service to support the move to public beta.
The continuing presence of a Clinician as part of the delivery team is an asset to the service. The integration of a Clinician as part of the delivery team should continue throughout the lifetime of the service. The content designers also have regular contact with clinicians working on the NHS 111 telephone service and have signed up to a short course on Pathways (the algorithm behind the triage tool). The content teams’ iterative practices incorporating both user and clinical feedback are commendable.
The team is not co-located but are working very well from different locations using tools like Slack and WebEx. The team has regular virtual meetings as well as regular face-to-face meetings to go through user research and to practice agile ceremonies.
The panel notes that the team shape is soon to change, as the service moves into public beta and scales. The service owner is in the process of putting in place the resources needed for the team to proceed through the public beta phase and a potential national rollout. The panel recommends the service owner must review the implementation team resourcing to ensure it is at adequate and sustainable levels for scaling the service.
Technology
The team has continued to refine the service since the last assessment using the same technology stack that is widely used within NHS Digital. They should be commended on their efficient and secure use of public cloud infrastructure for a service with privacy implications.
Changes to the infrastructure of the service are expected to provide a high level of resilience. Significant performance, stress and load testing have shown that the service is up to the availability challenge of being in public beta. As various regions are added to the service, the team expect to continuously measure the impact and perform further capacity planning to ensure smooth running.
A lot of time has been spent considering the security and privacy implications of the service, and the team has done a good job in ensuring that they err on the side of security. Consideration has been given to the potential for denial of service attacks, not just to the service but indirectly to the IT of the suppliers themselves, and the team has a good approach to ensuring that this doesn’t become an issue.
The team has iterated on their use of Elasticsearch ensuring it handles typographical errors, word stemming and uses appropriate term weighting. They intend to continue to iterate on changes to the search to ensure the optimal weighting which should make the search very efficient within this limited domain. The approach to categorisation of search terms is pragmatic and the team is encouraged to continue to iterate to ensure that the taxonomy is complete, based on the experience gained during public beta.
Whilst the deployment process is automated, it isn’t performed continuously to production as there are still some remaining tests that are carried out manually. This is something that the team hopes to improve as they progress, although access to the source code for dependent services so that they could contribute would help in this regard.
The requested changes from the last assessment have largely been met, but there are some recommendations that the panel feels would improve the service yet further as it heads towards becoming live. These should by no means stop the service from entering public beta but are recommended before the next assessment. The team has agreed to investigate increasing the frequency of the source code release and to ensuring that any microservices are versioned, and independently deployable as the opportunity arises.
Design
The team demonstrated good collaborative working processes identifying user needs, sketching and prototyping solutions, iterating based on results of usability testing. The team also demonstrated they were are sharing and feeding back what they are learning with the rest of the organisation and are involved in regular design huddles.
The team has iterated the appointment booking and maps pages to make the interface consistent with NHS UK patterns - they identified issues with some patterns using voice assisted technology in the GDS empathy lab and managed to fix the issue quickly. In another example, the team iterated the 999 checklist to make it clearer to users when they need to contact 999; however, the team did not show their user research to demonstrate this.
It was noted that where users were being asked to give some particularly personal answers the questions were more verbose, for example, “To make sure you get the support you need…” the team explained that the content was iterated and tested and users found the experience less obtrusive and jarring.
With GDPR coming into force the panel recommends the team review whether they need to be explicit with users that their answers will be shared with clinicians if a booking is made through the service rather than rely on the user reading a privacy policy which they probably would not do. The panel also recommends the team review how they expose the cookies statement to users of the service.
There are various possible entry points to the service and many Clinical Commissioning Groups (CCGs) have commissioned their own online versions of NHS 111 which could be confusing for users. The team explained that they will be working with the wider organisation and CCGs on these entry points as part of their digital uptake strategy.
The panel acknowledges the need to align the online service with the phone service and that many users are aware of the name NHS 111 through CCG campaigns. However, the panel recommends the team consider how you might name the service in a way that focuses directly on the problem that the service solves for the user, something along the lines of ‘Find out immediately what urgent medical care you need.’ This will help with other entry points to the service, for example Google. Other services with well known names have included both, for example, the ‘Register your vehicle as off the road (SORN)’ service on GOV.UK.
There is a strategy in place to review the content for the 1700 questions and the team is making good progress on this working closely with the clinicians and call handlers. The team tested 250 questions with users and are using what they’ve learned there, analytics data and online feedback from users to inform their content review.
Analytics
The team presented a strong articulation of how the service’s KPIs have been developed from the business proposition and business needs. The team identified four ‘buckets’ of data:
- GDS KPIs
- strategic information for readiness CCGs
- service data
- evaluation data journeys and impact
From here the team has identified ‘macro’ and ‘micro’ KPIs.
Macro, for example:
- completion rate/ satisfaction
- cost per transaction
- reduce pressure on phones - uptake of online service
- abandonment rates
- roll-out across England
The wider outcomes for the NHS are complex and the Service will be tendering for an external academic evaluation. For example, there is the challenge of capturing where users turn up next in the NHS after they’ve used 111 Online.
Micro, for example:
- page dwell times
- operational data - telephone, feed data to CCGs
- search queries
The team explained that, subject to big clinical governance issues, there are big opportunities to do predictive analytics, but the number of online service users is too small at present.
The team has two part-time data scientists (one recently recruited) who come from a statistical background. They mainly use Power BI. Currently, the teams’ developers own the Google Analytics and Google Tag Manager implementation and governance. The data scientists are starting to be involved in stories, are building useful team dashboards, and commencing dives on product pages. The analysts and developers are pairing to share work, expertise and analysis stories driven by both user research and quantitative data.
The team expects a Performance Platform dashboard to go live imminently.
Recommendations
To pass the next assessment, the service team must:
- do more contextual research with people with disabilities to learn about problems they may have understanding and answering questions, and understanding and acting on their disposition
- ensure all team members are in place for the team to sustain the public beta phase and scaling of the service with a particular emphasis on the implementation team
- recruit a second delivery manager to help the service move through public beta and the scaling of the service
- with GDPR coming into force, the team must review whether they need to be explicit with users that their answers will be shared with clinicians if a booking is made through the service
- the team has built up its analysis capability with two part-time data scientists, but must ensure the analysts in the team are fully involved in the agile team, helping prioritise stories, and identifying success criteria
- do more analysis of how people search on the web for user needs that 111 Online can meet - what opportunities are there to do Search Engine Optimisation to help users find the service. This will be increasingly important as the service is rolled out
- test the service again with the minister responsible for it
The service team should also:
- approach the wider organisation to evaluate the multiplicity of online versions of NHS 111 that have been commissioned by some CCGs. Particular focus should be on the effect the multiplicity of offerings has on potential users of the online service including an investigation into whether or not a single universal online offering would best meet user needs
- consider how the service might be named in a way that that focuses directly on the problem that the service solves for the user, something along the lines of ‘Find out immediately what urgent medical care you need.’
- have a clearer strategy for testing the comprehension of the triage questions in context
- do more research on users’ reaction to their disposition
- explore how notifications about their disposition and from follow-up care services affect users’ behaviour
- explore further how users can skip a question or express they do not know the answer to a question or that they do not understand a question
- revisit and improve the user experience for when the service is offline. Currently, when the service is offline, the user is sent to the NHS 111 telephone service that has an automated message (if busy) to send the user back to the online service
- maintain the integration of a Clinician as part of the delivery team. The hands on involvement of a Clinician in the project is an asset to the service
- approach the organisation to make the urgent care Directory of Services (DOS) service code open source, so that the team can contribute to a key component
- approach the wider organisation to release the Pathways data
- provide some documentation to enable other teams within the organisation to reuse microservices and other components of 111 Online
- review how the cookies statement is exposed to users of the service
Next Steps
You should follow the recommendations made in this report before arranging your next assessment.
This service now has permission to launch on a GOV.UK service domain. These instructions explain how to set up your *.service.gov.uk domain.
Submit feedback
Submit feedback about your assessment.
Get advice and guidance
The team can get advice and guidance on the next stage of development by:
- searching the Government Service Design Manual
- asking the cross-government slack community
- contacting the Service Assessment team