Covid Pass assessment

The live assessment report for the Department of Health and Social Care's Covid Pass service on 21st July 2022.

From:
Central Digital and Data Office
Published
17 October 2022
Assessment date:
21 July 2022
Result:
Not met
Stage:
Live

Service Standard assessment report

Covid Pass

From: Central Digital & Data Office (CDDO)
Assessment date: 21/07/2022
Stage: Live
Result: Not Met
Service provider: Department of Health and Social Care (DHSC)

Related report

Reassessment report

Service description

The NHS Covid Pass facilitates evidencing and checking of Covid-19 status, both in domestic and international settings.

Service users

Summary of user groups:

  • Vaccination status

    • Fully vaccinated
    • Negative test
    • Unvaccinated
    • Recovery
    • Not fully vaccinated

  • Vaccine recorded out-side England

    • DACDOT users
    • Vaccinated Overseas
    • MOD & FCDO

  • Inclusive Needs or Exempt

    • Clinical Trials (blinded and unblinded)
    • Inclusive
    • Medical Exemptions
    • Young People (under 12s)

  • Other Users

    • Asking on behalf of others
    • Verifiers

1. Understand users and their needs

Decision

The service met point 1 of the Standard.

What the team has done well

The panel was impressed that:

  • the team conducted a range of appropriate research at pace and under a number of constraints (including exploratory research, usability testing, and content testing)
  • there has been a consideration of the subtle ways in which branding and language can affect user behaviour and how attitudes towards vaccination might impact on behaviour

What the team needs to explore

The team needs to:

  • continue to explore the experiences of users who might have distinct or more complex needs. For example, those who need to engage with the service but who have less positive attitudes towards vaccination
  • understand any distinct needs of people in secure settings such as prisons (both staff and prisoners)

2. Solve a whole problem for users

Decision

The service met point 2 of the Standard.

What the team has done well

The panel was impressed that the team:

  • successfully brought together a complicated landscape of related services into a simple experience for users
  • designed the whole end-to-end, front-to-back experience of using a covid pass, and created a verifier app and guidance for venues needing to check passes
  • has a very good understanding of the many and ever-changing constraints that affect their work - policy constraints in particular. As things have changed that affected the scope of the service, they have constantly approached this in a user-centered way to make sure the service still meets user needs
  • considered the future scope of the service and how it could change depending on Covid-19 policy. They were able to explain the future direction of the service in line with current policy, their contingency plans for bringing back retired elements of the service if needed, and how elements of the service might be reused in other contexts

What the team needs to explore

The team needs to:

  • continue to work on any unmet needs - for example making passes available online for 5 to 11 year olds
  • continue to review the scope of the service as the policy environment changes, and to take a user-centered approach to any changes to the service - especially after it’s handed over to a live service team

3. Provide a joined-up experience across all channels

Decision

The service met point 3 of the Standard.

What the team has done well

The panel was impressed that:

  • there is a well-developed offline route for users that struggle to use the digital service, or cannot use it at all
  • the team explained how they work with teams providing user support and have processes to make sure guidance stays accurate and up to date
  • the team is in regular contact with related services like Test and Trace so messaging is accurate and consistent
  • use insights from user support to drive updates to the service

What the team needs to explore

The team needs to:

  • continue to work with the support team and related services to provide consistent messaging and identify pain points

4. Make the service simple to use

Decision

The service met point 4 of the Standard.

What the team has done well

The panel was impressed that:

  • made iterations that make the service simpler to use. In particular, they were able to work with the information governance team to remove unnecessary screens from the journey while still meeting requirements
  • worked with related services and medical experts to fix the problems behind the problems - for example, making sure the service can receive high quality data in the long run as well as putting workarounds in place for data quality issues
  • used NHS design patterns and clear language throughout

What the team needs to explore

The team needs to:

  • monitor whether encouraging users to use the digital journey rather than requesting letters is making the journey harder to use for those who do need a letter. Make sure the letter journey is not hidden while the digital version is promoted above it. When measuring the success of this initiative, make sure not to see the numbers of letter requests decreasing as a sign of success - it may actually be that the journey is now too difficult for people to get what they need

  • consider whether more usability improvements can be made as the service scope changes. A couple of examples:

    • Users still have to select to see their international travel pass even though the domestic option is no longer available and there’s nothing else users can do on that page
    • As more vaccination records are added, it’s quite hard to tell the different QR codes apart when swiping between them
    • In the journey where users get a pass on behalf of someone else, it could be clearer who it’s asking about. For example, the headings could be “Enter the person’s name” instead of “Enter your name” and the useful instruction about filling it in for someone else should not be hidden in a details component

5. Make sure everyone can use the service

Decision

The service did not meet point 5 of the Standard.

What the team has done well

The panel was impressed that:

  • there are multiple ways of getting and presenting a covid pass, which accounts for the technology users might have access to and makes the service accessible
  • the team designed for a wide range of access needs from the start - the wide range of formats and languages that the covid pass letter was available in was particularly impressive
  • the team has tested with users with a range of access needs and made improvements in response
  • the service has had internal and external accessibility audits to check for issues, including an end-to-end test of the whole NHS app journey

What the team needs to explore

To meet the standard the team needs to:

  • fix outstanding accessibility issues so the service meets the WCAG 2.1 AA standard and make sure the accessibility statements reflect the current accessibility status while this work is being carried out

It’s also recommended that:

  • the team improves the browser-based journey so it still works without JavaScript - services should be built using progressive enhancement

6. Have a multidisciplinary team

Decision

The service met point 6 of the Standard.

What the team has done well

The panel was impressed that:

  • the team was spun up and then maintained using a blended resourcing model of civil servants, contractors, policy, finance and corporate leads. This means that knowledge is passed onto the organisation. As the team stated, when the operations side is run by people doing development this gives a greater degree of agility
  • the team structure has changed during the delivery as needs and direction have changed with a focus on rebalancing teams to make sure the skills were there to deliver and increment on the product end to end
  • the team is supported by skills from other teams in the organisation and in partner organisations as required

What the team needs to explore

The team needs to:

  • the team would still like to make some minor improvements to the service, and it’s important the team remain in place until the backlog is clear and any remaining recommendations (for example technology, accessibility, performance analytics) are addressed before the planned handover to the Live Services team or Business Owner
  • ensure the transition plan considers and includes a costed model for what continuous improvement looks like for the service and who will be needed in the team to get the balance in case policy changes quickly

7. Use agile ways of working

Decision

The service met point 7 of the Standard.

What the team has done well

The panel was impressed that:

  • the team has adhered to agile best practices (Scrum and Kanban), delivering at pace and improving ways of working through regular retrospectives
  • this is backed up with the use of collaboration tools such as Jira, Confluence, Github and Slack for continuous collaboration
  • the transition and progression of work through the team (from user research to deployment) was very smooth, which has been achieved through continuous backlog refinement and collaboration to get a shared understanding of policy, user needs, design, analytics and technical implementation
  • overall the panel was impressed with the team’s way of working, which has been evidenced by how they pivoted when they realised the structure was not allowing for rapid delivery and iteration of the service

What the team needs to explore

The team needs to:

  • focus on the internal process and document internal decisions and the process for knowledge handover before offboarding valued team members
  • some suggestions to consider further:
    • Encourage experts to document and share their knowledge on a daily basis – not just when they are about to leave
    • Develop learning content prepared in short that can be quickly grasped and easily digested
    • Consider the knowledge base platform and make it available to teams
  • continue to prepare for the future with transition planning for the future business owner and planned releases to clear the backlog between July 2022 and March 2023. The team should continue to be ambitious and challenge themselves in the roadmap, including providing support to others who face similar problems and challenges in the service community

8. Iterate and improve frequently

Decision

The service met point 8 of the Standard.

What the team has done well

The panel was impressed that:

  • the team spun up a service that became mandatory very quickly, they have had to work to aggressive timescales and under extreme pressure where there were no examples to lean on
  • the team focussed on actively releasing every week, driving service forward with insights from live use and produced 38 releases over last year. The team was continuously addressing constraints faced by users in using the service, gathering some feedback from telephone complaints and an online survey
  • the team gathered for weekly meetings to look at trends, prioritise improvements to the product, reviewing against the context and prioritised awareness campaigns around changes as they learned more

What the team needs to explore

The team needs to:

  • show how iterative releases to the service have shown a measurable impact on the user journey and addressed user needs
  • have a plan to continue to measure the impact of iterations on the user journey, and measure if user needs are being met going forwards

9. Create a secure service which protects users’ privacy

Decision

The service met point 9 of the Standard.

What the team has done well

The panel was impressed that:

  • the team has well established cookie and privacy policy and also published them
  • the data is held only for the time it is required and data is not stored
  • only authorised people can access the data which is again with a well established and audited process
  • team has done the DPIA, penetration tests and risk assessments and appropriate actions are planned

What the team needs to explore

Before the next assessment, the team should be able to:

  • address and resolve the risks highlighted in the penetration test and risk assessment

10. Define what success looks like and publish performance data

Decision

The service did not meet point 10 of the Standard.

What the team has done well

The panel was impressed that the service team:

  • has good MI that is publishable outside of the team
  • has a plan for how to monitor the service for any issues when it’s in live
  • has a good variety of offline data including telephony and letters
  • has considered and used cost per transaction to show how iterations have reduced the cost of the service

What the team needs to explore

Before the next assessment, the team needs to:

  • develop success measures around the defined user needs to show success in using the service
  • implement user behaviour tracking, if this meets the definitions of success
  • understand the “so what” of the data collected and show how it’s been used in decision making, not just reporting

11. Choose the right tools and technology

Decision

The service met point 11 of the Standard.

What the team has done well

The panel was impressed that:

  • the service is hosted in Azure cloud and team is using Jira/ Confluence/ Google Suite
  • tools like Azure Devops, .Net Core, Splunk, Terraform, Cosmos, PostgreSql are used
  • the team uses appropriate tools for the front end. The service uses JavaScript. Users can request a paper copy. Please see the related recommendation at point 5

12. Make new source code open

Decision

The service did not meet point 12 of the Standard.

What the team has done well

The panel was impressed that:

What the team needs to explore

Before the next assessment, the team needs to:

  • ensure that coding in the open is normalised for the service unless there is an explicit reason to exclude components from the public domain, in line with guidance at https://www.gov.uk/service-manual/technology/making-source-code-open-and-reusable

13. Use and contribute to open standards, common components and patterns

Decision

The service met point 13 of the Standard.

What the team has done well

The panel was impressed that:

  • the team has used the NHS design system
  • the team has used GOV.UK Notify
  • the service has good integration with NHS App and NHS Login
  • the team is using 2 APIs - 2D Barcode Generator API (An API which creates EU Digital COVID Certificates) and COVID Isolation Exemption Service API (An API to present a citizens COVID status)

What the team needs to explore

The team needs to:

  • make sure they’ve documented and shared what they’ve learnt with the NHS service manual team where they’ve needed to introduce new design patterns - particularly the 2D barcode and information governance screens

14. Operate a reliable service

Decision

The service met point 14 of the Standard.

What the team has done well

The panel was impressed that:

  • the team has well established SLAs, plan and processes for the operation and running of the service
  • the infrastructure has been designed and analysed to deliver the required capacity and capabilities and seems to be scalable for future requirements
  • the technology and architecture looks good to support the operation and running of the services
Published 17 October 2022
Contents