Policy paper

Coronavirus press conferences quality and methodology information report

Published 26 June 2020

Quality and methodology information for data slides presented at the coronavirus press conferences.

1. About this Quality and Methodology Information (QMI) Report

This quality and methodology information report contains information on the quality characteristics of the data (including the European Statistical System’s five dimensions of quality) as well as the methods used to create it.

The information in this report will help you to:

• understand the strengths and limitations of the data
• learn about existing uses and users of the data
• understand the methods used to create the data
• help you to decide suitable uses for the data
• reduce the risk of misusing data

2. Important points

• This report presents information for the following slides which have regularly been included in the press conferences at 10 Downing Street in response to coronavirus: cases, tests, data from hospitals (admissions, ventilators, people in hospital), deaths, transport indicators and social impacts.
• The data for the slides are compiled from sources originally published elsewhere. These are brought together to provide a comprehensive picture of the UK COVID-19 outbreak.
• This report outlines the quality characteristics of the data presented at the coronavirus press conferences. Some trade-offs have been made between the different quality dimensions since these data were used to update the public about the spread of the virus in the UK in a timely manner and during an unprecedented situation. In addition many data sources have been developing over time.

3. Overview of the coronavirus press conference data production

Slides and data on the latest COVID-19 statistics are published for the coronavirus press conferences when these occur. Data are sourced from the Civil Contingencies Secretariat (CCS) and a series of published official statistics. The data underpinning the charts presented in the press conference briefings is published alongside the slides on gov.uk. Data are presented on a UK basis, where appropriate, by aggregating regional statistics.

3.1 Production Process

• Key data on the COVID-19 pandemic is submitted to the Civil Contingencies Secretariat (CCS) to accurately monitor the pandemic across the UK.
• Data for the press conferences are sourced from CCS (via data submitted by NHS England and the devolved administrations) or from other published official statistics.
• The briefing slides and accompanying data pack are compiled by analysts across statistics, economist and social research professions.
• Data is independently produced and quality assured within the team and sent back to data owners for final clearance.
• Full information and links to data sources are included in the data pack, on the slides and statistical annex in the slide pack. Caveats and footnotes detail revisions and limitations of the data.
• Any changes to presentation and use of data are cleared with the appropriate data owners and Heads of Professions in other government departments and devolved administrations.
• The final conference slides are signed-off by a Deputy Director analyst before broadcast.

3.2 Orderly release

• Conference slides, statistical annexes and data packs are published on a dedicated gov.uk page each day there is a press conference. These are usually published during the press conference so between 5pm and 6pm each day there is a conference.
• On days where a press conference isn’t held, published data can be sourced from departmental sources that are linked in the slides and data pack.

4. Quality characteristics of the press conference data

This part of the report is grouped into different sections:

4.1 Cases

Confirmed cases (coverage UK, published by Department of Health and Social Care)

• Relevance: Confirmed cases data published by the Department of Health and Social Care (DHSC) incorporates data submitted from across the whole of the UK. Confirmed cases data has been used to monitor progress against satisfying the UK Government’s five tests for adjusting lockdown, in particular cases data has been used for test 3 “rate of infection is decreasing to manageable levels”.
• Accuracy: COVID-19 cases are identified by taking specimens from people and sending these specimens to laboratories around the UK to be tested. If the test is positive, this is referred to as a lab-confirmed case. The data published by DHSC on confirmed cases only report on cases tested positive so this is likely to undercount the total number of cases across the UK. Cases in people who have not been tested are not included in the confirmed case counts. There may be a small percentage of cases where the same person has had more than one positive test result for COVID-19.
• Coherence and comparability: There are separate reporting processes for each of the 4 Nations of the UK. England, Northern Ireland and Scotland each provide data based on tests carried out in NHS (and PHE) laboratories. These represent ‘pillar 1’ of the UK Government’s mass testing programme. Wales provides data based on tests carried out in NHS laboratories (‘pillar 1’) and testing on critical workers processed in NHS laboratories (‘pillar 2’ of the mass-testing programme). DHSC combines the counts from the 4 Nations, and adds data from tests carried out by commercial partners (‘pillar 2’) in England, Northern Ireland and Scotland to give daily and total (cumulative) counts of lab-confirmed cases. These are submitted to Public Health England (PHE) to display on the dashboard. The 4 figures are not all taken from the same cut-off time: England and Northern Ireland counts are as at 9am on the day of publication, Scotland counts are as at 8am on the day of publication; Wales counts are as at 1pm on the day before publication. Despite the slightly different definitions data are aggregated to provide a headline UK case count, however users should be mindful of the different definitions and where possible look at the national level data as well to understand differences in trends.
• Reliability: The data published on the DHSC website are constantly being reviewed and corrected. Cumulative counts can occasionally go down from one day to the next, and on some occasions there have been major revisions that have a significant effect on local, regional, National or UK totals. Data are provided daily from several different electronic data collection systems and these can experience technical issues which can affect daily figures, usually resulting in lower daily counts.
• Timeliness: A time series of confirmed cases data is updated each day. The headline UK case count is published by DHSC each day via Twitter and on the DHSC website.

Coronavirus Infection Survey pilot (coverage England, published by Office for National Statistics)

• This survey is produced by the Office for National Statistics (ONS) in collaboration with research partners at the University of Oxford, the University of Manchester, Public Health England and Wellcome Trust. It is being delivered in partnership with IQVIA, the University of Oxford and UK Biocentre. These statistics have been produced quickly in response to developing world events.
• The pilot phase of the study only includes private households in England. People in care homes, hospitals, and other communal establishments are not included. Work is underway to consider further expansion of the survey across the UK.
• The Office for Statistics Regulation (OSR), on behalf of the UK Statistics Authority, has reviewed them against several important aspects of the Code of Practice for Statistics and regards them as consistent with the Code’s pillars of trustworthiness, quality and value.
• Accuracy: The estimates presented in this bulletin contain uncertainty. Uncertainty relates to how the estimate might differ from the “true value” and these measures help users to understand the degree of confidence in the outputs. There are many sources of uncertainty, but the main sources in the information presented include each of the following:
  • Uncertainty in the test: These results are directly from the test, and no test is perfect. There will be false-positives and false-negatives from the test, and false-negatives could also come from the fact that participants in this study are self-swabbing. More information about the potential impact of false-positives and false-negatives is provided in the ONS publication, Section 8: Measuring the data.
  • The data are based on a sample of people, so there is some uncertainty in the estimates: Any estimate based on a random sample contains some uncertainty. Confidence intervals are used to provide an indication of the degree of uncertainty. If we were to repeat the whole process many times, we would expect the true value to lie within the upper and lower confidence intervals 95% of the time. A wider confidence interval indicates more uncertainty in the estimate.
  • Quality of data collected in the questionnaire: As in any survey, some data can be incorrect or missing. For example, participants and interviewers sometimes misinterpret questions or skip them by accident. To minimise the impact of this, ONS clean the data, editing or removing things that are clearly incorrect. In these initial data, ONS identified some specific quality issues with the healthcare and social care worker question responses and have therefore applied some data editing (cleaning) to improve the quality.

Further information can be found in the methods and further information publication on the ONS website.

• Comparability: the figures produced from the coronavirus infection survey are not directly comparable to the confirmed cases published by DHSC.
  • Data published by DHSC on confirmed cases figures relate to those cases which are the result of a positive COVID-19 test confirmed by sending specimens to laboratories. The infection survey tests people from a random section of households which means it can produce statistics about the total number of people infected, whether or not they have symptoms.
  • The infection survey is a point-in-time estimate of the number of people who are infected with COVID-19, whereas the data published by DHSC are a cumulative count of all confirmed cases of COVID-19 since testing began, which will include people who have since recovered.
  • Figures from the infection survey are currently for England only, whereas data published by DHSC are for the UK.
• Timeliness: Users are provided with timely estimates for the percentage of the target population in England testing positive for COVID-19. For example, in the publication released on 12 June 2020, the results presented the number of people in England who had coronavirus based on the results of swab tests performed between 25 May and 7 June.

Reproduction number (coverage UK, published by Government Office for Science)

• The Government Office for Science publish the latest reproduction number (R) for the UK. This includes the latest R number range for the UK, a definition of what R is, limitations of R, how R is estimated and who estimates R.
• Accuracy: R is an average value that can vary in different parts of the country, communities, and subsections of the population. It cannot be measured directly so there is always some uncertainty around its exact value.

4.2 Testing

Testing (coverage UK, published by DHSC)

• DHSC publish daily statistics on coronavirus testing, this incorporates data provided from across the UK. These statistics cover testing for coronavirus, including tests carried out and posted out, people tested and positive cases. Capacity to process tests is published weekly.
• DHSC have published a methodology note which sets out information on the data sources and methodology used to generate testing and testing capacity figures.
• Relevance: Testing data published by the Department of Health and Social Care covers the whole of the UK. Data is published by testing pillar.
• Accuracy: The number of tests is not the same as the number of people tested. For clinical reasons, some people are tested more than once. Therefore, the number of tests will be higher than the number of people tested. Coronavirus tests are processed in several separate labs. Projected lab capacity is an estimate of each lab’s constrained capacity each day based on the staff, chemical reagents and other resources it has available. These estimates are made locally by the labs themselves.
• Comparability: Although some data is being reported daily, direct comparisons of change should not be made between weekdays and weekends/bank holidays. Some testing streams do not provide any tests on weekends or bank holidays, and others may have constrained capacity at these times.
• Timeliness: A time series of daily testing data is updated each day. The early parts of these series include some historical data revisions and series backdating as administrative systems were being developed at the time. This means that individual figures will not necessarily align with what was published at the time in the daily tweet posted by DHSC (view an example tweet) or on GOV.UK. Different pillars started testing at different stages, and therefore a full time series of testing data is not available across all pillars and nations. The testing time series include data from 21 March 2020 onwards. The testing capacity data is updated weekly.

NHS Test and Trace (coverage England, published by DHSC)

• DHSC publish a weekly NHS Test and Trace bulletin. This is an experimental statistics release first published on 11 June 2020. DHSC have published a statement outlining the steps they have taken towards voluntary compliance with the Code of Practice for Statistics. DHSC have published a methodology note to accompany the statistical release.
• Accuracy: The figures are based on management information from the NHS Test and Trace Service, covering England only. There are separate test and trace services in other parts of the UK. Given the importance of this service and the commitment of the NHS Test and Trace Service to be open and transparent with the public it serves, these data are being released at the earliest possible opportunity. However, new IT systems and statistical outputs often take a period of time to bed in. These data should therefore be treated with caution as the system and understanding of the data develops.
• Coherence and comparability: The data presented is derived from the NHS Test and Trace Service (England only) in response to the coronavirus pandemic. Tracing data cannot be compared against other sources as part of the quality assurance process; testing data, when included, will be checked for consistency against published daily testing figures.
• Timeliness: Given the importance of this service and the commitment of NHS Test and Trace to be open and transparent with the public it serves, these data are being published at the earliest possible opportunity. These figures will be initially published weekly, and we will continuously evaluate the frequency based on need and public interest. The timeliness of the data allows for regular monitoring of the public’s response to the contact tracing programme.

4.3 Data from hospitals

Hospital admissions (coverage UK)

• Relevance: Covers the whole of the UK, data is sourced from NHS England, Welsh Government, Scottish Government and Northern Ireland Executive (figures for devolved administrations are in turn provided by the relevant NHS bodies). Accompanying data is published for England and the devolved administrations. Admissions data has been used to monitor progress against satisfying the UK Government’s five tests for adjusting lockdown, in particular admissions data has been used for test 1 “Protect the NHS’s ability to cope”. In Scotland admissions data has been used in the assessment of the evidence on the criteria for progressing to Phase 1 of Scotland’s route map through and out of the crisis.
• Coherence and comparability: There are different approaches to data collection across the UK, which will impact the comparability of both levels and trends given the inclusion, or otherwise, of different classification of COVID-19 patients. England data captures people admitted to hospital who already had a confirmed COVID-19 status at point of admission, and adds those who tested positive in the previous 24 hours whilst in hospital. Inpatients diagnosed with COVID-19 after admission are assumed to have been admitted on the day prior to their diagnosis. England data covers NHS Trusts and the Independent Sector. Wales data includes confirmed and suspected cases, and is the number of admissions to the hospital in the previous 24 hour period up to 9am. The status of COVID/non-COVID is as at the time of reporting not at time of admission. Wales data excludes Private hospitals, field hospitals were included from 20 April 2020 and community hospitals included from 23 April 2020. Northern Ireland data includes suspected and confirmed COVID-19 admissions by admission date. Northern Ireland data covers Health and Social Care Trusts. Data for Scotland provides the profile of admissions into hospital for patients who tested positive for COVID-19 in the 14 days prior to admission to hospital, on the day of their admission or during their stay in hospital. Scotland data covers admissions to all Acute sites. The chart on admissions published in the briefings covers the UK however the comparison text excludes Scotland due to Scotland admissions data being updated weekly rather than daily. Despite the slightly different definitions data are aggregated to provide a headline UK admissions count, however users should be mindful of the different definitions and where possible look at the national level data as well to understand differences in trends.
• Timeliness: Data on NHS activity and capacity during the COVID-19 pandemic is published weekly by Welsh Government. Scottish data are published weekly by Public Health Scotland. Northern Ireland Statistics and Research Agency (NISRA) statisticians within the Department of Health publish COVID-19 statistics via a daily dashboard. Data are also published on coronavirus.data.

COVID-19 patients occupying mechanical ventilator beds (coverage UK)

• Relevance: Covers the whole of the UK, data is sourced from NHS England, Welsh Government, Scottish Government and Northern Ireland Executive and aggregated to a UK total (figures for devolved administrations are in turn provided by the relevant NHS bodies). Accompanying data is published for England and the devolved administrations. Data on patients occupying mechanical ventilator beds has been used to monitor progress against satisfying the UK Government’s five tests for adjusting lockdown, in particular patients occupying mechanical ventilator beds data has been used for test 1 “Protect the NHS’s ability to cope”.
• Coherence and comparability: There are different approaches to data collection across the UK. England figures include the number of COVID-19 patients in beds which are capable of delivering mechanical ventilation. England data covers NHS Trusts and the Independent Sector. For Wales the figures cover people with confirmed or suspected COVID-19 in invasive ventilated beds in a critical care setting, plus those outside of a critical care environment. Wales data excludes Private hospitals, field hospitals were included from 20 April 2020 and community hospitals included from 23 April 2020. Scottish figures include people in ICU with confirmed or suspected COVID-19, and may include a small number of patients who are not on mechanical ventilation. Scotland data covers ICU Level 3 patients in Acute sites. Northern Ireland figures include the number of COVID-19 patients in beds which are capable of delivering mechanical ventilation. Northern Ireland data covers Health and Social Care Trusts.
• Timeliness: Data on NHS activity and capacity during the COVID-19 pandemic is published weekly by Welsh Government. Scottish data are published daily by Scottish Government. NISRA statisticians within the Department of Health publish COVID-19 statistics via a daily dashboard. Data are also published on coronavirus.data.

People in Hospital (coverage UK)

• Relevance: Covers the whole of the UK, data is sourced from NHS England, Welsh Government, Scottish Government and Northern Ireland Executive (figures for devolved administrations are in turn provided by the relevant NHS bodies) . Accompanying data is published for England (by region) and the devolved administrations. Data has been included regularly in press briefings to allow users to see the number of people in hospital with coronavirus.
• Coherence and comparability: England data covers Acute NHS Trusts and the Independent Sector. Wales data excludes Private hospitals, field hospitals were included from 20 April 2020 and community hospitals included from 23 April 2020. Scotland data covers Acute beds in all Acute sites. Northern Ireland data covers Health and Social Care Trusts (all A&E types).
• Timeliness: Data on NHS activity and capacity during the COVID-19 pandemic is published weekly by Welsh Government. The Chief Statistician in Wales has written a blog on measuring people in hospital, and some thoughts on data quality. Scottish data are published daily by Scottish Government. NISRA statisticians within the Department of Health publish COVID-19 statistics via a daily dashboard. Data are also published on coronavirus.data.

4.4 Deaths

• The recording of coronavirus (COVID-19) deaths and infections is a complex picture for the UK. The recording of health and death statistics in the UK is devolved, with information being produced on a number of government websites.
• There are two broad ways of counting COVID-19 deaths across the UK: statistics reported through health and care organisations, usually focused on cases where a positive test for COVID-19 has been confirmed, and statistics reported through the process of death registration, where COVID-19 is mentioned on the death certificate.
• ONS have published a guide on where to find statistics on UK deaths by country, this includes definitions of COVID-19 deaths between different sources.

Daily deaths (coverage UK, published by DHSC)

• Relevance: Daily deaths data published by the Department of Health and Social Care incorporates data provided from the whole of the UK. Daily deaths data has been used to monitor progress against satisfying the UK Government’s five tests for adjusting lockdown, in particular deaths data has been used for test 2 “sustained and consistent fall in the daily death rates”.
• Accuracy: Daily deaths data are deaths of people who have had a positive test result confirmed by a Public Health or NHS laboratory. In Northern Ireland the death must have occurred within 28 days of a positive test result to be included. They also include, for England, deaths of people who have had a positive test result confirmed from testing by commercial partners. The data do not include deaths of people who had COVID-19 but had not been tested, people who were tested positive only via a non-NHS or Public Health laboratory, or people who had been tested negative and subsequently caught the virus and died. Deaths of people who have tested positively for COVID-19 could in some cases be due to a different cause. The daily change in deaths column shows the difference between consecutive cumulative counts. This is not necessarily the same as the number of new deaths announced each day as the changes occasionally include corrections to earlier data.
• Coherence and comparability: There are separate reporting processes for England, Scotland, Wales and Northern Ireland. DHSC combines these 4 counts to give the overall UK daily and total (cumulative) counts. The 4 figures are not all taken from the same cut-off time: England and Wales counts are as at 5pm on the day before publication; Scotland counts are as at 9am on the day before publication; Northern Ireland counts are as at 10am on the day before publication. Details of the processes for counting deaths in England and the devolved administrations are available in this guide published by ONS on where to find statistics on UK deaths by country. Daily death figures are by date of reporting, not date of death, and therefore daily death figures tend to be lower at the weekend and higher on weekdays due to reporting lags. As a result, it can be difficult to interpret the trend. A seven day rolling average is used to smooth the time series and illustrate the trend.
• Timeliness: A time series of daily deaths data is updated each day.

Weekly deaths (coverage primarily England and Wales, published by ONS)

• Relevance: Weekly death data published by ONS primarily cover England and Wales, but the publication also includes weekly provisional figures on COVID-19 deaths registered in the UK. Further information can be found in the mortality statistics in England and Wales QMI.
• Coherence and comparability: Deaths reported by ONS are based on the causes of death recorded on death certificates. These can include cases where the doctor thought it likely that the person had COVID-19, even when there was no positive test result. These figures are different from the daily death figures on coronavirus deaths published by DHSC. Deaths reported by ONS are derived from the formal process of death registration and may include cases where the doctor completing the death certificate diagnosed possible cases of COVID-19, for example, where this was based on relevant symptoms but no test for the virus was conducted.
• Timeliness: Weekly death figures are published every week on a Tuesday. In England and Wales, deaths should ideally be registered within 5 days of the death occurring, but there are some situations that result in the registration of the death being delayed.

4.5 Transport indicators

Transport use during the coronavirus pandemic (coverage Great Britain, published by Department for Transport)

• This data was previously included in the press briefings and is now being published on a weekly basis by the Department for Transport (DfT).
• OSR, on behalf of the UK Statistics Authority, has reviewed them against several important aspects of the Code of Practice for Statistics and regards them as consistent with the Code’s pillars of trustworthiness, quality and value.
• DfT have published a methodology note alongside these statistics which provides an overview of the data sources used by mode of transport.
• Coherence and comparability: Although daily data is being reported, direct comparisons of change should not be made between weekdays and weekends/bank holidays. For public transport there are typically different levels of service/timetable in place on weekends and bank holidays than on weekdays; and for road traffic, there is a different profile on weekend days compared to weekdays.
• Timeliness: To monitor the use of the transport system during the coronavirus pandemic, DfT provides statistics on transport use by mode for each day since 1 March 2020. Data are published weekly by DfT.

4.6 Social impacts

Coronavirus and the social impacts on Great Britain (coverage Great Britain, published by ONS)

• This data was previously included in the press briefings and continues to be published on a weekly basis by ONS as part of the Opinions and Lifestyle Survey (OPN).
• More quality and methodology information on strengths, limitations, appropriate uses, and how the data were created is available in the QMI published alongside the OPN.
• Coherence and comparability: The sample size is relatively small: 2,500 individuals per week with fewer completed interviews, meaning that detailed analyses for subnational geographies and other sub-groups are not possible. Comparisons between periods and groups must be done with caution as estimates are provided from a sample survey; as such, confidence intervals are included in the datasets to present the sampling variability, which should be taken into account when assessing differences between periods, as true differences may not exist.
• Timeliness: The OPN allows for timely production of data and statistics that respond quickly to changing needs. In response to the coronavirus pandemic, the OPN was adapted to become a weekly survey used to collect data on the impact of the coronavirus on day-to-day life in Great Britain.

5. Revisions to data

Due to the nature of these statistics, revisions may be made to the data. Revisions are clearly marked in the data pack and noted in the statistical annex to the slides, if required. Further information is provided in footnotes.

6. Other information

Slides, datasets and transcripts from press conferences at 10 Downing Street in response to coronavirus

Coronavirus in the UK

Statistics on UK deaths involving the coronavirus (COVID-19) and infection rates by country

Number of cases and deaths, and breakdown of testing by testing strategy ‘pillars’ (UK)

England, COVID-19 Daily Deaths

Wales, NHS activity and capacity during the coronavirus pandemic

Scottish Government, Coronavirus daily data

National Records Scotland, Deaths involving coronavirus (COVID-19) in Scotland

Public Health Scotland, COVID-19 Statistical Report

Northern Ireland, Coronavirus Statistics