Guidance

Universal Credit statistics: background quality report

Published 10 November 2020

Applies to England, Scotland and Wales

The latest release of the Universal Credit statistics can be found in the Universal Credit statistics collection.

1. Introduction

This report assesses the quality of Universal Credit official statistics using the European Statistics System (ESS) Quality Assurance Framework (QAF). This is the method recommended by the Government Statistical Service (GSS) Quality Strategy. Statistics are of good quality when they are fit for their intended use.

The ESS QAF measures the quality of statistical outputs against the dimensions of

• relevance
• accuracy and reliability
• timeliness
• accessibility and clarity
• comparability and coherence

The GSS also recommends assessment against 3 other principles in the ESS QAF. These are

• trade-offs between output quality components
• confidentiality and transparency
• balance between performance, cost and respondent burden

These dimensions and principles cross the three pillars of trustworthiness, quality and value in the Code of Practice for Statistics.

The Background Information and Methodology document provides more information on Universal Credit and the methodology used to produce these statistics.

2. Relevance

Definition: Relevance is how the statistics meet the needs of current and potential users for both coverage and content. Innovation is pursued to continuously improve statistical output.

The purpose of Universal Credit statistics is to be the primary source of official information on the people and households on Universal Credit in Great Britain. The series has been developed to provide information to various users for policy development, monitoring and accountability as well as providing academics, journalists and the general public, data to aid informed public debate. More information on uses and users is provided in our background information and methodology document.

To understand user needs, regular engagement is made with users to seek their views. These include:

• consultation on release strategy in 2017
• consultation on Department for Work and Pensions (DWP) statistics in 2019
• user engagement survey of Universal Credit statistics users in 2020

The consultation on the release strategy provided information to help identify and prioritise the developments to be made to Universal Credit statistics over the next few years. Users will be consulted each time the release strategy is updated.

The 2019 consultation identified that timely, monthly, statistics was important to users. This informed a decision not to take forward a proposal to change the frequency of publication of monthly Universal Credit statistics to quarterly.

The 2020 user engagement survey identified the types of users and how they use Universal Credit statistics. The information gathered will inform decisions on how we disseminate our statistics, and what further information users want.

In addition to these structured forms of gathering information from users, every release contains a request to provide feedback on the statistics through a dedicated team email address. This email address also provides a single point of contact for users’ to enquire about the statistics.

Other platforms away from GOV.UK are also used to engage with users. These include:

• the publication being tweeted by the Head of Profession for Statistics on Twitter
• promoting each release and development on the Benefits and Welfare board on StatsUserNet
• issuing an email on each release of the statistics to users who have opted in to receiving the information

Analysis of usage of Universal Credit statistics through Google Analytics is being developed to inform future development of statistics.

User requests have led to:

• developing of new measures such as ethnicity and religion, including on-going work to improve the declaration rate
• the inclusion of an extra pre-defined table in the People in UC domain in Stat-Xplore

3. Accuracy and reliability

Definition: Accuracy is the proximity between an estimate and the unknown true value. Reliability is the closeness of early estimates to subsequent estimated values.

This statistical series is produced from administrative data using data from systems within local offices and records of Universal Credit benefit payments made by the department. There are processes in place at various points in the data journey from collecting and administering the data to the procedures to turn the administrative data to official statistics, including using evidence to verify information declared by claimants and quality assurance processes to monitor data in transforming from administrative data to statistics.

The possible errors that could occur in Universal Credit statistics include:

• data entry error – data is entered by claimants and Agents. Some data may be entered in a form that is incomplete or in a format that cannot be processed. This is minimised by using menu options or calendar entry for dates in place of free text
• fraud – intentional incorrect information may be entered by some claimants to receive benefit they are not entitled to. Although all information is validated to minimise this, inevitably some will appear in the data. Statistics are available on the estimated fraud and administrative error in the benefits system
• processing error – an error during processing caused by data or programmer error may cause an element of the processing to fail or produce inaccurate statistics. Checks are in place to pick up on these where possible
• non-response error – some information is optional. There may be particular cohorts that respond and others that do not respond that may lead to some measures being unreliable. Statistics will not be published until a suitable level of response is available to minimise the effects non-response error

Data are produced on a provisional basis and revised in the following release. Provisional data is only released where it is reliable and the revised figure will be close to the provisional figure. It is expected that the overall provisional figure for People on Universal Credit will be within 2% of the final revised figure.

Where there is evidence that the provisional figure is not a close measure of the final figure, provisional data will not be released. This is the case with the employment indicator for people on Universal Credit. Similarly, household data has a longer lag than the other measures in the series to avoid large revisions from provisional data.

4. Timeliness

Definition: This dimension measures that the statistics are released in a timely and punctual manner. Timeliness refers to the lapse of time between publication and the period to which the data refer. Punctuality refers to the time lag between the actual and planned dates of publication.

In deciding methodology and a publication schedule there is a trade-off between timeliness, quality and resources. Most users of Universal Credit official statistics require timely data to provide an early indication of the labour market for economic statistics, or for planning support in local authorities. For these reasons timeliness takes precedence over quality in the first release to provide an early indicator. Figures are revised when more data is available, with final figures expected to be within 2% of the overall total figure.

The statistics are released at the earliest possible point after the date for which the count of people on Universal Credit is taken. The process to produce Universal Credit official statistics is:

1. 2nd Thursday of month is the cut-off date for data to be included in statistics. Known as the count date.

2. 3 days after the count date the data extract is taken.

3. From 4 days after the count date to 8 days after the count date the data is transformed and prepared by data teams.

4. From 11 days after the count date to 15 days after the count date the data is analysed and the final datasets are produced.

5. From 18 days after the count date to 7 days before the publication date the outputs are quality assured. In the months which are a quarterly release, the bulletin and household dashboard are prepared.

6. 7 days before publication day the data on Stat-Xplore and the household dashboard is signed-off. No further changes to data on Stat-Xplore can be made after this time without delaying the release of the publication.

7. From 8 days before publication to 4 days before publication the process to clear the data and commentary with senior Statisticians takes place. Time is also allowed to incorporate feedback. Maps are also prepared for updating and the bulletin is prepared for web release including where necessary quality checking and redrafting.

8. 24 hours before publication the statistics are shared with Ministers and press office for briefing in line with DWP’s pre-release policy.

9. On publication day the statistics are released on GOV.UK and Stat-Xplore. This date is aligned with the release of Claimant Count statistics published by the Office for National Statistics (ONS), which uses in part some Universal Credit statistics.

The employment measure in the people series is not published for the latest month in the people series. This is because this measure requires information for the whole assessment period, which for around half of claims is not available until after the data extract is taken.

As people on Universal Credit is required monthly for the claimant count, Universal Credit official statistics aligns releases with the ONS Claimant Count, publishing monthly, one month in arrears.

Claims made, Starts and Households on Universal Credit are published quarterly. In each quarterly release Claims made and Starts are published up to the same date as People on Universal Credit as the latest data available. This publication schedule is to obtain a balance between user needs and available resources.

Households on Universal Credit are published 3 months in arrears. Due to the household series requiring statement and payment data, this is the earliest month for which there is complete data available for every quarterly release.

Households are counted in the ‘household on Universal Credit’ series if they have an assessment period spanning the count date, which is always the 2nd Thursday of the month. Most data in the ‘households on Universal Credit’ series is based on payment information contained in the statement detailing the claimant’s award. The statement is not produced until the end of the assessment period.

The household data cannot be produced for the most recent month. This is because nearly all assessment periods for households that will be included in that month will not have ended. Consequently, the statements for those households will not have been produced to be able to collate statistics on entitlements and payments.

Neither can the month before the latest month in the ‘people on Universal Credit’ be published as, in some months, when the data extract is taken there will still be some households whose assessment period has not ended and therefore entitlement and payment information is not available.

For example, for the February 2020 publication, the data extract is taken at the end of 11 January – 2 days after the 2nd Thursday in January. Statistics for households for December 2019 will include all assessment periods that span the December count date, which was the 12 December. Therefore, to collate statistics for December requires data for statements for assessment periods that end up to 11 December.

As statements are produced after the end of the assessment period, that means statements for assessment periods ending 11 December are produced no earlier than the 12 December and are not available for the data extract taken at the end of 11 January. Therefore, the latest complete month available to collate statistics on entitlements and payments for households is February. However, we also allow two complete months from the point of the data extract being taken for information to be verified and entitlements to be determined to avoid large revisions to the published statistics.

Payment timeliness is published 4 month in arrears. This is because the data is subject to a lot of retrospection. Publishing data earlier would lead to relative large revisions and lead to a loss of confidence in the earlier figures. This is the earliest payment timeliness can be published when balancing the needs of quality and timeliness of the statistics.

Punctuality

The statistics are published at 9.30am on the day that is pre-announced. The release calendar is updated at the earliest opportunity to inform users of any change to date of the statistical release and will include a reason for the change. All statistics will be published in compliance with the release policies in the Code of Practice for Statistics.

5. Accessibility and Clarity

Definition: Accessibility is the ease with which users are able to access the data, also reflecting the format in which the data are available. Clarity refers to the quality and sufficiency of the metadata, illustrations and accompanying advice.

Accessibility

Universal Credit statistics follow best practice and guidance from the Government Digital Service and Government Statistical Service in publishing statistics that are accessible to users with a range of disabilities.

The statistics are also released in open formats so they are accessible in a range of free software applications, including:

• statistics are released on GOV.UK and Stat-Xplore, supplemented by additional interactive tools
• the bulletin is published in HTML format on GOV.UK to Government Digital Service accessibility standards, giving equality of access to all users
• data is released on Stat-Xplore, a web-based interactive tool that allows users to produce tables and charts to meet their needs
• ready-made tables available on Stat-Xplore for users with less confidence in creating their own tables
• all data on Stat-Xplore, including the ready-made tables can be downloaded in open data formats and re-used
• supplementary tools, such as maps and dashboards, are also web based enabling users to visualise information on the web in a different form without needing other software

Universal Credit statistics are covered by the Accessibility Statement for DWP Statistics.

Clarity

For all available measures on Stat-Xplore metadata is available providing users with a description of the information and the quality of the data. Technical language is avoided where possible for example by using the term “households” as opposed to “benefit units” which is more likely to be understood by the users of our statistics.

Another example would be our use of “Conditionality regimes” which is a set of terms used to determine the level of support a claimant would get. These terms are used opposed to using “conditionality groups” and “labour market regimes”.

To help users each bulletin contains the definitions of key terms, with a more extensive glossary of terms used provided in the background information and methodology document.

Universal Credit statistics contain contact details for further information on the statistics, guidance on using the statistics, data sources, coverage, data limitations and other necessary relevant information to enable users of the data to interpret and apply the statistics correctly.

6 Comparability and Coherence

Comparability is the degree to which data can be compared over time, region or other domain. Coherence is the degree to which the statistical processes use the same concepts and harmonised methodology. With the goal being able to combine and make joint use of related data from different data sources.

Statistics are of consistent methodology over the time periods that they are available. As experimental statistics using data sources that are constantly being developed there are possibilities that changes to the system will affect the comparability of these statistics over time.

For example, a change to the system in January 2019 resulted in several not in payment claims being closed. This affected several measures in Universal Credit statistics and documentation is provided to explain these changes.

Similarly, temporary changes as a result of the coronavirus (COVID-19) pandemic has had an effect on the statistics, including increasing the proportion of not in-payment households and increases in the proportion of people in employment on Universal Credit. The changes made to Universal Credit that may affect the statistics are explained in the background information and methodology document.

There are differences in the methodology between people and households which mean that they cannot be compared. There are also methodological differences between employment and conditionality in the people on Universal Credit series preventing comparison between the two measures.

Terms and definitions used in Universal Credit statistics have been designed to be consistent with other government statistics wherever possible. Where it has been necessary to use different terms or definitions to other publications, the reasons for this are explained, for example when the term household is used in place of benefit units.

Universal Credit statistics is published on the same day as the ONS Claimant Count, which this data feeds into. Statistics on Universal Credit is also available in other DWP statistics publications. Details of these related statistics and an explanation of how they are related is provided in each release and in the background information and methodology document. Most of these publications are usually published on the same day as Universal Credit statistics.

7. Trade-offs between output quality and components

Trade-offs are the extent to which different dimensions of quality are balanced against each other

There are different definitions and points in the process for claiming Universal Credit where a count of people on Universal Credit could be taken. Each of the definitions have their advantages and disadvantages and the methodology chosen has to balance the competing requirements of different users.

There is always a trade-off between timeliness and quality in any statistics, and Universal Credit is no different. The following table highlights the trade-offs for each of the different definitions that could be used.

Counting rule	Rationale	Timeliness	Accuracy
1. Claimants who have an open claim on the count date	Count all people having submitted a claim for Universal Credit	Timely. Data can be extracted and produced within 1 month.	Includes people that have made a claim but are not eligible for Universal Credit because they do not meet the basic identity requirements for Universal Credit or do not agree to the commitments required to claim Universal Credit. Will also include people who have made a claim in anticipation of needing Universal Credit but do not complete the initial processes to assess their commitment requirements.
2. Open claim where claimant has verified their ID and accepted their claimant commitment on the count date	Counts people who met basic conditions for claiming Universal Credit and is subject, where applicable, to work related conditions to claim Universal Credit.	Timely. Data can be extracted and released within 1 month.	Excludes all people making a claim that do not meet the basic identity requirements or do not agree to the work related conditions required to claim Universal Credit. Includes people who withdraw their claim before first payment or people where, through further validation checks, are found not to be eligible. Excludes people with an open claim on the count date that have not verified their identity or agreed to their claimant commitment but go on to do so and receive a statement at the end of the assessment period.
3. Claimants who, for an assessment period on the count date, have received a statement	Counts people who are eligible for Universal Credit having passed verification checks for at least some entitlement and have had entitlement calculated.	Not timely. Data is not available for up to 2 months with a further month to prepare for publication. Many users require an earlier indication of the caseload.	Only includes claimants that are eligible for Universal Credit and have not withdrawn their claim before the first payment. Excludes open claims of claimants that may be eligible but who have not yet reached the end of the first assessment period or, where they have, have not yet passed verification checks for at least some entitlement.

The first definition is discounted on the basis that it includes a large amount of ineligible people.

The second definition is used for People on Universal Credit as it is timely and meets the needs of a range of users. It trades-off some accuracy by including some people who may not be eligible or withdraw their claim before the first payment, but on a lower scale than the first definition. But it also does not include some people who have an open claim and go on to be paid, because they have not, by the count date, verified their basic entitlement to Universal Credit.

The third definition is used for Households on Universal Credit as it requires payment information which cannot be provided by using the first two definitions.

8. Balance between performance, cost and respondent burden

Data is taken from administrative sources used for the operation of Universal Credit, supplemented by data from existing central data sources to minimise burden on data required from claimants and data providers.

Datasets produced for official statistics are re-used within the department for policy and operational analysis.

9. Confidentiality, transparency and security

Personal identifiers on the data provided from administrative systems are encrypted to prevent identification of individuals within the data.

Random error is introduced to statistical outputs to prevent identification of individuals in small areas. For more information on this please see the background information and methodology document.

Data is held securely with access given only to analysts who have completed internal data access and security training and have a need to access the data.

10. Contacts

Feedback on the content, relevance, accessibility and timeliness of these statistics and any non-media enquiries should be directed to:

Statistician: Stephen Slater

Email: team.ucos@dwp.gov.uk

For media enquiries on these statistics, please contact the DWP press office

For statistics enquiries only. These contact details are unable to provide any information or assistance with claiming Universal Credit.