Guidance

Assessment of compliance with the Code of Practice for Official Statistics: statistics on household resources

Published 7 December 2011

© Crown copyright 2011

This publication is licensed under the terms of the Open Government Licence v3.0 except where otherwise stated. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk.

Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned.

This publication is available at https://www.gov.uk/government/publications/assessment-of-compliance-with-the-code-of-practice-for-official-statistics-statistics-on-household-resources/assessment-of-compliance-with-the-code-of-practice-for-official-statistics-statistics-on-household-resources

1 Summary of findings

1.1 Introduction

1.1.1 This is one of a series of reports[footnote 1] prepared under the provisions of the Statistics and Registration Service Act 2007[footnote 2].

The Act requires all statistics currently designated as National Statistics to be assessed against the Code of Practice for Official Statistics[footnote 3].

The report covers the set of statistics produced by the Department for Work and Pensions (DWP), reported in:

1.1.2 This report is shorter than normal. In particular, section 3 adopts an ‘exception reporting’ approach – it includes text only to support the Requirements made to strengthen compliance with the Code and Suggestions made to improve confidence in the production, management and dissemination of these statistics.

This abbreviated style of report reflects the Head of Assessment’s consideration of aspects of risk and materiality[footnote 8].

The Assessment team nonetheless assessed compliance with all parts of the Code of Practice and has commented on all those in respect of which some remedial action is recommended.

1.1.3 This report was prepared by the Authority’s Assessment team, and approved by the Board of the Statistics Authority on the advice of the Head of Assessment.

1.2 Decision concerning designation as National Statistics

1.2.1 The Statistics Authority judges that the statistics covered by this report are readily accessible, produced according to sound methods and managed impartially and objectively in the public interest, subject to any points for action in this report. The Statistics Authority confirms that the statistics published in FRSAR, HBAI, PI and IRBTU can be designated as National Statistics, subject to DWP implementing the enhancements listed in section 1.5 and reporting them to the Authority by March 2012 (June 2012 for Requirements 2 and 4).

1.3 Summary of strengths and weaknesses

1.3.1 DWP has set up user groups for all these statistics apart from PI, and consults with users when making major changes to the statistics. It has published user feedback for all statistics apart from PI.

1.3.2 DWP carries out thorough quality assurance checks on the data collected using the Family Resources Survey (FRS) but, consequently, there is a substantial delay between the end of the survey period and publication of the statistics. It publishes detailed information about methods and quality, but it has not published information about the strengths and weaknesses of the statistics in relation to their potential uses.

1.3.3 FRSAR and HBAI contain detailed data tables but limited commentary to aid user interpretation of the statistics.

1.4 Detailed recommendations

1.4.1 The Assessment team identified some areas where it felt that DWP could strengthen its compliance with the Code. Those which the Assessment team considers essential to enable designation as National Statistics are listed in section 1.5.

Other suggestions, which would improve the statistics and the service provided to users but which are not formally required for their designation, are listed at annex 1.

1.5 Requirements for designation as National Statistics

Requirement 1

Take steps to develop a more complete understanding of the use made of PI, the needs of current and potential users, and user views on the service provided. Publish the relevant information and assumptions and use them to better support the beneficial use of the statistics (para 3.2).

Requirement 2

Improve the information about the quality of the statistics included in FRSAR, IRBTU and PI so that it describes the strengths and limitations of the statistics in relation to the range of potential uses (para 3.5).

Requirement 3

Review the balance between users’ needs for accuracy and timeliness and consider whether more timely statistics could be produced (para 3.6).

Requirement 4

Improve the commentary in FRSAR and HBAI so that it aids user interpretation of the statistics (para 3.7).

2 Subject of the assessment

2.1 The assessment covers 4 releases about household income published by DWP, all of which use data from the FRS. FRS is an annual survey which has been conducted since October 1992. Prior to this DWP used income data from the Family Expenditure Survey to compile the statistics presented in HBAI, PI and IRBTU.

2.2 The FRS has a stratified clustered probability sample design. In 2009 to 2010 48,315 UK addresses were included in the sample and it achieved a response rate of 58%. DWP commissions the Office for National Statistics [footnote 9] (ONS) and the National Centre for Social Research[footnote 10]: (NatCen) to administer the FRS for Great Britain (GB).

The fieldwork for the survey in Northern Ireland is managed by the Department for Social Development (DSDNI) and is currently carried out by the Northern Ireland Statistics and Research Agency[footnote 11].

NatCen is responsible for maintaining the electronic questionnaire and ONS is responsible for sampling in GB and initial data validation for the UK. From April 2012, it is proposed that the FRS will adopt the questions from ONS’s General Lifestyle Survey (GLF)[footnote 12] which collect data required by the EU on income and living conditions. The GLF will then be discontinued.

2.3 FRSAR, first published in 1994, presents statistics on the incomes and circumstances of private households in GB (prior to 2002 to 2003) or the UK (from 2002 to 2003). The report includes statistics on income and state support receipt, tenure, savings and investments, carers and disability, and occupation and employment.

HBAI was first published in 1988. In its present form it provides statistics on the income distribution[footnote 13] of individuals in the UK by a number of breakdowns including region, gender, ethnicity, family type, disability and economic status. It also presents statistics on material deprivation by the same breakdowns[footnote 14] I as well as information on the composition of low income groups.

The statistics include adjustments for household size and composition, to reflect the extent to which households of different size and composition require different levels of income to achieve the same standard of living.

PI, first published in 1995, presents statistics on the level and sources of income of UK pensioners by a number of breakdowns including age, region, ethnicity and household composition.

Finally, IRBTU, first published in 1993, presents statistics on the take-up rate of income related benefits[footnote 15] and analysis of those missing out on their entitlements by such characteristics as region, housing tenure and characteristics[footnote 16] of the benefit unit[footnote 17].

The statistics are compiled using modelling based on the FRS to estimate the number of eligible nonrecipients[footnote 18] and existing administrative data[footnote 19] on benefits claimants and recipients. The releases all cover the financial year and are usually published annually.

2.4 Government departments, including DWP and HM Treasury, use the statistics to monitor performance against poverty related targets, for example those set out in the Child Poverty Act, 2010[footnote 20]. They also use the related datasets to model the take-up of credit entitlements and to assess the impact of other government policies.

Academics use the statistics to inform research into financial inclusion and other poverty related themes, and to forecast future income distributions. Researchers also use the statistics on income to inform their selection process for recruiting suitable participants to take part in income-related research programmes.

Charities use the statistics to monitor affordable housing and characteristics of household wealth and spending. The statistics are also used by a variety of organisations in the production of other research reports and articles, for example in ONS’s Social Trends[footnote 21].

2.5 DWP has reported that it costs around £5m per year to produce its statistics on household resources.

3 Assessment findings

3.1 DWP has created separate groups for users of the statistics presented in HBAI, FRSAR and IRBTU. These groups include users mainly from government departments and meet at least once a year to discuss proposed changes to the publications and potential new methods.

DWP told us that the UK Data Archive[footnote 22] also hosts a separate FRS user group for academics. During 2009, DWP conducted an open consultation on the content of FRSAR and published the results on its website[footnote 23].

During 2010 and 2011 DWP undertook 2 consultation exercises to better understand the uses made of HBAI and IRBTU and to determine whether the releases meet users’ needs. DWP published the outcome of the consultations on its website[footnote 24].

3.2 DWP does not host a user group specifically for users of PI, but during 2011 it tried to obtain the views of these users via an online survey. DWP told us that it received a low response to the survey and has not published the results.

However, DWP told us that it receives queries and requests from users of PI, and that there is demand for the statistics. As part of the designation as National Statistics, DWP should take steps to develop a more complete understanding of the use made of PI, the needs of current and potential users, and user views on the service provided DWP should publish the relevant information and assumptions and use them to better support the beneficial use of the statistics.

DWP also maintains a record of all user queries it receives, and has introduced changes to the publications based on feedback from users. For example, it recently introduced in IRBTU additional analysis of benefit take up by government office region and ethnicity, following requests from users.[footnote 25].

3.3 HBAI, FRSAR, IRBTU and PI all include information about the methods used to compile the statistics. Several users told us that, generally, they found the explanations in the releases about methods and technical aspects of statistical production clear and useful.

The assessment team considers that, although there is methods information available to users within IRBTU, it does not clearly set out how the data are compiled to produce the statistics on benefit take-up.

We suggest that DWP improve the information about methods within IRBTU so that it clearly sets out how the data are compiled (Requirement 1).

3.4 DWP told us that the methods that it uses to compile income statistics is slightly different from that used by other European countries. DWP includes measures of income that are excluded by other European countries when compiling income statistics, for example the value of free school meals or free TV licences. DWP also deducts contributions to private pensions from incomes.

We suggest that DWP clarify within the poverty publications how the methods used for measuring income differ from other European measures of income, to aid international comparisons.

3.5 ONS has published a useful Summary Quality Report (SQR) for the FRSAR[footnote 26].

DWP also publishes information about the quality of the statistics within FRSAR, HBAI, IRBTU and PI. This includes a description of the accuracy of the statistics, potential sources of error or bias, and the coherence of the statistics.

However, it does not provide sufficient information about the strengths and weaknesses of the statistics presented in FRSAR, IRBTU and PI in relation to their potential uses. As part of the designation as National Statistics, DWP should improve the information about the quality of the statistics included in FRSAR, IRBTU and PI so that it describes the strengths and limitations of the statistics in relation to the range of potential uses[footnote 27].

3.6 Initial checks of the survey data are carried out by ONS, and DWP subsequently carries out its own checks on the data. This lengthy quality assurance process causes a lag of roughly 14 months between the end of the FRS survey period and the publication of FRSAR, HBAI, PI and IRBTU. The 2009 to 2010 IRBTU has been delayed for an additional 9 to 10 months due to developmental work.

The assessment team has not seen sufficient evidence that DWP has assessed the extent to which the quality assurance procedures create an appropriate balance between accuracy and timeliness in accordance with users’ needs. As part of the designation as National Statistics, DWP should review the balance between users’ needs for accuracy and timeliness and consider whether more timely statistics could be produced (Requirement 2). [footnote 28].

3.7 FRSAR and HBAI are large releases which include many tables of statistics but very little commentary to assist users’ understanding of the statistics. FRSAR does not include a summary of the main points so it is difficult for users to interpret the statistics and extract the main messages.

The releases all provide information about household resources in the UK but DWP does not publish information about how the findings in the releases relate to one another. As part of the designation as National Statistics, DWP should improve the commentary in FRSAR and HBAI so that it aids user interpretation of the statistics (Requirement 3)[footnote 29].

3.8 There has been a delay in the 2009 to 2010 FRS data being deposited at the UK Data Archive following a DWP data security review. DWP has been working to establish the discrepancy between its own data security requirements and the UK Data Archive’s practices which enable researchers to access the data under licence.

During the course of the assessment, DWP has deposited the FRS End User Licence dataset at the UK Data Archive, but it still has not deposited the Special User Licence dataset. Also during the course of the assessment, DWP has published information about the reason for the delay on its website (Requirement 4).

In meeting this requirement DWP should consider the points detailed in Annex 2 [footnote 30] and has been working to resolve the issue as quickly as possible.

3.9 In 2005, DWP started a work programme to link FRS data to other sources of income data with the purpose of validating survey data and allowing longitudinal analysis. It has introduced a consent question to the FRS, which has enabled it to start linking data for validation purposes.

We suggest that DWP publish an updated data linking strategy which explains the extent to which initial objectives are being met, plans for future data linking work, and information about the extent to which the outcomes will address users’ quality needs.

Annex 1: Suggestions for improvement

A1.1 This annex includes some suggestions for improvement to DWP’s income statistics in the interest of the public good. These are not formally required for designation, but the Assessment team considers that their implementation will improve public confidence in the production, management and dissemination of official statistics.

Suggestion 1

Improve the information about methods within IRBTU so that it clearly sets out how the data are compiled (para 3.3).

Suggestion 2

Clarify within the poverty publications how the methods used for measuring income differ from other European measures of income, to aid international comparisons (para 3.4).

Suggestion 3

Publish an updated data linking strategy which explains the extent to which initial objectives are being met, plans for future data linking work, and information about the extent to which the outcomes will address users’ quality needs (para 3.9).

Annex 2: Compliance with Standards for Statistical Releases

A2.1 In October 2010, the Statistics Authority issued a statement on Standards for Statistical Releases[footnote 31].

A2.2 In implementing any Requirements of this report (at paragraph 1.5) which relate to the content of statistical releases, we encourage the producer body to apply the standards as fully as possible.

Whilst this is not part of the Code of Practice for Official Statistics, the Authority regards it as advice that will promote both understanding and compliance with the Code. In relation to the statistical releases associated with the Statistics on Poverty, Benefit Take Up and Household Resources, this annex comments on compliance with the statement on standards.

Appropriate identification of the statistics being released

A2.3 The title of each release clearly describes the period to which the statistics relate. Only the title of FRSAR indicates the coverage of the statistics. None of the releases includes a statement about the frequency of publication, although it is possible to deduce this from the commentary.

The National Statistics logo is clearly displayed on all releases. The releases clearly identify the originating department and provide contact details for the responsible statistician. The 1st section of all releases includes a background and introduction section that outlines the structure of the report, identifying the main topics that the report covers.

It is reasonably clear which statistics are new.

Include commentary that is helpful to the non-expert and presents the main messages in plain English

A2.4 PI, IRBTU and HBAI include a summary of the main results at the start of the release but FRSAR does not. All of the releases include a section which explains definitions and concepts in a good level of detail. However, the complex nature of the topic area means that the narrative is still quite technical which could make the releases inaccessible for the non-expert user.

PI and IRBTU present detailed narrative alongside the statistics which in places goes beyond describing rises and falls in the data, and suggests potential reasons for observed changes. For example, IRBTU puts forward tentative reasons as to why eligible individuals may not access benefits.

Similarly, PI suggests possible reasons for age differences in pensioner income. FRSAR and HBAI do not provide the same level of commentary alongside the statistical tables and graphs that are presented in the release. These releases should include more depth of narrative which draws out the main messages from the data and provides users with an appropriate context within which to interpret statistical trends.

Use language that is impartial, objective and professionally sound

A2.5 The language used in all releases is objective and impartial throughout. The descriptive statements used are demonstrably consistent with the statistics. [footnote 31].

Include information about the context and likely uses

A2.6 HBAI refers to the Child Poverty Act 2010 and says that the HBAI statistics will be used to monitor 6 indicators in relation to the Act. The relevant statistics presented in HBAI are described in relation to these indicators.

PI, FRSAR and IRBTU could be improved by including more information about the policy and operational context within which the statistics have been produced and will be used. FRSAR and HBAI include sections which present information about the main uses made of the statistics. FRSAR, IRBTU and PI do not include adequate information about the strengths and weaknesses of the statistics in relation to their potential uses. Include, or link to, appropriate metadata

A2.7 The releases link to other useful and related statistical outputs and research. For example, HBAI links to Poverty and Income Inequality in Scotland [footnote 32]. Similarly, IRBTU compares the statistics that are presented on benefit take-up and marital status with findings from other published research, and suggests possible reasons for the findings. The releases could go into more detail about the degree to which the statistics presented are similar to other official statistics that relate to poverty, and discuss which sources are most appropriate for particular uses.

A2.8 All releases include detailed and useful information on the accuracy of the statistics and identify potential sources of error or bias. All releases include useful information on the data sources and methods used to compile the statistics. FRSAR has a detailed methodology section which describes sampling, data collection, how the data are validated, edited, and grossed, and how the statistics are compiled. This section also describes the use of harmonised concepts and questions and provides links to further information. PI presents less detailed methodology information but includes links to where users can find further details. The methods information presented in IRBTU could be improved by clarifying how each data source is used and how the statistics are compiled.

Annex 3: Summary of assessment process and users’ views

A3.1 This assessment was conducted from July to November 2011.

A3.2 The Assessment team – Emma Bowditch and Ruth James – agreed the scope of and timetable for this assessment with representatives of the DWP in July.

The Written Evidence for Assessment was provided on 30 July. The Assessment team subsequently met DWP during August to review compliance with the Code of Practice, taking account of the written evidence provided and other relevant sources of evidence.

Summary of users contacted, and issues raised

A3.3 Part of the assessment process involves our consideration of the views of users. We approach some known and potential users of the set of statistics, and we invite comments via an open note on the Authority’s website. This process is not a statistical survey, but it enables us to gain some insights about the extent to which the statistics meet users’ needs and the extent to which users feel that the producers of those statistics engage with them. We are aware that responses from users may not be representative of wider views, and we take account of this in the way that we prepare assessment reports.

A3.4 The Assessment team received 22 responses from the user consultation. The respondents were grouped as follows:

  • Government 12
  • Research Institution/Think Tank 6
  • Charity/non-profit sector 2
  • Other public sector 1
  • Private sector 1

A3.5 Users were complimentary about the range of data that are available and generally considered the commentary and technical guidance to be comprehensive. One user felt that the methods information provided in IRBTU needed to be clearer but several other users were complimentary about the methods information available for the releases.

Users generally considered the statistical teams within DWP to be very helpful and responsive. Some users identified timeliness as an area of concern. Several users mentioned that, at the time of consultation, they were still unable to access the full FRS dataset through the UK Data Archive.

Some users also requested further disaggregation of the data in order to distinguish between rural and urban areas.

Written Evidence for Assessment document

  1. http://www.statisticsauthority.gov.uk/assessment/assessment-reports/index.html ↩

  2. http://www.opsi.gov.uk/Acts/acts2007/pdf/ukpga_20070018_en.pdf ↩

  3. http://www.statisticsauthority.gov.uk/assessment/code-of-practice/index.html ↩

  4. http://statistics.dwp.gov.uk/asd/frs/index.php?page=intro ↩

  5. http://statistics.dwp.gov.uk/asd/index.php?page=hbai_arc ↩

  6. http://research.dwp.gov.uk/asd/index.php?page=pensioners_income_arc#PI_Latest ↩

  7. http://statistics.dwp.gov.uk/asd/index.php?page=irb_arc ↩

  8. http://www.statisticsauthority.gov.uk/assessment/assessment/guidance-about-assessment/criteria-fordeciding-upon-the-format-of-an-assessment-report.pdf ↩

  9. http://www.ons.gov.uk/ons/index.html ↩

  10. http://www.natcen.ac.uk/ ↩

  11. http://www.natcen.ac.uk/ ↩

  12. http://www.ons.gov.uk/ons/rel/ghs/general-lifestyle-survey/2009-report/index.html ↩

  13. Income is equivalised to take account of household size and composition ↩

  14. In 2004/05 DWP introduced a suite of questions to measure material deprivation amongst families with children. In May 2008, DWP introduced a suite of questions to measure material deprivation amongst pensioners ↩

  15. The income related benefits covered are: Income support and employment allowance, Pension Credit, Housing Benefit, Council Tax Credit and Jobseeker’s Allowance ↩

  16. For example, ethnicity ↩

  17. A benefit unit is a single adult or a co-habiting couple, with or without children ↩

  18. Eligible non recipients are benefits units that are entitled to benefit, but that are not currently claiming the benefits that they are entitled to ↩

  19. The statistics presented in IRBTU are produced using DWP’s Income Support database, Jobseekers Allowance database and Pension Service database, and also extracts from the Single Housing Benefit Databases and the Rent Officer data databases owned by Local Authorities ↩

  20. http://www.legislation.gov.uk/ukpga/2010/9/contents ↩

  21. http://www.ons.gov.uk/ons/rel/social-trends-rd/social-trends/social-trends-39/social-trends-full-report.pdf ↩

  22. http://www.data-archive.ac.uk/ ↩

  23. http://research.dwp.gov.uk/asd/frs/200708/frs_publication_content%20review_2009.pdf ↩

  24. The outcome of DWP’s IRBTU consultation can be accessed at: http://statistics.dwp.gov.uk/asd/index.php?page=irb_arc. The outcome of DWP’s HBAI consultation can be accessed at: http://research.dwp.gov.uk/asd/index.php?page=hbai_arc. The outcome of DWP’s HBAI usage survey can be accessed at: http://research.dwp.gov.uk/asd/index.php?page=hbai_arc#usage_survey ↩

  25. In relation to Principle 1, Practice 5 of the Code of Practice ↩

  26. http://www.ons.gov.uk/ons/guide-method/method-quality/quality/quality-information/social-statistics/index.html ↩

  27. In relation to Principle 4, Practice 3 and Principle 8, Practice 1 of the Code of Practice ↩

  28. In relation to Principle 4, Practice 2, Principle 7, Practice 5, Protocol 1, Practice 6 and Protocol 2, Practice 1 of the Code of Practice ↩

  29. In relation to Principle 8, Practice 2 of the Code of Practice ↩

  30. http://research.dwp.gov.uk/asd/frs ↩

  31. http://www.statisticsauthority.gov.uk/news/standards-for-statistical-releases.html ↩ ↩2

  32. https://www2.gov.scot/News/Releases/2011/05/12142035 ↩

Back to top