Consultation outcome

Annex 5: Tenant Satisfaction Measures - Tenant survey requirements (accessible)

Updated 21 September 2022

Introduction

1 - The Tenant Satisfaction Measures Standard requires all registered providers ^{[footnote 1]} to generate and report TSMs as specified by the regulator. This document, Tenant Satisfaction Measures: Tenant survey requirements, sets out the basis upon which providers are required to conduct tenant perception surveys to generate a subset of TSMs (‘tenant perception measures’). This document must be read in conjunction with Tenant Satisfaction Measures: Technical requirements.

2 - The requirements in this document aim to establish a robust basis for generating comparable tenant perception measures across providers. They represent a consistent approach to key aspects of survey methodology, while allowing sufficient flexibility for providers to meet requirements in a way that fits with wider objectives and a range of operating models. This document includes some statistical requirements which are necessarily technical, although it describes how the approach to meeting these requirements should be proportionate to the size and complexity of each provider.

3 - This document is not intended to offer guidance on how to conduct tenant perception surveys. Providers must determine the most appropriate way to meet the requirements in this document, taking advice where necessary.

4 -This remainder of this document is structured as follows:

• Section 1 – Scope of tenant perception surveys
• Section 2 – Sampling approach.

5 - The Annexes contain reference information relevant to our requirements. This includes a summary of requirements for providers with fewer than 1,000 dwelling units of relevant social housing stock (small providers) ^{[footnote 2]} (Annex A), a description of common sampling methods (Annex B), and illustrative sample sizes (Annex C).

6 - These requirements have been prepared by the regulator drawing on advice commissioned from BMG Research.

Scope of tenant perception surveys

7 - This section sets out the scope of tenant perception surveys that registered providers are required to undertake with tenants in order to generate tenant perception measures.

Type of survey

8 - The tenant perception measures listed below (TP01-TP12) must be generated using data from perception surveys only. A perception survey is intended to periodically capture tenants’ general views of landlord performance and is not triggered by a recent interaction with the landlord. Providers must not use any data from transactional surveys to calculate these TSMs ^{[footnote 3]}.

Table 1: Type of survey

	Tenant Satisfaction Measure	Collection method
TP01	Overall satisfaction	Perception survey
TP02	Satisfaction with repairs	Perception survey
TP03	Satisfaction with time taken to complete most recent repair	Perception survey
TP04	Satisfaction that the home is well maintained	Perception survey
TP05	Satisfaction that the home is safe	Perception survey
TP06	Satisfaction that the landlord listens to tenant views and acts upon them	Perception survey
TP07	Satisfaction that the landlord keeps tenants informed about things that matter to them	Perception survey
TP08	Agreement that the landlord treats tenants fairly and with respect	Perception survey
TP09	Satisfaction with the landlord’s approach to handling complaints	Perception survey
TP10	Satisfaction that the landlord keeps communal areas clean and well maintained	Perception survey
TP11	Satisfaction that the landlord makes a positive contribution to neighbourhoods	Perception survey
TP12	Satisfaction with the landlord’s approach to handling anti-social behaviour	Perception survey

9 - The remainder of this document sets out how providers must undertake perception surveys to generate the tenant perception measures.

Survey questions and response options

10 - Perception surveys must include the survey question wording and response options as specified in Table 2 below. ^{[footnote 4]} This is irrespective of the survey collection method used (see section 2), so for example in a face-to-face or telephone survey, the surveyor must read the question followed by the response options specified below in full.^{[footnote 5]} Written or online surveys must likewise include the response options below. Unless they are necessary to overcome specific barriers to a particular group of tenants completing the survey, providers must not introduce additional visual features such as emojis alongside the response options.^{[footnote 6]}

11 - Where [your landlord] or [my landlord] is written in Table 2 this can be substituted for the provider name (entity name, group name or recognised brand name), ‘your/my landlord’ or ‘your/my social housing provider’. For example, local authority registered providers are permitted to append ‘housing services’ to their name to distinguish the landlord function from other services provided by local authorities.

12 - Providers must follow the response scale for each question below. For example, a ‘not applicable/ don’t know’ response must be included where this is shown for a question (and must not be included where this is not shown). For all questions if tenants offer an unprompted ‘not applicable/don’t know’ response, for example during a telephone interview, providers must record this as an effective non-response to this particular question.

Table 2: Tenant perception survey question wording

	TSM	Survey question wording
TP01	Overall satisfaction	‘Taking everything into account, how satisfied or dissatisfied are you with the service provided by [your landlord]?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP02	Satisfaction with repairs	‘Has [your landlord] carried out a repair to your home in the last 12 months?’ - Yes - No If yes, ‘How satisfied or dissatisfied are you with the overall repairs service from [your landlord] over the last 12 months?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied.
TP03	Satisfaction with time taken to complete most recent repair	‘Has [your landlord] carried out a repair to your home in the last 12 months?’ - Yes - No If yes, ‘How satisfied or dissatisfied are you with the time taken to complete your most recent repair after you reported it?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP04	Satisfaction that the home is well maintained	‘How satisfied or dissatisfied are you that [your landlord] provides a home that is well maintained?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP05	Satisfaction that the home is safe	‘Thinking about the condition of the property or building you live in, how satisfied or dissatisfied are you that [your landlord] provides a home that is safe?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP06	Satisfaction that the landlord listens to tenant views and acts upon them	‘How satisfied or dissatisfied are you that [your landlord] listens to your views and acts upon them?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP07	Satisfaction that the landlord keeps tenants informed about things that matter to them	‘How satisfied or dissatisfied are you that [your landlord] keeps you informed about things that matter to you?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP08	Agreement that the landlord treats tenants fairly and with respect	‘To what extent do you agree or disagree with the following “[my landlord] treats me fairly and with respect”?’ Response options: - Strongly agree - Agree - Neither agree nor disagree - Disagree - Strongly disagree - Not applicable/ don’t know.
TP09	Satisfaction with the landlord’s approach to handling complaints	‘Have you made a complaint to [your landlord] in the last 12 months?’ - Yes - No If yes, ‘How satisfied or dissatisfied are you with [your landlord]’s approach to complaints handling?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP10	Satisfaction that the landlord keeps communal areas clean and well maintained	‘Do you live in a building with communal areas, either inside or outside, that [your landlord] is responsible for maintaining?’ - Yes - No - Don’t know If yes, ‘How satisfied or dissatisfied are you that [your landlord] keeps these communal areas clean and well maintained?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied
TP11	Satisfaction that the landlord makes a positive contribution to neighbourhoods	‘How satisfied or dissatisfied are you that [your landlord] makes a positive contribution to your neighbourhood?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.
TP12	Satisfaction with the landlord’s approach to handling anti-social behaviour	‘How satisfied or dissatisfied are you with [your landlord]’s approach to handling anti-social behaviour?’ Response options: - Very satisfied - Fairly satisfied - Neither satisfied nor dissatisfied - Fairly dissatisfied - Very dissatisfied - Not applicable/ don’t know.

Questionnaire structure

13 - Registered providers must ensure the purpose of the perception survey is explicitly communicated to the tenant at the start of each survey conducted. Specifically, alongside any reference to other purposes, providers must state that the survey will be used to calculate annual TSMs to be published by the landlord. Local authority registered providers are permitted to include appropriate introductory wording to clarify that the survey relates to housing services provided by the local authority as part of its landlord function (as distinct from wider local authority services).

14 - In addition to the questions above, providers are permitted to include other questions in the same tenant perception survey questionnaire. These additional questions may be used, for example, to better understand responses to the TSM questions, seek responses on wider or more detailed aspects of performance, or to generate data on tenant characteristics. Additional questions can be in any appropriate format (e.g. lists or free text), and can include questions that seek more information about a tenant’s response to TSM questions.

15 - The question to generate overall satisfaction (TP01) must appear as the first question in any perception survey questionnaire used to generate TSMs. The question on satisfaction with repairs (TP02) must appear before the question on timeliness of repairs (TP03). Questions to generate the other TSMs must precede any other question(s) that a provider chooses to include on the same topic. This means that the questions to generate TSMs (TP01 - TP12) may appear together at the start of a survey or separately at the start of relevant sections with related questions. For example, if the survey includes a section with several questions on repairs, the TSM questions on repairs (TP02 - TP03) must appear before any other questions on repairs – either at the start of this specific section or at the start of the survey. Beyond these requirements, there is no prescribed order for the questions.

16 - Providers must take reasonable action and care when undertaking or commissioning surveys to ensure that participants are not led toward a particular point of view. In particular, beyond any necessary clarification, providers must avoid introducing any additional wording or preceding questions that are likely to have a significant impact on responses to TSM questions. This principle applies to any introductory wording used to frame the purpose of the survey.

17 - Providers must consider the most appropriate length of the survey in order to maximise response rates across the range of households, given the information the provider needs to generate, and to avoid survey fatigue having a significant impact on responses. Providers must advise respondents of the approximate length of completion at the start of the survey.

Relevant tenant population

18 - Providers must use the relevant tenant population as set out in Table 3 as the basis for tenant perception measures. This means that all households in the relevant tenant population must be included within the sampling frame to generate a perception measure ^{[footnote 7]}. For providers that own at least 1,000 dwelling units of relevant social housing stock, this means that:

• Providers that own 1,000 or more Low Cost Rental Accommodation dwelling units must report tenant perception measures for LCRA specifically. For these perception measures, the relevant population comprises all households residing in LCRA owned by the provider.
  
  

• Providers that own 1,000 or more Low Cost Home Ownership units must report tenant perception measures for LCHO specifically. For these perception measures, the relevant population comprises all households residing in LCHO owned by the provider.
  
  

• Providers that own 1,000 or more LCRA dwelling units, but fewer than 1,000 LCHO dwelling units, are only required to report tenant perception measures for LCRA (and vice versa).
  
  

• Providers that own 1,000 or more LCRA dwelling units and 1,000 or more LCHO dwelling units must report tenant perception measures for LCRA and LCHO separately.

19 - Registered providers with fewer than 1,000 dwelling units of LCRA and fewer than 1,000 dwelling units of LCHO can choose to calculate and report perception measures for LCRA (only), LCHO (only), both stock types separately, or both stock types combined. These providers must be able to demonstrate a proportionate and rational approach to determining the basis for reporting perception measures in this respect, following their stock profile. The relevant tenant population must follow the basis such a provider selects to report perception measures. For example, if a provider reports perception measures for LCRA and LCHO combined, the relevant tenant population must comprise all households residing in LCRA and LCHO owned by the provider.

Table 3: Relevant tenant population

Tenant perception measures	Relevant tenant population	Providers required to report tenant perception measures on this basis
Tenant perception measures for LCRA	All households residing in LCRA owned by the provider.	All providers that own 1,000 or more LCRA dwelling units.
Tenant perception measures for LCHO	All households residing in LCHO owned by the provider.	All providers that own 1,000 or more LCHO dwelling units.
Tenant perception measures for LCRA and LCHO combined	All households residing in LCRA and LCHO owned by the provider.	Providers that own fewer than 1,000 dwelling units of LCRA and fewer than 1,000 dwelling units of LCHO can choose to report perception measures for LCRA (only), LCHO (only), LCRA and LCHO separately, or LCRA and LCHO combined.

Units of measurement

20 - In order to generate tenant perception measures and assess statistical accuracy, providers must record the number of responses to each survey question and the total responses to the perception survey as a whole. For each question, there must be a maximum of one response recorded per household irrespective of how many legal tenants there are in a household. For the purposes of these requirements, a household refers to the group of people that occupies a single dwelling unit (as it is defined in Tenant Satisfaction Measures: Technical requirements ^{[footnote 8]}.

21 - Providers must record the number of responses to each question as the total number of surveyed households that have provided one of the following responses: Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied or Very dissatisfied ^{[footnote 9]}. Providers must ensure that the count of responses for a question does not include any households that have either answered ‘Not applicable/don’t know’ or otherwise have not responded to the question.

22 - Providers must include partial survey responses within the count of responses – for example, if a household has responded to some but not all questions in the survey, they must be included within the responses for those questions they have responded to.

23 - Providers must record the total responses to the tenant perception survey as the total number of surveyed households that have provided one of the following responses – Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied or Very dissatisfied (or the equivalent agreement responses) – to at least one of the tenant perception survey questions.

Survey timings

24 - Data used to generate tenant perception TSMs must be collected as part of a single integrated survey exercise which has been designed to meet the requirements in this document. This requires that all survey responses used by a provider must be generated using a questionnaire consistent in length and structure ^{[footnote 10]}, an integrated sampling approach (e.g. to avoid double counting of households) and be subject to an overarching assessment of statistical accuracy and representativeness which meets the requirements set out in this document. The fieldwork undertaken as part of this exercise may happen at a single point in time, through a phased approach, or on a periodic rolling basis (e.g. quarterly or monthly). ^{[footnote 11]} All valid survey responses collected through this exercise must be included in the calculation of the perception measures ^{[footnote 12]}. Informal survey data which does not meet the requirements set out in this document must be excluded from the responses used to calculate the perception measures.

25 - Providers that own 1,000 or more dwelling units of relevant social housing stock ^{[footnote 13]} must carry out tenant perception surveys of the relevant tenant population (as specified in Table 3) at least annually. This means that tenant perception measures for these providers must only be calculated using responses from surveys conducted within the reporting year the TSMs relate to. The reporting year for all providers with 1,000 or more dwelling units of relevant social housing stock is 1 April to 31 March.

26 - Providers that own fewer than 1,000 dwelling units of relevant social housing stock (small providers) must carry out perception surveys at least once every two years.^{[footnote 14]} This means that tenant perception measures for these providers must only be calculated using responses from surveys conducted within the past two reporting years.

27 - Providers with fewer than 1,000 dwelling units of relevant social housing stock are permitted to collect and report annual TSMs according to a reporting year other than 1 April to 31 March. For example, small providers are permitted to use a reporting year for TSMs which aligns with their particular financial reporting year.

28 - In calculating the perception measures, providers must use the most recent survey exercise completed that meets the requirements in this document.

In-house and outsourced surveys

29 - Providers must ensure that they employ a suitable level of expertise to design and apply a survey methodology to generate tenant perception measures in a way that meets the requirements of this document. The level of expertise required will be proportionate to the size and complexity of each provider’s tenant base and sampling approach chosen. For example, the level of required statistical and market research expertise is significantly greater for a large provider sampling across a diverse tenant base, than the level required for a provider with fewer than 1,000 dwelling units undertaking a relatively simple census across all relevant tenant households.

30 - In meeting the requirements of this document, providers can choose to conduct surveys and calculate the perception measures in-house, or through external contractors. Providers must as far as possible ensure that tenant perception measures are collected and reported in a way that adheres to the Market Research Society Code of Conduct where its provisions are relevant. Providers must have adequate assurance that any external contractor employed in this capacity adheres to the MRS Code of Conduct. Providers must ensure that calculations used to generate perception measures have been subject to an appropriate level of verification and validation. It is ultimately the responsibility of Boards of private registered providers and governing bodies of local authority registered providers to ensure that the tenant perception TSMs are accurate and built on a survey methodology that meets the requirements set out in this document.

Data protection and confidentiality

31 - Registered providers must ensure that they and any contractors are compliant with the relevant privacy and data protection legislation when processing personal data. This includes all processing, including during the conduct of the survey itself and in the publication of results.

32 - Providers must protect respondent confidentiality in order to protect tenants and give respondents confidence to give honest feedback. This includes ensuring that identifiable individual responses to survey questions are not shared internally or externally beyond those who require the information to produce the TSMs, unless the respondents give explicit consent and this is managed appropriately (or an alternative lawful basis is identified).^{[footnote 15]}

33 - Providers with fewer than 1,000 dwelling units of relevant social housing stock must take a proportionate approach to tenant perception measures in order to protect tenant confidentiality. For the smallest of these providers, it may be difficult to protect the anonymity of individual tenants if tenant perception information is published in full. It is permissible for at least some tenant perception measures and supporting analysis be omitted from published information if providers judge this to be a material risk.

Publication of summary of approach

34 - Providers must publish a summary of the survey approach used to generate published tenant perception measures. This must be made clearly available alongside each set of tenant perception measures published by the provider. It must include at a minimum:

a. a summary of achieved sample size (number of responses)

b. timing of survey

c. collection method(s)

d. sample method

e. summary of the assessment of representativeness of the sample against the relevant tenant population (including reference to the characteristics against which representativeness has been assessed)

f. any weighting applied to generate the reported perception measures (including a reference to all characteristics used to weight results)

g. the role of any named external contractor(s) in collecting, generating, or validating the reported perception measures

h. the number of tenant households within the relevant population that have not been included in the sample frame due to the exceptional circumstances described in paragraph 63 with a broad rationale for their removal ^{[footnote 16]}

i. reasons for any failure to meet the required sample size requirements summarised in Table 5

j. type and amount of any incentives offered to tenants to encourage survey completion

k. any other methodological issues likely to have a material impact on the tenant perception measures reported.

35 - The summary of approach must be proportionate to the complexity of the sampling methods employed and must include sufficient information to enable reasonable assessment of the validity of the published tenant perception measures. For example, the level of detail required from a relatively large provider applying stratified sampling and weighting of responses is significantly greater than that required from a small provider employing a simple census approach. Alongside this summary, all providers must publish the questionnaire(s) used to generate survey responses. This must include any additional questions and introductory or explanatory wording communicated to tenants alongside the TSM questions.

36 - As part of the summary of the assessment of representativeness, all providers that own 1,000 or more dwelling units of relevant social housing stock must set out the following: proportion of the relevant (a) tenant population and (b) total survey responses that share the principal characteristics for which representativeness has been assessed (see illustration in Table 4). For these providers, a rationale for the choice of characteristics included must be set out with reference to tenant and stock profile. Where weighting has been used to ensure that the sample is as far as possible representative, (b) must reflect the weighted total survey responses used to generate reported TSMs.

37 - The summary of approach must state if the provider has undertaken any tenant perception surveys which include TSM questions but has not included these responses in the calculation of the TSMs. A rationale for why this information has been excluded must be provided. The provider must include a summary of responses by survey collection method and the rationale for the survey collection method(s) chosen. Where there are any material year-on-year changes in survey methodology, for example in survey collection method(s), a summary of these changes must be included with the reason for any such changes. Further, any analysis of year-on-year changes in tenant perception measure performance published by the provider must refer to any material changes in survey methodology that are likely to have significantly affected satisfaction scores.

Table 4: Illustration of summary of representativeness

^{[footnote 17]}

Tenant perception measures	Relevant tenant population (% total)	Total survey responses (% total)*
Housing type
Sheltered housing	30%	30%
General needs housing	70%	70%
Age of respondent
75+	25%	28%
55-75	25%	26%
…	…	…

If weighting has been used to generate reported TSMs, this must reflect weighted total responses. If such weighting has not been used, this should reflect total responses without any adjustments.

Regulatory data returns

38 - Registered providers must submit to the regulator information relating to TSMs as specified by the regulator. We expect that this will, in general, include a standard data return prepared by the regulator covering both calculated tenant perception measures as well as summary information on the survey approach used.

Sampling approach

39 - This section sets out the approach providers must take to generating a sample of responses used to calculate tenant perception measures. There are two broad approaches to collecting such a sample:

• A census approach – where all households in the relevant tenant population are invited to participate in the survey.
  
  
• A sample approach – where a sample of relevant tenant households are invited to participate in the survey.

40 - With either approach, the accuracy of tenant perception measures depends on how many households respond to the survey (‘the sample size’) and how well these respondents represent the relevant tenant population (‘representativeness’).

Sample size

41 - When undertaking a survey, providers must, as far as possible, generate a sample size for overall satisfaction (TP01) that meets the minimum level of statistical accuracy set out in Table 5 ^{[footnote 18]}. For example, a provider with 10,000 LCRA dwelling units must, as far as possible, achieve a sample where estimated overall satisfaction for LCRA tenants is accurate to a margin of no more than +/-3% at a 95% confidence level. With this level of accuracy, the provider can be 95% sure that – as long as the sample is representative – the estimated level of satisfaction is right (to within 3% above or below) for the whole population ^{[footnote 19]}.

Table 5: Required minimum levels of statistical accuracy for overall satisfaction

Population	Required minimum statistical accuracy (margin of error at 95% confidence level)
Fewer than 2,500 dwelling units	+/- 5%*
2,500 – 9,999 dwelling units	+/- 4%
10,000 – 24,999 dwelling units	+/- 3%
25,000 dwelling units or more	+/- 2%

*Where it is not possible to achieve this level of statistical accuracy (for example for many providers with fewer than 1,000 dwelling units of relevant social housing stock), employing a census approach is sufficient to meet this requirement.

42 - Providers must determine the sample size required to achieve the required levels of statistical accuracy, for example using an online calculator or expert statistical input, ^{[footnote 20]} in order to plan survey activity. The sample size required to achieve a given level of statistical accuracy applies to each relevant tenant population for which separate perception measures are required ^{[footnote 21]}.The required sample size depends primarily on the size of this relevant tenant population. Where weighting has been used these requirements relate to the effective sample size. Annex C sets out indicative sample size corresponding to example relevant tenant populations.

43 - The sample size refers to the total number of responses to the tenant perception survey ^{[footnote 22]}. It is not the number of surveys issued or contacts attempted but the number of responses received. Since precise response rates are unknown in advance, providers must incorporate an appropriate degree of contingency in their planned survey activity in order that the achieved sample is likely to be at least the size required to meet the minimum level of statistical accuracy set out in Table 5 ^{[footnote 23]}.

44 - Providers that own 1,000 or more dwelling units of relevant social housing stock ^{[footnote 24]} must as far as is possible meet these minimum standards of statistical accuracy – whether they undertake a census or other sampling approach – unless there are exceptional circumstances that mean this is not practically possible using standard survey approaches. Providers may choose to exceed minimum sample sizes, for example to generate robust estimates of satisfaction for particular groups of tenants.

45 - Where it is practically difficult to achieve minimum sample sizes using conventional survey collection methods, providers may undertake a census as a means of meeting the requirements of this document in relation to sample size (i.e. inviting all households in the relevant tenant population to participate in a survey). This is irrespective of whether a census approach generates the minimum level of statistical accuracy set out in Table 5. This is likely to apply in particular to small providers with fewer than 1,000 dwelling units.

Representativeness of responses

46 - A survey that has met the sample size requirements can still be biased if groups of tenants who on average have higher satisfaction are over-represented in the sample (or vice versa).

47 - Providers must ensure that, as far as possible, survey responses used to calculate perception TSMs are representative of the relevant tenant population ^{[footnote 25]}. Providers can meet this requirement through one of two routes:

• a. representative sample: This means there is no material under – or over – representation of tenant groups (compared to the relevant tenant population) that is likely to affect calculated satisfaction scores. Using this approach, providers must ensure that the achieved sample is representative of the relevant tenant population.
  
  

• b. Weighting responses: If the achieved sample is not representative of the tenant population, then providers must appropriately weight the responses to ensure the TSMs reported are representative as far as possible ^{[footnote 26]}.

48 - All providers must assess the extent to which the achieved sample is representative of the relevant tenant population. Specifically, providers must undertake reasonable checks for differences between total survey responses and the relevant tenant population in terms of characteristics associated with different average satisfaction scores. For example, for many providers tenants in sheltered housing or in larger properties typically have higher satisfaction than average – for large providers where this is likely to have a material impact on satisfaction scores, reasonable checks are likely to include assessing how the proportion of survey responses from such households compares to the relevant tenant population (see illustration in Table 6 below).

Table 6: Illustration of a check for representativeness of survey responses

	Relevant tenant population (dwelling units % total)	Total survey responses (% total)	Calculated satisfaction score
Provider 1
LCRA	25,000 (100%)	2,500 (100%)	75%
Check: Housing type
Sheltered housing	7,500 (30%)	1,250 (50%) x	85%
General needs housing	17,500 (70%)	1,250 (50%) x	65%
Provider 2
LCRA	50,000 (100%)	5,000 (100%)	71%
Check: Housing type
Sheltered housing	15,000 (30%)	1,500 (30%) /	85%
General needs housing	35,000 (70%)	3,500 (70%) /	65%

x - In this example provider 1’s sheltered housing (30% of all units) is over-represented in the sample (50% of responses) and general needs housing (70% of units) is under-represented (50% of responses). As there is a large difference in the satisfaction score recorded (85% in sheltered housing, 65% in general needs housing) it is likely to have a material impact on the calculated satisfaction score. Therefore, the sample is not representative and the perception measures will be biased estimates of the satisfaction score for the relevant tenant population. To address this, responses must be appropriately weighted to ensure the calculated perception measures are representative of the relevant tenant population.

/ - Provider 2 has surveyed a representative proportion of tenants based on stock type. 30% of tenants live in sheltered housing and 30% of those who responded live in this stock type. Similarly, 70% of tenants live in general needs housing and 70% of those who responded live in this stock type. Provider 2 does not need to weight its responses based on stock type.

This illustration focusses on stock type. Providers need to consider what other characteristics should be included in their assessment of representativeness.

49 - Providers must reach a balanced judgement as to which characteristics to include in this assessment of representativeness based on their particular tenant profile, evidence or rationale for potential different satisfaction scores by characteristic, and available data. Current evidence from providers’ analysis of tenant satisfaction suggests that the following characteristics are material for many large providers:

• stock type (e.g. general needs, housing for older people/sheltered housing, other supported housing, temporary social housing)
• age of respondent
• ethnicity
• building type (e.g. high rise/flats)
• property or household size
• geographical area or estate.

50 - Providers must ensure they hold requisite information on tenant characteristics to undertake this analysis. This includes ensuring data for the relevant tenant population is sufficiently robust.

51 - For most large providers, stock type is likely to be a significant influence on satisfaction scores and is likely to be included within the scope of an assessment that meets this requirement.

52 - The scope of the assessment must be proportionate to the size and complexity of each provider’s stock, tenant base and sampling approach. Providers with fewer than 1,000 dwelling units of relevant social housing stock (small providers) ^{[footnote 27]} are only required to undertake a high-level assessment of representativeness. For example, by checking their average results against a small number of key characteristics.

53 - If this assessment confirms the sample is unrepresentative, and this is likely to have a material impact on satisfaction scores, then providers must appropriately weight the responses to ensure the TSMs reported are representative as far as possible. Weighting needs to be conducted appropriately to meet this objective and may require expert input. Weighting used to calculate perception TSMs must not be undertaken to meet other objectives, for example to attempt to control for potential effects of different collection methodology or to seek to reflect a national tenant profile.

54 - Weights must be calculated on the basis of total responses to the perception survey and applied consistently across all tenant perception measures.

55 - Providers with fewer than 1,000 dwelling units of relevant social housing stock are not required to weight responses, unless it is possible to generate a sample large enough to meet the minimum statistical accuracy set out in Table 5 and there is strong evidence of a significant bias in estimated scores.

56 - Providers are permitted to sample a larger proportion of tenants in any particular group, either due to a known issue of low response rates or to gain more detailed statistically valid information. Providers that choose to ‘over-sample’ particular groups must assess the impact on the representativeness of the sample, and if required weight calculated TSMs results to ensure that they are representative of the overall relevant tenant population as far as possible.

57 - In order to calculate representativeness and minimum sample size, providers must use a relevant tenant population based on the number of tenant households at a particular point in time. For example, temporary social housing might account for 10% of a provider’s LCRA stock (at 31 March) but due to high turnover it could account for the majority of households residing in LCRA stock over the course of a year (1 April – 31 March). In this case, analysis of representativeness and minimum sample size must assume that temporary social housing (and the characteristics of resident households) accounts for 10% of the relevant tenant population ^{[footnote 28]}.

58 - Providers must have an appropriate approach to retaining survey data, details of the survey approach, and calculations used to generate perception measures for a reasonable period of time ^{[footnote 29]}. At a minimum, this must include sufficient data and information to permit replication of the last two iterations of calculated tenant perception measures and assess their validity. For each iteration, this must include the number of responses per response option for each tenant survey question (including filter questions), details of the assessment of representativeness, any weighting applied, as well as the number of responses by survey collection method and the average overall satisfaction score (TP01) for each method.

Survey collection method

59 - Providers must use an appropriate survey collection method or methods considering factors such as likely response rate, cost, addressing barriers to participation, tenant profile, and the representativeness of responses. As long the requirements of this document (and other applicable requirements) are met, providers can use any standard collection method including face-to-face, telephone, postal, or email/online delivery and may use more than one collection method.

60 - Providers must be able to demonstrate a rationale for the survey collection methods chosen with reference to the size and characteristics of their relevant tenant population. In particular, survey collection methods where staff or contractors ask tenants the questions and report their response (e.g. face-to-face, telephone) can generate higher satisfaction scores than other methods. They may however be important to address barriers to participation, meet minimum sample size requirements, enable richer information to be generated, and for many small providers may be more cost-effective than setting up postal or online surveys.

Sampling method

61 - Providers must use a sampling method consistent with the requirement of this document, in particular that survey responses used to generate perception measures are representative of the tenant population as a whole. The following are examples of common sampling methods that are generally consistent with this requirement: computer-generated random sampling, systematic sampling, stratified sampling, or a census approach. In contrast, cluster sampling carries significant risks of generating a non-representative sample and must only be used where the provider has a strong practical rationale for adopting this method and can demonstrate that the risk is fully understood and addressed. Common sampling methods are described in Annex B.

Accessibility and barriers to responding

62 - Providers must take reasonable steps to assess, identify and remove barriers to certain groups of tenants participating in surveys used to generate the TSMs. In particular, this is in respect to tenants who share one or more protected characteristics under the Equality Act 2010, and in respect of duties of that Act. Barriers may include, but are not limited to, language barriers, visual impairment, literacy or lack of access to digital media ^{[footnote 30]}. Where necessary to overcome barriers to participation, it is permissible for surveys to be completed by a carer, another household member on behalf of a tenant or through an interpreter.

63 - Under exceptional circumstances, providers are permitted to remove some tenant households from the sample frame where there are significant capacity issues or health and safety risks that cannot be reasonably surmounted. Such capacity issues are those that are likely to prevent a meaningful response to the vast majority of the TSM questions set out in Table 2. Tenant households must not be removed where available methods to reduce barriers (e.g. face-to-face interviews or translations) could feasibly allow them to participate in the survey.

64 - In exceptional cases where a material number of tenants have been removed from the sample frame, providers must take an appropriate approach to seek the views of these tenants on relevant topics covered by tenant perception measures. This may include using ‘easy read’ versions of the TSM questions suitable for these tenants ^{[footnote 31]}. Providers must communicate a summary of results from these approaches in a manner they consider appropriate to support effective scrutiny by tenants of their landlord’s performance.

65 - Providers can choose to incentivise tenants to encourage survey completion (e.g. by including all survey participants in a prize draw). When considering the use of incentives, providers must consider the potential impacts on the representativeness of responses.

Annex A – Summary of requirements for small providers

66 - For the purposes of this document, small providers are those which own fewer than 1,000 dwelling units of relevant social housing stock ^{[footnote 32]}.

67 - All requirements set out in this document apply to small providers unless stated. This section summarises the provisions set out above that specifically apply to small providers. These recognise the practical barriers for small providers in conducting surveys to the same statistical accuracy and detail as larger providers, and the likelihood that many small providers will need to employ a census survey approach to meet the requirements of this document.

Survey frequency

68 - Small providers must carry out perception surveys at least once every two years. This reduced frequency recognises the greater relative costs and capacity constraints for small providers in conducting a census of all households and/or achieving relatively high response rates consistent with minimum levels of statistical accuracy. Small providers are permitted to collect and report annual TSMs according to a reporting year other than 1 April to 31 March.

Relevant tenant population

69 - Small providers must take a proportionate approach to defining the relevant tenant population depending on their stock profile. Small providers can choose to report tenant perception measures for households in LCRA, LCHO stock, or both stock types combined.

Sampling approach

70 - Small providers must, as far as possible, generate an estimate for overall satisfaction accurate to a margin of no more than +/-5% at a 95% confidence level. While this is a wider margin than for many large providers, it may still be practically difficult to achieve for many small providers. In this instance, small providers can choose to undertake a census. A small provider that undertakes a census will be considered to have met the requirement for statistical accuracy, irrespective of the number of responses they achieve.

71 - All providers must assess the extent to which the achieved sample is representative of the relevant tenant population. However, the scope of this assessment must be proportionate to the size and diversity of each provider’s tenant base and sampling approach. Small providers are only required to undertake a high-level assessment of representativeness of the achieved sample against the relevant tenant population, for example against a small number of key characteristics. They are not required to weight responses unless it is possible to generate a sample size large enough to meet the minimum statistical accuracy set out in Table 5 (a margin of +/-5% at a 95% confidence level) and there is strong evidence of a significant bias in estimated scores. In their published summary of approach, small providers are not required to include detail of the representativeness of the sample compared to the relevant tenant population (as illustrated in Table 4).

Confidentiality

72 - Small providers must take a proportionate approach to tenant perception measures in order to protect tenant confidentiality. For the smallest of these providers, it may be hard to protect the anonymity of individual tenants if tenant perception information is published in full. It is permissible for at least some tenant perception measures to be omitted from published information if small providers judge this to be a material risk.

Annex B – Description of common sampling methods

Computer-generated random sampling

73 - A list of all the relevant tenant population is produced and the required number of households to be surveyed are randomly selected, for example using a computer-generated random sampling.

Systematic sampling

74 - A list of all the relevant tenant population is produced and every nth household is selected for sampling. For example, a provider with 10,000 units decides to sample 2,000 properties ^{[footnote 33]}. For a required sample of 2,000 of 10,000 households, every fifth (10,000 / 2,000) household is selected for surveying.

Stratified sampling

75 - The relevant tenant population is divided into separate groups called strata based on characteristics of interest and then a proportionate sample is drawn from each group using random or systematic sampling. Strata may include, for example, stock types (general needs and supported housing), geography (schemes, local authorities) or tenant characteristics. For a provider with 10,000 units, of which 8,000 (80%) are general needs and 2,000 (20%) are supported housing, using stratified sampling to generate a proportionate sample of 2,000 responses across stock types would mean as far as possible generating a sample of 1,600 responses from general needs households (2,000 x 80%) and 400 in supported housing (2,000 x 20%).

Cluster sampling

76 - The relevant tenant population is divided into separate groups, called clusters, such as by street or postcode areas. A random sample of clusters is selected from the population, capturing a sufficient number of households to satisfy sample size requirements. There may be practical benefits for some survey methods (e.g. face-to-face), but this method relies on chosen clusters being sufficiently representative of the population as a whole.

Census approach

77 - An entire population of relevant tenant households is invited to participate in the perception survey, although typically not all invited to participate will respond. This is commonly used by small providers where it may be challenging to achieve a minimum sample size associated with standard levels of statistical accuracy.

Annex C – Illustrative sample sizes

78 - Providers must determine the sample size required to achieve the required levels of statistical accuracy. The table below provides an illustration of achieved sample sizes required to ensure statistical accuracy based on example population sizes, in order to give providers a sense of scale for what achieved sample sizes will be required. There are numerous free-to-use online calculators that can generate sample sizes from known population sizes using a standard formula consistent with the table below ^{[footnote 34]}.

79 - Indicative sample sizes in this table are consistent with an estimated satisfaction score of 50%. While a higher or lower satisfaction score would reduce the number of responses required, there is a risk that assumptions over average satisfaction before a survey has been conducted are inaccurate, especially as providers transition to the methodology set out in this document to generate satisfaction scores. Using an assumed 50% satisfaction score allows for a degree of contingency in planning survey activity.

80 - The indicative sample sizes in the next table assume no weighting of survey responses. Weighting will reduce the achieved sample size (a lower “effective sample size”) and may therefore require a higher number of responses. Estimation of the effective sample size may require expert input.

Annex table: Indicative sample sizes consistent with meeting minimum levels of statistical accuracy for example population sizes

Population (dwelling units/households)	Margin of error at 95% confidence level	Indicative achieved sample size
100	+/- 5%	80
250	+/- 5%	152
500	+/- 5%	218
750	+/- 5%	255
1,000	+/- 5%	278
1,500	+/- 5%	306
2,000	+/- 5%	323
2,500	+/- 4%	485
3,000	+/- 4%	501
4,000	+/- 4%	522
5,000	+/- 4%	536
7,500	+/- 4%	556
10,000	+/- 3%	965
15,000	+/- 3%	997
20,000	+/- 3%	1,014
25,000	+/- 2%	2,191
50,000	+/- 2%	2,291
75,000	+/- 2%	2,327
100,000	+/- 2%	2,345

1. Every reference of ‘registered providers’ and ‘providers’ in this document relates to registered providers of social housing. ↩

2. For the purposes of this document, a small provider is defined as one that owns fewer than 1,000 units of low cost rental accommodation and low cost home ownership dwelling units (combined). For providers in a group, dwelling units must be measured at a registered group level. ↩

3. Transactional surveys are triggered by a recent interaction (such as, by way of example only, completion of a repair, or response to a complaint). ↩

4. Questions TP02, TP03 and TP04 need to be completed by LCRA tenants only. If providers use a separate questionnaire for households residing in LCHO these questions must be removed. ↩

5. Where consecutive questions share the same response scale, it is permitted to: (a) set out the questions and response scale in a grid format within a written survey; or (b) state that the response scale is unchanged when reading consecutive questions as part of a telephone or face-to-face survey (however, if tenants ask to be reminded of the response scale, this must be repeated in full). ↩

6. Where providers use additional visual features such as emojis to overcome specific barriers, these features must be used alongside the standard response options, with the visual feature used (or emoji unicode) included as part of the published summary of approach. ↩

7. The sampling frame is the population from which a sample is drawn. As explained below (para 45 onwards), the calculated satisfaction scores must be representative of this tenant population. ↩

8. This means that a single household refers to the occupant(s) of each self-contained unit or each bedspace in non-self-contained housing. ↩

9. Or equivalently, for TP07 and TP12, one of the following responses: strongly agree, agree, neither agree nor disagree, disagree, strongly disagree. ↩

10. Apart from where TP02, TP03 and TP04 have been removed for households residing in LCHO units. ↩

11. Providers are permitted to survey tenants who started tenancies after fieldwork used to generate the TSMs has commenced; in particular this may apply when there is phased or rolling approach. ↩

12. This includes additional survey responses generating via ‘over-sampling’ to ensure minimum sample sizes for particular groups of tenants. Where the effect of any over-sampling is material, results must be weighted. ↩

13. That is, 1,000 or more dwelling units of low cost rental accommodation and low cost home ownership dwelling units (combined). For providers in a group, dwelling units must be measured at a registered group level. ↩

14. Providers with fewer than 1,000 dwelling units of relevant social housing stock must still report tenant perception TSMs in their first published set of TSMs. All providers must undertake the necessary perception surveys to allow them to report tenant perception TSMs for the first year that TSMs are to be reported. ↩

15. For example, explicit consent may be required if information on tenant characteristics is collected through the tenant perception survey in order to update a provider’s general data on each tenant household. ↩

16. Providers must ensure this rationale does not breach requirements on confidentiality. ↩

17. The sample of categories in this table is purely illustrative. As set out in paragraph 49, each provider must reach a balanced judgement as to which characteristics to include in their assessment of representativeness. This analysis is only required for providers with 1,000 or more dwelling units of relevant social housing stock. ↩

18. As the response rates for different questions will vary (for example based on those who choose not to respond to a particular question) this requirement applies specifically to estimated overall satisfaction. ↩

19. Strictly speaking, this requires a sample to be random and representative. ↩

20. There are numerous free-to-use online calculators that generate sample sizes from known population sizes using a standard equation as described in Annex C. ↩

21. For example, a provider with 10,000 LCRA units and 2,500 LCHO units is likely to require total responses for overall satisfaction from approximately 965 LCRA and 485 LCHO tenant households respectively in order to meet the minimum level of statistical accuracy. ↩

22. Total responses to the tenant perception survey is defined in this document as the total number of surveyed households that have provided one of the following responses – Very satisfied, Fairly satisfied, Neither satisfied nor dissatisfied, Fairly dissatisfied or Very dissatisfied (or the equivalent agreement responses) - to at least one of the tenant perception survey questions. ↩

23. The estimated satisfaction score - and any weighting requirements - are also unknown at the outset. See Annex C for more information. ↩

24. That is, 1,000 or more dwelling units of low cost rental accommodation and low cost home ownership dwelling units (combined). ↩

25. Equivalently, this means that providers must ensure that as far as possible calculated perception measures are unbiased estimates of the satisfaction score for the relevant tenant population. ↩

26. Providers with fewer than 1,000 units are not generally required to weight responses (see paragraph 55). ↩

27. That is, providers that own fewer than 1,000 units of low cost rental accommodation and low cost home ownership combined at a registered group level. ↩

28. Within the general principle outlined in this paragraph, providers should take a proportionate approach to technical issues such as factoring in vacant dwelling units (e.g. in general it is permissible to use total stock owned as a proxy for the size of the relevant tenant population), determining the precise approach to the point in time tenant characteristics are measured, or applying particular characteristics to households (e.g. where they may be a mix of characteristics within a particular household). ↩

29. With respect to retaining any personal data, providers must ensure that they and any contractors are compliant with the relevant privacy and data protection legislation. ↩

30. A form of the specified question wording must be used in all surveys that generate the TSMs. Where required to overcome specific barriers this question wording may be represented in braille, translated into another language (verbally or in writing) or supplemented with emojis. ↩

31. Any data derived from surveys that use different versions of the TSM question wording must not be included in the calculation of perception TSMs. ↩

32. That is, providers that own fewer than 1,000 units of low cost rental accommodation and low cost home ownership combined. For providers in a group, units are measured at a registered group level. ↩

33. A provider with 10,000 units would need 965 responses to achieve minimum levels of statistical accuracy. Sampling 2,000 properties means they expect a response rate greater than 48%. ↩

34. For the avoidance of doubt, the standard equation for calculating the sample size (n) referred to in these requirements is as follows: [z2 * p(1-p)] / e2 / 1 + [z2 * p(1-p)] / e2 * N] , where z is the Z-score related to the confidence level (1.96 for 95%), P is the estimated satisfaction score, N is the relevant number of units held by the provider and ε is the margin of error for the population size shown in the table. This equation assumes there is no weighting. ↩