Guidance

Community and Engagement Survey 2025/26: Contact Strategy Trial Report

Updated 2 July 2026

Applies to England

1. Abstract

This report summarises the findings from a trial of different contact strategies within the context of the new DCMS Community and Engagement Survey: a large scale ‘ABOS’ (push-to-web) survey from an address sample frame.

The report quantifies the additive effect of a second reminder over just one as well as the impact of including paper questionnaires in the final reminder pack. The scale of the trial also allows some analysis of the demographic profile yielded by each contact strategy. This suggests that a relatively low-cost strategy can work as well or better than a high-cost strategy.

The report includes some consideration of the potential for mixed contact strategies, differing between sample strata, and then concludes with a short review of the effect of contact strategy on substantive data profiles.

2. The Community and Engagement Survey

The Community and Engagement Survey (CES) is a new source of official statistics in England, commissioned by the Department for Culture, Media and Sport (DCMS). It brings together the content of the Community Life Survey (2012-25) and the Participation Survey (2021-25) into a single study with two ‘pathways’, one for each of the progenitor surveys (‘Community’ and ‘Engagement’ respectively). CES respondents are allocated at random to one of these two pathways.

The CES is an ‘ABOS’ survey: a push-to-web survey from an address sample frame, with a secondary paper questionnaire mode. All individuals aged 16+ resident at the sampled address may take part but the number of web survey logins is constrained between two and four with the number determined by auxiliary data attached to each address (the mean is 2.8).

The CES was launched in October 2025, with a target respondent sample size of 191,500 (175,000 directed down the Community pathway and the remaining 16,500 directed down the Engagement pathway). This total had to be achieved by the end of March 2026. As well as the overall targets, DCMS set a sub-target for each of the 296 Local Authorities in England.

The survey was managed via four sample releases, two in the first quarter (October to December 2025) and two in the second quarter (January to March 2026). These sample releases varied in scale and geographic profile in an effort to get as close as possible to each sub-target by the close of fieldwork. In the event, 192,244 completed questionnaires passed the quality control filters, and three quarters of the Local Authority level totals were within 90-110% of the relevant sub-target.

3. The contact strategy trial design

For the first quarter (covering the first two sample releases), Verian designed a contact strategy trial so that the additive value of each component of that strategy could be measured. These components were: (i) the addition of a second reminder letter, and (ii) the inclusion of up to two paper questionnaires^{[footnote 1]} (and a pre-paid return envelope) in the final reminder. That meant there were four arms to the trial, labelled ‘WW’, ‘WP’, ‘WWW’ and ‘WWP’:

WW: invitation and one reminder, no paper questionnaires included in any pack

WP: invitation and one reminder, up to two paper questionnaires included in reminder

WWW: invitation and two reminders, no paper questionnaires included in any pack

WWP: invitation and two reminders, up to two paper questionnaires included in second reminder

Otherwise, the survey participation instructions were the same and the incentive was uniform (a £10 voucher from a selection provided by Merit, conditional on completing the questionnaire)^{[footnote 2]}.

Because Verian held substantial information from the Community Life and Participation Surveys about how each contact strategy was likely to work, the allocation of addresses to each arm of the trial was probabilistic but not uniform across the sample. This approach was a compromise between the ideal trial design – equal probability allocation to each trial arm - and an ‘optimised’ design in which contact strategy allocation is selective in an effort to minimise the variation in response rates between strata within a set budget.

In total, 322,092 addresses were issued in the first quarter and included in the trial. The per-strategy totals ranged from 14,038 (‘WWP’) to 152,340 (‘WW’). Design weights have been applied to the data to ensure the results reflect what would have happened had (i) the address sample been nationally representative, and (ii) allocation to contact strategy been unrelated to address characteristics.

4. Trial Results

4.1 Return rates

Table 1 shows the mean number of completed questionnaires per sampled address per contact strategy.

First, it shows that the addition of a second push-to-web reminder increased the mean number of returns by 0.05, a relative uplift of +20% (‘WWW’ v ‘WW’).

Second, it also shows that including two paper questionnaires in the final reminder increased the return rate by 0.05-0.06 (a relative uplift of +25% if this was the first reminder (‘WP’ v ‘WW’); +19% if it was the second (‘WWP’ v ‘WWW’)).

Third, it shows that the inclusion of paper questionnaires in the final reminder displaces some web returns, replacing them with paper returns. This effect was strongest if the paper questionnaires were included in the first reminder (‘WP’ v ‘WW’): 14% of web returns were displaced, compared to just 4% if the paper questionnaires were included in a second reminder (‘WWP’ v ‘WWW’). Paper returns tend to have more missing data than web returns so the displacement of the latter with the former is sub-optimal.

Table 1: Mean number of returns by mode, by contact

Demographic variable (number of categories)	WW	WP	WWW	WWP
Web returns	0.23	0.19	0.27	0.26
Paper returns	0.01	0.09	0.01	0.06
Total returns	0.23	0.28	0.27	0.32

The great scale of the trial means that these results can be disaggregated by any sample frame variable that informed trial arm allocation.

One of these variables was the local English index of multiple deprivation (discretised into five classes based on national quintile points). Return rates varied significantly between these five classes: for example, under the ‘WW’ contact strategy, the returns rate ranged from 0.16 in the most deprived class to 0.29 in the least deprived class. However, the relative effects of each intervention did not differ much between the classes.  The relative effect of adding a second push-to-web reminder (‘WWW’ v ‘WW’) ranged from +16% to +24%; the relative effect of including paper questionnaires in the final reminder (‘WP’ v ‘WW’ averaged with ‘WWP’ v ‘WWW’) varied a little more, ranging from +12% to +32%. However, this variation was not correlated with deprivation class order.

A more discriminating sample variable was one based on imputed address-level data supplied to Verian by CACI. This variable indicated whether anyone living at the address was expected to be aged 65+. Including paper questionnaires in the final reminder had a much greater impact on the return rate at these addresses than on others. The averaged relative uplift was +38% compared to +16% at other addresses. The displacement effect was also stronger: 15% of web responses were replaced with paper responses compared to 7% at other addresses^{[footnote 3]}.

The trial results allowed Verian to fit an expected return rate for each of the four contact strategies for every address sampled for the second quarter of the CES. Various rules were applied to ensure cost control but, within these constraints, Verian was able to allocate each sampled address to an ‘optimal’ contact strategy, designed to minimise the variation in return rates between strata but also achieve a mean return rate of approximately 0.25 per sampled address.

However, an optimal design that is focused entirely on return rates risks ecological fallacy. Minimising the variation in return rates between sample strata does not necessarily mean that the variation in response rates between different types of people is minimised. It probably helps, but by how much?

4.2 Demographic profiles

We can begin to evaluate this by comparing the demographic profiles yielded by each contact strategy against the benchmarks used to weight the CES data. For the first quarter, these were derived from the July-September 2025 Labour Force Survey.

This type of nonresponse bias evaluation is limited to those survey variables that have measurement equivalence with available population benchmarks. It provides a useful but inevitably partial view.

For each contact strategy, Verian has estimated a systematic error score for each of nine demographic variables included in the CES weighting matrix. The smaller the score the better (see footnote for details)^{[footnote 4]}. 

Table 2 reveals that the best demographic profile is produced by the lowest cost contact strategy ‘WW’: the mean systematic error score across the nine demographic variables was 3.9%pts. The error level was slightly higher under ‘WWW’ (mean systematic error score = 4.4%pts) meaning that the additional push-to-web reminder raised the response rate without obviously reducing non-response bias. The systematic error score means were higher for the two strategies that included proactive provision of paper questionnaires: 6.3%pts for ‘WP’ and 5.0%pts for ‘WWP’. These were the most expensive strategies but yielded the worst profiles overall.

Table 2: Demographic profile systematic error scores, by contact strategy

Demographic variable (number of categories)	WW	WP	WWW	WWP
Age group (7)	4.6%pts	7.8%pts	4.8%pts	6.2%pts
Sex (2)	3.1%pts	4.1%pts	3.9%pts	4.3%pts
Housing tenure (3)	9.9%pts	14.9%pts	10.6%pts	11.2%pts
Education level (5)	4.7%pts	4.0%pts	5.5%pts	4.5%pts
Number of co-resident adults (3)	2.0%pts	5.8%pts	2.6%pts	3.5%pts
Whether co-resident with u16s (2)	1.4%pts	5.1%pts	1.1%pts	4.3%pts
Ethnic group (5)	3.6%pts	5.6%pts	4.0%pts	5.0%pts
Whether use the internet (3)	4.1%pts	2.4%pts	3.7%pts	2.8%pts
Whether in work (2)	1.6%pts	7.1%pts	3.4%pts	3.4%pts
AVERAGE	3.9%pts	6.3%pts	4.4%pts	5.0%pts

One question is whether the best contact strategy differs between strata, meaning that the optimal strategy might be a mixed strategy (different contact strategies for different strata). The principal difficulty in evaluating this is generating a benchmark for each stratum because these are not necessarily available externally. For this work, Verian has used a reasonably logical approach to generate the benchmarks (see footnote) but it is certainly not the only one available^{[footnote 5]}. 

Research designers may stratify samples differently depending on their measurement objectives. Consequently, the results presented here are for illustrative purposes only, treating each of the five classes based on the index of multiple deprivation as potential sample strata. Table 3 shows the stratum-level mean systematic error scores for each of the four contact strategies.

The results suggest that the best strategy is a mixed one, with ‘WWW’ best in the most deprived fifth of the country, ‘WW’ best in the least deprived fifth of the country, and ‘WWP’ best in the middle three fifths of the country. This would yield an overall mean systematic error score of 2.9%pts, lower than even the best uniform strategy (‘WW’ = 3.9%pts).

Table 3: Mean systematic error scores per deprivation stratum, by contact strategy

Stratum	WW	WP	WWW	WWP
Most deprived 1/5	3.3%pts	5.0%pts	3.0%pts	4.6%pts
Class 2	3.9%pts	5.1%pts	3.6%pts	3.5%pts
Class 3	4.0%pts	6.7%pts	5.3%pts	2.6%pts
Class 4	3.6%pts	5.9%pts	3.2%pts	2.5%pts
Least deprived 1/5	2.4%pts	3.7%pts	3.6%pts	6.3%pts

However, even with a trial of this scale, there are some fairly wide margins of error around these stratum-level error scores. The ranks for each contact strategy are not particularly stable and the benchmarks themselves are quasi-benchmarks at best. This might make research designers want to default back to the simpler strategy of minimising response rate differences between strata, irrespective of the risk of ecological fallacy.

If this approach was taken, the selection would be somewhat different, favouring the highest response rate strategy (‘WWP’) in the most deprived fifth of the country, the lowest response rate strategy (‘WW’) in the least deprived three classes, and either ‘WP’ or ‘WWW’ in class 2 areas.

In practice, the CES design for the second quarter (January to March 2026) was implemented in the spirit of this latter approach, albeit with several additional subtleties. In particular, the ‘WP’ and ‘WWP’ designs were almost – but not quite – restricted to addresses coded by CACI as containing someone aged 65+. These were the addresses most likely to yield paper returns if paper questionnaires were included in a reminder pack, so it made economic sense to focus resources this way.

4.3 Substantive variables

The analysis above focuses on the stratum-level response rates and the demographic profiles obtained by each contact strategy. That is mainly because good benchmark data is available to evaluate the samples in these respects. In contrast, the effect of contact strategy on the distribution of substantive variables can be measured but it is not obvious which strategy produces the most accurate sample profile^{[footnote 6]}. 

To explore this, six important variables from the Community pathway were selected for study: (i) strength of ‘belonging’ to the local neighbourhood; (ii) whether engaged in formal volunteering at any point in the last twelve months; (iii) whether engaged in informal volunteering over the same period; (iv) whether personally involved in helping out on a local ‘issue’ in the last twelve months; (v) level of pride in the local area, and (vi) frequency of feeling lonely.

The only statistically significant difference (p=.014) was over the level of pride in the local area: 65% of respondents from the ‘WW’, ‘WWW’ and ‘WWP’ strategies expressed pride in the local area compared to 69% of respondents from the ‘WP’ strategy. That was it. No other difference exceeded three percentage points and none was statistically significant, despite the very large sample sizes per contact strategy.

In short, while different contact strategies produce different response rates and demographic profiles, the effect on substantive profiles – at least at the national level – may have been rather weaker.

5. Conclusions

The Community and Engagement Survey contact strategy trial is remarkable for its scale, allowing direct comparison of four different contact strategies with respect to both response rates and demographic profiles. It is even possible to review performance at a subnational level.

It yielded some counter-intuitive results. Principal among these is that the lowest cost contact strategy – one invitation and one reminder, no paper questionnaires included in either pack (‘WW’) – produced a better demographic profile than the higher cost strategies. Furthermore, the trial results suggest that a mixed contact strategy, differing between strata, might be better than even the best uniform strategy.

Finally, at a national level at least, the substantive data did not vary much as a function of contact strategy, or at least not so much as the demographic data.

There are, inevitably, caveats to place around these findings.

First, although the scale of the trial is exceptional, the highly variable allocation of addresses to trial arm makes the results a little less certain than ideal, especially at the subnational level.

Second, it is not obvious whether the research designer should aim for the highest response rate, the least variable response rate (between sample strata), or the best demographic profile. All three evaluation criteria are considered in this report and the logical recommendation that follows is dependent on how the criteria are ranked.

Third, although the substantive data produced by each contact strategy looked remarkably similar, this analysis was limited to a small number of variables (six) and to the national level data. It does not rule out rather stronger effects for other variables or for some population subgroups.

Fourth, the findings are context-specific (topic, sponsor, length, incentive offer etc.). Although there is a lot of scope for researchers to use this data to help develop their survey designs, we should not expect a single ‘best practice’ to emerge as a consequence. The optimal design for a new survey is rarely lifted straight off the shelf.

1. A maximum of two; where prior returns suggested only one non-responding eligible individual, just one questionnaire was included. The realised mean number of paper questionnaires per pack was 1.9. ↩

2. A Love2Shop voucher was offered to those completing on paper. ↩

3. For both statistics, the relative effects of ‘WP’ v ‘WW’ and ‘WWP’ v ‘WWW’ have been averaged for simpler presentation. ↩

4. To derive this, Verian estimated a D score for each category of the variable (the square of the observed error against the benchmark) and a V score (the square of the estimated standard error). The D and V values were summed for all categories of the variable (∑D and ∑V). The systematic error score is equal to the square root of (0.5∑D - 0.5∑V) but with a minimum value of zero. The 0.5 multiplier is included to limit the effects of ‘double-counting’ errors. ↩

5. Verian calibrated each of the four samples (one per contact strategy) to the national benchmarks, filtered each sample by stratum, and then treated the weighted proportions as stratum-level quasi-benchmarks. To smooth out the potential effects of contact strategy on the generation of these benchmarks, Verian averaged the four benchmark values to create a single quasi-benchmark for each category of each of the nine demographic variables. ↩

6. Small measurement differences between web and paper questionnaires may also mask or, alternatively, exaggerate profile differences. ↩