Research and analysis

Comparability between the People and Nature Survey and the Monitor of Engagement with the Natural Environment Report

Published 11 September 2023

Applies to England

An assessment of the comparability between the People and Nature Survey (PaNS) and the Monitor of Engagement with the Natural Environment (MENE) considering sample design, survey mode and questionnaire changes and the impact for users of the statistics.

1. Key Findings

There are significant differences between the People and Nature Survey (PaNS) and its predecessor, the Monitor of Engagement with the Natural Environment (MENE).

The key differences between the two surveys are:

• Sample design – PaNS uses a smaller sample size than MENE and respondents are selected using a panel of individuals who have agreed to respond to surveys. MENE sampled from the whole English population using probability sampling methods;

• Survey mode – PaNS is conducted as an online survey where MENE was conducted using face-to-face interviews; and

• Questionnaire design – Significant differences exist between the MENE and PaNS questionnaires in questionnaire design, question wording and response categories. As a result, no identical questions exist in both PaNS and MENE.

• Timing – MENE was conducted from 2009/10 to 2018/19 while PaNS began in 2020/21. No data were collected by either survey for 2019/20.

The extent of these differences means that direct comparisons cannot be made between the two surveys. However, for some key topics, it can be appropriate to compare high-level trends observed in MENE with those observed in PaNS. For example, it is appropriate to say that the proportion of individuals who reported that they had made a recent visit to a green or natural space tended to increase over the period 2009/10 to 2018/19 (MENE) but tended to decrease between 2020/21 and 2021/22 (PaNS). This type of comparison is only possible where individual question wording and definitions are broadly similar between the two surveys.

Calculation of percentage changes between the two surveys, for example comparing the percentage of individuals who reported visits to green or natural spaces between 2009/10 (MENE) and 2021/22 (PaNS) is not possible for any variable.

This report provides more detail on the differences between the two surveys, the impact of these differences and further information about the types of comparisons that can be made. A summary of the key findings of this report is shown in Figure 1 and a summary of recommendations for the comparison of the two surveys is found in Figure 2.

Table 1. Summary of comparability between PaNS and MENE

Method Change	MENE	PaNS	Impact
Sample Size	Approximately 46,000 individuals each year	Approximately 25,000 individuals each year	Reduction in sample size will tend to lead to larger confidence intervals around estimates, but otherwise does not affect comparability.
Sample Selection	Probability sample - all English households have chance of selection	Non-probability sample - “opt in” panel survey, quota sampling	Change from probability to non-probability sample is likely to have introduced some bias, limiting comparability, although the impact of this appears small.
Survey Mode	Face-to-face household survey	Online survey	Change in survey mode is likely to have caused a discontinuity between the two survey which may vary between questions and respondents and prevent comparability.
Questions	Consistent questionnaire over the 10 years of MENE survey	New questionnaire introduced for PaNS	No identical questions exist in both MENE and PaNS. Even small changes to questions can have significant impacts so this is a big factor preventing direct comparisons between the two surveys.
Timing	MENE ran from 2009/10 to 2018/19	PaNS was introduced in 2020/21, a year after MENE finished	The year break between the two surveys, and the fact that the introduction of PaNS coincided with the onset of the COVID-19 pandemic makes it very difficult to assess the impact of changes to survey mode and questions.

Table 2. Key recommendations for the comparison of PaNS and MENE

Do	Don’t
Use surveys independently	Make direct comparisons between measures reported by the two surveys. For example, it is not appropriate to report a percentage change from 2018/19 (MENE) to 2021/22 (PaNS).
Compare key trends. For example, ‘data from PaNS shows an increasing trend, continuing the trend observed through MENE’.

2. Background to PaNS and MENE

he People and Nature Survey (PaNS) has been delivered by Natural England in its current format since 2020 and forms part of Natural England’s statutory function to inform evidence-based policy decisions on English residents’ (both adults and children) engagement with the natural environment and assess the impact of such engagement and changes through time.

PaNS replaced the Monitor of Engagement with the Natural Environment (MENE) survey which ran from 2009/10 to 2018/19. In 2017, a strategic review of MENE was conducted to consider ways to ensure the survey continued to meet both current and emerging user needs and how to respond to falling response rates and increasing costs. The recommendations of this review ultimately led to the development of PaNS which was conducted for the first time in April 2020.

Both survey design and methodology differ significantly between the two surveys. While a parallel run of the two surveys was planned to quantify the impact of these changes, this was not possible due to the COVID-19 pandemic which prevented the face-to-face interviews required by MENE. In addition to the methodological differences, no data was collected by either survey for 2019/20.

3. Impact of Differences in Sample Size and Design on Comparability

MENE was based upon a sample of approximately 45,000 households per year, selected using a Probability Proportionate to Size (PPS) sample of English households from the Postcode Address File (PAF). This is the approach traditionally used for household surveys. It means that every household has a chance of being included in the survey and it minimises bias.

PaNS is based on a quota sample of approximately 25,000 individuals per year, selected from an online panel of individuals who have opted-in to join the panel. The use of a panel means that not every individual resident in England has a chance of being selected. It is common for such panels to under-represent particular age or ethnic groups. Quotas for age and gender (combined), English region, ethnicity and education level were used to try to minimise such bias.

The samples achieved by PaNS and MENE were analysed to identify any biases in age, gender and ethnicity. There was no indication of significant bias in either of the surveys and the differences between the surveys were not significant. This analysis is only indicative since the data collected on age, gender and ethnicity differs between the two surveys and the surveys relate to different time periods. There are other potential sources of bias between the two samples, for example characteristics such as socioeconomic or working status, but this was not possible to analyse with the data available.

No sample is likely to achieve the exact demographic breakdown shown by the entire population, primarily due to sampling error and non-response bias. Weights are applied to survey data to correct for this and to bring the sample more in line with the target population. Weights are applied to both MENE and PaNS data but, due to the differences in sampling methodology, the method used varies between the two surveys.

The sample size used for PaNS is substantially smaller than that used for MENE. When considered alone, reducing the sample size will reduce the accuracy of the estimates produced. This will produce estimates with wider confidence intervals, but it won’t directly affect comparability between the two surveys. The impact of this sample size reduction was considered during the development of PaNS and deemed to be minimal – with 95% confidence intervals increasing from ±0.6% to ±0.7% for national respondent-level estimates and from ±0.8% to ±1% for national visit-level estimates.

While changes in sample design can cause a large discontinuity, this analysis of the PaNS and MENE samples suggests that the impact of the change is relatively small in this case.

4. Impact of Differences in Survey Mode on Comparability

MENE was conducted as a face-to-face survey where trained interviewers visited households and interviewed respondents directly. This survey mode has the advantages that it is accessible to all (for example not only those with internet access) and that the interviewers can provide prompts and additional explanation to respondents to help gain responses. However, it is the most expensive survey mode available and response rates are often very low.

PaNS is conducted entirely as an online survey. This has the advantage of being relatively cheap to administer and results are typically available more quickly compared with a face-to-face survey. However, the main limitation is that responses are only possible from individuals with access to the internet.

While the majority of households do now have an internet connection, internet use is known to vary by age and ethnicity. For example, according to ONS statistics on internet users%20in%202019) for 2020, at least 95% of individuals in the UK aged 16 to 64 were estimated to have used the internet in the last three months. This fell to around 85% for those aged 65-74 years and 54% for those aged 75 years and over. Similarly, while 92% of individuals aged 16 years and over had used the internet in the last three months overall, this fell to 88% for individuals from a Bangladeshi ethnic group. Differences in internet use may result in bias when using an online-only survey mode and, due to the broad age and ethnicity categories currently published for PaNS, may not be fully detected by the sample analysis described above. In addition, since individuals who make up the panel have chosen to be part of it, this could lead to self-selection bias, where the demographics or behaviours of panel members can differ from the wider English population.

A change in survey mode can also affect the way in which respondents answer questions which will have an impact on the results. For example, while a trained interviewer can provide additional prompts and clarification to encourage individuals to provide an answer, respondents may provide less detailed answers in an online survey. Also, where questions are more sensitive or ask for a personal opinion, respondents may be more honest in their answers when providing information online.

The impact of a change in survey mode may vary between question types or respondents. Without conducting a parallel run or additional studies, it cannot be known with any certainty whether the change in mode has had an impact on comparability or the likely extent of this impact.

As a result, it is advisable to assume that the change in mode has caused a discontinuity between the two surveys.

5. Impact of Differences in Questionnaire Design on Comparability

The questionnaire used for PaNS is substantially different from that used for MENE. Changes were made both to accommodate the change in survey mode, from face-to-face to online, and to improve question clarity so that it could be better understood by respondents. Extensive cognitive testing and pilot exercises were carried out to ensure the PaNS questionnaire collects the data required in the best possible way. However, the way in which a question is worded, as well as its placement within a questionnaire, is known to have an impact on the responses provided.

No single question included in the PaNS questionnaire is unchanged from its equivalent question in MENE. For some questions, alterations have been made to question wording, and for others changes have been made to the possible response categories given to respondents. In some cases, both question wording and response categories have been changed. This means that no single variable has completely comparable questions in both MENE and PaNS.

Questionnaire changes represent the biggest difference between PaNS and MENE. Without equivalent questions in both surveys, there is no way that direct comparisons of estimates can be made.

6. Impact of Survey Timing on Comparability

MENE was conducted until the end of March 2019 (covering the 2018/19 financial year) while PaNS began in April 2020 (covering the 2020/21 financial year). No data were collected by either survey for 2019/20.

The COVID-19 pandemic, which was declared in March 2020, coincided with the launch of PaNS. The pandemic prevented the planned parallel run – where both surveys are conducted for the same point in time to assess the impact of the change – from taking place. Furthermore, the pandemic itself will have had a significant impact on the way that individuals engaged with the natural environment. Without a parallel run, it is impossible to determine whether observed changes in the data arise from changes to the survey or reflect actual behaviour changes.

This timing issue compounds the challenges of making any comparisons of data from the two surveys.

7. Comparing Key Variables

Due to the discontinuity caused by differences in sampling methodology, survey mode and questionnaire design, direct comparisons between PaNS and MENE are not possible for any variable. Furthermore, no data are available the 2019/20 financial year. However, broad statements to compare the high-level trends observed by the two surveys may be possible in some cases for both the whole population and for most sub-groups of the population.

Comparisons of trends are only possible where questions are broadly similar in terms of their wording, definitions, structure and the response options given to respondents. Comparing trends is not recommended for estimates for individuals aged 65 years and over as a separate sub-group. This is because the change in sampling method and survey mode is likely to have introduced the most bias for this age group.

Care must be taken when considering the trends seen by PaNS and making comparisons with MENE. This is because the PaNS series available is currently very short and is likely to have been influenced by the COVID-19 pandemic.

For key topics that are most important to users of PaNS, a summary of questionnaire differences between PaNS and MENE, and whether it is appropriate to compare trends is shown in table 1. Example statements are included in table 2.

Table 3. Comparison of questionnaire differences between PaNS and MENE for key variables.

Variable	PaNS question number	Differences between PaNS and MENE questionnaire				Comparison of trends possible?
		Question wording	Definitions	Structure	Response options
Percentage of individuals who visited a green or natural space in the last 7/14 days	Q6	Minor	Major	None	N/A	Yes – only where consistent definitions used (see below)
Frequency of visits to a green or natural space in the last 12 months	M1 Q1	Minor	None	None	Minor	Yes
Health and wellbeing benefit	M2A Q9	Minor	Minor	Minor	None	Yes
Activities to protect the environment	M4 Q11	Major	Major	Major	None	No
Concern about damage to the natural environment	M4 Q4	None	None	Minor	Minor	Yes
Green spaces	M1 Q5	Minor	None	Minor	Minor	Yes

Table 4. Example statements covering high-level trends for key variables where it is appropriate to make comparisons.

Variable	Example statement
Percentage of individuals who visited a green or natural space in the last 7/14 days	The proportion of individuals who reported that they had made a recent visit to a green or natural space tended to increase over the period 2009/10 to 2018/19 (MENE) but tended to decrease between 2020/21 and 2021/22 (PaNS).
Frequency of visits to a green or natural space in the last 12 months	The proportion of individuals who reported that they spent time outdoors at least once a week tended to increase over the period 2009/10 to 2018/19 (MENE) but tended to decrease between 2020/21 and 2021/22 (PaNS).
Health and wellbeing benefit	The proportion of individuals who agreed that an outdoor visit made them feel closer to nature was broadly unchanged between 2009/10 and 2017/18 but increased slightly between 2017/18 and 2018/19. The proportion who agreed that a visit made them feel closer to nature also increased over the period 2020/21 to 2021/22.
Concern about damage to the natural environment	There was very little change overall in the proportion of individuals who agreed that they were concerned about damage to the natural environment between 2009/10 and 2018/19. This trend continued between 2020/21 and 2021/22.

Notes
1. No data are available for the 2019/20 financial year
2. Only short time-series is currently available for PaNS with trends likely to be influenced by the COVID-19 pandemic

As an example, this section outlines key trends across PaNS and MENE for individuals making visits to green or natural space. The percentage of individuals who made a recent visit to a green or natural space is a key variable for many users of PaNS, and has been included in the ONS Measures of National Wellbeing dashboard. In MENE, ‘recent’ was defined as the last 7 days but in PaNS this recall period was extended to the last 14 days. This definition change would prevent comparison of trends between the two surveys. However, PaNS also collects data using a 7-day recall period which does allow comparisons of trends to be made. The comparison of this variable, using the consistent 7-day recall period is shown in Figure 1.

It is important that any graphical representations of key trends covering both MENE and PaNS data indicate clearly that there is a break in the series between MENE and PaNS so that direct comparisons cannot be made. This can be indicated, for example, by including a vertical dotted line for the year 2019/20.

Figure 1. Percentage of individuals (16 years plus) who visited a green or natural space in the last 7 days (MENE: 2009/10-2018/19, PaNS: 2020/21-2021/22)