Guidance

Participation Survey 2023/24: Pilot Report

Updated 9 November 2023

Applies to England

1. Introduction

In 2021 , the Department for Digital, Culture, Media and Sport (DCMS)^{[footnote 1]} commissioned Kantar Public to design and deliver a new, nationally representative ‘push-to-web’ survey to assess adult participation in DCMS sectors across England. The survey served as a successor to the Taking Part Survey), which ran for 16 years as a continuous face to face survey.

The 2023/24 Participation Survey was commissioned by DCMS in partnership with Arts Council England (ACE). The scope of the survey is to deliver a nationally representative sample of adults (aged 16 years and over) and to assess adult participation in DCMS sectors across England, targeting enough households to allow for Local Authority representation of the data. The data collection model for the Participation Survey is based on ABOS (Address-Based Online Surveying), a type of ‘push-to-web’ survey method. Respondents take part either online or by completing a paper questionnaire. In 2023/24 the target respondent sample size increased to 175,000 – which was previously 33,000 per survey year in the interim survey from 2021 to 2023. Fieldwork will run across four quarters (May-June 2023, July-September 2023, October-December 2023 and January-March 2024).

Survey objectives:

• To inform and monitor government policy and programmes in DCMS, ACE and other government departments (OGDs) on adult engagement with the DCMS and digital sectors. The survey will also gather information on demographics (for example, age, gender, education).
• To assess the variation in engagement with cultural activities across DCMS sectors in England, and the differences in social-demographics such as location, age and education,
• To monitor and report on progress in achieving the Outcomes set out in Let’s Create^{[footnote 2]} – Creative People, Cultural Communities, and A Creative and Cultural Country (as set out in the Arts Council England Impact Framework).

2. Questionnaire development

The online questionnaire was designed to take an average of 30 minutes to complete. A modular design was used with around half of the questionnaire made up of a core set of questions asked of the full sample. The remaining questions were split into three separate modules, randomly allocated to a subset of the sample.

The postal version of the questionnaire included the same set of core questions asked online, but the modular questions were omitted to avoid overly burdening respondents who complete the survey on paper, and to encourage response. Copies of the online and paper questionnaires are available online.

Given the extent of questionnaire changes in 2023/24 Participation Survey, it was important to implement a comprehensive development and testing phase. This was made up of three key stages:

2.1 Questionnaire review

As a result of the DCMS and ACE partnership, we have a more granular dataset on adult engagement in DCMS sectors. This has meant that there have been some new questions and changes to existing questions, response options and definitions in the 2023/24 questionnaire compared to the 2021/22 or 2022/23 survey. The questionnaire for 2023/24 has been developed collaboratively to adapt to the needs and interests of both DCMS and ACE.

Questionnaire changes

Questions on the following cultural topics of interest were added to the Participation Survey 2023/24, as requested by ACE and/or DCMS:

• Environment, which included questions on mode of transport taken while travelling to an arts and cultural event, distance travelled, and reason(s) for transportation choice.
• Social prescribing, which included questions on the respondent’s experience with social prescribing, and the types of activities they were referred to.
• Further questions on arts and culture engagement, which included questions on the types of classes and clubs respondents have taken part in, the frequency and reasons(s) for their involvement, the impact/benefits of participating, and for non-participants, the reason for not participating.
• Pride in Place, which included questions on respondents’ sense of belonging and pride of their local area, the role culture plays in choosing where to live, and the current arts and culture scene in their local area.

The following changes to the digital questions were also made to the Participation Survey 2023/24:

• Smart devices: New devices listed, response method changed to collect the number of devices owned and new questions added on whether respondents considered security features when purchasing said devices.
• Digital skills training: New question asked to see how confident respondents were with digital skills and devices, response options changed to differentiate whether respondents did training as part of work or in their own time and new question added on whether this training resulted in an academic qualification.
• A new concept for digital identity was introduced, followed by a short list of factors one might consider when choosing a company or agency to process their digital identity, and the respondent was asked to rank the importance of each.
• A new question was added to ask how respondents react to cookie banners.
• In these sections, online safety and security, 5G, comfort around use of data, there were adjustments made to questions & response options to make statements easier to scan, shorter, and neutral.

2.2 Cognitive testing

Cognitive testing explores how participants understand, process, and respond to survey questions. Two stages of online cognitive testing were conducted, the first round took place in January 2023 and the second round in February 2023. A total of 18 participants, comprised of diverse genders, ages, highest educational qualifications and ethnicities, were a part of the cognitive testing.

In the first round of testing, participants enjoyed the questions about arts and culture activities, historic places visited and their local area. The questions towards the end of the interview about digital skills and data were not as salient and there was a concern that this could lead to participants satisficing^{[footnote 3]}. Participants regularly reported there was too much to read on the screen, especially where examples and bracketed text were used. Another finding was some questions were perceived as repetitive where a similar list of options was used. Also, where there were many statements in agree/disagree batteries, there was the risk of participants straight-lining without engaging with the text properly due to the onerousness nature of the task.

The aim of the second round of cognitive testing was to address the changes made following the first round. Where wording had been simplified and amount of text had been reduced (for example removing brackets and complex language) this meant questions were easier to answer and participants were more willing to read through all the answer codes. Moreover, where examples were provided, the question was more meaningful than the more conceptual/theoretical version tested in round one.

Further findings from cognitive testing are outlined in Annex A.

2.3 Usability testing

The primary focus of these interviews was to explore the usability of the paper questionnaire, presented as an A4 booklet, that is sent out with the second reminder letter. How participants approached the booklet and answered these questions was explored in detail. On the 22nd and 23rd March 2023, usability testing was carried out face-to-face with five participants comprising of diverse genders, social grades, highest educational qualifications and ethnicities.

Generally, the paper questionnaire was reviewed as relatively straightforward to complete and contained questions that were engaging. Following feedback from the usability testing, some changes to the paper questionnaire were made which included

• ensuring all essential information is included within the questions as some respondents skimmed through the front page or skipped it entirely
• making instructions clearer by underlining the number of responses respondents should cross and using darker shading
• emphasising filters by changing its colour, moving the filter closer to the wording and placing filters at the top of the page rather than mid-page.

More detailed findings from the usability testing are outlined in Annex B.

1. In February 2023, there was a Machinery of Government (MoG) change and responsibility for digital policy now sits within the Department for Science, Innovation and Technology (DSIT). This MoG change did not affect the contents of the Participation Survey for 2023/24 - digital questions are still part of the survey. ↩

2. Let’s Create, a strategic vision by ACE, sets out that by 2030 they want England to be a country in which the creativity of each of us is valued and given the chance to flourish and where everyone has access to a remarkable range of high quality cultural experiences. They invest public money from the government and The National Lottery to help support the sector and to deliver this vision. ↩

3. Satisficing happens when respondents provide quick, “good enough” answers to complete a survey faster rather than carefully considering the answers. ↩