Guidance

A guide to inclusive social research practices

Published 1 December 2022

1. Foreword by the Deputy Heads of the Government Social Research profession

As Deputy Heads of Profession, we are proud to support this Government Social Research (GSR) Inclusive Research Guidance, which was developed by one of our excellent GSR Strategy Working Groups.

Understanding and representing a diverse range of experiences from different groups is essential to developing and delivering effective government policy. Government decision-making affects different groups and individuals in different ways. Embedding an understanding of these differences in the analysis and evaluation conducted to support decisions, both before and after they have been made, is an important step towards ensuring equity of access, opportunity, and of the impact of policy decisions.

This guide, produced as part of the delivery of the GSR 2021-2025 Strategy, provides advice to researchers across all professions within Government on how to improve the generation and use of data relating under-represented groups, whose views are less likely to be heard by decision-makers through traditional methods.

The guide focuses on the four key elements of any research project: research design, data collection, analysis and reporting. In each section issues around inclusivity are explored and recommendations highlighted.

We truly thank William Nicholson, Jamie Juniper, Beth Holloway, and Kat Wingfield, for all their efforts as members of this Working Group – as volunteers and on top of their day-to-day roles – and for the passion they have demonstrated on this important topic.

This Inclusive Research Guidance will be a fundamental tool for social research across government to ensure the equity and representativeness of our work. We encourage all GSR members, and those involved in research for government more broadly, to read and make use of it.

Siobhan Campbell and Ed Dunn

Deputy Heads of GSR

2. Acknowledgements

This guidance was produced by four members of the GSR, as part of a working group that was formed in Spring 2021. The working group members were:

• William Nicholson,
• Jamie Juniper,
• Beth Holloway and,
• Kat Wingfield.

The working group consulted with a variety of experts and stakeholders throughout the Civil Service and other Public Bodies in the creation of this document. Specifically, the working group would like to extend their thanks to the experts listed below whose comments were invaluable in the development of this Guidance:

• Dr Emma Gordon, Economic and Social Research Council,
• Dr Atiya Kamal, Birmingham City University and,
• The Equality and Human Rights Commission (EHRC) who provided comments on an earlier version of the guide.

3. Introduction

This guidance has been produced in response to the GSR 2021 – 2025 strategy, focussing on the priority to create ‘a diverse and inclusive profession: representative of the society we serve through our values, profile and perspectives of our membership’; with this being the first professional guidance on inclusive social research practices produced by the GSR.

It supports the GSR commitment to continue developing improved people-focused decision-making at the heart of Government and allow greater understanding of the individuals and groups affected by those decisions.

The guidance provides advice to researchers across all professions within Government on how to improve the generation and use of data relating to seldom heard groups. In the context of this guidance the term seldom heard groups refers to under-represented groups whose views are less likely to be heard by decision-makers. Such groups can include those who are not often included in government surveys or those whose issues receive little coverage by public debate.

Representing a diverse range of experiences from different groups is essential to delivering effective policy making. Changes to policy may affect different groups and individuals in different ways; embedding an attentiveness to these differences in the analysis conducted to support decisions, or in evaluation after a policy decision, is an important step towards ensuring equity of access, opportunity, and impact of policy decisions.

The Public Sector Equalities Duty (PSED), created under the Equality Act 2010, sets out the need to eliminate unlawful discrimination, harassment and victimisation, and advance equality of opportunity and good relations between people who share a protected characteristic and those who do not. A guide to Public Sector Equalities Duty is available .

It can be challenging to develop projects that capture input from all parts of society if researchers do not have a clear set of tools and guidelines at their disposal. It is hoped that this guidance can help address this challenge. It needs to be highlighted that this guide inherently only provides a high-level view, and readers are advised to use this guidance as a starting point for further research if they are looking for specific guidance. The guidance should be seen as part of the work by the wider analytical community to improve the inclusivity of data generation and analytical processes. For more information on the wider work by the Analytical Function please see the Inclusive Data Taskforce’s report and Annex A for a summary of the Taskforce’s principles.

The main recommendations of this guidance are:

• leave sufficient time and resources for the design stage of the research process so that the project can accurately capture the views of all relevant groups from the beginning
• consider the different needs of various groups when collecting data and be open to using a range of methods and approaches. Researchers need to be aware of the additional costs associated with using a range of methods to reach seldom heard groups and factor this into their research plans
• analytical plans need to consider equalities issues and analysis should be taken on disaggregated data where possible
• when reporting research, the voice of seldom heard groups needs to be heard, with the diversity of participants reflected in the write up of analysis and conclusions drawn.

The table below highlights the key questions that researchers should consider in order to produce research that is inclusive and representative of the UK population as a whole. These are explored in more detail in the subsequent sections.

Research stage	Key questions
Research design	Who do I need to sample?
	Do I have the right sampling approach?
Data collection	Are my research materials accessible?
	Is my recruitment strategy inclusive?
Conducting analysis	Is my analytical approach representative of the different groups in the research?
	How will I overcome statistical issues with sub-group analysis?
Reporting	Have I reported enough detail on the groups involved in my research?
	Do my conclusions reflect results for sub-groups and not just the sample as a whole?

4. Research Design

From the outset of a project, researchers should endeavour to design inclusive research. This means taking time to understand different groups or sub-groups within the research and acknowledging the barriers and enablers of different designs. For example, using a mixed methods or purely qualitative approach can prove advantageous when looking at seldom heard groups, as they provide the researcher with a deeper level of insight and understanding.

Questions to consider at the research design stage include:

• have I scoped the project thoroughly and ensured that I am aware of all seldom heard groups I need to engage with during the project?
• have I considered who to sample?
• how will I sample? How will this approach reach the seldom heard groups I want to engage with?

The discussion below highlights the complexities of creating an inclusive research design. Such designs can take additional time and resource but help provide strong foundations for ensuring research reflects the views of all sections of society. This in turn helps inform policy that works for all, and which can help to improve the lives of all members of society.

4.1 The research question

The research question is key to developing research and informs all subsequent design decisions. Findings from the scoping stage should be used to ensure that the research question is clear and focused on the issue under investigation, guides the data collection and analysis, and is sensitive to the research context. Researchers should be aware of how the definition of research question(s) may inherently exclude some groups, and thus it is important to devote particular attention to the cascade effect that the design of the question(s) will have on inclusivity. In some cases, non-specific research questions may be the most appropriate. In this instance, engagement with communities to establish what is important to them should take precedent over narrowly defined researcher-specified questions. Co-production can help to ensure that decisions about what is measured and prioritised are fair and equitable.

4.2 Choosing your research method

Research projects can use qualitative methods, quantitative methods, or a combination of the two approaches to address the research question. A qualitative or mixed methods approach can be particularly insightful when conducting research with seldom heard groups as they allow participants more freedom to share their experiences in detail and can paint a clearer picture of their experiences than quantitative methods alone. Though often more time consuming it is possible to scale up such mixed methods approach and achieve appropriate sample sizes, see below for more details.

4.3 Who to sample

Within the context of this guidance a key part of this stage is to identify which groups the research project needs to engage with. One approach to achieve this would be to undertake a stakeholder mapping exercise early in the project lifespan. Such exercises can help identify the range and types of stakeholders, including seldom heard groups, allowing for the use of appropriate methods subsequently in the project to engage with all groups. When designing the stakeholder map, it is important to be mindful of ‘groupthink’ and avoid focusing exclusively on groups that past research has already investigated. It maybe, that since the completion of past research, new seldom heard groups could have emerged, for example a new refugee community, and it is important to include any new groups in your stakeholder map. Undertaking literature reviews could also help when scoping the project. Such work can help researchers identify specific groups that relevant work engaged with and identify best practice for bringing seldom heard groups into the research process. Undertaking these steps would help researchers align with the second recommendation of the Inclusive Data Taskforce’s report which emphasises the need to ensure all groups are captured in UK data.

4.4 Sampling and sample size

If you wish to use data to make statistical assertions, then not all sampling techniques will be appropriate. For example, in instances where no sampling frame exists or is not appropriate, snowball or other self-selection techniques can be valuable (e.g., the National LGBT Survey 2017). However, this approach removes the ability to produce conclusions that are inferable to a wider population as respondents are self-selected and are not randomised. Specific issues for different research methodologies are discussed below.

Sampling for quantitative research

Depending on the types of questions you are interested in answering, randomised sampling with standard sampling sizes could lead to sub-groups in the sample that are too small for robust statistical analysis. To overcome this issue, researchers should undertake a power analysis to understand the appropriate sample size at a sub-group level. This guide provides more details on sample selection and statistical power for sub-group analysis. Sampling methods such as stratified sampling, quota sampling or sample boosts could then be employed to reach the required sample size. For more details on these sampling methods see this GSR guidance. It is important to note that such sampling strategies do not increase participation for seldom heard groups, but they increase sub-sample sizes to allow for effective analysis.

Sampling for qualitative research

Qualitative research represents a crucial tool for developing an understanding of the rich experiences and complex issues affecting seldom heard groups. However, the principles applied to sampling for quantitative studies are not transferrable to qualitative approaches. Qualitative methods do not aim to produce generalisable findings and should be expected to involve purposive rather than random or representative samples. There is a lively discussion in academic circles on how best to approach sampling techniques and sample sizes in qualitative research, see work by Marshall (1996) and Vasileiou (2018) for introductions.

Sampling for evaluations

When evaluating whether a policy is having differing impacts on different groups, it is important to ensure adequate baseline data collection before the policy comes into effect. If baseline data, be it qualitative or quantitative, does not contain the views of and/or data from a range of groups, it can be difficult to post-hoc evaluate the impact of a policy on different groups. For guidance on assessing the quality of qualitative research evaluations see this GSR report.

4.5 How to sample

To decide how to sample, it is important to explore how other relevant research has developed its sample and if sampling frames exist. The Office for National Statistics (ONS) can provide further details on available sampling frames. There is often the concern around selection bias, as there is a chance that certain groups (e.g., young men) are less likely to complete surveys, with such underrepresentation skewing results. This is difficult to design for and weighting will only help to some extent. Liaising with staff networks in your workplace may help as these may offer pathways into specific communities.

4.6 Co-production

Co-production, whereby projects are created and delivered by researchers alongside community leaders, and communities can be an effective way to ensure that the voices of seldom heard groups are captured. Working with charities and community groups may also be productive as they are often trusted bodies and can be seen as approachable by seldom heard groups. For more guidance see the principles for co-production of guidance by the Independant Scientific Pandemic Insights Group on Behaviours (SPI-B) and a report by the Institute for Community Studies. These approaches can also overcome issues around selection bias and research fatigue which can be an issue when sampling from a specific group.

5. Collecting Data

Once the research design has been scoped out and planned, data collection can begin. At this stage it can be helpful to refer back to existing research (you may have undertaken a literature review while scoping your research design) to note how other researchers have collected similar data.

Questions to consider at the data collection stage include:

• are the research materials accessible?
• what demographic information is needed?
• how will recruitment of participants be inclusive?
• how can questions be worded, presented, and probed sensitively? Could harmonised standards help?
• is online recruitment and/or data collection appropriate?

The below section emphasises the importance of considering how to engage with seldom heard groups as a distinct aspect of planning research. Importantly, simply promoting opportunities to participate in the research using different mediums (whilst worthwhile), is unlikely to lead to significant interaction with a target group. To have a high chance of engaging with seldom heard groups, consideration for their needs must be embedded into all aspects of data collection. However, this can take time, and so researchers must ensure they build in sufficient resource into project plans to develop suitable data collection practices.

This section focuses on the mechanics of data collection, but there are situations where data from seldom heard groups is needed but no research is planned. In these instances, the Equalities Data Navigator Tool could provide a route to identifying and, in some cases, accessing the required data.

5.1 Research materials

When collecting data, it is important that research materials are accessible to as many people as possible. This includes the format of the materials as well as the language used. There are specific legal accessibility requirements for websites of public bodies that need to be considered but, more broadly, when formatting materials researchers should bear in mind the need to keep them accessible (see Government Accessibility Guidance for more information; additional guidance has also been published by the University of Edinburgh).

As well as format, the length of materials (for example the length and number of questions in a survey) should be no longer than absolutely necessary — keeping materials short and to the point will reduce demand on participants and make your study more accessible to some groups.

5.2 Language

Language is also an important consideration when conducting research with diverse groups — English might not be the first language for all participants. A comprehensive example of this can be seen from the North Cumbria Clinical Commissioning Group who translated COVID-19 guidance into a range of languages. It should be noted that the requirements for health-related materials may be different to other areas of research. It is reasonable to assume that health guidance should be translated into languages spoken locally, whereas researchers should determine a proportionate number of languages for inclusion when producing other research materials. It is important to have these in place in advance to avoid delays to the research process. One issue with direct translations is that they can be problematic and lose the intended meaning, and as a result SPI-B recommend pre-testing research materials to ensure language is accessible.

5.3 Easy read documents

Another approach is the use of Easy Read documents which can enable people to understand questions and concepts without the need for a direct translation, see this blog by the UK Health Security Agency detailing how they made an Easy Read document explaining how to book a PCR test, and a 2010 guide from the Department of Health and Social Care on making written information easier to understand for people with learning disabilities. Easy Read documents are essential for those with learning difficulties, or those who may have difficulty understanding more complex concepts. The Home Office has also provided guidelines for producing materials for six diverse groups, including those with dyslexia and those who use screen readers.

Undertaking the above steps could help researchers work towards the Inclusive Data Taskforce’s first recommendation, which focuses on creating an environment which allows and encourages everyone to count and be counted in UK data and evidence.

5.4 Consent

The GSR ethical assurance guidance notes the importance of consent forms for gaining full assent from participants, yet consent forms (and information sheets) are also important tools to promote trust in a research programme. Clearly stating, in accessible language, why, how, and when personal data and research data will be used is a crucial step in helping build trust with participants and can help overcome barriers to participation. The British Psychology Society Code of Human Research Ethics provides further guidance on consent forms and how to gather informed consent from participants. For advice on wider data protection issues pertaining to the UK General Data Protection Regulation (GDPR) see the Information Commissioner’s Office website.

5.5 Demographics

When collecting demographic information, researchers could consider collecting data on all Protected Characteristics in line with the Public Sector Equalities Duty. However, it is important to ensure that questions are not overly burdensome to ensure participants do not drop out of the research. Furthermore, Government researchers need to have due regard to Principle 6 of the GSR Professional Guidance and avoid unnecessary questions. The balance between reducing burden and collecting equalities data will be specific to each individual research project, however without such equalities data it may be difficult for a project to meet any requirement under the Public Sector Equality Duty. Therefore, research must think carefully when deciding against the inclusion of questions collecting data on protected characteristics.

With regards to phrasing demographic questions, think carefully about your options. The term ‘BAME’ is not accepted by all and should be avoided see writing about ethnicity. Similarly, the term ‘other’ should be avoided when providing a list of categories for respondents to choose from: the use of ‘another’ is generally more inclusive in this scenario. The ONS publishes guidance on the use of terms, for example the differences between gender and sex.

When collecting demographic data, researchers should always try to use the most disaggregated taxonomy where resource allows. Offering the lowest disaggregated options can help avoid cultural sensitivities arising from using higher level group options and so create a more inclusive data collection method. Having lower-level categories will also enable more detailed data analysis to be undertaken and capture intragroup differences. See this report from the Race Disparity Unit for more guidance. However, it is also important to consider anonymity throughout, and ensure you do not use low level categories that are so specific they could lead to anyone in your sample being identified. Researchers also need to be mindful of different requirements for asking questions in Scotland and Northern Ireland.

Overall taking such a disaggregated approach to demographics would allow researchers to work towards the fourth recommendation of the Inclusive Data Taskforce’s report, which is to enable robust and reliable disaggregation and intersectional analysis across the full range of relevant groups and populations.

5.6 Harmonised standards

The Government Statistical Service (GSS) Harmonisation Team at the ONS have published harmonised standards which can be helpful when choosing demographic options. Using their set terms can also improve the coherence and comparability of statistics. Further information on the importance of harmonisation can be found on the dedicated GSS harmonisation page; with further information in the UK Statistics Authority’s explanation of how coherence relates to the Code of Practice for Statistics.

5.7 Facilitating participation

In addition to considering research materials it is important to be aware of different approaches to recruit your participants and how to gather data from them. To secure participation from seldom heard groups, a range of strategies may need to be considered. For a more systemic review of strategies to increase participation of seldom heard groups see Bonevski et al (2014).

Recruitment of participants

When looking to recruit from seldom heard groups, it is important to consider your methods and how to make sure they are accessible. For example, not all individuals have internet access, computer skills, or a permanent address, so it is crucial to consider alternative approaches such as post, telephone calls and leaflets in public places. Additionally, while noting the limitations of snowballing raised in the Research Design section, it can be a useful method for recruitment from seldom heard groups.

Population-based probability sampling can be a time and cost inefficient method of sampling seldom heard groups due to an inherently smaller participant pool to draw from. The technique can be improved by collaborating with third party organisations (e.g., charities) that are already well connected with the communities you wish to engage with. It can also be useful to consider whether particular geographic locations may have a higher proportion of groups – such as ethnic minority groups – who you might want to sample.

You should also consider how best to promote research activities. For example, not everyone will be motivated by engaging in research but might have other motivations and priorities that the research could align with. Engaging with community groups could help uncover people’s motivations. Incentives may be used as a means of encouraging participation, but the appropriateness of different types of incentive should be considered. For example, a cash incentive may not always be appropriate and vouchers could be used instead. Please consult the GSR Professional Guidance for further advice.

Key to recruitment with any groups, but arguably more pertinent when engaging with participants from seldom heard groups, is the ethical principle of minimising harm. This principle holds that researchers must minimise the risk that participants taking part in their study will come to harm. For further details on this see the British Psychology Society Code of Human Research Ethics.

Protecting researchers

As well as protecting participants from harm, it is also important to protect researchers working on sensitive topics or when out in the field. Guidance by the Social Research Association covers the key elements to consider in order to protect researchers. Also see Fahie (2014) for a further discussion of the issues. Additionally, this blog covers steps to protect both participants and researchers.

Data collection

When collecting data from seldom heard groups it is important to note that the researcher may be viewed as an ‘outsider’ and care should be taken in how things are worded, presented and probed. Using community-based approaches such as co-production involves working with ‘insiders’ – such as local researchers or community leaders – and can help to address issues of researcher mistrust. As with all research it is important to consider data protection and GDPR issues. See this guide to user research for more information.

When collecting data from seldom heard groups a range of different data collection methods need to be considered. This includes a mix of online and face-to-face approaches. If using face-to-face methods researchers should be mindful of the location chosen for research. Research should be conducted in a relaxed setting where participants feel comfortable, for example a local community centre rather than an office.

Lastly, researchers should consider how they can support third party organisations helping to facilitate participation. This could include helping to cover printing or refreshment costs.

Online methods

While online research, both in the form of online recruitment (e.g., posting on community interest groups), and online data collection (e.g., online surveys), can have benefits, in that it removes geography and travel as exclusionary barriers to participation (see the BMJ Journal for an online research example), it is not a panacea .Online research can still exclude people, for example those without access to online devices, or those who might not have the technical ability to use online video conferencing. This might lead to sampling biases in your study; for example, by potentially excluding those on low income and older people. With this in mind, using a mixed methods approach to recruitment and data collection is recommended.

6. Conducting Analysis

Researchers should have a clear analysis plan that considers how their work affects and is affected by different groups or individuals, and factors in the time and resources to undertake appropriate group and subgroup analysis.

Questions to consider during the planning process include:

• is there previous evidence of differences in outcomes that can inform the analysis plan?
• are you conducting a comparison of groups specifically to examine equalities, or ensuring that the analysis is inclusive of different groups?
• how can the analysis be designed to ensure reflexivity in qualitative research?
• can you design your analysis so that you remain open to the issues and experiences that are important to the research participant(s)?
• is it recommended that an equality analysis is included to ensure compliance with the Public Sector Equality Duty?
• if an equality analysis is being carried out, how can the sampling and analysis plans be designed to ensure comparison between the relevant groups?
• what level of disaggregation is necessary? Assume the lowest level if possible.
• if resources are not available for lowest disaggregation, what steps can be taken to ensure aggregated analysis is caveated?

The issues outlined below highlight the importance of looking deeper into the data and the value of taking time at the outset to consider the equalities implications and impacts of analysis. However, while there will sometimes be clear reasons to disaggregate data, aggregating ethnicity categories might be more appropriate with other data, such as when it is difficult to achieve a large enough sample for robust analysis. The point here is that researchers need to always consider their work from an equalities perspective to determine how to build this into their analysis approach. This is in addition to maintaining due regard towards the specific issues outlined in the Public Sector Equalities Duty, a summary of which can be found in the EHRC’s Essential Guide to the Public Sector Equalities Duty.

6.1 Disaggregating broad, heterogeneous categories

Once the data has been collected, efforts should be made to make sure that seldom heard groups are visible in the analysis. Such an approach will align with the Inclusive Data Taskforce’s first and fourth recommendations. These put emphasis on creating an environment which allows and encourages everyone to count and be counted in UK data and evidence, and the need to enable robust and reliable disaggregation and intersectional analysis across the full range of relevant groups and populations.

Disaggregation of data is important as the aggregation of data into broad categories has the potential to mask critical within-group differences and disparities. The Race Disparity Unit of the UK government has produced a range of methodology papers including a discussion of the similarities and differences between aggregated ethnic groups. This outlines issues with heterogeneity as well as some of the known differences within groups which can help inform planned analysis. For example, in the UK, the category Asian encompasses a diverse population and there is significant variation in measures such as educational attainment among the five ONS-recommended Asian/Asian British sub-groups. In the 2020-21 academic year, 50.3% of those identifying as Pakistani gained a grade 5 or above in English and maths GCSE, whereas 72.4% of those identifying as Indian achieved this. This distinction would be lost if analysis was only conducted at the Asian/Asian British level. Importantly, including only broad ethnic categories in analysis perpetuates the representation of the population as a homogenous group and fails to take account of different experiences and needs. Therefore, when analysing results, researchers should ensure the analysis of group differences at the lowest possible level of characteristics.

6.2 Compounding disadvantage

Assuming the possibility of compounding disadvantage helps to analyse how different forms of oppression (e.g., racism and sexism) interact and intersect to influence lived experiences. When conducting analysis, it is important to consider how the complexity of individual experience can be highlighted. Qualitative research approaches, such as life-story narratives, can provide productive means of expressing the complexity which can often be more difficult to convey via statistical summaries. A ‘two pronged approach’ can help to develop analysis which is based on: 1) the assumption that an individual’s experiences can be linked to multiple forms of discrimination; and 2) an understanding of context, be it social, political, legal, or regulatory. It is the responsibility of researchers to highlight the existence and interaction of these two components. Quantitative analysis may also be helpful for breaking down categories and identifying compounding effects.

6.3 Qualitative analysis

Particular care should be taken to ensure reflexivity in qualitative analysis when conducting research into the experiences of seldom-heard groups. Well-facilitated qualitative methods should allow for the representation of issues and experiences that are important to the participant (rather than just the researcher) in the data generated. This is applicable when working with research participants as well as when undertaking documentary or observational analysis. It is also necessary to acknowledge the positionality of the researcher when analysing this data and how this might influence the conclusions reached. Grounded theory-based approaches can help to focus on the substance of the data itself and generate theories that reflect what matters to seldom heard groups.

6.4 Statistical analysis issues

The most common approach to analysing quantitative data is to consider parametric/non-parametric tests, but there are many approaches beyond this, for example Generalized Linear Modelling and multilevel modelling.

One issue with using null-hypothesis significance testing to undertake analysis on disaggregated groups is the increased risk of a of a Type 1 error occurring. This error is the chance of falsely detecting a statistical effect (i.e., a difference between two groups) where one does not exist. While the chance of a Type 1 error occurring for one test is 5% (when using a standard alpha of p = .05), the chance of a Type 1 error occurring increases for each additional statistical test undertaken. To reduce the chance of a Type 1 error occurring when undertaking multiple comparisons there are several corrections that can be applied, see Chen (2017) for details. Other analytical approaches avoid such issues for example Bayesian analysis, which expresses the likelihood of obtaining the data from one hypothesis relative to another. See Wagenmakers (2016) for further details on this analytical approach.

A risk when undertaking subgroup analysis to explore the data from seldom heard groups is that sample sizes may be small. For example, secondary datasets often have representative samples, and therefore small sample sizes for lower group levels (e.g., specific ethnic groups). Focusing on a specific outcome – such as unemployment – can reduce the size of the groups even further. With the addition of two or three more variables, the effective sample can become very small. This increases the risk that your data will be non-parametric (e.g., non-normal distribution), so it is crucial to check that your data meets the assumptions for your identified statistical analysis to ensure your analysis rests on accurate models. If your data does not meet the required assumptions, you should consider an alternative approach. The most common approach is to consider non-parametric tests, but there are many approaches beyond this, such as Multilevel Modelling and Generalized Linear Modelling. The University of Bristol has a comprehensive guide on these approaches.

7. Reporting

Research outputs need to be tailored for their intended audience. After collecting and analysing your data, it is important to consider how you reflect your findings appropriately by how you report your data.

Not all data sets will be the same, and in some circumstances, it will not be necessary to comment on your specific sub-group analysis all the way through your report. There is no ‘one-size-fits-all’ template for writing up your analysis, but you should aim to include:

• an overview of the problem – why was this research or analysis conducted?
• how you collected and analysed your data – including recruitment details, details of participant demographics, and analytical methods used – particularly details of any quantitative or qualitative methods.
• the results of your analysis, in which all sub-group analysis conducted should be reported. Take care not to unintentionally marginalise and/or stigmatise groups in the language that is used to report the results.
• if you have used a qualitative approach, you may need to consider a longer results section, for example if you are including anonymised quotes.
• conclusions of your analysis or research, and recommendations for how it should be considered and applied in the real world, particularly for seldom-heard groups.

You will also need to consider the appropriate place to publish your findings, making sure you make them as accessible as possible.

After collecting and analysing your data, you will need to report it in a way that continues to reflect the diversity of your participants. This section broadly outlines how you should do this, and what you should consider including in your report.

To overcome the difficulty in comparing different analysis on different topics, the ONS has produced guidance on reporting and presenting data for a range of topics. Harmonised reporting can help increase the usefulness of analysis as results can be easily compared across research. This is especially important if conducting research including seldom heard groups, so that key messaging can be clearly articulated across studies. Analytical reports may be structured differently but should tend to include four main areas: 1) an overview of the problem or the topic being explored, 2) collecting and analysing data, 3) reporting the results, 4) drawing out conclusions and recommendations. As outlined above in the Conducting Analysis section, it is important to report meaningful disaggregation of your data and to consider what is important to present based on the research aims.

7.1 Overview of the problem

In this section of a report, you should outline why this particular project or piece of analysis was conducted, and what the analysis was trying to achieve. If the analysis is specifically looking at a range of groups or trying to focus on an equalities angle, highlight this in the overview, including the specific groups you are focussing on, before going into the methodology. The research design and analysis sections emphasise the need to factor in this focus from the outset. Referring back to available research design or analysis plan documentation may be helpful. It is important to fully highlight an understanding of the problem at hand, as this will allow you to showcase why your collection and analysis of data relating to seldom heard groups is necessary and encourage readers to think about what the outcomes from the analysis may help to achieve.

7.2 Data and methodology

This section should be used to outline how you collected and analysed your data. This is an important place to highlight the different groups you may have collected data from — all of these groups should be briefly outlined, even if you do not comment on them further in the report. It is also an opportunity to outline how you collected data from seldom heard groups, for example, by using co-production methods. Such detail will aid the replicability of the project and could also encourage others to consider how their research or analysis can also include these groups. This information should typically include how participants were recruited (sampling methods) and how data was collected (for example, online surveys or face-to-face interviews). More generally, detail on ethical considerations and sampling methods should also be included in this section — research methods should be well documented and transparent, with assumptions and limitations clearly detailed. Specific attention should be paid to certain known biases that can occur, e.g., under-sampling of men in surveys.

Report the demographic details of all groups in the same manner in which they were displayed at the data collection stage. For example, if you conducted a survey and asked respondents to select their ethnicity, it is important to report ethnicity options on this section exactly as they were written in the survey. This should follow the ONS’ harmonised standards; this link provides the recommended approach to classifying ethnic groups. You need to report all the demographic groups that were recorded at the data collection stage, not just ethnicity.

One important factor to consider here is disclosure. If you have a small sample size, or individuals with specific attributes (particularly if collecting lots of details on diversity) it may be possible to identify these individuals from the data. Refer to ONS guidance on suppressing data fields for further advice.

It is important to highlight the sample size of each of these groups as you want to make sure you do not overrepresent or underrepresent the view or finding from one specific group. Explain the statistical tests conducted between groups, and the statistical software used to do this. If any groups had to be excluded from the analysis (e.g., for disclosure issues or error in data collection) this needs to be made clear. If qualitative methods were used, outline which data management tool, if any, was used to manage data and outline the analytic approach used to analyse data.

7.3 The results of your analysis

If you have conducted analysis to determine statistical differences between sub-groups, it is important you report (ideally in a table) the outcomes of all groups analysed, and then draw attention to the areas that you stated were the focus in the aims of the study. The areas of interest can be highlighted further, or drawn attention to, using figures such as graphs, which should be clear and appropriately labelled. It is important that in your presentation of the results you draw attention to subgroup analyses to prevent them being overlooked against the all-groups analysis. Care should also be taken to explain and make clear any concepts discussed to avoid confusion among those using your analysis (e.g., clearly distinguishing between concepts such as sex and gender). Terms can change over time and it is important you reflect the current concepts in your reporting. Published work by ONS can help you understand differences between concepts. In this section it is also important to be transparent about the assumptions and limitations of the research method and the analytic approach so that audiences can use your analysis appropriately. This should also involve publishing and sharing data wherever possible.

If you have used a qualitative approach in your data collection and analysis, you will likely have a larger results section than if you are reporting on quantitative data, and you may need to take some of the following considerations into account when reporting demographic information. If your analysis is specifically looking at differences between demographic groups, it is important to be representative in your reporting of all groups studied and to reflect the diversity and range of voices of the groups you have collected qualitative data from. If you are reporting quotes from participants, you may want to provide demographic information on these individuals if this is relevant to the point you are making, for example providing their age band and ethnic group. However, it is important to remember that including too much or too specific information could compromise confidentiality agreements, particularly if you have researched at-risk groups (for example, victims of domestic violence). More information on how to report quotations from research participants can be found in this American Psychological Association guide. Kirklees Council have also published a useful ‘how to…’ guide on analysing and reporting qualitative data.

7.4 Conclusions and recommendations

If you have completed sub-group analyses on several variables, you may not be able to write about all significant differences found in your conclusion. The parts of the analysis reported in the narrative will differ in different pieces of research. However, as a rough guide this selection should be data driven and should include the parts of the analysis that are pertinent to the research question. You may want to direct respondents to an annex with full sub-group analyses. This final stage of reporting is also an opportunity to reflect on the wider impact of your analysis and what the findings may mean in context, particularly for seldom-heard groups.

7.5 Publishing

Following completion of your write up, you may choose to publish your work, either in an academic journal or on GOV.UK. The GSR provides clear guidelines on the mechanics of publication in the GSR Publication Protocol, so these will not be covered here. Rather it is important to reflect on the needs of different groups when publishing research.

One issue regarding publishing in academic journals is access. Many journals are only accessible via a subscription. This may be suitable for academic audiences, but less so for policy makers and members of the public. Open access journals are those that are free to access, and so can help overcome this barrier to sharing knowledge. Publishing via GOV.UK can overcome some of the access issues as the site is open to all with internet access (see guide on how to publish on GOV.UK).

Additionally, care should be taken to avoid inaccessible technical language which may be off putting to those who participated in the research. Ensuring that you give presentations to participants where appropriate, or share lay summaries, can help participants understand the benefit of their engagement with the research, and hopefully encourage them to continue to engage with research in the future. The first recommendation from the Inclusive Data Taskforce highlights the importance of providing feedback to participants at the end of research in building trust.

More broadly, all the considerations highlighted regarding data collection apply equally to its dissemination. It is important to ensure you address a wide range of accessibility issues to make your data and evidence as accessible as possible. This could include producing easy-to-read documents, translations of reports, and non-digital options. See the Inclusive Data Taskforce’s eighth recommendation for further guidance. In all cases, dissemination activities should be mindful of the language that is used when reporting the results to avoid unintentionally stigmatising groups.