Guidance

Evaluation methods

Published 7 August 2018

This guidance was withdrawn on 17 September 2021

For up-to-date guidance see Evaluation in health and wellbeing.

Choosing methods for evaluation

A wide variety of research methods and data collection tools are available for use in evaluation: qualitative and quantitative. Different methods are suitable for answering different types of evaluation questions. This section provides an overview of the most common methods. The resources referred to at the end of this section give more detail on different methods and how to use them.

Qualitative research

Introduction to qualitative research

Qualitative research encompasses a variety of methods, but is often defined in terms of using words and text in data collection and analysis, rather than using numerical measurement and quantification (Bryman, 2016). It commonly investigates how people make sense of the world and experience events. For example, it might explore how it feels to be a patient with a long-term condition or what it is like to live with someone with a terminal illness, and aim to gain insight into how people make sense of and manage these situations.

Qualitative research is usually ‘bottom up’ rather than ‘top down’. That is, a theory or explanation is developed from the data rather than data being collected to test a theory or hypothesis. Qualitative research often aims to describe and explain rather than predict or identify cause and effect. It can be used to enrich our understanding of an issue or to improve services (Willig, 2008).

Qualitative research is not concerned with representing a population, in the way that quantitative research is, but with investigating topics in depth (Willig, 2008). Usually, a relatively small number of people are included in a sample because of the richness of the data collected and the time it takes to collect and analyse information. Sometimes, simple qualitative research methods are combined with quantitative research, for example, via the inclusion of a comments section within a questionnaire.

Benefits of qualitative research

The benefits to using qualitative methods are that they:

• are useful for investigating the perspectives and interpretations of participants in a holistic fashion
• are less dependent than quantitative methods on the pre-conceived ideas of the researcher(s)
• facilitate new or deeper insights into, and understanding of, a phenomenon (this can be very useful where little is known about a topic and exploratory research is required)
• can be used to investigate unusual cases from which valuable insights can be gained (in quantitative research these ‘outliers’ may be discarded)
• can usually be modified on an ongoing basis as new insights are gained; for example, to explore if unexpected findings emerge
• is adaptable; for example, the order and style of questions in an interview or focus group can be altered to fit or accommodate the needs of different participants or groups
• can be coded and summarised in a quantitative way, if desired

Limitations of qualitative research

The limitations of qualitative methods are that:

• they are generally time-consuming to use - it can take a lot of time to arrange and undertake data collection (such as via interviews and focus groups), transcribe discussions, analyse data, and interpret and present findings in a meaningful way
• data analysis can be challenging and relies on having the necessary knowledge and skills in qualitative data analysis approaches, techniques, analysis software, etc
• they produce data and findings which may vary depending on the skills and theoretical perspectives of the researcher(s) collecting, analysing and interpreting the data; however, the reporting of qualitative research should make this process transparent to allow the reader to judge findings in light of this
• they usually collect and analyse data from relatively small numbers of participants, meaning that the generalisability of findings is not addressed and trends or distributions of views within a population are not ascertained
• the anonymity and confidentiality of participants is more difficult to maintain

Qualitative research tools

Interviews

Interviews are frequently used to collect data in qualitative research. They are usually based on a topic guide. In evaluation studies, interviews are commonly semi-structured where questions or themes are decided in advance, but the interviewer has flexibility to re-order the questions and follow-up with further questions, if necessary.

Interviews allow topics to be explored in depth. These topics can be adapted to take account of participant’s needs (for example, if there are language or disability challenges) or experiences (for example, skipping questions which are not relevant). An interviewee might also discuss a topic without being prompted or the interviewer might change the order of the questions depending on the situation.

In semi-structured interviews, questions are usually relatively ‘open’; that is, they encourage a detailed answer from the interviewee rather than a ‘yes’ or ‘no’ response. Generally, the interview topic guide/schedule would contain prompts in case a topic doesn’t naturally arise. Leading questions or otherwise indicating to an interviewee that a certain type of answer is the ‘right’ one should be avoided as this would bias the data.

Interviews can be conducted by telephone, through an online video resource or in person. There are benefits and drawbacks of using the telephone or online video compared to face-to-face interviews, summarised below.

Advantages of telephone/online video interviews are:

• they are cheaper in terms of staff time and travel
• they avoid wasted time if the sample population has a high rate of ‘no-shows’ for interviews
• they are more practical if the interviewee lives far away, or if they have problems travelling
• they allow more people to be interviewed due to reduced cost and time

Limitations of telephone/online video interviews are:

• they make it more difficult to establish a rapport with the interviewee
• they make it difficult to assess how interviewees are reacting to questions because their body language cannot be observed
• they tend to be shorter and are less suitable for sensitive topics
• they rely on having access to online video packages or good phone connection/reception, which may be problematic for some harder-to-reach participants

Advantages of face-to-face interviews are:

• they capture verbal and non-verbal cues and responses to questions
• they observe emotions and body language
• they establish a rapport with the interviewee

Limitations of face-to-face video interviews are:

• they require a number of people to conduct the interviews, which means there will be personnel costs
• quality of data you receive will often depend on the ability of the interviewer
• size of the sample can be limited to the size of your interviewing staff, the area in which the interviews are conducted, and the number of qualified respondents within that area
• they are more expensive and require time for travelling

In advance of the interview, researchers should decide if: the interview should be recorded, how it will be transcribed and analysed and how excerpts will be presented in any write up. In line with research ethics, it is good practice to discuss confidentiality issues and ask interviewees to read and sign a consent form to confirm that they are happy with the arrangements. Complete transcription of recordings is very time-consuming and partial transcriptions or summaries are sometimes used, depending on the type of evaluation.

Interviews can be tiring for interviewers, as they have to listen carefully to determine which topic to go to next and to ensure understanding. If the interviews are lengthy, it is generally a good idea not to arrange more than 2 in a day.

Focus groups

Focus groups bring together a small number of selected participants to discuss a specific topic and are usually facilitated by someone who does not know the group. The participants might be strangers or people who meet together already. In general, it is best not to include people with more power than others in the same group, such as teachers and pupils or junior and senior staff, as this can stop participants being open with their views or opinions. It takes time to set up a focus group and to ensure that everyone attends. Set up can be easier if the group is run alongside a regular meeting. For example, if you are interested in the views of men with an alcohol problem, you might try to get permission to conduct a focus group before or after an alcohol support group session.

Focus groups have similarities to interviews in that a facilitator uses a topic guide or scenario (see below) to ask particular questions and steer the discussion. The person who facilitates the focus group should be experienced, and there is training available to support development of relevant skills. Facilitators must ensure that one person does not dominate the discussion and that everyone gets the opportunity to have their views heard. The facilitator should not have a particular vested interest in the findings, so they do not lead the discussion or impose their own views. The facilitator should always be neutral.

It is useful to record a focus group or at least have a second person taking notes of the discussion as it is difficult to do this alongside facilitating the group. As with interviews, issues of consent and confidentiality should be discussed with participants.

Advantages of focus groups are that they:

• are useful when the interactions and contributions within the group may prompt additional, interesting themes to arise
• are useful when time is short and there is a need to get a range of people’s views quickly
• tend to work well for existing groups where people are comfortable with speaking to each other and are confident in expressing their opinion in a group situation
• allow people to build on and challenge each other’s views
• can be useful for understanding how and why different groups or types of participants have differing views (for example, men versus women, patients versus health professionals)

Disadvantages of focus groups are that:

• they are generally less useful for recording the responses of individual participants, as it can be difficult to note this down accurately during a discussion, or to identify individuals from a recording during transcription
• they are less suitable to use with some participants; especially if the topic to be discussed is sensitive
• people with hearing problems might also find a group discussion difficult
• they can be challenging and require training to facilitate, especially if there are disagreements amongst participants, dominant participants, or discussions get side-tracked away from the main topics of interest
• they produce complex data which can be particularly difficult and time-consuming to transcribe, summarise, and analyse

Topic guide

Topic guides include questions and prompts used when conducting a focus group or interview. They are called guides because they are meant to be adapted to the needs and experiences of the interviewee or focus group and used in a flexible way. The content of a guide would be decided based on the evaluation question(s) and objectives as well as after discussions with any oversight or steering group and/or with the type of people you will be interviewing. It is a good idea to test questions with a pilot group to ensure they are understandable and appropriate for the type of participants in the sample.

For example, if you were interviewing smokers about resources and services for quitting smoking in Manchester, you may wish to meet with a group of smokers in Liverpool (to talk to smokers in a similar area who are likely to have similar experiences) to discuss what topics should be included and what language should be used. It is most useful to ask open-ended questions to encourage participants to speak, and it is usual to start with some generic questions at the beginning to relax the participants and build a rapport.

For example, you might want to start with a question which asks for some background and gauges the participants’ knowledge of the area. This will help with adapting later topics and deciding in which order to ask questions. The opening questions also give insight into how forthcoming participants are likely to be. Some people speak readily and at length whereas others can be more reluctant to express their views and need prompts to explore issues in depth. This is particularly important if the topic is of a sensitive nature.

Scenarios

Scenarios can also be used in interviews or focus groups to depersonalise an issue. If a topic is sensitive and people might be uncomfortable talking about their own experiences, they could be asked to give advice to someone else in a similar situation or to imagine a person and what their responses would be.

For example, if you wanted to interview someone about addictions and their views on the services available, you could ask: ‘If a friend or neighbour wished to access addiction services in this area what advice would you give them? What problems might you highlight? Where would you suggest that they start?’

Alternatively, if you wanted to gain insight into the concerns of someone with terminal illness regarding hospices, you might say: ‘Mr McKenzie is moving to a hospice 20 miles from his home after a long illness. His wife is elderly and 2 of his 4 children have moved away from the area. What do you think his concerns might be? What about those of his wife and children?’

This would allow participants to discuss a topic in a more general way without feeling that the questions are intrusive or require sharing too much personal information.

Other qualitative methods

There are many more qualitative research methods such as naturalistic or ethnographic observations, video recordings of behaviour, open-ended questions on questionnaires, and analysis of text (for example, in advertising or educational materials) or photographs. More information about these different qualitative methods are described in the useful resources material below.

Analysing qualitative data

In-depth qualitative analysis requires training and experience. It can be conducted in different ways but usually proceeds in the following stages.

1. It is usually helpful to thoroughly familiarise yourself with the data by reading and re-reading transcripts or notes, or by listening to recordings.
2. Next, segments of data are labelled through ‘coding’, and then higher level, more interpretive themes are developed from these codes in one or more further stages.
3. Later stages often involve organising themes (for example, into groups), exploring their relationships with each other and observing patterns in the data.

In this way, qualitative analysis goes beyond merely summarising data, to drawing out underlying themes that provide in-depth insight into an issue.

There are computer packages that can help with analysis of qualitative data. The most common of these are NVivo, Atlas and MAXQDA. These programmes allow large amounts of qualitative data to be stored, analysed and summarised in a systematic way. Use of these packages requires training and a good understanding of qualitative methods.

Quantitative methods

Introduction to quantitative methods

Quantitative methods are used to investigate things that can be measured or quantified to generate numerical data. For example, what percentage of people hold a particular view, does use of a service vary by gender and age, what has changed after a particular intervention? Quantitative research aims to be objective and usually collects data in a pre-defined, structured way (Black, 1999). It is used to describe general trends or distributions, or to test a theory or hypothesis about relationships between things, including cause and effect.

Quantitative research methods usually aim to tell you something about a population (for example, smokers or older adults) based on a representative sample selected for a study. However, evaluation studies do not always make claims for the population at large, and may only provide more specific information about the effects of an intervention in a specific group of people or context.

Often, the aim of an evaluation is to find out how things have changed after an intervention. For example, are clinicians more knowledgeable about correct hand washing techniques after a training session? Alternatively, an evaluation may assess whether a service achieves a pre-existing standard. For example, have young people with mental health problems been offered support within a defined time period? See the section on outcome evaluation.

When planning an evaluation, you should identify what data is needed, from whom and when. This will be influenced by the questions being addressed and the most suitable study design for answering these questions within the resources available. It is important to determine if the required data are already available and accessible, or whether the data will need to be collected as this will have time and resource implications. It may be possible to directly observe (for example, to produce counts) or measure (for example, using specialist equipment or monitoring to assess physiological markers) the phenomena of interest. However, evaluation often relies on self-reported or pre-existing data, and various tools are available to support collection of this type of data. See the section on evaluation planning.

Quantitative research tools

Surveys and questionnaires

A common and quick way of collecting quantitative data from large samples of people is through surveys and questionnaires. Data collected from questionnaires and surveys allow comparisons between groups and subgroups to decide if there are differences between them, and to explore potential reasons for these differences.

Questionnaires are less useful when you wish to conduct more exploratory research (for example, where you might not have enough understanding of something to know what questions to include). They are also unsuitable for understanding aspects such as the meaning of participant experiences or for exploring issues in-depth. In these cases, qualitative methods are usually more suitable.

Questionnaire administration

There are different ways of administering surveys and questionnaires. The advantages and limitations of some of the more common methods are outlined below.

Advantages of administering surveys in person:

• you can lead participant through appropriate sections, explaining what is required, if necessary
• high response rate and minimal errors/missing responses
• good for overcoming disability and language problems

Limitations of administering surveys in person:

• expensive
• time consuming
• geographically limited
• problematic for sensitive topics

Advantages of administering surveys online:

• quick
• cheap
• wide geographical reach
• can ‘force’ responses and lead people through questions to reduce missing data
• people can do in own time
• can ensure anonymity

Limitations of administering surveys online:

• response rate may be low
• internet access may be a problem and exclude certain groups (for example, those with sensory impairments or older people)
• requires technical knowledge to set up

Advantages of mailing out surveys:

• anonymous
• cheap

Limitations of mailing out surveys:

• risk of very low response rates
• may need to build in time and budget for reminders or incentives
• need to be very clearly designed and self-explanatory

Designing a questionnaire

Questionnaires are usually accompanied by some demographic questions about age, gender, employment, place of residence and so on. For evaluations, specific questions are then based on what the intervention to be evaluated aims to achieve. If possible, it is better to use or adapt a questionnaire that has been used for similar purposes or in a similar sample and, ideally, has been demonstrated to be reliable and valid – see outcome evaluation section.

When selecting or designing a questionnaire, these issues may be considered:

• who is your sample; who do you need to answer the questions; will they complete the questionnaire
• where/how can you best reach these people
• how can you ensure a good response rate
• if the answers are confidential, how to ensure this
• are the questions simple and clear and only ask about one issue at a time
• avoid jargon in questions - this can be tested by piloting the questionnaire
• avoid leading questions (for example you would not ask ‘why did you like the service?’; ask ‘did you like the service? If so can you say why?’) -exclude any questions that are not necessary for the study
• exclude any questions to which the respondent might not know the answer
• include an ‘other’ option when necessary and provide space for respondents to write/add their own answers
• keep the questionnaire as short as possible
• leave any sensitive or confidential questions to the end to avoid putting people off at an early stage
• check whether the questionnaire is reliable and valid for the sample to be studied
• plan how the data generated by the questionnaire will be captured and stored (for example, entered into a spreadsheet), summarised (for example, to produce overall scores) and analysed

Secondary data collection

It is not always necessary to collect new data as data relevant to an evaluation may already exist; for example, data that’s been collected for administrative purposes or added to patient records. Certain permissions (for example ethics approval, letters of access) may need to be obtained to allow access to these data. A data collection ‘pro forma’ which specifies the data required is sometimes designed and can then be populated using the existing data sources. Alternatively, a request can be made for data or reports for a sample of patients to be transferred directly to the evaluator.

Secondary data collection, however, can also have its own challenges. Some data may be confidential or stored in a way that makes the data hard to access or to use for the desired purpose. Sometimes, data should be available in theory, but, in practice, they are missing, misfiled or not collected consistently.

Analysing quantitative data

Some quantitative analyses required for evaluation will be quite simple; for example, describing the number or percentage of people with a certain characteristic, achieving a goal or expressing a particular opinion. This analysis can usually be managed in Excel. More complex analyses (for example, comparing groups, looking at difference between subgroups, or controlling for extraneous factors) will require knowledge of statistical analyses and associated software packages (for example, SPSS, Stata, SAS). Support for statistical training or knowledge of the relevant software packages will need be to identified when planning an evaluation.

Combining qualitative and quantitative research

Qualitative and quantitative methods can usefully be combined in ‘mixed methods’ research to provide more comprehensive information for an evaluation (Andrew and Halcomb, 2009). You may wish to use qualitative research to find out how people feel and follow this with quantitative research to find out if these feelings are commonly experienced.

For example, if you want to explore pregnant mothers’ understanding and knowledge of antenatal care and their compliance with recommendations, you could start by conducting a focus group of mothers from different areas and backgrounds. This qualitative research would give you an insight into their knowledge and other the main topics that are important. You could then use these topics to develop a questionnaire to discover what is happening across the population of expecting mothers as a whole.

Alternatively, you may have obtained findings from a questionnaire and wish to discuss these with a focus group to get a deeper understanding of what the results mean. For example, if data from a survey or questionnaire shows that one hospital has more infections than others, you may wish to follow this up by interviewing staff or visiting the hospital and undertaking observations to get a more in-depth understanding of why that might be and how it could be resolved.

Selection of useful resources

BetterEvaluation website

Big Lottery Fund (2008). ‘Well-being evaluation tools: a research and development project’

Centers for Disease Control and Prevention (1999). ‘Framework for program evaluation in public health’

Centers for Disease Control and Prevention (2011.) ‘Developing an effective evaluation plan’

Centers for Disease Control and Prevention (2010). ‘Learning and growing through evaluation: state asthma program evaluation guide’

Centers for Disease Control and Prevention (2011). ‘Implementing evaluations: learning and growing through evaluation module 2’

Evaluation Support Scotland

Food Standards Agency (2015). ‘Introduction to evaluation for local authorities: a brief guide to evaluating food enforcement projects and innovative approaches’

HM Treasury (2011). ‘The Magenta Book: guidance for evaluation’

Department for International Development (2013). ‘Planning evaluability assessments: a synthesis of the literature with recommendations’

Medical Research Council Guidance (2008). ‘Developing and evaluating complex interventions: new guidance’

Medical Research Council Guidance (2015). ‘Process evaluation of complex interventions’

Medical Research Council Guidance (2010). ‘Using natural experiments to evaluate population health interventions’

National Science Foundation (2002). ‘The 2002 user-friendly handbook for project evaluation’

Avon Primary Care Research Collaborative website

NHS Health Scotland (2003). ‘LEAP for health: learning, evaluation and planning’

The World Bank (2004). ‘Monitoring and evaluation: some tools, methods and approaches’

Treasury Board of Canada, Secretariat (nd). ‘Program evaluation methods: measurement and attribution of program results (third edition)’

United Nations Evaluations Group (2005). ‘Norms for evaluation in the UN systems’

United Nations Development Programme (2009). ‘Handbook on planning, monitoring and evaluating for development results’

UN Women (2013). ‘Ending violence against women and girls: programming essentials’

US Department of Health and Human Services (2006). ‘Guide to analyzing the cost-effectiveness of community public health prevention approaches’

US Department of Health and Human Services (2010). ‘The program manager’s guide to evaluation’

WK Kellogg Foundation (2004). ‘WK Kellogg Foundation evaluation handbook’

World Health Organisation (2013). ‘WHO evaluation practice handbook’

Acknowledgement

Written by Sarah Denford, Jane Smith, Sarah Morgan Trimmer, Charles Abraham, and Krystal Warmoth, Psychology Applied to Health, University of Exeter Medical School.

This work was partially funded by the UK National Institute for Health Research (NIHR) School for Public Health Research, the NIHR Collaboration for Leadership in Applied Health Research and Care of the South West Peninsula (PenCLAHRC) and by Public Health England. However, the views expressed are those of the authors.

References

Andrew, S and Halcomb, E J (2009). ‘Mixed methods research for nursing and the health sciences’ Blackwell, West Sussex

Black, T R (1999). ‘Doing quantitative research in the social sciences: an integrated approach to research design, measurement and statistics’ Sage Publishing

Bryman, A (2015). ‘Social research methods’ (fifth edition) Oxford University Press, Oxford.

Guest, G. and Namely, E E (2014). ‘Public health research methods’ Sage Publishing

Willig, C (2008). ‘Introducing qualitative research in psychology: adventures in theory and method’ Open University Press, UK

Access a series of supporting materials:

Evaluation podcasts

Evaluation PowerPoint presentations

Evaluation videos