Before-and-after study: comparative studies
How to use a before-and-after study to evaluate your digital health product.
This page is part of a collection of guidance on evaluating digital health products.
A before-and-after study (also called pre-post study) measures outcomes in a group of participants before introducing a product or other intervention, and then again afterwards. Any changes in the outcomes are attributed to the product or intervention.
This study design cannot rule out that something other than the product may have caused a change. Randomised controlled trials (RCTs) are considered the most reliable way to show that your digital product has caused an outcome. However, it is not always possible to run an RCT. Before-and-after studies are more flexible and generally cheaper to run.
The NICE Evidence Standards Framework for digital health technologies considers before-and-after studies evidence for demonstrating effectiveness of tier C products (broadly, these are digital products that seek to prevent, manage, treat or diagnose conditions).
What to use it for
Use a before-and-after study when:
- you want to identify changes in particular outcomes following the use of your digital product
- you are launching your digital product, or soon after (summative evaluation), and you want to explore if it can be effective
- there are practical limits to what you can do, such as small available samples, difficulty around randomising participants, or limited budget
Pros
Benefits include:
- they can tell you something about the effect of your digital product
- they do not require randomisation
- they are less resource-intensive than an RCT
- they can run alongside normal use of the product
Cons
Drawbacks include:
- they cannot tell you for certain that your digital product was the cause of improvements for users
- if there is no difference in the outcomes before and after the intervention, it is difficult to interpret findings – it might be helpful to use qualitative studies to find out why this happened
How to carry out a before-and-after study
Start by planning what you want to find out and clearly state what outcomes you want to assess. Use your model of how your product will create change to help you.
It is important to recruit participants who are as similar as possible to the people you want to actually use your digital product. You will need to recruit enough participants to give you some confidence in your findings (read more about statistical power).
You want to assess and compare the outcomes before the introduction of the digital product and after the intervention period. You may want to assess the outcomes immediately at the end of the intervention period and later, to see if the effect continued after time has passed.
The length of the study will vary depending on the digital product and what you want to find out. For example, it might take a shorter time to show an increase in physical activity (behaviour outcome) than weight loss (the effect of doing physical activity).
Make the outcomes as specific as possible. For example, when assessing the effects of a digital toothbrush to encourage children to brush their teeth regularly, your outcome might be measured as total time (in seconds) the child brushed their teeth and the daily average time spent brushing the teeth over a certain time period (to assess regularity).
In some contexts, you will need to take into account that some change over time is expected anyway. For example, you could carry out a before-and-after study on a mental health app in a group of participants showing high levels of depression symptoms. However, some people with depression symptoms recover over time without any intervention. If participants in the study show an improvement from the before measurement to the after measurement, you would not know whether that is because of the app or because they would have shown some improvement anyway.
If there are 2 groups and one group receives the intervention and the other doesn’t, this is called a controlled before-and-after study. A control group can give you more confidence in your results by telling you what the typical change over time is. Instead of a control group, you can use a historical control: historical data that follows up some participants. This tells you what the typical change over time is, but will miss any current events that might have an effect.
Example: effects of an app intervention to help reduce self-harming in young people
See Stallard and others (2019): A Smartphone App (BlueIce) for Young People Who Self-Harm: Open Phase 1 Pre-Post Trial.
The team developed a mobile app to support young people who experience self-harming, as a supplement to face-to-face therapy.
They conducted a before-and-after study to assess the potential effects of the app. They recruited patients from specialist mental health services aged 12 to 17 years old who had a history of self-harming or were currently self-harming. They assessed the outcomes 3 times: at the start of the study, at 2 weeks (end of app familiarisation period), and again at 12 weeks (post-app use).
They gathered data on widely-accepted, self-reported measures – self-harming, depression, anxiety, behavioural issues – and also app-specific measures of safety and acceptability of the app. They also used existing data from clinical records to assess self-harming behaviour 4 weeks before each participant started the study.
They worked in collaboration with healthcare professionals who referred potential participants to the researchers.
Parental consent was obtained for those under 16 years old and those above 16 years old could consent themselves. They got approval from a research ethics committee.
Of 54 participants who were referred by healthcare professionals, 40 were eligible and 37 wanted to use the app after familiarising themselves with it.
In their analysis comparing the participants’ data at the start of the study and after 12 weeks of using the app, the team found that the app was highly acceptable and helpful to participants. They also found a decrease in anxiety and depression.
In this study, 3 participants did not find the app helpful because their distress was too strong to open the app. The team recommended that the level of distress should be assessed before introducing the app. Then the app could be tailored to those that might be most responsive to the intervention.
More information and resources
Robson and others (2001): Guide to evaluating the effectiveness of strategies for preventing work injuries: how to show whether a safety invervention really works. Chapter 3 (‘Before-and-after design: A simple evaluation design’) talks about the before-and-after study design and explains potential issues.
National Institutes of Health: Quality Assessment Tool for Before-After (Pre-Post) Studies With No Control Group. This discusses the criteria for a high-quality before-and-after study.
Examples of before-and-after studies in digital health
Martens and others (2019): Remediating Reduced Autobiographical Memory in Healthy Older Adults With Computerized Memory Specificity Training (c-MeST): An Observational Before-After Study. The team developed computer-based memory training for older adults and used a before-and-after study to find out about the effectiveness of the training.
Huurne and others (2013): Web-based treatment program using intensive therapeutic contact for patients with eating disorders: before-after study. In this study, the researchers evaluated the effects of a web-based programme to support recovery from an eating disorder.
Ruwaard and others (2012): The effectiveness of online cognitive behavioral treatment in routine clinical practice. This study evaluated the effectiveness of online cognitive behavioural treatment in a real world setting.