Randomised controlled trial: comparative studies
How to use a randomised controlled trial to evaluate your digital health product.
This page is part of a collection of guidance on evaluating digital health products.
A randomised controlled trial (RCT) tests the effect of a digital health product compared to an alternative. The alternative could be nothing, current best practice, or an alternative version of the product. Participants are recruited for the trial and then randomly assigned to one of 2 or more trial groups. This makes sure that participants in each group are comparable.
What to use it for
RCTs can prove whether a product has caused an outcome, for example, whether your app helped users to stop smoking. The NICE Evidence Standards Framework for digital health technologies recommends RCTs for evaluations of tier C products (broadly, these are digital products that seek to prevent, manage, treat or diagnose conditions).
RCTs can also be used during product development, for example, to compare different versions of a product.
Pros
Benefits include:
- they can produce definitive answers because randomisation makes sure that participants in each group are similar
- they are often needed to meet regulatory requirements
Cons
Drawbacks include:
- they are often expensive to do well
- choosing an appropriate comparison (the control) can be difficult
- they often require ethics committee approval
How to carry out an RCT
The main idea in an RCT is to have 2 or more trial groups in which everything is the same except for the thing you want to evaluate. This means any difference you observe in the outcome you are measuring must be because of what you did differently in the trial groups.
Recruit participants and allocate them to the different groups at random. This makes sure that the participants in each group are, on average, the same. It removes any bias in who is in which group.
Randomisation can be difficult. People involved in the study may try to undermine the random allocation deliberately or because they are biased without realising it. Computer randomisation or remote randomisation by an independent group can minimise this.
Each trial group receives a different intervention, although that can include doing nothing. In a factorial RCT design, you can look at combinations of interventions. Outcomes are assessed in the same way for all participants.
Study duration
Trial length varies. For example, in an A/B study, which is a type of randomised trial, you might direct users to one of 2 versions of a website and see which encourages most clicks on a particular link. The trial duration is just seconds. Most clinical trials, however, will have follow-ups over months.
Choosing controls and interventions
Choosing a control can be difficult. In some cases, the control condition will involve doing nothing to participants. One option is a waiting list control. If there is an existing service that participants are on a waiting list for, the 2 trial groups could be either on the waiting list (which would have happened anyway) or receiving the digital health product. After the trial period, all participants still receive the existing service.
If there is an existing intervention that works, it would be unethical not to give it to participants, so the control could be current best practice and the intervention could be the digital health product.
You might develop a minimal version of the digital health product as a comparison, so that all participants experience receiving something but only one group receives the full version of the product.
Before you carry out the study, you will often need to register a protocol of the trial stating what you will do. Usually, RCTs are used to test whether one intervention is better than another. You can also test whether a new intervention is as good as an existing intervention (equivalence or non-inferiority trials). For example, you might trial a digitally-delivered intervention against an existing non-digital service you already know is effective.
Who do you randomise
Normally, when carrying out an RCT, you randomise individuals to different treatments. Sometimes this is not practical. For example, a health service may find it impractical to offer 2 different treatment regimens to different patients. Instead, you can randomise some larger unit, eg different health centres. This is called a clustered RCT. Each health centre only delivers one treatment arm, which is simpler to organise, but which treatment arm is offered in which centre is still picked at random. The study is larger as you need multiple health centres in each arm.
A stepped wedge design (a type of clustered RCT) has several clusters, which again might be health centres. All the centres are currently delivering one treatment (the control) and are moving to delivering a new treatment. However, you can randomly allocate the centres to switch over at different times. So at any time during the study, some centres, picked at random, are offering the new treatment and the rest are offering the control treatment. The proportion offering the new treatment starts small and grows.
You can combine an RCT with a before/after trial in a crossover design, where individuals receive all the different treatments, but in a randomly allocated order.
Allocating participants to groups
All RCTs use random allocation to the different trial arms. This is often equal allocation: the probability of a participant going into each group is the same. This produces trial groups with the same number of people in them at the end, maximising the statistical power to make a comparison between the groups. However, we can use other allocation ratios, putting more people into certain groups than others. For example, you might do this if:
- one treatment is much more expensive than another
- there is a limited resource and only a certain number of participants can be put through one arm
In an adaptive trial, allocation probabilities dynamically change as the trial progresses. The most common reason for this is to balance the characteristics of the participants in the different arms. For example, you might want each arm to have equal numbers of older and younger participants.
Response-adaptive randomisations change the allocation probabilities based on the results to date. Imagine having 10 variants of a digital intervention and a corresponding 10-arm trial. As the trial proceeds, if how a participant responds to the intervention can be determined quickly, you might start to see that some variants are performing better or worse than others. You can change the allocation probabilities so that new participants are more likely to go into the arms that have good results than the arms with poor results. This means more participants experience the more successful versions of the intervention and gives us more statistical power to talk about the more successful versions. These trials end up with unequal numbers of participants in different arms and require complex statistical analysis.
Example: StopAdvisor
Brown and others (2014), Internet-based intervention for smoking cessation (StopAdvisor) in people with low and high socioeconomic status: a randomised controlled trial
StopAdvisor is an interactive website for smoking cessation. The team carried out an RCT from December 2011 to October 2013 with 4,613 participants. These were randomly assigned (1:1) to StopAdvisor or an information-only website.
Individuals were recruited through a website, which also carried out the random allocation. Participants were assessed as of low or high socioeconomic status. The researchers thought the website might be more useful for people with low socioeconomic status, so they recruited enough participants for two trials, one for people of low socioeconomic status and one for people of high socioeconomic status.
The researchers had to invent a control website. To do this, they created a simple, one-page static website giving brief standard advice, based on existing and widely used materials.
The main outcome was abstinence from smoking, biochemically verified, at 6 months. Follow-up data was obtained by emailing participants. If a participant reported a successful quit attempt, they were sent a dental roll to provide a saliva sample, which was posted back. This was analysed using a standard method to demonstrate abstinence. If a participant did not respond, it was assumed they were still smoking.
Participants knew they were in a trial and could see the website allocated to their trial group, but they did not know what was happening in the second group. The researchers who analysed the saliva samples also did not know which group participants were in. These are forms of blinding and they reduce any chance of bias arising because of individuals’ expectations that one alternative will be better.
The study was registered as an International Standard Randomised Control Trial, number ISRCTN99820519. The study was approved by a university ethics committee.
Types of randomised controlled trials
Read more about specific RCT designs:
Factorial randomised controlled trial
Crossover randomised controlled trial
More information and resources
The Revised CONSORT Statement for Reporting Randomized Trials. This explains what you should report when describing an RCT. It can also be used as a checklist for designing a trial.
Bricker and others (2014), an RCT of 2 different smartphone apps for smoking cessation.
Garnett and others (2016), a protocol for a factorial RCT. The results of the study were published as Crane and others (2018).
Gielen and others (2018), an RCT to evaluate an educational app on car seat use.
Hemming and others (2015), an introduction to the stepped wedge design.
Klasnja and others (2018), a micro-randomised trial of whether an app to promote physical activity should offer activity suggestions.
Patel and others (2016), a crossover trial comparing different technology to support communication between clinicians.
Miloff and others (2016), a study protocol for a non-inferiority trial of virtual reality versus traditional exposure therapy to treat spider phobia.
Last updated 4 October 2021 + show all updates
-
Added explanation of cluster and stepped wedge designs and unequal allocation of participants.
-
First published.