Choose evaluation methods: evaluating digital health products
Compare different evaluation approaches and choose an appropriate method.
This page is part of a collection of guidance on evaluating digital health products.
These resources look at 4 main types of study. If you are unfamiliar with them, read about each one before choosing your method. You can also combine methods.
1. Descriptive studies
Understand how things are going.
These types of studies describe what is going on. They can give you descriptive statistics (what proportion, how many, how much), or investigate correlations: relationships between variables.
Descriptive studies are weaker at providing evidence of cause and effect. They can’t tell you what would have happened if you had done something else, for example, if a different product would have worked better.
Descriptive studies are often simple. For digital products, they can be quick and cheap if you can use data that is already being collected. Sometimes you will want to collect additional data specifically for the evaluation. This may cost more.
Most descriptive studies use quantitative methods, but some are qualitative or incorporate qualitative methods.
When to use them
Use them when:
- your product is in use, to check it works and has no unintended consequences (ex-post evaluation)
- your product is about to be in use and can be tested with a pilot group to see how it works in practice
Questions they answer
How many people are using your product?
How many people stay engaged with your product?
How often do they use it?
Do they like the product?
Descriptive study methods
Analysis of routinely collected data
Behaviour change techniques review
2. Comparative studies
Check whether your product works.
To judge whether a digital health product or service is effective, you need to make a comparison to what the alternative would be. If this product didn’t exist, what would have happened? The alternative might be nothing, an existing non-digital service, or a different digital product or service.
For example, a group of people who have some anxiety problems use your new app. Six months later, 30% report they no longer have anxiety problems. Does this prove that the app is effective? No. Some people recover from anxiety problems over time without outside assistance. You need to compare whether more people recovered using the app than would have without the app.
Comparative studies usually involve collecting quantitative data on individuals who are not given your product. This is more complex and expensive than a typical descriptive study.
The choice of comparison is important. Some comparisons are easier and cheaper to make, but may not be fair. You will have to find a balance between a very robust evaluation and what is practical and affordable. Some types of comparative study are very robust. They are recommended in the NICE Evidence Standards Framework for several types of digital product.
When to use them
Use them when:
- you want to know how effective your product is, for example, in comparison to not using your product or to using another product
- you are launching your product, or soon after (summative evaluation) and want to know how effective it is
Questions they answer
What is the difference in health outcomes between people using your product and people using a rival product?
What is the difference in health outcomes between people using 2 versions of your product?
What is the difference in health outcomes between people using your product now and how people did before your product was available?
Comparative study methods
Crossover randomised controlled trial
Factorial randomised controlled trial
3. Qualitative studies
Understand how people feel about your product.
Qualitative approaches can be good for understanding the thoughts and experiences of users of the digital health product, or of other stakeholders. Qualitative methods can give a richer explanation of what is happening. For example, a quantitative approach could describe patterns of disengagement with an app and what demographic factors predict disengagement, but a qualitative approach can describe why users stopped using the app. It could also focus on giving a voice to particular users who might otherwise be overlooked in evaluations. Qualitative studies may be more subjective than quantitative studies.
While qualitative studies can describe the range of views among participants, they are weak at saying how many participants hold different views. Results from qualitative studies are hard to generalise to other participants. The number of participants needed can be much lower than for other evaluation methods. However, the amount of information you get from each participant can be much higher, so usually more time is spent with each participant.
When to use them
Use them when:
- you want a more in-depth understanding of users’ thoughts and experiences of your product
- you’re developing your product, to work out how to improve it (formative or iterative evaluation)
- you have open-ended questions
Questions they answer
What was users’ overall experience of the product?
How did users feel during their journey through the product?
What do users want to see in a new version of the product?
Qualitative study methods
4. Health economic studies
Understand what value for money your product provides and the cost implications of implementing it.
Health economic studies typically involve 2 broad types of economic analyses: economic evaluation and budget impact analysis.
Economic evaluation can help you assess the costs and effects of a digital health product and compare it with alternative options. It provides information about the relative effectiveness and cost-effectiveness of your product to help inform decisions about which digital health interventions to provide. Economic evaluation enables decision-makers to make efficient use of available resources for improving a population’s health. Pursuing efficiency involves maximising the benefits of the available resources, not just reducing costs.
Budget impact analysis evaluates the affordability of a digital health product given the available budget constraints. This involves a comprehensive assessment of the costs and savings of implementing a new digital health product. Budget impact analysis usually complements value for money assessments and is often conducted after an economic evaluation.
Health economic studies can tell you whether the benefits of investing in your product exceed the benefits you would have got from investing the same amount in an alternative option. For example, if you are launching a new mobile app to facilitate remote GP consultations, economic evaluation will tell you whether your product provides most benefits for each additional pound invested in it compared to competing options. The alternative options could be other mobile apps or face-to-face appointments. It will also tell you about the likely cost implications of rolling out your product within the decision-maker’s available budget.
When to use them
Use them when:
- your product is being launched or tested in a pilot study to help guide decisions about whether it should be adopted
- you want to understand how your product compares with competing alternatives in terms of both benefits and costs
- you need to know whether your product offers the most benefits given the available budget
- you need a more comprehensive assessment of the broader economic implications of your product, and its impact on the decision-maker’s budget
Questions they answer
How much benefit is gained for each additional pound invested in your product?
How cost-effective is your product compared to alternative options when competing for local or national resources?
Is the adoption and spread of your product feasible within the decision-maker’s available budget?
Economic study methods
Approaches that cut across categories
The categorisation in this guide is one way of classifying different methods. Some approaches cut across this distinction. You can use qualitative or quantitative methods or both, depending on the objectives of your evaluation. Mixed methods evaluations combine quantitative and qualitative methods – for example, you could collect data on everyone using a product in a before-and-after study and also interview a subset of users in more depth.
Cross-category methods
Ecological momentary assessment
Last updated 19 May 2021 + show all updates
-
Added link to case-control studies under comparative studies.
-
Added information on evaluation methods that straddle categories.
-
First published.