Define how your product works: evaluating digital health products
How to create a model of how your digital health product works and choose measures for your evaluation.
This page is part of a collection of guidance on evaluating digital health products.
Map how your product will work
To conduct an evaluation, you need to know what your product does and how you expect this to create the change you want. This can help you to identify gaps in the evidence for how your product works and to improve the design of the product. For example, how will your app designed to stop people smoking actually increase the proportion of people giving up smoking?
You might know this mapping process by names such as logic model, theory of change or benefit mapping. If you have already mapped how your product works, it is still useful to revisit this when planning your evaluation.
Why create a product model
A product model represents how a product is expected to work. This can be shown as a diagram to help you visualise the different processes and how they fit together.
It is a high-level view of the programme’s components and how you intend them to produce certain health outcomes. It doesn’t need to model the programme in step-by-step detail. The focus is on understanding how the digital product should work for the purposes of evaluating it.
Create your model in a workshop
Product models are usually designed by a team rather than individuals. You can do this in a workshop. The aim is to produce a living document that can be updated as your digital health product changes.
Creating your model in a workshop helps your team understand:
- what the project aims are
- how you will achieve your project aims
- what evidence you need to prove you have achieved your aims
- what you need to find out from your evaluation
Creating a product model may also be useful beyond planning an evaluation, for example:
- to help a development team agree on plans
- to help describe the product to future users or commissioners
When to run a product model workshop
We recommend developing your model as soon as you have an idea of the outcomes your product is aiming to achieve. This will help you evaluate your design process, as well as the product itself. However, you can also develop or revisit your model when planning your evaluation.
A product model workshop can be carried out:
- at an early stage in product development, as soon as teams have an idea of how they expect their product to work, to help shape the product. You can create your model late in the discovery phase if you are using an agile delivery process.
- at a later stage in product development, to help inform the evaluation plan. Do this in in the early stages of planning an evaluation.
How to structure a model for a digital health product
The different elements are usually shown in a simple flow chart from left to right, with arrows indicating how one step leads to another.
Functions lead to responses, which lead to outcomes (short-term, medium-term and long-term). The other elements – hypothesis, assumptions, context and barriers – sit around this central model.
You may have large-scale long-term outcomes in mind, but for the purpose of the model, keep the outcomes as specific and focused as possible. It’s important to identify specific outcomes that will be measurable.
Functions
Functions are things the product does to lead the user towards outcomes that will improve their health. For example, ‘the product breaks down the goal of increasing fitness levels into manageable steps’. Your product will probably have several functions, leading to different responses and outcomes.
Responses
Responses are how your user responds to the functions. This will often be a change in their behaviour. For example, ‘the user feels confident enough to start following the programme’.
Outcomes
How does the team expect their digital health product to improve the public’s health or wellbeing? Health outcomes are the changes they are trying to bring about in their users’ health or wellbeing. For example:
-
short-term outcome: user increases amount of exercise they do each week
-
medium-term outcome: user has improved fitness levels
-
long-term outcome: user maintains good fitness levels
Digital health products and services ultimately want to help the user to be healthier and live longer through long-term outcomes such as reducing their risk of diabetes, cardiovascular disease or cancer.
For this to happen, the user needs to attain some behavioural outcomes by engaging with your digital product. For example, they might increase their activity, lose weight, and better manage their stress levels. These are examples of medium-term outcomes.
Measuring engagement with your digital product, such as downloads and usage, is a short-term outcome.
Responses and short-term outcomes – proximal outcomes – are often easier to measure and easier to see a change on than longer-term outcomes – distal outcomes. Laying out which outcomes are which helps the team to decide between different sorts of evaluation, weighing the cost and difficulty of doing the evaluation against the rigour of its findings.
Hypothesis
Explain the problem that you are trying to solve or the opportunity you are exploring. Think about what your digital health product needs to do to be successful.
Assumptions
Consider any assumptions that you have about the users of your product. For example, you might assume they want to change their behaviour.
Organise the workshop
Decide who needs to be involved in your workshop. Involve key members of your team who will bring different expertise and perspectives. A group size between 3 and 8 works well. If you have a larger group, you may need to split into sub-groups for the exercises.
Arrange a time when you can meet for 1 to 2 hours. You’ll need sticky notes and pens. Print out the templates in the download below and stick them on the walls. You can also run a workshop online. There are several free online notepads that let groups collaborate on notes.
We’ve created an optional template for the workshop which you can download and print.
Introduce the activity
Explain the purpose of creating a model to show how your product is expected to work. Summarise what you will do in the workshop.
Stick up headings
Stick up the headings from the template on the wall. In the middle, from left to right, stick up functions, responses and outcomes (short-term, medium-term, long-term). Your product will probably have several functions, leading to different responses and outcomes.
Stick up the other headings – hypothesis, assumptions, context and barriers – around the central model. If you’re running the workshop online, create boxes or columns in your online notepad for each section.
Fill out sections
Use the headings (functions, responses, outcomes) and questions in the downloadable PDF template (link above) to guide you. Put your answers on sticky notes so they can be moved around easily.
Encourage workshop attendees to talk to each other as they do this, especially if you’re conducting the workshop online.
Approach the sections in roughly the following order. Consider the hypothesis, assumptions, context and barriers. Then think about the start and end points in how your product works: the outcomes and the functions. Once you have recorded these, consider how the functions lead to the responses and outcomes. Refer back to the hypothesis, assumptions, context and barriers when appropriate.
If your product is trying to change behaviour, you may want to describe how the functions of your product use appropriate techniques based on behaviour change theory. This is recommended by the NICE evidence standards framework for digital health technologies for tier C products. For example, the Behaviour Change Technique Taxonomy lists established techniques with examples.
Record the model
At the end of the workshop, you should photograph the model to capture the visual representation. Keep the sticky notes you have been working on. If you’re using an online notepad, export the file or take a screenshot.
After the workshop, type up your model. You should keep it in an editable format. The aim is to produce a living document that can be updated as the digital health product develops.
Use the model
Use your model to help design your evaluation. What evidence do you need to show that your product is working as intended? How will you measure this?
Case study: Couch to 5K
The Couch to 5K team created a model of how their product works at an in-person workshop.
They started with their hypothesis: the app would help users be more active and gradually work up to running 5K. They thought about how this would be broken down as individual health outcomes, from short to long term. For example:
- using the app makes people more confident about being active
- users start running as directed by the app, meaning they are successful at following the instructions
- this means users are more likely to make small steps towards being active
- this increase in physical activity improves fitness, motivation and confidence
- these changes, if sustained, would improve individuals’ long-term health
Next, they considered how the functions of their product would help users to reach those outcomes. For example:
-
function: app asks you to record when you have been for a run and provides a mechanism to do this
-
response: user feels encouraged by their progress and continues with the programme
- outcomes (short-term):
- user engages with app (downloads it and uses it)
- user increases level of exercise week by week
- outcomes (medium-term):
- user runs 5k
- user has increased fitness levels
- outcomes (long-term):
- user maintains good fitness levels
- user is less likely to suffer from poor health related to lack of physical activity
Decide what to measure
When planning your evaluation, you need to choose appropriate measures of your product’s effectiveness. Think about the health outcomes you expect from your product, such as making people more active. How will you know whether you have achieved them?
Health indicators are measurable signs of the impact your digital health product is having on people’s health and wellbeing. They include behaviours, attitudes and skills. For example, an indicator of improved physical fitness could be level of physical activity.
Choosing relevant indicators will help you to understand what impact your digital health product is having.
How to choose your health indicators
Prioritising health outcomes
If you don’t already know which health outcomes your digital health product is trying to influence, create a model of how your product works. The indicators you choose should be closely related to your chosen health outcomes. For each outcome you want to measure, you’ll need at least one indicator.
When deciding which outcomes to focus on, consider:
- what you can practically measure – think about cost, likely response rates, the burden on participants, complexity and what evaluation method you might use
- which outcomes are most important to your project’s aims
- the priorities of your organisation, stakeholders or partners
It might be useful to create a table or simple diagram to show how your indicators relate to the outcomes you included in your model of how your product works.
You may also want to collect information that doesn’t have indicators, for example, descriptive feedback.
Using existing indicators
Developing your own indicators can be time-consuming so start by searching for existing indicators. Using existing indicators also helps you compare your digital health product to other products. You can use the Office for Health Improvement and Disparities’ database of regional health data, Fingertips, to find common indicators in some public health areas.
Creating your own indicators
Indicators should tell you whether your product has achieved its aims. Looking at your chosen health outcomes, think about what will indicate that you have reached that outcome. Some indicators will be specific to your product, for example, number of downloads or number of people self-reporting closer to average body mass index (BMI) after using your product. Choose indicators that you can measure. You may already be collecting data that you can use.
Case study: Couch to 5K
When the Couch to 5K team were looking for relevant indicators, they started by searching Fingertips.
They found indicators relevant to their project, like the proportion of adults doing less than 30 minutes of physical activity a week. This indicator could help them prove that they’d promoted physical activity to their target audience. Using common indicators would also let them compare their app to similar services.
They also designed indicators specific to their app, using data collected from within the app itself – for example, the proportion of users who went for a run after getting a push notification.
Last updated 6 July 2020 + show all updates
-
Added details of how to run a workshop online.
-
First published.