Guidance

Define how your product works: evaluating digital health products

How to create a model of how your digital health product works and choose measures for your evaluation.

This page is part of a guide to evaluating digital health products.

Map how your product will create change

To conduct an evaluation, you need to know what your product does and how you expect this to create the change you want. This can help you to identify gaps in the evidence for how your product works and to improve the design of the product. For example, how will your app designed to stop people smoking actually increase the proportion of people giving up smoking?

You might know this mapping process by names such as logic model, theory of change or benefit mapping. If you have already mapped how your product works, it is still useful to revisit this when planning your evaluation.

Why create a model

It’s useful to create a model to represent how a product is expected to work. This can be shown as a diagram to help you visualise the different processes and how they fit together.

It is a high-level view of the programme’s components and how you intend them to produce certain health outcomes. It doesn’t need to model the programme in step-by-step detail. The focus is on understanding how the digital product should work for the purposes of evaluating it.

How to structure a model for a digital health product

The different elements are usually shown in a simple flow chart from left to right, with arrows indicating how one step leads to another.

Functions lead to responses, which lead to outcomes (short-term, medium-term and long-term). The other elements – hypothesis, assumptions, context and barriers – sit around this central model.

Hypothesis

Explain the problem that you are trying to solve or the opportunity you are exploring. Think about what your digital health product needs to do to be successful.

Assumptions

Consider any assumptions that you have about the users of your product. For example, you might assume they want to change their behaviour.

Context

Describe the context for your users. For example, what support do they have and what are their motivations?

Barriers

Discuss any constraints that might prevent users from taking actions you would expect, or from using your product successfully. For example, users might lack the motivation to change their behaviour.

Outcomes

Think about how you expect your digital health product to improve the public’s health. Health outcomes are the changes you are trying to bring about in your users’ health or wellbeing. For example, you might want to improve their fitness levels.

Functions and responses

Functions are things your product does to encourage the user towards outcomes that will improve their health.

Responses are how your user responds to the functions. This will often be a change in their behaviour.

Consider how your digital health product, and its interactions with your users, could achieve your chosen health outcomes. What specific things will your intervention do to make them happen? You should also think about any negative impacts your product could have on health outcomes.

A template product model, as described in the webpage text

Create your model in a workshop

Product models are usually designed by a team rather than individuals. You can do this in a workshop. Develop your model as early as possible – as soon as you have an idea of the outcomes your product is aiming to achieve. This will help you evaluate your design process, as well as the product itself. Create your model late in the discovery phase if you are using an agile delivery process.

The aim is to produce a living document that can be updated as your digital health product changes. If you have already developed a model, it’s useful to revisit this when planning your evaluation.

Creating your model in a workshop helps your team understand:

  • what the project aims are
  • how you will achieve your project aims
  • what evidence you need to prove you have achieved your aims
  • what you need to find out from your evaluation
Decorative image of a team running a workshop to model how their product works

Organise the workshop

Decide who needs to be involved in your workshop. Involve key members of your team who will bring different expertise and perspectives. Arrange a time when you can meet face to face for 1 to 2 hours. You’ll need sticky notes and pens. Print out the templates in the download below and stick them on the walls.

We’ve created an optional template for the workshop which you can download and print.

Introduce the activity

Explain the purpose of creating a model to show how your product is expected to work. Summarise what you will do in the workshop.

Stick up headings

Stick up the headings from the template on the wall. In the middle, from left to right, stick up functions, responses and outcomes (short-term, medium-term, long-term). Your product will probably have several functions, leading to different responses and outcomes.

Stick up the other headings – hypothesis, assumptions, context and barriers – around the central model.

Fill out sections

Use the template headings and questions to guide you. Put your answers on sticky notes so they can be moved around easily.

Approach the sections in roughly the following order. Consider the hypothesis, assumptions, context and barriers. Then think about the start and end points in how your product works: the outcomes and the functions. Once you have recorded these, consider how the functions lead to the responses and outcomes. Refer back to the hypothesis, assumptions, context and barriers when appropriate.

If your product is trying to change behaviour, you may want to describe how the functions of your product use appropriate behaviour change techniques. This is recommended by the NICE Evidence Standards Framework for digital health technologies for tier 3a products. For example, the Behaviour Change Technique Taxonomy lists established techniques with examples.

Record the model

Once you are happy with your model, you could photograph it to capture the visual representation. Type up your model and share it with your team. Keep it in an editable format.

Use your model

Use your model to help design your evaluation. What evidence do you need to show that your product is working as intended? How will you measure this?

Case study: Couch to 5K

The Couch to 5K team created a model of how their product works at an in-person workshop.

They started with their hypothesis: the app would help users be more active and gradually work up to running 5K. They thought about how this would be broken down as individual health outcomes, from short to long term. For example:

  • using the app makes people more confident about being active
  • users start running as directed by the app, meaning they are successful at following the instructions
  • this means users are more likely to make small steps towards being active
  • this increase in physical activity improves fitness, motivation and confidence
  • these changes, if sustained, would improve individuals’ long-term health

Next, they considered how the functions of their product would help users to reach those outcomes. For example:

Function 1

App breaks down overall goal (running 5k) into manageable steps, gradually building up activities over 9 weeks

Response

User feels confident enough to begin programme

Outcome (short-term)

User increases moderate-intensity physical activity

Function 2

App asks you to record when you have been for a run and provides a mechanism to do this

Response

User feels encouraged by their progress and continues with the programme

Outcome (short-term)

User increases level of exercise week by week

Outcomes (medium-term)

User runs 5k

User has increased fitness levels

Outcomes (long-term)

User maintains good fitness levels

User is less likely to suffer from poor health related to lack of physical activity

Decide what to measure

When planning your evaluation, you need to choose appropriate measures of your product’s effectiveness. Think about the health outcomes you expect from your product, such as making people more active. How will you know whether you have achieved them?

Health indicators are measurable signs of the impact your digital health product is having on people’s health and wellbeing. They include behaviours, attitudes and skills. For example, an indicator of improved physical fitness could be level of physical activity.

Choosing relevant indicators will help you to understand what impact your digital health product is having.

How to choose your health indicators

Prioritising health outcomes

If you don’t already know which health outcomes your digital health product is trying to influence, create a model of how your product works. The indicators you choose should be closely related to your chosen health outcomes. For each outcome you want to measure, you’ll need at least one indicator.

When deciding which outcomes to focus on, consider:

  • what you can practically measure – think about cost, likely response rates, the burden on participants, complexity and what evaluation method you might use
  • which outcomes are most important to your project’s aims
  • the priorities of your organisation, stakeholders or partners

It might be useful to create a table or simple diagram to show how your indicators relate to the outcomes you included in your model of how your product works.

A diagram showing an example of outcomes with their corresponding indicators. Outcome: user increases level of exercise week by week. Indicator: continued user engagement with the app, shown by the app recording runs.

You may also want to collect information that doesn’t have indicators, for example, descriptive feedback.

Using existing indicators

Developing your own indicators can be time-consuming so start by searching for existing indicators. Using existing indicators also helps you compare your digital health product to other products. You can use Public Health England’s database of regional health data, Fingertips, to find common indicators in some public health areas.

Creating your own indicators

Indicators should tell you whether your product has achieved its aims. Looking at your chosen health outcomes, think about what will indicate that you have reached that outcome. Some indicators will be specific to your product, for example, number of downloads or number of people self-reporting closer to average body mass index (BMI) after using your product. Choose indicators that you can measure. There may be data that you are already collecting that you can use.

Case study: Couch to 5K

When the Couch to 5K team were looking for relevant indicators, they started by searching Fingertips.

They found indicators relevant to their project, like the proportion of adults doing less than 30 minutes of physical activity a week. This indicator could help them prove that they’d promoted physical activity to their target audience. Using common indicators would also let them compare their app to similar services.

They also designed indicators specific to their app, using data collected from within the app itself, for example, the proportion of users who went for a run after getting a push notification.

Published 30 January 2020