Guidance

Get started: evaluating digital health products

How to begin with evaluating a digital health product, including when to evaluate, budgeting and getting support.

This page is part of a guide to evaluating digital health products.

What evaluation is

Evaluation is essential to finding out what works and what does not work, and why. It involves collecting data that enables you to assess the effectiveness of your product.

Challenges of digital evaluation

Non-digital healthcare products and services are routinely evaluated to make sure they are safe, effective and a good use of public money. For example, pharmaceuticals are rigorously tested.

Digital health products should also be evaluated, for the same reasons. Digital health poses particular challenges for evaluation but also offers opportunities. Traditional approaches to evaluation in healthcare may not work well. For example:

  • digital health products often change rapidly, which can be difficult to align with the time it takes to do some evaluations, but the speed of digital can also mean evaluations can be done more quickly

  • most digital health products are complex, with many different functionalities and features, so evaluations may focus on specific features or take a more global approach

  • the regulatory framework for digital health products is evolving: new methods of evaluation are also being developed and tested and it is not always clear what current best practice is – this guide will help you to navigate your options

Choosing your evaluation approach

Evaluation is a broad term covering a range of different methods. Some are cheaper and quicker; some are more expensive and involved. There is no one correct way of carrying evaluation out. However, some methods will give more useful results and some will be more approximate.

You will need to decide the scope of your evaluation. This will depend on:

  • what the purpose is
  • who your audience is
  • what stage you are at in creating your product

Also consider practicalities, such as:

  • who will carry it out
  • what resources you can give to it – time, money, people

Data collection or evaluation

Collecting data is not evaluation, although it is an important step in the process. Evaluation means taking your data and using it to assess the impact of your product and how that compares to your stated goals.

Consider when to evaluate

You can evaluate your product at different times in its lifecycle. You might want to evaluate it:

  • as you’re developing it, to work out how to make it better (formative evaluation)
  • when you’re launching it, or soon after, to find out whether it achieves its aims (summative evaluation)
  • when you’re are scaling up use or using the product in new contexts (implementation evaluation)
  • once your product is in use, to check it still works in practice and has no unintended consequences (post-implementation or ex-post evaluation) – in some contexts, this means monitoring its safety for the public (post-market surveillance)

Digital products are often developed iteratively – testing products repeatedly and improving them throughout the development. Evaluation of one version will act as a formative evaluation for designing the next version.

You may need to use different evaluation methods at different times, but evaluation is much easier if you start planning for it during the development of your product (in the discovery phase if you are using an agile delivery process).

Do I need to evaluate?

Yes – all digital health products should be evaluated. However, the scope of the evaluation can vary depending on the situation. If your product or service does something for which evidence already exists, a lower-level evaluation may be enough.

For example, if you have developed a game and already demonstrated in a thorough evaluation that it helps increase physical activity, then a light-touch evaluation would be enough to test new levels you add to the game.

What evaluation is needed

In some circumstances, you might want your evaluation to meet certain standards.

The National Institute for Health and Care Excellence (NICE) has developed the NICE evidence standards framework for digital health technologies. This suggests what evidence you should gather to demonstrate the value of different types of intervention. This is based on:

  • risk of harm to users
  • economic risk

If you are developing an app and want it to be made available on the NHS Apps Library, you will need to pass the NHS apps assessment process. This asks for specific types of evidence of the benefits of using your product. The assessment process is changing so visit the NHS Apps Library website for the latest information and the new Digital Technology Assessment Criteria.

BSI PAS 277 sets out quality criteria for developing, testing and releasing health and wellness apps.

If your product is classed as a medical device, it must meet the Medicines and Healthcare products Regulatory Agency (MHRA) guidelines. For some devices, you will need to follow the ISO 14155 standard for clinical investigation of medical devices for human subjects.

Get support

You may need help to carry out the evaluation. Sources of support will depend on the context you’re working in.

Knowledge-sharing services

Academic Health Science Networks (AHSNs) are regional hubs that encourage the NHS, higher education institutes, private sector organisations and other partners to work together and share knowledge. They promote health innovations. You can access support from the AHSNs and other organisations by registering on HealthTechConnect, which connects digital health technologies with the NHS and wider health and care system.

If you work in the NHS, the Global Digital Exemplars may provide guidance.

Other regional research alliances can provide support. For example:

Partnerships

You might want to carry out an evaluation together with another organisation, such as a university or healthcare provider. AHSNs can support you to do this.

If you want to work with a university, look for departments with digital expertise as well as expertise in the subject you’re evaluating.

There might be groups in your organisation with specific evaluation expertise who you could work with.

Paid external consultants provide a range of services, including help to develop your evaluation, collect evidence and analyse your data. These services might be supplied by contract research organisations (CROs), higher education institutes or freelancers.

You could also pay an external organisation to conduct the evaluation. If the evaluation of your digital health product is carried out by a group who are independent of you, this can make the evaluation more impartial.

Training

This will depend on the organisation you’re working for. Higher education institutes often offer training. You may be able to buy training from them, including bespoke training.

If you are working in the NHS, there is relevant training at various levels:

Funding

You might be able to apply for government grants from the National Institute for Health Research (NIHR) or UK Research and Innovation (UKRI). Some charities also fund studies.

The NIHR Research Design Service provides advice and guidance to researchers on all aspects of preparing grant applications. This includes support with study design, research methods and identifying suitable sources of applied health and social care research funding.

If you are working in the private sector, there is government support for research and development – for example, Innovate UK provides grants. There may be tax exemptions. There are also sources of private funding for start-ups – for example, angel groups, venture capital firms and crowdfunding platforms.

App development

If you are developing an app you want to be available through the NHS Apps Library, you will need to pass their assessment. The assessment process is changing – visit the NHS Apps Library website for the latest information and the new Digital Technology Assessment Criteria.

Budget for evaluation

As a rough rule, you should allocate 10% of your total project budget to evaluation. However, this may not be possible, or it may not be adequate for your project. Your evaluation budget should be proportionate to what you need to learn from your evaluation.

Evaluation costs can range from around £500 to several million pounds. Typical costs include:

  • staff salaries
  • equipment and materials
  • travel

If you are carrying out research in the NHS, there may be interventional costs. The NIHR website has costing templates for planning commercial trials in the NHS.

Example of an evaluation with a smaller budget

This is the budget for a micro-randomised trial of an app to see whether reminder messages to encourage engagement could be improved. In the trial, certain users of the app are randomised each day to receive a reminder message in the late morning or early evening.

Developer salary: Making changes to the app to get participant consent, carry out the randomisation and send messages at different times £2,500
Staff salaries: 1 staff member leading the evaluation (50% FTE); 5 staff members contributing to the work over approx. 6 months No budget – absorbed into usual business costs
Total £2,500

Based on Bell L, Potts H, Williamson E (2019), ‘Improving engagement in a health app: Considerations in designing a micro-randomised trial.’ Trials, 20(Suppl 1): P-17.

Example of an evaluation with a larger budget

This is the budget for a 4-year randomised controlled trial, including some initial work to check the feasibility of doing a trial. It will evaluate the effectiveness of an app to promote physical activity among people diagnosed with cancer. It is a standalone trial using an existing app.

Lead researcher (100% FTE throughout) £206,771.88
Trial manager (40% FTE throughout) £82,434.55
Statistician (100% FTE for 4 months) £20,519.50
Research assistant (100% FTE for 30 months) working on participant recruitment and data collection £108,947.39
Research assistant (100% FTE for 18 months) working on participant recruitment and data collection £66,359.64
Research associate (50% for 6 months) providing health economic input £13,212.00
Consultant nurse (5% FTE throughout) providing clinical input £14,227.27
Travel £23,400
Equipment (mainly wearables for collecting data from individuals) £47,504.13
User involvement £1783.20
Transcription £1,170
Data entry £5,220
Postage £28,486.80
Printing £383.04
Total £620,419.40

Based on Yorkshire Cancer Research investment for a project using a mobile application to promote physical activity after cancer.

Acknowledgements

Content written by Henry Potts (University College London), Flora Death (Public Health England) (PHE), Paulina Bondaronek (University College London), Manuel Gomes (University College London) and James Raftery (University of Southampton) with additional input from PHE.

Project advisory group: Ann Blandford, Jamie Brown, Tim Chadborn, Darren Curry, Kenny Deighton, Pall Johnson, Indra Joshi, Scott Mahony, Elizabeth Murray, Felix Naughton, Claudia Pagliari, Neelam Patel, Colin Pattinson, Quynh Pham, John Powell, Susannah Robinson, Ian Roddis, Mark Salmon, Tero Väänänen, Matthew Walmsley.

We would like to acknowledge the PHE team behind ‘Evaluating Digital Health Products’: Hasan Ali, Rachel Clark, Iain Cooper, Simon Dixon, Charlotte Fountaine, Felix Greaves, Pouria Hadjibagheri, Bernie Hannigan, Kassandra Karpathakis, Vicki Litherland, Ros-Mari Mitova, Fionnuala O’Toole, Claire Rackstraw, Bobak Saadat, Anya Zeitlin.

We would like to acknowledge Livework Studio’s contribution to the earlier phases of the project: Anton Drachuk, Lilli Graf, Claire Henderson (King’s College London), Gene Libow, Nick Manson, Ben Reason, Paul Sims. This work was solely funded by PHE.

Published 30 January 2020