How to begin when evaluating a digital health product, including when to evaluate, budgeting and getting support.
This page is part of a guide to evaluating digital health products.
What evaluation is
Evaluation is essential to finding out what works and what does not work, and why. It involves collecting data that enables you to judge the value of your product.
Challenges of digital evaluation
Non-digital healthcare products and services are routinely evaluated to make sure they are safe, effective and a good use of public money. For example, pharmaceuticals are rigorously tested.
Digital health products should also be evaluated. However, digital health poses particular challenges for evaluation. Traditional approaches to evaluation in healthcare may not work well. Challenges include:
digital health products often change rapidly, which can be difficult to align with the time it takes to do some evaluations
most digital health products are complex, with many different functionalities and features
the regulatory framework for digital health products is evolving. New methods of evaluation are also being developed and tested. It is not always clear what current best practice is
Approaches to evaluation
Evaluation is a broad term covering a range of different methods. Some are cheaper and quicker; some are more expensive and involved. There is no one correct way of carrying evaluation out. However, some methods will give more useful results and some will be more approximate. You will need to decide the scope of your evaluation. This will depend on:
- what the purpose is
- who your audience is
- what stage you are at in creating your product
Also consider practicalities, such as:
- who will carry it out
- what resources you can give to it – time, money, people
Data collection versus evaluation
Collecting data is not evaluation, although it is an important step in the process. Evaluation means taking your data and using it to assess the impact of your product and how that compares to your stated goals.
Consider when to evaluate
You can evaluate your product at different times in its lifecycle. You might want to evaluate it:
- as you’re developing it, to work out how to make it better (formative evaluation)
- when you are launching it, or soon after, to find out whether it achieves its aims (summative evaluation)
- once your product is in use, to check it still works in practice and has no unintended consequences (post-implementation or ex-post evaluation). In some contexts, this means monitoring its safety for the public (post-market surveillance)
Digital products are often developed iteratively – testing products repeatedly and improving them throughout the development. This means evaluation of one version will act as formative evaluation for designing the next version.
You may need to use different evaluation methods at different times, but evaluation is much easier if you start planning for it during the development of your product (in the discovery phase if you are using an agile delivery process).
Do I need to evaluate?
Yes – all digital health products should be evaluated. However, the scope of the evaluation can vary depending on the situation. If your product or service does something for which evidence already exists, a lower-level evaluation may be enough.
For example, if you have developed a game and already demonstrated in a thorough evaluation that it helps increase physical activity, then a lighter evaluation would be enough for testing the addition of new levels to the game.
What evaluation is needed
In some circumstances, you might want your evaluation to meet certain standards.
The National Institute for Health and Care Excellence (NICE) has developed the NICE evidence standards framework for digital health technologies. This suggests what evidence you should gather to demonstrate the value of different types of intervention. This is based on:
- risk of harm to users
- economic risk
If you are developing an app and want it to be made available on the NHS Apps Library, you will need to pass the NHS apps assessment process. This asks for specific types of evidence of the benefits of using your product.
BSI PAS 277 sets out quality criteria for developing, testing and releasing health and wellness apps.
If your product is classed as a medical device, it must meet the Medicines and Healthcare products Regulatory Agency (MHRA) guidelines.
You may need support to carry out evaluation. Sources of support will depend on the context you’re working in.
Academic Health Science Networks (AHSNs) are regional hubs that encourage the NHS, higher education institutes, private sector organisations and other partners to work together and share knowledge. They promote health innovations.
If you work in the NHS, the Global Digital Exemplars may provide guidance.
Other regional research alliances can provide support. For example:
- MedCity (London and Greater South East England)
- Northern Health Science Alliance (North England)
- Health Innovation Research Alliance Northern Ireland
- Life Sciences Scotland
You might want to carry out an evaluation together with another organisation, such as a university or healthcare provider. AHSNs can support you to do this.
If you want to work with a university, look for departments with digital expertise as well as expertise in the subject you’re evaluating.
There might be groups in your organisation with specific evaluation expertise who you could work with.
Paid external consultants provide a range of services, including help to develop your evaluation, collect evidence and analyse your data. These services might be supplied by contract research organisations (CROs), higher education institutes or freelancers.
You could also pay an external organisation to conduct the evaluation. If the evaluation of your digital health product is carried out by a group who are independent from you, this can make the evaluation more impartial.
This will depend on the organisation you’re working for. Higher education institutes often offer training. You may be able to buy training from them, including bespoke training.
If you are working in the NHS, there is relevant training at various levels. For new graduates, there are relevant streams in the NHS Graduate Management Training Scheme. There are also relevant streams in the Modernising Scientific Careers framework. The NHS Digital Academy is open to Chief Information Officers (CIOs), Chief Clinical Information Officers (CCIOs) and prospective CIOs and CCIOs.
The NIHR Research Design Service provides advice and guidance to researchers on all aspects of preparing grant applications. This includes support with study design, research methods and identifying suitable sources of applied health and social care research funding.
If you are working in the private sector, there is government support for research and development. For example, Innovate UK provides grants. There may be tax exemptions. There are also sources of private funding for start-ups – for example, angel groups, venture capital firms and crowdfunding platforms.
If you are developing an app you want to be available through the NHS Apps Library, you will need to pass their assessment. Their website has guidance on how the assessment process works. It also has guidance on commissioning health apps.
Budget for evaluation
As a rough rule, you should allocate 10% of your total project budget to evaluation. However, this may not be possible, or it may not be adequate for your project. Your evaluation budget should be proportionate to what you need to learn from your evaluation.
Evaluation costs can range from around £500 to several million pounds. Typical costs include:
- staff salaries
- equipment and materials
If you are carrying out research in the NHS, there may be interventional costs. The NIHR website has costing templates for planning commercial trials in the NHS.
Example of an evaluation with a smaller budget
This is the budget for a micro-randomised trial of an app to see whether reminder messages to encourage engagement could be improved. In the trial, certain users of the app are randomised each day to receive a reminder message in the late morning or early evening.
|Developer salary: Making changes to the app to get participant consent, carry out the randomisation and send messages at different times||£2,500|
|Staff salaries: 1 staff member leading the evaluation (50% FTE); 5 staff members contributing to the work over approx. 6 months||No budget – absorbed into usual business costs|
Based on Bell L, Potts H, Williamson E (2019), ‘Improving engagement in a health app: Considerations in designing a micro-randomised trial.’ Trials, 20(Suppl 1): P-17.
Example of an evaluation with a larger budget
This is the budget for a four-year randomised controlled trial, including some initial work to check the feasibility of doing a trial. It will evaluate the effectiveness of an app to promote physical activity among people diagnosed with cancer. It is a standalone trial using an existing app.
|Researcher salaries, including overhead costs||£512,471.23|
|Lead researcher (100% FTE throughout)||£206,771.88|
|Trial manager (40% FTE throughout)||£82,434.55|
|Statistician (100% FTE for 4 months)||£20,519.50|
|Research assistant (100% FTE for 30 months) working on participant recruitment and data collection||£108,947.39|
|Research assistant (100% FTE for 18 months) working on participant recruitment and data collection||£66,359.64|
|Research associate (50% for 6 months) providing health economic input||£13,212.00|
|Consultant nurse (5% FTE throughout) providing clinical input||£14,227.27|
|Equipment (mainly wearables for data collection from individuals)||£47,504.13|
Content written by Henry Potts (University College London), Flora Death (Public Health England), and Paulina Bondaronek (University College London), with additional input from the Public Health England team.
Project advisory group: Ann Blandford, Jamie Brown, Tim Chadborn, Darren Curry, Kenny Deighton, Pall Johnson, Indra Joshi, Scott Mahony, Elizabeth Murray, Felix Naughton, Claudia Pagliari, Neelam Patel, Colin Pattinson, Quynh Pham, John Powell, Susannah Robinson, Ian Roddis, Mark Salmon, Tero Väänänen, Matthew Walmsley.
We would like to acknowledge the Public Health England team behind ‘Evaluating Digital Health Products’: Hasan Ali, Rachel Clark, Iain Cooper, Simon Dixon, Charlotte Fountaine, Felix Greaves, Pouria Hadjibagheri, Bernie Hannigan, Kassandra Karpathakis, Vicki Litherland, Ros-Mari Mitova, Fionnuala O’Toole, Claire Rackstraw, Bobak Saadat, Anya Zeitlin.
We would like to acknowledge Livework Studio’s contribution to the earlier phases of the project: Anton Drachuk, Lilli Graf, Claire Henderson (King’s College London), Gene Libow, Nick Manson, Ben Reason, Paul Sims. This work was solely funded by Public Health England.