Patient-reported outcomes and experiences study
How to use a patient-reported outcomes and experiences study to evaluate your digital health product.
This page is part of a collection of guidance on evaluating digital health products.
Patient-reported outcome measures (PROMs) and patient-reported experience measures (PREMs) are used to assess the quality of healthcare experiences, focusing on patients. These measures help healthcare providers, commissioners and other stakeholders to make informed changes to their services.
Showing the benefits of your intervention to the patient and healthcare delivery is important if you aim to have your digital product or service embedded within the healthcare system.
The NICE evidence framework for digital health technologies (tier C products) recommends including patient-reported outcomes in comparative studies to provide evidence of the effectiveness of your digital product or service.
What to use it for
Use a patient-reported outcomes and experiences study when you want to show the impact of your digital health service on the quality of healthcare patients receive (summative evaluation).
Evaluating the effectiveness and experience of care using patient-reported outcomes is becoming standard practice in healthcare. It has been widely used in randomised controlled trials assessing the effect of new medications or medical procedures. You might want to use patient-reported outcomes to demonstrate the benefit of your digital product to the quality of healthcare in comparison to standard care.
Pros
Benefits include:
- there is a wide variety of high-quality PROMs and PREMs and some of them have been used extensively
- some of the standard measures are used to determine the cost-effectiveness of particular interventions
Cons
Drawbacks include:
- it can be challenging to select appropriate measures
- questionnaires can be burdensome for patients to complete, so it is important to balance between the need to show impact and burden
- you should use disease-specific measures if you are assessing the impact of your intervention on a particular health condition, but disease-specific measures don’t always exist
How to carry out a patient-reported outcomes and experiences study
Patient-reported outcomes and experiences are commonly collected using questionnaires. Use existing standardised measures if they already exist. For example, there is a current national PROMs programme in England which measures PROMs before and after 4 inpatient procedures. It uses the EQ-5D Index, a standard PROM which measures overall level of health and combines 5 aspects of quality of life. The data from the current PROMs programme in England is used to compare the quality of care of different trusts and is published on NHS Choices.
There are resources to help you choose standard outcome measure instruments, and tools that can help you assess the quality of PROMs (see ‘More information and resources’ below).
Broadly, PROMs evaluate either the outcomes of care or the process of care. PROMs look at the outcomes of care as perceived by patients, including safety and effectiveness. For example, they can cover changes in:
- symptoms
- pain or discomfort
- mobility
- health-related quality of life, which includes various aspects of wellbeing
PREMs assess the process of care. For example, patients’ experience of:
- the information they were provided with
- the level of trust in staff
- the waiting time
- hospital hygiene
Digital devices enable data collection beyond the traditional pen and paper. Many PROMs and PREMs are now available electronically, as ePROMs. You might send questionnaires via the patient’s smartphone, or using a kiosk or tablet in a hospital.
You could use PROMs and PREMs to help answer questions like: does using your digital service improve patient pain or mobility beyond the standard care? Does your digital product enhance the communication between hospital staff and patients when compared to standard care?
You should use existing measures if possible. If there are no PROMs available for the health condition your outcome relates to, you might want to develop a measure. Developing high-quality outcome measures takes time and involves making sure that that the outcome:
- measures what it is supposed to measure (validity)
- measures it consistently in different circumstances (reliability)
You should develop the measures in collaboration with patients and experts in the relevant health area. There are also companies that specialise in developing PROMs.
Example: patient-reported outcomes collected using digital technology
See Tran and others (2018): Utilizing Digital Health to Collect Electronic Patient-Reported Outcomes in Prostate Cancer: Single-Arm Pilot Trial.
The team developed an app to gather ePROMs. They used a validated and widely-used quality of life questionnaire specifically for patients diagnosed with prostate cancer.
They conducted a feasibility study to understand if gathering PROMs data through a mobile app was possible and acceptable to patients. They used a mixed methods design. The quantitative part assessed the completion of the measure and the qualitative interviews were used to identify obstacles to using digital technology to support patients with prostate cancer.
Participants were asked to complete the ePROM once a week for 12 weeks. To help them draw conclusions on the feasibility of the ePROM, the researchers categorised a satisfactory completion as when participants filled in 60% of the questionnaire over the study duration.
Although the researchers did not provide the number of participants who were approached, the main reason why participants did not enrol was that they did not own an iPhone (the app was built using Apple ResearchKit). Of 30 participants who consented to the study, one person did not finish any of the questionnaires and was excluded. Eighty-six per cent (25 of 29) of patients’ responses were considered as satisfactorily completed.
The themes identified in the qualitative analysis showed potential avenues for further focus:
- patients wanted to see more emphasis on emotional health rather than just symptoms treatment
- many patients used online patient communities and networks
- patients expressed concern over the use of their data and wanted to see more personalised digital tools
This study showed that gathering ePROMs electronically using an app is a viable method of collecting data from patients with prostate cancer. It has to be noted that most participants identified themselves as having high digital literacy levels, so these findings might not apply for patients who are less confident using technology.
Examples of PROMs in digital health
MacLean and others (2020): Coach-Facilitated Web-Based Therapy Compared With Information About Web-Based Resources in Patients Referred to Secondary Mental Health Care for Depression: Randomized Controlled Trial. The team assessed the effects of online therapy on depression in a trial.
Matsumoto and others (2019): Long-Term Effectiveness and Cost-Effectiveness of Videoconference-Delivered Cognitive Behavioral Therapy for Obsessive-Compulsive Disorder, Panic Disorder, and Social Anxiety Disorder in Japan: One-Year Follow-Up of a Single-Arm Trial. In this study, PROMs were used to assess the effectiveness of a remote therapy on anxiety.
Maninin and others (2018): Perception of Older Adults Toward Smartwatch Technology for Assessing Pain and Related Patient-Reported Outcomes: Pilot Study. The team wanted to find out if collecting PROMs using wearables was feasible in an older patient population.
More information and resources
COSMIN is a resource to help with the selection of high-quality health outcomes. It includes a database of reviews, checklist to appraise the measurement, and further explanation of measurement properties (reliability and validity).
Reviews of PROMs and PREMs conducted by the University of Oxford: Patient Reported Outcomes Measurement Group: Reports and Publications.