Guidance

Evidence - what is good impact evidence?

A summary about how to use evidence to find out if interventions reduce reoffending.

How do we know if an intervention is effective in reducing reoffending? How can we find out why some things work, while others fail to have an impact?

To answer these questions we need good quality, carefully controlled evaluation.

What is strong evidence?

Strong evidence helps us understand what is effective and what is not effective. Strong evidence comes from good quality research and evaluation studies, replicated many times across different groups. To be sure something works, it needs to include a matched comparison group to help determine what would have happened without the intervention. This helps rule out the possibility that the results were down to something else.

Evaluations should also be repeated on different groups of people and in different contexts. This can help show how useful the findings are and whether they can be applied to different people and settings. Evaluations should also be repeated because the results from one study alone could be a fluke, due to something specific to that study, or something unique about the participants.

Good impact evaluations:

  • are reviewed by experts (peer reviewed) and published
  • look at reliable and relevant outcomes, examples are reconviction or something research has shown is strongly related to reconviction
  • describes the intervention or service, as well as why and how it aims to help people change (a model and theory of change)
  • compare those who have been through an intervention with a similar group who have not (this is a control group). The best studies randomly allocate people to the intervention and control groups. Other studies match the intervention and control groups on important characteristics affecting reoffending or the outcome measure. These types of studies can also produce strong evidence
  • look at impact across large numbers of people closely resembling those who might receive the intervention in the future
  • account for ‘drop-outs’. These people start the intervention but fail to complete it. Good evaluations describe the outcomes for these people as well
  • identify appropriate conclusions based on the data

Good studies that explore why something does or doesn’t work:

  • are clear about the approach they take, and provide a rationale for why this is a suitable way to answer the question
  • provide information about what they did, with whom, and follow established steps for summarising and interpreting the information they get
  • outline the limitations of the study

Poor or unreliable evidence

People can provide compelling arguments and give interesting and persuasive examples of success. Methods which do not provide evidence of impact include:

  • case studies
  • before and after studies with no matched comparison groups
  • descriptions of the process someone went through
  • accounts or someone’s positive experience

These can give us some useful information, but don’t tell us about the impact of an intervention of service. They don’t give the full picture and can’t tell us how most people are likely to be affected. They don’t tell us if there could be any negative impact, either in the short or longer-term.

They don’t tell us what would have happened if the person had not received the intervention. This is why it is so important that a study has a control or matched comparison group.

Evidence is not:

  • Intuition or common sense. Many common-sense approaches to reducing reoffending have no impact. Worse, some have actually made it more likely that people will reoffend. Just because something makes intuitive sense, does not mean it will work and could even make people worse. This highlights the importance of rigorous, objective evaluation of interventions and services.

  • ‘Good practice’. This can be based on opinion and experience, rather than on good quality evaluation and research.

  • Asking people how they were changed by an intervention or service can be helpful. People can, however, be unreliable at assessing the causes of their feelings, thinking and behaviour. This is because they: tend to think they were worse than they really were before an intervention, and are likely to see a link between any perceived changes and the intervention, even if there isn’t any

  • Opinion, or a consensus of stakeholder views

Services and interventions with a strong evidence base have the best chance of reducing reoffending. They are based on sound, good quality, research. When innovating services, the only responsible approach is to include a well-designed, robust evaluation.

Further reading

Offending Behaviour Programmes and Interventions page Information on evidence based accredited interventions in custody and community

Evidence and Segmentation (NOMS - now HMPPS 2014) Looks at factors linked to reoffending / desistance and ways to address them

Transforming Rehabilitation: a summary of evidence on reducing reoffending MoJ (2013)

This page summarises the available evidence base and is informed by independent academic peer review. It does not represent Ministry of Justice or Government policy.

Published 15 May 2019