Measuring the benefits of your service
Building the wrong thing can be very costly. It wastes users’ time, causes frustration and leads to people getting in touch via less cost effective channels.
It’s sometimes hard to make the case for building the right thing - especially if you’ve been funded to address a problem in a specific way.
The best way to overcome this is to explore different options and make sure you’ve got a deep understanding of:
- the benefits a particular solution delivers
- how those benefits help address wider departmental or governmental objectives
You can use the fact that you’re working in an agile way - and are constantly learning about your users and their behaviour - to your advantage.
Benefits realisation is a team sport, but it’s usually owned by a product manager or service owner.
If you’re looking to build a business case for spend approval from HM Treasury (HMT), refer to the Green Book guidance.
Discovery: spot problems and estimate how much they’re costing
Your discovery research should help you understand who your users are and identify problems in the area you’re working in. These could be things like:
- poor user experiences that cause people to use less cost effective channels
- inefficient or time consuming processes
- legacy systems that need replacing
As you do this, it’s useful to work out how big the problems are - how long it takes users to do a thing or how much a process is costing, for example.
It’s important to establish this baseline figure. You can only know if you’ve improved things by the end of your project if you know where you’re starting from.
To achieve this, your user researcher should work with an analyst (an economist or business analyst, for example) to come up with research questions that will help you quantify these things.
Your team can combine your research findings with insights from desk-based research (like academic articles or ONS data) to build a clearer picture of the size of the problem.
You can use a business model canvas to weigh up costs against benefits to users and to help you understand:
- the problems you’re solving
- if you’re providing value to users
It’s fine if you’re using assumptions to make estimates at this point. You’ll have the chance to test these assumptions as you move through the development phases.
Download an example of a team’s baselining work during their discovery (.ods).
Estimate how much it’ll cost to run an alpha
Once you’ve got an idea of the biggest problems, you can start thinking about how much it might cost to put a team together and run an alpha.
When you’re estimating, think about the number of people you might need and their salary costs, whether you’ll need any non-civil servant support and any software, hosting or technology costs you’re likely to incur.
Download an example of how to estimate delivery costs (.ods) for alpha.
Alpha: test different solutions
Having spotted and sized problems you think are worth solving, you should spend the alpha phase exploring how to solve them.
Test different ideas and scrutinise your riskiest assumptions. This should help you figure out how difficult it’d be to deliver those ideas for real and understand the benefits each solution would deliver.
At this stage, it’s still okay to base estimations of benefits on assumptions. You’ll gather data throughout your alpha that’ll help you test and validate them.
Any of the following sorts of things would be considered benefits.
Cashable benefits
Cashable benefits are changes that result in your organisation having more money to spend, either through savings or through additional revenue.
Non-cashable benefits
Non-cashable benefits are changes that don’t lead to an immediate benefit, but address problems you’d have had to fix eventually. They include things like saving money in future budgeting periods, or avoiding future procurement costs.
Wider economic benefits
These improve things for your users or other groups outside your organisation and include things like:
- saving users time or improving their experience
- reducing private sector costs or the burden on charities
Benefits don’t have to be financial. It’s okay to focus on realising qualitative benefits - especially if your end goal is something like improving user experience.
Show how fixing the problems would address wider strategies
During your alpha, it’s useful to think about what might happen if you were able to fix one or more of the problems you identified during discovery. And, if possible, how addressing them might help your organisation meet its wider strategic objectives.
This helps you understand the wider context you’re working in and demonstrate the value of your work to stakeholders.
One way of doing this is to create a benefits map. This helps you track the impact of fixing some of the problems and the changes this might bring about.
Creating a benefits map
Follow these steps to put your map together.
-
Get your team and stakeholders together.
-
Identify the wider departmental goal or strategy your project has been set up to address. This is known as your ‘strategic objective’.
-
Work out the things you’d need to do to meet that strategic objective - for example making a process more secure or easier to use. These things are known as your ‘end benefits’.
-
Figure out what you’d need to do to realise your end benefits - for example improving accessibility. These things are known as ‘intermediate benefits’.
-
Identify the changes you’d need to bring about to make your intermediate benefits possible. For instance, to improve accessibility, you might need to develop a new user interface. These are known as ‘enabling changes’.
-
List some immediate, tangible deliverables for your project. For example, building a new system.
-
List the problems or opportunities you identified at discovery. These are your ‘drivers’ and could include things like low user satisfaction.
You can then plot these benefits into a flow diagram, starting with your drivers and ending with your strategic objective, to show the knock-on effect of your work.
It’ll be much easier to make the case for doing your work a certain way if your solution not only makes things better for users, but helps government deliver the things it’s committed to deliver.
Create an economic model at the end of alpha
By the end of alpha, you’ll have an understanding of the size of the problem and a few potential solutions.
Create an economic model to help you:
- summarise what you’ve learned up to this point
- choose which solution you take forward to beta
The economic model should show an estimate of the costs and benefits of your chosen solution in beta and in live.
There are a few principles you should follow when doing this.
Show how much you’ll improve things by
Work out the difference between the baseline figure you identified during discovery and your estimate of how much you might be able to improve things by.
Repeat this process for each of the solutions you explored at alpha.
For example, if it’s currently taking users an average of 45 minutes to submit a claim, but your work will lead to it taking 15 minutes, then you’re saving users 30 minutes on average.
Or, if a process is currently costing £10 per transaction and you think it might cost £4 per transaction once you’ve finished your project, then the benefit is £6 per transaction.
It’s human nature to overestimate things. This is called ‘optimism bias’.
Make sure to be realistic about what benefits you can deliver given your budget and the amount of time you’ve got. This might mean saying you’ll deliver a smaller number of features by public beta than you think you could if things went perfectly.
It’s also important to think about what happens if things don’t go to plan once you’re in public beta. So make sure to think about all possible outcomes. For example, what would happen if:
- take up of your service was as low as it could conceivably be
- take up of your service was as high as it could conceivably be
This is known as ‘sensitivity analysis’.
Scrutinise your assumptions
The second part of sensitivity analysis involves scrutinising the assumptions your predictions are based on. This is particularly important early on in a project when your assumptions might not be based on much research.
For instance, predicting that you’ll save £6 per transaction could be based on assuming that:
- it’ll take a certain amount of time for contact centre staff to deal with an application or claim
- a certain number of users will need to contact you via offline channels
Tweak some of these numbers and see how this affects your economic model. For instance, what would your cost per transaction be if contact centre staff were able to process claims in 5 minutes rather than 10?
Focus on scrutinising your riskiest assumptions. Include this sensitivity analysis in your economic model.
Estimate how much your team will cost in beta and live
Include your estimated delivery costs for each of your potential solutions. This means thinking about the team you’ll need and any other costs (such as hosting or licensing) you’d incur.
Sometimes projects are impacted by issues you couldn’t possibly have predicted. So it’s worth overestimating how much a project will cost or how long it’ll take to deliver.
Download an example of an economic model (.ods) for beta and live.
Advocate for the right solution
Following this process can help you make the case for building the right solution, especially if you’ve been given a project brief you don’t agree with.
It gives you a firm idea of the costs and benefits of your potential solution, which you can compare to the thing you’ve been asked to do.
Beta and live: monitor and evaluate your project
Agree a set of metrics to track at the start of beta (including benefits) with stakeholders. These could be things like the number of people using your service, volumes or cost and should relate to the end benefits and strategic objectives you listed in your benefits maps.
In order to monitor and evaluate benefits being delivered you may want to gather data from your users through:
- user research
- case studies
- questionnaires
- analytics data
Make sure to ask the questions in a way that means the user only gives you information about the benefits of your project.
This will help you demonstrate how the benefits compare to what you forecast in your end of alpha economic model.
You should have decided how you’re going to gather data before you move into private beta - and should make data gathering and iteration a constant part of the beta and live stages.
Iterate your models
Working in an agile way means you can test things and get rich data fairly quickly - especially in beta and live when your service is being used by a high number of users.
Feed your findings back into your economic model and benefits map after each iteration period. And turn any success stories into case studies - these are an excellent way of showing the value of your service, especially to stakeholders.
Decide whether you should continue your project
At the end of beta you should have an iterated economic model and benefits map that demonstrate the value for money your service will deliver if progressed into live.
You should also have data showing how the benefits compare with the forecasts outlined in your end of alpha economic model. Support this data with case studies and user research findings.
All of this information should help establish whether it’s worth continuing your project into live, or whether you should stop or change what you’re doing.
- Last update:
-
Guidance first published