Deploying software regularly
You must release updates to your software regularly so you:
- avoid making changes to one part of your software system that cause other parts to fail (these are known as ‘breaking changes’)
- have constant feedback from users - this will help you design and build your system in an iterative manner
- create a resilient and easy to change system
Your software release cycle should be based on how your team can support updates.
Some teams release weekly, some release daily and some release constantly throughout the day.
You’ll have to explain how you’re doing this at your service assessments.
Principles for deploying regularly
When deploying software you need to:
- deploy little and often
- deploy quality software
- use auditable deployments
- use zero downtime deployments where possible
Deploy little and often
You should deploy software often and with small changes, so that:
- users benefit from new features or improvements quickly, instead of waiting months for an update
- users can give you feedback quicker, allowing you to iterate your software to meet their needs
- your team gets better at releasing (as they do it often)
- you identify bugs more easily because fewer things are changing at the same time - this means they’re cheaper to fix
- you can react quickly to urgent fixes, such as security updates and last minute changes
- your team can move faster through the Build-Measure-Learn loop needed for iterative development
You should measure how long it takes for code changes to hit production so you are aware of how regular your deployment cycle is.
Deploy quality software
To deploy quality software, make sure:
- deployments are consistent across environments - this gives you confidence that your testing is always relevant
- you automate the testing process of the software to routinely check for issues
Deployment should be a low-risk process. By the time you’re deploying a new version of software into production, you should be sure it’ll work.
To check the new version will work, you can:
- set up automated testing
- deploy your software to staging servers first
- use blue green deployment mechanisms
Regular deployments which use the same tools and technologies repeatedly and reliably give you confidence that your deployment mechanism works. This is because you’re constantly testing deployment.
Use auditable deployments
To keep track of deployment history and get immediate feedback on your deployment, you should:
- know which version of your service is running in each environment at all times
- be able to trace changes back to the initial code commits in the source code repositories after a deployment hits production
If you combine this approach with small and frequent releases, you can:
- narrow the source of any problems down to a small number of commits
- roll back to a previous version more easily
- confidently ‘roll forward’ with a code change to fix a production issue - you can be a lot more confident your fix will work if you’re making frequent releases
Aim for zero downtime deployment
Deployments that minimise or limit service disruption are known as zero downtime deployments.
These can take place during normal business day, which means staff don’t have to work unsociable hours.
Zero downtime deployments can:
- increase user satisfaction
- lower service operation costs - release windows are often in unsociable hours, and require hands-on skills from operations staff
- prevent difficulties when negotiating emergency releases - for example, for security or critical issues
You may think that setting up your application to do zero downtime deployments involves lots of work, but downtime can cost you money. You should find out the cost of downtime for your service and decide if you can accept that cost.
Use either of the following to carry out zero downtime deployments:
How to deploy regularly
It’s easier to deploy software regularly if you:
- build a single artifact rather than variations for different environments
- have multiple deployment environments
- manage variable configuration
- secure passwords and keys
- use smoke tests - software tests that check if the most important functions are working
Building a single artifact
An artifact can be:
- the compiled binary and its dependencies
- a .jar file for JVM languages
- a compressed archive of the source - not including compilation artifacts
- an entire virtual machine image with the application pre-deployed
You should try to build a single application artifact that’s deployed into each environment, from development machines through test into production.
Having the same application artifact deployed everywhere means that you:
- can be confident the artifact is tested in the same way as in production
- have a single artifact to represent a deployment, meaning it’s auditable and easy to roll back or upgrade
Teams often add debugging symbols, optimisations or test code solely to their pre-production environments. This makes it hard to have confidence in the artifact that will go to production.
You should keep all your artifacts in a central repository. Having one place for your artifacts allows you to:
- give your team access to a shared set of artifacts
- promote your artifacts across environments
- rollback or upgrade more easily
Multiple deployment environments
You should have multiple deployment environments so you can phase the rollout of your software and ensure it’s adequately tested.
At the very least you need a:
- development environment running the latest version of the software
- production environment hosting live users
You may also have a staging environment prior to production, or environments dedicated to:
- exploratory testing
- user testing
- performance testing
Using a pipeline
You may choose to manage the deployment of your software to your environments using a pipeline.
This will make sure a version of software isn’t deployed to a later environment before it has been deployed and tested at an earlier stage.
For example, GOV.UK deploys to its staging environment first so the development team can be confident the software does what it should before it’s deployed to production.
Deciding which environments to deploy
You also need to decide which of your environments you must deploy to in order to have confidence in your product. Emergency fixes can then skip unnecessary stages that aren’t part of the normal deployment pipeline so that development isn’t held up.
For example, a product owner shouldn’t have to sign off a security fix in a user acceptance environment.
The order of your environments doesn’t have to be strictly linear - you can run some sets of software tests in parallel, for example user acceptance tests and performance tests.
However, you may find there’s a single production environment that’s later than all others and a single entry point that precedes all others.
Managing variable configuration
To build a single artifact, your deployment mechanism should provide a way to inject configuration that varies between environments, like URLs of dependent services. For example, passwords and locations for external services like databases.
Your application should follow the 12 factor principles so that you can inject specific configuration for your environments.
Securing passwords and keys
To secure your software, you must take extra care when managing database passwords or Secure Sockets Layer (SSL) keys. Make sure:
- confidential information can’t be accessed outside of the environment it’s intended for
- confidential information is known only by the machines that need to know it For example, in a 3-tier app with database, application and web servers:
- the database server doesn’t need to know the TLS (transport layer security) private keys for the site
- the web server doesn’t need to know the database credentials
Using smoke tests after you deploy
Once you’ve deployed your software, you should check whether it’s working as expected using a ‘smoke test’.
If it’s not working, you can cancel or roll back the deployment.
A good smoke test should:
- be simple and fast
- exercise the software and all its essential dependencies
If your software needs a database to be present to operate effectively, your smoke test should exercise an application code path that will fail if the database is not present or returns an error.
If a smoke test fails
You should plan what to do if a smoke test fails - the simplest option is to manually roll back to a previous version of the application.
You could also have a system that automatically detects smoke test failures and cancels the deployment, or rolls back to the previous version.
Ideally the system you have won’t add the updated application to the production load balancer until it has been smoke tested and verified (if the application fails the smoke test, it’s discarded).
This ensures no rollback is necessary, and no interruption in service happens. This works particularly well with the immutable server pattern.
Get in touch with the web operations community directly to:
- discuss this page
- share ideas across government
- find support from teams that have worked on similar services
You may also find the Uptime and availability guide useful.