A few years ago, I was afraid of updating the Ubuntu system, that our applications ran on. We chose the LTS (long-term support) versions of the OS releases so we wouldn’t have to do this too often. Essentially, we updated when we heard of new critical security updates or when we needed a new feature that was only available in a more recent version (think a Redis update or other services).
The most horrendous possible outcome of this update was the following: The server did not start anymore after rebooting it. Other very annoying outcomes included
- the application did not start
- data could not be loaded
- the application did not work as expected (there was a bug somewhere)
In these cases you had to choose between three options:
- Do nothing (this is always an option 😉)
- Try to locate and fix the problem
- Rollback to an earlier version
If you application has an error you can revert to an earlier version quite easily. You may have build artifacts that you deploy (think
jar in Java-land). For Ruby on Rails apps you can rollback to an earlier release. Capistrano offers to keep ~5 releases (or more if you tell it to do so).
Sidenote: I also had situations where I ran out of releases to rollback to, because the error was there all the time—we just didn’t know about it!
If the problem is not inside your application but in the area surrounding it (the application server, external libraries, network stack etc.) could you rollback there? I know I couldn’t have.
These days I suggest to keep as much information under version control as possible. This includes all the external libraries you are using (
Gemfile.lock) as well as the configuration of your servers, your services and databases and their specific versions. Depending on your deployment setup it still might be hard to recreate a certain environment for your application given these configuration points. But at least you know which version worked. You know the specific versions of all services that worked.
Tools like Puppet, Chef, Ansible and, of course, Docker and Kubernetes make it way easier to (re)create environments today. But that does not free you of the responsibility to know which configuration to actually use. For that you still need proper configuration management.
I am preparing a workshop for a client about this topic. They noticed their needs and want to improve their situation. If this is something that you would be interested as well, then let me know. I could give you some things to read about it or ask the right question so you could figure things out on your own?