In today’s continuous integration/deployment oriented world, applications constantly evolve, and frequent changes make it confusing for the end user to keep track of the current version and status, especially if an upgrade involves different services and components.
In such a case, when following the traditional upgrade approach, one has to stop the application, redeploy it with the latest code, possibly include the update of the database schema, and prepare it for remote debugging, patching and rollback.
No wonder why zero downtime deployment is a much more favored deployment pattern, where applicable. In today’s post, we will see when and how we can implement it, and learn the pros and cons of leveraging Amazon Relational Database Service (RDS) for its execution.
Upgrade & Deploy, But How?
There are several ways to make deployments practically invisible for the end users. Regular application deployments that don’t involve database scheme modification are not so much of an issue. However, when you need, for instance, to add or remove some tables or columns, that’s when the real challenge begins.
In the zero downtime deployment approach, if you go with the blue/green deployment pattern, a new version of the application may not work with the old database scheme.
Similarly, if you update your database before the application deployment, the old version of the application may cause issues with the new database scheme. Amazon RDS will help you create database snapshots to protect yourself against data corruption, but the service has its weaknesses.
How do we overcome these concerns? Maybe the maintenance mode is the best option? What about database modification?
To answer these questions, we will go through several scenarios:
- Traditional deployment pattern with the maintenance window
- Blue/green deployment
- Zero downtime deployment with DB updates
Traditional Deployment Approach
With the traditional approach, the deployment process is quite simple. It consists of several uncomplicated steps:
- Stop the service
- Stop the application
- Create backups
- Update the database
- Migrate the data
- Deploy the new version of the application
- Start the new version
- Verify if the upgraded environment works as expected
- Reestablish the service
If anything goes wrong, you can still roll back and restore the service from the backups. Amazon RDS database makes it particularly easy to do so.
Creating snapshots, modifying instances, and deploying new instances from snapshots with RDS is very straightforward. Here’s an example of the panel for snapshot creation:
You can make all modifications in the database using the new DB instance and deploy the application with the new database. The potential rollback will be easier, as you won’t have to restore the entire database but only switch back to the previous one.
The same can be done with the entire application. You may deploy a new version to an entirely new environment, launched and configured from a CloudFormation template for instance.
Traditional Approach? Yes, But
One important thing to remember is that the above approach is perfect when your application and database are in the maintenance mode.
You have scheduled time for the upgrade and deployment, you switched off your service, your backups are in place, and you’re ready to roll. No end user will be able to connect to the service until it is prepared and confirmed working.
If you cannot suspend access to your service, you will need to look for alternatives.
Zero Downtime Deployment with Blue/Green Approach
The Blue/green deployment is a technique that involves running two identical environments. One environment is the current one (blue), while the other one is a version behind (green).
During the upgrade process, the deployment is made to the green environment, and once complete, the traffic is switched to the freshly-deployed version.
The following are the components of the above solution:
- Amazon Route 53 that allows you to switch traffic between the blue and green environments easily.
- Load balancers.
- EC2 instances.
- Amazon RDS database.
The blue/green approach is very flexible as both environments co-exist and can be deployed in parallel. If there are any issues with the new version, it can be easily rolled back to the previous one. The risk is negligible as traffic redirection is performed on Route 53.
However, this approach isn’t flawless, either. It has one vulnerability – shared database.
Blue/Green Approach – Two Environments, One Database
Even if you have two parallel environments running with a load balancer and DNS, they have a single DB behind. It doesn’t matter if it is a multi-AZ with plenty of read replicas because all data is amassed in a single database.
Application deployment steps to the next level of hard when we deal with a change in the database schema or database modification, so if database upgrade is required for your deployment, you have to look for a method other than blue/green.
Zero Downtime Deployment with DB Upgrade
The most significant challenge of zero downtime deployment is that there may be two different versions of an application running in parallel and the database has to be compatible with both.
- The first thing to remember is that the database schema has to be updated before the application deployment so that new version can use the DB with no issues.
- Now we need to migrate data from the old version to the new one. The new version (let’s call it V2) can be backward compatible with the old one (V1) and present less information in the context of old data.
- If there are more changes to the database, there are two possible ways to migrate data:
- After you launch services for V2.
- Before you launch services for V2.
The first approach is faster because you don’t have to wait for data migration, but the second approach is simpler.
If you migrate data after launching V2 services, V1 and V2 may become available online at once, and you will need to keep checking that all data written in V1 is migrated to V2.
The second approach requires you to migrate data to V2 before you switch application to that version. At the switch, data has been already migrated, and V2 can run seamlessly.
- You should avoid exposing two versions in parallel to the end users to avoid problems with data synchronization. As long as you maintain both versions, you need to keep migrating the new data from V1 to V2, which can be painful. It can get even more problematic in the other direction, as V1 may not be able to read data in the new V2 format.
Amazon RDS in Zero Downtime Deployment
Can you benefit from Amazon RDS in zero downtime deployment similarly as in the traditional approach? You can, but it is a double-edged sword.
Amazon’s service can make it easier for you to manage snapshots and roll back to a previous version. You can also launch a new database instance from a snapshot and force V2 to use it. Everything you know from the traditional approach with a maintenance window applies here, with one significant difference.
Bear in mind that your application is up and running during the migration, so all data is live.
The snapshot provides a kind of point in time recovery, so when rolling back, you will lose all data written to the database after the snapshot was taken.
One way to sidestep that issue is to operate on a master database copy and to synchronize data manually once the upgrade is done. Then you can switch the application to V2 with the new database and synchronize the missing data:
Zero Downtime Deployment or Maintenance Window?
Zero downtime deployment offers many benefits, but requires careful planning, especially if it involves a database scheme update. If so, it’s still feasible, but much more challenging:
- The new database has to be compatible with the old version and needs to be reflected in the application.
- You have to plan the data migration process.
- Using Amazon RDS may help, but requires extra caution as there may be more disadvantages than advantages of its application.
As a rule of thumb, regular blue/green deployments work best for application upgrades, but whenever you need to modify the database, the traditional approach with a maintenance window seems to be the safest and easiest option.