CI/CD Rollback Strategies

A team does not have a deployment strategy unless it has a rollback strategy.

This is where a lot of pipelines are dishonest. They automate build and deploy, but rollback remains a vague sentence in a runbook. Under pressure, rollback must be obvious.

Why rollback deserves first-class treatment

Releases fail for predictable reasons:

• bad config
• missing secret
• startup regression
• migration mismatch
• dependency incompatibility
• resource pressure
• latent bugs triggered by production traffic

None of these become less real because the pipeline had a green checkmark.

Common rollback strategies

There are three common ways to handle rollbacks. The right one depends on your infrastructure and codebase.

1. Revert and Redeploy (The Standard Way)

If a release causes issues, you revert the bad commit in Git and run the deployment pipeline again.

• How it works: Git revert ➔ git push ➔ pipeline builds a new image ➔ deploys.
• Pros: Simple to understand. No complex infrastructure needed.
• Cons: It is slow. Building, testing, and deploying takes time (often 5 to 15 minutes). If production is down, waiting is painful.

2. Version Toggling (The Fast Way)

Instead of building a new image, you keep the previous version of your application running and route traffic back to it.

• How it works: In platforms like ECS or Kubernetes, you change load balancer traffic weights or container targets to route requests back to the previous deployment.
• Pros: Near-instant rollback (takes seconds).
• Cons: Requires running both versions of the application at the same time during the release window.

3. Feature Flags (The Safe Way)

You deploy the code behind a conditional configuration switch (a feature flag). If the new feature breaks, you toggle the flag off without deploying any code.

• How it works: Code uses an if condition: if (features.enableNewCheckout). If payment failures spike, you set the flag to false in a dashboard.
• Pros: No deployment pipeline needed. Disables the broken code path instantly.
• Cons: Adds conditional code complexity, and you must clean up old flags later.

Handling the Database (The Real Challenge)

No rollback strategy works if database schema changes are not backward-compatible. If you deploy a database change that deletes a column, and then roll back your code to the older version, the older code will crash because it expects that column to be there.

Every database migration should be split into backward-compatible steps:

1. Add new columns/tables first, while keeping the old ones.
2. Deploy code that writes to both old and new columns.
3. Deploy code that only reads from the new column.
4. Delete the old columns/tables in a separate, later release.

Final thought

Deployment speed is overrated if rollback is clumsy. Good release engineering is not about pushing faster than everyone else. It is about recovering faster, with less confusion, when reality does what it always does: refuse to follow the happy path.