How to Roll Back a Bad Release Quickly

The moment a deploy goes wrong

Error rate spiking, users complaining, and the deploy you just shipped is the obvious cause. The instinct to debug in production is the wrong one. The right move is almost always: roll back first, debug after. Get back to a known-good state, then investigate calmly.

This is a playbook for rolling back fast and safely — including the part everyone forgets: database migrations.

The mental model: deploys are versioned artifacts

A good platform keeps every deploy as an immutable, versioned artifact. Rolling back is just re-pointing traffic at the previous artifact — no rebuild, no waiting. This is why "redeploy the old commit" is slower and riskier than a real rollback: rebuilding can pull different dependencies and takes minutes you don't have.

PandaStack keeps deploy history (10 days on free, 30 on Pro, 90 on Premium) so you can roll back to a previous successful deploy without rebuilding.

Step 1: confirm it's the deploy

Before rolling back, take 30 seconds to confirm causality (rolling back an unrelated issue wastes time):

• Did the error rate jump *at the deploy time*? Check metrics.
• Are the errors in the live logs related to the change?
• Is it all users or a subset?

If the timeline lines up with the deploy, roll back.

Step 2: roll back the application

On PandaStack:

1. 1Open the service → Deploy history.
2. 2Find the last successful deploy before the bad one.
3. 3Click Rollback (or Redeploy this version).
4. 4The platform re-points traffic to that artifact — typically seconds, since it's already built.

Watch the live logs and metrics confirm recovery. Communicate in your incident channel: "Rolled back myapp to , recovering."

Step 3: the database migration problem

This is where rollbacks get dangerous. If your bad release ran a database migration, rolling back the *code* doesn't undo the *schema change* — and the old code may not understand the new schema (or vice versa).

The rule that prevents this pain: make migrations backward-compatible (expand/contract pattern). Never ship a migration that the previous version of the code can't tolerate.

Expand / contract

Instead of a breaking change in one deploy, split it across deploys:

Goal: rename column `username` -> `handle`

Deploy 1 (expand): add `handle`, write to BOTH columns, read from `username`.
Deploy 2: switch reads to `handle`.
Deploy 3 (contract): stop writing `username`, then drop it.

At every step, the previous deploy still works, so a rollback is safe. The naive single-deploy rename is exactly what makes rollbacks impossible.

If you already shipped a breaking migration

If the bad release dropped or renamed a column and you must roll back:

• Additive migrations (added a column/table): safe to roll back code; the new column is just unused.
• Destructive migrations (dropped/renamed): rolling back code may break. You may need to restore from backup or write a forward-fix migration instead of rolling back. This is why destructive migrations should never ship in the same release as the code that needs them.

This is the strongest argument for backward-compatible migrations: it keeps rollback as a one-click option.

Step 4: roll back config and env vars too

A "bad release" isn't always code — sometimes it's a changed env var or feature flag. If you flipped a flag or changed config, revert that too. Flags are often the *fastest* rollback: flip the flag off, no redeploy needed.

Step 5: verify recovery

• Error rate back to baseline?
• Live logs clean?
• A manual smoke test of the broken flow passes?
• Update the incident channel: "Recovered."

Step 6: post-incident — prevent the next one

Once you're stable, *then* debug. And put guardrails in place:

• Backward-compatible migrations (expand/contract) so rollback is always safe.
• Health checks that fail a deploy before it takes all traffic.
• Gradual rollout / canary if your platform supports it, so a bad deploy hits 5% before 100%.
• Fast rollback drills — practice it so it's muscle memory during a real incident.
• Deploy notifications so the team knows when something shipped.

Quick reference

BAD DEPLOY PLAYBOOK
1. Confirm: did errors start at deploy time?      (check metrics/logs)
2. Roll back app to last good deploy.             (deploy history -> rollback)
3. Revert any config/flag changes.
4. Check: did it run a destructive migration?     (if so, forward-fix or restore)
5. Verify recovery (metrics, logs, smoke test).
6. Communicate. THEN debug.

References

• Expand/contract migration pattern: https://martinfowler.com/bliki/ParallelChange.html
• Database migrations best practices (Flyway): https://documentation.red-gate.com/fd/migrations-184127470.html
• Google SRE incident response: https://sre.google/sre-book/managing-incidents/
• Canary deployments overview: https://martinfowler.com/bliki/CanaryRelease.html
• Blameless postmortems: https://sre.google/sre-book/postmortem-culture/

---

Want one-click rollback to a previous good deploy without rebuilding? PandaStack keeps deploy history so you can recover fast. Try it free at https://dashboard.pandastack.io