How to Roll Back a Failed Deployment

# How to Roll Back a Failed Deployment

Every team ships a bad deploy eventually. What separates a calm five-minute fix from a two-hour incident is whether you can roll back quickly and confidently. This tutorial covers how rollbacks work, where they get tricky, and how to make them boring.

What a rollback actually is

A rollback returns your running application to a previous known-good version. Because modern deploys produce immutable, versioned artifacts (a container image tagged with a commit SHA, a Helm release revision), rolling back is usually a matter of pointing production at the previous artifact and letting the orchestrator replace the current instances.

Live: v5 (broken)
   │ rollback
   ▼
Live: v4 (last known good)

The key enabler is that you kept the previous artifact around. If you overwrite images with a mutable latest tag, you've thrown away your rollback target — which is one more reason to tag images with immutable SHAs.

Rollback vs. roll-forward

There are two ways to recover from a bad deploy, and the right choice depends on the situation.

Rollback is faster and usually the right first move. But roll-forward is sometimes the only safe option — most importantly when the broken deploy already made a forward-only database change.

The hard part: database changes

This is the rollback's Achilles' heel. Rolling back *code* does not roll back *data*. If v5 ran a migration that dropped a column, rolling back to v4 leaves v4 running against a schema that no longer has the column it expects — so v4 breaks too.

This is exactly why backward-compatible (expand/contract) migrations matter: if your migrations never destroy anything the previous version needs, a code rollback is always safe. The discipline:

• Never combine a destructive schema change with the deploy that introduces it.
• Deploy additive (expand) migrations separately and earlier.
• Save destructive (contract) migrations for *after* you're confident the new code is stable and won't be rolled back.

If you follow that, rollbacks stay trivial. If you don't, a rollback can make things worse.

How to roll back, by mechanism

Kubernetes / Helm:

# See revision history
helm history myapp
# Roll back to the previous revision
helm rollback myapp 0   # or a specific revision number

Plain Kubernetes Deployment:

kubectl rollout undo deployment/myapp
kubectl rollout status deployment/myapp

Git-driven platforms: select a previous deploy from history and promote it, or revert the commit and let the platform redeploy.

Make rollbacks a non-event

The goal is for rolling back to feel routine, not heroic. The practices that get you there:

• Immutable, versioned artifacts — every deploy is independently re-deployable.
• Retained deploy history — you can see and pick previous versions.
• Backward-compatible migrations — code rollback is always data-safe.
• Fast health detection — good monitoring tells you a deploy is bad in seconds, not after a customer complains.
• Practiced procedure — everyone on call knows the rollback command without looking it up.

Rolling back on PandaStack

PandaStack keeps deploy history and supports rollbacks as a built-in feature — you can revert to a previous release when a new one misbehaves, with how far back you can go depending on your plan's deploy-history retention (10 days on Free, 30 days on Pro, up to 90 days on Premium). Because the platform deploys versioned images via Helm, reverting points production back at a prior known-good release and rolls it out the same zero-downtime way a normal deploy goes out.

The one responsibility that stays with you — on any platform — is database safety. PandaStack's managed databases offer scheduled and manual backups, so take a manual backup before risky schema changes and keep migrations backward-compatible. Do that, and a rollback on PandaStack is exactly what it should be: a quick, low-drama click back to the version that worked.

A rollback runbook template

1. Confirm the deploy is the cause (check metrics/logs/error rate).
2. Decide: rollback (default) or roll-forward (if DB migrated destructively).
3. Execute rollback to last known-good version.
4. Verify health checks and error rate recover.
5. Communicate status.
6. Post-incident: write up cause, add a test/guard to prevent recurrence.

References

• [Kubernetes: Rolling back a Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/#rolling-back-a-deployment)
• [Helm: helm rollback](https://helm.sh/docs/helm/helm_rollback/)
• [Google SRE Book: Emergency Response](https://sre.google/sre-book/emergency-response/)
• [Martin Fowler: Parallel Change](https://martinfowler.com/bliki/ParallelChange.html)

Get built-in rollbacks and deploy history on PandaStack — try the free tier at [dashboard.pandastack.io](https://dashboard.pandastack.io).