How to Achieve Zero-Downtime Deployments

# How to Achieve Zero-Downtime Deployments

"Zero-downtime" is one of those phrases everyone wants and few fully implement. A rolling deploy is necessary but not sufficient — without proper health checks, graceful shutdown, and backward-compatible data changes, you'll still drop requests. Here's the complete picture.

What "zero-downtime" actually requires

During a deploy you're replacing running instances of your app. Zero-downtime means: at every instant during that transition, there is at least one healthy instance serving traffic, and no in-flight request is killed. Achieving that has several independent requirements that all have to hold.

Requirement 1: A rolling update strategy

The baseline. Instead of stopping all old instances and starting new ones (which causes an outage window), you replace them incrementally — bring up a new instance, confirm it's healthy, route traffic to it, then retire an old one.

# Kubernetes rolling update — never drop below capacity
strategy:
 type: RollingUpdate
 rollingUpdate:
 maxUnavailable: 0 # never have fewer than desired healthy pods
 maxSurge: 1 # add one extra during rollout

maxUnavailable: 0 is the key: it guarantees full capacity is maintained throughout.

Requirement 2: Health checks that mean something

The orchestrator only routes traffic to instances that report healthy. But a naive health check that just returns 200 OK lies — it'll pass before the app can actually serve requests. You need two distinct checks:

// Readiness: check real dependencies
app.get('/healthz/ready', async (req, res) => {
 try {
 await db.query('SELECT 1');
 res.sendStatus(200);
 } catch {
 res.sendStatus(503); // not ready — don't send me traffic
 }
});

// Liveness: cheap, no external deps
app.get('/healthz/live', (req, res) => res.sendStatus(200));

A new instance should only receive traffic once readiness passes — that's what prevents routing requests to an app that's still warming up.

Requirement 3: Graceful shutdown and connection draining

When an old instance is retired, it gets a termination signal (SIGTERM). If it dies immediately, every in-flight request is dropped. Instead, it should stop accepting new connections, finish what it's doing, then exit.

process.on('SIGTERM', () => {
 server.close(() => { // stop accepting new, finish in-flight
 db.end().then(() => process.exit(0));
 });
 // Safety net: force exit if draining hangs
 setTimeout(() => process.exit(1), 30_000).unref();
});

The orchestrator also needs to stop routing new traffic to the instance *before* sending SIGTERM — there's a brief window (the preStop hook / termination grace period) that handles this. Get this wrong and you'll see occasional connection-refused errors during every deploy.

Requirement 4: Backward-compatible database changes

This is the requirement people forget, and it's the one that bites hardest. During a rolling deploy, old and new code run *simultaneously* against the *same* database. If your new version renames a column the old version still reads, the old instances break mid-deploy.

The rule: schema changes must be compatible with both the old and new code. Use the expand/contract pattern:

1. 1Expand — add the new column (nullable), deploy code that writes both old and new.
2. 2Migrate — backfill data.
3. 3Contract — once all instances run new code, stop using and then drop the old column.

Never combine a destructive schema change with the deploy that depends on it. (This is covered in depth in our database migrations guide.)

Requirement 5: Idempotent, retry-safe clients

Even with everything above, a client may occasionally need to retry a request that landed on an instance being drained. Design write endpoints to be idempotent (e.g. via idempotency keys) so a retry doesn't double-charge or double-create.

How PandaStack handles this

PandaStack deploys via Helm on GKE, which uses Kubernetes rolling updates by default — new pods come up, readiness is checked, and traffic shifts over before old pods retire, with Kong ingress handling routing. You get rollbacks and deploy history, so if a new version misbehaves you can revert to a known-good release.

The platform manages the orchestration layer, but two responsibilities stay with you, because they live in your application code: implementing a real readiness check that reflects your dependencies, and handling SIGTERM for graceful shutdown. And the database discipline — backward-compatible migrations — is always the developer's job regardless of platform, because only you know your schema's semantics. Get those three right and the rolling deploy underneath gives you genuine zero-downtime releases.

Checklist

• ✅ Rolling update with maxUnavailable: 0
• ✅ Readiness probe that checks real dependencies
• ✅ Liveness probe that's cheap and dependency-free
• ✅ SIGTERM handler that drains connections
• ✅ Backward-compatible (expand/contract) schema changes
• ✅ Idempotent write endpoints

References

• [Kubernetes: Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
• [Kubernetes: Pod termination lifecycle](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)
• [Martin Fowler: Evolutionary Database Design](https://martinfowler.com/articles/evodb.html)
• [Google SRE Book: Release Engineering](https://sre.google/sre-book/release-engineering/)

Get rolling deploys, rollbacks, and deploy history out of the box on PandaStack. Start free at [dashboard.pandastack.io](https://dashboard.pandastack.io).