Canary Deployments: Ship to 5% of Users Before 100%

Every deployment is a bet that the new version behaves like the old one plus your change. Most of the time you win. The times you lose, you'd much rather lose in front of 5% of your users than all of them. That's the entire idea of a canary deployment — and it's simpler, and also messier, than most explanations make it sound.

What a canary deployment actually is

You run the new version alongside the old one and route a small slice of real production traffic to it — say 5 or 10%. You watch error rates and latency on that slice. If the new version holds up, you ramp the percentage: 10 → 25 → 50 → 100. If it doesn't, you shift traffic back to the old version and nobody files a support ticket.

This is different from blue/green, where you flip 100% of traffic at once between two environments. Blue/green gives you fast rollback but zero blast-radius reduction — every user hits the new code the moment you flip. A canary trades a slower rollout for the ability to catch problems while they're still small.

The catch: a canary only tells you something if the slice gets enough traffic to be statistically meaningful. If your service does 50 requests a minute, a 5% canary sees 2.5 requests a minute — a 1% error-rate regression could take hours to show up. Low-traffic services often get more value from fast rollback and good alerting than from a canary at all. Be honest about which situation you're in.

Where the traffic split happens

There are four common places to split traffic, and the right one depends on what you already run:

DNS weighting. Don't. Resolvers cache aggressively and ignore your TTLs, so your "10%" is whatever the internet feels like, and rollback takes as long as the caches take to expire.

Load balancer / ingress. The most practical option for most teams. NGINX Ingress on Kubernetes has first-class canary support via annotations — you create a second Ingress pointing at the canary Service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-canary
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-canary
                port:
                  number: 80

10% of requests hit api-canary, the rest hit the primary. There's also canary-by-header, which routes only requests carrying a specific header — useful for letting your own team dogfood the canary at 0% public weight before opening it up. The annotation reference is worth reading in full: https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary

Service mesh. Istio and Linkerd split at the sidecar level with finer control (per-route weights, retries, mirroring). Powerful, but don't adopt a mesh just for canaries — the operational cost is real.

Application-level flags. A feature flag that gates the new code path inside one deployment. This is technically a canary of the *code*, not the *deployment* — it won't catch a bad container image, a broken healthcheck, or a dependency bump that fails at startup. Use it to complement, not replace, traffic-level canaries.

One thing weight-based splitting won't give you: stickiness. A user can hit the canary on one request and the stable version on the next. If your change alters response shapes or session state, split by header or cookie instead of raw weight.

Picking the metrics that decide pass/fail

A canary without defined success criteria is just a slow rollout with extra YAML. Before you ship, write down the comparison you'll make:

• Error rate: 5xx ratio on the canary vs. the stable version, over the same window. Compare against the baseline, not an absolute number — if stable is also erroring at 2% because a downstream is flaky, don't fail the canary for matching it.
• Latency: p95/p99 on the canary vs. stable. Watch for the cold-start effect — a freshly started instance with empty caches and an un-warmed JIT will look worse for the first few minutes. Give it a warm-up window before judging.
• Saturation: CPU and memory per pod. A memory leak that OOMs after 40 minutes will pass a 10-minute canary. Longer bake times catch slower failures; pick a duration that matches how your service actually breaks.

And one guard metric people forget: business signals. A canary that returns 200s with an empty response body passes every infra metric while quietly zeroing your checkout conversion. If you have one number the business cares about, watch it.

Automating the rollback

Manual canaries fail in a predictable way: the person watching the dashboard gets pulled into a meeting. The fix is to make the promotion/rollback decision mechanical.

On Kubernetes, Argo Rollouts is the standard tool. You replace your Deployment with a Rollout that defines the ramp steps and an analysis that runs between them:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
spec:
  replicas: 5
  strategy:
    canary:
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

The analysis template queries Prometheus and fails the rollout if the condition breaks:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      successCondition: result[0] < 0.01
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="api",status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{app="api"}[5m]))

If the error rate crosses 1%, Argo aborts the rollout and shifts traffic back to stable — no human in the loop. Flagger does the same job as an operator that manages canaries on top of your existing mesh or ingress, and it's worth evaluating if you're already on Istio or Linkerd. Docs: https://argoproj.github.io/argo-rollouts/ and https://docs.flagger.app/

The part canaries can't protect: the database

Traffic splitting works because the two versions are independent. Your database isn't — both versions share one schema. If v2 renames a column, v1 starts erroring the moment the migration runs, no matter how carefully you weighted the traffic.

The answer is expand/contract migrations: add the new column first (both versions work), deploy v2 which writes to both, backfill, then remove the old column in a later release once v1 is gone. It's more releases, but it's the only pattern that makes rollback safe — because rolling back the app doesn't roll back the schema.

A practical checklist

1. 1Confirm you have enough traffic for the canary slice to mean something within your bake time.
2. 2Write down pass/fail criteria *before* deploying — error-rate delta, latency delta, minimum bake duration.
3. 3Split at the ingress or mesh, not DNS. Use header-based routing to dogfood at 0% weight first.
4. 4Automate the abort. A canary that requires a human watching Grafana is a canary that fails on Friday evening.
5. 5Keep migrations expand/contract so the rollback path is always real.
6. 6After every aborted canary, keep the failed version's logs and metrics around — that's your debugging material.

That last point is where deployment history earns its keep. On PandaStack, every deploy is recorded with its build and runtime logs streamed live, rollbacks are built in, and metrics are captured server-side without an SDK — so even a plain rolling deploy gets you the observe-and-revert half of the canary workflow, with history retained from 10 days on the free tier up to 90 on Premium. If you want to see how that feels with a git push, try it at https://pandastack.io.