The problem with manual migrations
Manual migrations fail in predictable ways: someone forgets to run them, runs the wrong one, runs them against the wrong database, or runs them *after* the new code is already serving traffic against an old schema. Automating migrations into the deploy pipeline removes the human error — but only if you do it carefully, because a migration is the one part of a deploy that can corrupt data.
The goal: every deploy applies pending migrations exactly once, in a controlled phase, before the new code serves traffic.
Release phase vs run phase
The single most important concept is *when* migrations run. There are two distinct phases:
- Release phase — runs once, before new instances start, while old ones may still be serving. This is where migrations belong.
- Run phase — your app processes, which may be 1 pod or 50.
If you naively run migrations on every container start, you get a race: ten pods boot simultaneously and all try to migrate at once. That's why migrations belong in a dedicated step that runs once per release, not once per process.
In Kubernetes the idiomatic way to express this is a Job (or a Helm pre-install/pre-upgrade hook) that runs to completion before the new Deployment rolls out:
apiVersion: batch/v1
kind: Job
metadata:
name: db-migrate
annotations:
"helm.sh/hook": pre-upgrade,pre-install
"helm.sh/hook-weight": "-5"
"helm.sh/hook-delete-policy": before-hook-creation
spec:
backoffLimit: 1
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry/app:v2
command: ["npm", "run", "migrate:up"]
envFrom:
- secretRef: { name: app-secrets }If the Job fails, the rollout doesn't proceed — exactly what you want.
Make migrations idempotent and locked
Good migration tools already track which migrations have run (a schema_migrations table) and take an advisory lock so two runners can't collide. If yours doesn't, add one:
-- Postgres advisory lock: only one migrator proceeds
SELECT pg_advisory_lock(12345);
-- ... run pending migrations ...
SELECT pg_advisory_unlock(12345);Most mature tools (Flyway, Liquibase, Alembic, Prisma Migrate, Rails, knex, golang-migrate) handle the tracking table and locking for you. Use a real tool; don't hand-roll ALTER TABLE scripts.
The backward-compatible rule (this is the big one)
During a rolling deploy there's a window where *old code and new code run simultaneously against the same database*. If your migration breaks the old code, you get errors during every deploy.
The rule: a migration must be backward-compatible with the currently-running code. Use expand-and-contract:
- 1Expand (deploy N): add the new column/table. Make it nullable or give it a default. Old code ignores it; new code can use it.
- 2Transition (deploy N): new app code reads/writes the new shape, but tolerates the old shape too.
- 3Contract (deploy N+1, later): once no running code references the old column, a *separate* migration drops it.
Never combine "add new column" and "drop old column" in the same release. A renamed column is really *add new + backfill + switch reads + drop old*, spread across releases.
-- DEPLOY 1 (expand): safe, additive
ALTER TABLE users ADD COLUMN full_name text;
UPDATE users SET full_name = first_name || ' ' || last_name; -- backfill
-- DEPLOY 2 (contract, weeks later): only after old code is gone
ALTER TABLE users DROP COLUMN first_name;
ALTER TABLE users DROP COLUMN last_name;Beware long-locking migrations
Some operations take exclusive locks that block reads/writes on large tables. On Postgres, adding an index without CONCURRENTLY locks the table; adding a NOT NULL column with a non-constant default historically rewrote the whole table.
-- BAD on a big table: locks writes for the duration
CREATE INDEX idx_orders_user ON orders(user_id);
-- GOOD: builds without blocking writes (cannot run in a transaction)
CREATE INDEX CONCURRENTLY idx_orders_user ON orders(user_id);For large tables, test the migration's lock behavior in staging with production-like data volume *before* it ships.
How this looks on a managed platform
PandaStack builds your image with rootless BuildKit in an ephemeral Kubernetes Job, ships it to Artifact Registry, and deploys via Helm. The clean place to hang migrations is a pre-upgrade hook or a release command that runs once before the new revision takes traffic, using the same image and the auto-injected DATABASE_URL so the migrator always points at the right managed database. Because deploy history and rollbacks are first-class, you can roll the *code* back instantly — but remember a code rollback does not undo a schema change, which is the whole reason the expand-and-contract discipline matters.
A checklist for safe auto-migrations
| Concern | Practice |
|---|---|
| Runs once per release | Pre-deploy Job / Helm hook, not per-pod startup |
| Concurrency | Tool-level advisory lock + tracking table |
| Backward compatible | Expand-and-contract; never add+drop in one release |
| Big-table locks | CREATE INDEX CONCURRENTLY, test on prod-size data |
| Failure handling | Failed migration blocks the rollout |
| Rollback awareness | Code rollback ≠ schema rollback; keep migrations additive |
Bottom line
Automate migrations as a single release-phase step that fails the deploy if it fails, use a real migration tool with locking, and make every migration backward-compatible. Do that and migrations become the most boring part of your deploy — which is exactly what you want them to be.
References
- [The Twelve-Factor App — Admin processes / release phase](https://12factor.net/admin-processes)
- [PostgreSQL — CREATE INDEX CONCURRENTLY](https://www.postgresql.org/docs/current/sql-createindex.html)
- [Helm — Chart hooks](https://helm.sh/docs/topics/charts_hooks/)
- [Martin Fowler — Evolutionary Database Design](https://martinfowler.com/articles/evodb.html)
Want migrations that run on deploy against an auto-wired database? PandaStack injects DATABASE_URL for managed databases on every tier — start free: [dashboard.pandastack.io](https://dashboard.pandastack.io)