Most CI security incidents and "works on my machine but not in CI" mysteries trace back to one design flaw: shared, long-lived build infrastructure. When many builds run on the same persistent host, they contaminate each other and share an attack surface. The fix is an architecture where every build runs in a fresh, isolated, ephemeral pod that's destroyed afterward. This post explains how that works and why it's worth it.
The problem with shared build hosts
The classic CI setup is a pool of long-running build agents. Each agent runs many builds over its lifetime. That introduces several problems:
- State leakage: build A leaves files, caches, env vars, or processes that affect build B. "Why did my build pass yesterday and fail today on the same commit?" — often this.
- Security blast radius: a malicious or compromised build (think a poisoned dependency running arbitrary code during
npm install) can read secrets, other tenants' source, or the host itself. - The Docker socket problem: builds that need to build images often get access to the host's Docker daemon socket — effectively root on the host. A single compromised build owns the machine.
- Noisy neighbors: one heavy build starves others on the shared host.
For a multi-tenant platform, the security points are disqualifying. You cannot let one customer's build threaten another's.
The ephemeral build pod model
The alternative: treat each build as a disposable unit of compute.
Git push
│
▼
Create a fresh Kubernetes Job pod ← isolated, single-use
│
▼
Build image (rootless BuildKit) ← no host Docker socket
│
▼
Push image to Artifact Registry
│
▼
Pod is destroyed ← no residueEvery build gets a brand-new pod. The pod does exactly one build, pushes the result, and is torn down. Nothing persists between builds.
Why this design is more secure
1. Clean-room isolation
Each build starts from an identical, empty state. No leftover files, no cached env vars, no lingering processes from a previous build. This kills state-leakage bugs *and* prevents one build from observing another's data — there's no shared filesystem to snoop.
2. Rootless builds, no Docker socket
The most important security property. Instead of handing builds the host Docker daemon, the image is built with rootless BuildKit — building images without root and without a privileged daemon. A compromised build can't escalate to host root via the Docker socket because there is no socket to abuse. PandaStack builds this way: rootless BuildKit in the pod, image pushed to Google Artifact Registry, then Helm deploys it.
3. Bounded blast radius
If a build *is* malicious, the damage is confined to a single throwaway pod with scoped permissions (just enough to pull source and push to its target registry repo). When the pod dies, so does anything the build did. Compare that to a shared host where a compromise persists and spreads.
4. Resource isolation
Kubernetes resource requests/limits cap each build pod's CPU and memory. A runaway build can't starve other builds — the scheduler isolates them onto appropriate capacity.
Shared host vs ephemeral pod
| Property | Shared build host | Ephemeral build pod |
|---|---|---|
| State between builds | Persists (contamination) | None (clean room) |
| Privilege model | Often Docker socket = host root | Rootless, no socket |
| Blast radius of compromise | Host + other builds | Single throwaway pod |
| Reproducibility | Drifts over time | Identical every run |
| Resource isolation | Weak (noisy neighbors) | Per-pod limits |
| Startup cost | None (already running) | Pod scheduling time |
The trade-offs (being honest)
Ephemeral pods aren't free:
- Startup latency. Spinning up a fresh pod adds time versus an already-warm shared agent. Mitigated by fast scheduling and lean builder images, but it's real.
- Cache strategy required. A clean room means no local cache by default — you'd re-download dependencies every build. The fix is *external* cache: BuildKit cache mounts and registry-backed cache import/export, so caching survives without persisting pod state. (See our reduce-build-times guide.)
- Orchestration complexity. You need a system to create, monitor, and reap Job pods, ship their logs out before they die, and handle failures.
The security and reproducibility wins justify the cost for any multi-tenant or security-conscious setup. The caching caveat is the main thing to engineer around.
Getting logs out before the pod dies
Since the pod is destroyed after the build, logs can't live on it. The architecture ships build output to a central store as the build runs — PandaStack streams build logs to a self-hosted Elasticsearch cluster, which is what makes them both live and searchable after the pod is gone. Without this, your post-mortem evidence vanishes with the pod.
How it fits the bigger picture
Ephemeral build pods are one half of a clean build/deploy separation:
- 1Build (ephemeral pod, rootless BuildKit) produces an immutable image.
- 2Package pushes it to Artifact Registry — content-addressed, durable.
- 3Deploy (Helm, on multi-region GKE) pulls that image.
The registry is the durable handoff; the build pod is intentionally disposable. This separation is what makes builds reproducible *and* deploys/rollbacks reliable.
References
- [Kubernetes Jobs](https://kubernetes.io/docs/concepts/workloads/controllers/job/)
- [BuildKit rootless mode](https://github.com/moby/buildkit/blob/master/docs/rootless.md)
- [The dangers of mounting the Docker socket](https://docs.docker.com/engine/security/)
- [BuildKit cache export/import](https://docs.docker.com/build/cache/backends/)
- [SLSA: Supply-chain security framework](https://slsa.dev/)
---
Every build on PandaStack runs in a fresh, rootless, ephemeral pod — secure isolation and reproducible builds with no setup on your part. See the architecture in action on the free tier at [dashboard.pandastack.io](https://dashboard.pandastack.io).