# Using Spot/Preemptible Instances to Cut Cloud Costs
Cloud providers have a huge amount of spare capacity sitting idle at any moment. Rather than waste it, they sell it at a steep discount — with one catch: they can take it back with little warning. These are spot instances (AWS), preemptible / Spot VMs (Google Cloud), and Spot Virtual Machines (Azure). Used correctly, they cut compute costs dramatically. Used carelessly, they take down your production app. This guide is about using them correctly.
Why they are so cheap
A spot instance is the *same physical hardware* as on-demand. The only difference is the contract: the provider can reclaim it when they need the capacity back. Because demand for that capacity fluctuates, they sell it at a discount that commonly reaches up to ~90% off on-demand pricing.
The tradeoff is interruption. AWS gives a 2-minute warning before reclaiming a spot instance; Google's Spot VMs give roughly 30 seconds. Your workload must tolerate a node disappearing.
The mental model: cattle, not pets
Spot only works if your compute is stateless and replaceable. If losing a single node loses data or breaks a long-running session, spot will hurt you. The golden questions:
- Can this workload be killed and restarted elsewhere without data loss?
- Is the work idempotent or checkpointed?
- Can the system run with N-1 nodes for a few minutes?
If yes to all three, spot is a near-free win.
Workloads that fit
| Workload | Spot fit | Why |
|---|---|---|
| Stateless web/API replicas | Great | Load balancer reroutes; just keep some on-demand |
| CI/CD build jobs | Excellent | Short-lived, retryable, no persistent state |
| Batch / data processing | Excellent | Checkpoint and resume |
| ML training (checkpointed) | Great | Save state periodically, resume on new node |
| Background workers / queues | Good | Re-queue the in-flight message |
| Dev / preview environments | Great | Interruptions are tolerable |
| Stateful databases | Poor | Reclamation risks data/availability |
| Sticky-session apps | Poor | Sessions die with the node |
Handling interruptions gracefully
The key engineering practice is to listen for the termination notice and drain cleanly. On AWS the instance metadata exposes a rebalance/interruption signal:
# Poll the instance metadata for an interruption notice
curl -s http://169.254.169.254/latest/meta-data/spot/instance-action
# Returns the action + time when termination is scheduledIn Kubernetes, run a handler (e.g. the AWS Node Termination Handler or GKE's graceful node shutdown) that cordons and drains the node on notice, letting pods reschedule onto healthy capacity.
For message workers, ack messages only after work completes so an interrupted job is automatically redelivered.
Patterns that make spot safe
- Mix spot and on-demand. Run a baseline of on-demand for reliability and burst with spot. A common ratio is a small on-demand floor plus a large spot ceiling.
- Diversify instance types and zones. Spreading across many instance families reduces the chance of a simultaneous mass reclamation.
- Make everything idempotent. Retried work should be safe to run twice.
- Checkpoint long jobs. Save progress so a restart resumes near where it left off.
- Cap your budget. Set max prices/budgets so you never overpay during a price spike.
Kubernetes makes this easier
Kubernetes is the natural home for spot because its scheduler already treats nodes as disposable. A node vanishes, pods are rescheduled, the deployment self-heals. Tools like Karpenter (AWS) and cluster-autoscaler can automatically provision spot capacity and fall back to on-demand when spot is unavailable.
# Schedule tolerant workloads onto spot nodes
nodeSelector:
cloud.google.com/gke-spot: "true"
tolerations:
- key: "cloud.google.com/gke-spot"
operator: "Equal"
value: "true"
effect: "NoSchedule"The honest downsides
- No availability guarantee. During capacity crunches spot may be unavailable entirely.
- Operational complexity. You must build interruption handling; it is not free.
- Not for stateful single points of failure. Databases belong on stable capacity.
The right framing: spot is a discount you *earn* by making your architecture resilient. The resilience is good engineering anyway.
How PandaStack uses spot
PandaStack runs free-tier apps on spot nodes inside a gVisor sandbox, combined with KEDA scale-to-zero so idle apps consume nothing. That is how the free tier stays free: spare, interruptible capacity for workloads where a cold start is acceptable. The platform handles rescheduling for you — you get the cost benefit without writing your own node-termination handler. (The known tradeoff, stated plainly: free-tier apps can cold-start when they scale back up from zero on preemptible nodes.)
References
- [AWS EC2 Spot Instances](https://aws.amazon.com/ec2/spot/)
- [Google Cloud Spot VMs](https://cloud.google.com/compute/docs/instances/spot)
- [AWS Node Termination Handler](https://github.com/aws/aws-node-termination-handler)
- [Karpenter documentation](https://karpenter.sh/docs/)
- [KEDA: Kubernetes Event-driven Autoscaling](https://keda.sh/docs/)
---
Spot instances reward resilient architecture with massive savings. PandaStack already runs free-tier workloads on spot with scale-to-zero so you benefit automatically — try it at [dashboard.pandastack.io](https://dashboard.pandastack.io).