How to Deploy a Stable Diffusion Inference API

Serving Stable Diffusion is different from serving a normal API in one fundamental way: each request is expensive, slow, and memory-hungry. A single image generation can take seconds even on a GPU and far longer on CPU, and the model weights consume several gigabytes of memory. Deploying it well is mostly about managing that cost: loading the model once, queuing requests instead of running them concurrently, and being honest about hardware requirements. This guide covers the architecture and packaging; it's framework-agnostic but uses the Hugging Face diffusers library.

The hardware reality up front

Let's be direct: Stable Diffusion is designed for GPUs. On a modern GPU, a 512x512 image with 30 steps generates in a couple of seconds. On CPU, the same generation can take minutes. You can serve it on CPU for low-volume or batch use cases, but for any interactive workload you want a GPU. Plan your deployment around the hardware you actually have access to, and set user expectations accordingly.

The inference service

The core pattern is: load the pipeline once at startup, then reuse it for every request. Never reload weights per request, model loading is the slowest part.

from diffusers import StableDiffusionPipeline
import torch, os

MODEL_ID = os.getenv("MODEL_ID", "runwayml/stable-diffusion-v1-5")
device = "cuda" if torch.cuda.is_available() else "cpu"

pipe = StableDiffusionPipeline.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.float16 if device == "cuda" else torch.float32,
)
pipe = pipe.to(device)
if device == "cuda":
    pipe.enable_attention_slicing()  # lower peak VRAM

Then a FastAPI endpoint:

from fastapi import FastAPI
from pydantic import BaseModel
import base64, io

app = FastAPI()

class GenReq(BaseModel):
    prompt: str
    steps: int = 30

@app.post("/generate")
def generate(req: GenReq):
    image = pipe(req.prompt, num_inference_steps=req.steps).images[0]
    buf = io.BytesIO(); image.save(buf, format="PNG")
    return {"image": base64.b64encode(buf.getvalue()).decode()}

Why you must queue requests

Here's the trap: a naive FastAPI app will happily accept ten concurrent generation requests and try to run them all at once. On a single GPU, that exhausts VRAM and crashes with an out-of-memory error. The fix is to serialize generation, only one (or a small fixed number) running at a time, and queue the rest.

The robust pattern separates the API from the work: the API enqueues a job (to Redis via RQ or Celery) and returns a job ID immediately; a worker with the loaded model processes the queue one job at a time and stores the result. The client polls for completion. This decouples request acceptance from the expensive compute and protects your GPU memory.

# API side: enqueue and return a job id
job = queue.enqueue("tasks.generate", prompt, steps, job_timeout=600)
return {"job_id": job.id}

For smaller deployments, an in-process semaphore that limits concurrency to one is a simpler alternative to a full queue.

Memory management techniques

Several diffusers features reduce memory pressure:

• enable_attention_slicing() trades a little speed for lower peak VRAM.
• enable_model_cpu_offload() keeps parts of the model on CPU and moves them to GPU as needed, useful on smaller GPUs.
• Use torch.float16 on GPU to roughly halve memory versus float32.

For CPU-only serving, expect high RAM usage and slow generation; pick a memory-optimized tier.

Packaging in a container

Model weights are large. Two strategies: bake them into the image (predictable, large image) or download at startup and cache to a persistent volume (smaller image, slower first boot). For most deployments, caching to a volume is better:

FROM python:3.11-slim
WORKDIR /app
RUN pip install --no-cache-dir torch diffusers transformers accelerate fastapi uvicorn
COPY . .
ENV HF_HOME=/cache/huggingface
EXPOSE 8000
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

Pointing HF_HOME at a mounted volume means weights download once and persist across restarts.

Deploying on PandaStack

1. 1Connect your repo as a container app. The build runs in an ephemeral Kubernetes Job pod with rootless BuildKit and deploys via Helm.
2. 2Provision a managed Redis instance if you're using the queue pattern; reference it via REDIS_URL.
3. 3Deploy the API as one container app and the GPU worker as a separate background container app sharing the queue.
4. 4Choose a memory-optimized tier for CPU inference; sizing matters because the model resident in memory is large.
5. 5Tail live logs to confirm the model loaded once at startup, not per request.

An honest note on the free tier

Stable Diffusion is heavy. The free tier (0.25 CPU / 512MB) cannot load these models, this is a workload for larger compute or memory-optimized tiers, and ideally GPU hardware. Use the free tier to deploy the API skeleton, queue, and managed Redis, then size the inference worker appropriately. Scale-to-zero is attractive here because GPU/large compute is expensive when idle, but accept a meaningful cold start while the model reloads on wake.

Verifying

Submit a prompt, poll for the result, and decode the base64 PNG. Watch memory in the metrics view during generation, the spike tells you how close you are to the tier's limit and whether you need slicing or offload.

References

• Hugging Face Diffusers: https://huggingface.co/docs/diffusers/index
• Diffusers memory optimization: https://huggingface.co/docs/diffusers/optimization/memory
• Stable Diffusion pipeline: https://huggingface.co/docs/diffusers/using-diffusers/conditional_image_generation
• FastAPI: https://fastapi.tiangolo.com/
• PyTorch CUDA memory management: https://pytorch.org/docs/stable/notes/cuda.html

Serving Stable Diffusion is an exercise in managing expensive, memory-heavy requests: load once, queue everything, and size the hardware honestly. Stand up the API and queue with managed Redis on PandaStack's free tier, then scale the worker to fit your model: https://dashboard.pandastack.io