How to Deploy an Ollama Server Without a GPU

Ollama makes running open-weight models locally almost boring — ollama run llama3.2 and you're chatting. Deploying it as a shared server is a different exercise, and most guides skip the three decisions that actually matter: where the model weights live (image, volume, or downloaded at boot), how much RAM the container really needs, and what you do about the fact that Ollama ships with no authentication at all. Let's do it properly, on CPU, without pretending you need a GPU cluster for a 1B model.

First, be honest about CPU inference

Without a GPU, model size is everything. As rough, deliberately conservative guidance for 4-bit quantized models:

On CPU, expect a 7–8B model to generate at low single-digit tokens per second; 1–3B models are noticeably quicker. For internal tools, classification, summarization, and structured extraction, small models are genuinely enough — that's the honest use case for a CPU-hosted Ollama. If you need a 70B chat experience, this isn't the architecture.

Budget RAM for the model plus the KV cache (grows with context length) plus the OS. When in doubt, one size up.

Decision one: bake the model into the image

Ollama stores models under /root/.ollama inside the container. If you run the stock ollama/ollama image with no model, your first request fails until something pulls one. You have two sane options.

Option A (recommended): pull the model at build time so it ships inside the image:

FROM ollama/ollama:latest

ENV OLLAMA_HOST=0.0.0.0:11434
ENV OLLAMA_KEEP_ALIVE=30m

# Start the server just long enough to pull the model into an image layer
RUN ollama serve & \
    server=$! && \
    sleep 5 && \
    ollama pull llama3.2:1b && \
    kill $server

EXPOSE 11434

The base image's entrypoint already runs ollama serve, so no CMD needed. The RUN trick works because Docker RUN ignores the entrypoint: we start the server in the background, pull, and kill it before the layer commits.

Tradeoffs, stated plainly: the image grows by the model size (~1.3 GB for llama3.2:1b), so builds and pushes take a few minutes longer. In exchange, every deploy and every cold start is reproducible — the model is *there*, no runtime download, no dependency on Ollama's registry being reachable at 3 a.m.

Option B: pull at boot with an entrypoint script:

#!/bin/sh
# start.sh
ollama serve &
pid=$!
sleep 3
ollama pull llama3.2:1b
wait $pid

FROM ollama/ollama:latest
ENV OLLAMA_HOST=0.0.0.0:11434
COPY start.sh /start.sh
ENTRYPOINT ["/bin/sh", "/start.sh"]
EXPOSE 11434

Smaller image, but every fresh container downloads the model before it can serve. Unless you have persistent storage mounted at the model path, that's minutes of dead time on every cold start. Bake it in.

Decision two: the environment variables that matter

Ollama is configured entirely through environment variables. The ones worth setting:

• OLLAMA_HOST=0.0.0.0:11434 — listen on all interfaces so the platform router can reach it. Port 11434 is the default and there's no reason to change it.
• OLLAMA_KEEP_ALIVE — how long a model stays loaded in RAM after the last request. Default is 5 minutes, after which the *next* request pays the full model-load time again. For a dedicated inference box, 30m or -1 (never unload) makes response times consistent — just make sure the RAM budget assumes the model is always resident.
• OLLAMA_NUM_PARALLEL — concurrent requests per model. On CPU, keep this low (1–2); parallel requests share the same cores and everyone gets slower.
• OLLAMA_MAX_LOADED_MODELS — leave at 1 on a small box. Two loaded models means two models' worth of RAM.
• OLLAMA_ORIGINS — allowed CORS origins, if a browser app calls the API directly (think hard before allowing that; see security below).

Decision three: authentication, because Ollama has none

This is the part that bites people. Ollama's API is wide open — anyone who can reach port 11434 can run generations on your CPU, list your models, and pull new ones. Exposed to the public internet with no protection, your inference server becomes someone else's free compute within days. Options, in order of preference:

1. 1Don't expose it publicly. Call Ollama only from your own backend, server-to-server, and never hand the URL to browsers.
2. 2Restrict by source. Platform firewall rules limiting who can reach the service cover most internal-tool cases.
3. 3Put a thin auth proxy in front. A 20-line FastAPI or Express app that checks a bearer token and forwards to localhost:11434 is unglamorous and works.

Pick at least one before you deploy, not after the bill arrives.

Deploying on PandaStack

With the Option A Dockerfile committed, the deploy itself is short:

1. 1Push the repo to GitHub and connect it as a container app on PandaStack. A Dockerfile in the root is used as-is — no buildpack guessing.
2. 2Watch the live build logs. The ollama pull layer is the slow step; you'll see the model download happen inside the build. Builds run as rootless BuildKit in ephemeral Kubernetes Job pods, and the finished image goes to the registry with the model already inside.
3. 3Pick a tier that matches your model. The free tier's 0.25 CPU / 512 MB won't fit even a 1B model in RAM — this is one workload where you genuinely need a paid compute tier. A 1–3B model wants one of the memory-friendly mid tiers; at the top end, the 8 CPU / 16 GB tier ($0.300/hr, roughly $219/mo if run flat out) handles 7–8B quantized models with room for context.
4. 4Set OLLAMA_KEEP_ALIVE and any other env vars in the app's environment settings, add firewall rules so only your services can reach it, and deploy. Custom domain and SSL are automatic if you want a named endpoint.

One scale-to-zero note: free-tier apps on PandaStack scale to zero when idle, which is great for websites and rough for model servers — a cold start means booting the container *and* loading gigabytes of weights into RAM before the first token. Paid tiers stay warm, which is what you want here anyway.

Ollama itself needs no database — the model files are its only state. If you're building a chat app around it, that app is a separate service, and attaching a managed Postgres to it gets DATABASE_URL injected automatically for storing conversations.

Verify it works

Basic liveness — Ollama answers with plain text on the root path:

curl https://your-app.example.com/
# Ollama is running

A real generation:

curl https://your-app.example.com/api/generate -d '{
  "model": "llama3.2:1b",
  "prompt": "Explain CIDR notation in one paragraph.",
  "stream": false
}'

And because Ollama exposes an OpenAI-compatible endpoint, most existing SDKs work by changing the base URL:

curl https://your-app.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:1b",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

If the first request after a deploy is slow and the second is fast, that's the model loading into RAM — exactly the behavior OLLAMA_KEEP_ALIVE controls. If *every* request is slow, check whether you set KEEP_ALIVE too low or the tier too small; watching memory on the app's metrics view will tell you which within a minute.

That's the whole thing: one Dockerfile, four environment variables, one non-negotiable security decision, and an honest RAM budget. If you want to see it running, connect the repo and push on https://pandastack.io.