How to Deploy Django with Celery and Redis

# How to Deploy Django with Celery and Redis

The moment your Django app needs to send an email, process an upload, or call a slow third-party API, you should not make the user wait for it. The standard solution is Celery — a distributed task queue — with Redis as the message broker. The tricky part is that this is a *multi-process* architecture: your web server and your workers are separate deployments. This tutorial covers getting all the pieces deployed and talking to each other.

The architecture

Django web (Gunicorn) ──enqueue──▶ Redis (broker) ──▶ Celery worker(s)
        ▲                                                    │
        └──────────── PostgreSQL (results / app data) ◀──────┘
   Celery Beat ──schedule──▶ Redis

The key insight: three or four separate processes, all sharing the same codebase and config:

1. 1Web — Gunicorn serving Django, enqueues tasks.
2. 2Worker — celery worker, executes tasks.
3. 3Beat (optional) — celery beat, schedules periodic tasks.
4. 4Redis — the broker connecting web and workers.

Step 1: configure Celery

Point Celery at Redis via environment variables:

# myproject/celery.py
import os
from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myproject.settings")
app = Celery("myproject")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

# settings.py
CELERY_BROKER_URL = os.environ["REDIS_URL"]
CELERY_RESULT_BACKEND = os.environ.get("REDIS_URL")
CELERY_TASK_ACKS_LATE = True            # re-deliver if a worker dies mid-task
CELERY_WORKER_PREFETCH_MULTIPLIER = 1   # fairer distribution for long tasks

ACKS_LATE = True matters in production: if a worker crashes (or a spot node is reclaimed) mid-task, the message is redelivered instead of lost.

Step 2: write a task

# tasks.py
from celery import shared_task

@shared_task(bind=True, max_retries=3, default_retry_delay=10)
def send_welcome_email(self, user_id):
    try:
        user = User.objects.get(id=user_id)
        deliver_email(user.email)
    except SMTPException as exc:
        raise self.retry(exc=exc)

Enqueue from a view with .delay():

send_welcome_email.delay(user.id)
return Response({"status": "queued"})  # returns instantly

Step 3: run each process

This is the heart of the deployment — each process is its own service with a different start command but the same image and env vars.

# Web
gunicorn myproject.wsgi:application --bind 0.0.0.0:${PORT:-8080} --workers 3

# Worker
celery -A myproject worker --loglevel=info --concurrency=4

# Beat (only ONE instance, ever)
celery -A myproject beat --loglevel=info

Critical rule: run exactly one Beat instance. Two Beat schedulers means every periodic task fires twice. Workers can scale horizontally; Beat must be a singleton.

Step 4: provision Redis and Postgres

You need two data services:

• Redis as the broker (and optionally result backend). Read REDIS_URL from the environment.
• PostgreSQL for your application data. Read DATABASE_URL.

Both web and workers need both connection strings, because workers run the same Django code and touch the database.

Step 5: the shared container

One image, multiple start commands:

FROM python:3.12-slim
ENV PYTHONUNBUFFERED=1
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
# start command set per service (web / worker / beat)

Step 6: scaling and reliability

• Scale workers horizontally for throughput — add more worker instances behind the same Redis.
• Use task time limits so a stuck task cannot hold a worker forever:

CELERY_TASK_TIME_LIMIT = 300        # hard kill at 5 min
CELERY_TASK_SOFT_TIME_LIMIT = 270   # raise exception first for cleanup

• Make tasks idempotent. With acks_late, a task may run twice after a crash — design for it.
• Monitor queue depth. A growing Redis queue means workers cannot keep up; add capacity.

Common pitfalls

• Running Beat in multiple replicas. Duplicate scheduled tasks. Run one.
• Workers missing DATABASE_URL. Tasks that touch the DB crash. Set all env vars on every service.
• Broker not reachable. .delay() blocks or errors — check REDIS_URL.
• Tasks not idempotent. Redelivery causes double-charges, double-emails. Guard with idempotency keys.
• Forgetting time limits. One runaway task starves the pool.

Deploying on PandaStack

PandaStack's model maps cleanly onto this multi-process setup. Deploy the web as one container service and the Celery worker as another container service (same repo/image, different start command). Provision a managed Redis for the broker and a managed PostgreSQL for app data — PandaStack injects REDIS_URL and DATABASE_URL into your services automatically. For periodic jobs you can run a single Beat service, or use PandaStack cronjobs to trigger scheduled work directly. Workers scale horizontally, you get live logs across each process, and rollbacks cover bad deploys. The free tier includes web services plus one database (Redis is available among the managed databases) to prototype the whole pipeline.

References

• [Celery documentation](https://docs.celeryq.dev/en/stable/)
• [First steps with Celery and Django](https://docs.celeryq.dev/en/stable/django/first-steps-with-django.html)
• [Celery: acks_late and reliability](https://docs.celeryq.dev/en/stable/userguide/configuration.html#task-acks-late)
• [Redis as a Celery broker](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html)

---

Django + Celery + Redis is really an exercise in deploying cooperating processes. PandaStack lets you ship web, worker, and Redis from one repo with auto-wired connection strings — start free at [dashboard.pandastack.io](https://dashboard.pandastack.io).