How to Deploy Celery Beat for Scheduled Tasks

If you're running a Python app and need periodic tasks — nightly reports, cache warmers, data syncs — Celery Beat is the standard scheduler. But Beat is deceptively easy to misconfigure: run two Beat processes and every task fires twice; lose the schedule file and you lose history; put Beat inside a worker and a worker restart kills your cron.

This guide deploys Celery Beat correctly alongside workers and a broker.

The moving parts

A production Celery setup has three distinct processes:

1. 1Broker — a message queue (Redis or RabbitMQ) that holds tasks.
2. 2Worker(s) — celery worker, which consume and execute tasks. Scale these horizontally.
3. 3Beat — celery beat, which publishes scheduled tasks to the broker on time. Run exactly one.

The golden rule: there must be exactly one Beat process. Workers can be many; Beat must be a singleton, or every schedule fires N times.

Step 1: Define tasks and the schedule

A minimal celery_app.py:

from celery import Celery
from celery.schedules import crontab
import os

app = Celery(
    "myapp",
    broker=os.environ["CELERY_BROKER_URL"],
    backend=os.environ.get("CELERY_RESULT_BACKEND"),
)

@app.task
def generate_daily_report():
    # ... do work ...
    return "ok"

app.conf.beat_schedule = {
    "daily-report": {
        "task": "celery_app.generate_daily_report",
        "schedule": crontab(hour=6, minute=0),  # 06:00 UTC daily
    },
    "warm-cache": {
        "task": "celery_app.warm_cache",
        "schedule": 300.0,  # every 5 minutes
    },
}
app.conf.timezone = "UTC"

Keep timezone explicit — defaulting to the container's local time has burned many teams during DST.

Step 2: Containerize

One image, three start commands. Build once, deploy three apps.

FROM python:3.12-slim

WORKDIR /app
ENV PYTHONUNBUFFERED=1

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

requirements.txt:

celery[redis]==5.4.0
redis==5.0.8

Step 3: Choose a broker

Redis is the simplest broker for most teams. Provision a managed Redis instance (PandaStack offers Redis via KubeBlocks) and use its connection string as CELERY_BROKER_URL.

CELERY_BROKER_URL=redis://:<password>@<host>:6379/0
CELERY_RESULT_BACKEND=redis://:<password>@<host>:6379/1

Use separate logical DBs (or separate instances) for broker and result backend so result keys don't collide with queues.

Step 4: Deploy worker and beat as separate apps

Deploy the worker as a container app with this start command:

celery -A celery_app worker --loglevel=info --concurrency=4

Deploy Beat as a *separate* app, with a single replica, using:

celery -A celery_app beat --loglevel=info --schedule /tmp/celerybeat-schedule

The --schedule path is where Beat tracks last-run timestamps. Because containers are ephemeral, this file resets on restart — which is fine for interval/crontab schedules (the next tick simply recomputes). If you need durable, database-backed schedules editable at runtime, use django-celery-beat with a database scheduler instead.

Critical: set the Beat app's replica count to exactly 1 and do not enable autoscaling on it. Multiple Beat replicas cause duplicate task dispatch.

Step 5: Avoid the duplicate-run trap

Beyond keeping a single Beat replica, add idempotency to tasks that must not double-execute. Two patterns:

• Idempotent tasks — design generate_daily_report so running it twice is harmless (upsert, not insert).
• Distributed lock — wrap the task body in a Redis lock keyed to the period:

from redis import Redis
r = Redis.from_url(os.environ["CELERY_BROKER_URL"])

@app.task
def generate_daily_report():
    lock = r.lock("lock:daily-report", timeout=600)
    if not lock.acquire(blocking=False):
        return "skipped: already running"
    try:
        # ... work ...
        return "ok"
    finally:
        lock.release()

Beat vs. platform cronjobs

There's a real architectural choice here:

If your only need is "run this script at 6am," a PandaStack cronjob that runs a one-off container is simpler and gives you per-run logs without a long-lived Beat process. Reach for Beat when scheduling is deeply integrated with your task workflows.

Step 6: Observe it

• Tail worker logs to confirm tasks are received and executed.
• Tail Beat logs to confirm schedules fire on time (Scheduler: Sending due task ...).
• Add a result backend so you can inspect task outcomes.

On PandaStack, build/app logs stream live (backed by self-hosted Elasticsearch), so you can watch Beat dispatch and workers pick up in real time.

References

• [Celery Periodic Tasks (Beat)](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html)
• [Celery worker guide](https://docs.celeryq.dev/en/stable/userguide/workers.html)
• [django-celery-beat (database scheduler)](https://django-celery-beat.readthedocs.io/)
• [Redis as a Celery broker](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html)

---

The trick with Celery Beat is discipline: one Beat, many workers, idempotent tasks. PandaStack lets you run the worker, the singleton Beat, and managed Redis side by side, with live logs to watch it all work. Start free at [dashboard.pandastack.io](https://dashboard.pandastack.io).