How to Deploy a Celery Worker with Redis as Broker

Celery is the standard distributed task queue for Python. Almost every Django or Flask app eventually needs it — to send emails, process uploads, call slow third-party APIs, or run scheduled jobs without blocking web requests. This guide deploys Celery with Redis as the message broker, the most common and simplest production setup.

The mental model

Three roles, each its own process:

The web app enqueues tasks into Redis; workers pull and execute them. Critically, the web app and the worker are separate deployments running the same codebase. Bundling them into one process defeats the purpose.

Minimal setup

# app/celery_app.py
from celery import Celery
import os

app = Celery(
    'myapp',
    broker=os.environ['REDIS_URL'],
    backend=os.environ.get('RESULT_BACKEND', os.environ['REDIS_URL']),
)
app.conf.task_acks_late = True
app.conf.worker_prefetch_multiplier = 1

@app.task
def send_welcome_email(user_id):
    ...

Enqueue from the web app:

send_welcome_email.delay(user.id)

Deploying the worker

Your web service and worker service share one image but run different commands:

# web service
gunicorn myapp.wsgi --bind 0.0.0.0:$PORT

# worker service
celery -A app.celery_app worker --concurrency=4 --loglevel=info

On PandaStack, deploy both from the same Git repo as two container services. Attach a managed Redis and inject REDIS_URL into both. The worker needs no public port — it's a pure background service.

Concurrency and prefetch tuning

Two knobs determine throughput and fairness:

• --concurrency: number of worker child processes (prefork). Start around the CPU count for CPU-bound work; go higher for I/O-bound tasks, or use the gevent/eventlet pools for highly concurrent I/O.
• worker_prefetch_multiplier: how many tasks each worker reserves ahead of time. The default (4) is bad for long, uneven tasks because one worker hoards the queue. Set it to 1 for long tasks so work distributes evenly.

Pair task_acks_late = True with prefetch 1 so a task is only acknowledged after it finishes — if a worker crashes mid-task, the task is redelivered instead of lost.

Scheduled tasks: Celery Beat

For cron-like schedules, Celery Beat is a separate process that publishes scheduled tasks onto the broker:

celery -A app.celery_app beat --loglevel=info

Run exactly one Beat instance — two will double-fire every scheduled task. Deploy Beat as its own single-replica service. Alternatively, if your platform offers native cronjobs, you can skip Beat for simple schedules and let the platform invoke a management command on a cron expression — PandaStack cronjobs do exactly this, which removes the need to keep a Beat process alive and singleton.

The result backend decision

The result backend stores task return values and states. Be deliberate:

• If you don't need results, set task_ignore_result = True — it saves Redis memory and round-trips.
• If you do, Redis works but set a TTL (result_expires) so results don't accumulate forever.
• For durable, queryable results, some teams use the database backend instead.

Redis as both broker and backend is fine for most apps; just don't let result data balloon your Redis memory.

Redis vs RabbitMQ as broker

Redis is simpler and great for most workloads. RabbitMQ offers stronger delivery guarantees and advanced routing. For the common case — fire tasks, run them reliably — Redis with acks_late and prefetch 1 is the pragmatic choice and what this guide uses.

Go-live checklist

• Web + worker as separate services, same image
• Managed Redis; REDIS_URL in both
• --concurrency matched to workload; prefetch 1 for long tasks
• acks_late on for crash safety
• Single Beat instance, or platform cronjobs
• Result backend TTL or ignore_result

References

• [Celery documentation](https://docs.celeryq.dev/en/stable/)
• [Celery + Redis broker](https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html)
• [Celery worker optimization (prefetch)](https://docs.celeryq.dev/en/stable/userguide/optimizing.html)
• [Celery periodic tasks (Beat)](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html)

Celery fits PandaStack cleanly: web and worker as two services from one repo, a managed Redis as broker, and native cronjobs that can stand in for Beat. Deploy your task queue on the free tier at [dashboard.pandastack.io](https://dashboard.pandastack.io).