How to Deploy a Celery Background Worker

Celery is the standard task queue for Python. Deploying it correctly means treating the worker as a long-lived process distinct from your web app, choosing a broker, and getting concurrency and graceful shutdown right so tasks aren't lost on deploy. Here's a production-focused walkthrough.

The architecture: three moving parts

A Celery deployment has three components:

1. 1Producer — your web app, which enqueues tasks.
2. 2Broker — Redis or RabbitMQ, which holds the queue.
3. 3Worker — a separate process that consumes and runs tasks.

The worker is *not* your web server. It runs as its own deployable unit so you can scale it independently.

Defining the app and a task

# celery_app.py
import os
from celery import Celery

app = Celery(
    'tasks',
    broker=os.environ['CELERY_BROKER_URL'],
    backend=os.environ.get('CELERY_RESULT_BACKEND')
)

app.conf.update(
    task_acks_late=True,
    worker_prefetch_multiplier=1,
    task_reject_on_worker_lost=True
)

@app.task
def send_email(user_id):
    ...

The three config options matter for reliability:

• task_acks_late=True acknowledges a task only after it completes, so a crash mid-task re-queues it instead of losing it.
• worker_prefetch_multiplier=1 stops a worker from hoarding tasks it can't get to, which improves fairness for long tasks.
• task_reject_on_worker_lost=True re-queues a task if the worker dies.

Running the worker

celery -A celery_app worker \
  --loglevel=info \
  --concurrency=4 \
  --max-tasks-per-child=100

• --concurrency sets parallel task slots. For CPU-bound work, match it to available cores; for I/O-bound work you can go higher or use the gevent/eventlet pools.
• --max-tasks-per-child=100 recycles worker processes periodically to bound memory leaks in long-running deployments.

Graceful shutdown

Celery handles SIGTERM with a warm shutdown: it stops accepting new tasks and finishes in-flight ones before exiting. The catch is the platform's grace period — if your tasks run longer than it, they'll be killed. For long tasks, make them idempotent and keep acks_late on so a killed task re-runs cleanly on restart.

Scheduled tasks with Beat

For periodic jobs, Celery Beat is the scheduler. Run it as *one* process (never multiple, or you'll double-fire schedules):

celery -A celery_app beat --loglevel=info

app.conf.beat_schedule = {
    'cleanup-nightly': {
        'task': 'tasks.cleanup',
        'schedule': 3600.0,
    }
}

Dockerfile

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["celery", "-A", "celery_app", "worker", "--loglevel=info", "--concurrency=4"]

The worker has no HTTP port — it's a pure background process.

Deploying on PandaStack

A Celery setup maps onto PandaStack as multiple deployables sharing one broker:

1. 1Provision a managed Redis instance as the broker from the [dashboard](https://dashboard.pandastack.io).
2. 2Deploy your web app as a container app; it enqueues tasks.
3. 3Deploy the worker as a separate container app using the worker CMD above. Set CELERY_BROKER_URL to the Redis connection string.
4. 4Scale the worker independently when your queue grows.

For the Beat scheduler, you have two options: run it as a single dedicated container app (concurrency 1), or — if your periodic jobs are coarse — model them as PandaStack cronjobs that enqueue Celery tasks. The cronjob approach avoids running a permanent Beat process.

A caution on scale-to-zero

A Celery worker must stay running to consume the queue. Don't put the worker on a free-tier app that scales to zero, or tasks will sit unprocessed until something wakes it. Run workers on a tier that keeps at least one instance alive.

References

• [Celery: Daemonization and deployment](https://docs.celeryq.dev/en/stable/userguide/workers.html)
• [Celery task reliability and acks_late](https://docs.celeryq.dev/en/stable/userguide/tasks.html)
• [Celery Periodic Tasks (Beat)](https://docs.celeryq.dev/en/stable/userguide/periodic-tasks.html)
• [Celery configuration reference](https://docs.celeryq.dev/en/stable/userguide/configuration.html)

Run your web app, Celery worker, and managed Redis broker together on PandaStack's free tier — get started at [dashboard.pandastack.io](https://dashboard.pandastack.io).