How to Deploy a Python RQ Background Worker

RQ (Redis Queue) is the simplest serious job queue in the Python ecosystem. It uses Redis as the broker, has no heavyweight dependencies, and its mental model is plain functions enqueued for background execution. The deployment question that trips people up is structural: a worker is a long-running process that is not an HTTP server, so it deploys differently from your web app, and it needs its own connection to Redis.

The two-process model

A typical setup has two deployable units sharing one Redis instance:

• Web app: enqueues jobs (queue.enqueue(...)) and serves HTTP.
• Worker: pulls jobs off the queue and executes them.

They share code (the task functions) but run as separate processes, often scaled independently. Your enqueuing side:

from redis import Redis
from rq import Queue
import os

redis_conn = Redis.from_url(os.environ["REDIS_URL"])
q = Queue("default", connection=redis_conn)

def kick_off_report(user_id):
    q.enqueue("tasks.generate_report", user_id, job_timeout=600)

And the task itself in tasks.py:

def generate_report(user_id):
    # heavy, slow work that shouldn't block a web request
    ...

The worker entrypoint

The worker is started with the rq worker command, pointed at the same Redis and the queues it should consume:

rq worker default high low --url $REDIS_URL

Workers process jobs in priority order of the queues listed. For containerized deployment, wrap this in a small script or use it directly as the container command. A clean Dockerfile for the worker:

FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["rq", "worker", "default", "high", "low", "--url", "${REDIS_URL}"]

Note that CMD in exec form doesn't expand environment variables, so for the URL substitution use a shell entrypoint:

CMD ["sh", "-c", "rq worker default high low --url $REDIS_URL"]

Reliability: retries, timeouts, and crashes

Three things make a worker production-grade:

1. 1Job timeouts prevent a stuck job from blocking the worker forever. Set job_timeout per job as shown above.
2. 2Retries handle transient failures. RQ supports retry policies:

from rq import Retry
q.enqueue("tasks.send_email", to, retry=Retry(max=3, interval=[10, 30, 60]))

1. 1Graceful shutdown. RQ workers handle SIGTERM by finishing the current job before exiting (a "warm shutdown"). This is exactly what you want during a rolling deploy so you don't kill a job mid-flight.

A dead-letter pattern is worth adding: failed jobs land in a failed-job registry where you can inspect and requeue them rather than losing them silently.

Deploying on PandaStack

A worker has no HTTP port, so it deploys as a container app whose role is a long-running background process rather than a web service. The steps:

1. 1Provision a managed Redis instance (via KubeBlocks). PandaStack injects the connection details; reference them via REDIS_URL.
2. 2Deploy your web app as one container app (it gets the public URL).
3. 3Deploy the worker as a separate container app pointing at the same Redis, using the worker command as its entrypoint.
4. 4Watch live logs from the worker to confirm it connected to Redis and is waiting for jobs (*** Listening on default, high, low...).

The build runs in an ephemeral Kubernetes Job pod with rootless BuildKit and deploys via Helm, the same pipeline for both the web app and the worker.

Scaling concurrency

There are two ways to add throughput: run more worker replicas, or run more worker processes per container. For CPU-bound tasks, more replicas on a compute-optimized tier is cleaner. For I/O-bound tasks, you can run several worker processes. Be careful with scale-to-zero on the worker: a worker that scales to zero won't pick up jobs while idle. For a steady stream of jobs keep at least one worker warm; for sporadic batch jobs, scaling down between bursts saves money but adds startup latency before the queue drains.

Monitoring

Keep an eye on queue depth and failed-job count. The simplest dashboard is rq info run against your Redis, or the rq-dashboard web UI deployed as a small extra service. Server-side metrics on the worker container give you CPU and memory; if memory climbs steadily, you likely have a leak in a task or are holding large objects between jobs.

Common pitfalls

• Forgetting the worker is a separate deploy. Enqueuing without a running worker means jobs pile up and never execute.
• No job timeout. One hung job can stall a single-process worker indefinitely.
• Importing the web app in tasks. Keep task modules importable without booting the whole web framework, or worker startup gets slow and fragile.
• Scaling Redis too small. Free-tier managed databases are sized for dev/hobby; a high-throughput queue needs more memory.

References

• RQ documentation: https://python-rq.org/docs/
• RQ workers and shutdown semantics: https://python-rq.org/docs/workers/
• RQ retry and exceptions: https://python-rq.org/docs/exceptions/
• redis-py client: https://redis.readthedocs.io/en/stable/
• rq-dashboard: https://github.com/Parallels/rq-dashboard

RQ keeps background processing simple, and the only deployment trick is treating the worker as its own long-running process beside your web app. Deploy a web app, a worker, and managed Redis together on PandaStack's free tier to try the pattern: https://dashboard.pandastack.io