How to Deploy a BullMQ Job Worker

BullMQ is the modern standard for background jobs in Node.js. Built on Redis, it gives you delayed jobs, repeatable jobs, rate limiting, priorities, and flow/parent-child job graphs. Deploying it well means understanding the producer/consumer split and getting Redis connection settings right, because BullMQ is particular about how it talks to Redis.

Producer and consumer

Your application enqueues jobs (the producer) and a separate process executes them (the worker/consumer). They share a queue name and a Redis connection.

Producer, typically inside your API:

import { Queue } from 'bullmq';

const connection = { url: process.env.REDIS_URL };
const emailQueue = new Queue('email', { connection });

await emailQueue.add('welcome', { userId }, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 1000 },
  removeOnComplete: 1000,
  removeOnFail: 5000,
});

Worker, a separate deployable:

import { Worker } from 'bullmq';

const worker = new Worker('email', async (job) => {
  if (job.name === 'welcome') {
    await sendWelcomeEmail(job.data.userId);
  }
}, {
  connection: { url: process.env.REDIS_URL },
  concurrency: 10,
});

Redis connection caveats

BullMQ uses blocking Redis commands (BRPOPLPUSH and friends), which requires maxRetriesPerRequest: null on the underlying ioredis connection. If you pass a raw connection object, set it:

const connection = {
  url: process.env.REDIS_URL,
  maxRetriesPerRequest: null,
};

This is the single most common BullMQ production error: a worker that connects but throws maxRetriesPerRequest warnings or silently stops processing. Also ensure your managed Redis allows enough connections; each worker and queue instance opens its own.

Concurrency and rate limiting

The concurrency option controls how many jobs a single worker processes in parallel. Because Node is single-threaded, set this based on how I/O-bound your jobs are. For jobs that mostly await network calls, a concurrency of 10-50 is reasonable. For CPU-heavy jobs, keep concurrency low and scale by adding replicas instead.

Rate limiting protects downstream services. To cap throughput, configure a limiter on the worker:

const worker = new Worker('email', processor, {
  connection,
  limiter: { max: 100, duration: 1000 }, // 100 jobs/sec
});

Graceful shutdown

A worker should finish in-flight jobs before exiting during a deploy. Handle SIGTERM:

process.on('SIGTERM', async () => {
  await worker.close();
  process.exit(0);
});

worker.close() stops accepting new jobs and waits for active ones to complete. Without this, a rolling deploy can kill jobs mid-execution and force them through your retry path unnecessarily.

Dockerfile

FROM node:20-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
CMD ["node", "worker.js"]

The worker has no HTTP port; it's a long-running background process.

Deploying on PandaStack

1. 1Provision a managed Redis instance (KubeBlocks-backed). Reference it via REDIS_URL.
2. 2Deploy your API as one container app (gets the public URL).
3. 3Deploy the worker as a separate container app, with node worker.js as the entrypoint and the same REDIS_URL.
4. 4Tail live logs to confirm the worker connected and is processing. BullMQ logs job completions and failures, which stream through the platform's Elasticsearch-backed log pipeline.

Both apps build with rootless BuildKit in ephemeral Job pods and deploy via Helm.

Scaling

Scale worker throughput by adding replicas (horizontal) for CPU-bound work, or raising concurrency for I/O-bound work. BullMQ coordinates through Redis, so multiple replicas of the same worker safely share the queue without double-processing, each job is delivered to exactly one worker. Watch out for scale-to-zero on the worker: an idle worker scaled to zero won't drain the queue until traffic or a scheduled trigger wakes it. For steady job volume keep one worker warm; for bursty workloads, scaling down between bursts is fine if a small startup delay is acceptable.

Monitoring and the dashboard

Bull Board is a popular web UI for inspecting queues, jobs, and failures. You can deploy it as a small extra container app pointed at the same Redis to get a visual on queue depth, active jobs, and the failed-job set. Pair that with the platform's server-side container metrics for CPU and memory.

Common pitfalls

• Missing maxRetriesPerRequest: null breaks blocking commands.
• Too few Redis connections on a small managed instance, BullMQ opens several per process.
• No graceful shutdown causes avoidable retries on every deploy.
• Over-high concurrency on CPU-bound jobs starves the event loop; scale replicas instead.

References

• BullMQ documentation: https://docs.bullmq.io/
• BullMQ connection and Redis options: https://docs.bullmq.io/guide/connections
• BullMQ workers and concurrency: https://docs.bullmq.io/guide/workers
• ioredis: https://github.com/redis/ioredis
• Bull Board UI: https://github.com/felixmosh/bull-board

BullMQ is robust once the Redis connection details and graceful shutdown are in place. Deploy an API, a BullMQ worker, and managed Redis together on PandaStack's free tier to try the full pattern: https://dashboard.pandastack.io