How to Add Background Jobs to a Web App

Why background jobs exist

When a user clicks "sign up," they shouldn't wait while you send a welcome email, generate a thumbnail, sync to a CRM, and warm a cache. Any work that's slow, can fail independently, or doesn't need to block the response belongs in the background. The pattern: the web request enqueues a job and returns immediately; a separate worker process picks it up and does the work.

Good candidates for background processing:

• Sending emails and notifications
• Image/video processing and thumbnailing
• Calling slow third-party APIs
• Generating reports and exports
• Data syncs and ETL
• Anything you'd otherwise cram into a request and pray it finishes

The architecture

[Web request] --enqueue--> [Queue] <--dequeue-- [Worker(s)] --> does the work

Three pieces: a producer (your web app), a queue (a broker that holds jobs), and one or more consumers/workers (separate processes that execute jobs). Decoupling them means you can scale workers independently and the web tier stays fast and responsive.

Choosing a queue

For most web apps, a Redis-backed queue is the sweet spot, fast and easy. Postgres-backed queues are underrated when you want jobs enqueued in the *same transaction* as your data change (no "committed the row but the job vanished" race).

A minimal worker (Node + BullMQ)

// producer (in your web app) — enqueue and return immediately
const { Queue } = require('bullmq');
const emailQueue = new Queue('emails', { connection: { url: process.env.REDIS_URL } });

app.post('/signup', async (req, res) => {
  const user = await createUser(req.body);
  await emailQueue.add('welcome', { userId: user.id }); // enqueue
  res.status(201).json({ ok: true });                   // respond now
});

// worker (separate process)
const { Worker } = require('bullmq');
new Worker('emails', async (job) => {
  if (job.name === 'welcome') {
    await sendWelcomeEmail(job.data.userId);
  }
}, { connection: { url: process.env.REDIS_URL }, concurrency: 10 });

The web process and the worker process are separate deployments sharing the same Redis.

Design rule #1: make jobs idempotent

Jobs *will* run more than once, retries, redeliveries, at-least-once queues. Your job must produce the same result if executed twice. Use a natural idempotency key:

new Worker('emails', async (job) => {
  const sent = await db.wasWelcomeEmailSent(job.data.userId);
  if (sent) return;                       // already done; no-op
  await sendWelcomeEmail(job.data.userId);
  await db.markWelcomeEmailSent(job.data.userId);
});

Idempotency is the single most important property of a reliable job. Design for "this might run twice" from the start.

Design rule #2: retries with backoff and a dead-letter

Transient failures (a flaky API, a network blip) should retry, but with exponential backoff so you don't hammer a struggling dependency. After N failures, move the job to a dead-letter queue (DLQ) for inspection instead of retrying forever.

await emailQueue.add('welcome', { userId }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 2000 }, // 2s, 4s, 8s, ...
});

Monitor the DLQ. Jobs landing there are real problems wearing a disguise.

Design rule #3: don't lose jobs to the enqueue race

A classic bug: you commit a database row, then try to enqueue a job, but the process crashes in between, the row exists, the job doesn't. Solutions:

• Use a transactional outbox: write the job into a DB table in the same transaction as your data change, and a relay moves it to the queue.
• Or use a Postgres-backed queue so enqueue is part of the same transaction.

Deploying workers alongside your app on PandaStack

Workers are just long-running processes, which means they deploy like any other service. On PandaStack you run your web service and your worker as separate container deployments from the same (or related) repo, both connected to a shared managed Redis (or Postgres) for the queue.

• The web service handles HTTP and enqueues jobs.
• The worker service runs the consumer loop and processes jobs.
• A managed Redis (via KubeBlocks) acts as the broker, with its connection injected via env vars.

Because PandaStack auto-wires managed databases, the worker reads DATABASE_URL (and your Redis URL) from the environment rather than hardcoding them. You scale the worker's compute independently, memory-optimized (m1/m2) tiers for memory-heavy jobs like image processing, compute-optimized (c1/c2) for CPU-bound work.

# Two services in one project:
#   web:    runs the HTTP server (enqueues)
#   worker: runs the BullMQ Worker (consumes)
# Shared broker: managed Redis. Shared data: managed Postgres (DATABASE_URL injected).

Scheduled vs. on-demand jobs

Not all background work is event-driven. Recurring work (nightly cleanup, daily digests) is a cron concern, not a queue concern. PandaStack supports cronjobs natively, so periodic tasks run on a schedule without you building a scheduler, while your queue handles the event-driven jobs. Use the right tool: queue for "do this soon, triggered by an event," cron for "do this every day at 2am."

Conclusion

Background jobs keep your app fast by moving slow, failure-prone work out of the request cycle. The reliability comes from a few rules: idempotent jobs, retries with backoff and a dead-letter queue, and avoiding the enqueue race with a transactional outbox or DB-backed queue. Deploy the worker as its own service next to your web app, sharing a managed broker.

PandaStack lets you run web services, workers, managed Redis/Postgres, and cronjobs together in one project, the full background-job stack without stitching providers. Try it on the free tier at https://dashboard.pandastack.io.

References

• BullMQ documentation: https://docs.bullmq.io/
• Celery documentation: https://docs.celeryq.dev/en/stable/
• Transactional outbox pattern: https://microservices.io/patterns/data/transactional-outbox.html
• AWS SQS developer guide: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/welcome.html