Worker Processes: Background Job Processing for Web Apps

What Are Worker Processes?

In a web application, the web process handles incoming HTTP requests and returns responses. The worker process runs in the background, consuming jobs from a queue or executing scheduled tasks. This separation is fundamental to building responsive, scalable applications.

Worker processes allow you to offload slow, resource-intensive, or deferred work from the request cycle. Image processing, email delivery, PDF generation, data enrichment, and third-party API calls all belong in a worker — not in an HTTP handler.

The Procfile Model

The 12-factor app methodology formalizes this separation with process types. A Procfile defines the distinct process types in your application:

web: node server.js
worker: node worker.js
scheduler: node scheduler.js

Each process type runs independently and can be scaled separately. You might run 3 web dynos and 2 worker dynos depending on your traffic and job volume.

Anatomy of a Worker Process

A worker process typically follows this pattern:

1. 1Connect to a job queue or scheduler
2. 2Pull a job off the queue (or wait for one to arrive)
3. 3Execute the job
4. 4Acknowledge completion or report failure
5. 5Loop back to step 2

// worker.js — simple job consumer
const queue = require('./queue');

async function runWorker() {
  console.log('Worker started, waiting for jobs...');

  queue.process('email', async (job) => {
    console.log(`Processing email job ${job.id} for ${job.data.to}`);
    await sendEmail(job.data.to, job.data.template, job.data.context);
    console.log(`Email job ${job.id} completed`);
  });

  queue.process('report', async (job) => {
    console.log(`Generating report ${job.id}`);
    const pdf = await generateReport(job.data.reportId);
    await uploadToStorage(pdf, job.data.outputPath);
  });
}

runWorker().catch(console.error);

Concurrency Models

Workers can process jobs in three concurrency models:

Sequential: One job at a time. Simple to reason about, but slow for I/O-bound work.

// Sequential — process one job at a time
queue.process(1, async (job) => { ... });

Concurrent: Multiple jobs processed in parallel within a single process. Ideal for I/O-bound work (network calls, database queries) where jobs spend most time waiting.

// Concurrent — process up to 5 jobs at once
queue.process(5, async (job) => { ... });

Multi-process: Multiple worker processes running in parallel. Ideal for CPU-bound work or when you need process-level isolation.

# Run 4 worker processes
for i in {1..4}; do node worker.js & done

Containerizing Worker Processes

Packaging workers as Docker containers is best practice for cloud deployments. Each worker type becomes its own image:

# Dockerfile.worker
FROM node:20-slim

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

CMD ["node", "worker.js"]

Build and run:

docker build -f Dockerfile.worker -t your-registry/worker:latest .
docker run -d \
  -e REDIS_URL=redis://your-redis:6379 \
  -e DATABASE_URL=postgresql://user:pass@host/db \
  your-registry/worker:latest

Deploying Workers on PandaStack

PandaStack is a cloud PaaS that runs Docker containers. You can deploy worker processes as long-running containers alongside your web application.

For scheduled workers that run on a cron schedule — executing a batch job, syncing data, or generating reports at a fixed time — PandaStack's cronjob platform is the right fit:

npm install -g @pandastack/cli

# Deploy a batch worker that runs nightly at 1 AM
panda cronjob create \
  --name batch-processor \
  --image your-registry/batch-worker:latest \
  --schedule "0 1 * * *"

# Deploy a recurring sync worker every 20 minutes
panda cronjob create \
  --name sync-worker \
  --image your-registry/sync-worker:latest \
  --schedule "*/20 * * * *"

PandaStack tracks execution history for each run and streams logs in real time. If a worker crashes or exits with a non-zero code, the failure is recorded and visible in the dashboard at [dashboard.pandastack.io](https://dashboard.pandastack.io).

Graceful Shutdown

Workers should handle termination signals gracefully — finish the current job before exiting, rather than being killed mid-task:

process.on('SIGTERM', async () => {
  console.log('SIGTERM received — finishing current job before shutdown');
  await queue.close(5000); // wait up to 5 seconds for active jobs to finish
  process.exit(0);
});

Scaling Workers

Scale workers horizontally by running more instances. Because each worker pulls from the same queue, adding more workers automatically increases throughput — no code changes required.

Monitor these metrics to know when to scale:

• Queue depth: Number of jobs waiting to be processed. Rising queue depth means workers can't keep up.
• Job latency: Time from enqueueing to processing start. Increasing latency signals worker saturation.
• Error rate: Percentage of jobs failing. Spikes indicate upstream dependency issues.

Summary

Worker processes are the backbone of asynchronous processing in web applications. Containerize them for portability, design them for graceful shutdown, and right-size their concurrency for your workload. For scheduled batch workloads, PandaStack provides first-class cronjob support with execution history and log streaming. Visit [docs.pandastack.io](https://docs.pandastack.io) to get started.