How to Deploy a Scrapy Spider to Production (with Postgres)

Scrapy is the odd one out among things people deploy. It's not a web server — there's no port to expose, no health check to pass. A spider is a batch job: it starts, crawls, writes items somewhere, and exits. Most deployment guides ignore this and tell you to wedge it into a web-service slot, where the platform then restarts your "crashed" app every time the crawl finishes successfully.

The right shape for a production spider is a scheduled job that writes to a real database. Here's the full setup: a working spider, a Postgres item pipeline driven by DATABASE_URL, a Dockerfile, and a scheduled deployment.

The project

Assuming you have a Scrapy project — if not:

pip install scrapy psycopg2-binary
scrapy startproject pricewatch
cd pricewatch
scrapy genspider books books.toscrape.com

A minimal spider that follows pagination:

# pricewatch/spiders/books.py
import scrapy

class BooksSpider(scrapy.Spider):
    name = "books"
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        for book in response.css("article.product_pod"):
            yield {
                "title": book.css("h3 a::attr(title)").get(),
                "price": book.css(".price_color::text").get(),
                "url": response.urljoin(book.css("h3 a::attr(href)").get()),
            }
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, self.parse)

Locally you'd dump this to a file with scrapy crawl books -O books.json. In production that's a trap: a container's filesystem is ephemeral, so any feed export written to local disk vanishes when the job pod exits. Items need to leave the container — a database is the simplest destination.

Writing items to Postgres with a pipeline

Scrapy's item pipelines are the right hook. This one reads a single DATABASE_URL environment variable — the convention every managed database platform uses — and upserts rows:

# pricewatch/pipelines.py
import os
import psycopg2

class PostgresPipeline:
    def open_spider(self, spider):
        self.conn = psycopg2.connect(os.environ["DATABASE_URL"])
        self.cur = self.conn.cursor()
        self.cur.execute("""
            CREATE TABLE IF NOT EXISTS books (
                url        TEXT PRIMARY KEY,
                title      TEXT NOT NULL,
                price      TEXT,
                scraped_at TIMESTAMPTZ NOT NULL DEFAULT now()
            )
        """)
        self.conn.commit()

    def process_item(self, item, spider):
        self.cur.execute(
            """
            INSERT INTO books (url, title, price, scraped_at)
            VALUES (%s, %s, %s, now())
            ON CONFLICT (url) DO UPDATE
                SET title = EXCLUDED.title,
                    price = EXCLUDED.price,
                    scraped_at = now()
            """,
            (item["url"], item["title"], item["price"]),
        )
        return item

    def close_spider(self, spider):
        self.conn.commit()
        self.cur.close()
        self.conn.close()

Two deliberate choices here. ON CONFLICT ... DO UPDATE makes the crawl idempotent — re-running the spider updates prices instead of duplicating rows, which matters because scheduled crawls *will* re-run. And CREATE TABLE IF NOT EXISTS in open_spider stands in for migrations: a scraper's schema is small enough that a self-initializing table beats maintaining an Alembic setup. If your schema grows real relationships, graduate to proper migrations run as a separate step.

Batching note: this commits once at the end. For long crawls, commit every N items in process_item so a crash near the end doesn't lose the whole run.

Register the pipeline:

# pricewatch/settings.py
ITEM_PIPELINES = {
    "pricewatch.pipelines.PostgresPipeline": 300,
}

Production settings worth setting

The defaults that bite people in production, all in settings.py:

# Be a polite crawler — this also keeps you off IP blocklists
ROBOTSTXT_OBEY = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0
DOWNLOAD_DELAY = 0.5
CONCURRENT_REQUESTS_PER_DOMAIN = 4

# Scrapy's default log level is DEBUG — one line per request.
# In a log-aggregated environment that's pure noise and cost.
LOG_LEVEL = "INFO"

# Fail the job visibly instead of retrying forever
RETRY_TIMES = 2
DOWNLOAD_TIMEOUT = 30

The LOG_LEVEL one is underrated. A 10,000-page crawl at DEBUG produces tens of thousands of log lines; at INFO you get the stats summary Scrapy prints at the end — items scraped, response codes, errors — which is exactly what you want to see in a job log.

One more Scrapy-specific gotcha: the process exit code. scrapy crawl exits 0 even when the spider finished with errors, unless you tell it otherwise. If your scheduler alerts on failed runs (it should), close the spider explicitly when something's wrong — raise CloseSpider in the spider — or check the finish_reason in a spider-closed signal handler and sys.exit(1) on anything other than "finished".

The Dockerfile

FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# No EXPOSE — this is a batch job, not a server
CMD ["scrapy", "crawl", "books"]

With requirements.txt:

scrapy>=2.11
psycopg2-binary>=2.9

python:3.12-slim works because psycopg2-binary ships prebuilt wheels — no compiler needed. If a target site forces you into headless-browser territory (scrapy-playwright), you'll need a heavier base image with browser dependencies; cross that bridge only when a site actually requires JavaScript rendering.

Deploying it on PandaStack

This is where the batch-job shape pays off, because PandaStack has cronjobs as a first-class deployment type — you don't pretend the spider is a web service.

1. 1Provision a managed PostgreSQL (14.x or 16.x) from the dashboard.
2. 2Connect the Git repo as a cronjob and set the schedule — standard cron syntax, e.g. 0 6 * * * for a daily 6:00 UTC crawl. The build uses your Dockerfile; images are built with rootless BuildKit in ephemeral Kubernetes Job pods, so there's no Docker daemon to babysit.
3. 3Attach the database to the job. DATABASE_URL is injected into the container automatically — the pipeline above works with zero configuration, and no credentials ever live in your repo.
4. 4Push. The build runs, logs stream live, and the spider fires on schedule from then on. Each run's output is in the job logs, including Scrapy's closing stats block.

Any other configuration — a SCRAPE_START_URL, API keys for a proxy service — goes in as ordinary environment variables through the dashboard, read with os.environ like the database URL.

If you need on-demand crawls instead

A schedule covers most scraping workloads, but if you need to trigger crawls via API — say, a user action kicks off a scrape — deploy [Scrapyd](https://scrapyd.readthedocs.io/) as a container app instead. Scrapyd is a small server that listens on port 6800 and accepts spider runs over HTTP. That variant *is* a web service, so deploy it as a container app rather than a cronjob, with the same attached database. On the free tier, idle container apps scale to zero, which suits a Scrapyd instance that only wakes up for occasional jobs — just budget for the cold start on the first request.

The checklist

• Items go to Postgres via a pipeline, not to the container filesystem
• Upserts (ON CONFLICT) so reruns are idempotent
• LOG_LEVEL = "INFO", autothrottle on, robots.txt respected
• Non-zero exit on failed crawls so scheduled-run failures are visible
• DATABASE_URL from the environment — never hardcoded

That's a spider you can leave alone for months. If you want to try the setup end to end, the whole thing deploys with a git push on https://pandastack.io.