How to Deploy Prefect Data Pipelines

Prefect is a modern workflow orchestration tool for Python — you write flows as decorated functions and Prefect handles scheduling, retries, concurrency, and observability. Deploying it self-hosted involves three pieces that confuse first-timers: the server (API + UI), the worker (which runs your flows), and the flow code/deployments themselves.

This guide deploys a self-hosted Prefect setup for running data pipelines in production.

Prefect's architecture

The key insight: the server stores state and schedules work; the worker does the actual execution by polling. They're separate processes. Your flow code runs inside (or is launched by) the worker.

Step 1: Provision PostgreSQL for the server

Prefect Server defaults to SQLite, which is unsuitable for production on ephemeral containers. Use PostgreSQL. On [PandaStack](https://dashboard.pandastack.io), create a managed PostgreSQL and point the server at it:

PREFECT_API_DATABASE_CONNECTION_URL=postgresql+asyncpg://<user>:<password>@<host>:5432/prefect

Prefect uses the async driver, so the URL uses postgresql+asyncpg://.

Step 2: Deploy the Prefect Server

FROM prefecthq/prefect:3-latest
CMD ["prefect", "server", "start", "--host", "0.0.0.0"]

1. 1Push the repo (or reference the image) to GitHub.
2. 2Create a container app on PandaStack for the server.
3. 3Set PREFECT_API_DATABASE_CONNECTION_URL (with the managed PostgreSQL) and PREFECT_SERVER_API_HOST=0.0.0.0.
4. 4Expose port 4200.
5. 5Add a custom domain for the UI/API; SSL is automatic.

The server runs migrations against PostgreSQL on startup. Once up, the Prefect UI is available at your domain.

Keep the server always-on (no scale-to-zero) — it's the scheduler and API; if it sleeps, nothing gets scheduled.

Step 3: Write a flow

A minimal flow with retries:

# flows.py
from prefect import flow, task
import httpx

@task(retries=3, retry_delay_seconds=10)
def fetch(url: str) -> dict:
    return httpx.get(url, timeout=30).json()

@task
def transform(data: dict) -> int:
    return len(data)

@flow(name="daily-etl")
def daily_etl(url: str):
    raw = fetch(url)
    count = transform(raw)
    print(f"Processed {count} records")
    return count

Retries, logging, and run tracking come for free from the decorators.

Step 4: Deploy a worker

The worker polls a work pool on the server and executes runs. Build an image that contains your flow code and its dependencies:

FROM prefecthq/prefect:3-latest
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["prefect", "worker", "start", "--pool", "default-pool"]

The worker needs to reach the server's API:

PREFECT_API_URL=https://prefect.example.com/api

1. 1Create a second container app for the worker.
2. 2Set PREFECT_API_URL to your server's API endpoint.
3. 3No public port needed — the worker only makes outbound calls to the server.
4. 4Keep it always-on so it's polling for scheduled runs.

Before the worker can pick up work, create the work pool (once) via the UI or CLI:

prefect work-pool create default-pool --type process

Step 5: Register deployments

A "deployment" tells Prefect how and when to run a flow. Define it in code and apply it (pointing at your self-hosted API):

# deploy.py
from flows import daily_etl

if __name__ == "__main__":
    daily_etl.from_source(
        source=".",
        entrypoint="flows.py:daily_etl",
    ).deploy(
        name="daily-etl-prod",
        work_pool_name="default-pool",
        cron="0 6 * * *",       # daily at 06:00 UTC
        parameters={"url": "https://api.example.com/data"},
    )

Run this once (as a job pointed at your server) to register the schedule. The server will now trigger daily-etl at 6am, and your always-on worker will execute it.

Step 6: Observe runs

Open the Prefect UI at your domain to see flow runs, logs, retries, and timing. This observability is Prefect's biggest advantage over hand-rolled cron scripts — you see exactly what ran, what failed, and why.

Self-hosted Prefect vs. Prefect Cloud

Prefect Cloud is excellent if you want zero orchestration-server ops and only run workers yourself. Self-host when you need the control plane on your own infrastructure. A nice hybrid pattern many teams use: Prefect Cloud for the control plane + self-hosted workers for your data.

Prefect vs. a platform cronjob

If all you need is "run one script daily," a PandaStack cronjob is far simpler. Reach for Prefect when you have *pipelines* — multiple dependent tasks, retries, fan-out, observability, and a UI to debug failures. It's orchestration, not just scheduling.

Operating tips

• Back up the server's PostgreSQL — it's your run history and state.
• Keep server and worker always-on so schedules fire and get executed.
• Bake dependencies into the worker image so flows have what they need.
• Pin Prefect versions across server and worker to avoid API mismatches.

References

• [Prefect documentation](https://docs.prefect.io/)
• [Self-hosting Prefect Server](https://docs.prefect.io/v3/manage/self-host)
• [Prefect workers and work pools](https://docs.prefect.io/v3/deploy/infrastructure-concepts/workers)
• [Prefect deployments](https://docs.prefect.io/v3/deploy/index)

---

Prefect's three-part model — server, worker, deployments — maps cleanly onto two always-on container apps plus a managed PostgreSQL, all of which PandaStack runs together with injected connections and automatic SSL on the UI. For simple schedules a cronjob may be enough; for real pipelines, deploy Prefect at [dashboard.pandastack.io](https://dashboard.pandastack.io).