How to Deploy a RAG Chatbot to Production

A RAG (retrieval-augmented generation) chatbot has more moving parts than people expect. It's not just a wrapper around an LLM API, it's an ingestion pipeline, a vector store, an embedding step, a retrieval step, and a generation step, each with its own failure modes and scaling characteristics. Deploying it to production means treating those parts as distinct components and wiring them together cleanly. This guide lays out a pragmatic architecture and how to ship it.

The two pipelines

Every RAG system has two distinct flows:

1. 1Ingestion (offline): documents are chunked, embedded, and stored in a vector database. This runs when your knowledge base changes, often as a batch job or cronjob, not on every request.
2. 2Query (online): a user question is embedded, similar chunks are retrieved from the vector store, and those chunks plus the question are sent to an LLM to generate a grounded answer.

Keeping these separate is the single most important architectural decision. Ingestion is heavy and intermittent; query is light and latency-sensitive. They scale differently and should deploy as different units.

Ingestion as a job

Chunking and embedding a large corpus is a batch operation. Run it as a cronjob (for periodically refreshed sources) or a one-off worker, not inside your web app:

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=120)
chunks = splitter.split_documents(docs)
vectors = embed_model.embed_documents([c.page_content for c in chunks])
vector_store.add(vectors, metadatas=[c.metadata for c in chunks])

Chunk size is a real tuning parameter: too small and you lose context, too large and retrieval gets noisy. 500-1000 tokens with some overlap is a sensible starting point.

The query service

The online path embeds the question, retrieves top-k chunks, and asks the LLM to answer using only that context:

@app.post("/chat")
async def chat(q: Query):
    q_vec = embed_model.embed_query(q.message)
    hits = vector_store.search(q_vec, top_k=5)
    context = "\n\n".join(h.text for h in hits)
    prompt = f"Answer using only the context.\n\nContext:\n{context}\n\nQuestion: {q.message}"
    return StreamingResponse(llm.stream(prompt), media_type="text/event-stream")

Stream the response. RAG answers can be long, and streaming tokens to the client makes the bot feel responsive instead of making users stare at a spinner for several seconds.

Choosing and wiring the vector store

You have options: a dedicated vector database (Qdrant, Weaviate, Milvus), or PostgreSQL with the pgvector extension. For many production chatbots, pgvector is the pragmatic choice: it keeps your vectors in the same managed Postgres you're already running, so there's one fewer system to operate. A separate vector DB makes sense at very large scale or when you need advanced filtering and hybrid search.

With pgvector, your retrieval is a SQL query using a distance operator:

SELECT text, metadata
FROM chunks
ORDER BY embedding <=> $1
LIMIT 5;

Secrets and cost control

Your LLM and embedding API keys are the crown jewels and the cost center. Keep them in environment variables, never in code or logs. Add caching: identical or near-identical questions can reuse retrieved context, and you can cache embeddings for repeated queries in Redis to cut API spend. Set token limits on generation to bound per-request cost.

Deploying on PandaStack

The components map cleanly onto distinct deployables:

Steps:

1. 1Connect your repo and deploy the query API as a container app. Build runs in an ephemeral Job pod with rootless BuildKit and deploys via Helm.
2. 2Provision a managed PostgreSQL database for vectors (enable pgvector) and managed Redis for caching. DATABASE_URL is auto-wired.
3. 3Add LLM and embedding API keys as environment variables.
4. 4Set up the ingestion step as a cronjob so your knowledge base refreshes on a schedule.
5. 5Tail live logs across components to watch retrieval quality and generation latency.

Streaming responses use Server-Sent Events or chunked HTTP, both of which pass cleanly through Kong ingress, so the streaming UX works out of the box with automatic SSL.

Free tier fit

The query API is light and fits the free tier well, including scale-to-zero for low-traffic bots (accept a cold start on the first request after idle). The free-tier managed database is small, fine for a modest knowledge base or a demo; size up as your corpus grows since vector indexes consume real storage and memory. Ingestion as a cronjob is a great free-tier use: it runs on a schedule and costs nothing while idle.

Quality and observability

RAG fails in subtle ways: retrieval returns irrelevant chunks, or the LLM ignores the context and hallucinates. Log the retrieved chunks alongside each answer (the Elasticsearch-backed log pipeline captures this) so you can debug bad answers. Track retrieval relevance and response latency through the server-side metrics. Add a simple feedback signal (thumbs up/down) and store it, it's the cheapest way to know whether your chunking and top-k settings are working.

Common pitfalls

• Embedding on every request without caching burns API budget.
• Running ingestion in the web app makes deploys slow and couples two very different workloads.
• No streaming makes the bot feel sluggish.
• Undersized database for a large vector index causes slow retrieval.

References

• pgvector extension: https://github.com/pgvector/pgvector
• LangChain RAG guide: https://python.langchain.com/docs/tutorials/rag/
• OpenAI embeddings guide: https://platform.openai.com/docs/guides/embeddings
• Qdrant documentation: https://qdrant.tech/documentation/
• Server-Sent Events spec: https://html.spec.whatwg.org/multipage/server-sent-events.html

A production RAG chatbot is several specialized components wired together, separate the heavy ingestion from the light query path and the system becomes manageable. Deploy the query API, a pgvector-backed managed Postgres, Redis cache, and an ingestion cronjob together on PandaStack's free tier: https://dashboard.pandastack.io