How RAG works inside OpenAgent: from PDF upload to grounded answer

What RAG actually is

Retrieval-Augmented Generation is a two-step pipeline. Step one is retrieval: given a user query, find the chunks of your private data most likely to contain the answer. Step two is generation: paste those chunks into the model's context window, then ask it to answer using only what it sees.

That's it. There's no fine-tuning, no continued pre-training, no “training the AI on your data” in any literal sense. The model weights never see your refund policy. They see your refund policy at inference time, the same way a new agent sees it: by reading it from a document.

This is a feature, not a limitation. Because nothing is baked into the weights:

• You can edit your knowledge base at 11pm and the next answer reflects the change. No retraining cycle.
• You can delete a customer's data and the system genuinely forgets, because the data lives in your database, not in model weights you don't control.
• You can swap the model without rebuilding the knowledge base. The chunks are model-agnostic; only the embeddings are tied to a model and those are cheap to recompute.

The pipeline, end to end

Here's what happens between a visitor typing “What's your refund window?” and the AI returning a grounded answer in ~1.4 seconds:

           ┌─────────────────────────────────────────────────────────┐
   YOU →   │  Knowledge base ingest (one-time per document)            │
           │                                                            │
           │   PDF/URL/text                                             │
           │       │                                                    │
           │       ▼                                                    │
           │   ┌───────────┐  ┌───────────┐  ┌───────────┐              │
           │   │ Extract   │→ │ Chunk     │→ │ Embed     │→ pgvector    │
           │   │ (text +   │  │ (~800 tok │  │ (768-dim  │              │
           │   │ markdown) │  │ + overlap)│  │ vectors)  │              │
           │   └───────────┘  └───────────┘  └───────────┘              │
           └─────────────────────────────────────────────────────────┘
                                                          │
                                                          │
           ┌─────────────────────────────────────────────────────────┐
 VISITOR → │  Query time (every message)                              │
           │                                                            │
           │   "What's your refund window?"                             │
           │       │                                                    │
           │       ▼                                                    │
           │   ┌───────────┐  ┌───────────┐  ┌───────────┐              │
           │   │ Embed     │→ │ Vector    │→ │ Build     │→ LLM →       │
           │   │ query     │  │ search    │  │ prompt    │  answer +    │
           │   │ (same dim)│  │ (top-k=8) │  │ (chunks   │  citations   │
           │   │           │  │           │  │ + rules)  │              │
           │   └───────────┘  └───────────┘  └───────────┘              │
           └─────────────────────────────────────────────────────────┘

Step 1: extract

OpenAgent accepts three knowledge source types out of the box: PDFs, URLs (the platform fetches and parses the page), and raw text or markdown pasted into the dashboard. Extraction normalises everything into a single internal format: clean text, optional markdown formatting, and a stable source URL we can cite back to.

PDFs are the messy one. A PDF can be born-digital (real text inside) or scanned (an image of text inside). For scanned PDFs the platform falls back to a vision-capable model to OCR the page, because a chunker fed an empty string of text won't help anyone.

Step 2: chunk

Models have context windows. Even a million-token window is a bad place to dump your entire knowledge base for every request, because the model gets distracted and the bill explodes. So we cut the document into chunks small enough to be precise but large enough to carry meaning.

The default OpenAgent chunker targets ~800 tokens per chunk with a 100-token overlap at the boundaries. Overlap matters: if a sentence about “30-day refund window” straddles a chunk boundary, neither chunk on its own retrieves cleanly. The overlap means at least one chunk has the full sentence.

Headings, lists, and tables get extra weight. A chunk that opens with an H2 retains the heading text as a prefix, because the heading is usually a search term in disguise.

Step 3: embed

Embeddings turn each chunk into a fixed-length vector of floating-point numbers (768 dimensions in OpenAgent's default config, using Google's gemini-embedding-001 model). Two chunks whose meanings are similar end up close together in that 768-dimensional space; two chunks about wildly different topics end up far apart.

We store the vectors in Postgres, using the pgvectorextension. That's deliberate: we want vector search to be a boring database query, not a separate piece of infrastructure with its own credentials, region, and outage page. The vectors live next to the source row, in the same transaction.

-- The relevant table, roughly
CREATE TABLE knowledge_chunk (
  id          uuid PRIMARY KEY,
  document_id uuid REFERENCES knowledge_document(id),
  body        text NOT NULL,
  embedding   vector(768) NOT NULL,
  start_token int,
  end_token   int,
  created_at  timestamptz DEFAULT now()
);

CREATE INDEX ON knowledge_chunk
  USING hnsw (embedding vector_cosine_ops);

Step 4: retrieve

When a visitor message arrives, OpenAgent embeds the query with the same model used at ingest time and runs a cosine-similarity search against the chunk index. The default top_k is 8: we fetch the eight chunks whose vectors are closest to the query vector.

Eight is a deliberate sweet spot. Fewer than that and a relevant chunk often gets edged out by a near-duplicate. More than that and the model loses focus over the long context, which shows up as watery, off-topic answers. You can override top_k per agent in OpenAgent's dashboard.

We also apply per-site scoping: a visitor on yoursite.com's widget only retrieves chunks from documents that agent on that site has been granted. If you run a multi-brand workspace, your Brand-A knowledge never bleeds into Brand-B answers.

Step 5: prompt construction

The retrieved chunks get pasted into a structured prompt the model sees. The shape, simplified:

SYSTEM:
You are Aria, the support agent for Your Company.
Answer ONLY from the context below. If the context
doesn't cover the question, say "I don't have that
information" and offer to escalate to a human.
Always cite the source at the end like [1], [2].

CONTEXT:
[1] (knowledge_doc: refunds.pdf, chunk 3 of 12)
"We offer a 30-day refund window from the date of
delivery. Refunds are processed to the original
payment method within 5 business days..."

[2] (knowledge_doc: shipping.pdf, chunk 1 of 4)
"Damaged-on-arrival claims must be filed within
48 hours of delivery..."

USER:
What's your refund window if the item arrived damaged?

The system prompt is the most important thing in the pipeline that teams overlook. “Answer only from the context” isn't decorative; it's the difference between an AI that admits when it doesn't know and an AI that confidently makes up a refund policy that gets you legally exposed. OpenAgent ships a default grounded prompt and lets you override it per agent.

Step 6: generate and cite

The model returns an answer plus citation markers. OpenAgent post-processes the response: it expands each [1] into a clickable link back to the source document, so the visitor (and your team) can verify the answer didn't hallucinate.

Citations are not optional. The single highest-leverage thing you can do to make AI support feel trustworthy is to show the visitor which document the answer came from. They click it once, see the canonical source, and stop second-guessing the bot.

How to add your data, step by step

The path inside the OpenAgent dashboard is Dashboard → Knowledge → Add document. Three input modes:

• Upload file: drop in a PDF, DOCX, TXT, or Markdown file. Up to 25 MB per file (covered by the 5 GB attachment storage on the Pro plan).
• From URL: paste a URL. The platform fetches the page server-side (no CORS dance), strips boilerplate, and indexes the article body. Great for help-center pages and product docs you already publish.
• Paste text: for inline knowledge like “our office hours are 9-6 IST” that doesn't deserve a whole document.

Once a document lands, indexing is asynchronous and idempotent. You can keep editing the title, swap which agents have access, or replace the file. Re-indexing only embeds the chunks that changed.

Three things teams get wrong when they build RAG themselves

We've looked at a lot of homegrown RAG implementations. Three recurring mistakes:

1. Chunking by fixed character count. A 1,500-character chunker happily slices a sentence at character 1,500 mid-clause. Retrieval then returns a half-sentence the model has to guess at. Chunk by tokens, with overlap, and respect paragraph boundaries.
2. Embedding the question, retrieving by raw text similarity. A bunch of teams ship with TF-IDF or BM25 because it's in their existing search stack. It works for keyword-rich queries and falls apart on natural-language paraphrases. Use a real embedding model and a vector index.
3. No system prompt grounding. They embed and retrieve beautifully, then send the chunks to the model with a prompt like “Answer the user's question.” The model happily ignores the chunks and answers from training data. Always pin the model to “only use the context below” in the system prompt, and instruct it to escalate when the context is silent.

When RAG fails, and what to do

RAG isn't a perfect retrieval algorithm. Two failure modes you should plan for:

• The answer isn't in the knowledge base. A well-prompted agent will say “I don't have that information” and offer to escalate to a human. OpenAgent ships with a built-in escalate_to_human tool the agent can call; configure the escalation policy in Settings → Escalations.
• The chunk got retrieved but the model glossed over it. This shows up as “I'm not sure, but…” followed by an answer that ignores the source. Fix the prompt: add “If the context contains the answer, cite it and be concrete; do not hedge.”

Bottom line

RAG isn't magic and it isn't hard. It's a five-step pipeline (extract, chunk, embed, retrieve, prompt) that any decent platform can run for you, and the hard part is the boring part: clean source data, sensible chunking, a grounded system prompt, and citations on every answer. OpenAgent does the pipeline; you bring the documents and the LLM keys.