What is RAG? A plain-English explanation for product and ops teams

The short version

RAG stands for Retrieval-Augmented Generation. It's a way to make a general-purpose language model answer questions about your data without retraining the model on it.

Every time a visitor asks something, two things happen:

1. Retrieval. The system searches your knowledge base and pulls back the few chunks of text most likely to answer the question.
2. Generation. Those chunks get pasted into the model's prompt, along with the question, and the model writes an answer grounded in what it sees.

That's the whole technique. Two steps, no fine-tuning, no special model. Any modern LLM can do it.

What problem RAG solves

A general-purpose language model has seen most of the public web. It has not seen your refund policy, your shipping cutoffs, or the engineer's note you pasted into your help center yesterday. Asked about any of those, it will either say “I don't know” (best case) or confidently make something up (worst case).

RAG closes that gap without changing the model. You give the model the relevant pages at request time, the same way a new agent would read the help center on their first day.

Three things RAG is NOT

It's not “training the AI on your data”

Marketing copy regularly says “our AI is trained on your help center.” Almost always, what they mean is RAG. Training rewrites the model's weights and is a multi-day, multi-thousand-dollar endeavour. RAG runs in 1.4 seconds per request and updates the moment you edit a document.

The practical difference matters: with RAG you can delete a customer's data from your knowledge base and the system genuinely forgets, because the data lives in your database, not baked into model parameters you can't inspect.

It's not a search engine bolted onto a chatbot

Keyword search returns documents. RAG returns answers. The retrieval step uses embeddings (vectors that capture meaning), not keyword matching, so a question phrased differently from how the document is written still finds the right chunk. The generation step synthesises an actual response, not a list of links.

It's not infallible

RAG can fail in two ways. The retrieval can miss the relevant chunk (because the chunking strategy was bad, or the question was ambiguous). Or the model can be handed the right chunk and still hedge. A well-designed system catches both: it falls back to “I don't have that information” and offers escalation to a human, rather than guessing.

How RAG shows up in customer support

The three places you'll see RAG in your support stack:

• AI agent answering FAQs. The visitor types a question; the AI looks up your help center; it returns a grounded answer with citations. This is the bread-and-butter use.
• Agent assist for humans. The agent is typing; the system suggests an answer pulled from your docs. The human edits and sends. Speeds up tier-1 handle time without removing the person.
• Internal knowledge search. Your team asks “what did the engineering team decide about timezone handling last month?” and gets an answer from the wiki. Same pipeline, different audience.

What makes RAG actually work in production

Four things in order of importance:

1. Clean source data. If your help center is contradictory or out of date, RAG will faithfully reproduce the mess. Clean the docs before you blame the AI.
2. Sensible chunking. Cut the docs into pieces small enough to be precise but large enough to carry meaning. ~800 tokens with overlap at boundaries is a strong default.
3. A grounded system prompt. Tell the model explicitly: “Answer only from the context. If the context doesn't cover the question, say so.” This is the difference between a useful AI and a hallucination machine.
4. Citations on every answer. Show the visitor which document the answer came from. Trust compounds, distrust compounds faster.

Bottom line

RAG is a clean engineering pattern: retrieve the right chunks, paste them into the model's context, ask for a grounded answer. Most of the “AI customer support” products you'll evaluate run a flavour of it. The differences are in chunking quality, retrieval precision, prompt grounding, and how the system behaves when it doesn't know.

If you want the depth version of how OpenAgent runs the pipeline end to end, read how RAG works inside OpenAgent.