Picture the support head of a hospital chain asking for an assistant that answers staff questions about the leave policy and the discharge procedure. A language model cannot do that on its own, because it has never read those documents. Retrieval-augmented generation (RAG) is the usual way to close the gap: find the relevant passages first, then have the model write an answer from them and point to where each part came from. Knowing what each step does is what lets you judge a proposal, a demo or a bug report later.
Why the model alone cannot answer
A language model produces text from two sources: the patterns it absorbed in training and whatever is in the current request. Your leave policy, last quarter's incident reports and the discount rules in the sales manual were not in the training data. If an older copy somehow was, the model has no record of which version it absorbed, and nothing in its output will flag the difference.
The obvious workaround is to paste the whole document into every request. It works for one PDF and stops working at ten. The context window (the amount of text a single request can carry) has a limit, cost rises with every token sent, and a model answers more reliably from a short relevant passage than from a long block where the answer is buried on page nine.
RAG splits the job in two. A retrieval step picks the few passages that matter for this question. The model then reads only those and writes the answer.
Five steps from a document to an answer
- 01Ingestconvert to text, record origin
- 02Chunksplit into passages that stand alone
- 03Embedstore vectors with metadata
- 04Retrievefind the closest passages
- 05Generateanswer from them, cite the source
The first three steps run when documents change; the last two run on every question.
Ingestion collects the files from wherever they live. In most mid-size companies that means a mix: a shared drive of PDFs, an internal wiki, the HR portal, and a folder of scanned circulars somebody kept. Each file is converted to text, and the pipeline records where it came from, who owns it and when it last changed.
Chunking splits each document into passages small enough to retrieve on their own. A twelve-page leave policy might become forty chunks, roughly one per clause or section, each carrying its heading so it still makes sense when read alone. Where you cut matters more than most teams expect, and we cover it in detail in how document splitting changes the answers.
Embedding turns each chunk into a vector, a list of numbers that places passages with similar meaning near each other. The vectors go into an index along with the chunk text and its metadata. Our preference is to start with a vector extension inside the Postgres database the application already runs, and move to a separate vector store only if volume forces it.
Retrieval runs at question time. The question is embedded the same way, and the index returns the chunks whose vectors sit closest to it. A keyword search usually runs alongside, to catch exact terms like a form number or a ward code, which embeddings match poorly. A reranking step can reorder the candidates before the top few go forward.
Generation sends the question and those chunks to the model, with instructions to answer only from that material and to name the chunk that supports each claim. What reaches the user carries a link to the source document and section.
One question, followed through
Suppose a staff nurse types: "Can I carry forward unused casual leave to next year?"
Retrieval embeds that sentence and searches the index. Among the five chunks that come back, the top one reads: "Leave policy > Casual leave > Carry forward. Casual leave lapses at the end of the calendar year and cannot be carried forward or encashed." The heading path at the start is there because chunking attached it; without it, the passage would not say which kind of leave it covers.
The model receives the question, the five chunks and an instruction to answer from them alone. It writes: "No. Casual leave lapses on 31 December and cannot be carried forward (Leave policy, section 4.2)." The application checks that section 4.2 was among the retrieved chunks before showing the answer.
Now change one thing. Suppose the policy was revised in March to allow carrying forward three days, and the index still holds the old version. Every step above runs correctly, and the answer is wrong. The model does not know the document is outdated. It only sees the chunk it was given.
What the citation is for
The citation lets a person check an answer in a few seconds instead of trusting it. It also makes the system debuggable. When an answer is wrong, the citation tells you whether retrieval brought back the wrong passage or the model misread the right one. Those are different bugs, and the fix for one does nothing for the other.
So the chunk identifiers, the document versions and the retrieval scores go in the log for every answer. Without them, a complaint about a wrong answer cannot be reproduced, and the review meeting has nothing to look at except someone's memory of what the screen said.
Three ways the answers go wrong
Most complaints about a RAG system trace back to one of three failures, and each has a different cause.
- Quotes a replaced policy
- Ingestion ran once, at launch
- Fix: re-index on change, version chunks
- Fluent answer, off topic
- Passage lost its heading or table
- Fix: test retrieval alone, then chunking
- Confident answer, no real citation
- Retrieval found nothing useful
- Fix: require refusal, reject uncited answers
The stale index is the one that turns up most. The documents changed and the index did not, as in the carry-forward example above. Ingestion has to run on a schedule or on change events, and each chunk has to carry its document version so old chunks are removed when a new version arrives.
The failure looks different with a wrong chunk. The passage holding the answer was split badly, often through a table, or the question uses words the passage does not. The answer still reads well, which hides the fact that it was written from the wrong material. The only way to see this is to test retrieval separately from generation: ask the question, read the five chunks that came back, and leave the model out of it.
A confident answer with no real source appears when retrieval returns nothing relevant and the model produces an answer anyway. The fix sits in two places. The prompt instructs the model to say it cannot answer when the passages do not cover the question, and the application refuses to display any answer whose citation does not match a retrieved chunk. The prompt instruction alone is not enough, because the model follows it most of the time rather than every time.
A fourth problem matters as soon as documents have different audiences: the index holds everything, so the answer can draw on a document the asker was never allowed to open. We treat that as its own design question in access control in RAG.
What to build first
- Pick one document set with one owner and one clear audience.
- Collect twenty real questions people have asked about it, and note the passage that answers each one.
- Build ingestion, chunking and retrieval, and check that the right passage comes back for those twenty before adding a model.
- Add generation with citations required, and drop any answer that arrives without one.
- Schedule re-indexing and log chunk identifiers with every answer.
Everything else in the pipeline gets tuned against those twenty questions, and measuring a RAG system explains how to turn them into scores you can track. Without that list you cannot tell a retrieval bug from a writing bug, which is why we ask for it before discussing which model to use. If you are scoping an assistant over your own documents, our AI solutions page lists how we approach RAG work.
