"Hostel ka fee semester wise dena hai ya yearly?" is the kind of question a parent might type into the admissions assistant of an engineering college. The answer sits in section 3 of the fee circular, under a heading that says "Payment schedule for residential students", which shares almost no words with the question. Whether that section comes back from the index, and whether the model then reports it correctly, are two separate events. A RAG system needs a score for each, or nobody can say whether last week's change made it better or worse.
Build the evaluation set from real questions
The evaluation set is a list of questions, each paired with the passage that answers it and a reference answer written by someone who knows the material. Collect the questions from where people already ask them: the helpdesk inbox, the website search log, the questions typed into the pilot. Those arrive the way people actually phrase them, in Hinglish, with the wrong terminology and missing context. That is exactly what makes retrieval hard, and exactly what the set needs to contain.
For each question, record the chunk identifiers (or the document and section) that hold the answer. That reference passage is what the retrieval score is measured against. Then add a short reference answer, or a list of facts the answer must contain, for the answer score.
Two kinds of question are easy to forget. Include questions the documents cannot answer, such as next year's cut-off ranks, and questions a given user is not allowed to see answered, such as a staff salary band asked from a student account. They test behaviour most teams never measure.
Start with fifty questions and add every failure found in production. Keep the set in the repository next to the chunking rules and the prompt. All three change together, and a pull request should show all three.
A model can draft questions from the documents to get started, but keep those apart from the real ones. Generated questions borrow the document's wording, so retrieval scores well on them and poorly on what parents and students actually type.
Did the right passage come back
The retrieval score asks one thing: for this question, is a reference passage among the chunks returned? Counted across the set, that is the hit rate. Record the rank as well. A passage at position one and a passage at position nine both count as hits, but the model treats them very differently once it has eight other chunks to read.
This score needs no model call. It runs in seconds on every change to the chunker, the embedding model or the index settings, which is why we make it the first gate. An answer cannot be better than the passages it was written from, and how documents are split is usually the first thing a low hit rate points to.
Measure it separately by document type and by question source. A healthy overall hit rate can hide a very poor one for the scanned PDFs from the accounts office, or for questions typed by parents rather than students.
Is the answer faithful and correctly cited
Once the right passages come back, judge the answer on three things.
Faithfulness asks whether every claim in the answer is supported by the retrieved chunks. An answer that adds a plausible detail from the model's training data is unfaithful even when that detail happens to be true, because next time it will not be. Correctness asks whether the answer matches the reference answer or contains the required facts. Citation accuracy asks whether the cited chunk actually supports the sentence it is attached to.
- 01Question setreal questions, reference passages
- 02Retrieverecord chunk identifiers
- 03Score retrievalhit rate and rank
- 04Generateanswer from retrieved chunks
- 05Score answersfaithfulness, correctness, citations
Run on every change to chunker, index, prompt or model
Retrieval is scored before the model runs, so each stage has its own number.
People score a sample of answers against written criteria. A second model (a judge model) can score the rest at scale, but only after its verdicts have been compared with the human-scored sample, and its prompt should be versioned like any other code. A judge model is wrong in consistent ways, and those ways need to be known before anyone trusts its numbers.
The questions it should refuse
A question outside the documents has one correct answer: the system says it does not have that information, and ideally says where to ask instead. The same applies to a question whose answer sits in a document the asker may not see, which depends on access control inside retrieval working as designed.
Measure both directions on the evaluation set. On out-of-scope questions, the system should refuse nearly every time. On in-scope questions, it should rarely refuse. A system that never says "I don't know" is not more capable. Its failures are hidden inside confident answers, and users stop trusting it the first time they check one against the actual circular.
Reading the two scores together
The two numbers side by side tell you where to look, which is the reason for keeping them apart.
- Model working from the wrong material
- Check chunking, metadata, embeddings
- Right passages, answer drifts
- Check prompt, chunk count, model
- Set no longer matches real questions
- Refresh it from last month's log
The third column is easy to miss. The scores can stay high while the questions people ask move on, for example when admissions season ends and the questions shift from fees to exam schedules. The evaluation set has to move with them.
Where evaluation goes wrong
The most common mistake is one overall score that mixes retrieval and generation, so a drop tells you something broke and nothing about where. Next is an evaluation set written by the team that built the system, in the words of the documents, which flatters retrieval. Then an evaluation that ran once, at launch, and never again.
Three more. A judge model trusted without ever being checked against people. No out-of-scope questions, so refusal behaviour is never tested. And evaluation run under an admin account, so access-related failures never appear. Once a change passes offline, shadow runs and canaries are how to check it against live traffic without exposing users to it.
Getting started
- Pull fifty real questions from the helpdesk and search logs.
- For each, record the passage that answers it, and mark the ones the documents cannot answer.
- Run retrieval alone and record hit rate and rank.
- Add generation, and have people score a sample for faithfulness and citations.
- Put the run in the build pipeline so it executes on every change.
The reference passages in step two are the part we do first. Without them there is no retrieval score, and without a retrieval score every other number is a guess about where the problem is.
