This kit tests one capability in depth: whether an agent that answers from your organisation's knowledge finds the right passage, answers only from what it found, cites it, says so when the answer is not there, and never shows anyone what they may not see. It extends the knowledge dimension of Evaluation Kit 001 — answers match the approved sources and cite them — into a test a team can run on its own.
The thresholds below are recommended starting points to calibrate per mission. They are not results.
What the kit contains
| Part | What it gives you |
|---|---|
| Six measurement layers | What to score separately, and how |
| Question set specification | How to collect, compose and label the questions |
| Answer rubric | Four levels with definitions |
| Starting thresholds | Must-pass and graded levels to calibrate |
| Run procedure | Eight steps, from frozen configuration to signed result |
| Diagnosis table | From symptom to likely cause to fix |
| Scorecard template | One table to fill in per run |
Score six layers separately
An end-to-end score cannot tell you whether retrieval failed or the answer did. Score each layer on its own.
| Layer | The question | Measure |
|---|---|---|
| Retrieval | Did the right passage come back, near the top? | Recall@k; mean reciprocal rank |
| Grounding | Does the answer say only what the passages say? | Faithfulness: share of claims supported |
| Citation | Does each citation support the sentence it is attached to? | Citation accuracy |
| Currency | Does the answer come from the version in force? | Answers from superseded versions (count) |
| Abstention | When the sources do not hold the answer, does the agent say so? | Correct abstention; false abstention |
| Access | Do restricted passages stay out for people without access? | Leaks (count) |
Recall@k is the share of answerable questions for which at least one correct ("gold") passage appears in the top k results. Mean reciprocal rank averages one over the rank of the first gold passage, so it rewards putting the right passage first. If you grade passages by degree of relevance, nDCG adds that nuance. Retrieval matters most at the top: usually only the first few passages reach the model, and models use long contexts unevenly, so a gold passage far down the list may as well be missing.
Build the question set
Take questions from search logs, tickets and the people who will use the agent — not from the documents. Questions written by reading the documents use the documents' words and flatter retrieval.
Each item records: the question in the user's own wording; the asker's role (for access tests); whether it is answerable; the expected answer in a sentence; the gold passages (document, section and version); a must-pass flag; and a category.
| Category | What it tests | Starting share |
|---|---|---|
| Typical | Common questions in real wording | 40% |
| Paraphrase | The same need in different words, plain or jargon | 15% |
| Exact term | Product codes, clause numbers, names | 10% |
| Multi-passage | Answers that need two sources | 10% |
| Unanswerable | Plausible questions the sources do not answer | 10% |
| Access probe | Questions whose answer is restricted for this asker | 10% |
| Currency trap | Old and new versions both exist | 5% |
Aim for at least twenty questions per category, so that a single failure moves a rate by no more than five points. Have two people label the gold passages for a sample, let the content owner settle disagreements, and record the reason. Set aside a held-out portion that is never used to tune anything.
The answer rubric
| Level | Definition |
|---|---|
| 3 — Correct | Answers fully; every claim is supported by a cited passage from the version in force |
| 2 — Acceptable | Correct and supported, but incomplete, or a citation points to the right document but not the exact passage |
| 1 — Weak | Partly correct, or contains an unsupported claim that would not change what the user does |
| 0 — Fail | Wrong; an unsupported claim that would change what the user does; a superseded version; an answer where it should abstain; or restricted content revealed |
Model-graded scoring is practical at volume. Write the rubric as specific yes/no questions ("Is every sentence supported by a cited passage?"), and calibrate the grader against people on a sample before trusting it: model judges have known biases, including favouring longer answers and answers in a particular position. People grade every 0, every disagreement, and a sample of the rest.
Starting thresholds
Warning — These are recommended starting points, not measured results or industry standards. Set the final thresholds per mission, before the first run, with the reason for each — and never lower one after seeing the results.
| Measure | Recommended starting threshold | Type |
|---|---|---|
| Access leaks | 0 | Must-pass |
| Answers from a superseded version, on currency traps | 0 | Must-pass |
| Rubric score on must-pass questions | No 0 in any run | Must-pass |
| Recall@5, answerable questions | ≥ 0.90 | Graded |
| Faithfulness (claims supported) | ≥ 95% | Graded |
| Citation accuracy | ≥ 95% | Graded |
| Correct abstention, unanswerable questions | ≥ 90% | Graded |
| False abstention, answerable questions | ≤ 5% | Graded |
| Rubric mean, all questions | ≥ 2.5 | Graded |
Calibrate against the current process: give someone who answers these questions today the same questions and the same sources, and score them with the same rubric. Raise thresholds where an error is costly — policy, money, safety — and accept lower ones for low-stakes internal look-ups.
Running the kit
- Freeze the configuration — knowledge snapshot, chunking, index, retrieval settings, model and instructions — and record every version.
- Run retrieval alone and score recall and rank.
- Run end to end, each question at least three times. For must-pass questions the worst run counts.
- Score grounding, citation, currency, abstention and access, and apply the rubric.
- Read every failure and classify it with the diagnosis table.
- Fix causes, not questions. Never edit a question or a gold label to make it pass, unless the label was wrong — and record why.
- Record and sign the result against the configuration it tested.
- Re-run after any change to sources, chunking, embeddings, retrieval settings, model or instructions.
Diagnosing failures
| Symptom | Likely cause | Try |
|---|---|---|
| Low recall on paraphrases | Vocabulary mismatch | Hybrid keyword and vector search; query rewriting |
| Low recall on exact terms | Vector search alone | Add keyword search |
| Right document, wrong passage | Chunks split a rule from its conditions | Chunk by structure; keep headings with chunks |
| Good recall, poor faithfulness | The answer adds outside knowledge; too many passages | A cite-or-abstain contract; fewer, reranked passages |
| Citations that do not support the sentence | Citations attached after writing | Cite per sentence, from passage identifiers |
| Superseded answers | Old versions still indexed | Effective dates; retire superseded versions |
| Leaks | Filtering after ranking, or in the prompt | Filter by the asker's identity before retrieval |
| False abstentions | Retrieval misses, or an over-strict instruction | Check retrieval first |
Checklist
- Questions come from real users, in their own words.
- Every category in the mix is represented, with at least twenty questions each.
- Gold passages name the document, section and version.
- A held-out set exists and is never tuned against.
- Retrieval is scored on its own before end-to-end runs.
- The model grader is calibrated against people on a sample.
- Access probes and currency traps run in every evaluation.
- Thresholds are written down before the first run, with reasons.
- Every 0 is read by a person.
- Results are stored with the exact configuration tested.
Worked example
Illustrative example, not a client engagement.
Suppose a company prepares an internal assistant for HR policies. The team collects questions from two years of HR inbox messages, removes personal data, and builds a set of about two hundred, following the starting mix. Access probes ask, as a regular employee, about salary bands that only managers may see.
The first retrieval-only run shows a pattern rather than a number to worry about: paraphrased questions ("can I carry over holiday?") miss the policy titled "annual leave entitlement". Adding keyword search alongside vector search, and the policy's common names as metadata, fixes most of them. End to end, one must-pass question fails in one of three runs: the assistant quotes last year's leave policy. The superseded document was still indexed. It is retired, the full set re-runs clean, and the result is signed with the configuration it tested.
Scorecard template
| Measure | Threshold (set before the run) | Result | Must-pass | Evidence | Owner |
|---|---|---|---|---|---|
| Access leaks | 0 | Yes | Probe log | ||
| Superseded answers | 0 | Yes | Currency-trap results | ||
| Must-pass rubric | No 0 | Yes | Graded answers | ||
| Recall@5 | Retrieval log | ||||
| Faithfulness | Claim checks | ||||
| Citation accuracy | Citation checks | ||||
| Correct abstention | Unanswerable results | ||||
| False abstention | Answerable results | ||||
| Rubric mean | Graded answers |
In the Guild's Evaluation Lab, retrieval accuracy and citation quality are two of the sixteen dimensions on the scorecard Oracle owns.
