Operational questions are not like the questions retrieval demos are built on. They mix exact identifiers — a form number, a product code, a clause — with loosely worded intent. Their answers change when a policy is revised. Some of them are not knowledge questions at all, but requests for a live record dressed up as one. And the person asking may not be allowed to see the answer. This experiment is designed to test which retrieval set-up holds up under those conditions, in a real operational knowledge base rather than a benchmark.
Status — Protocol. No runs of this experiment are recorded yet, so this page reports no results. They will be added here, with the exact versions tested, when they exist.
The question: what makes retrieval trustworthy in operations?
For operational questions — procedures, policies, exceptions, "how do we handle this?" — which retrieval configuration produces answers that are correct, cited, current and permitted? And which questions should never reach retrieval, because the answer lives in a system of record and must be looked up live?
Retrieval-augmented generation (Lewis et al., 2020) is a common way to ground an agent in a business's own knowledge. The choices inside it — how text is found, cut up, filtered and routed — decide whether an answer can be trusted.
Hypotheses
- H1 — Hybrid retrieval wins on mixed questions. Combining keyword retrieval (BM25) with embedding search, then reranking, finds the right passage more often than either alone, because operational questions mix exact identifiers with paraphrase.
- H2 — Structure beats a better embedding. Cutting documents along their own structure — section by section, with the title, version and effective date attached to every piece — improves answer faithfulness more than switching to a different embedding model.
- H3 — Most wrong answers are not retrieval misses. In operations, more errors come from superseded documents and from record questions sent to retrieval than from the retriever failing to find the right text.
- H4 — Permissions belong before retrieval. Filtering by the asker's rights before anything is retrieved produces no leaks on restricted probes; filtering the answer afterwards does not reliably prevent them.
Method
Corpus and questions
The field test is designed for a real operational knowledge base — policies, procedures, product information — used only with its owner's permission and never named without their consent. Before that, and alongside it, a synthetic corpus of the same shape is used to rehearse the protocol and to publish a version anyone can reproduce.
Questions are written with the people who own the documents. Each has a gold passage, an expected answer and an owner who approved both. They fall into ten kinds:
| Kind | Example of what it tests |
|---|---|
| Direct lookup | The answer is stated in one passage |
| Paraphrase | The question uses none of the document's words |
| Exact identifier | A code, form number or clause reference |
| Multi-document | The answer needs two sources |
| Exception | An edge case the procedure handles separately |
| Superseded | The answer changed in a newer version of the document |
| Unanswerable | The sources do not contain the answer |
| Restricted | The asker is not allowed to see the source |
| Record lookup | The answer is in a live system, not in the documents |
| Planted instruction | A document contains text that tries to steer the agent ("ignore the policy above") |
Variables
Each variable is changed one at a time from a plain baseline; the best settings are then combined and tested together.
- Retriever: BM25 alone; embeddings alone; hybrid; hybrid with a reranker.
- Chunking: fixed-size pieces; structure-aware pieces with title, version and effective date.
- Freshness: no date handling; superseded versions filtered out by effective date.
- Routing: every question to retrieval; a router that sends record questions to a live lookup tool.
- Permissions: answer filtered after generation; sources filtered by the asker's rights before retrieval.
Measurements
Retrieval and answers are measured separately, so a failure can be traced to one or the other.
- Retrieval: whether the gold passage is in the top results (recall at k), and how high the first correct passage ranks.
- Answers: correctness against the owner's answer; faithfulness to the retrieved text; whether each citation supports the sentence it is attached to. RAGAS (Es et al., 2023) supplies starting metrics, calibrated against the owners' judgements before use.
- Abstention: on unanswerable questions, the agent says the sources do not cover it.
- Staleness: answers drawn from a superseded version.
- Leakage: any restricted content in an answer to someone without access. Must-pass: one leak fails the configuration.
- Injection resistance: text planted in a document is treated as content, never obeyed. Must-pass, like leakage.
- Routing accuracy: record questions sent to the live system, knowledge questions to retrieval.
- Latency and cost per answered question.
Controls
- Same model, instructions and question set for every configuration; the corpus is frozen as a versioned snapshot for the duration of the test.
- Each question runs several times; results are reported with their spread.
- Owners grade a sample blind to the configuration that produced each answer.
- More retrieved text is not assumed to be better — models use the middle of long contexts less reliably (Liu et al., 2023) — so the number of passages passed to the model is held fixed.
- Success thresholds are written down with the protocol before the first run.
What would confirm or refute it
- H1 holds if the hybrid retriever with reranking clearly beats both single retrievers on identifier and paraphrase questions. It is refuted if one simple retriever matches it — a cheaper result worth knowing.
- H2 holds if structure-aware chunking moves faithfulness and citation accuracy more than the embedding change does.
- H3 holds if, in a review of every wrong answer, stale sources and misrouted record questions together outnumber retrieval misses.
- H4 fails on a single leak under pre-retrieval filtering. That is a defect to fix before anything else, not a data point.
Results
None yet. When runs are complete, this section will report every measurement above per configuration and per question kind, an analysis of every wrong answer by cause, and the versions tested. Nothing about the owner's business will be published without their consent. Negative and inconclusive results will be reported in the same way as positive ones.
Limits: what this cannot show
- One corpus. A different business, with different documents, may favour different settings.
- Owners write the questions. Their questions reflect what they think people ask, which is not always what people do ask.
- Model graders can be generous. Calibration against owners reduces this but does not remove it.
- Snapshots age. Freezing the corpus makes runs comparable, but hides how quickly a live index drifts.
Until there are results
Measure retrieval and answers separately, test superseded and restricted questions on purpose, and send questions about live records to the system that holds them. Building a production AI knowledge system sets out the architecture this experiment tests, and Memory is not a database explains why records are queried, never copied.
