The Guild designs its operational missions around several agents, not one. Athena receives and routes the work, specialists act in the client's systems, Sentinel holds the approvals and boundaries, and Oracle measures what happened. The Evaluation Lab scores agents against sixteen dimensions before they are trusted with real work. This experiment asks whether scoring agents one at a time is enough when the work passes between them.
Status — Protocol. No runs of this experiment are recorded yet, so this page reports no results. They will be added here, with the exact versions tested, when they exist.
The question: is a good agent enough?
Does a scorecard that rates each agent in isolation predict how a multi-agent mission performs from start to finish? If it does not, what is the smallest addition to the evaluation that closes the gap?
The question matters because evaluating the whole mission end to end is slow and expensive, and when it fails it often cannot say where. Evaluating each agent is cheaper and points straight at the agent at fault. The trade-off is only worth making if per-agent results can be trusted for the mission as a whole.
Public research gives a reason for doubt. A taxonomy of failures in multi-agent language-model systems (Cemri et al., 2025) sorts them into three groups: problems in how the system and its roles are specified, misalignment between agents, and weak verification of the work. The second and third groups live between agents. A test that looks at one agent at a time cannot see them by construction.
Hypotheses
- H1 — Hand-off blindness. Per-agent evaluation misses most failures that only occur at hand-offs: context lost in transit, work dropped or done twice, a decision that falls between two agents' boundaries, an escalation each agent assumes the other made.
- H2 — Contracts close most of the gap. Writing down a contract for each hand-off — what must pass, in what form, and who owns the next step — and checking it mechanically detects most hand-off failures, at a fraction of the cost of full end-to-end simulation.
- H3 — End-to-end scoring finds but does not locate. Scoring only the final state of a mission detects that it failed, but more often leaves a person to trace which agent or hand-off caused it.
Method
The reference mission
Illustrative mission, not a client engagement. Supplier invoices arrive by email. An intake agent classifies each one and extracts the fields. An action agent prepares a match against the purchase order in a finance sandbox. A boundary rule sends anything above an approval threshold, or with a mismatch it cannot explain, to a person. A monitoring agent reports exceptions daily.
Every tool is a sandbox or a simulator that can return realistic errors, timeouts and refusals. The scenario library covers typical cases, variations, edge cases, out-of-scope requests, escalation cases and adversarial inputs. Each scenario states the expected end state of the system of record, not only the expected messages.
Seeded defects
To measure whether an evaluation catches failures, plant known failures and count how many it catches. Software testing has done this for decades as mutation testing (DeMillo, Lipton and Sayward, 1978). Each defect below is injected on its own into an otherwise identical build:
| Class | Seeded defect | Where it lives |
|---|---|---|
| Context loss | A field (currency, supplier reference) is dropped at hand-off | Between agents |
| Ambiguous ownership | Two agents may both act on the same item, or neither is told to | Between agents |
| Duplicate action | A retry without an idempotency key applies the same update twice | Between agents |
| Escalation gap | Each agent assumes the other escalated | Between agents |
| Contract drift | One agent's output format changes in a new version; the next agent misreads it silently | Between agents |
| Single-agent fault (control) | A wrong tool parameter inside one agent | Inside one agent |
The control class should be caught by every design. If it is not, the harness is broken and the run is discarded.
Three evaluation designs
| Design | What it scores |
|---|---|
| A — Per-agent | Each agent against its own scorecard, with recorded or mocked inputs from its neighbours |
| B — End-to-end | The mission's final state in the sandbox against the expected state |
| C — Per-agent plus contracts | Design A, plus rule checks on every message between agents: required fields, types, owner of the next step, idempotency key, escalation flag |
Measurements
- Detection rate per defect class and design: the share of seeded defects the evaluation flags.
- Localisation: when a defect is flagged, whether the evaluation names the agent or hand-off at fault.
- False alarms on clean builds.
- Agreement with people: on a sample of runs, how often the design's verdict matches two human reviewers, reported as Cohen's kappa.
- Reliability: every scenario runs several times. Results are reported as pass^k — the chance that all k runs pass, the measure τ-bench (Yao et al., 2024) introduced — rather than a single pass rate.
- Cost and time to run each design's full suite.
Controls
- Model versions, instructions, tool definitions, knowledge snapshot and scenario set are fixed and recorded with every result, so runs can be compared.
- Whatever can be decided from the logs is decided by a rule, not a model. Model-graded rubrics are kept for what rules cannot see, written as yes/no questions and calibrated against people first, because model graders have known biases (Zheng et al., 2023).
- The human reviewers do not know which defect, if any, was seeded.
- Success thresholds for each hypothesis are written down with the protocol before the first run, so the results cannot be read to fit.
What would confirm or refute it
- H1 holds if design A's detection rate on the five hand-off classes is clearly below its rate on the control class. It is refuted if A catches hand-off defects about as well as single-agent ones — in which case per-agent scorecards are enough, and this page will say so.
- H2 holds if design C detects hand-off defects at close to design B's rate and localises them better, for materially less run cost. It is refuted if contracts miss whole classes — escalation gaps are the likeliest — that only an end-to-end state check reveals.
- H3 holds if failures flagged by design B need manual tracing more often than those flagged by C.
Results
None yet. When runs are complete, this section will report detection, localisation and false-alarm rates per defect class and design, pass^k per scenario, reviewer agreement, run cost, and the version of everything tested. Negative and inconclusive results will be reported in the same way as positive ones.
Limits: what this cannot show
- We only seed the defects we thought of. Real failures may be stranger. Defect classes found later in production should be added to the set.
- One reference mission. Results may not transfer to missions with more agents, longer chains or human steps in the middle.
- Simulators are kinder than real systems. Sandboxes rarely reproduce the full range of real errors and delays.
- Graders can share blind spots with the agents they grade, especially when both use the same model family.
- Small samples are noisy. Rates from a few runs per scenario need intervals, not single numbers.
Until there are results
The practical step does not need to wait. When a mission uses more than one agent, write down each hand-off as a contract and test it the way you would test a tool call: required fields present, formats as expected, one named owner for the next step, and escalations recorded rather than assumed. The Agent Production Readiness Kit already covers each agent; the evaluation framework guide describes the harness, the scorecard and the gates this experiment builds on.
