Most agents are tested the way demos are built. Someone tries a dozen questions, the answers look sensible, a few people say it is impressive, and the agent goes live. A few weeks later it refunds the wrong order, answers from last year's price list, or keeps a frustrated customer talking when it should have handed over to a person. Nobody tested for any of it, because nobody wrote down what "working" meant.
This is the short version of how the Guild approaches testing before an agent touches real work. It is opinionated on purpose. The full instruments are Evaluation Kit 001, which packages the go/no-go decision, and Field Guide 028, which builds the machinery that runs it again and again. This is the reasoning behind both.
Start from the mission, not the model
You cannot test against "be helpful". Before writing a single test, write one page: what the agent does, for whom, what counts as done, what it must never do, which systems it may touch, and when it must hand over to a person.
Many "model problems" turn out to be mission problems: the agent did something reasonable that nobody had said it must not do. In the Guild's Evaluation Lab, defining the mission is the first stage, and Athena owns it. Every test afterwards is a test against that page. When a test reveals a gap, the page is fixed first.
Write the failures before the successes
Most test sets list what the agent should do. The valuable half is what it should not do — and those are the cases that end projects. Write them first:
- requests that are out of scope and should be declined or redirected;
- situations the mission says must reach a person;
- requests just above a limit, and one large action split into two small ones;
- a customer asking the agent to ignore its instructions;
- a document or email with instructions planted inside it.
Then write the everyday cases, taken from real work — the inbox, the ticket queue, the process notes — with personal data removed. Imagined scenarios test what you expect. Real ones test what happens.
Four things to measure
The Evaluation Lab's scorecard has sixteen dimensions, but four questions decide most go/no-go calls.
Reliability: run it more than once
Agents are not deterministic. The same input can produce a good answer on Monday and a poor one on Tuesday. Run every scenario several times, and run paraphrases of it: incomplete, misspelled, rude, in another language if your users write in one, buried in a long conversation. Report the worst run, not the best. A scenario that passes nine runs in ten fails one customer in ten.
Tool use: test the hands, not just the voice
Tools are where an agent changes things, so test them in a sandbox that behaves like production. Does the agent choose the right tool and leave the others alone? Are the parameters right? When the tool errors or times out, does it say so instead of inventing a result? Does a retry create a second invoice? When a permission is refused, does it stop?
Read the tool-call log, not just the final message. An agent can write a perfect reply on top of a failed call.
Escalation: score the hand-over
Escalation is a feature, not a failure, and it deserves its own score. Test both directions. Do the cases that must reach a person arrive — at the right person, with enough context to decide in a minute? And do the cases that need no person stay with the agent? An agent that escalates everything is a slower inbox. Of the two errors, missed escalations are the dangerous ones; weight them accordingly.
Business outcome: compare with today, not with perfection
An agent does not have to be flawless. It has to be better, safer or cheaper than the way the work is done now, at the level of autonomy you are about to grant. Measure the current process first — time to completion, rework, errors, cost per completed task — and compare the agent with that. Without a baseline, every result is an opinion.
Some tests are must-pass
Averages hide the failures that matter. An agent with an excellent overall score may still have failed the one scenario in which it sent a customer's details to someone else.
Split the scorecard in two. Must-pass tests — policy, permissions, privacy, approvals — allow no failures: one is a no-go. Graded tests — task success, knowledge, tone — need a threshold, set per mission, written down with its reason, before the first run. Never lower a threshold after seeing the results. If it was wrong, change it deliberately, record why, and run everything again.
Read the failures; do not just count them
Scores tell you where to look. Transcripts tell you what happened. A person should read every failed scenario and name the cause: the right passage was never retrieved, the tool description was ambiguous, the mission did not cover the case, the threshold was unrealistic. Fix the cause and keep the scenario. It becomes part of the regression suite for the life of the agent.
Merlin note — The question is never "is this agent good?" It is "is this agent good enough at this mission, at this level of autonomy, with these limits?" Change any one of the four and you are testing a different agent.
A person signs, and the testing does not stop
Automated evaluators can score every run; they should not make the decision. In the Guild, Oracle owns the scorecard, Sentinel owns the boundaries, and a Human Wizard reviews the evidence and signs the go/no-go. Kit 001 adds a third outcome worth having — go with conditions: a narrower scope or a lower level of autonomy until named conditions are met. The decision, the evidence and the name sit together, so the question "why did we trust this agent?" always has an answer.
Then the agent starts small — shadowing the current process, or preparing work for a person to approve — and earns more only on evidence. Re-run the full suite whenever the model, the instructions, a tool or a knowledge source changes, and turn every real failure in production into a new scenario. Readiness is not a certificate. It is a habit.
Where to go next
- Evaluation Kit 001 — the mission sheet, scorecard, thresholds, review checklist and go/no-go for one decision.
- Field Guide 028 — the scenario library, harness, evaluators and regression runs behind it.
- Five measurements that tell you whether an agent is helping — Oracle on what to watch once the agent is live.
- The Evaluation Lab — how the Guild evaluates an agent you are building, or one you already run.
