Agent Production Readiness Kit.

What the kit contains

Write the mission sheet

You cannot test an agent against a vague goal. Before any scenario is written, fill in one page:

• Mission — what the agent does, for whom, and the outcome that counts as done.
• Never — what it must never do, even if asked. Be concrete: "never issue a refund", "never email a customer without approval".
• Tools and data — every system it may read or change, and with which permissions.
• Escalation — the situations in which it must hand over to a person, and to whom.
• Starting autonomy — observe, assist, operate within limits, or optimise.
• Owner — the person accountable for the agent in production (in a Guild engagement, a named Human Wizard).

In the Guild's Evaluation Lab, defining the mission is the first of five stages, and Athena owns it. When a later test reveals a gap, fix the sheet first, then the agent.

Build the scenarios

Good scenarios come from real work, not imagination. Collect examples from the inbox, the ticket queue or the process documentation, remove personal data, and write down what a good outcome looks like for each. Cover six kinds:

1. Typical cases — the everyday requests that make up most of the volume.
2. Variations — the same request phrased differently, incomplete, or in another language your users write in.
3. Edge cases — unusual but legitimate requests near the limits of the mission.
4. Out-of-scope requests — things the agent should decline or redirect.
5. Escalation cases — situations the mission sheet says must reach a person.
6. Adversarial cases — attempts to make the agent break a "never" rule, including instructions hidden in documents or messages.

Every scenario is recorded the same way, so that anyone can rerun it and agree on the result.

Keep the scenarios in version control. They become your regression suite.

Test every tool

Tools are where an agent changes things, so test each one it can call, in an environment that behaves like production but is not production:

• Selection — the agent chooses this tool when it should, and not when it should not.
• Parameters — the values passed are correct and complete.
• Failure — when the tool errors or times out, the agent reports it and does not invent a result.
• Repetition — retrying does not create duplicates: a second invoice, a second email.
• Denial — when a permission is refused, the agent stops and escalates instead of looking for a way round.

Read the tool-call log, not only the final message. A polished reply can sit on top of a failed call.

Set the thresholds

Thresholds are set per mission, before the first run, and written down with the reason for each.

• Separate must-pass from graded. Policy and approval tests are must-pass: one failure means no-go. Task success and knowledge are graded: you decide the level that is acceptable.
• Set the level by the cost of an error. An agent drafting internal summaries can tolerate more misses than one updating customer records.
• Compare with the current process. If people complete the task correctly most of the time, measure the agent against that, not against perfection.
• Never lower a threshold after seeing the results. If it was wrong, change it deliberately, record why, and run everything again.

Run and score

Run every scenario more than once — agents do not give the same answer every time — and score each run against the scorecard.

These six are the readiness view of the sixteen dimensions the Evaluation Lab measures across quality, tools and safety, operations and business outcome. Add latency, cost per mission, exception rate or human intervention to the scorecard when the mission makes them matter.

When a scenario fails on some runs and passes on others, record the worst run. A scenario that passes nine runs in ten fails one customer in ten.

The human review

Automated evaluators score the runs; a person decides. Before signing, the reviewing Human Wizard checks:

• The mission sheet is complete, and the owner has accepted it.
• Every must-pass test passed, and every failure in the graded tests has been read, not just counted.
• The escalation cases reached the right person with enough context to decide.
• No adversarial case produced an action outside the boundaries.
• Tool failures were reported honestly, never papered over.
• Cost and latency per mission are acceptable for the business case.
• Monitoring and a way back to the previous stage are in place before go-live.

The go/no-go decision

The review ends in one of three signed outcomes:

• Go — staged rollout at the proposed autonomy level.
• Go with conditions — rollout with a narrower scope or a lower autonomy level (for example, the agent prepares and a person approves) until named conditions are met.
• No-go — with the failures listed and the fixes required before the next evaluation.

Keep the decision, the evidence and the name of the person who signed it together. When someone later asks why the agent was trusted with this work, the answer exists.

Deploy in stages

Move in stages, and widen only on evidence:

1. Shadow — the agent works alongside the current process; its output is compared, not used.
2. Assist — the agent prepares the work; a person approves each action.
3. Operate within limits — the agent acts alone inside narrow policies, with exceptions routed to people.
4. Expand — scope or limits grow through measured, reviewed changes.

At every stage, keep a quick way to return the agent to the stage before.

Monitor and re-test

Readiness is not a one-time certificate. In production, keep measuring task success, escalations, exceptions, human interventions, policy events, latency and cost per mission, and compare them with the evaluation results.

Run the full scenario suite again whenever the model, the instructions, a tool or a knowledge source changes, and turn every real failure in production into a new scenario. An agent that passed last quarter has not necessarily passed this one.

Scorecard template

Copy this table for each evaluation. Fill in the thresholds before the first run.

For the reasoning behind the kit, read How to test an agent before production. To build the scenario library and harness that run it again and again, read Field Guide 028. If you would rather the Guild ran the evaluation, the Evaluation Lab applies the kit to your agent and hands back the scorecard, the evidence and the go/no-go.