An agent that has not been evaluated has not been trusted; it has been hoped for. Evaluation is how the Guild turns "it seems to work" into evidence a person can sign: what the agent was tested on, how it scored, against which thresholds, and who decided it was ready. Trust isn't assumed. It is measured — before launch, and every time something changes.
The methods here cover the whole life of an agent: testing before production, the framework that tests it again after every change, specialist kits for knowledge, tools, escalation and safety, and the measurements that show whether it is helping once it is live.
Questions this topic answers
- What should we test before an agent goes live, and what counts as passing?
- Which failures are never acceptable, however good the average?
- How do we know a change of model, prompt, tool or knowledge has not made it worse?
- Does the agent find and cite the right knowledge, use its tools correctly, and hand over when it should?
- Once it is live, how do we know it is actually helping?
Start here
- How to test an agent before production — the short entry point: reliability, tool use, escalation quality and business outcome, with the failures written down first.
- Agent Production Readiness Kit — Evaluation Kit 001: mission sheet, scenarios, scorecard, thresholds and a signed go, go with conditions, or no-go.
- Evaluation Framework for AI Agents — Field Guide 028: the scenario library, harness, evaluators and gates that keep deciding every time the agent changes.
- Agent regression test framework — proving that a change has not made the agent worse.
- Five measurements that tell you whether an agent is helping — Oracle on outcome, intervention, rework, time and cost once the agent is live.
The specialist kits go deeper on one capability each: the Knowledge retrieval evaluation framework, the Tool-use evaluation kit, the Human escalation benchmark and the Production safety checklist. The Forge's research protocols, such as the Multi-agent evaluation framework, test the methods themselves.
Key ideas
- The mission sheet — every test traces back to what the agent should do and must never do.
- Scenarios from real work — including the cases that should make the agent stop or hand over.
- Must-pass and graded — policy, permission and approval tests allow no failures; the rest have thresholds set before the run.
- Sixteen dimensions, one scorecard — quality, tools and safety, operations, and business outcome.
- Automated scoring, human decision — evaluators score every run; a Human Wizard reviews the exceptions and signs the decision.
- Regression — any change of model, instructions, tools or knowledge means testing again.
- Baselines — an agent is compared with the process it replaces, not with perfection.
If you only do three things
- Write the failure cases first. Out-of-scope, escalation and adversarial scenarios are the ones that end projects.
- Score must-pass tests separately. An excellent average can hide the one failure that matters.
- Re-run on every change. A new model, prompt, tool or knowledge source is a new agent until the suite says otherwise.
Everything else in this topic makes those three faster, cheaper and more convincing.
Who knows this topic
In the Guild's Evaluation Lab, every layer of trust has an owner:
- Oracle — scoring, performance measurement, accuracy monitoring and business outcomes. Owns the scorecard.
- Merlin — model selection, agent architecture, prompt and memory evaluation. Owns the experiments.
- Sentinel — security, permissions, policy adherence and escalation rules. Owns the boundaries.
- Athena — mission definition, intake quality and human approval points. Owns the mission.
A Human Wizard owns the decision to deploy.
Where this knowledge is put to work
The Evaluation Lab explains the five stages from mission to deployment; the evaluation architecture shows the nine layers and thirteen gates in full. Bring an agent you are building, or one you already run, and the Lab will show how it scores and what it needs before it deserves more autonomy. AI Agent Development counts evaluation, audit and support as part of what makes an agent production-ready, and managed digital operations keeps monitoring agent performance and drift after launch.
