Agent regression test framework

What the framework contains

What counts as a change

In machine-learning systems, changing anything changes everything: a prompt edit for one case can break another, and published studies have documented behaviour shifting between versions of models served under the same name. So every change goes through the same gate, however small — and the unchanged agent is re-tested on a schedule, because providers and data move underneath it.

Record a configuration fingerprint with every run: the version of each item in the table. A result without one cannot be compared with anything.

Suite tiers

No change reaches production without the release suite — including the "quick" instruction edit made directly in a console.

The golden set

Scenarios come from the original evaluation set; every production failure, which becomes at least one scenario before the incident is closed; near misses that people caught; confirmed missed escalations; and adversarial cases from security reviews.

Each scenario records an identifier, an owner, its origin, the date added, a must-pass flag, tags (dimension, request type, risk), the expected behaviour and the pass condition.

Keep it honest:

• Never edit a scenario to make it pass. If the scenario is wrong, fix it deliberately, with a reason, and re-run the baseline.
• Retire with a reason, when policy or product changes — never silently.
• Keep known failures visible. Mark them; do not delete them.
• Hold out a set that is never used to tune instructions.

Non-determinism

Agents do not give the same output twice, so a single run proves little.

• Must-pass scenarios use pass^k: the scenario passes only if every one of k runs passes.
• Measure the noise floor. Run the production baseline twice. The difference between two runs of the same version is noise; a candidate's drop smaller than that is not evidence of anything.
• Compare scenario by scenario. Use the same scenarios for both versions, and look at what changed: newly failing (passed before, fails now), newly passing, still failing. Newly failing scenarios matter more than the net rate.
• Mind the sample size. On two hundred scenarios, a rate of 90% carries a standard error of about two points, so a two-point drop can be noise and an eight-point drop rarely is.
• Quarantine flaky scenarios with an owner and a date. A flaky must-pass scenario still blocks.

Comparison rules

Run the baseline — the production configuration — on the same suite, the same day, with the same number of repeats. Report by dimension and by category, not only overall: an overall gain can hide a loss in one category. Then apply four rules:

1. Any newly failing must-pass scenario blocks the release.
2. A graded drop beyond tolerance blocks, unless a named person signs the trade-off.
3. An improvement does not count if a guardrail — cost, latency, escalation rate — worsens beyond tolerance.
4. A person reads every newly failing scenario.

Starting thresholds

Warning — These are recommended starting points, not measured results or industry standards. Set tolerances per mission and per suite size, before the run, and never widen one after seeing the result.

The gate

Every release suite ends in one of three signed outcomes. Pass — the change proceeds to staged rollout. Pass with sign-off — a named person accepts a documented trade-off. Block — the failures are listed with the fixes required.

A pass is not the end. Release in stages: a canary on a small share of traffic, live measures compared with the baseline, widening only on evidence, and a rollback that is one action.

Running it

1. Record the fingerprint of the production configuration and the candidate.
2. Run the smoke suite on every change; fix before review.
3. Run baseline and candidate on the release suite, with the same repeats.
4. Compare with the rules above, by dimension and category.
5. Read every newly failing scenario, and decide.
6. Sign the gate, and store the report with both fingerprints.
7. Canary, then widen.
8. Add new production failures to the golden set.
9. Run the scheduled suite even when nothing has changed.

Checklist

• Every kind of change in the register triggers the release suite.
• Every run stores a configuration fingerprint.
• Smoke, release and scheduled tiers exist and run automatically.
• Every production incident adds at least one scenario.
• Must-pass scenarios use pass^k.
• The noise floor is measured before tolerances are applied.
• Results are compared by dimension and category, scenario by scenario.
• Every newly failing scenario is read by a person.
• Guardrails on cost, latency and escalation are part of the gate.
• Rollback is tested and takes one action.

Worked example

Illustrative example, not a client engagement.

Suppose a support agent's model is to be upgraded. The release suite shows the overall rubric score up, and cost per mission down. By category, refunds have dropped: the new model is more willing to grant exceptions. One must-pass scenario — a refund above the limit — fails in one of five runs. The gate blocks.

The team enforces the refund limit in the tool rather than relying on the instructions, re-runs baseline and candidate, and finds no newly failing scenarios and every guardrail within tolerance. The gate passes; the canary starts with a small share of conversations, and the refund scenario stays in the smoke suite for good.

Regression report template

In the Guild, Oracle owns regression performance on the scorecard, and regression tests are part of the deployment stage Sentinel owns in the Evaluation Lab.