Production safety checklist

How to use it

• Run it per agent and per environment, before go-live and before every step up in autonomy.
• Attach evidence to every item — a link, a log, a test result, a document. A tick without evidence does not count.
• Any open Must item blocks go-live. Every open Should item gets a named owner and a date.
• The person responsible for security prepares it; the person accountable for the agent signs it. In the Guild, Sentinel owns the boundaries and a Human Wizard owns the decision to deploy.

Identity and access

• Must — The agent has its own identity, separate from every person and every other agent.
• Must — Its credentials are isolated, held in a secrets store, and never appear in prompts, code or logs.
• Must — Every permission is the least the agent's job needs; nothing is standing that could be granted per task.
• Must — The agent cannot change its own permissions, instructions or tools in production.
• Must — Access can be revoked in one action, and revocation has been tested.
• Should — Permissions are reviewed on a schedule and after every change of scope.

Approved knowledge

• Must — The agent answers only from approved sources, each with a named owner.
• Must — Retrieval enforces the requester's permissions before anything reaches the model.
• Must — Retrieved text is treated as data; tests with planted instructions have passed.
• Should — Every source has a freshness rule and a review date.
• Should — Deletions and retirements reach indexes and caches.

Controlled tools

• Must — Tools are allowlisted; every other system is unreachable.
• Must — Limits on amounts, records and rates are enforced in the tool or the system, not only in instructions.
• Must — Irreversible, financial, identity and access actions need approval, or are not available at all.
• Must — Write tools are idempotent; a retry cannot duplicate an effect.
• Must — Tools have been tested against failing test versions (see the Tool-use evaluation kit).
• Should — Each task has a call budget; loops stop and escalate.

Evaluation

• Must — The readiness evaluation is complete and signed (see Evaluation Kit 001).
• Must — Every must-pass scenario, including adversarial and injection cases, passes in every run.
• Must — Results are stored with the exact configuration tested.
• Must — The regression suite runs before every change (see the Agent regression test framework).
• Should — Scheduled runs detect drift when nothing has changed.

Human escalation

• Must — A written escalation policy names triggers, routes and response times, and the people who receive escalations have agreed to it.
• Must — The escalation benchmark shows no missed escalations on critical triggers (see the Human escalation benchmark).
• Must — A person is available on request, in every channel.
• Should — Every approver has a backup, and out-of-hours cover is defined.

Audit and improvement

• Must — Every action is recorded: who or what acted, the inputs, the evidence, any approval and the result.
• Must — The agent cannot edit or delete that record.
• Must — Cost and usage are visible, with an alert on unusual spend.
• Should — Every production failure becomes a regression scenario.

Incident response

• Must — A stop control pauses the agent, or drops it to a lower autonomy level, in one action. The on-call person knows where it is, and it has been tested.
• Must — A runbook covers detect, contain, explain and improve, and names who is called.
• Must — Alerts fire on policy events, error spikes, cost spikes and a sudden fall in escalations.
• Should — The team has rehearsed an incident before go-live.

Data protection and transparency

• Must — Data flows are mapped: what personal data the agent sees, where it goes, and which vendors — model providers and subprocessors — process it.
• Must — No secrets or payment data reach prompts, memory or transcripts; logs are masked.
• Must — People are told they are dealing with an AI agent. The EU AI Act requires this for systems that interact directly with people, unless it is obvious; elsewhere it is good practice.
• Should — Retention periods for transcripts and memory are set and enforced.
• Should — The model and vendor register is current.

Mapping to public frameworks

For teams that already report against public frameworks, an indicative mapping. It is a reading aid, not a compliance claim.

The Guidelines for secure AI system development, published by the UK's NCSC and the US CISA with partner agencies, follow a similar lifecycle — secure design, development, deployment, and operation and maintenance — and are a useful companion for identity and access, controlled tools and incident response.

Sign-off

When to run it again

Re-run the full checklist before every step up the trust model — from Assist to Operate above all, because that is the step at which the agent first acts without a person approving each action. Re-run identity and access and controlled tools whenever a tool or permission changes, evaluation with every release, and data protection whenever a new vendor or data source is added. A step down in autonomy needs no checklist: it should take one action, and nobody should hesitate to take it.

Example

Illustrative example, not a client engagement.

Suppose a team prepares an invoice-processing agent for go-live. The checklist turns up two open Must items. The agent's credentials are shared with an integration account that people also use, so its actions cannot be told apart from theirs in the record. And the stop control exists in the code, but nobody on call knows how to use it. Both are fixed — a separate identity, and a rehearsal in which the on-call person pauses the agent in under a minute — before the accountable owner signs. Two Should items, permission reviews and transcript retention, go into the plan with owners and dates.