A field guide for teams putting an AI agent in front of customers. Reliable, here, has a precise meaning: the agent does the same right thing on the thousandth conversation as on the first — whatever the phrasing, the pressure or the time of day, and after every change to its model, knowledge or tools. It follows the Guild's twelve-chapter format.
The problem
Support is where an agent is most visible and least forgiven. Four failures recur:
- The polite wall. The customer cannot reach a person, and the agent keeps answering a question they did not ask.
- The confident wrong answer. A returns policy that does not exist, stated warmly.
- The overreach. A refund, an address change or an account update made without the authority or the identity check it needed.
- The lost thread. A handover with no context, so the customer explains everything again.
Each is a design failure, not a model failure. And measured on deflection alone, an agent can look successful while committing all four.
Concept overview
A reliable support agent keeps five promises:
- It answers from approved knowledge and cites it, or says it does not know.
- It reads records live and acts only through bounded tools, within limits enforced outside the model.
- It knows its catalogue: which requests it handles and which it hands over.
- It hands over with a brief, at the right moment, to the right person.
- It is measured on verified resolution, with guardrails.
It also says that it is an AI. In the EU, the AI Act requires systems that interact with people to be designed so that people know they are dealing with an AI, unless that is obvious; everywhere else it is simply honest.
Support work comes in three tiers. Inform — answer a question. Act — change something: reschedule, update an address, open a return. Decide — exceptions, complaints, goodwill, disputes. Agents start with Inform, earn Act within limits, and leave Decide with people.
When to use this
Good signs: a large share of contacts are repeat request types; the answers are written down, or can be; policies are clear; the systems involved have APIs; and a team exists to take the handovers. Poor signs: most contacts are complaints or negotiations; policy lives in people's heads; there is no ticketing system to hand over into; or the advice is regulated and needs a qualified person.
Architecture
| Stage | What happens | Guild role |
|---|---|---|
| 1. Intake | Channel, language, identity; the agent says it is an AI | Hermes — voice and communication |
| 2. Understand | Intent, request type, risk flags (complaint, vulnerability, legal, money) | Athena — intent and coordination |
| 3. Ground | Approved knowledge with citations; the customer's records through read tools | Atlas — customer and system context |
| 4. Act | Bounded tools; approval for anything above a limit | Sentinel — consent and boundaries |
| 5. Respond | The answer, what changed, and what happens next | — |
| 6. Hand over or close | A brief to a person, or the outcome recorded | — |
| 7. Learn | Unanswered questions and corrections sent to knowledge owners | — |
In the Guild's voice and customer experience work, conversation designers, integration specialists and Human Wizards define the tone, the policy, the knowledge and the moment a person takes over. The agents carry it out.
Core components
The request catalogue
The catalogue is the agent's job description, one line per request type.
| Request | Tier | The agent may | Hand over when |
|---|---|---|---|
| Order status | Inform | Read the order and carrier status | The parcel is past the carrier's window |
| Address change | Act | Change it before dispatch, after an identity check | Already dispatched; the check fails |
| Return | Act | Open a return within policy and send the label | Outside policy, damaged or high value |
| Refund | Act | Refund within a set limit, where policy allows | Above the limit; any dispute |
| Complaint | Decide | Acknowledge and gather the facts | Always |
Anything not in the catalogue is out of scope. The agent says so and offers a person.
Knowledge
Answer from approved, current sources and cite them, or abstain and route the question to its owner. Policy is quoted, not paraphrased into promises. Field Guide 017 covers the knowledge system behind this.
Tools and permissions
Read tools broadly, write tools narrowly. Identity verification is a tool step with a result, not the agent's impression of the conversation. Limits — refund amounts, which fields may change — are enforced in the tool, not in the prompt. Write tools are idempotent, so a retry never refunds twice.
The handover brief
The person taking over should never have to ask the customer to repeat. Every brief carries: who the customer is and whether identity was verified; what they want, in their words; what the agent checked and tried, with links; why it is handing over; what the person needs to decide; urgency; and what the customer was told about what happens next.
Conversation design
One clarifying question at a time. Confirm before acting. Say no with a reason and an alternative. Offer a person when frustration shows, and always when asked. Never argue.
Measurement
Optimise verified resolution — resolved, correct and stayed resolved — with policy breaches, missed escalations, customer effort and cost per verified resolution as guardrails. Oracle sets out the case in The KPI your support agent should actually optimise.
Implementation
- Read the real conversations. Group a few months of contacts by request type; note the volume, the risk and how people resolve each.
- Pick the first types: frequent, documented, low-risk, resolvable with the systems available.
- Write the catalogue and the "never" list with the support lead.
- Fix the knowledge for those types first. Every gap becomes a wrong answer later. Capture new answers as part of resolving tickets, not as a separate project.
- Build read tools, then write tools with limits and identity checks.
- Design the handover with the people who receive it.
- Evaluate before any customer sees the agent.
- Roll out in stages: the agent drafts and a person sends; then a person approves each action; then the agent acts within limits. One request type at a time.
Failure modes
| Failure | What you see | What fixes it |
|---|---|---|
| The polite wall | Customers ask for a person twice, or give up | A person on request, always; frustration triggers |
| Invented policy | A plausible rule nobody wrote | Cite or abstain; faithfulness checks |
| Pressure compliance | The agent bends after the third insistence | Limits in tools; pressure scenarios in every run |
| Identity bypass | An account changed for a convincing stranger | Verification as a tool step that must pass |
| The lost thread | Customers repeat themselves after handover | A required brief; scored by the people who receive it |
| Silent tool failure | "Done" in the reply, an error in the log | Replies built from tool results; failure tests |
| Metric gaming | Deflection rises; repeat contacts rise with it | Verified resolution with guardrails |
Evaluation
Build scenarios per request type from real conversations: typical cases; variations (vague, rude, misspelt, other languages your customers use); two requests in one message; out-of-scope requests; every escalation trigger; and adversarial cases — pressure, social engineering, instructions hidden in an attachment.
Run each scenario several times. A public customer-service benchmark for agents, τ-bench, reports reliability as pass^k — the share of tasks that succeed in all of k attempts — because an agent that succeeds once may not succeed every time. Set must-pass scenarios that allow no failure in any run: no account action without a passed identity check, no refund above the limit, every request for a person honoured, nothing about another customer disclosed.
Evaluation Kit 001 gives the readiness scorecard; the Human escalation benchmark tests the handover in depth.
Security and governance
- Identity before account. No change, and no account detail disclosed, until verification passes.
- Customer text is data, not instructions. Messages and attachments can carry injected instructions; the agent must not follow them.
- Minimise what is kept. Mask payment details in transcripts; retain conversations only as long as policy says.
- No commitments outside policy. Goodwill and exceptions are Decide work, for people.
- Least agency. Only the tools the catalogue needs, each with the narrowest permission.
- Audit. Every action is recorded with the conversation and evidence that led to it.
Cost considerations
Count cost per verified resolution, not per conversation. The direct costs are model tokens across turns, retrieval, tool calls, and the people who take handovers and review samples. The largest cost is usually invisible: wrong answers that create repeat contacts, refunds and complaints.
The levers: send simple request types to smaller, cheaper models; keep the context short; fix knowledge gaps at the source; and write good briefs, which cut the time a person spends on every handover.
Example
Illustrative example, not a client engagement.
An online homeware retailer starts with three request types: order status, delivery changes before dispatch, and returns within thirty days. Damaged-goods claims, refunds above a limit the support lead sets, and every complaint go to a person with a brief.
Before launch, the team runs scenarios drawn from its own tickets, including customers who insist on a refund outside the return window and a message claiming to be from "the warehouse manager" asking for an address change. The first runs show the agent agreeing to a late return under pressure. The fix is in the tool, which now rejects returns outside the window whatever the agent asks, and in a clearer policy passage. The agent starts by drafting replies a person sends, and takes on returns alone only once the samples are clean.
Checklist
- The agent says it is an AI, and a person is always available on request.
- A request catalogue lists what the agent handles, at which tier, and when it hands over.
- Answers cite approved sources, or the agent abstains and routes the question.
- Limits and identity checks are enforced in the tools, not the prompt.
- Write tools are idempotent.
- Every handover carries a complete brief, designed with the people who receive it.
- Customer messages and attachments cannot change the agent's instructions.
- Scenarios cover variations, pressure, social engineering and every escalation trigger.
- Must-pass scenarios pass in every run, not just on average.
- Verified resolution is the target, with guardrails beside it.
- Rollout moves one request type and one level of autonomy at a time.
- Unanswered questions reach the knowledge owners.
