In most businesses, work moves between systems because a person moves it. Someone sees an order in the shop, types it into the finance system, emails the warehouse and updates the customer record. People act as the message bus: reading, copying, forwarding and chasing. Event-driven operations replace that with one principle: when something that matters happens, the system where it happened announces it, and everything that needs to react does so — automations, agents and, where judgement is needed, people. This guide shows how to design that, and how to keep it safe when agents are among the listeners.
The problem: people as the message bus
When people carry information between systems, three things go wrong. Work waits until someone notices it. Data is re-entered, and each copy drifts. And nobody can see the whole flow, because it exists only as a series of personal habits.
Point-to-point integrations fix part of this and create a new problem. Each system calls the next directly, so every new consumer means changing the producer, and one failure can stop the chain. As the business adds systems — and now agents — the web of direct connections becomes the thing nobody dares to touch. The hidden cost of disconnected systems sets out what that costs.
Concept: business events
A business event is a record that something meaningful happened: order placed, invoice overdue, ticket escalated, contract signed, stock below threshold. It is a fact about the past, named in the past tense, and it does not say what anyone should do about it.
| Kind | Example | Says | Who decides what happens |
|---|---|---|---|
| Event | OrderPlaced | This happened | Each consumer, independently |
| Command | CreateInvoice | Do this | The sender |
| State | Order status = "shipped" | This is how things are now | Whoever reads it |
Producers publish events without knowing who listens. Consumers subscribe to the events they care about. Adding a reaction — a report, an agent, an alert — means adding a consumer, not changing the system that produced the event. "Event-driven" covers several patterns; this guide uses the simplest, event notification, with a durable record of every event.
When to use it, and when not
Event-driven design pays off when:
- several systems or teams need to react to the same thing;
- timing matters, and work currently waits for someone to notice;
- work crosses team boundaries, and handoffs are where it gets lost;
- you need a reliable record of what happened, in what order.
It is the wrong tool when a single system does the whole job, when a simple request and response is enough, when volumes are very low, or when the team cannot yet operate the extra moving parts. An event backbone nobody monitors is worse than a spreadsheet everyone checks.
Architecture
- Producers — the shop, CRM, finance system, support desk, forms and agents publish events when something meaningful happens.
- Event backbone — a broker, bus or queue receives events, keeps them durably and delivers them to subscribers.
- Consumers — deterministic automations, AI agents and approval queues react to the events they subscribe to.
- Systems of record — consumers update the systems that own each piece of data.
- Record — every event, action, approval and outcome is kept with a correlation id, so a whole flow can be traced.
Around these sit three supporting parts: an event catalogue that defines every event, a dead-letter queue for events that could not be processed, and monitoring that shows flow, delay and failure.
Components
The event catalogue
Every event the business relies on has an entry.
| Field | Example |
|---|---|
| Name | InvoiceOverdue |
| Meaning | An issued invoice has passed its due date unpaid |
| Owner | Finance lead |
| Producer | Accounting system (nightly check) |
| Payload | Invoice id, customer id, amount, due date, days overdue |
| Consumers | Reminder automation; collections agent; account manager notification |
| Personal data | Customer id only; details fetched by permitted consumers |
| Version | 2 |
The catalogue is a business document as much as a technical one. If nobody can say what an event means or who owns it, it should not be published.
Event design rules
- Name events in the past tense for what happened, not what should happen next.
- Give every event an id, a source, a type and a time, plus a correlation id that links it to its flow. The CloudEvents specification is a ready-made envelope for the first four.
- Keep payloads small. Carry identifiers and the facts consumers need; let consumers fetch the rest with their own permissions.
- Version deliberately. Add fields compatibly; publish a new version for breaking changes and retire the old one on a schedule.
- Publish reliably. Write the event in the same transaction as the change that caused it — the transactional outbox — so a crash can neither update the system without announcing it nor announce a change that never happened.
Consumers: automation, agent or person
Decide for each reaction who should handle it:
- deterministic automation for rule-based reactions with clear inputs — send the reminder, update the status, create the task;
- an AI agent where the reaction needs reading, judgement or drafting — classify the request, gather context, propose a resolution;
- a person where the decision is consequential — approve the credit, agree the exception, contact the customer about a complaint.
Most flows use all three.
Agents as consumers
Events give agents a clean way to work: they subscribe to the events in their mission, act within their boundaries and publish events of their own, so every agent action sits in the same flow as everything else. Four rules keep it safe:
- Subscribe narrowly. An agent listens only to the events its mission needs and reads only the data its role permits.
- Act through allowlisted tools. The events an agent may publish and the systems it may change are defined in advance and enforced outside the agent.
- Route consequential actions through approval events. The agent proposes; a person decides.
- Cap reactions to its own output. An agent responding to events it caused is the fastest way to generate thousands of actions.
In the Citadel's operating layer these are separate roles — Athena on intake and routing, Hephaestus on integrations, Sentinel on approvals and boundaries, Oracle on monitoring — and the Activity Ledger records what changed, who acted and on what evidence. An event-driven design is what lets that record be complete.
Approvals as events
An agent that wants to take a sensitive action publishes ApprovalRequested with the proposal and the evidence. The approval queue routes it to the right person. Their decision is published as ApprovalGranted or ApprovalDeclined, and only then does the action proceed. The approval becomes part of the traceable flow rather than an email on the side. Field Guide 014 covers the patterns.
Dead letters
Some events will fail: a malformed payload, a downstream system that is down, a rule that does not cover the case. They go to a dead-letter queue with the reason. Somebody owns that queue and reviews it daily. A dead-letter queue nobody watches is how event-driven systems lose work silently.
Implementation
- Map one value stream end to end — order to cash, say — and list the moments where work changes hands.
- Name the events at those moments, and write their catalogue entries with owners.
- Start with one event and two consumers. Prove the backbone, the monitoring and the dead-letter handling before adding more.
- Make every consumer idempotent. Delivery is usually at least once, so the same event can arrive twice. Record the ids of processed events so a second delivery does nothing.
- Decide what ordering you need. Most flows need order only per entity — per order, per customer — so design for that rather than global order.
- Retry with limits. Back off between attempts; after a set number, send the event to the dead-letter queue and alert its owner.
- Instrument the flow. Measure the time from event to completed reaction, and make it visible.
- Expand one value stream at a time, retiring the manual steps and point-to-point links the events replace.
Failure modes
- Event storms and loops. One event triggers reactions that trigger the original event again.
- Silent consumers. A consumer fails, nothing alerts, and work stops without anyone noticing.
- Duplicate processing. A redelivered event charges a customer twice because the consumer was not idempotent.
- Schema drift. A producer changes a field and three consumers break.
- Hidden coupling. Consumers depend on undocumented details of a payload.
- Invisible flow. A process spread across many events is in no single program, so nobody can see it whole. The catalogue and correlation ids are how you get it back.
- Everything is an event. Technical noise published as business events, burying the ones that matter.
Evaluation
Before production, test the flow with scenarios: duplicate events, events out of order, missing fields, a downstream system that is unavailable, a slow consumer, and a replay of a day's real events into a test environment. In production, track:
- time from event to completed reaction, per value stream;
- dead-letter volume and age;
- processing success rate per consumer;
- time waiting for human approval;
- for agents: proposals approved, edited and declined.
Security and governance
Events often carry personal or commercial data. Grant access per event type, not to the whole backbone. Keep sensitive details out of payloads and let consumers fetch them with their own permissions. Authenticate producers, so a rogue system cannot publish PaymentApproved. Set retention for stored events in line with your data policies. Keep each agent's rights to publish and consume as narrow as its mission — excessive functionality, permissions and autonomy are how an agent does damage — and record every action with the identity that took it.
Cost considerations
The infrastructure for an event backbone is rarely the largest cost; operating it is. Budget for monitoring, the ownership of dead-letter queues and the discipline of the catalogue. With agents, the main variable cost is model usage per event: filter and route with cheap deterministic rules first, and invoke an agent only for the events that need reading or judgement.
Example
Illustrative example, not a client engagement.
A wholesaler's order-to-cash flow. The shop publishes OrderPlaced. An automation reserves stock and publishes StockReserved, or StockShort when it cannot. StockShort goes to an agent, which gathers the customer's history and alternative products, proposes a resolution and publishes ApprovalRequested; the account manager decides. OrderShipped triggers invoicing. A nightly check publishes InvoiceOverdue; an automation sends the first reminder, and after a set period an agent drafts a personal follow-up for the credit controller to approve. Every step carries the order's correlation id, so anyone can trace an order from placement to payment.
Checklist
- One value stream is mapped and its handoff moments are named as events.
- Every event has a catalogue entry with a meaning, an owner and a version.
- Events are published reliably with the change that caused them.
- Every consumer is idempotent and has bounded retries.
- The dead-letter queue has an owner who reviews it daily.
- Consequential actions go through approval events.
- Agents subscribe narrowly, act through allowlisted tools, and cannot react to their own output without a limit.
- Flow time, failures and approval waits are measured and visible.
- Access is granted per event type; payloads carry minimal personal data.
- A replay of real events has been tested before production.
