Analytics is only useful when it changes what someone does. The Guild's version of the topic starts from that test: every measure exists for a decision someone owns, every figure has one definition and a traceable source, and every report says what changed, compared with what, and what might be done about it.
The same discipline applies to a sales pipeline, a support queue, a website or an AI agent. Agents in particular are easy to measure badly — by messages sent, tokens used or tickets deflected — and harder to measure well, by what they actually achieved and what it cost to achieve it.
Questions this topic answers
- Which decisions should our reporting serve?
- Is this change a signal, or normal variation?
- What should a useful alert contain, and who should receive it?
- How do we know whether an AI agent is actually helping?
- Which number should a support agent be judged on?
Start here
- Operational intelligence is the missing layer — why dashboards alone do not change outcomes.
- Signal in the noise — Oracle's tests for a signal, and the shape of a message an owner can act on in a minute.
- Five measurements that tell you whether an agent is helping — outcome, intervention, rework, time to outcome and cost per completed mission, read together.
- The KPI your support agent should actually optimise — why deflection rewards the wrong behaviour, and the case for verified resolution.
- Measuring agent productivity beyond tokens — the Forge protocol that tests whether token and activity counts say anything about value.
- Operational Intelligence for SMBs — Field Guide 033: decisions, owners, signals and a weekly review for a small team.
Key ideas
- Decisions first — start from the decisions people own, then measure what feeds them. A measure nobody acts on is a cost.
- One definition per measure — written down, owned and used in every report.
- Baselines — compare with what normal looks like for that day, season and segment, not with a number picked once.
- Signal or context — a change is a signal when it matters to a decision and someone can act on it; otherwise it is context.
- Correlation is labelled — two things moving together is a finding, not a cause.
- Honest confidence — every figure comes with its period, its sample and how sure we are.
- Outcomes over activity — report what the business gained or lost, not how busy its systems were.
- Agents are measured like operations — task success, accuracy, escalation, cost per mission and business outcome, with baselines and trends.
Who knows this topic
- Oracle — the Guild's analytics and operational intelligence agent. Evaluates results, detects patterns, tests assumptions and turns evidence into decisions; owns scoring in the Evaluation Lab.
- Atlas — maps the sources and lineage that every analysis depends on.
On managed operations the division of labour is short: Sentinel watches, Oracle interprets, humans decide. Oracle's human checkpoint is the Data Wizard, the role that turns raw data into strategic clarity through pipelines, models and dashboards.
Where this knowledge is put to work
Data & Intelligence builds the trusted definitions and reporting that analytics depends on. The Evaluation Lab applies the same measurement to AI agents across sixteen dimensions, from task success to business outcome. Managed operations puts signals and evaluation to work every day.
Conflicting reports and unclear KPIs? Start with one decision and the measure it needs.
