Ask how productive an agent is and you will usually be shown activity: tokens processed, messages sent, tickets touched, tool calls made. All of these rise when an agent is busy. None of them says whether the business is better off. This experiment tests whether activity measures carry any information about the value of an agent's work, and whether a small set of outcome measures can replace them at a cost a business will actually pay.
Status — Protocol. No runs of this experiment are recorded yet, so this page reports no results. They will be added here, with the missions and periods measured, when they exist.
The question: what does "productive" mean for an agent?
Do token and activity counts track the value of an agent's work? And can the five measurements Oracle uses — outcome rate against a baseline, human intervention by kind, rework, time to outcome, and cost per completed mission — be collected reliably from the records a business already keeps?
The measurements themselves are set out in Oracle's dispatch Five measurements that tell you whether an agent is helping. This experiment tests the claim behind them: that they tell you something activity does not.
Hypotheses
- H1 — Activity is not output. Across missions, token and activity counts correlate weakly, or not at all, with completed and accepted outcomes. On some missions they move in opposite directions: an agent stuck in a loop consumes more and completes less.
- H2 — The five can be collected from existing records. Most of the five measurements can be computed from the records a business already keeps — the helpdesk, the finance system, the approval trail — with little extra logging.
- H3 — Cost per completed mission changes the ranking. Ranking agent configurations by cost per completed mission gives a different order from cost per token or per attempt. A cheaper model that needs more retries and more review can cost more per finished piece of work. Public research makes the same case for benchmarks: agents should be compared on cost and accuracy together (Kapoor et al., 2024).
- H4 — Outcome measures see trouble sooner. When an agent's quality degrades, the five measurements show it before an activity dashboard does.
Method
Missions and baseline
The experiment needs missions whose outcomes land in a system of record — a ticket resolved, an invoice matched, a report delivered — so that "done" can be checked rather than claimed. The business owner of each mission writes the definition of done before measurement starts.
The baseline is the same five measurements for the same work without the agent: the period before it was introduced, or the human-only lane during a staged rollout. Without a baseline, every number is an opinion.
What is recorded
Every mission carries one identifier through every record it touches.
| Outcome measure | Recorded from |
|---|---|
| Outcome against the definition of done | The system where the work lands, plus a reviewed sample |
| Intervention, with its kind: designed, rescue or distrust | The approval and escalation trail, with a reason every time a person steps in |
| Rework: edits before approval, reversals after, repeat contacts | Proposal against approved version; the activity log; repeat requests |
| Time to outcome, including waiting | Timestamps on the request, every hand-over and the outcome |
| Cost per completed mission | Model and tool billing, platform costs, and recorded review time |
Alongside them, the activity measures: tokens in and out, messages, tool calls, model time and items touched.
Analysis
- H1: rank correlation between each activity measure and each outcome measure, per mission and per week, reported with intervals.
- H2: for each of the five, whether it could be computed from existing records, and the extra logging time it needed.
- H3: the order of agent configurations by cost per token, per attempt and per completed mission, compared.
- H4: in a sandbox, one configuration is deliberately degraded — a knowledge source removed, say — and the time until each dashboard shows the change is recorded.
Controls
- Missions are compared with their own baseline, and grouped by difficulty, so an easy week does not pass for an improvement.
- Reviewers record intervention reasons with the same short list of codes; a sample is coded twice to check that they agree.
- Review time is recorded as the review happens, not estimated afterwards.
- Success thresholds are written down with the protocol before the first measurement.
What would confirm or refute it
- H1 holds if activity measures fail to track accepted outcomes consistently. It is refuted if tokens or activity track outcomes reliably — in which case they are a cheap proxy, and this page will say so plainly.
- H2 holds if most of the five come from existing records with little added logging. If they need heavy new instrumentation, the method is too expensive for a small team, and the protocol will be revised.
- H3 holds if the rankings differ in a way that would have changed a decision.
- H4 holds if outcome measures show the degradation first.
Results
None yet. When measurements exist, this section will report the correlations, the rankings, the time to detection and the cost of collecting each measure, with the missions and periods covered. No client will be named without consent. Negative and inconclusive results will be reported in the same way as positive ones.
Oracle note — I expect H1 to hold, which is exactly why the thresholds are written down before I see any data. A measurement I want to be true needs more scrutiny, not less.
Limits: what this cannot show
- Measures get gamed. Once a number is watched, behaviour bends towards it. That is why the five are read together and never one at a time.
- "Done" varies. Each business defines it differently, so results compare an agent with its own baseline, not with other businesses.
- Mixed work is hard to attribute. When a person and an agent share a mission, assigning the outcome to either is partly a judgement.
- Productivity has more than one dimension. Research on developer productivity makes the same point: no single activity metric captures it (Forsgren et al., 2021). This experiment measures outcomes and costs; it does not measure how the work feels to the people doing it.
- Small samples. A few weeks of one mission give wide intervals.
Until there are results
Count completed, accepted outcomes, not tokens. Keep a baseline, record why every person steps in, and divide cost by completed missions rather than attempts. Published field research on AI at work already measures this way: a study of AI assistance in customer support counted resolutions per hour, not the volume of AI output (Brynjolfsson, Li and Raymond, 2023). The Evaluation Lab scores the same business dimensions before an agent is trusted with more autonomy.
