Most organisations have now run at least one AI pilot. Far fewer can point to a process that works differently because of it. The pilot impressed the people who saw it, a report was written, and the work carried on as before. The gap between a promising experiment and a change in how work gets done is rarely technical. It is operational: ownership, process, measurement and adoption. This article sets out a practical framework for closing it.
Why pilots stall
Pilots tend to stall for the same handful of reasons, and none of them is the model.
- The pilot was designed to prove the technology, not to change the work. Its success criterion was "it works", which a good demo can meet without anything in the business changing.
- Nobody in operations owns it. It belongs to an innovation team or to IT. When the pilot ends, there is no one whose job it is to carry it into the process.
- It runs beside the process, not inside it. People do the work the old way and then try the new way as well. After a few weeks of double effort, they stop.
- It is measured on activity. Number of users, number of prompts, number of documents processed. None of these says whether the outcome improved.
- There is no decision at the end. The pilot is neither scaled nor stopped. It lingers, consuming attention and budget, until someone quietly turns it off.
Each of these is a design choice made before the pilot starts. That is good news: it means they can be made differently.
Treat the pilot as the first stage of production
The single most useful change is to stop treating a pilot as an experiment and start treating it as the first, deliberately small stage of production.
| Pilot as experiment | Pilot as stage one of production | |
|---|---|---|
| Owner | An innovation or IT team | The person who runs the process today |
| Scope | Whatever looks impressive | One workflow, one outcome |
| Data | A sample or a sandbox | Real, permitted data, in a controlled scope |
| Users | Enthusiasts | The team that actually does the work |
| Success | "It works" | The outcome moved against a baseline |
| Autonomy | Whatever the demo showed | Recommend or assist first; more on evidence |
| Ending | A report | A recorded decision: scale, adjust or stop |
The right-hand column takes longer to set up. It is much faster to turn into value, because nothing has to be rebuilt or re-argued when the pilot ends.
Five conditions for productivity
A pilot becomes productivity when five conditions are in place. If one is missing, it is worth fixing before anything else.
- A named operational owner. Someone who runs the process today and will run it tomorrow, and who wants it fixed. Not a sponsor; an owner.
- A baseline and a target. Measure the current process before changing it: how long a case takes, how many errors, how big the backlog, what each completed case costs. Without a baseline, any result is an opinion.
- A workflow redesigned around the agent. Decide who does what in the new process, where a person approves, and what happens to the exceptions. An agent added to an unchanged workflow usually adds a step rather than removing one.
- Bounded autonomy. Start where the risk says you should. The Citadel's trust model describes the path: the agent first observes and recommends while a person executes, then assists and waits for approval, then operates within narrow limits, and only then optimises. Each step is a decision a person makes on evidence.
- A decision date. Agree in advance when the pilot will be judged and what result means scale, adjust or stop. A decision date turns a pilot from an open question into a commitment.
Adoption is the work
A pilot can meet every technical criterion and still fail because the people who do the work never adopt it. Adoption is not a communication exercise at the end; it is part of the design.
- Explain what changes for each role. Not "we are introducing AI", but "you will no longer re-key orders; you will review the ones the agent flags".
- Keep a person accountable. People adopt tools they can override. They resist ones that seem to decide for them.
- Make the agent's work visible. A record of what the agent did, on what evidence, and who approved it builds trust faster than any training session.
- Give people a way to say "this is wrong". Corrections and declined approvals are the best information you will get about where the agent and the business disagree. Capture the reasons and act on them.
- Train on the new process, not the tool. The question people need answered is "what do I do now?", not "which button do I press?".
Measure what changed
Three groups of measures tell you whether a pilot is becoming productivity.
- Outcome measures — cycle time, error rate, backlog, customer waiting time, cost per completed case. These are the reason the pilot exists.
- Operating measures — how often a person has to intervene, how many exceptions occur, whether escalations reach the right person. These tell you whether the agent can be trusted with more.
- Adoption measures — the share of eligible work that actually goes through the new process, and how often people override it, and why.
The Guild's evaluation scorecard puts human intervention and business outcome alongside quality and safety for the same reason: an agent that is accurate but unused, or used but not trusted, has not yet produced value. Evaluation is also not a launch event. When the model, the prompts, the tools or the data change, the measures need to be checked again.
From one pilot to many
When the first pilot has become part of the process, the most valuable thing it leaves behind is not the agent. It is the set of habits around it: a named owner, a baseline, bounded autonomy, a visible record, a regular review.
The second initiative should reuse all of them. Keep a short portfolio view — each initiative, its owner, its stage and its next decision — and review it on a fixed rhythm. Stop the initiatives that are not moving their outcome. Widen an agent's boundaries only when the evidence supports it.
This is how the Guild structures its own engagements: discover, design, build, integrate, deploy and evolve, with a decision, an owner and evidence at every gate. The specific gates matter less than the principle. Every step should end in a decision someone owns.
Before your next pilot
- Can you name the person who runs this process today, and have they agreed to own the pilot?
- Have you measured the current process, so you will know whether anything improved?
- Is the pilot running inside the real workflow, with real (permitted) data and the real team?
- Have you decided which actions need a person's approval?
- Do you know what result means scale, what means adjust and what means stop?
- Is there a date on which that decision will be made?
- Will you be able to see afterwards what the agent did, and who approved it?
If the answer to most of these is yes, you are no longer running an experiment. You are making the first, careful change to how the business works — and that is where productivity comes from.
