Key takeaways
- Measure the current process first, or you cannot prove improvement.
- Pick one primary success metric and a few guardrail metrics.
- Agree on stop rules in writing before the pilot begins.
- Test on real, messy inputs, not on the cases that make the demo shine.
Why pilots drift
An AI pilot begins with enthusiasm and a working demo. Six weeks later, people disagree about whether it worked. One group remembers the impressive examples, another remembers the failures, and nobody has numbers to settle it. The pilot then either lingers indefinitely or gets scaled on the strength of a good feeling. Both outcomes are expensive.
The cure is to treat the pilot as an experiment with a hypothesis, a measurement plan and an ending, all written down before the first prompt is tested.
Capture the baseline first
You cannot show improvement without knowing where you started. Before the pilot, measure how the task is done today: how long it takes, how often it needs rework, what it costs in staff time and how satisfied the people downstream are. Even a rough sample of a few dozen real cases is better than none. If the task is not measured today, the first deliverable of the pilot is the measurement.
Be honest about the human baseline. People are not perfect either, and comparing an AI system to an imagined flawless employee sets an impossible bar. Compare it to the actual current process, including its errors and delays.
Choose the success metric and the guardrails
Pick one primary metric that captures the value you hope for, such as time per case, cases handled per person or first-pass accuracy. Then add guardrail metrics that catch harm you would not accept even if the primary metric improves.
- Quality guardrails: error rate on a reviewed sample, and severity of the worst errors.
- Risk guardrails: any exposure of restricted data, and any output that breaks a policy.
- Experience guardrails: reviewer effort and user satisfaction, since a tool that saves time but frustrates people rarely lasts.
Write the target as a range, not a single number, for example the minimum improvement that would justify the cost of running it in production. That threshold is a business judgment, so involve the budget owner.
Test on real inputs
Demos use clean, favorable examples. Production does not. Build the evaluation set from real cases, including the awkward ones: incomplete records, ambiguous requests, unusual formats and edge cases your team complains about. Keep a portion of the set hidden from anyone tuning prompts, so you have an honest final check.
Where possible, have reviewers grade outputs without knowing whether they came from the system or from a person. It removes a surprising amount of bias in both directions.
Write the stop rules
Stop rules are the part teams skip, and the part that saves the most money. Agree in advance on conditions that end the pilot early or send it back for redesign.
- The primary metric does not beat the baseline by the agreed margin by the review date.
- A guardrail is breached in a way that cannot be fixed with a configuration change.
- The cost of supervision and correction consumes the time savings.
- The data or access needed turns out to be unavailable or not permitted.
Also define what success unlocks: which team, which volume, which controls and which budget come next. A pilot without a next step is a demo.
Decide, then document
At the end, hold a short review with the numbers on one page: baseline, result, guardrails, cost and a recommendation to scale, revise or stop. Stopping is a legitimate and valuable outcome. It means the organization learned something cheaply. Record what you learned so the next pilot starts smarter.
