Playbook · Rollout and training
Rolling an AI agent out to a team
Most AI tools fail after the demo, when real people meet them on real work. This is the plan: who learns first, what the first four weeks look like, and the numbers that decide whether the tool stays. The timer at the bottom does the arithmetic.
Who learns first
Training goes in this order because each group sets up the next one. The people who check the output learn before the people who produce it.
- The sponsor and the process owner. They agree what success means, the rule for stopping, and who approves the tool’s output. Nothing else starts until this is written down.
- The reviewers. They set the quality bar. They train on the cases the tool gets wrong in testing, so they know what a bad output looks like before they see one on live work.
- Two or three champions, week 1. They use it on real work every day, with a person approving every output, and log each correction and why.
- The wider team, from week 2. Groups of five to eight, led by a champion, working on their own tasks rather than a scripted example.
- New starters, after week 4. The tool goes into onboarding once the week 4 check passes, not before.
The first four weeks
| When | What happens | Done when |
|---|---|---|
| Week 0 | Time ten runs of the task by hand. Agree the success measure and the stop rule with the sponsor. Write the review checklist. | Baseline medians written down and signed off. |
| Week 1 | Champions use the tool on real work. Every output is approved by a person. Every correction goes in a log with its cause. | Ten timed runs with the tool, and the correction log reviewed. |
| Weeks 2 and 3 | The team learns in small groups, led by champions. Office hours twice a week. The three most common corrections get fixed in the prompt, the data or the template. | Most of the team has used it on real work at least three times. |
| Week 4 | Time ten runs again, the same way as week 0. Compare. Decide: keep, fix or stop. | The decision is written down with the numbers behind it. |
What gets measured
Six measures, taken at week 1 and again at week 4, the same way both times.
| Measure | How it is taken | Week 1 | Week 4 |
|---|---|---|---|
| Time per task | Median of ten timed runs, start to finish, corrections included. | No slower than by hand | At least 30% faster |
| Adoption | Share of eligible tasks done with the tool. Logins do not count. | Champions use it daily | 60% of eligible tasks |
| Edit rate | Share of outputs a person changed before they were used. | Logged, no target yet | Falling week on week |
| Errors caught | Mistakes the review step stopped before they reached a customer, each with its cause. | Every one logged | No cause repeats after its fix |
| Escalations | Cases the tool sent to a person, each read to check the reason was right. | All read | Reasons right on a sample of ten |
| Confidence | One question, 1 to 5: “Would you use this output after a quick check?” | Baseline taken | Higher than week 1 |
These thresholds are examples. The real ones are set with the sponsor in week 0, before anyone sees a result.
When people stop using it
Ask before you remind. Sit with two people who stopped and watch them do one task. The reason is usually one of four, and each has its own fix.
- It is slower. Fix the step that costs the time, or narrow the tool to the tasks where it is faster.
- It is wrong too often. The correction log shows where. Change the prompt, the data or the template, then tell people what changed.
- It lives somewhere else. Put it where the work already happens, not in another tab.
- Nobody said it was allowed. The manager says, in writing, which tasks it is for and that checking its output is part of the job.
Timing it the same way twice
- The same kind of task and the same mix of people, timed from start to finish.
- At least ten runs on each side. A run is left out only when it was interrupted, and the write-up says so.
- Correction time counts. An AI draft that takes 4 minutes and needs 6 minutes of fixing is a 10 minute task.
- Medians, not averages, so one stuck task does not decide the result.
Pilot timer
Did it actually save time?
Paste the minutes from your timed runs. The timer pairs each correction with its AI run, compares medians and gives the week 4 call. When it saves time, one click prices it in the ROI calculator.
Your timings
Try a scenario
What the timings say
Comparing…
Real vs simulated
- Real
- The method: the training order, the four weeks, the six measures and the timing rules. The timer’s arithmetic is exact, and its tests run with the rest of the site.
- Simulated
- Every timing in the scenarios is invented to show one verdict each. The thresholds in the table are examples, not results from a client.
- Why it matters
- A rollout is judged on the team’s own work, not on the demo. If the numbers do not move by week 4, stopping is a good result: it costs less than a tool nobody uses.