Almost every organisation I talk to has run an AI pilot. Far fewer can tell you what percentage of the intended users touched it last week. That gap — between something that works in a demo and an organisation that reliably operates differently because of it — is the whole problem.
At Amazon I led an AI productivity charter end-to-end: from proof of concept with Engineering and Data Science, into production, then monitor-and-iterate. We embedded a GenAI toolkit into the daily work of 5,000+ operators across five regions, and went from 0% to 98% adoption in six months, retiring roughly 50,000 hours of manual effort per week. Here is what I think actually mattered — and what didn’t.
Adoption is a change-management problem wearing a technology costume
The instinct is to treat low adoption as a product gap: the output isn’t good enough, so people don’t use it. Sometimes that’s true. More often, people don’t use it because it isn’t in the path of the work they already do, nobody told them what “good” looks like, and no one is accountable for whether they use it at all.
Once I started treating adoption as a change problem rather than a tooling problem, the levers changed completely. Less time tuning prompts. More time on where the tool sits in the workflow, who owns the outcome, and what happens in week three when the novelty wears off.
Ground it, cite it, or refuse
Trust is the constraint that decides everything downstream. An operator who gets one confidently wrong answer will quietly stop using the tool, and they will not tell you.
So the core use case — an SOP lookup built as grounded retrieval on AWS Bedrock — was deliberately constrained: answer only from the retrieved source, cite that source, and escalate rather than invent when nothing matches. That is a worse demo. It is a far better product. The visible citation is what lets someone check the answer in five seconds instead of abandoning the tool after one bad experience.
Alongside it we ran two more: decision support, and an onboarding buddy that let new joiners interrogate institutional knowledge instead of interrupting a colleague. Three use cases, not thirty. Narrow and trusted beats broad and doubted.
Measure usage that means something
The easiest metric to move is the least useful one. Daily active users tells you people opened something. It doesn’t tell you the work changed.
What I pushed for instead was active, purposeful usage — is this tool being used for the task it was built for, by the people whose workflow it was meant to change, repeatedly? That number is harder to move and much harder to fake, which is precisely why it’s worth reporting.
The same discipline applies to the business case. I pitched the charter conservatively — roughly 4x return assuming only half the target population ever adopted it, with a payback measured in months. We realised about 8.6x in the first year. Promising conservatively and delivering above it buys you the credibility to do the next thing. Promising 10x and landing at 3x ends the programme, regardless of whether 3x was a good outcome.
Codify it or it leaves when you do
The part I’d most defend in hindsight is the least glamorous. We wrote down the prompt-engineering and LLM-evaluation guidelines, and they were adopted as the organisation’s production standard.
That’s what turns a successful programme into infrastructure. Without it, adoption is a personality — it works while the person driving it is in the room, and decays when they leave. Governance gets a bad name because people confuse it with bureaucracy. Done properly it’s the opposite: it’s what lets the next team start from your ending point instead of your starting one.
What I’d tell someone starting now
- Pick three use cases, not thirty. Depth of trust beats breadth of coverage.
- Design the citation before the answer. Verifiability is the adoption feature.
- Name an owner for the outcome, not just for the tool.
- Report usage that would embarrass you if it were vanity.
- Write the standard down while you still remember why you chose it.
None of this is about having the best model. It’s about building the operating system around it — which is, more or less, what I now spend my days doing at Agoda.