It is almost never the model
The pilot hit its accuracy target. Six months later it still runs in a notebook on somebody’s laptop. We have seen this often enough that we now scope against it explicitly, and the causes are organisational every time.
One: no owner on the business side
A model that nobody is accountable for is a curiosity. The question to answer before building is not “can we predict this” but “whose decision changes when we do, and will they change it”.
Two: the data was hand-assembled
Pilots run on an extract someone prepared carefully. Production needs that extract every day, tested, with someone paged when it breaks. If the pipeline was not part of the pilot, the pilot proved less than it looked.
Three: no path to deployment was agreed
Deployment turns out to require a review board, a security assessment and a platform team with its own roadmap. None of that is unreasonable, and all of it takes months if discovered late.
Design the production path in week one, even if you never walk it. Discovering it in month five is how pilots die.
Four: success was never defined in business terms
A five-point accuracy gain does not release budget. “Nine fewer stockouts a week” does. Agree the threshold before starting, in the units the person signing the invoice already uses.
What we do differently
We scope the pilot and the production path together, and we write down the number that counts as success. Four times we have reached that number and recommended against scaling anyway, because the operating cost outweighed the gain. Saying so is cheaper than finding out in year two.
Tasha Bramley
Head of Machine Learning