The AI pilot that never becomes a product

There is a particular kind of AI project that everyone agrees went well and no one can point to in production. The demo landed. The accuracy numbers were respectable. The executive sponsor was pleased. Eighteen months later the notebook is still a notebook, and the team has quietly moved on to the next pilot.

This is not a failure of ambition or of talent. It is a failure of framing. A pilot is designed to answer the question "can this work?" — and it answers it honestly. The trouble is that nobody funded the far more expensive question underneath it: "can this work every day, for every user, when the data is late and the vendor changes their API and the person who built it has left?"

The demo is allowed to skip the hard parts

Pilots run on curated data, on a fixed snapshot, with a human in the loop who happens to be the author. Production runs on whatever arrives. The move from one to the other is where the real engineering lives, and it looks nothing like model work.

What separates the two is rarely intelligence. It is the unglamorous scaffolding: a data pipeline with contracts and alerting, a serving path with a latency budget, an evaluation harness that runs on every change, a fallback for when the model is unavailable or wrong, and an owner whose job description includes the words "when this breaks."

What tends to separate the pilots that ship

• A named production owner from day one — not a sponsor, an owner, on a team with an on-call rotation and a budget line.
• A defined decision the output feeds. If no downstream process changes because of the prediction, the model is a very expensive opinion.
• An evaluation set that reflects reality rather than the happy path, including the messy, ambiguous and adversarial cases the pilot quietly excluded.
• An honest cost model covering inference, retraining, monitoring and human review — priced per unit of business value, not per API call.
• A rollback story. Every AI feature needs a defined behaviour for the day the model is worse than the thing it replaced.

Scope the pilot around the production question

The most useful correction is also the least popular: design the pilot to fail for production reasons rather than modelling ones. Feed it live data. Run it on the slowest, least reliable input source you have. Give it to someone who did not build it. Ask what happens at ten times the volume.

A pilot that survives those conditions is worth building. One that only survives a curated dataset has told you something about your dataset, not about your business. Learning that in six weeks is cheap. Learning it in year two, after the roadmap has been rewritten around a capability that never arrived, is not.