From AI Pilot to Production: The Operating Gap

Everybody talks about the pilot. Almost nobody talks about the gap between the pilot and production — the unglamorous engineering work that determines whether an AI system ever delivers a dollar of value. This is about that gap: what production actually requires, why most pilots never cross it, and how to build the bridge on purpose.

The gap is not a technology gap. The models are good enough. The gap is a discipline gap — the difference between a system that works when watched and a system that works when ignored. Crossing it requires doing a specific, finite list of boring things, and most organizations do not know the list exists, let alone what is on it.

Production is a list of boring commitments

A production system is not defined by its technology. It is defined by a set of commitments the team makes about how the system behaves under stress, and the engineering that backs those commitments up. The list is remarkably stable across domains — the same commitments apply to an AI workflow that processes invoices and to one that triages support tickets.

• A named owner: A human, on a team, whose performance review mentions this system. Not 'the data team.' A name.
• An on-call rotation: Someone takes a pager. If nobody is paged when it breaks, it is not in production.
• A runbook: What to do when it breaks, written down before it breaks, tested at least once by someone who did not write it.
• Service level objectives: A number that defines 'working,' measured continuously, with an alert when it is breached.
• Monitoring on quality, not just uptime: The system can be 'up' and producing wrong answers. Quality must be measured too.
• Drift detection: A mechanism that notices when inputs or outputs shift away from the baseline, before users do.
• A rollback path: A way to undo the last deploy, tested, that does not require the person who shipped it to be awake.
• An eval suite: A held-out set of inputs that runs on every change, so regressions are caught before they ship.

Read that list slowly. Most pilots have none of these. Most “production” AI systems have two or three. A system with all eight is rare, and is also the only kind that reliably pays off. The list is the gap. The gap is the list.

Most teams cannot tell where on the ladder they are

One of the more striking findings from our audits is that teams systematically overestimate where they sit on the pilot-to-production ladder. A team that has a dashboard and a Slack channel will describe their system as “in production” when, by the list above, it is at best hardened, and often still a pilot. The dashboard is checked occasionally, the Slack channel is where outages get noticed, and there is no runbook, no on-call, and no rollback path. The system works — until it does not, at which point it is a surprise to everyone.

The diagnostic below is the short version of the audit we run with clients. Answer it honestly. If you score as a pilot, that is not a failure — it is a map of the work that needs doing, and it is closeable.

The gap is crossed one boring commitment at a time

The good news about the gap is that it is finite. The list above is the whole list. There is no secret second list of harder things you discover after you finish the first. Cross the eight commitments and you have a production system. The work is not glamorous — it is runbooks and dashboards and on-call schedules and eval suites — but it is knowable, and it is the difference between an AI initiative that compounds and one that quietly dies.

The order in which you cross the commitments matters less than the willingness to start. Our recommendation, in practice, is to begin with the named owner and the runbook, because those two unlock everything else — you cannot build an on-call rotation without an owner, and you cannot take a pager without a runbook. From there, monitoring and SLOs come next, then drift detection and eval suites, then the rollback path that ties it all together. Eight weeks of focused work, for a typical pilot, is enough to cross the gap. The value that unlocks is usually a multiple of what the pilot was already nominally delivering.

That last point is worth sitting with. The pilot was already doing the work. The production wrapper does not make the system do more work — it makes the work reliable enough to count on, which is what unlocks the business to depend on it, which is what turns a pilot into value. The gap is not between the pilot and more value. The gap is between the pilot and the value it was always capable of, if anyone had built the wrapper. Build the wrapper.