In This Article
Key Takeaways
- A pilot and a production ML system are different artifacts. The pilot proves a mapping is learnable; production commits to serving it reliably, monitoring it, retraining it, and documenting it.
- Sculley et al. (NeurIPS 2015) described the surrounding infrastructure — configuration, data dependencies, serving, monitoring — as the bulk of a real ML system, with the model code a small piece of it.
- Drift detection has a test-selection problem: Evidently reports the Kolmogorov-Smirnov test becomes over-sensitive on large datasets, which is how teams end up with alarms nobody reads.
- Most stalled pilots fail on organizational questions — who owns it after go-live, whose budget runs it next year, what number defines "still working" — not on model quality.
- For federal systems, OMB M-25-21 makes ongoing monitoring, human oversight, and remedies mandatory practices for high-impact AI, not optional engineering polish.
The demo worked. That was never the question.
You have a model that hits its number on a held-out set. Somebody put it in a deck. Leadership liked it. Six months later it is still not serving a single real request, and no one can name the specific thing blocking it. Every status meeting produces a slightly different answer: security, data access, the platform team, headcount.
This is the normal outcome, and the model is almost never the cause. A pilot demonstrates that a mapping from inputs to outputs is learnable on data you already had, selected by you, at a moment of your choosing, with you watching. Production is a commitment that the mapping will still work at 3 a.m. on a Tuesday when an upstream system changes a field name, on infrastructure somebody has to patch, producing an audit trail somebody has to hand over, owned by a person whose phone rings. Those are different artifacts. Very little built for the first one carries over to the second.
The clearest description of that gap is still Sculley et al., "Hidden Technical Debt in Machine Learning Systems" (NeurIPS 2015). Its argument is that the ML code in a real system is a small component surrounded by configuration, data collection, feature extraction, serving infrastructure, process management, and monitoring — and that the surrounding parts carry ongoing maintenance costs that a quick win hides. The paper names the specific failure modes: entanglement between features, hidden feedback loops, undeclared consumers of a model's output, unstable data dependencies, configuration debt, and changes in the external world. None of those are visible from inside a notebook.
What a pilot proves, and what it does not
Take the questions production actually asks, and check them against your pilot:
- Where does each input come from at 3 a.m., and what happens when it arrives late, malformed, or not at all?
- Who gets paged, and what does the runbook tell them to do?
- Which model version produced the output a person is now disputing, and what data was that version trained on?
- What number, checkable without a human eyeballing outputs, means "still working"?
- What happens when the answer is wrong — rollback, manual override, an appeal path for the person affected?
If a pilot cannot answer those, it is not eighty percent done. It is a different project that has not started. The most useful diagnostic here is "The ML Test Score" (Breck et al., IEEE Big Data 2017), a rubric of 28 tests spanning data, model development, infrastructure, and monitoring, which deliberately treats data and infrastructure testing as first-class concerns rather than afterthoughts. Score a stalled pilot against it honestly, section by section, and the shape of the blockage tends to appear quickly. The monitoring tests are the ones a pilot has the least reason to have built — and monitoring is precisely what production depends on.
Evaluation is the step people skip
A single aggregate accuracy number is the least informative artifact in the project, because it hides three things that decide whether the system ships.
Time. If the train/test split was random rather than chronological, the model was trained on the future. Features derived from downstream outcomes leak backward. This surfaces in the first week of live traffic, long after the deck was approved.
Slices. Aggregate accuracy is a weighted average over subpopulations. A model that is strong overall and useless on the small fraction of cases that generate every escalation will be pulled by the people handling escalations, no matter what the headline number says. Per-slice reporting is not a fairness nicety; it is how you find out where the system is unusable.
The bar. Nobody wrote down what "good enough to serve" means before the build, so the launch decision has no criterion and defaults to whoever is most nervous in the room. Write the threshold first, in a document, with the slices named, before anyone tunes a hyperparameter.
The fix is unglamorous: a frozen evaluation set under version control, a written pass bar, per-slice results, and a run that reproduces end to end for someone who is not the original author. Then shadow deployment — the candidate model runs against live traffic and its predictions are logged but never served — which converts an argument into a measurement. If you are evaluating an LLM-based system rather than a classifier, the same discipline applies with different instruments; our guides on LLM evaluation and how teams run agent evals go into the mechanics.
Drift is several problems wearing one name
Three distinct things get called drift. Input drift is a change in the distribution of features arriving at the model. Prediction drift is a change in the distribution of what the model outputs. Concept drift is a change in the relationship itself — the same input should now produce a different answer. Input drift is the easiest to detect and the least conclusive. Concept drift is the most damaging and normally cannot be confirmed without ground-truth labels, which usually arrive late or never.
Then there is the trap underneath: choosing the detector. Evidently's comparison of drift methods on large datasets is worth reading before you pick one. They report that the Kolmogorov-Smirnov test becomes over-sensitive as sample size grows — with enough data, trivial differences become statistically significant, so the alarm fires constantly — and suggest sampling, noting KS works best below roughly a thousand observations. They position Wasserstein distance as a compromise between over- and under-sensitivity, Population Stability Index for when only major changes should register, and Jensen-Shannon distance as stable on large data. Evidently also ships a selector that picks a test from feature type, observation count, and unique values, so this does not have to be decided by hand for every column.
The engineering point matters more than the statistics: an alarm that fires every day is the same as no alarm. Wire each detector to an action — a ticket with a named owner, a retraining trigger, an automatic rollback — rather than to a colored square on a dashboard nobody opens. Monitoring not connected to an action is decoration, and it is the most common thing built in the final month of a stalled pilot.
Retraining is a release, not a refresh
Teams picture retraining as pressing a button. In production it is a software release, with everything that implies: a reproducible data snapshot, a pinned code commit, evaluation against the frozen bar, a promotion gate a human signs (and not the same human who trained the model), a canary rollout at increasing traffic percentages, and a tested path back to the previous version.
Two traps are worth naming. The first is retraining on data the model itself shaped. If model outputs determine which cases get reviewed, approved, or escalated, then next quarter's training data encodes last quarter's model opinions — the hidden feedback loops in the Sculley paper. They are hard to see from inside the metrics and they compound quietly. The second is retraining when labels arrive months late or never. If ground truth is delayed, a "performance decay" alarm is measuring something you cannot yet know. Say so out loud, and use a deliberately labeled canary sample rather than pretending the number means what it would mean with fresh labels.
All of this rests on the data layer. If your feature pipeline is not versioned and reproducible, none of the above is possible, which is why pipeline design and MLOps practice are the same conversation, not two.
Somebody has to be on call
An ML system in production is an operational service, and operational services need the ordinary machinery: a service level objective, an owner, a rotation, a runbook. Google's SRE book frames the tradeoff with error budgets — the difference between the reliability you committed to and the reliability you actually delivered, which becomes the remaining budget of unreliability for the period. That number is what lets a team decide whether to ship a change or spend the quarter on stability, instead of arguing about it.
A stalled pilot usually cannot answer who owns the system twelve months out, who is in the rotation, or whose budget line pays for serving and retraining next fiscal year. The team that built it is being reassigned. The platform team never agreed to inherit it. No runbook exists. That is an organizational blocker wearing a technical costume, and it will not be resolved by improving the model.
The organizational reasons pilots die
Five, stated plainly. One: no named owner after go-live. Two: the success metric was never written down before the build, so there is no criterion to pass. Three: funding covered the build, not the run — serving, monitoring, retraining, on-call and re-authorization were never in anyone's out-year budget. Four: compliance and security arrived last, after architecture decisions had hardened. Five: the pilot proved the model, leadership concluded the project was nearly finished, and the remaining work got no schedule.
For federal readers, item four has a specific current shape. OMB Memorandum M-25-21, issued April 3, 2025, rescinded and replaced M-24-10 and set minimum risk management practices for what it designates "high-impact" AI: pre-deployment testing, an AI impact assessment, ongoing monitoring for performance and adverse impacts, adequate human training and assessment, additional human oversight and intervention, consistent remedies or appeals, and consultation with end users and the public. Agencies were directed to document implementation within 365 days of the memo — a date that has now passed. Read that list against a typical pilot: at least four of the seven are things a pilot never builds, and each one is a system component, not a paragraph.
NIST's AI Risk Management Framework 1.0 (NIST AI 100-1) makes a similar point in a different vocabulary. Its Measure and Manage functions assume evidence that exists continuously — evaluation over the system's life, incident records, monitoring telemetry, documented limitations and intended use. A notebook produces none of that. A properly built pipeline produces it as a byproduct, which is the strongest practical argument for building one before you need the paperwork. Researchers have also started giving the recurring mistakes shared names; "Using AntiPatterns to avoid MLOps Mistakes" (Muralidhar et al.) catalogs defective practices observed in enterprise deployment precisely so teams can point at a problem instead of describing it from scratch each time.
A short readiness check
Before the next go/no-go, answer these in writing:
- Can you retrain the exact model now serving traffic, from a frozen data snapshot and a pinned commit, without the original author in the room?
- Can you name, right now, the model version in production and the dataset behind it?
- What single number tells you it stopped working, who is paged when it crosses, and how long does a rollback take?
- Which features are computed differently online than offline? Train/serve skew is the quiet killer, because nothing about it looks wrong in training.
- Who owns this in twelve months, and whose budget pays for it?
- What documentation could you hand a security control assessor tomorrow, without writing anything new?
If several answers are "we would have to find out," the blocker is not the model, and another round of tuning will not move it.
If you want help with this
The pilot works. The path to production does not exist yet.
That is the situation most teams are in when they start looking: a model somebody already decided to run, and no registry, no drift monitoring, no retraining gate, no rollback, and no documentation an assessor would accept. Precision Federal — a federal software and AI firm, and the sister company of this site — builds that operational layer.
Concretely, per its MLOps capability page, the firm builds:
- A model registry that functions as the authorization artifact — semantic versioning, immutable pointers to the training data snapshot and code commit, a required model card, a two-person promotion gate so an author cannot self-promote, and a rollback path to any prior version.
- Reproducible training and data lineage — versioned data snapshots, a feature store that keeps online and offline features identical, and orchestrated training pipelines on Kubernetes, AWS GovCloud, or Azure Government.
- Monitoring wired to action — input and prediction drift, performance decay measured on a held-out canary when labels are delayed, fairness slices, latency and error SLOs, each breach opening a ticket rather than turning a square red.
- CI/CD with real gates — security and dependency scanning, SBOM generation, model evaluation as a build step, shadow deployment, and progressive canary rollout with automatic rollback on regression.
- The compliance byproduct — NIST AI RMF artifacts and continuous-ATO change records generated from pipeline logs, so authorization evidence is produced by the system rather than assembled by hand at the end.
What an engagement looks like. A scoped assessment first: read the existing pipeline, the model, the authorization boundary, and whatever monitoring exists, then hand back a written readiness picture and a build plan you own whether or not you continue. If it goes forward, the build follows the sequence on the capability page — experiment tracking and versioning, data validation, training automation, registry and approval gate, serving with canary deployment, then drift monitoring and retraining triggers.
Where this is not the right fit — worth saying plainly:
- If no one on your side will own the system after go-live, an MLOps build produces shelfware. The firm will raise this before scoping rather than after delivery.
- If what you need is a data science team to find a use case or improve model accuracy, this is the wrong service. This work is the operational layer under a model somebody has already decided to run.
- If ground-truth labels never arrive for your problem, an automated retraining loop cannot be built honestly, and you should be told that at assessment rather than sold a pipeline that pretends otherwise.
- A dashboard as the deliverable is not something the firm will build. Monitoring ships wired to a ticket, a trigger, or a rollback, or it does not ship.
Sources: Sculley et al., "Hidden Technical Debt in Machine Learning Systems," NeurIPS 2015; Breck et al., "The ML Test Score," IEEE Big Data 2017; Evidently — comparing drift detection methods on large datasets; Google SRE Book — Embracing Risk (SLOs and error budgets); NIST AI Risk Management Framework 1.0 (NIST AI 100-1); Wiley — analysis of OMB M-25-21; Muralidhar et al., "Using AntiPatterns to avoid MLOps Mistakes". Analysis and framing by Precision AI Academy. Verify any policy text against the primary source before acting on it.
Common questions
Why do most AI pilots never reach production? Because a pilot and a production system are different artifacts. A pilot proves a mapping from inputs to outputs is learnable on data you already had. Production requires reproducible data pipelines, versioned models, evaluation with a written pass bar, drift monitoring wired to action, retraining gated like a software release, an on-call owner, and documentation an assessor can read. Almost none of that is built during a pilot.
What is the difference between data drift, prediction drift, and concept drift? Data or input drift is a change in the distribution of features arriving at the model. Prediction drift is a change in the distribution of model outputs. Concept drift is a change in the underlying relationship, so the same input should now produce a different answer. Input drift is easiest to detect and least conclusive; concept drift is most damaging and usually needs ground-truth labels to confirm.
Which statistical test should be used to detect data drift? It depends on sample size and how sensitive the alarm should be. Evidently reports that the Kolmogorov-Smirnov test becomes over-sensitive on large datasets and suggests it works best below roughly a thousand observations, positioning Wasserstein distance as a compromise, Population Stability Index for when only major changes matter, and Jensen-Shannon distance as stable on large data. Evidently also ships a selector that chooses a test from feature type, observation count, and unique values.
What does OMB M-25-21 require for high-impact federal AI? Issued April 3, 2025, it rescinded and replaced M-24-10 and set minimum risk management practices for high-impact AI: pre-deployment testing, an AI impact assessment, ongoing monitoring for performance and adverse impacts, adequate human training and assessment, additional human oversight and intervention, consistent remedies or appeals, and consultation with end users and the public. Agencies were directed to document implementation within 365 days of the memo.