What a production MLOps workflow needs to prove
MLOps connects model development to a service that can be tested, released, observed, and recovered. The objective is a repeatable decision about whether a particular model and its surrounding software are safe to release for a defined task. Faster deployments can be useful, but deployment frequency alone does not establish prediction quality or reliability.
This guide provides a practical release and monitoring plan for predictive machine learning. It also identifies additional checks for language-model applications. It does not report a TensorBlue benchmark or promise a percentage improvement. Set acceptance thresholds from your product requirements, historical performance, and the consequences of a wrong decision.
CI, CD, and continuous training
Google's MLOps architecture guide describes continuous integration, delivery, and training for predictive ML. Beyond testing software, the workflow needs checks for data and model quality. Continuous training is a separate decision: automating a training run does not mean every resulting model should reach production.
A small team can start with a scheduled training job, versioned artifacts, a repeatable evaluation command, and a reviewed release. Add orchestration when manual steps create measurable risk or repeated work. Avoid adopting a complicated platform before identifying who will operate it and what failures it must handle.
Version the complete prediction contract
A model file is only one part of a release. Record the code revision, dependency lockfile, training configuration, feature definitions, dataset snapshot or immutable reference, label rules, evaluation results, and serving configuration. Include the owner and intended use. A release should identify exactly which inputs it accepts and what its outputs mean.
For a fraud score, document the event timestamp, missing-value behavior, feature availability at decision time, score range, and decision threshold. A training query that uses information arriving after the decision creates leakage. A schema-compatible request can still be wrong if units, time windows, or feature definitions changed.
Keep sensitive training examples out of ordinary build logs. Store access-controlled references and integrity checks instead. Agree on retention and deletion behavior with the data owner so reproducibility does not silently become unlimited retention of personal information.
Build CI around failure modes
- Software checks: validate transformations, serialization, API contracts, and deterministic fixtures. Confirm the serving package can load the selected artifact in a clean environment.
- Data checks: validate required columns, types, time boundaries, duplicates, missing values, and label coverage. Compare expected ranges with a documented reference period.
- Evaluation checks: run the candidate and baseline on the same held-out examples. Review task metrics and important slices, including rare but costly errors.
- Integration checks: exercise the feature source, model endpoint, authorization, timeouts, and fallback path together.
- Release checks: confirm the artifact identity, reviewer decision, deployment configuration, rollback target, and monitoring owner.
Make failures actionable. A failed data check should identify the affected field and reference window without exposing raw sensitive records. Decide which checks block training, which block release, and which open an investigation. A warning that nobody owns is unlikely to protect a production system.
Keep evaluation separate from tuning
Reserve a test set that is not used for repeated parameter selection. For time-dependent tasks, choose splits that reflect when predictions will actually be made. Document changes in label definitions and the delay before reliable outcomes become available. Treat evaluation-set revisions as reviewed changes.
Compare with the current production model and a simple baseline. Choose a task metric that reflects the decision: precision and recall at an operating threshold, ranking quality, or error by business-relevant range may be more useful than a single aggregate score. Review important slices even when the overall metric improves.
A release gate can state that the candidate must meet an agreed quality floor, avoid unacceptable regression on critical slices, pass the input contract, and satisfy a measured latency budget. The numbers belong to your application. Preserve the results and the reasoning for exceptions so the next reviewer can understand the decision.
Track experiments and release artifacts
MLflow's experiment-tracking documentation is a useful starting point for organizing run records. Whichever tooling you choose, ensure a reviewer can trace a deployed artifact back to its training inputs, configuration, and evaluation. A dashboard of experiment names is insufficient if the winning run cannot be reproduced or loaded.
Separate experiment permission from production release permission. Keep immutable release artifacts and record which environment uses each version. Decide how credentials, feature-store access, and dependency updates are managed. Test artifact retrieval during recovery; a rollback plan cannot depend on a model that has already been deleted.
Choose a deployment strategy that matches the risk
In a shadow deployment, a candidate receives representative inputs but does not make the user-facing decision. This can reveal integration errors and differences from the incumbent. It cannot by itself prove the business effect of acting on the candidate's outputs, particularly when those actions change future data.
A canary routes a bounded portion of eligible traffic to the candidate. Define routing, comparison windows, stopping conditions, and the person responsible before starting. Check whether traffic groups are comparable and whether rare events require a longer observation period. Do not let a small, unrepresentative sample override offline evidence.
A blue-green release keeps a previous environment available while traffic moves to the new one. Confirm that both versions can consume current features and that stateful dependencies remain compatible. For batch prediction, use a bounded output partition or date range, validate the output, and provide a way to stop or replace the affected batch.
Monitor service health and prediction quality separately
Service monitoring covers errors, latency percentiles, request volume, resource saturation, queue delay, and dependency failures. Data monitoring covers missing inputs, changed categories, freshness, range violations, and distribution changes. Outcome monitoring compares predictions with sufficiently mature labels and tracks the cost of relevant mistakes.
A distribution shift is an investigation signal, not automatic proof that the model is failing. Conversely, a stable input distribution does not guarantee good outcomes. Connect alerts to a documented action: inspect a feature pipeline, pause a batch, route to a fallback, or investigate delayed labels. Assign an owner and escalation path for each production alert.
For example, a hypothetical support-priority classifier might monitor missing account fields immediately, endpoint failures continuously, and priority accuracy only after reviewed ticket outcomes arrive. These are different clocks. Reporting the latest predictions as if their outcomes were already known would make the quality dashboard misleading.
Design rollback before launch
Document the previous model, compatible feature schema, routing mechanism, and person authorized to roll back. Rehearse loading the previous artifact and switching traffic in a non-production environment. Measure the procedure and confirm that monitoring distinguishes the active versions.
Some failures require more than changing the model. If a feature pipeline is corrupt, the previous model may consume the same bad input. Define a safe fallback such as a reviewed rule, a manual queue, or temporarily disabling an automated decision. For irreversible downstream actions, a rollback stops further harm but may require a separate remediation workflow.
When to retrain
Choose retraining triggers from the task: a scheduled refresh, sufficient new reviewed labels, a material data change, or a confirmed quality regression. Check that the new data is valid before starting. Reuse the same evaluation and approval gates after retraining; recent data is not necessarily better data.
Google's Rules of Machine Learning offers engineering guidance for keeping ML systems manageable. Our practical recommendation is to start with an understandable baseline and reliable instrumentation. Automate repeated steps once their input contract and failure behavior are clear.
Additional gates for language-model applications
For an LLM application, version prompts, retrieval configuration, tools, and model settings alongside code. Evaluate representative tasks, unsupported requests, permission boundaries, malformed tool arguments, and missing evidence. A changed prompt or retrieval index can alter behavior even when the model version is unchanged.
Track task completion, evidence correctness where required, human escalation, latency, and cost per completed task. Redact sensitive payloads from traces and document sampling and retention. If an external model changes, rerun your application evaluation rather than assuming a stable provider name guarantees stable behavior.
A practical first release checklist
- The product owner defines the task, prohibited uses, and acceptable failure behavior.
- The release record identifies code, data references, features, artifact, configuration, and evaluation.
- Quality and integration checks run against a documented baseline.
- Deployment has a bounded exposure plan and explicit stopping conditions.
- Alerts have owners, and delayed-label monitoring has a defined review schedule.
- Rollback and fallback are rehearsed with compatible dependencies.
Use this checklist to find gaps, then prioritize the failures that would most affect your users. For implementation context, read our AI development guide and explore application monitoring, LLM inference, and AI consulting. Bring your task, label availability, deployment environment, and current failure modes to an architecture discussion.