Skip to main content

MLOps

MLOps takes traditional ML models from data to production and keeps them healthy there. You can:

  • define features once and serve them for training and inference;
  • train models with AutoML or your own code;
  • track every training run;
  • register, version and deploy models;
  • compare models on live traffic with A/B tests;
  • detect when production data or accuracy drifts from what a model was trained on.

MLOps is under MLOps in the sidebar, with four pages: Feature Store, Model Registry, AutoML and Experiments. Drift Detection and A/B Testing open from the Model Registry header.

How the pieces fit​

  1. Features. Define entities, feature views and feature services in the Feature Store. It keeps their history in Postgres and their latest values in Redis.
  2. Training.
    • AutoML trains and tunes many models on a volume file, a data source, an add-on or a feature service, and keeps the best one.
    • Or train in your own code in a workspace and track it in Experiments.
  3. Registry. Publish the AutoML model, register the run's model, or upload one trained elsewhere. The Model Registry keeps its versions and serves it always on, on demand or on a schedule, with autoscaling and optionally on spot capacity.
  4. Predictions. Apps, workflows, workspaces, the REST API and the Python SDK call the model. Every prediction is recorded with the version that made it.
  5. Comparison. An A/B test splits live traffic between models and decides the winner in a statistical experiment. It can split traffic in four ways:
    • weighted random;
    • feature-based;
    • multi-armed bandit;
    • canary.
  6. Drift. Drift Detection compares each version's production predictions with its baseline, feature by feature. With actuals, it also measures the model's real performance and warns or alerts when it drifts.

Sections​

SectionWhat it covers
Feature StoreEntities, batch, streaming and on-demand views, feature services, materialization, online serving, point-in-time training data, models bound to features
AutoMLCreating a job from any source, every training and advanced setting, results, publishing
ExperimentsTracking runs from Python, autolog, comparing runs, sharing, registering a run's model
Model RegistryRegistering and uploading models, the model page, deploying and scaling, versions, monitoring settings
A/B TestingRouting strategies, creating and running a test, recording outcomes, experiments
Drift DetectionBaselines, actuals, running and scheduling analysis, reading results, algorithms

Getting started​

Train and deploy a model​

  1. Open MLOps > AutoML and select New AutoML Job.
  2. Pick a dataset and the column to predict, and start training.
  3. When the job completes, select Publish to Registry. The model is registered and deployed, and its drift baseline is built.
  4. Call it from its Test tab, the REST API, the Python SDK, a workflow or an app.

Monitor a model for drift​

  1. Register and deploy the model in the Model Registry.
  2. On its Monitoring tab, check its Task and select Build Baseline with the CSV the version was trained or validated on. Turn on Analyze drift daily.
  3. Use the model. Every prediction is recorded.
  4. Import the true outcomes with Import Actuals as they become known.
  5. Read the results on the Monitoring tab and the drift dashboard.

Compare two models​

  1. Register both models and deploy them.
  2. Select A/B Testing in the Model Registry header and create a test with both models as variants. Make the current model the control.
  3. Deploy the test and send your application's predictions through it.
  4. Create and start an experiment on the metric you care about, and run its analysis.
  5. When a variant wins, switch your application to it and stop the test.