Drift Detection

Drift detection tells you when the data a model sees in production no longer looks like the data it was trained on, and, once you record the true outcomes of its predictions, whether it is still as accurate. It covers traditional ML models in the Model Registry.

Each analysis compares the production predictions of a model's active version with that version's baseline: a summary of the rows it was trained or validated on.

Why Drift Detection Matters

A model learns from historical data, but the world keeps changing. Customer behaviour shifts, a data source changes its format, a season turns. When production inputs drift away from the training data, predictions degrade, often silently.

Type	What changes	How it is detected
Data drift	The distribution of an input feature	Per-feature tests such as PSI, Kolmogorov-Smirnov and Chi-Square
Multivariate drift	How features move together, even when each one alone looks stable	Domain classifier, PCA reconstruction error
Prediction drift	The model's performance against the true outcomes, by its task	Accuracy (classifiers), mean absolute error (regressors) or the reviewers' mean verdict (multilabel, time series and other models) on predictions that have actuals, against the baseline's
Estimated performance	Likely accuracy before the true outcomes arrive	CBPE, from the model's confidence

Where to Find It

• The model's Monitoring tab. Open a model from MLOps > Model Registry and select Monitoring. It holds the model's monitoring settings, its latest drift status, its predictions, actuals imports, performance and baselines, with Build Baseline, Import Actuals, Run Analysis and Full Dashboard.
• The Drift Detection page. Select Drift Detection in the Model Registry header. It lists every model you can see with its drift status, drift score, sample size and when it was last analyzed. Filter by status (Healthy, Warning, Alert, Could not analyze, No data) or search by name. A model without a drift result says what it still needs: No task (set its task in Monitoring settings), Record inputs is off, No baseline for its active version, or Not analyzed yet (or why its latest analysis could not run). View Drift Details opens a model's drift dashboard.
• A model's drift dashboard has these tabs: Overview, Feature Drift, Predictions, History, Baselines and Algorithms.

Predictions Are Recorded Automatically

Every prediction a registry model makes is recorded once, whether it comes from the model page, the REST API, the Python SDK, a workflow, an app or an A/B test. Each record keeps the model version that made it, its prediction and score, how long it took, the entity id you sent (if any) and a prediction id you can attach an actual to later.

• Predictions made on the model page's Test tab are marked Test and never count in drift or performance results.
• Predictions routed through an A/B test also record the variant that served them.
• A model served outside the platform can send its predictions through the REST API or SDK so drift covers them too.
• Predictions are kept for 90 days, then move to archive storage.

The model's task

Drift scores a model's predictions by its task, so a model needs one:

Task	Scored by	Actuals are
Classification, Multiclass	Accuracy: the share of predictions whose label is the true one	The true label
Regression	Mean absolute error: the mean distance between predicted and true numbers (lower is better)	The true number
Multilabel, Time series, Other	Review score: the mean of the reviewers' verdicts	A reviewer's verdict on the prediction: true or false (correct/incorrect, yes/no), or a quality score from 0 to 1

A model registered without a task shows a warning on its drift dashboard and its drift settings cannot be saved until it has one. Set it under Task in Monitoring settings on the model's Monitoring tab (see The model's task).

Record inputs and outputs

In Monitoring settings on the Monitoring tab, Record inputs and outputs decides whether each prediction's inputs and the model's whole output are stored with it. It is on by default.

Turn it off for models whose data is sensitive, such as personal data or document text, or a model whose output holds what it found (for example a PII model). Predictions are still recorded (version, prediction and score, latency), but input drift cannot be measured without inputs. The change applies from the model's next prediction.

Baselines

A baseline summarizes a model version's reference rows: the data it was trained or validated on. Drift compares the version's production predictions with its own baseline.

Building a baseline

1. On the model's Monitoring tab, select Build Baseline.
2. Choose the Version the rows describe (the active version is selected).
3. Upload a CSV, or pick one from a shared volume. It can be any size. It needs a column for every input feature the model declares. Optionally add:
   • actual and prediction to record the version's performance on its reference rows (accuracy, or mean absolute error for a regressor). For a multilabel, time series or other model, actual alone: each row's reviewed verdict on the version's output (true/false or a 0 to 1 score);
   • confidence (the probability of the predicted class) as well, so the platform can estimate how often the version is right on live traffic before actuals arrive. A regressor has no confidence calibration: its performance is its error, measured from actuals.
4. Optionally turn on Analyze drift daily once the baseline is built. It turns on the model's daily drift schedule; you can change it any time in the drift settings.
5. Select Build Baseline. Its progress shows live; you can close the dialog and it keeps running.

When the baseline is built it becomes the version's active baseline, and the version's first drift analysis starts, over the last 7 days of production predictions. If the version has no production predictions in that time yet, the job says so, and the first analysis runs once it has some (with Run Analysis, or on the drift schedule).

Models published from AutoML get a baseline automatically, built from the reference rows their training job saved, and their drift is analyzed daily from then on: the first day the model has production predictions, it gets a result without anyone starting one. A baseline needs at least the model's minimum baseline rows (30 unless its drift settings say otherwise) for drift to be measured against it. A model trained on a small set may get a smaller baseline from its few held-back rows: the model's Monitoring tab then says how many rows the baseline has and how many drift needs, drift is not analyzed or scheduled, and Run Analysis is not offered until you build a baseline from more rows.

Models served outside the platform

A model registered without versions (served elsewhere, with its predictions sent to the platform) has one baseline for the whole model. Build Baseline then has no version to choose, and drift compares every prediction sent for the model with that baseline.

Managing baselines

A version keeps one active baseline. When you build a new one, the one it replaces is kept, inactive. The Baselines tab of the drift dashboard lists every baseline with its version, number of features and rows, its performance on its reference rows (by its task) and the file it was built from. Activate makes an earlier baseline its version's active one again; drift compares with it from the next analysis on.

A version without a baseline cannot be analyzed. The Monitoring tab says so and offers Build Baseline.

Actuals

Actuals are the true outcomes of predictions (the customer churned, the claim was fraud), usually known some time after the prediction. With actuals, drift measures the version's real performance, by its task (see The model's task).

Importing actuals

1. On the model's Monitoring tab or drift dashboard, select Import Actuals.
2. Upload a CSV (any size) or pick one from a shared volume, with:
   • a predictionId column (the id a prediction returned) or an entityId column (your id sent with the prediction),
   • an actual column (the true outcome),
   • optionally actualAt (when the outcome happened).
3. Select Import. The import shows the rows read, recorded, replaced and refused as it runs, and lists every refused row with its line in the file and the reason.

Importing an actual again for the same key replaces it, so correcting a file and importing it again is safe. The Actuals Imports list on the Monitoring tab keeps every import.

You can also record an actual for a single prediction an A/B test routed, from that prediction's details (see A/B Testing).

How actuals match predictions

• An actual with a prediction id labels that prediction.
• An actual with an entity id labels that entity's predictions made within the model's Actuals window before the outcome (its actualAt, else when it was imported).

Either may arrive first. Set the Actuals window (days) in Monitoring settings: 1 to 90 days, 14 by default. In the Predictions list each prediction shows its actual, marked Correct or Incorrect, or Pending until one arrives.

Running an Analysis

Select Run Analysis on the Monitoring tab or the drift dashboard. The analysis compares the active version's production predictions from the last 7 days with its active baseline.

While it runs, the page shows Analyzing with what it is doing. If it cannot produce a result, it says why; for example, the version has no production predictions in the window, or its inputs are not recorded. The drift list shows these models as Could not run with the reason.

An analysis window needs enough predictions to mean anything: with a handful of values, every feature reads as drifted. A window with fewer predictions (with recorded inputs) than the model's Minimum predictions per analysis (30 by default) is not scored, and the analysis says so.

Scheduled analysis

On the drift dashboard's Algorithms tab, under Automated schedule:

• Enable scheduled analysis;
• choose the Frequency: hourly, daily or weekly;
• set the Analysis window (days) each run looks back over;
• select Save Configuration.

The first scheduled run happens shortly after you enable it. Building a baseline with Analyze drift daily turns on a daily schedule the same way; the baseline's own first analysis counts as that day's run. If the schedule cannot start an analysis (for example, the version has no baseline), the Algorithms tab says why.

Notifications

When an analysis finds drift (status Warning or Alert), the model's owner is notified in Strongly (the notification bell and push). On the drift dashboard's Algorithms tab, under Notifications, you can also:

• turn on Email to email the Email recipients (separated by commas; leave empty to email the model's owner);
• set a Slack incoming webhook to post a message to a Slack channel.

Select Save Configuration. An address that is not an email address, or a webhook pointing at localhost, an internal host or a private network address, is refused. Each result is announced once, and an analysis that could not run sends no notification (the dashboard says why it could not run). Email needs the platform's mail server to be configured; when it is not, the result records that the email could not be sent.

Reading the Results

Drift status

Status	Meaning
OK (No Drift)	The production data matches the baseline.
Warning	Some drift. Investigate the features driving it.
Alert (Drift Detected)	Significant drift. Consider retraining.
Could not analyze	No algorithm could score the window; each row says why.

Drift score

The drift score is the mean PSI (Population Stability Index) across the model's features: a plain number, not a percentage. The drift status comes from it: by default a score of 0.1 or more is a warning and 0.2 or more an alert (the PSI thresholds can be changed through the REST API or Python SDK). When PSI scores no feature, there is no drift score and the status is the worst any other algorithm gave. When a window holds few predictions, neighboring value ranges are compared together, so chance differences between two small samples do not read as drift; the same window always gives the same score.

Overview

The Overview tab shows the drift score, the analysis window, the number of predictions analyzed, how many features drifted out of those analyzed, and the most drifted features. Algorithms that were skipped are listed with why. For a result that found drift, Notified shows how it was announced: the owner in Strongly, email (to whom) and Slack, or why a channel was not sent.

Feature Drift

The Feature Drift tab lists one row per feature, most drifted first, with every algorithm's result on it: its value, threshold and status. Algorithms about the dataset as a whole are rows of their own, named like [multivariate] domain_classifier or [performance] cbpe. Search by feature name, or sort by status, score or name. A row where nothing could be scored says why.

Performance

When predictions in the window have actuals, Performance Tracking on the Monitoring tab and Prediction Drift on the drift dashboard's Overview show the version's performance on them (accuracy, mean absolute error or review score, by its task) against its performance on its reference rows, the change, and how much better or worse that is than the baseline. Until actuals arrive it says so. With a baseline built from rows that include actual, prediction and confidence, CBPE estimates the accuracy drop of a classifier or reviewed model before actuals arrive; a regressor skips CBPE, saying why.

Prediction drift raises the analysis's status: under Prediction drift on the drift dashboard's Configuration tab, Warning when performance is worse by (%) (default 5) and Alert when performance is worse by (%) (default 10) set how much worse than the baseline, relatively, its performance may get. Lower accuracy or review score is worse, as is a higher mean absolute error. A warning or alert makes the whole analysis one, so it is announced like feature drift.

History

The History tab lists every analysis with its status, drift score, sample size and duration.

Algorithms

Choose which algorithms run for a model on the drift dashboard's Algorithms tab. All built-in algorithms run by default.

Algorithm	What it detects	Default warning / alert
PSI (Population Stability Index)	Distribution shift of a feature	0.1 / 0.2
Kolmogorov-Smirnov	Shift in a numerical feature's cumulative distribution	0.1 / 0.2 (p-value below 0.05)
Chi-Square	Shift in a categorical feature's categories	10 / 20 (p-value below 0.05)
Jensen-Shannon	Symmetric difference between distributions	0.1 / 0.2
Wasserstein	How far a numerical distribution moved	0.1 / 0.2
Domain classifier	Correlated shifts that single-feature tests miss	0.1 / 0.2
PCA reconstruction	Structural change in high-dimensional data	1.5 / 2.0
ADWIN	A change point within the window	0.5 / 0.5
CBPE	Estimated accuracy drop without actuals	0.05 / 0.10

Custom algorithms

Add your own on the Drift Detection page with Add Algorithm: a name, the feature types it supports (numerical, categorical or both), a description and Python code that defines detect_drift(reference_data, current_data).

• reference_data and current_data are one feature's values: numbers for a numerical feature, text values for a categorical one. The reference values are drawn from the version's baseline; the current ones are the analysis window's.
• It returns {"score": number, "threshold": number, "status": "ok" | "warning" | "alert", "details": {...}} (details is optional).
• The code can use numpy, math, random, json and scipy.stats. It cannot read files, reach the network or import other modules.

Test runs the code the way an analysis does, on a sample of each feature type it supports, and shows what it returned for each, or why it returned nothing (for example a syntax error or a result without a status). Last Tested shows when it was last tested and whether it passed. Once added, turn it on per model on the Algorithms tab. In an analysis, a feature the algorithm could not score shows the reason in the Feature Drift tab.

Interpreting Common Scenarios

Seasonal drift. Drift appears at regular intervals. This may be expected rather than a problem; consider season-aware models.

Sudden spike. Drift jumps in a short time. Look for recent changes: a new data source, a pipeline bug, an upstream system change.

Gradual increase. Drift slowly rises over weeks. Plan regular retraining, and build a new baseline for the retrained version.

Single feature. One feature drifts while others are stable. Check that feature's source and processing.

High drift, stable accuracy. Inputs shifted but the model still performs. Keep watching; if this is the new normal, retrain and build a new baseline.

Best Practices

• Build baselines when you train. The reference rows are easiest to capture then. Include actual, prediction and confidence (or, for a reviewed model, actual verdicts and confidence) so performance and estimated accuracy are available.
• Schedule analysis. Daily or weekly analysis catches gradual drift a one-off check misses.
• Record actuals. Data drift says inputs changed; actuals say whether it matters.
• Keep test traffic on the Test tab. It never skews production results.
• Retrain and compare safely. When drift is confirmed, retrain, register the new version with its baseline, and compare it with the current version in an A/B test before switching.