Monitoring a Model

Every prediction a registry model makes is recorded, and the model's Monitoring tab shows how it is doing in production. This page covers the tab and the model's monitoring settings; Drift Detection covers baselines, actuals, analysis and results in full.

Predictions are recorded automatically

Every prediction a registry model makes is recorded once, whether it comes from the model page, the REST API, the Python SDK, a workflow, an app or an A/B test. Each record keeps:

• the model version that made it;
• its prediction and score;
• how long it took;
• the entity id you sent, if any;
• a prediction id you can attach an actual to later;
• with Record inputs and outputs on, its inputs and the model's whole output.

Predictions made on the model page's Test tab are marked Test and never count in drift or performance results. Predictions routed through an A/B test also record the variant that served them. A model served outside the platform can send its predictions through the REST API or SDK, so drift covers them too. Predictions are kept for 90 days, then move to archive storage.

Monitoring settings

On the Monitoring tab, Monitoring settings holds:

• Task: Classification, Regression, Multiclass, Multilabel, Time series or Other (see The model's task).
• Record inputs and outputs: on by default. Turn it off for models whose data is sensitive (personal data, document text, a model whose output holds what it found, such as a PII model). Predictions are still recorded (version, prediction and score, latency), but input drift cannot be measured without inputs. The change applies from the model's next prediction.
• Actuals window (days): 1 to 90, default 14. How far before an outcome a prediction for the same entity can be, for an actual imported by entity id to label it (see Actuals).

Changes are saved at once by the model's owner and editors.

The model's task

Drift and actuals evaluate a model by its task, shown as Task on the Overview tab:

Task	Scored by	Actuals are
Classification, Multiclass	Accuracy: the share of predictions whose label is the true one	The true label
Regression	Mean absolute error (lower is better)	The true number
Multilabel, Time series, Other	Review score: the mean of the reviewers' verdicts	A reviewer's verdict: true/false (correct/incorrect, yes/no), or a quality score from 0 to 1

A model gets its task when it is registered: AutoML records it, an uploaded file's Task or a bundle's manifest (problem_type) declares it, and a run's model takes the one log_model recorded. A model without one shows Not set: drift cannot be measured for it, its drift settings cannot be saved, and the Drift Detection page says it has no task. Its owner sets it under Task in Monitoring settings; it is saved at once and kept with the active version.

From Python: client.model_registry.update(model_id, problem_type="regression").

The Monitoring tab

• Drift: the latest analysis's Drift Status (No Drift, Warning, Drift Detected, Could Not Analyze), Drift Score (mean PSI), Features Analyzed and when it was Last Analyzed, or what the model still needs before it can be analyzed.
• Predictions: the model's recorded predictions, newest first, each with its time, entity id, version, Source (Production or Test tab; filter by it), prediction, actual (Correct, Incorrect or Pending) and latency. A failed prediction reads Failed, with the error on hover.
• Actuals Imports: every import of actuals with its counts.
• Performance Tracking: the active version's performance on predictions with actuals, against its baseline's.
• Baselines: the active version's baseline: when it was Built from which file, its Reference rows, its Performance on its reference rows, whether it Estimates accuracy before actuals (the rows had a confidence column) and how long it has been Active for.

Its actions:

Action	Does
Build Baseline	Summarizes a version's reference rows so drift can be measured (see Baselines)
Import Actuals	Records the true outcomes of predictions (see Actuals)
Run Analysis	Compares the active version's recent production predictions with its baseline (see Running an analysis)
Full Dashboard	Opens the model's drift dashboard (see Reading results)