Algorithms
A model's drift dashboard has an Algorithms tab that holds all of its drift settings:
| Section | Settings |
|---|---|
| Built-in Algorithms | A switch for each algorithm below |
| Custom Algorithms | A switch for each custom algorithm (shown once one exists) |
| Notifications | Email and its Email recipients, and a Slack incoming webhook (see Notifications) |
| Analysis window | Minimum predictions per analysis (default 30): a window with fewer predictions is not scored |
| Prediction drift | Warning when performance is worse by (%) (default 5) and Alert when performance is worse by (%) (default 10) (see Performance) |
| Automated schedule | Enable scheduled analysis, its Frequency and Analysis window (days) (1 to 90) (see Scheduled analysis) |
Save Configuration saves them. The model's owner and editors change them; a model without a task cannot save them until it has one.
Built-in algorithms
All built-in algorithms run by default, with the thresholds below. The PSI thresholds, which set the drift status, can be changed per model through the REST API or Python SDK.
| Algorithm | What it detects | Default warning / alert |
|---|---|---|
| PSI (Population Stability Index) | Distribution shift of a feature | 0.1 / 0.2 |
| Kolmogorov-Smirnov | Shift in a numerical feature's cumulative distribution | 0.1 / 0.2 (p-value below 0.05) |
| Chi-Square | Shift in a categorical feature's categories | 10 / 20 (p-value below 0.05) |
| Jensen-Shannon | Symmetric difference between distributions | 0.1 / 0.2 |
| Wasserstein | How far a numerical distribution moved | 0.1 / 0.2 |
| Domain classifier | Correlated shifts that single-feature tests miss | 0.1 / 0.2 |
| PCA reconstruction | Structural change in high-dimensional data | 1.5 / 2.0 |
| ADWIN | A change point within the window | 0.5 / 0.5 |
| CBPE | Estimated accuracy drop without actuals | 0.05 / 0.10 |
Custom algorithms
Add your own on the Drift Detection page with Add Algorithm: a name, the feature types it supports (numerical, categorical or both), a description and Python code that defines detect_drift(reference_data, current_data).
reference_dataandcurrent_dataare one feature's values: numbers for a numerical feature, text values for a categorical one. The reference values are drawn from the version's baseline; the current ones are the analysis window's.- It returns
{"score": number, "threshold": number, "status": "ok" | "warning" | "alert", "details": {...}}(detailsis optional). - The code can use numpy, math, random, json and scipy.stats. It cannot read files, reach the network or import other modules.
Test runs the code the way an analysis does, on a sample of each feature type it supports, and shows what it returned for each, or why it returned nothing (for example a syntax error or a result without a status). Last Tested shows when it was last tested and whether it passed. Once added, turn it on per model on the Algorithms tab. In an analysis, a feature the algorithm could not score shows the reason in the Feature Drift tab.