A/B Testing

An A/B test splits live prediction traffic between two or more versions of a model, records which one served each prediction, and compares them. Use it to decide, on real traffic, whether a new model should replace the current one. A/B tests cover traditional ML models in the Model Registry.

Why Use A/B Testing?

• Validate a change. Try a retrained or new model on real traffic before it replaces the current one.
• Measure the difference. Compare success rate, latency, or an outcome you score, between variants, with statistical significance.
• Reduce risk. Send a small share of traffic to the new model first.
• Optimize automatically. Let a multi-armed bandit route more traffic to whichever variant earns the best reward.

Concepts

• Variant. One deployed registry model in the test (running, or idle and woken by its first prediction), with an id you choose (for example control or challenger). Only deployed models can be picked: deploy a model in the Model Registry before adding it. One variant is the control: the current model the others are compared with.
• Strategy. How the test picks a variant for each prediction.
• Experiment. A statistical comparison of the control with the other variants on one metric, over the predictions made while the experiment runs.

Routing Strategies

Weighted random

Each prediction goes to a variant at random, in proportion to the variants' weights. Weights are fractions of traffic from 0 to 1, and the enabled variants' weights add up to 1 (for example 0.5 and 0.5, or 0.9 and 0.1).

Best for: a standard A/B test with a controlled traffic split.

Feature-based

Rules pick the variant from the prediction's inputs. Rules are checked in priority order (lower number first), and the first that matches picks its variant; a prediction no rule matches goes to the control. Each rule names a feature, an operator, a value and the variant to route to.

Operator	Matches when the feature
equals / not_equals	equals, or does not equal, the value
in / not_in	is, or is not, one of a list of values
greater_than / less_than	is above, or below, a number
contains	contains the text
regex	matches a regular expression

Add rules under Routing Rules when you create the test: pick the Feature (an input feature of the variant models), the operator, the value and the variant to route to, top to bottom in the order they are checked. A feature-based test needs at least one rule; with none, every prediction would go to the control. The value's shape depends on the operator: a comma-separated list for in / not_in, a number for greater_than / less_than, and a valid regular expression for regex. A rule that cannot be applied (no feature, a value of the wrong shape, a pattern that does not compile, a variant the test does not have) is refused when you create the test, naming the rule and why.

Best for: sending a segment (a region, a customer tier) to a specialized model.

Multi-armed bandit

The test learns which variant does best and sends it more traffic, while still trying the others. It learns from the reward you record for each prediction (see Recording outcomes).

Algorithm	How it chooses
Epsilon-greedy	Usually the variant with the best average reward; with probability epsilon (0.1 by default) a random one. Variants without rewards are tried first.
Thompson sampling	Samples each variant's likely reward and picks the highest sample, so uncertain variants still get tried.
UCB1	Picks the variant with the best reward plus an exploration bonus that shrinks as a variant is used more.
LinUCB	A contextual bandit: it learns, per variant, how the reward depends on the prediction's context features and picks the variant expected to do best for this prediction, plus an exploration bonus (Exploration Bonus (α)).

For LinUCB, choose the Context features: numeric input features that every variant model takes. Each prediction must send each of them as one number, and every variant model needs Record inputs on in its Monitoring settings, because LinUCB learns from each prediction's recorded inputs when its reward arrives. The create page warns about a model with Record inputs off.

Best for: limiting how much traffic a weaker model receives while you learn which is best; LinUCB when the best model depends on the prediction (for example one model for small accounts, another for large ones).

Canary

The target variant (the new model) receives a percentage of traffic, and the control receives the rest. The test starts at Initial % and, once deployed, the platform moves it through its stages (1%, 5%, 10%, 25%, 50% and 100% of traffic by default) on its own:

• A stage is judged only after it has run for Bake minutes and the canary has served enough predictions in it (20 by default).
• The canary passes when its error rate and its p95 latency in the current stage are no worse than the control's by more than Max error Δ vs control and Max p95 Δ vs control (ms). It then advances to the next stage.
• If it fails, it is rolled back to 0% (all traffic to the control) with the reason, and it stays there.

The test page shows the current share, the stages, the health gate, the latest check (advance, wait and why, rollback, or complete) and, after a rollback, its reason.

Best for: a cautious first exposure of a new model version.

Sticky routing

With Sticky routing on, the same entity (a user or account, identified by the entity id sent with each prediction) always gets the same variant for the life of the test, so one customer never flips between models. A prediction without an entity id is refused. With a canary, an entity moves to the new model only as the canary's share grows past it, and back to the control on a rollback. Sticky routing works with every strategy.

Creating an A/B Test

1. Open MLOps > Model Registry and select A/B Testing in the header, then Create A/B Test.
2. Under Basic Information, enter a Name and an optional Description.
3. Choose a Routing Strategy: Weighted Random, Feature-Based, Multi-Armed Bandit or Canary Deployment.
4. Turn on Sticky routing if each entity should keep its variant. For a feature-based test, add at least one Routing Rule. For a bandit, pick the Algorithm and its Epsilon or Exploration Bonus (α), and for LinUCB its Context features. For a canary, set the Initial %, Bake minutes and the health gate (Max error Δ vs control, Max p95 Δ vs control (ms)).
5. Under Model Variants, add at least two variants: a Variant ID, the Model, its Weight (0-1), and mark one as Control.
6. Check the Summary and select Create A/B Test.

A new test is registered: it does not serve traffic until you deploy it.

Deploying and Running a Test

Open the test from the A/B Tests list and select Deploy. The variants' models must be deployed. The test's status moves through:

Status	Meaning
Registered	Created, never deployed
Deploying	Starting to route traffic
Running	Routing predictions to its variants
Stopped	Not routing; its predictions, counters and experiments are kept
Failed	Could not deploy; the Logs tab says why

Stop pauses a running test and Start resumes it. Deleting a test stops it first; its recorded predictions are kept for their normal retention.

Applications send predictions to the test's endpoint (shown on its Overview tab), through the REST API or the Python SDK. The test picks the variant, and the response names the variant that served the prediction and returns a prediction id.

The A/B Test Page

The header shows Total Requests, Success Rate, Avg Latency and Active Variants since the test was created. The tabs:

• Overview. Strategy, dates, the endpoint and the traffic distribution, and how the test routes: sticky routing, the bandit algorithm and its context features, the canary's share, stages, health gate, latest check and any rollback reason, and the routing rules in order.
• Variants. Each variant's model, weight, requests and status. Change a variant's weight (the other enabled variants are rescaled so the weights still add up to 1; weights apply to weighted random tests) or enable and disable a variant. At least two variants stay enabled; a disabled variant receives no traffic.
• Traffic. Requests per variant, average latency and error rate over time.
• Test. Once the test is deployed, send a prediction through it and see which variant served it.
• Predictions. Every prediction the test routed: when, which variant and why, latency, success, its actual and its reward. Filter by variant and search. Details and feedback opens a prediction.
• Logs. Deployment logs.
• Experiments. Statistical comparisons of the variants.

Recording Outcomes

A prediction's details (Details and feedback on the Predictions tab) show its input and output (when the model records them), its actual and its reward, and let you record:

• The actual: the true label for a classifier, the true number for a regression model, or for a model whose predictions are reviewed, a Verdict (is this prediction correct?): Correct or Incorrect. It is recorded with the model's actuals, so drift and performance count it too.
• Reward (0 to 1): how good the outcome was. A multi-armed bandit learns from rewards (LinUCB together with the prediction's recorded inputs), and an experiment on Feedback reward measures them. Recording a reward again replaces it; a bandit counts only the difference.

Applications can record rewards and actuals through the REST API or the Python SDK, keyed by the prediction id the test returned. To record actuals in bulk, import them on the model's Monitoring tab (see Drift Detection).

Experiments

An experiment compares the control with every other variant on one metric, over the predictions the test routes while the experiment runs.

Creating an experiment

On the Experiments tab select New Experiment and set:

Field	Meaning
Name	What you are testing
Primary Metric	Success Rate (higher is better), Latency (ms) (lower is better), Feedback reward (higher is better), or each prediction against its actual, by the variants' models' task: Accuracy (on actuals) for classification and multiclass models (higher is better), Absolute error (on actuals) for regression models (lower is better), Review score (on actuals) for multilabel, time series and other models (the reviewers' verdicts, higher is better). The form offers the one that fits the test's models; a test whose variants have different tasks has none.
Hypothesis	What you expect to happen
Confidence Level	90%, 95% (default) or 99%
Min Detectable Effect	The smallest difference that decides it, relative to the control's mean (default 0.05, 5%). A significant difference smaller than this is inconclusive
Min Samples per Variant	Observations every variant needs before the experiment can decide (default 100); with fewer, the result is inconclusive

A new experiment is a draft. Select Start to begin counting predictions. Starting can require governance approval when a governance solution covers the model.

Analyzing and concluding

While an experiment runs, Run Analysis compares each variant with the control and shows, per variant, the sample size, mean, standard deviation, p-value, whether the difference is significant and the relative improvement, with a recommendation: a variant that wins (significantly better than the control by at least the min detectable effect), the control (every variant significantly worse by at least that much), or inconclusive, with the reason when a setting decided it (for example "control has 15 of the 100 per variant the experiment needs"). The relative improvement is shown as - when the control's mean is 0.

An analysis concludes a running experiment (status completed) when a sequential test shows a significant difference, or when the experiment has run for its maximum duration (83 days, so every prediction it counts is still within the 90-day prediction retention). Through the REST API or Python SDK you can set a shorter maximum duration, or a maximum sample size per variant that concludes it once every variant reaches it. Stop cancels an experiment. A completed experiment without results offers Run Final Analysis.

Access

The owner of a test and administrators can change, deploy, stop and delete it. People it is shared with can view it and its results.

Best Practices

• Decide the metric first. Create the experiment before you judge the variants.
• Start small. Give a new model a small weight or a low canary percentage, then increase it.
• Wait for enough samples. Judge on the experiment's significance, not on early numbers.
• Record outcomes. Rewards drive bandits and reward experiments; actuals measure real accuracy.
• Watch the Traffic tab. Rising errors or latency on a new variant are the first sign of trouble.
• After the test, switch your application to the winning model and stop the test.