Create and Run a Test
Creating an A/B Test
- Open MLOps > Model Registry and select A/B Testing in the header, then Create A/B Test.
- Under Basic Information, enter a Name and an optional Description.
- Choose a Routing Strategy: Weighted Random, Feature-Based, Multi-Armed Bandit or Canary Deployment.
- Turn on Sticky routing if each entity should keep its variant. For a feature-based test, add at least one Routing Rule. For a bandit, pick the Algorithm and its Epsilon or Exploration Bonus (α), and for LinUCB its Context features. For a canary, set the Initial %, Bake minutes and the health gate (Max error Δ vs control, Max p95 Δ vs control (ms)).
- Under Model Variants, add at least two variants: a Variant ID, the Model, its Weight (0-1), and mark one as Control.
- Check the Summary and select Create A/B Test.
A new test is registered: it does not serve traffic until you deploy it.
Deploying and Running a Test
Open the test from the A/B Tests list and select Deploy. The variants' models must be deployed. The test's status moves through:
| Status | Meaning |
|---|---|
| Registered | Created, never deployed |
| Deploying | Starting to route traffic |
| Running | Routing predictions to its variants |
| Stopped | Not routing; its predictions, counters and experiments are kept |
| Failed | Could not deploy; the Logs tab says why |
Stop pauses a running test and Start resumes it. Deleting a test stops it first; its recorded predictions are kept for their normal retention.
Applications send predictions to the test's endpoint (shown on its Overview tab), through the REST API or the Python SDK. The test picks the variant, and the response names the variant that served the prediction and returns a prediction id.
The A/B Test Page
The header shows Total Requests, Success Rate, Avg Latency and Active Variants since the test was created. The tabs:
- Overview. Strategy, dates, the endpoint and the traffic distribution, and how the test routes: sticky routing, the bandit algorithm and its context features, the canary's share, stages, health gate, latest check and any rollback reason, and the routing rules in order.
- Variants. Each variant's model, weight, requests and status. Change a variant's weight (the other enabled variants are rescaled so the weights still add up to 1; weights apply to weighted random tests) or enable and disable a variant. At least two variants stay enabled; a disabled variant receives no traffic.
- Traffic. Requests per variant, average latency and error rate over time.
- Test. Once the test is deployed, send a prediction through it and see which variant served it.
- Predictions. Every prediction the test routed: when, which variant and why, latency, success, its actual and its reward. Filter by variant and search. Details and feedback opens a prediction.
- Logs. Deployment logs.
- Experiments. Statistical comparisons of the variants.