A/B Tests
Create, deploy and analyze A/B tests between registry models. A test splits prediction traffic across variants (each serving one registry model) by its strategy: weighted random, feature based, multi-armed bandit or canary. Every prediction it routes is recorded with the variant that served it, and experiments compare the variants statistically.
All endpoints require authentication via the X-API-Key header and the scope shown. List endpoints are paged (see Pagination) and return the items in data with total, limit, cursor, hasMore and requestId in meta.
A/B Test Object
{
"_id": "ab_abc123",
"name": "Churn v1 vs v2",
"description": "Compare the retrained churn model with the current one",
"strategy": "weighted_random",
"variants": [
{
"variantId": "control",
"modelId": "model_churn_v1",
"weight": 0.5,
"isControl": true,
"isEnabled": true,
"metrics": { "requestCount": 1200, "successCount": 1195, "errorCount": 5, "totalLatencyMs": 14300 }
},
{
"variantId": "challenger",
"modelId": "model_churn_v2",
"weight": 0.5,
"isControl": false,
"isEnabled": true,
"metrics": { "requestCount": 1180, "successCount": 1178, "errorCount": 2, "totalLatencyMs": 12100 }
}
],
"stickyRouting": false,
"featureRules": [],
"deployment": { "status": "running", "endpoint": "/api/v1/mlops/ab-tests/ab_abc123/predict", "deployedAt": "2026-09-20T10:00:00Z" },
"metrics": { "totalRequests": 2380, "totalSuccesses": 2373, "totalErrors": 7, "totalLatencyMs": 26400 },
"tags": ["churn"],
"workspaceId": "ws_001",
"organizationId": "org_xyz",
"createdAt": "2026-09-19T15:00:00Z",
"updatedAt": "2026-09-27T09:12:00Z"
}
deployment.status is registered (created, never deployed), deploying, running, stopped or failed.
A multi_armed_bandit test also has banditConfig: algorithm, epsilon, explorationBonus, rewardMetric, each variant's reward statistics in armStats (pulls, totalReward, avgReward, by variant id), and for linucb its contextFeatures and each variant's linear model in linucbState (A, b).
A canary test has canaryConfig: controlVariantId, targetVariantId, stages, currentPercentage (the canary's current share of traffic), targetPercentage, the health gate (bakeMinutes, errorRateDelta, latencyP95DeltaMs, minRequestsForDecision), and the platform's state as it moves the canary: lastAdvancedAt, lastRollbackAt, rollbackReason and lastCheck ({ at, action: "advance" | "rollback" | "wait" | "complete", reason }).
GET /api/v1/mlops/ab-tests
The A/B tests you can see.
Scope: mlops:read
| Parameter | Type | Required | Description |
|---|---|---|---|
status | string | No | registered, deploying, running, stopped or failed |
strategy | string | No | weighted_random, feature_based, multi_armed_bandit or canary |
modelId | string | No | Only tests with a variant serving this registry model |
workspaceId | string | No | Only tests in this workspace |
q | string | No | Name and description |
sort | string | No | -updatedAt (default), createdAt, name, strategy, -metrics.totalRequests |
POST /api/v1/mlops/ab-tests
Create an A/B test. It is registered until deployed.
Scope: mlops:write
{
"name": "Churn v1 vs v2",
"strategy": "weighted_random",
"variants": [
{ "variantId": "control", "modelId": "model_churn_v1", "weight": 0.5, "isControl": true },
{ "variantId": "challenger", "modelId": "model_churn_v2", "weight": 0.5 }
]
}
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Test name |
strategy | string | Yes | weighted_random, feature_based, multi_armed_bandit or canary |
variants | array | Yes | At least 2 { variantId, modelId, weight?, isControl? }, variant ids unique, at most one control (the first is the control when none is marked). weight is a 0-1 fraction (default an equal share); for weighted_random the weights must add up to 1. Each modelId must be a registry model you can use (yours, shared with you, or public): another id is refused with 403, naming it. |
description, tags, workspaceId | No | Metadata | |
stickyRouting | boolean | No | Keep an entity on one variant for the test's life, keyed on the entityId sent with each prediction (a prediction without one is then refused). A canary moves an entity to the target only as its share grows, and back on a rollback. Default false. |
featureRules | array | For feature_based | At least one { featureName, operator, value, targetVariantId, priority? }. Operators: equals, not_equals, in, not_in, greater_than, less_than, contains, regex. value is a list for in / not_in, a number for greater_than / less_than, a valid pattern for regex. Lower priority is checked first (default the order given); a prediction no rule matches goes to the control. A rule that cannot be applied is refused with 400, naming the rule. |
banditConfig | object | No | For multi_armed_bandit: { algorithm: "epsilon_greedy" (default) | "thompson_sampling" | "ucb1" | "linucb", epsilon (0-1, default 0.1), explorationBonus (default 2.0; alpha for linucb), contextFeatures, rewardMetric }. linucb routes by contextFeatures: numeric input features every variant model declares as numbers; every variant model needs Record inputs on (it learns from each prediction's recorded inputs). |
canaryConfig | object | No | For canary: { controlVariantId, targetVariantId (default the second variant), stages (default 1, 5, 10, 25, 50, 100), currentPercentage (default the first stage), targetPercentage (default 100), bakeMinutes (default 30), errorRateDelta (default 0.01), latencyP95DeltaMs (default 500), minRequestsForDecision (default 20) }. Once deployed, the platform advances the canary to its next stage after bakeMinutes and minRequestsForDecision canary predictions in the stage, while its error rate and p95 latency in the stage stay within the deltas of the control's; otherwise it rolls it back to 0% with the reason. |
Response 201 Created
The new A/B test, status registered.
GET /api/v1/mlops/ab-tests/:id
An A/B test.
Scope: mlops:read
PATCH /api/v1/mlops/ab-tests/:id
Change a test's name, description or tags. Scope: mlops:write
Response 200 OK
The updated A/B test.
DELETE /api/v1/mlops/ab-tests/:id
Delete a test (stopped first when running). Its recorded predictions are kept for their retention period. Scope: mlops:write. Response: 204
POST /api/v1/mlops/ab-tests/:id/deploy
Start routing predictions to the variants' models (status running). The variants' models must be deployed. Scope: mlops:write
POST /api/v1/mlops/ab-tests/:id/stop
Stop routing (status stopped); predictions, counters and experiments are kept. Scope: mlops:write
POST /api/v1/mlops/ab-tests/:id/start
Start a stopped test again (re-deploys it). Scope: mlops:write
PATCH /api/v1/mlops/ab-tests/:testId/variants/:id
Update one variant and get it back. Send either field or both (enabling applies first, so one call can enable a variant and give it its weight):
weight: an enabled variant's share of traffic in aweighted_randomtest, a 0-1 fraction (not a percentage). The other enabled variants are rescaled so the weights still add up to 1.enabled:falseto stop routing traffic to the variant,trueto route to it again. At least 2 stay enabled.
Scope: mlops:write | Body: { "enabled": true, "weight": 0.2 }
POST /api/v1/mlops/ab-tests/:id/predict
Predict through a running test. The test picks the variant.
Scope: mlops:write
{ "inputData": { "age": 42, "plan": "pro" }, "entityId": "customer_456" }
entityId is the user or entity the prediction is for. A test with stickyRouting requires it (400 without it) and keeps the entity on one variant. A linucb test needs each of its contextFeatures in inputData as one number (400 otherwise).
Response 200 OK: data is the serving model's output, plus predictionId and the variantId that served it. Use predictionId to record the outcome.
GET /api/v1/mlops/ab-tests/:id/metrics
The test's totals since it was created: totalRequests, successCount, errorCount, successRate and avgLatencyMs (serving latency of successful predictions), and the same per variant in variantMetrics. Rates are null before any request.
Scope: mlops:read
GET /api/v1/mlops/ab-tests/:id/traffic
Traffic over time from the test's recorded predictions: bucketMs, buckets (the ISO start of each bucket) and, per variant, requests, errors and avgLatencyMs per bucket.
Scope: mlops:read
GET /api/v1/mlops/ab-tests/:id/predictions
The predictions the test routed (see the prediction record): the variant that served each and why, latency, success, its actual and its reward.
Scope: mlops:read
| Parameter | Type | Required | Description |
|---|---|---|---|
variantId | string | No | One variant's predictions |
q | string | No | Prediction id, entity id, variant, routing reason or error, or the prediction exactly |
sort | string | No | -timestamp (default), latencyMs, variantId, success, confidence |
POST /api/v1/mlops/ab-tests/-/predictions/:id/feedback
Record the outcome of a prediction the test routed. Send reward, label or both.
Scope: mlops:write
| Field | Type | Description |
|---|---|---|
reward | number | 0 to 1, kept on the prediction (sending it again replaces it, and a bandit counts only the difference). A multi_armed_bandit test learns from it (linucb over the prediction's recorded inputs, refused when the model does not record them), and an experiment on custom measures it. |
label | string | The prediction's actual (its true outcome), recorded with the model's actuals |
Response 200 OK: the prediction.
Experiments
An experiment compares each treatment variant with the control on one metric, over the predictions the test routes while it runs. Its status is draft, running, completed or cancelled.
POST /api/v1/mlops/ab-tests/:id/experiments
Scope: mlops:write
{
"name": "v2 succeeds more often",
"controlVariantId": "control",
"treatmentVariantIds": ["challenger"],
"primaryMetric": "success_rate",
"confidenceLevel": 0.95
}
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Experiment name |
controlVariantId | string | Yes | The control variant |
treatmentVariantIds | array | Yes | At least one, none equal to the control |
primaryMetric | string | Yes | success_rate (higher is better), accuracy (each prediction against its actual: a label sent as feedback or the model's actuals; higher is better), latency (lower is better) or custom (the recorded reward, higher is better) |
description, hypothesis | string | No | What you are testing and expect |
confidenceLevel | number | No | Between 0 and 1 (default 0.95) |
minimumDetectableEffect | number | No | Between 0 and 1 (default 0.05): the smallest difference, relative to the control's mean, that decides the experiment |
minSamplePerVariant | number | No | Observations every variant needs before the experiment can decide (default 100) |
maxSamplePerVariant | number | No | Concludes the experiment once every variant has this many samples |
maxDurationDays | number | No | Concludes it after this many days (default and maximum 83, so every prediction it counts is within the 90-day prediction retention) |
Response 201 Created
The new experiment, status draft: its _id, name, routerId (the A/B test), config and createdAt, as listed.
GET /api/v1/mlops/ab-tests/:id/experiments
The test's experiments, newest first, with their config, status, startedAt, endedAt, concludedReason (sequential_significant, max_samples, max_duration, manual) and latest results.
Scope: mlops:read | Sort: -createdAt (default), name, status, startedAt
POST /api/v1/mlops/ab-tests/-/experiments/:id/start
Start a draft experiment: it counts the predictions from now until it ends. Starting can require governance approval (403 governance-blocked says what is missing). Scope: mlops:write
Response 200 OK: the experiment, status running.
POST /api/v1/mlops/ab-tests/-/experiments/:id/analyze
Analyze the experiment: each treatment against the control on the primary metric.
Scope: mlops:read
Response 200 OK
{
"data": {
"controlStats": { "sampleSize": 1195, "mean": 0.9958, "stdDev": 0.064, "ci95Lower": 0.992, "ci95Upper": 0.9994 },
"treatmentStats": {
"challenger": {
"sampleSize": 1178, "mean": 0.9983, "stdDev": 0.041, "ci95Lower": 0.996, "ci95Upper": 1.0,
"pValue": 0.21, "isSignificant": false, "relativeImprovement": 0.0025
}
},
"recommendation": "inconclusive",
"winnerVariantId": null,
"reason": null,
"metric": "success_rate",
"higherIsBetter": true
},
"meta": { "requestId": "req_abc123" }
}
recommendation is treatment (with winnerVariantId: the treatment significantly better than the control by at least minimumDetectableEffect, by the most), control (every treatment significantly worse by at least that much), or inconclusive. reason says why when the settings decided it: a variant with fewer than minSamplePerVariant observations, or a significant difference below minimumDetectableEffect. relativeImprovement is null when the control's mean is 0. A running experiment is concluded (status completed) when a sequential test is significant, maxSamplePerVariant is reached, or maxDurationDays has passed.
POST /api/v1/mlops/ab-tests/-/experiments/:id/stop
Cancel a draft or running experiment (status cancelled). Scope: mlops:write
Response 200 OK: the experiment, status cancelled.
POST /api/v1/mlops/ab-tests/-/experiments/:id/conclude
End a running experiment as a finished run (status completed, concludedReason manual). Scope: mlops:write
Response 200 OK: the experiment, status completed.
DELETE /api/v1/mlops/ab-tests/-/experiments/:id
Delete an experiment and its results; the test's predictions are kept. Scope: mlops:write