Routing Strategies
A test's strategy decides which variant serves each prediction. It is chosen when the test is created.
Weighted random
Each prediction goes to a variant at random, in proportion to the variants' weights. Weights are fractions of traffic from 0 to 1, and the enabled variants' weights add up to 1 (for example 0.5 and 0.5, or 0.9 and 0.1).
Best for: a standard A/B test with a controlled traffic split.
Feature-based
Rules pick the variant from the prediction's inputs. Rules are checked in priority order (lower number first), and the first that matches picks its variant; a prediction no rule matches goes to the control. Each rule names a feature, an operator, a value and the variant to route to.
| Operator | Matches when the feature |
|---|---|
equals / not_equals | equals, or does not equal, the value |
in / not_in | is, or is not, one of a list of values |
greater_than / less_than | is above, or below, a number |
contains | contains the text |
regex | matches a regular expression |
Add rules under Routing Rules when you create the test: pick the Feature (an input feature of the variant models), the operator, the value and the variant to route to, top to bottom in the order they are checked. A feature-based test needs at least one rule; with none, every prediction would go to the control. The value's shape depends on the operator: a comma-separated list for in / not_in, a number for greater_than / less_than, and a valid regular expression for regex. A rule that cannot be applied (no feature, a value of the wrong shape, a pattern that does not compile, a variant the test does not have) is refused when you create the test, naming the rule and why.
Best for: sending a segment (a region, a customer tier) to a specialized model.
Multi-armed bandit
The test learns which variant does best and sends it more traffic, while still trying the others. It learns from the reward you record for each prediction (see Recording outcomes).
| Algorithm | How it chooses |
|---|---|
| Epsilon-greedy | Usually the variant with the best average reward; with probability epsilon (0.1 by default) a random one. Variants without rewards are tried first. |
| Thompson sampling | Samples each variant's likely reward and picks the highest sample, so uncertain variants still get tried. |
| UCB1 | Picks the variant with the best reward plus an exploration bonus that shrinks as a variant is used more. |
| LinUCB | A contextual bandit: it learns, per variant, how the reward depends on the prediction's context features and picks the variant expected to do best for this prediction, plus an exploration bonus (Exploration Bonus (α)). |
For LinUCB, choose the Context features: numeric input features that every variant model takes. Each prediction must send each of them as one number, and every variant model needs Record inputs on in its Monitoring settings, because LinUCB learns from each prediction's recorded inputs when its reward arrives. The create page warns about a model with Record inputs off.
Best for: limiting how much traffic a weaker model receives while you learn which is best; LinUCB when the best model depends on the prediction (for example one model for small accounts, another for large ones).
Canary
The target variant (the new model) receives a percentage of traffic, and the control receives the rest. The test starts at Initial % and, once deployed, the platform moves it through its stages (1%, 5%, 10%, 25%, 50% and 100% of traffic by default) on its own:
- A stage is judged only after it has run for Bake minutes and the canary has served enough predictions in it (20 by default).
- The canary passes when its error rate and its p95 latency in the current stage are no worse than the control's by more than Max error Δ vs control and Max p95 Δ vs control (ms). It then advances to the next stage.
- If it fails, it is rolled back to 0% (all traffic to the control) with the reason, and it stays there.
The test page shows the current share, the stages, the health gate, the latest check (advance, wait and why, rollback, or complete) and, after a rollback, its reason.
Best for: a cautious first exposure of a new model version.
Sticky routing
With Sticky routing on, the same entity (a user or account, identified by the entity id sent with each prediction) always gets the same variant for the life of the test, so one customer never flips between models. A prediction without an entity id is refused. With a canary, an entity moves to the new model only as the canary's share grows past it, and back to the control on a rollback. Sticky routing works with every strategy.