Experiments
An experiment compares the control with every other variant on one metric, over the predictions the test routes while the experiment runs.
Creating an experiment
On the Experiments tab select New Experiment and set:
| Field | Meaning |
|---|---|
| Name | What you are testing |
| Primary Metric | Success Rate (higher is better), Latency (ms) (lower is better), Feedback reward (higher is better), or each prediction against its actual, by the variants' models' task: Accuracy (on actuals) for classification and multiclass models (higher is better), Absolute error (on actuals) for regression models (lower is better), Review score (on actuals) for multilabel, time series and other models (the reviewers' verdicts, higher is better). The form offers the one that fits the test's models; a test whose variants have different tasks has none. |
| Hypothesis | What you expect to happen |
| Confidence Level | 90%, 95% (default) or 99% |
| Min Detectable Effect | The smallest difference that decides it, relative to the control's mean (default 0.05, 5%). A significant difference smaller than this is inconclusive |
| Min Samples per Variant | Observations every variant needs before the experiment can decide (default 100); with fewer, the result is inconclusive |
A new experiment is a draft. Select Start to begin counting predictions. Starting can require governance approval when a governance solution covers the model.
Analyzing and concluding
While an experiment runs, Run Analysis compares each variant with the control and shows, per variant, the sample size, mean, standard deviation, 95% confidence interval, p-value, whether the difference is significant and the relative change from the control (green when it is better for the metric, red when it is worse: for latency, lower is better), with a recommendation: a variant that wins (significantly better than the control by at least the min detectable effect), the control (every variant significantly worse by at least that much), or inconclusive, with the reason when a setting decided it (for example "control has 15 of the 100 per variant the experiment needs"). The relative change is shown as - when the control's mean is 0.
An analysis concludes a running experiment (status completed) when a sequential test shows a significant difference, or when the experiment has run for its maximum duration (83 days, so every prediction it counts is still within the 90-day prediction retention). Through the REST API or Python SDK you can set a shorter maximum duration, or a maximum sample size per variant that concludes it once every variant reaches it. Stop cancels an experiment. A completed experiment without results offers Run Final Analysis.