Skip to main content

A/B Tests

A/B tests between registry models, their predictions and experiments.

Access it as client.ab_tests on a Strongly client, or the same path on AsyncStrongly with await. All methods exist on both with identical signatures.

Quick start​

from strongly import Strongly

client = Strongly()

# List (auto-paginates as you iterate)  # filters: status, strategy, model_id, workspace_id, search, sort
for a/b in client.ab_tests.list():
    print(a/b.id)

Methods​

Core​

list​

list(*, status: str | None = None, strategy: str | None = None, model_id: str | None = None, workspace_id: str | None = None, search: str | None = None, sort: str | None = None, limit: int | None = None) -> SyncPaginator[ABTest]

Return the A/B tests you can see (newest update first).

Parameters

  • status (str | None, optional): registered, deploying, running, stopped or failed.
  • strategy (str | None, optional): weighted_random, feature_based, multi_armed_bandit or canary.
  • model_id (str | None, optional): Only tests with a variant serving this registry model.
  • workspace_id (str | None, optional): Only tests in this workspace.
  • search (str | None, optional): Matches name and description.
  • sort (str | None, optional): -updatedAt (default), createdAt, name, strategy, -metrics.totalRequests.
  • limit (int | None, optional): Maximum number of tests (default: all).

create​

create(*, name: str, strategy: str, variants: Sequence[Mapping[str, Any]], description: str | None = None, tags: Sequence[str] | None = None, workspace_id: str | None = None, sticky_routing: bool | None = None, feature_rules: Sequence[Mapping[str, Any]] | None = None, bandit_config: Mapping[str, Any] | None = None, canary_config: Mapping[str, Any] | None = None) -> ABTest

Create an A/B test (status registered; deploy starts serving it).

Parameters

  • name (str): Test name.
  • strategy (str): weighted_random (each variant gets its weight's share), feature_based (the first matching rule by priority picks the variant, else the control), multi_armed_bandit (routes by the rewards recorded with prediction_feedback) or canary (the target variant gets the canary's current percentage of traffic; once deployed, the platform advances it through its stages while it stays as healthy as the control, or rolls it back to 0% with the reason, shown in canary_config.last_check).
  • variants (Sequence[Mapping[str, Any]]): At least two {"variantId", "modelId", "weight"?, "isControl"?}` dicts, each serving a registry model. `weight is a 0-1 fraction (default an equal share); for weighted_random the weights must sum to 1. The first variant is the control unless one is marked.
  • description, tags, workspace_id: Metadata.
  • sticky_routing (bool | None, optional): Keep an entity on one variant for the test's life, keyed on the entity_id sent with each predict (a prediction without one is then refused). A canary moves an entity to the target only as its share grows, and back on a rollback.
  • feature_rules (Sequence[Mapping[str, Any]] | None, optional): For feature_based, required (at least one): {"featureName", "operator", "value", "targetVariantId", "priority"?}`. Operators `equals`, `not_equals`, `in, not_in, greater_than, less_than, contains, regex; value is a list for in / not_in, a number for greater_than / less_than, a valid pattern for regex. priority (lower first) defaults to the order given. A rule serving could not apply is refused, naming it.
  • bandit_config (Mapping[str, Any] | None, optional): For multi_armed_bandit: {"algorithm": "epsilon_greedy" | "thompson_sampling" | "ucb1" | "linucb", "epsilon", "explorationBonus" (LinUCB's alpha), "contextFeatures", "rewardMetric"}`. `linucb routes by the request's contextFeatures: numeric input features every variant model declares as numbers, and every variant model must have Record inputs on (LinUCB learns from each prediction's recorded inputs).
  • canary_config (Mapping[str, Any] | None, optional): For canary: {"controlVariantId", "targetVariantId", "stages", "currentPercentage", "targetPercentage", "bakeMinutes", "errorRateDelta", "latencyP95DeltaMs", "minRequestsForDecision"}. A stage advances only after bakeMinutes and minRequestsForDecision canary predictions, judged on the current stage's traffic against the control.

retrieve​

retrieve(test_id: str) -> ABTest

Return an A/B test.

Its variants and their counters, strategy config and deployment (status).

update​

update(test_id: str, *, name: str | None = None, description: str | None = None, tags: Sequence[str] | None = None) -> ABTest

Change an A/B test's name, description or tags and return it.

delete​

delete(test_id: str) -> None

Delete an A/B test (stopped first when running).

Its recorded predictions are kept for their retention period.

Lifecycle & actions​

start​

start(test_id: str) -> ABTest

Start a stopped A/B test again (re-deploys it). Returns the test.

stop​

stop(test_id: str) -> ABTest

Stop routing (status stopped); predictions, counters and experiments are kept.

deploy​

deploy(test_id: str) -> ABTest

Start routing predictions to the variants' models (status running).

The variants' models must be deployed. Returns the test.

Other​

analyze_experiment​

analyze_experiment(experiment_id: str) -> dict[str, Any]

Analyze an experiment: each treatment against the control on the primary metric.

Returns controlStats, treatmentStats (mean, sample size, p-value, confidence interval, relative improvement, significance), recommendation (treatment, control or inconclusive), winnerVariantId and reason. A winner needs min_sample_per_variant observations in every variant and a significant difference of at least minimum_detectable_effect relative to the control's mean; reason says which setting left the result inconclusive. relativeImprovement is None when the control's mean is 0. A running experiment concludes when a sequential test is significant or its sample or duration limit is reached.

conclude_experiment​

conclude_experiment(experiment_id: str) -> ABTestExperiment

End a running experiment as a finished run (status completed).

create_experiment​

create_experiment(test_id: str, *, name: str, control_variant_id: str, treatment_variant_ids: Sequence[str], primary_metric: str, description: str | None = None, hypothesis: str | None = None, confidence_level: float | None = None, minimum_detectable_effect: float | None = None, min_sample_per_variant: int | None = None, max_sample_per_variant: int | None = None, max_duration_days: float | None = None) -> ABTestExperiment

Create an experiment (status draft) and return its id.

It compares each treatment variant with the control on the primary metric over the predictions the test routes while it runs.

Parameters

  • primary_metric (str): success_rate (higher is better), accuracy (each prediction against its actual: a label sent with prediction_feedback or the model's actuals, higher is better), latency (lower is better) or custom (the reward from prediction_feedback, higher is better).
  • confidence_level (float | None, optional): Between 0 and 1 (default 0.95).
  • minimum_detectable_effect (float | None, optional): Between 0 and 1 (default 0.05).
  • min_sample_per_variant (int | None, optional): Samples per variant before a result is meaningful (default 100).
  • max_sample_per_variant, max_duration_days: Conclude the experiment once every variant has this many samples, or after this many days (default and maximum 83, within the 90-day prediction retention).

delete_experiment​

delete_experiment(experiment_id: str) -> None

Delete an experiment and its results (the test's predictions are kept).

list_experiments​

list_experiments(test_id: str, *, search: str | None = None, sort: str | None = None, limit: int | None = None) -> SyncPaginator[ABTestExperiment]

Return the test's experiments (newest first) with their status and latest results.

Parameters

  • sort (str | None, optional): -createdAt (default), name, status, startedAt.

metrics​

metrics(test_id: str) -> ABTestMetrics

Return the test's totals since it was created, overall and per variant.

predict​

predict(test_id: str, input_data: Any, *, entity_id: str | None = None) -> dict[str, Any]

Predict through a running A/B test.

The response is the serving model's output plus prediction_id and the variant_id that served it; pass prediction_id to prediction_feedback.

entity_id is the user or entity the prediction is for. A test with sticky_routing requires it (the entity keeps its variant); a linucb test also needs each of its context features in input_data as one number.

prediction_feedback​

prediction_feedback(prediction_id: str, *, reward: float | None = None, label: str | None = None) -> dict[str, Any]

Record the outcome of a prediction a test routed.

Parameters

  • reward (float | None, optional): 0 to 1, kept on the prediction (sending it again replaces it, and a bandit counts only the difference). A multi_armed_bandit test learns from it (linucb over the prediction's recorded inputs, so the variant model needs Record inputs on) and an experiment with primary_metric="custom" measures it.
  • label (str | None, optional): The prediction's actual (its true outcome), recorded with the model's actuals.

predictions​

predictions(test_id: str, *, variant_id: str | None = None, search: str | None = None, sort: str | None = None, limit: int | None = None) -> SyncPaginator[PredictionRecord]

Return the predictions the test routed (newest first).

The variant that served each and why, latency, success, and its actual and reward when recorded.

Parameters

  • variant_id (str | None, optional): Only one variant's predictions.
  • search (str | None, optional): Matches prediction id, entity id, variant, routing reason or error, or the prediction exactly.
  • sort (str | None, optional): -timestamp (default), latencyMs, variantId, success, confidence.
  • limit (int | None, optional): Maximum number of predictions (default: all).

start_experiment​

start_experiment(experiment_id: str) -> ABTestExperiment

Start a draft experiment: it counts the predictions from now until it ends.

stop_experiment​

stop_experiment(experiment_id: str) -> ABTestExperiment

Cancel a draft or running experiment (status cancelled).

traffic​

traffic(test_id: str) -> ABTestTraffic

Return requests, errors and latency per variant over time.

update_variant​

update_variant(test_id: str, variant_id: str, *, weight: float | None = None, enabled: bool | None = None) -> dict[str, Any]

Update a variant: its share of traffic, whether it is enabled, or both.

Parameters

  • weight (float | None): The enabled variant's share of traffic (a 0-1 fraction) in a weighted_random test; the other enabled variants are rescaled to fill the rest.
  • enabled (bool | None): False disables the variant (it receives no traffic); at least two stay enabled.