Skip to main content

A/B Tests

A/B tests, variants, predictions, and Bayesian experiments.

Access it as client.ab_tests on a Strongly client, or the same path on AsyncStrongly with await. All methods exist on both with identical signatures.

Quick start​

from strongly import Strongly

client = Strongly()

# List  # filters: status, strategy, workspace_id
for test in client.ab_tests.list():
    print(test.id)

# Create
created = client.ab_tests.create(
    name="champion-vs-challenger",
    strategy="weighted_random",
    variants=[
        {"variantId": "champion", "modelId": "model_abc", "weight": 0.5},
        {"variantId": "challenger", "modelId": "model_xyz", "weight": 0.5},
    ],
)
print(created.id)

# Read it back
fetched = client.ab_tests.retrieve(created.id)

# Delete (returns None; raises on failure)
client.ab_tests.delete(created.id)

Methods​

Core​

list​

list(*, status: Optional[str] = None, strategy: Optional[str] = None, workspace_id: Optional[str] = None) -> List[ABTest]

List A/B tests.

Args: status: Filter by test status. strategy: Filter by allocation strategy. workspace_id: Restrict to a workspace.

Returns: A list of ABTest.

create​

create(*, name: str, strategy: str, variants: Sequence[Mapping[str, Any]], description: Optional[str] = None, tags: Optional[Sequence[str]] = None, feature_rules: Optional[Sequence[Mapping[str, Any]]] = None, bandit_config: Optional[Mapping[str, Any]] = None, canary_config: Optional[Mapping[str, Any]] = None) -> ABTest

Create an A/B test.

Args: name: Test name. strategy: Routing strategy: weighted_random, feature_based, multi_armed_bandit, or canary. variants: At least two variants, each a plain dict with camelCase wire keys (each points a deployed model at a share of traffic). description, tags: Optional metadata. feature_rules: Routing rules for the feature_based strategy. bandit_config: Config for the multi_armed_bandit strategy. canary_config: Config for the canary strategy.

from strongly import Strongly

test = client.ab_tests.create(
    name="champion-vs-challenger",
    strategy="weighted_random",
    variants=[
        {"variantId": "champion", "modelId": "model_abc", "weight": 0.7, "isControl": True},
        {"variantId": "challenger", "modelId": "model_xyz", "weight": 0.3},
    ],
)

retrieve​

retrieve(test_id: str) -> ABTest

Retrieve a single A/B test by id.

update​

update(test_id: str, *, name: Optional[str] = None, description: Optional[str] = None, tags: Optional[Sequence[str]] = None, strategy: Optional[str] = None, status: Optional[str] = None) -> ABTest

Update an A/B test and return the updated record.

delete​

delete(test_id: str) -> None

Delete an A/B test. Raises on failure; returns nothing.

Lifecycle & actions​

start​

start(test_id: str) -> Dict[str, Any]

Start an A/B test.

stop​

stop(test_id: str) -> Dict[str, Any]

Stop an A/B test.

deploy​

deploy(test_id: str) -> Dict[str, Any]

Deploy an A/B test's variants to serving.

Other​

analyze_experiment​

analyze_experiment(experiment_id: str) -> Dict[str, Any]

Run analysis on a Bayesian experiment.

create_experiment​

create_experiment(test_id: str, *, name: str, control_variant_id: str, treatment_variant_ids: Sequence[str], description: Optional[str] = None, hypothesis: Optional[str] = None, primary_metric: Optional[str] = None, confidence_level: Optional[float] = None, minimum_detectable_effect: Optional[float] = None, min_sample_per_variant: Optional[int] = None) -> Dict[str, Any]

Create a formal (Bayesian) experiment under an A/B test (control vs treatments).

client.ab_tests.create_experiment(
    test_id,
    name="ctr-lift",
    control_variant_id="champion",
    treatment_variant_ids=["challenger"],
    primary_metric="click_through_rate",
    confidence_level=0.95,
)

delete_experiment​

delete_experiment(experiment_id: str) -> None

Delete a Bayesian experiment. Raises on failure; returns nothing.

metrics​

metrics(test_id: str, *, start_date: Optional[str] = None, end_date: Optional[str] = None) -> Dict[str, Any]

Get metrics for an A/B test over an optional date window.

prediction_feedback​

prediction_feedback(prediction_id: str, *, reward: Optional[float] = None, label: Optional[str] = None) -> Dict[str, Any]

Record ground-truth feedback for a prediction.

Args: reward: Reward score (0-1) for bandit optimization. label: Feedback label (e.g. correct / incorrect).

predictions​

predictions(test_id: str, *, limit: Optional[int] = None, offset: Optional[int] = None, variant_id: Optional[str] = None) -> Dict[str, Any]

List predictions recorded for an A/B test.

set_variant_weight​

set_variant_weight(test_id: str, variant_id: str, *, weight: float) -> Dict[str, Any]

Set the traffic weight for a variant (0-1 fraction of traffic).

start_experiment​

start_experiment(experiment_id: str) -> Dict[str, Any]

Start a Bayesian experiment.

stop_experiment​

stop_experiment(experiment_id: str) -> Dict[str, Any]

Stop a Bayesian experiment.

toggle_variant​

toggle_variant(test_id: str, variant_id: str, *, enabled: bool) -> Dict[str, Any]

Enable or disable a variant.