Skip to main content

Experiments

Experiments track ML runs with parameters, metrics, and artifacts. Use this resource to create experiments, log results, compare runs, and register models.

Access it as client.experiments on a Strongly client, or the same path on AsyncStrongly with await. All methods exist on both with identical signatures.

Quick start​

from strongly import Strongly

client = Strongly()

# List (auto-paginates as you iterate)  # filters: search, status, tag, pinned, parent_run_id
for experiment in client.experiments.list():
    print(experiment.id)

Methods​

Core​

list​

list(*, search: str | None = None, status: str | None = None, tag: str | None = None, pinned: bool | None = None, parent_run_id: str | None = None, limit: int | None = None) -> SyncPaginator[Experiment]

List experiments, or the runs of one, newest first.

Parameters

  • search (str | None, optional): Search by name or description.
  • status (str | None, optional): Filter by status.
  • tag (str | None, optional): Filter by tag.
  • pinned (bool | None, optional): If True, only return pinned experiments.
  • parent_run_id (str | None, optional): The run_id of an experiment (or run): list its runs. Without it the top-level experiments are listed.
  • limit (int | None, optional): Maximum number of items to return (default: all matching items).

create​

create(*, name: str, description: str | None = None, tags: Sequence[str] | None = None, parent_run_id: str | None = None, status: str | None = None, params: Mapping[str, ParamValue] | None = None, system_info: Mapping[str, Any] | None = None, git_info: Mapping[str, Any] | None = None) -> Experiment

Create an experiment, or a run of one.

Parameters

  • name (str): Its name.
  • description (str | None, optional): What it is.
  • tags (Sequence[str] | None, optional): Tags.
  • parent_run_id (str | None, optional): For a run: the run_id of its experiment (or parent run).
  • status (str | None, optional): pending (default) or running (records when it started).
  • params (Mapping[str, ParamValue] | None, optional): Params to log with it.
  • system_info (Mapping[str, Any] | None, optional): The machine it runs on (python version, CPUs, GPUs, ...).
  • git_info (Mapping[str, Any] | None, optional): Where its code came from: commit, branch, repo and dirty.

retrieve​

retrieve(experiment_id: str) -> Experiment

Get an experiment or run by id.

update​

update(experiment_id: str, *, name: str | None = None, description: str | None = None, tags: Sequence[str] | None = None, status: str | None = None, framework: str | None = None, pinned: bool | None = None) -> Experiment

Update an experiment or run.

Parameters

  • experiment_id (str): Its id.
  • name, description (optional): New values.
  • tags (Sequence[str] | None, optional): The tags; they replace the current ones.
  • status (str | None, optional): running, completed, failed or cancelled; ending a run records when it finished and how long it ran.
  • framework (str | None, optional): The framework of the model the run trained (log_model sets it).
  • pinned (bool | None, optional): True pins the experiment (list(pinned=True) lists the pinned ones), False unpins it.

delete​

delete(experiment_id: str) -> None

Delete an experiment or run, with its runs, metric points and artifact files.

Other​

add_member​

add_member(resource_id: str, *, user_id: str, role: str) -> Permissions

Share the resource with a user.

Parameters

  • resource_id (str): The resource's id.
  • user_id (str): The user to share it with.
  • role (str): "editor" (use and change it) or "user" (use it only; types without a use-only tier, such as knowledge bases, take "editor" only).

compare​

compare(ids: list[str]) -> dict[str, Any]

Compare runs: their params and metrics side by side.

Parameters

  • ids (list[str]): The ids of the runs (or experiments) to compare (at least 2).

download_artifact​

download_artifact(experiment_id: str, path: str, dst_path: str | os.PathLike[str] | None = None) -> str

Download one of a run's artifact files and return where it was saved.

Parameters

  • experiment_id (str): The run's id.
  • path (str): The artifact's path, as listed.
  • dst_path (str | os.PathLike | None, optional): A file or folder to save it to (default: its name, in the current folder).

get_metric_history​

get_metric_history(experiment_id: str, key: str) -> list[MetricPoint]

Every logged point of one metric, in step order.

Parameters

  • experiment_id (str): The run's id.
  • key (str): The metric's key.

list_artifacts​

list_artifacts(experiment_id: str) -> list[ExperimentArtifact]

List the artifact files recorded on a run.

list_metric_keys​

list_metric_keys(experiment_id: str) -> list[str]

List the keys a run has logged metrics under.

list_runs​

list_runs(experiment_id: str, *, status: str | None = None) -> list[Experiment]

List the runs of an experiment (or the nested runs of a run), newest first.

Parameters

  • experiment_id (str): The experiment's (or run's) id.
  • status (str | None, optional): Only runs with this status.

log_artifact​

log_artifact(experiment_id: str, local_path: str | os.PathLike[str], *, path: str | None = None, type: str = "file", description: str | None = None, content_type: str | None = None) -> ExperimentArtifact

Upload a file of any size as an artifact of a run.

The file goes straight to storage through a presigned URL (a file over 512MB in parts, up to 5TB), then is recorded on the run with the size storage reports.

Parameters

  • experiment_id (str): The run's id.
  • local_path (str | os.PathLike): The file to upload.
  • path (str | None, optional): Its path among the run's artifacts, e.g. "model/model.joblib" (default: the file's name).
  • type (str, optional): file (default), model, dataset or figure.
  • description (str | None, optional): What it is.
  • content_type (str | None, optional): Its MIME type (default: from its name).

log_batch​

log_batch(experiment_id: str, *, metrics: Sequence[Mapping[str, Any]] = (), params: Mapping[str, ParamValue] | None = None) -> None

Log many metric points (any keys, any steps) and params, a request each.

Parameters

  • experiment_id (str): The run's id.
  • metrics (Sequence[Mapping[str, Any]], optional): Points {"key", "value", "step"?, "timestamp"?}, e.g. a whole training history.
  • params (Mapping[str, ParamValue] | None, optional): Params to log.

log_metrics​

log_metrics(experiment_id: str, metrics: Mapping[str, float], *, step: int | None = None, timestamp: datetime | int | str | None = None) -> None

Log metric points, one per key, e.g. {"loss": 0.21, "val/accuracy": 0.91}.

Every point is kept; the run's latest_metrics holds each key's value at its highest step.

Parameters

  • experiment_id (str): The run's id.
  • metrics (Mapping[str, float]): Key to value. A key is 1-250 letters, digits, _ - . / or spaces.
  • step (int | None, optional): A whole number, 0 or more (default 0), e.g. the epoch.
  • timestamp (datetime | int | str | None, optional): When they were measured: a datetime, epoch milliseconds or an ISO string (default now).

log_params​

log_params(experiment_id: str, params: Mapping[str, ParamValue]) -> None

Log params, e.g. {"learning_rate": 0.01, "max_depth": 6}.

A key logged again takes the new value.

Parameters

  • experiment_id (str): The run's id.
  • params (Mapping[str, ParamValue]): Key to a string, number or boolean.

permissions​

permissions(resource_id: str) -> Permissions

Return who can reach the resource: its owner, members and their roles, and visibility.

Parameters

  • resource_id (str): The resource's id.

register​

register(*, name: str, description: str | None = None, tags: Sequence[str] | None = None) -> Experiment

Your experiment with this name, created if you have none.

Parameters

  • name (str): Experiment name.
  • description (str | None, optional): Its description, when it is created.
  • tags (Sequence[str] | None, optional): Its tags, when it is created.

register_model​

register_model(experiment_id: str, *, name: str | None = None, model_id: str | None = None, artifact_path: str = "model", description: str | None = None) -> RunModelRegistration

Register the model a run logged (log_model) in the model registry.

The model file is copied into the registry and the run recorded as its training: params as hyperparameters, metrics such as accuracy, f1, auc, rmse or r2 on the model card, and its machine. scikit-learn, XGBoost, LightGBM, PyTorch and pickled models register.

Parameters

  • experiment_id (str): The run's id.
  • name (str | None, optional): The name of a new registry model (or give model_id).
  • model_id (str | None, optional): A registry model to add a version to.
  • artifact_path (str, optional): The folder log_model wrote to (default model).
  • description (str | None, optional): What the model (or version) is.

remove_member​

remove_member(resource_id: str, user_id: str) -> None

Stop sharing the resource with a user.

Parameters

  • resource_id (str): The resource's id.
  • user_id (str): The user to stop sharing it with.

set_visibility​

set_visibility(resource_id: str, visibility: str) -> Permissions

Make the resource public (every user can find and use it) or private.

Parameters

  • resource_id (str): The resource's id.
  • visibility (str): "public" or "private".

stats​

stats() -> ExperimentStats

Get experiment overview stats.