Tracking Runs

Track a run

import strongly

strongly.set_experiment("churn-model")   # your experiment of this name, created the first time

with strongly.start_run("xgb-depth-6", params={"max_depth": 6, "eta": 0.1}) as run:
    for epoch in range(50):
        loss, val_auc = train_one_epoch()
        strongly.log_metrics({"loss": loss, "val/auc": val_auc}, step=epoch)

    strongly.log_metric("auc", final_auc)
    strongly.set_tags(["xgboost", "baseline"])

Leaving the with block ends the run as completed, or failed if an exception left it. A failed run records why: the exception's type, message and traceback, shown with its status on the run's page, in its experiment's Runs and in the experiments list. Its duration is recorded. You can also call strongly.end_run("cancelled"), or strongly.end_run("failed", error=exc) to record an exception you caught.

Every run records the machine it ran on (Python version, platform, CPUs, memory, GPUs) and, when the code runs from a git repository, its commit, branch, remote and whether it had uncommitted changes. Pass log_system_metrics=False, log_environment=False or log_git_info=False to start_run to leave them out.

Params

strongly.log_param(key, value) and strongly.log_params({...}) record a run's configuration. A value is a string, number or boolean; a key logged again takes the new value.

Metrics

strongly.log_metric(key, value, step=None) and strongly.log_metrics({...}, step=None) record measurements. Every point is kept, so a run has the full history of each metric over steps, and its training curves can be drawn. Without step, log_metrics logs at the run's current step and then moves it on by one, so calling it once per epoch draws a curve.

A key is 1 to 250 letters, digits, _, -, ., / or spaces, such as val/loss.

Read a metric's points back with run.get_metric_history("val/auc").

Artifacts

Any file a run produces can be kept with it: plots, reports, datasets, checkpoints.

strongly.log_artifact("roc.png", "plots")           # plots/roc.png
strongly.log_artifacts("outputs/", "eval")         # every file under outputs/, keeping its folders
strongly.log_text(report, "notes/summary.md")
strongly.log_dict(config, "config.json")
strongly.log_figure(fig, "confusion-matrix")       # figures/confusion-matrix.png (Matplotlib or Plotly)
strongly.log_dataset(train_df, "train")            # datasets/train.parquet, with its version and shape as params

Files of any size are uploaded straight to storage: a file over 512MB goes in parts, up to 5TB. run.list_artifacts() lists them and run.download_artifact(path, "./downloads") fetches one back.

Models

strongly.log_model(model)          # model/model.joblib + model/MLmodel.json
strongly.log_model(model, input_example=X_train.head(5))   # and its typed inputs
strongly.log_model(net, problem_type="classification")     # a model whose task it cannot tell

log_model saves the model in the format the Model Registry serves: scikit-learn, XGBoost and LightGBM models with joblib, a PyTorch module whole (model.pt), anything else pickled (a Keras model is saved as model.keras). MLmodel.json beside it records the framework, the model class and version, the Python version, and the model's feature names (the columns it was fitted on, or feature_names=[...]) and target (target_name=...). The run's framework becomes the logged model's, as the run page shows it.

MLmodel.json also records the model's task (problem_type), which drift detection needs. log_model reads it from the fitted model where the scikit-learn API says it: a scikit-learn, XGBoost or LightGBM classifier (binary or multiclass) is classification, a regressor is regression. For a model whose task it cannot tell, such as a PyTorch module, pass problem_type=: classification, regression, multiclass, multilabel, timeseries or other. A given problem_type wins over the model's. Without either, no task is recorded.

input_example (a pandas DataFrame of a few input rows) records the model's signature: each column's name and type (int, float, boolean or string), as inputs in MLmodel.json. A model registered from the run takes them as its input schema: its Test tab asks for typed values, and a LinUCB A/B test can route on its numeric inputs (a model with no input schema cannot take part in one). Autolog records them from the DataFrame the model was fitted on.

The model is served with the library version and Python it was saved with (both recorded in MLmodel.json), so a model pickled with one scikit-learn release loads in that release.

Nested runs

with strongly.start_run("cv-search") as search:
    for fold in range(5):
        with strongly.start_run(f"fold-{fold}", nested=True) as run:
            run.log_metric("auc", fit_fold(fold))

A run started while another is active must be nested=True; ending it makes its parent the active run again.