Metadata-Version: 2.4
Name: chronobench
Version: 0.1.3
Summary: A flexible benchmarking framework for ML models.
Author-email: Louis Carpentier <louis.carpentier@kuleuven.be>
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tqdm
Requires-Dist: tqdm; extra == "tqdm"
Provides-Extra: simple-example
Requires-Dist: scikit-learn>=1.3; extra == "simple-example"
Dynamic: license-file

Chronobench
===========

A registry-driven ML experiment benchmarking framework. You register your own data loaders, models, and metrics (or refer to library ones by import path) and write one training loop — Chronobench handles configuration, parameter sweeps, parallelism, and CSV output. Your domain code stays in its own modules and never needs to import Chronobench or change to add an experiment.

## Installation

Install Chronobench as follows:
```bash
pip install chronobench
```

## Quick Start

Below example shows a minimal working experiment with scikit-learn. A data loader for the Iris dataset is registered, a runner is defined, and chronobench is started:
```python
import chronobench as cb
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split


@cb.data_loaders("iris")
class IrisLoader:
    def __init__(self, test_size: float = 0.2, random_state: int | None = None):
        self.test_size = test_size
        self.seed = random_state

    def load(self):
        X, y = load_iris(return_X_y=True)
        return train_test_split(X, y, test_size=self.test_size, random_state=self.seed)


@cb.runner
def runner(data_loader, model, metrics) -> dict:
    X_train, X_test, y_train, y_test = data_loader.load()
    y_pred = model.fit(X_train, y_train).predict(X_test)
    results = {metric.func.__name__: metric(y_test, y_pred) for metric in metrics}
    return results


if __name__ == "__main__":
    cb.main()

```

Experiment TOML entries pick an implementation by registered `name` or by dotted `target` import path:
```toml
[[data]]
name = "iris"                              # a registered name
test_size = 0.2

[[models]]
target = "sklearn.svm.SVC"                 # an import path - no registration
kernel = "linear"

[[metrics]]
target = "sklearn.metrics.accuracy_score"  # an import path — no registration
```
Then run it from the command line:
```bash
python script.py my_experiment
```

See [`examples/`](examples/) for complete working examples.

## Concepts

### Registering components

Chronobench keeps three registries — `cb.data_loaders`, `cb.models`, and `cb.metrics` — plus one `runner`. A *factory* is any callable that produces a component: a class, or a function defined in your script. Call a registry to register factories — either with keyword pairs or as a decorator:

```python
cb.data_loaders(<short-name>=<object>)   # keyword pairs

@cb.data_loaders                          # bare decorator — name taken from the function
def iris(...): ...

@cb.data_loaders("all")                   # decorator with an explicit name
def all_datasets(...): ...
```

Registration is optional: anything importable can be referenced by its `target` path instead (see below), so in practice you register the components for which you want a short, stable name — typically your own project code — and refer to library components by import path.

Because `data_loader` and `model` are whatever the factory returns, you are not constrained to object-oriented wrappers — returning raw data (e.g., a tuple of arrays), a file path, or a config dict is equally valid as long as `runner` knows how to use it. The `runner` returns a dict whose keys become CSV columns.

### `name` vs `target`

Each TOML entry selects its implementation with **exactly one** of:

- `name` — a name you registered in the script. Short, refactor-safe, and validated (an unknown name fails fast with the list of available names). Best for your own, frequently used components.
- `target` — a dotted import path (e.g. `"sklearn.svm.SVC"`). Resolved by import, so any library class or function works with no wrapper and no registration. Best for third-party components.

All other keys in the entry are forwarded to the factory as `**kwargs`.

### Fan-out: many variants from one entry

A factory may return a **single instance** (one job variant) or **several instances**, which fan out into one variant per element — each crossed with every other-side variant, i.e. one CSV row per `(item × other-side-config)`. This runs multiple datasets and/or models in one experiment without loops or separate config files. There are three ways to return several, differing only in how each variant is *labeled* in the final resutls (its `Dataset` / `Model` column):

**A list** — labeled by the entry label plus an index (`all[0]`, `all[1]`):

```python
@cb.data_loaders("all")   # one [[data]] entry -> two dataset rows
def all_datasets(test_size=0.2, random_state=None):
    return [
        IrisLoader(test_size=test_size, random_state=random_state),
        BreastCancerLoader(test_size=test_size, random_state=random_state),
    ]
```

**A dict** — keys become the labels:

```python
@cb.data_loaders("named")
def named_datasets(test_size=0.2):
    return {
        "iris": IrisLoader(test_size=test_size),
        "breast_cancer": BreastCancerLoader(test_size=test_size),
    }
```

**`cb.variant(instance, name=..., **params)`** — for full control: `name` sets the label and any extra keyword arguments are recorded as CSV columns. Because the distinguishing value is a recorded parameter rather than part of the label, several variants can share one label yet stay distinct. The following example illustrates how you can create a model with 10 different random seeds programmatically, without having to define this in the configuration file. In the final results file, the "Model" column will be the same for all 10 variants, but each will have a different `seed` column.
```python
@cb.models("my-seeded-model")
def my_model(**kwargs): 
    n_seeds = 10
    return [
        cb.variant(MyModel(seed=i, **kwargs), name="my_model", seed=i)
        for i in range(n_seeds)
    ]
```

### Metrics: scoring functions without wrappers

A metric is a callable the `runner` invokes with data (e.g. `metric(y_true, y_pred)`). You can use a class whose instance is that callable, but a **bare scoring function works directly** — by `target` (no registration) or by a registered `name`:

```toml
[[metrics]]
target = "sklearn.metrics.fbeta_score"
beta = 1.0
average = "macro"   # bound into the function; remaining y_true/y_pred come from the runner
```

When the config kwargs don't satisfy every required parameter (here `fbeta_score` still needs `y_true`/`y_pred`), Chronobench binds the given arguments with `functools.partial` and lets the `runner` supply the rest, instead of calling the function immediately. The result is a bound function, so name the CSV column from it in your runner (e.g., `metric.func.__name__`). Note that a bare function uses *its own* defaults (e.g., `fbeta_score` defaults to `average="binary"`, which fails on multiclass data), so set what you need in the TOML.

To register such a function under a short name instead, the optional convenience is:

```python
from sklearn.metrics import accuracy_score, fbeta_score
cb.metrics(accuracy=accuracy_score, fbeta=fbeta_score)
```

### Context and environment injection

Factories are called with their TOML `**kwargs`. Chronobench also passes `environment` and/or `context` **only when the factory declares them as explicit parameters**, so plain and library classes work untouched while advanced factories can opt in:

```python
def make_model(context, n_layers=2):     # receives the cross-initializer context
    return MyModel(n_classes=context.data_loader.n_classes, n_layers=n_layers)
```

`environment` is the parsed `environment.toml`. `context` is a `Context` dataclass built progressively across stages:

| Stage | Populated `context` fields |
|---|---|
| data loaders | none |
| models | `data_loader`, `data_loader_name`, `data_loader_kwargs` |
| metrics | above + `model`, `model_name`, `model_kwargs` |

> `name`, `target`, `context`, and `environment` are reserved keys and must not be used as TOML parameter names.

### Configuration files

**`environment.toml`** — global settings, required keys:

```toml
path-to-results     = "./results"
path-to-experiments = "./experiments"
n-jobs              = 4
```

**`experiments/<name>.toml`** — one file per experiment, with `[[data]]`, `[[models]]`, and `[[metrics]]` arrays-of-tables. `[[data]]` and `[[models]]` entries produce job variants (their Cartesian product is the final job list); `[[metrics]]` entries are collected into a single flat list used for every job.

### Sweep parameters

Add `sweep.<key> = [v1, v2, ...]` to any entry to generate a parameter grid. The framework expands all sweep keys within a block via Cartesian product, then takes the product of data configs × model configs to produce the final job list. Sweep expansion on `[[metrics]]` entries only adds more metric instances per job — it does not create additional jobs.

```toml
[[data]]
name = "iris"
sweep.random_state = [0, 1, 2]   # 3 variants

[[models]]  # 2 × 2 = 4 model variants
target = "sklearn.ensemble.RandomForestClassifier"
sweep.criterion = ["gini", "entropy"]  # 2 variants
sweep.max_depth  = [5, 10]             # 2 variants

[[metrics]]
target = "sklearn.metrics.fbeta_score"
average = "macro"
sweep.beta = [0.5, 1.0, 2.0]  # 3 variants
```

This produces 3 × 4 = 12 jobs total. For each job, 3 metrics are computed ($F_\beta$ for all $\beta \in \{0.5, 1.0, 2.0\}$).

### Results

Results are written to:

```
{path-to-results}/raw/{experiment_name}/YYYYMMDD-HHMMSS.csv
```

Columns are the union of all keys returned by `runner()`, plus `Dataset` and `Model` metadata columns. If a key appears in both data loader and model kwargs, it is prefixed with `Dataset.` / `Model.` to avoid collision.

Run `python script.py --help` for the full list of options.

### Roadmap

This project is in early development, and many changes are still expected. However, the main interface should remain stable. Go to the [issue page](https://github.com/LouisCarpentier42/Chronobench/issues) for an overview of the tasks we plan to add in the near future.
