Metadata-Version: 2.4
Name: chronobench
Version: 0.1.1
Author-email: Louis Carpentier <louis.carpentier@kuleuven.be>
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: tqdm
Requires-Dist: tqdm; extra == "tqdm"
Provides-Extra: simple-example
Requires-Dist: scikit-learn>=1.3; extra == "simple-example"
Dynamic: license-file

Chronobench
===========

A registry-driven ML experiment benchmarking framework. You register your own data loaders, models, and metrics (or refer to library ones by import path) and write one training loop — Chronobench handles configuration, parameter sweeps, parallelism, and CSV output. Your domain code stays in its own modules and never needs to import Chronobench or change to add an experiment.

## Installation

Install Chronobench as follows:
```bash
pip install git+https://github.com/LouisCarpentier42/Chronobench
```

## Quick Start

Keep your components in plain modules (here `loaders.py`), then write a `script.py` that registers them, defines a `runner`, and calls `chronobench.main()`:

```python
import chronobench as cb
from loaders import IrisLoader, BreastCancerLoader

cb.data_loaders.register(iris=IrisLoader, breast_cancer=BreastCancerLoader)

@cb.runner
def runner(data_loader, model, metrics) -> dict:
    X_train, X_test, y_train, y_test = data_loader.load()
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    return {m.func.__name__: m(y_test, y_pred) for m in metrics}

if __name__ == "__main__":
    cb.main()
```

Experiment TOML entries pick an implementation by registered `name` or by dotted `target` import path:

```toml
[[data]]
name = "iris"                              # a registered name
test_size = 0.2

[[models]]
target = "sklearn.svm.SVC"                 # an import path - no registration
kernel = "linear"

[[metrics]]
target = "sklearn.metrics.accuracy_score"  # an import path — no registration
```

Then run it from the command line:

```bash
python script.py my_experiment
```

See [`examples/simple/`](examples/simple/) for a complete working example with scikit-learn.

## Concepts

### Registering components

Chronobench keeps three registries — `cb.data_loaders`, `cb.models`, and `cb.metrics` — plus one `runner`. A *factory* is any callable that produces a component: a class, or a function defined in your script. Register factories by short name:

```python
cb.data_loaders.register(<short-name>=<object>)
```

Registration is optional: anything importable can be referenced by its `target` path instead (see below), so in practice you register the components for which you want a short, stable name — typically your own project code — and refer to library components by import path.

Because `data_loader` and `model` are whatever the factory returns, you are not constrained to object-oriented wrappers — returning raw data (e.g., a tuple of arrays), a file path, or a config dict is equally valid as long as `runner` knows how to use it. The `runner` returns a dict whose keys become CSV columns.

### `name` vs `target`

Each TOML entry selects its implementation with **exactly one** of:

- `name` — a name you registered in the script. Short, refactor-safe, and validated (an unknown name fails fast with the list of available names). Best for your own, frequently used components.
- `target` — a dotted import path (e.g. `"sklearn.svm.SVC"`). Resolved by import, so any library class or function works with no wrapper and no registration. Best for third-party components.

All other keys in the entry are forwarded to the factory as `**kwargs`.

### Factories return lists or single instances

Each factory may return a single instance or a list of instances. A single instance produces one job variant, a list produces one variant per element, each crossed with every other-side variant. This allows you to easily run multiple datasets and/or models in the same experiment without writing any loops or separate config files.


A factory may return a **list** of instances. A single `[[data]]` (or `[[models]]`) entry then fans out into one variant per element, each crossed with every model (resp. data) variant — i.e. one CSV row per `(item × other-side-config)`. Each row is labeled by the instance's `name` attribute when present, otherwise by the entry label plus an index.

```python
def all_datasets(test_size=0.2, random_state=None):
    return [
        IrisLoader(test_size=test_size, random_state=random_state),
        BreastCancerLoader(test_size=test_size, random_state=random_state),
    ]

cb.data_loaders.register(all=all_datasets)   # one [[data]] entry -> two dataset rows
```

To instead keep a single job that internally manages several splits (e.g. a tuning + evaluation set in one folder), just return one object whose `load()` exposes all splits and let `runner` use them.

### Metrics: scoring functions without wrappers

A metric is a callable the `runner` invokes with data (e.g. `metric(y_true, y_pred)`). You can use a class whose instance is that callable, but a **bare scoring function works directly** — by `target` (no registration) or by a registered `name`:

```toml
[[metrics]]
target = "sklearn.metrics.fbeta_score"
beta = 1.0
average = "macro"   # bound into the function; remaining y_true/y_pred come from the runner
```

When the config kwargs don't satisfy every required parameter (here `fbeta_score` still needs `y_true`/`y_pred`), Chronobench binds the given arguments with `functools.partial` and lets the `runner` supply the rest, instead of calling the function immediately. The result is a bound function, so name the CSV column from it in your runner (e.g., `metric.func.__name__`). Note that a bare function uses *its own* defaults (e.g., `fbeta_score` defaults to `average="binary"`, which fails on multiclass data), so set what you need in the TOML.

To register such a function under a short name instead, the optional convenience is:

```python
from sklearn.metrics import accuracy_score, fbeta_score
cb.metrics.register(accuracy=accuracy_score, fbeta=fbeta_score)
```

### Context and environment injection

Factories are called with their TOML `**kwargs`. Chronobench also passes `environment` and/or `context` **only when the factory declares them as explicit parameters**, so plain and library classes work untouched while advanced factories can opt in:

```python
def make_model(context, n_layers=2):     # receives the cross-initializer context
    return MyModel(n_classes=context.data_loader.n_classes, n_layers=n_layers)
```

`environment` is the parsed `environment.toml`. `context` is a `Context` dataclass built progressively across stages:

| Stage | Populated `context` fields |
|---|---|
| data loaders | none |
| models | `data_loader`, `data_loader_name`, `data_loader_kwargs` |
| metrics | above + `model`, `model_name`, `model_kwargs` |

> `name`, `target`, `context`, and `environment` are reserved keys and must not be used as TOML parameter names.

### Configuration files

**`environment.toml`** — global settings, required keys:

```toml
path-to-results     = "./results"
path-to-experiments = "./experiments"
n-jobs              = 4
```

**`experiments/<name>.toml`** — one file per experiment, with `[[data]]`, `[[models]]`, and `[[metrics]]` arrays-of-tables. `[[data]]` and `[[models]]` entries produce job variants (their Cartesian product is the final job list); `[[metrics]]` entries are collected into a single flat list used for every job.

### Sweep parameters

Add `sweep.<key> = [v1, v2, ...]` to any entry to generate a parameter grid. The framework expands all sweep keys within a block via Cartesian product, then takes the product of data configs × model configs to produce the final job list. Sweep expansion on `[[metrics]]` entries only adds more metric instances per job — it does not create additional jobs.

```toml
[[data]]
name = "iris"
sweep.random_state = [0, 1, 2]   # 3 variants

[[models]]  # 2 × 2 = 4 model variants
target = "sklearn.ensemble.RandomForestClassifier"
sweep.criterion = ["gini", "entropy"]  # 2 variants
sweep.max_depth  = [5, 10]             # 2 variants

[[metrics]]
target = "sklearn.metrics.fbeta_score"
average = "macro"
sweep.beta = [0.5, 1.0, 2.0]  # 3 variants
```

This produces 3 × 4 = 12 jobs total. For each job, 3 metrics are computed ($F_\beta$ for all $\beta \in \{0.5, 1.0, 2.0\}$).

### Results

Results are written to:

```
{path-to-results}/raw/{experiment_name}/YYYYMMDD-HHMMSS.csv
```

Columns are the union of all keys returned by `runner()`, plus `Dataset` and `Model` metadata columns. If a key appears in both data loader and model kwargs, it is prefixed with `Dataset.` / `Model.` to avoid collision.

Run `python script.py --help` for the full list of options.

### Roadmap

This project is in early development, and many changes are still expected. However, the main interface should remain stable. Go to the [issue page](https://github.com/LouisCarpentier42/Chronobench/issues) for an overview of the tasks we plan to add in the near future.
