Metadata-Version: 2.4
Name: ibl-bwb-eval
Version: 0.0.2
Summary: Evaluation contract and scorers for the IBL BrainWideBench benchmark
Author: The IBL BrainWideBench authors
License-Expression: MIT
Project-URL: Homepage, https://bwb.iblcore.org
Project-URL: Repository, https://github.com/brainbench-org/ibl-bwb
Project-URL: Issues, https://github.com/brainbench-org/ibl-bwb/issues
Project-URL: Changelog, https://github.com/brainbench-org/ibl-bwb/blob/main/packaging/ibl-bwb-eval/CHANGELOG.md
Keywords: neuroscience,benchmark,electrophysiology,neural-decoding
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy==2.2.6
Requires-Dist: rich==15.0.0
Requires-Dist: safetensors==0.8.0
Requires-Dist: scikit-learn==1.7.2
Requires-Dist: scipy==1.15.3
Requires-Dist: torch==2.10.0
Requires-Dist: torchmetrics==1.9.0
Dynamic: license-file

# ibl-bwb-eval

The evaluation contract for the IBL BrainWideBench benchmark: everything needed to build
a valid submission and to score one, and nothing else. It is independent of the
benchmark's training code, so it installs on a CPU-only box without torch-brain, hydra,
ray or wandb.

## Install

```bash
pip install ibl-bwb-eval --extra-index-url https://download.pytorch.org/whl/cpu
```

Without that index torch resolves from PyPI to a CUDA build, around 4 GB of NVIDIA
packages the scorer never calls, so keep the flag in automated scoring environments too.
uv needs `--index-strategy unsafe-best-match` as well, since it will not fall back to
PyPI for a package found on another index.

The import name is `ibl_bwb_eval`:

```python
import ibl_bwb_eval
```

## What it gives you

```python
import ibl_bwb_eval

ibl_bwb_eval.EVAL_RECORDING_IDS  # path to the eval session list
ibl_bwb_eval.EVAL_SEEDS          # the seeds a reported number is averaged over
ibl_bwb_eval.SELECTION_METRIC    # what model selection optimizes
ibl_bwb_eval.SUITE_TASKS         # task vocabulary, per suite
```

Reading the protocol or the task list costs no torch or numpy import. The task names and
readout specs live in `ibl_bwb_eval.tasks`, the reported metrics (bits-per-spike, Poisson D²)
in `ibl_bwb_eval.metrics`, and the write side of the submission format in
`ibl_bwb_eval.predictions.PredictionsWriter`.

## Scoring a submission

```bash
python -m ibl_bwb_eval.scoring.ts1 --pred_dir predictions/ts1 --gt_dir ground_truth/ts1
python -m ibl_bwb_eval.scoring.ts2 --pred_dir predictions/<label> --gt_dir ground_truth/ts2
python -m ibl_bwb_eval.scoring.ts3 --pred_dir predictions/<label> --gt_dir ground_truth/
# all three suites, aggregated and ranked
python -m ibl_bwb_eval.scoring.aggregation --pred_dir predictions/ --gt_dir ground_truth/
```

## Versioning

Metric definitions and readout specs are part of published numbers, so this package is
versioned as API. `COSMOS_LABELS` is the sharpest case: it is the column order of every
TS3 `pred_proba`, and the TS3 scorer rejects a submission whose `label_names` disagree
with the ground truth. Pin the version you scored with.

## License

MIT.
