Metadata-Version: 2.4
Name: synthonbench
Version: 0.2.0
Summary: Reference baselines, DCRF metrics and datasets for budgeted search over synthon combinatorial spaces.
Author-email: Miroslav Lzicar <miroslav.lzicar@deepmedchem.com>
License-Expression: MIT
Project-URL: Homepage, https://github.com/mireklzicar/synthonbench
Project-URL: Documentation, https://github.com/mireklzicar/synthonbench#readme
Project-URL: Repository, https://github.com/mireklzicar/synthonbench
Project-URL: Issues, https://github.com/mireklzicar/synthonbench/issues
Keywords: cheminformatics,benchmark,synthon,combinatorial-chemical-space,virtual-screening,drug-discovery
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Chemistry
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Provides-Extra: minimal
Provides-Extra: ml
Requires-Dist: scikit-learn>=1.1; extra == "ml"
Provides-Extra: chem
Requires-Dist: rdkit>=2023.3; extra == "chem"
Requires-Dist: pandas>=2.0; extra == "chem"
Requires-Dist: pyarrow>=14; extra == "chem"
Requires-Dist: rdfp>=0.1; extra == "chem"
Provides-Extra: hf
Requires-Dist: huggingface_hub>=0.24; extra == "hf"
Requires-Dist: hf_transfer>=0.1; extra == "hf"
Requires-Dist: pandas>=2.0; extra == "hf"
Requires-Dist: pyarrow>=14; extra == "hf"
Provides-Extra: surrogate
Requires-Dist: lightgbm>=4.0; extra == "surrogate"
Requires-Dist: xgboost>=2.0; extra == "surrogate"
Provides-Extra: cuda
Requires-Dist: lightgbm>=4.0; extra == "cuda"
Requires-Dist: xgboost>=2.0; extra == "cuda"
Requires-Dist: torch>=2.0; extra == "cuda"
Provides-Extra: all
Requires-Dist: scikit-learn>=1.1; extra == "all"
Requires-Dist: rdkit>=2023.3; extra == "all"
Requires-Dist: pandas>=2.0; extra == "all"
Requires-Dist: pyarrow>=14; extra == "all"
Requires-Dist: rdfp>=0.1; extra == "all"
Requires-Dist: huggingface_hub>=0.24; extra == "all"
Requires-Dist: hf_transfer>=0.1; extra == "all"
Requires-Dist: lightgbm>=4.0; extra == "all"
Requires-Dist: xgboost>=2.0; extra == "all"
Provides-Extra: dev
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: bump2version>=1.0; extra == "dev"
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: scikit-learn>=1.1; extra == "dev"
Requires-Dist: pandas>=2.0; extra == "dev"
Requires-Dist: pyarrow>=14; extra == "dev"
Requires-Dist: twine>=5.0; extra == "dev"
Dynamic: license-file

# synthonbench

[![CI](https://github.com/mireklzicar/synthonbench/actions/workflows/ci.yml/badge.svg)](https://github.com/mireklzicar/synthonbench/actions/workflows/ci.yml)
[![Demo Website](https://img.shields.io/badge/demo-website-0A7B83)](https://synthonbench.mireklzicar.com/)
[![Dataset](https://img.shields.io/badge/🤗%20dataset-mireklzicar%2Fsynthonbench-yellow)](https://huggingface.co/datasets/mireklzicar/synthonbench)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

> **SynthonBench is a benchmark for budgeted search over synthon combinatorial
> (make-on-demand) chemical spaces.**

Search methods over synthon spaces never enumerate the full product library;
they propose **reaction-component tuples** `(reaction_id, synthon_ids)` and pay
a fixed oracle-call budget. SynthonBench compares such methods *fairly and
reproducibly*:

- a **frozen space + oracle** so every run sees the same objective;
- a **shared budget** with a **unique-query guard** (duplicate oracle calls are
  never charged);
- the **top-k DCRF** headline metric — how many random oracle calls a run's
  top-k recall is worth, divided by what it actually spent. Random search ≈
  **1.0**; the exact oracle gives the ceiling.

This package ships the synthon-space task API, the frozen scalar oracles, the
reference baselines, the **top-k DCRF** metric, the camera-ready **method
presets**, and a one-line **downloader** for the released giga-scale datasets.

## Install — dependency tiers

The core runs every native baseline on the synthetic oracle with **NumPy only**.
Add tiers for parquet score tables, dataset downloads, or surrogate training:

```bash
pip install synthonbench                # core (NumPy only)
pip install "synthonbench[minimal]"     # explicit alias for the core install
pip install "synthonbench[ml]"          # + scikit-learn  → RF / ISE learned baselines
pip install "synthonbench[chem]"        # + RDKit/pandas/pyarrow → read parquet tables, build descriptors/FPs
pip install "synthonbench[hf]"          # + huggingface_hub → download the released datasets (with progress)
pip install "synthonbench[surrogate]"   # + LightGBM/XGBoost → (re)train the docking surrogates
pip install "synthonbench[cuda]"        # GPU surrogate stack (install CUDA-matched wheels yourself)
pip install "synthonbench[all]"         # everything except env-specific CUDA wheels
```

## Quickstart (no data, no chemistry stack)

```python
from synthonbench import make_example_task, run_method

task = make_example_task(budget=2000, seed=0)
result = run_method("ga", task)            # typed global genetic algorithm

print(result.best_utility)
print(result.metrics["call_reduction_factor_top1000"])   # top-1000 DCRF
print(result.metrics["top1000_recall"])
```

Compare several methods from the command line:

```console
$ synthonbench compare --methods random_pw ts ga bb --budget 300 --seeds 0 1 2 3 4 --top-k 100
method     mean call_reduction_factor_top100  (random-equivalent baseline = 1.0)
bb         6.108
ts         3.190
ga         2.664
random_pw  1.154

$ synthonbench run --method ga --budget 2000 --seed 0
$ synthonbench list-methods
```

(Illustrative numbers on the bundled ~2.5k-product demo space. The paper's
full-scale results are on giga-scale spaces — see below.)

## Real-scale workflow (download → run → score)

```bash
pip install "synthonbench[hf,chem]"
synthonbench download --scale 1M          # spaces + glide_1M + surrogate_1M + seeds (~150 MB)
```

```python
from synthonbench import load_space, load_oracle, build_with_preset
from synthonbench.runner import run_optimizer
from synthonbench.task import GlobalSynthonTask

space  = load_space("1M")
oracle = load_oracle("1M", "kif11", kind="glide")     # frozen real-Glide LE oracle
task   = GlobalSynthonTask(space, oracle, budget=10_000, seed=0)

opt    = build_with_preset("ga", space=space, allowed_reactions=task.allowed_reactions, seed=0)
result = run_optimizer(opt, task, method="ga")
```

A full, copy-pasteable submission is in
[`examples/submission_template.py`](examples/submission_template.py); score it
against the released table with
[`scripts/score_submission.py`](scripts/score_submission.py).

## The baseline roster

Names, abbreviations and **Family** mirror the camera-ready leaderboard. Every
method is a **global** optimizer over the shared multi-reaction budget; most are
a single-reaction proposer made global by a reaction allocator, and the typed GA
is global by construction.

| name | label | family | what it is |
|------|-------|--------|------------|
| `ga` | GA | Evolutionary | **typed global genetic algorithm** — strongest in-house baseline |
| `ise` | ISE | Active learning | iterative surrogate expansion (SpaceHASTEN-*style*, benchmark-native) |
| `salsa` | SALSA\* | Bandit | SALSA-inspired tabular Thompson sampler |
| `bb` | BB\* | Bandit | ROCS-X-inspired Bayesian belief bandit |
| `ts` | TS | Bandit | reagent-prior Thompson sampling (Klarich/Walters-style prior) |
| `sh` | SpaceHASTEN | Similarity search | **external** — FTrees/SpaceLight + surrogate |
| `rf` | RF | Active learning | pooled per-slot random forest *(needs `[ml]`)* |
| `shc` | SpaceHASTEN-ctrl | Similarity search | **external** — SpaceHASTEN without the surrogate |
| `random_rw` | Rand-RW | Random | reaction-uniform random anchor |
| `mcts` | MCTS | Metaheuristic | Monte Carlo tree search made global |
| `hc` | HC | Metaheuristic | hill-climbing made global |
| `random_pw` | Rand-PW | Random | product-weighted random anchor (DCRF ≈ 1 cost baseline) |

Appendix / reference rows: `spacega`, `hasten`, `molpal`, `ts_walters`
(all **external**), and the registered ablations `fm`, `thompson_slot`,
`slot_ucb`. `synthonbench list-methods` prints the full roster with families.

> **Starred (`*`)** rows are benchmark-native analogues *inspired by* the named
> systems' public descriptions, not source-faithful reproductions. `ise` is a
> SpaceHASTEN-*style* analogue, not the SpaceHASTEN source adapter.

### External methods (optional, not bundled)

`sh`, `shc`, `spacega`, `hasten`, `molpal` and `ts_walters` are third-party or
licensed systems the paper drove through their own upstream code. They are in
the roster for completeness and result reproduction; running them means cloning
the upstream repo (and, for some, commercial software). Nothing is bundled.

```bash
synthonbench fetch                 # list the optional backends + their licenses/URLs
synthonbench fetch spacega         # git-clone one into ~/.cache/synthonbench/repos
```

| key | upstream | license | also needs |
|-----|----------|---------|------------|
| `spacehasten` (`sh`,`shc`) | TuomoKalliokoski/SpaceHASTEN | Proprietary (Orion) | Glide, BiosolveIT FTrees/CoLibri |
| `hasten` | TuomoKalliokoski/HASTEN | upstream | Glide, ChemProp/XGBoost |
| `spacega` | lmoesgaard/SpaceGA | MIT | BiosolveIT SpaceLight |
| `thompson_sampling` | PatWalters/TS | MIT | RDKit |
| `molpal` | coleygroup/molpal | MIT | RDKit, PyTorch |
| `rdkit_fp` | mireklzicar/rdkit-fp (`pip install rdfp`) | BSD-3 | RDKit |

`create_optimizer("sh", ...)` raises a clear, actionable `ExternalMethodError`
with the fetch + install instructions rather than silently failing.

## Datasets (Hugging Face)

Frozen tasks, score tables, synthon spaces and shared warm-start seeds live in
[`mireklzicar/synthonbench`](https://huggingface.co/datasets/mireklzicar/synthonbench).

```bash
synthonbench download --scale 1M                     # one scale
synthonbench download --scale all                    # incl. the 100M table (~5 GB)
synthonbench download --scale 10M --what scores      # just the 10M surrogate table
```

| group | files | notes |
|-------|-------|-------|
| `spaces/` | `synthon_space_{1M,10M,100M}.synthons.tsv` + properties + `reactions.tsv` | 42 canonical reactions (SMIRKS) |
| `scores/glide_1M.parquet` | real-Glide LE oracle, **3 targets** | headline docking track (medchem-filtered, −Glide−cLogP) |
| `scores/surrogate_{1M,10M,100M}.parquet` | LightGBM surrogate, 3 targets | scale tracks |
| `seeds/seeds_<target>_s{0,1,2}.parquet` | 3 × 100k shared warm-start pools / target | one per protocol seed |

Targets: **KIF11**, **PYRD**, **TGFR1**. Scales: 990,610 / 10,022,100 /
99,851,700 products. All tables key on the same md5 `product_id`; the surrogate
and seed tables also carry the `synthon{1,2,3}_id` tuple.

## The DCRF metric

The headline efficiency metric is the **top-k DCRF**:

```
recall@k = |observed ∩ exact_top_k| / k
DCRF_k   = recall@k / (B / N)          # = recall@k * N / B
```

where `B` is the charged (unique) oracle-call budget and `N` the number of
products in the (allowed) space. Random search has `DCRF = 1` in expectation;
the exact oracle gives the ceiling. The paper audits the **scale-normalized
top-0.1%**: top-1000 @ 1M, top-10,000 @ 10M, top-100,000 @ 100M.

- For small spaces, `synthonbench.metrics.compute_exact_audit` enumerates and
  ranks the space exactly (used by `run_method`).
- At 1M/10M/100M, `scripts/score_submission.py` ranks the released score table
  directly — no enumeration — and scores a submission's recall@k / DCRF@k.

## Method presets & curricula

`synthonbench.presets` carries the **exact camera-ready hyperparameters** (the
appendix table) mapped onto the optimizer constructors, plus the protocol
constants:

```python
from synthonbench import preset_kwargs, build_with_preset
from synthonbench.presets import BUDGET_SWEEP, SEEDS, TOPK_BY_SCALE

preset_kwargs("ga")     # {'population_size': 96, 'mutate_p': 0.35, ...}
BUDGET_SWEEP            # (250, 500, 1000, 2500, 5000, 10000, 100000)
SEEDS                  # (0, 1, 2, 3, 4)   — protocol minimum is (0, 1, 2)
TOPK_BY_SCALE          # {'1M': 1000, '10M': 10000, '100M': 100000}
```

## Reproduction scripts

See [`scripts/`](scripts/):

- [`scripts/score_submission.py`](scripts/score_submission.py) — at-scale DCRF scorer.
- [`scripts/fingerprints/compute_descriptors.py`](scripts/fingerprints/compute_descriptors.py)
  — RDKit physchem descriptors + ECFP4 fingerprints via [`rdfp`](https://github.com/mireklzicar/rdkit-fp).
- [`scripts/colibri/`](scripts/colibri/) — build BioSolveIT CoLibri/infiniSee
  `.space` files from the synthon TSVs (for the SpaceHASTEN/SpaceGA/FTrees
  backends; requires a CoLibri license).

## Practical notes — runtime & memory

Native baselines are pure-Python `ask`/`tell` loops; cost scales with the
**budget** (each oracle call is an `O(1)` table lookup), not with the space
size. Order-of-magnitude guidance on one modern core:

| scale | score table on disk | RAM to load/score | notes |
|-------|---------------------|-------------------|-------|
| 1M    | ~47 MB (surrogate) / ~93 MB (glide) | < 1 GB | frozen oracle dict builds in seconds |
| 10M   | ~480 MB | ~2–4 GB | read only the columns you need |
| 100M  | ~4.6 GB | **~16–32 GB** | load the `product_id` + one score column only; this is the RAM wall |

A 10,000-budget native run on the 1M oracle finishes in seconds to a couple of
minutes; `[ml]`/surrogate methods (RF/ISE) are dominated by periodic refits.
External backends (Glide docking, FTrees) are orders of magnitude heavier and
governed by their own tooling.

## Development

```bash
git clone https://github.com/mireklzicar/synthonbench && cd synthonbench
pip install -e ".[dev]"
ruff check src tests
pytest -q
python -m build
```

## Status

The camera-ready paper describes the benchmark design, tasks, oracles and full
baseline results. This repository is the public home for the package, the
reference baselines, the presets, and the reproduction scripts; the datasets are
on the [Hub](https://huggingface.co/datasets/mireklzicar/synthonbench).

## Links

- **Project / demo:** https://synthonbench.mireklzicar.com/
- **Dataset:** https://huggingface.co/datasets/mireklzicar/synthonbench
- **Companion packages:**
  [SynthonOR](https://github.com/mireklzicar/synthonor) (synthon similarity
  search), [SynthonMap](https://github.com/mireklzicar/synthonmap) (embedding +
  density rendering), [rdkit-fp](https://github.com/mireklzicar/rdkit-fp)
  (`rdfp`, fingerprints).

## Citation

If you use SynthonBench, please cite it using the metadata in
[`CITATION.cff`](CITATION.cff).

## License

Released under the [MIT License](LICENSE). External backends and datasets carry
their own licenses (see above).
