Metadata-Version: 2.5
Name: forestgump
Version: 0.1.1
Summary: Life is like a box of baselines. Did your cheminformatics model beat a random forest trained on Morgan fingerprints?
Author-email: Jenny Luo <jennyluo@berkeley.edu>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: baselines,cheminformatics,model-evaluation,morgan-fingerprint,random-forest,rdkit
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Scientific/Engineering :: Chemistry
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: rdkit>=2023.9
Requires-Dist: scikit-learn>=1.3
Requires-Dist: scipy>=1.10
Provides-Extra: test
Requires-Dist: pytest>=8; extra == 'test'
Description-Content-Type: text/markdown

# forestgump

*Life is like a box of baselines.*

**Morgan-fingerprint baselines for cheminformatics models.**

You trained a molecular property model for 3 days on 8×H100. Did it beat a random forest trained on Morgan
fingerprints?

forestgump takes your molecules as SMILES, plus your labels, your cross-validation folds and your model's
out-of-fold predictions. It fits three baselines on **exactly the same folds**: the training mean, Tanimoto kNN on
Morgan fingerprints, and a random forest on Morgan fingerprints. Name a context column, such as an assay, a
chemical series or a reaction's catalyst, and it adds a fourth: the context mean. Then it runs a paired test to
decide whether your model beat the forest.

```
Your model, trained on 8×H100 for 3 days, lost to a random forest. The forest trained in 31 seconds on a CPU.
The forest has no GPU. The forest has no feelings. The forest won.
```

## Install

```bash
pip install forestgump
```

## Try it

```bash
forestgump demo                             # a model that predicts noise. The forest wins.
forestgump demo --model oracle              # knows the answer, beats the forest on all 5 folds, and only ties
forestgump demo --model oracle --repeats 3  # 15 folds is enough evidence. You win.
```

The demo molecules are 105 para-disubstituted benzenes. Their labels are RDKit's Crippen logP.

The tie is not a bug. Five folds is not much evidence, and the test is built not to call a winner without it.

## Use it

You need one CSV with one row per molecule. Its columns are:

| Column | Meaning |
|---|---|
| `smiles` | the molecule (or several columns, one per reaction component; see below) |
| `target` | the label: a number, or one of two classes |
| `fold` | which test fold the row belongs to |
| `pred` | your model's prediction for the row, made while it was in the test fold. For a two-class target, this is the probability of the larger class label. |

```bash
forestgump run my_results.csv
forestgump run my_results.csv --smiles SMILES --target y --fold cv_fold --pred my_model
```

For **repeated** cross-validation, add one row per (molecule, repeat) and pass `--repeat repeat_column`. The test
set of (repeat r, fold f) is those rows. Its training set is every other row of repeat r.

For **reactions or mixtures**, name one SMILES column per component. Each column gets its own Morgan fingerprint
and the fingerprints are concatenated, so a substructure in the ketone and the same substructure in the ligand set
different bits. kNN's Tanimoto and the forest both use the concatenated fingerprint.

```bash
forestgump run my_results.csv --smiles ligand ketone base
```

To add the context mean, name one or more context columns. With several, a group is one combination of their values.

```bash
forestgump run my_results.csv --context catalyst base
```

From Python:

```python
import forestgump

report = forestgump.run(df, smiles="smiles", target="target", fold="fold", pred="pred", context=["catalyst", "base"])
print(report.verdict)  # "win", "lose", "tie", or "refuse"
print(report.render())
report.to_json("forest.json")
```

## The baselines

Each baseline is retrained on every training set, and its hyperparameters are picked from **that training set
only**. Test rows are only ever predicted.

- **The Mean** has nothing to tune. It predicts the training-set mean for every test row, or for a two-class
  target the training share of the larger class.
- **Retrieval-Augmented Mean** appears only with `--context`, and it has nothing to tune either. For each test
  row it retrieves the training rows with the same context values and averages their labels. A group with no
  training rows gets the training-set mean, and the report says how many test rows that happened to.
- **Large Lookup Model** uses inner cross-validation, which takes milliseconds for kNN: 5 folds, or 3 below 50
  training rows.
- **The Forest** fits each of its 12 configs once and keeps the one with the best out-of-bag score. Every tree
  skips about a third of the training rows, so the forest gets a validation score for free. If you'd rather tune
  it like everything else, `--forest-inner-cv` uses inner cross-validation instead. That fits each config once
  per inner fold and then refits the winner, so it is about 5× slower.

| Name | What it is | Grid |
|---|---|---|
| The Mean | the training-set mean, or the class prior | none (zero parameters, zero shots, zero GPUs) |
| Retrieval-Augmented Mean | only with `--context`: the mean label of the training rows in the test row's group, or the training-set mean if the group has no training rows | none (it is a groupby) |
| Large Lookup Model | k-nearest neighbors with Tanimoto distance | k ∈ {1, 3, 5, 10} × {uniform, distance} weights × radius ∈ {2, 3} |
| The Forest | a random forest with 500 trees | max_features ∈ {sqrt, 0.3, 1.0} × min_samples_leaf ∈ {1, 3} × radius ∈ {2, 3} |

Both fingerprint baselines use 2048-bit Morgan fingerprints from RDKit.

## How the verdict is decided

The primary metric is RMSE for numeric targets and ROC-AUC for two-class targets. The report also gives MAE, R²
and Spearman ρ, or accuracy.

1. Every method is scored on every test set. The full per-test-set table is printed **before** any verdict.
2. Your model is compared with each baseline on the per-test-set differences, using the **Nadeau–Bengio corrected
   resampled t-test**. CV training sets overlap, so the differences are correlated and a plain paired t-test is
   overconfident. The correction inflates the variance by `1/J + n_test/n_train` over J test sets
   (Nadeau & Bengio 2003; Bouckaert & Frank 2004 for repeated CV).
3. The p-values, three or four with `--context`, are Holm-adjusted together.
4. The verdict compares you with the forest at α = 0.05: **win**, **lose**, or **tie**. With fewer than 3 test
   sets the forest declines to be judged.

The report also gives the mean difference, its confidence interval and Cohen's d_z. Use `--json` to get every
number, the hyperparameters chosen on each test set, and the baselines' predictions.

## Flags

| Flag | Effect |
|---|---|
| `--seriously` | plain output with no jokes, for logs and papers |
| `--brag "a 3-billion-parameter foundation model"` | changes how the forest describes your training |
| `--forest-inner-cv` | picks the forest's hyperparameters by inner CV instead of out-of-bag score. About 5× slower, and it warns you. |
| `--fail-if-forest-wins` | exits 1 if the forest won, so a CI pipeline goes red when your model loses |
| `--context catalyst base` | adds the context mean, grouping rows by these columns. The report says how many test rows had a group with no training rows and fell back to the training-set mean. |
| `--task regression\|binary` | overrides the task. By default a target with 2 values is binary. |
| `--seed 0` | the random seed for the forest's trees and the inner CV splits. Default 0. In `forestgump demo` it also sets the demo's folds and noise. |
| `--n-jobs -1` | how many CPU cores the forest uses. Default -1, meaning all of them. |
| `--alpha 0.05` | the significance level: the forest wins or loses only when the Holm-adjusted p is below it. It also sets the confidence intervals to 1 − alpha. Default 0.05. |

## License

Apache-2.0.
