Metadata-Version: 2.5
Name: foldguard
Version: 0.1.0
Summary: Catch evaluation leaks at runtime: checkpoint picks, calibrator fits and threshold tuning that quietly use the test fold.
Project-URL: Homepage, https://github.com/really-notabot/foldguard
Project-URL: Repository, https://github.com/really-notabot/foldguard
Project-URL: Documentation, https://github.com/really-notabot/foldguard/tree/main/docs
Project-URL: Changelog, https://github.com/really-notabot/foldguard/blob/main/CHANGELOG.md
Project-URL: Issues, https://github.com/really-notabot/foldguard/issues
Author: Aswin Jose
License-Expression: MIT
License-File: LICENSE
Keywords: ci,cross-validation,data-leakage,evaluation,leakage,machine-learning,reproducibility,taint-tracking,testing
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Framework :: Pytest
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Provides-Extra: dev
Requires-Dist: lightgbm; extra == 'dev'
Requires-Dist: pandas; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=8; extra == 'dev'
Requires-Dist: ruff>=0.5; extra == 'dev'
Requires-Dist: scikit-learn>=1.3; extra == 'dev'
Requires-Dist: xgboost; extra == 'dev'
Provides-Extra: sklearn
Requires-Dist: scikit-learn>=1.3; extra == 'sklearn'
Description-Content-Type: text/markdown

# foldguard

Your train/test split is fine. Your numbers are still too good.

There's a family of leaks that never shows up in a diff of the modelling code,
because the split itself is textbook. The eval fold never touches training. It just
quietly steers decisions:

- you kept the epoch that scored best on the test fold (FG-1)
- you fit the calibrator on the labels you then report against (FG-2)
- you picked the threshold by maximising F1 on the scores you're about to publish (FG-3)

Each of those inflates the number you report, and none of them look wrong when you
read the code. Existing leakage tools go after preprocessing and feature leakage.
Nothing I could find catches these at runtime, so I wrote this.

> Version 0.1.0, the first release. Early enough that the API may still move.

## Getting started

```
pip install foldguard[sklearn]
```

You mark the eval fold once (the library calls it tainting, after the taint-tracking
idea it borrows from security tooling) and run your protocol inside a guard. The
snippet below works as-is:

```python
import foldguard as fg
from sklearn import metrics as skm
from sklearn.datasets import make_classification
from sklearn.isotonic import IsotonicRegression
from sklearn.linear_model import LogisticRegression

X, y = make_classification(n_samples=400, n_features=20, random_state=0)
X_tr, X_te, y_tr, y_te = fg.sklearn.tainted_train_test_split(
    X, y, test_size=0.2, random_state=0
)

with fg.guard(action="record") as g:      # "raise" (the default) stops at the first one
    model = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
    scores = model.predict_proba(X_te)[:, 1]
    auroc = fg.report("auroc", skm.roc_auc_score(y_te, scores))

    IsotonicRegression().fit(scores, y_te)         # FIT_ON_EVAL (FG-2)
    if skm.roc_auc_score(y_te, scores) > 0.5:      # DECISION_ON_EVAL (FG-1)
        best_model = model

print(g.summary())
```

The first three lines inside the guard are fine. Fitting on the training half, scoring
the eval fold, reporting the result: that's what a test set is for. The last two are
not, and you get the file and line for each, plus a note on what the honest version
looks like.

One rule covers it. Computing on eval data is fine. Letting eval data change what your
pipeline *does* is not.

One wrinkle worth knowing early: use `from sklearn import metrics as skm`, not
`from sklearn.metrics import roc_auc_score`. A from-import that runs before any guard
exists grabs the unwrapped function, so nothing downstream of it carries a mark. The
`foldguard run` CLI and the pytest plugin patch things before your imports and don't
have this problem. In a plain script you'll get a `FoldguardEarlyBindWarning` if you
trip it, so at least it's loud.

## What it watches

**numpy.** Marks propagate through ufuncs, slicing and dispatched `np.*` calls. Two
sinks: `bool()` of a marked value (`DECISION_ON_EVAL`) and the argmax/argsort family
(`SELECT_ON_EVAL`). Per-sample class prediction (`probs.argmax(axis=1)`) is exempt,
since that's a prediction and not a choice. Flattening argmax, which is how threshold
tuning looks, still fires.

**sklearn.** Every `BaseEstimator` fit method is a sink. `predict`, `transform`,
`score` and all of `sklearn.metrics` carry marks through instead of complaining.
Data arriving through `eval_set`-style keywords gets its own message, because that's
early stopping on the test fold rather than fitting on it, and the fix is different.

**xgboost and lightgbm, including the native APIs.** `xgb.train`/`DMatrix` and
`lgb.train`/`Dataset` are covered, and the check happens before any boosting round.
The field test is why: the sklearn wrappers were caught from day one while the native
path sailed straight through.

**pandas.** `fg.taint()` takes a Series or DataFrame and hands one back. The mark
survives slicing, masking, arithmetic and `train_test_split`, and fit guards see it.
`.to_numpy()` still drops it, which is in the limitations below.

Audit output is deduplicated by location with counts, so a loop that trips the same
line twenty thousand times costs you three lines of summary rather than a megabyte.
I know that number because a pipeline in the field test did exactly that.

## What a violation actually means

A violation says eval data reached a fit, a branch or a selection on that line. That
part is a fact about your run, not a guess.

Whether it's a *leak* is a separate question, and it's one you answer by looking at
the line. Picking a checkpoint that way is a leak. Asserting your class balance is
sane before you score is not, even though both branch on eval data. So there are two
ways to say "I meant that":

```python
# a deliberate check
with fg.allow("data validation"):
    if np.isnan(y_test).any():
        raise SystemExit("bad labels in the eval fold")

# a hand-rolled metric that has to sort or bin its own inputs
@fg.allow_metric
def ece(y_true, probs, n_bins=10):
    ...

ece_val = ece(y_test, scores)   # still marked, so tuning on it is still caught
auroc = fg.report("auroc", skm.roc_auc_score(y_test, scores))
assert auroc > 0.7              # report first, then assert on the plain float
```

Both get logged with a reason and a location, so anyone auditing the run can see what
you waved through and decide whether they agree.

The reverse doesn't hold, and you should hold this against the tool: a clean run is
not proof of a clean protocol. See the limitations.

## The three leaks, side by side

```
$ python examples/hero.py
foldguard: three textbook evaluation leaks, one guard
=====================================================

leak   protocol             metric    leaky  honest  optimism   foldguard verdict
-------------------------------------------------------------------------------------------
FG-1   checkpoint on test   AUROC     0.789   0.726    +0.063   LeakError: DECISION_ON_EVAL
FG-2   calibrator on test   AUROC     0.767   0.734    +0.033   LeakError: FIT_ON_EVAL
FG-3   threshold on test    F1        0.727   0.681    +0.046   LeakError: SELECT_ON_EVAL
-------------------------------------------------------------------------------------------
Same data, same model, same split in every row. The only difference is
whether the test fold was allowed to steer a choice.
```

Don't read too much into the size of those gaps. They're one pinned seed. Selection
optimism is never negative but it can be zero if you get lucky, and on FG-1 four of
ten held-out seeds came out at exactly +0.000, because validation happened to pick the
same epoch the test fold would have. The protocol is broken either way. Some seeds
just don't send you the bill.

## In CI

```python
@pytest.mark.leakfree
def test_eval_protocol():
    run_full_evaluation()
```

```
foldguard run eval_pipeline.py --report foldguard-report.json
```

`foldguard run` exits 0 when clean, 2 on violations, 1 if your script raised, 64 for
bad usage. A script that opens its own guard runs fine under it; the inner guard folds
into the outer audit instead of erroring. There's a GitHub Action in
[`action.yml`](https://github.com/really-notabot/foldguard/blob/main/action.yml).

## Does it work on real code?

I ran it against 30 public evaluation pipelines across about 20 domains before
releasing it: 10 executed directly, 20 re-typed faithfully from their eval code. 23
had genuine protocol leaks. The other 7 were pipelines I picked *because* they looked
clean, to find out how often it cries wolf. The corpus stays anonymous, since the point was to test my tool
rather than to publish an audit of other people's work.

It flagged the leak on the exact line in 15 of the 23, and in 20 of 23 once the same
leak was expressed on surfaces it can see. The three it can't reach are structural
and described below. Median cost to integrate was 6 changed lines, worst case 14.

On the clean pipelines it stayed quiet about protocol, which is the result I cared
about most: fit-on-validation, nested CV, stacking and internal model selection all
passed without a word. It did fire 10 times, every one of them inside a hand-rolled
metric helper that had to branch or sort on its own inputs. The most common shape
(per-sample argmax) is now exempt, and the rest take a one-line `@fg.allow_metric`.

The useful finding was that every miss was a *transport* failure rather than a
judgement failure. The sink logic was right every time; the mark just didn't survive
the trip, through pandas containers, `np.array` copies, a framework's internal
`float()`, or a native GBDT entry point. Native GBDT and pandas ingestion are fixed as
of this release. The rest are documented route by route, because a leak that audits
clean and silent is the worst thing this tool could do.

## Limitations

It's a bug finder, not a proof. Two kinds of gap.

**Structural.** Mark the eval fold before anything supervised touches those rows. If
your vectorizer or feature filter is fit on the whole dataset before the split, the
leak happens before the mark exists and nothing here can see it. Cross-fold row
identity (duplicate, grouped or overlapping rows, FG-7) is a different mechanism and
out of scope. And a threshold you picked by eyeballing a plot is invisible to any
runtime tool.

**Erasure.** Some operations drop the mark. Where a hook exists, you get a deduplicated
`taint-erased` event so at least there's a breadcrumb: `float()`, `int()`, `.item()`,
`.tolist()`, formatting or `repr()` of a marked scalar, `np.array([...])` over a list
of marked scalars, pandas ingesting marked scalars, and `fg.report()` handed something
already unmarked.

Some are simply invisible, and each one is pinned by a test so it can't quietly change:
`np.asarray` over a marked array (numpy short-circuits subclasses in C), stacking a
marked array with a plain one, indexing a *plain* array with a marked index or mask,
writes into plain buffers via `out=`, `np.copyto` or slice assignment,
`Series.to_numpy()`, pickling, a framework's own `float()` boundaries such as HPO trial
storage, and anything training outside `sklearn.base.BaseEstimator`, which means torch
and keras loops.

High precision on what it flags. Incomplete recall on what it can reach.

## Docs

- [Design and architecture](https://github.com/really-notabot/foldguard/blob/main/docs/design.md)
- [The leak taxonomy, FG-1 to FG-7](https://github.com/really-notabot/foldguard/blob/main/docs/taxonomy.md)
- [Integration recipes](https://github.com/really-notabot/foldguard/blob/main/docs/recipes.md) for pandas, GBDTs, CV loops, HPO without a holdout, and torch

## Built with Claude

Claude (Anthropic) did most of the typing on this, via Claude Code. The taxonomy and
the design came out of working through the problem with it, and two of the things I'm
most confident about were run as multi-agent Claude workflows: an adversarial review
that found 21 defects in my first version, including a crash and a sink that could be
bypassed entirely, and the 30-pipeline field test above.

I set the scope, made the design calls, and decided what to believe. Every number in
this README came from a run I checked, and the claims got weaker rather than stronger
each time something was verified properly. The original README said every violation
was a true positive. That was wrong, and the review proved it with a counterexample in
about a minute.

## License

MIT
