Metadata-Version: 2.4
Name: ltm-eval
Version: 0.1.0
Summary: Evaluation for large tabular models (LTMs), generative models of tabular data, with calibrated uncertainty.
Project-URL: Homepage, https://github.com/granica-ai/ltm-eval
Author: Granica Research
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: bootstrap,evaluation,metrics,synthetic-data,tabular
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering
Requires-Python: >=3.10
Requires-Dist: joblib>=1.1
Requires-Dist: numpy>=1.23
Requires-Dist: pandas>=1.5
Requires-Dist: scikit-learn>=1.2
Requires-Dist: scipy>=1.9
Provides-Extra: datasets
Description-Content-Type: text/markdown

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/granica-ai/ltm-eval/main/assets/granica-wordmark-dark.svg">
    <img alt="Granica" src="https://raw.githubusercontent.com/granica-ai/ltm-eval/main/assets/granica-wordmark-light.svg" height="40">
  </picture>
</p>

# ltm-eval

Evaluation for large tabular models (LTMs), generative models of tabular data, with calibrated uncertainty.

Most evaluators of generative tabular models report a single number — a Shape score of `0.92`, a precision of `0.97` — and leave you guessing. Is that good? Is it distinguishable from noise? Would it change at a different sample size? A bare value can't say.

`ltm-eval` reports every metric as three quantities (see [`docs/methodology.md`](docs/methodology.md) for the approach and the exact algorithm, and how the code computes them):

- **value** — the metric computed on your data;
- **confidence interval** — a two-sided bootstrap interval, so two models that differ only by sampling noise don't look different;
- **ideal value** — what two independent samples of the real holdout score against each other *at this sample size*. This, **not** a metric's theoretical best score (e.g. `1.0`), is the achievable target: with finite samples even a perfect model can't reach the theoretical best.

Together, these outputs allow you to interpret a metric finite-sample effects instead of against an out-of-reach perfect score. Every result also carries **reliability flags**, so the library never reports a number without the context to trust it.

**Holdout and generated sets.** Every evaluation takes two inputs: a **holdout set** — real data your model never saw in training, the stand-in for "real" — and a **generated set** — your model's synthetic output. Each metric scores the generated set against the holdout, while the **ideal value** is computed from the holdout alone (using two independent sub-samples, scored against each other).

## Installation

```bash
pip install ltm-eval
```

The core depends only on `numpy`, `pandas`, `scipy`, `scikit-learn`, and `joblib`. It is CPU-only; there is no `torch` or CUDA in the default path.

## Quickstart

Run the bundled example — it needs no data of your own. It pulls a real dataset from OpenML and scores a stand-in generator against the held-out real split:

```bash
python examples/openml_quickstart.py        # defaults to the "adult" dataset
```

It prints each metric's value beside its ideal value:

```
adult (OpenML id 179): train=24421 rows, holdout=24421 rows, 15 columns
[tutorial] generated = noised resample of train (not a trained model), so it should score below the ideal value.
config: n_bootstrap=50, max_eval_n=1000, seed=0 (max_eval_n is set only to keep this demo fast: it caps the per-side sample size to 1000, so each metric is computed on <= 1000 rows/side; the header's 'inputs' are the raw row counts)
ltm-eval report   (inputs: holdout=24421, generated=24421 rows; each metric computed on 1000 rows/side; B=50, 95% CI)

Sanity (Data Validity)
  NaN Cell% (Syn)        0.008441 —
  NaN Row% (Syn)         0.070964 —
  ...

Fidelity (Distribution Similarity)
  Shape                  0.9711 [0.9645, 0.9771]   ideal 0.9630 [0.9532, 0.9676]
  Trend                  0.9106 [0.9014, 0.9181]   ideal 0.9229 [0.9099, 0.9304]

Diversity (PRDC — Manifold Coverage)
  Precision              0.9261 [0.9041, 0.9489]   ideal 0.9194 [0.8955, 0.9382]
  Recall                 0.9048 [0.8855, 0.9221]   ideal 0.8956 [0.8751, 0.9202]
  ...

Conditional (Subpopulation Fidelity)
  Cond Shape             0.8928 [0.8823, 0.8991]   ideal 0.8875 [0.8788, 0.8957]
  Cond Shape (Rare)      0.8263 [0.8125, 0.8368]   ideal 0.8143 [0.8009, 0.8300]
  ...

flags (* marks an unreliable value):
  CATEGORY_VOCAB_MISMATCH — the generated set has categories absent in the holdout (or vice versa)
```

Each row is one metric: its value (with CI) and the ideal value (with CI). 

### Use it on your own data

`evaluate(holdout, generated)` takes two `pandas.DataFrame`s sharing the same columns; row counts need not match:

```python
import pandas as pd
from ltm_eval import evaluate, Schema

holdout   = pd.read_csv("real_holdout.csv")   # an unseen real holdout — the proxy for "real"
generated = pd.read_csv("generated.csv")      # your model's output; row count need not match

schema = Schema(numeric=[...], categorical=[...])   # optional; inferred and recorded if omitted
report = evaluate(holdout, generated, schema=schema)

print(report.summary())          # console table (the format shown above)
report.to_dataframe()            # tidy long form for analysis
report.to_markdown("eval.md")    # report table
report.to_json("eval.json")      # round-trippable
```

The unseen holdout is what the ideal depends on: train your model on real `train`, then score `generated` against a `holdout` it never saw. `load_openml` hands you that split directly, so you can try the workflow on real data without your own files:

```python
from ltm_eval import evaluate, load_openml

train, holdout = load_openml("adult")     # deterministic (train, holdout) split of real data
# ... train your model on `train`, sample `generated`, then: evaluate(holdout, generated) ...
```

## Metrics (v1)

All v1 metrics are *sample-based* — a statistic of the samples, so a bootstrap replicate is just "resample and recompute".

| Metric | Category | What it measures |
|---|---|---|
| Shape | fidelity | per-column **marginal** similarity — KS complement (numeric), total-variation complement (categorical) |
| Trend | fidelity | **pairwise-association** similarity over column pairs — correlation similarity (numeric–numeric), contingency similarity otherwise |
| PRDC | diversity | k-NN **manifold overlap** of the two point clouds: Precision, Recall, Density, Coverage |
| Conditional | conditional_fidelity | Shape, Trend & Recovery **within subpopulation slices**, averaged equally per slice (so rare slices count), reported over all slices and rare-only |
| Sanity | sanity | always-on data-validity checks: NaN / duplicate rates, size ratio, unseen-category count |

The statistical primitives (KS complement, TV complement, correlation similarity, contingency similarity) are implemented directly on numpy/scipy/scikit-learn. 

## What the library guarantees

1. If inputs pass validation, you get a `Report`. The library does not crash on data *content*.
2. Every number carries a CI (or a flag explaining its absence) and, where meaningful, an ideal value (computed on the holdout) at matched sizes — so the value and the ideal are directly comparable.
3. Results are deterministic given `(inputs, schema, config, seed)`.
4. A column's role is never silently coerced, and quantities are never compared at mismatched sample sizes, without a flag recording it.

The boundary between user error and data quality is explicit: malformed *input* (wrong type, mismatched columns, missing target) **raises** with an actionable message; degenerate but valid *data* (too few rows, constant columns, unseen categories) produces a **flagged** result. The full catalogue of handled failure modes is in [`docs/failure-modes.md`](docs/failure-modes.md).

## Documentation

> **Humans:** start with this README, [`CONTRIBUTING.md`](CONTRIBUTING.md), and [`docs/methodology.md`](docs/methodology.md).
> **AI agents:** the rest of [`docs/`](docs/) (usage, architecture, failure modes) is the detailed exposure; read it before you reason about the code.

- [`docs/methodology.md`](docs/methodology.md) — the approach: probability metrics, bootstrap uncertainty, and the ideal value.
- [`docs/usage.md`](docs/usage.md) — inputs, schema, the CLI, configuration, and data cleaning.
- [`docs/architecture.md`](docs/architecture.md) — how `evaluate()` flows through the modules; repository layout.
- [`docs/failure-modes.md`](docs/failure-modes.md) — the catalogue of statistical and "in the wild" failure modes, each with a covering test.
- [`tests/README.md`](tests/README.md) — the test layers (unit → property → metamorphic → calibration → fuzz → perf) and how to run them.

## Citation

If you use `ltm-eval` in your work, please cite:

```bibtex
@misc{ltm-eval,
  title   = {Large Tabular Models: The Next Frontier},
  author  = {Granica Research Team},
  year    = {2026},
  publisher = {GitHub},
  journal = {GitHub repository},
}
```

## Contributing

Contributions are welcome — see [`CONTRIBUTING.md`](CONTRIBUTING.md) for dev setup, the test layers, and how to add a metric.

## Project status

`0.x`, API stabilising. Versioned with SemVer; licensed under Apache-2.0.

## Changelog

Release notes are in [`CHANGELOG.md`](CHANGELOG.md), following Keep a Changelog + SemVer.
