Metadata-Version: 2.4
Name: mbe-eval
Version: 0.4.0
Summary: Marginal Baseline Evaluation for auditing machine-learning training metrics.
Author-email: Aparajeet Shadangi <aparajeet.shadangi@proton.me>
License-Expression: MIT
Project-URL: Homepage, https://github.com/AparajeetS/marginal-baseline-eval
Project-URL: Source, https://github.com/AparajeetS/marginal-baseline-eval
Project-URL: Issues, https://github.com/AparajeetS/marginal-baseline-eval/issues
Project-URL: Documentation, https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/README.md
Project-URL: Evidence, https://github.com/AparajeetS/marginal-baseline-eval/blob/master/SUPPORTING_EVIDENCE.md
Project-URL: Research program, https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/MBE_2_RESEARCH_PROGRAM.md
Project-URL: Replication protocol, https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/INDEPENDENT_REPLICATION_PROTOCOL.md
Project-URL: Kaggle notebook, https://www.kaggle.com/code/aparajeetshadangi/audit-ml-training-metrics-with-mbe
Keywords: machine learning,metrics,evaluation,generalization,partial correlation,reproducibility
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: scipy>=1.10
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Requires-Dist: torchvision>=0.15; extra == "torch"
Provides-Extra: checkpoint
Requires-Dist: h5py>=3.10; extra == "checkpoint"
Provides-Extra: flexible
Requires-Dist: scikit-learn>=1.3; extra == "flexible"
Provides-Extra: plot
Requires-Dist: matplotlib>=3.7; extra == "plot"
Requires-Dist: seaborn>=0.12; extra == "plot"
Provides-Extra: examples
Requires-Dist: torch>=2.0; extra == "examples"
Requires-Dist: torchvision>=0.15; extra == "examples"
Requires-Dist: scikit-learn>=1.3; extra == "examples"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4; extra == "dev"
Requires-Dist: scikit-learn>=1.3; extra == "dev"
Dynamic: license-file

# Marginal Baseline Evaluation

[![PyPI](https://img.shields.io/pypi/v/mbe-eval.svg)](https://pypi.org/project/mbe-eval/)
[![CI](https://github.com/AparajeetS/marginal-baseline-eval/actions/workflows/ci.yml/badge.svg)](https://github.com/AparajeetS/marginal-baseline-eval/actions/workflows/ci.yml)
[![Python](https://img.shields.io/pypi/pyversions/mbe-eval.svg)](https://pypi.org/project/mbe-eval/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/LICENSE)

**Marginal Baseline Evaluation (MBE)** is an audit protocol for testing whether
machine-learning training metrics still predict held-out performance after
controlling for ordinary baselines such as learning rate, weight decay,
optimizer, architecture, task, and training-state measurements.

The active research direction treats reliability as conditional on the target,
baseline information, and environment. MBE is being developed into a public
reliability atlas and an abstaining selector that recommends metrics only when
task-specific or calibrated transfer evidence supports them.

The project started from a concrete failure mode: a proposed metric can look
promising under raw pooled correlation while actually tracking easier baselines,
training loss, architecture mix, or other design variables. MBE makes that
failure visible by comparing raw association against controlled partial
rank-correlation.

## Current Status

This is an active research direction with an accompanying Python package,
Kaggle-scale experiment artifacts, and a public walkthrough notebook.

**Software status:** `mbe-eval` v0.4.0 implements the stable MBE v1
partial-rank audit. MBE 2.0 is the active research design and is not yet fully
implemented or empirically validated. See the [open research inventory](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/OPEN_RESEARCH.md)
for a precise map of what is available now.

The current direction is **MBE 2.0**, a multi-environment metric-validation
framework that separates unconditional association, incremental information,
transport, intervention response, and measurement reliability. The technical
[research program](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/MBE_2_RESEARCH_PROGRAM.md), gated [JMLR roadmap](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/JMLR_MILESTONE_ROADMAP.md),
and [compute plan](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/COMPUTE_AND_COST_PLAN.md) are the source of truth for new work.

MBE does not claim that partial correlation or hyperparameter conditioning is
new. The proposed contribution is their calibrated integration with a baseline
information ladder, five explicit estimands, deliberately deceptive controls,
configuration-blocked uncertainty, environment transport, matched
interventions, measurement reliability, and scoped metric claim cards. This
novelty claim must pass the public-corpus comparison gate or be narrowed.

- Package: [`mbe-eval`](https://pypi.org/project/mbe-eval/)
- Public notebook: [Audit ML Training Metrics with MBE](https://www.kaggle.com/code/aparajeetshadangi/audit-ml-training-metrics-with-mbe)
- Documentation map: [docs/README.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/README.md)
- Conditional reliability protocol: [atlas, selector, and abstention design](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/CONDITIONAL_METRIC_RELIABILITY_PROTOCOL.md)
- Metric audit service: [open-core product boundary](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/METRIC_RELIABILITY_AUDIT_SERVICE.md)
- Credibility ledger: [passed, failed, fixed, and blocked evidence gates](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/MBE_CREDIBILITY_LEDGER.md)
- Shared method comparison: [CMI, granulated Kendall, rank, and MBE benchmark](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/experiments/10_method_comparison/README.md)
- Preregistration and claim gate: [credibility freeze](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/experiments/11_credibility_freeze/PREREGISTRATION.md)
- Statistical specification: [estimand, inference, and assumptions](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/STATISTICAL_ESTIMAND_AND_INFERENCE.md)
- Paper workspace: [JMLR manuscript skeleton and generated tables](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/paper/README.md)
- Evidence ledger: [SUPPORTING_EVIDENCE.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/SUPPORTING_EVIDENCE.md)
- Reproducibility notes: [REPRODUCIBILITY.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/REPRODUCIBILITY.md)
- Legacy v1 protocol: [PROTOCOL_FREEZE.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/PROTOCOL_FREEZE.md)
- Metric taxonomy: [METRIC_TAXONOMY.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/METRIC_TAXONOMY.md)
- Figures: [FIGURES.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/FIGURES.md)
- Project brief: [PROJECT_BRIEF.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/PROJECT_BRIEF.md)
- Superseded v1 experiment protocol: [NEXT_EXPERIMENT_PROTOCOL.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/NEXT_EXPERIMENT_PROTOCOL.md)
- Paper skeleton: [PAPER_SKELETON.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/PAPER_SKELETON.md)
- Paper notes: [PAPER.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/PAPER.md) and [PUBLICATION_STRATEGY.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/PUBLICATION_STRATEGY.md)
- Contribution guide: [CONTRIBUTING.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/CONTRIBUTING.md)
- Open research inventory: [OPEN_RESEARCH.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/OPEN_RESEARCH.md)
- Grant execution plan: [GRANT_EXECUTION_PLAN.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/GRANT_EXECUTION_PLAN.md)
- Independent replication protocol: [docs/INDEPENDENT_REPLICATION_PROTOCOL.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/docs/INDEPENDENT_REPLICATION_PROTOCOL.md)
- Governance: [GOVERNANCE.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/GOVERNANCE.md)
- Roadmap: [ROADMAP.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/ROADMAP.md)

## Legacy Pilot Evidence

The existing **680-row pilot ledger** is exploratory evidence, not a
submission-grade independent model sample. It includes repeated configurations,
and the text experiment lacks a causal attention mask and permits label leakage.
Its results motivate the new protocol but must not support confirmatory claims.

The minimum corrected scale design is explicit: 240 image runs
(`2 datasets x 3 architectures x 8 configurations x 5 seeds`) plus 100
causally masked text runs
(`1 dataset x 2 model sizes x 10 configurations x 5 seeds`). The 340 total is
a blocked factorial design, not a claim of 340 independent observations. See
[GRANT_EXECUTION_PLAN.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/GRANT_EXECUTION_PLAN.md).

The ledger contains:

- 480 CIFAR-10 image models across CNN, ResNet, ViT, and WideResNet settings.
- 200 character-transformer language models.
- 40+ candidate metrics including gradient/Fisher metrics, feature metrics,
  confidence/calibration metrics, sharpness metrics, weight norms, and
  distance/update proxies.

## What The Evidence Suggests

The current Kaggle-scale runs support a selective audit story:

- MBE is selective rather than indiscriminate; many metrics retain signal under the declared controls.
- Several validation, confidence/logit, gradient/Fisher magnitude, and
  task-proximal metrics survive.
- Several feature-rank, weight-norm, distance/update, and sharpness/noise-scale
  metrics weaken, wash out, or invert under controls.
- FIM_norm is the motivating case study: it looked promising under conventional
  metric validation, then became task-dependent under MBE.

FIM_norm summary from the legacy pilot pool:

| Audit | n | Raw rho | MBE partial rho | Class |
|---|---:|---:|---:|---|
| Image only, default controls | 480 | -0.662 | -0.218 | survives |
| Image only, strict + validation loss | 480 | -0.662 | -0.383 | survives |
| Text only, default controls | 200 | -0.291 | +0.014 | washout |
| Text only, strict + validation loss | 200 | -0.291 | +0.188 | weak-or-mixed |
| Full image+text pool, default controls | 680 | +0.225 | -0.203 | reverse-inversion |
| Full image+text pool, strict + validation loss | 680 | +0.225 | -0.300 | reverse-inversion |

Full result tables and interpretation are in
[SUPPORTING_EVIDENCE.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/SUPPORTING_EVIDENCE.md).
CPU-only bootstrap confidence intervals and threshold sensitivity are summarized
in the no-compute uncertainty report listed from the reproducibility guide.

## Install

```bash
pip install mbe-eval
```

Supported Python versions are 3.9 and newer; CI currently exercises 3.9, 3.11,
3.13, and 3.14.

Optional FIM_norm extraction utilities require PyTorch:

```bash
pip install "mbe-eval[torch]"
```

For local development:

```bash
git clone https://github.com/AparajeetS/marginal-baseline-eval.git
cd marginal-baseline-eval
pip install -e ".[dev]"
```

## Try It In One Command

After installation:

```bash
mbe-eval-demo --bootstrap 200
```

This runs a CPU-only synthetic audit, prints the MBE table, and writes
`mbe_demo_report.md`. The demo is intentionally small; replace the synthetic
dataframe with your training-run ledger for real experiments.
Use `--no-output` if you only want the printed table.

To audit your own CSV ledger:

```bash
mbe-eval-audit \
  --csv runs.csv \
  --metrics fim_norm,val_loss_ep20,grad_norm \
  --target test_accuracy \
  --controls learning_rate,weight_decay,optimizer,arch \
  --groupby task \
  --bootstrap 200 \
  --seed 42 \
  --output audit_report.md \
  --results audit_results.json
```

The CLI fails closed if a requested metric, target, control, or grouping column
is absent. Use `--results audit_results.csv` or `--results audit_results.json`
for machine-readable output suitable for experiment pipelines and AI agents.

## Basic API

```python
import pandas as pd
from mbe_eval import audit_metrics, audit_report_markdown

df = pd.DataFrame(
    {
        "fim_norm": [0.42, 0.51, 0.37, 0.65, 0.62, 0.35],
        "val_loss_ep20": [1.2, 0.9, 1.4, 0.7, 0.8, 1.5],
        "learning_rate": [1e-3, 1e-3, 3e-4, 3e-4, 1e-4, 1e-4],
        "arch": ["cnn", "cnn", "resnet", "resnet", "vit", "vit"],
        "test_accuracy": [0.71, 0.78, 0.68, 0.82, 0.80, 0.66],
    }
)

report = audit_metrics(
    df,
    metrics=["fim_norm", "val_loss_ep20"],
    target="test_accuracy",
    controls=["learning_rate", "arch"],
    bootstrap=100,
)

print(report[["metric", "raw_r", "partial_r", "classification"]])
print(audit_report_markdown(report, target="test_accuracy", controls=["learning_rate", "arch"]))
```

Your dataframe should have one row per trained model/run, one held-out target,
candidate metric columns, and baseline/design columns to control.

## Reproduce Current Tables

The main paper-scale audit can be regenerated from saved result CSVs. See
[REPRODUCIBILITY.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/REPRODUCIBILITY.md) for the exact legacy artifact command.

The public notebook source lives in:

```bash
kaggle/mbe_metric_audit/how_to_audit_ml_training_metrics_mbe.ipynb
```

Kaggle training scripts and raw result manifests are documented in
[REPRODUCIBILITY.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/REPRODUCIBILITY.md) and the large-scale artifact manifest.

## Repository Layout

```text
marginal-baseline-eval/
+-- mbe_eval/                  # installable MBE package
+-- examples/                  # small local examples
+-- experiments/               # paper-scale and exploratory experiments
+-- figures/                   # generated no-compute evidence figures
+-- kaggle/mbe_metric_audit/   # public Kaggle notebook source
+-- docs/                      # documentation index
+-- SUPPORTING_EVIDENCE.md     # run-by-run evidence ledger
+-- REPRODUCIBILITY.md         # reproduction commands and expected artifacts
+-- PAPER.md                   # evolving paper direction
+-- PUBLICATION_STRATEGY.md    # publication strategy notes
```

## Research Claim

The claim is not that any one metric is universally good or bad. The active
claim is narrower and more useful:

> Metric reliability is conditional on a declared target, baseline information
> set, and environment. MBE audits that scope and tests whether it supports a
> useful recommendation or an honest abstention.

This is a methodological hypothesis under active validation. The current PGDL
pilot checks implementation behavior; it does not establish a general metric
router, universal metric failure, or causal effects.

## Citation

```bibtex
@article{shadangi2026mbe,
  title={Marginal Baseline Evaluation for Auditing Generalization Metrics},
  author={Shadangi, Aparajeet},
  year={2026},
  note={Preprint and open-source research artifact}
}
```

## License

MIT License. See [LICENSE](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/LICENSE).

## Community And Maintenance

Scientific challenges and independent replications are welcome. See
[CONTRIBUTING.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/CONTRIBUTING.md), [GOVERNANCE.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/GOVERNANCE.md),
[CODE_OF_CONDUCT.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/CODE_OF_CONDUCT.md), and [SECURITY.md](https://github.com/AparajeetS/marginal-baseline-eval/blob/master/SECURITY.md).
