Metadata-Version: 2.4
Name: mbe-eval
Version: 0.2.0
Summary: Marginal Baseline Evaluation for auditing generalization metrics.
Home-page: https://github.com/AparajeetS/metric-audit-paper-code
Author: Aparajeet Shadangi
Author-email: aparajeet.shadangi@proton.me
License: MIT
Project-URL: Source, https://github.com/AparajeetS/metric-audit-paper-code
Project-URL: Issues, https://github.com/AparajeetS/metric-audit-paper-code/issues
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: scipy>=1.10
Provides-Extra: torch
Requires-Dist: torch>=2.0; extra == "torch"
Requires-Dist: torchvision>=0.15; extra == "torch"
Provides-Extra: plot
Requires-Dist: matplotlib>=3.7; extra == "plot"
Requires-Dist: seaborn>=0.12; extra == "plot"
Provides-Extra: examples
Requires-Dist: torch>=2.0; extra == "examples"
Requires-Dist: torchvision>=0.15; extra == "examples"
Requires-Dist: scikit-learn>=1.3; extra == "examples"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: build>=1.0; extra == "dev"
Requires-Dist: twine>=4; extra == "dev"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: project-url
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# Metric Audit Paper Code

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

Code and Python package for experiments on **Marginal Baseline Evaluation (MBE)**: an audit protocol for testing whether a proposed generalization metric keeps predictive signal after ordinary training baselines and experimental design variables are controlled.

The current research direction is deliberately empirical. A metric can look useful under raw correlation, weaken after controls, invert sign, or survive in some architectures/tasks but not others. MBE is the framework for making those cases visible instead of treating one pooled correlation as the whole story.

## What This Repository Contains

- `mbe_eval/`: lightweight package for raw and partial rank-correlation audits.
- `examples/`: small demonstrations using FIM_norm and synthetic data.
- `experiments/`: paper-scale and exploratory experiments, including CIFAR-10, transformers/language-model probes, and Kaggle-scale runs.
- `PAPER.md` and `JMLR_STRATEGY.md`: evolving paper notes and publication strategy.

## Installation

Install the core MBE audit library:

```bash
pip install mbe-eval
```

For local development from this repository:

```bash
git clone https://github.com/AparajeetS/metric-audit-paper-code.git
cd metric-audit-paper-code
pip install -e .
```

The core install only requires NumPy, pandas, and SciPy. FIM_norm extraction uses PyTorch and is optional:

```bash
pip install "mbe-eval[torch]"
```

## Basic Usage

Audit several candidate metrics in one dataframe:

```python
import pandas as pd
from mbe_eval import audit_metrics

df = pd.DataFrame(
    {
        "fim_norm": [0.42, 0.51, 0.37, 0.65],
        "val_loss_ep20": [1.2, 0.9, 1.4, 0.7],
        "learning_rate": [1e-3, 1e-3, 3e-4, 3e-4],
        "weight_decay": [1e-4, 1e-5, 1e-4, 1e-5],
        "test_accuracy": [0.71, 0.78, 0.68, 0.82],
    }
)

report = audit_metrics(
    df,
    metrics=["fim_norm", "val_loss_ep20"],
    target="test_accuracy",
    controls=["learning_rate", "weight_decay"],
)

print(report[["metric", "raw_r", "partial_r", "classification"]])
```

Use the backward-compatible single-metric API:

```python
from mbe_eval import MBEEvaluator

evaluator = MBEEvaluator(metric_name="FIM_norm", baseline_name="Validation Loss")
report = evaluator.evaluate(metric_vals, baseline_vals, target_vals)
print(report.partial_r, report.classification)
```

## Reproducing Current Experiments

The current paper-scale audit lives in:

```bash
python experiments/07_jmlr_scale/analyze_jmlr_scale.py
```

Earlier falsification experiments are in:

```bash
python experiments/04_falsification/fim_unified_grid.py
python experiments/04_falsification/extract_tables.py
```

The Kaggle-scale scripts under `experiments/07_jmlr_scale/` train and audit image and text models. Their outputs are summarized in `jmlr_scale_v2_audit_summary.md` when downloaded.

## Repository Structure

```text
metric-audit-paper-code/
+-- mbe_eval/
|   +-- __init__.py
|   +-- core.py
|   +-- utils.py
|   +-- sample_eval.py
+-- examples/
|   +-- 01_run_acid_test.py
|   +-- 02_run_heterogeneous_grid.py
+-- experiments/
|   +-- 04_falsification/
|   +-- 05_kaggle/
|   +-- 06_independent_audit/
|   +-- 07_jmlr_scale/
+-- tests/
+-- pyproject.toml
+-- setup.py
+-- README.md
```

## Citation

```bibtex
@article{shadangi2026mbe,
  title={Marginal Baseline Evaluation for Auditing Generalization Metrics},
  author={Shadangi, Aparajeet},
  year={2026},
  note={Preprint}
}
```

## License

MIT License. See [LICENSE](LICENSE) for details.
