Metadata-Version: 2.4
Name: scimlrigorbench
Version: 4.0.0
Summary: Reproducible benchmarking and validation framework for methodological assurance in machine learning
Author: Dimitrios Galiatsatos
License: MIT License
        
        Copyright (c) 2026 Dimitrios Galiatsatos
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/dgaliatsatos2010/SciMLRigorBench
Project-URL: Repository, https://github.com/dgaliatsatos2010/SciMLRigorBench
Project-URL: Issues, https://github.com/dgaliatsatos2010/SciMLRigorBench/issues
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: scimlrigor==1.5.2
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: scikit-learn==1.8.0
Dynamic: license-file

# SciMLRigorBench v4.0.0

SciMLRigorBench is the reproducible benchmarking and assurance layer for **SciMLRigor**. Version 4.0.0 evaluates the released **SciMLRigor v1.5.2** engine across controlled methodological failure injections, a preregistered bespoke procedural reference, source-backed external-validation context, and a version-pinned capability landscape.

## Purpose

The benchmark asks whether a single standardized methodological-audit interface can recover heterogeneous validation failures at performance comparable to a hand-assembled set of specialized diagnostics. It does **not** claim that SciMLRigor is universally more accurate than all specialized tools.

## What is included

- Frozen comparative protocol: `METHODS_PROTOCOL.md`
- Exact benchmark runner: `src/run_comparative_benchmark.py`
- 60 paired controlled scenarios across R01-R10
- Classification and regression benchmark tasks
- SciMLRigor v1.5.2 vs scikit-learn v1.8.0 bespoke procedural reference
- Exact McNemar and paired-bootstrap statistics
- Version-pinned capability mapping for Deepchecks, Evidently, Great Expectations, mlinspect, and mlscorecheck
- Explicit third-party execution status
- Source-backed external-validation context from SciMLRigor v1.5.2
- Publication-oriented reports and figures

## Headline comparative results

| Metric | SciMLRigor 1.5.2 | Manual reference 1.8.0 |
|---|---:|---:|
| Expected-positive scenarios | 38 | 38 |
| Expected-negative controls | 22 | 22 |
| True positives | 33 | 32 |
| False negatives | 5 | 6 |
| True negatives | 20 | 22 |
| Detection recall | 0.8684 | 0.8421 |
| Benchmark-ground-truth specificity | 0.9091 | 1.0000 |
| Balanced accuracy | 0.8888 | 0.9211 |
| Overall accuracy | 0.8833 | 0.9000 |
| Localization recall | 0.8684 | 0.8421 |
| Total measured diagnostic runtime | 47.02 s | 5.98 s |

Paired comparison: exact two-sided McNemar p = **1.0**. The SciMLRigor-minus-reference balanced-accuracy difference was **-0.0323**, with a 10,000-replicate paired bootstrap 95% CI of **[-0.1375, 0.0638]**.

The evidence therefore supports **comparable aggregate diagnostic performance with different family-specific strengths and sensitivity/specificity trade-offs**, rather than universal superiority.

## Failure families

The benchmark covers ten methodological-risk families:

- R01: target-derived leakage
- R02: selection/tuning scope
- R03: repeated/group dependence
- R04: temporal transport sensitivity
- R05: duplicate contamination
- R06: signal/permutation integrity
- R07: identifier shortcut dependence
- R08: proxy dependence
- R09: domain transportability
- R10: reliability/calibration degradation

## Evidence grading

External-validation context follows the SciMLRigor v1.5.2 evidence tiers:

- **A**: authoritative/original source executed end-to-end
- **B**: verified mirror executed and independently cross-checked
- **C**: protocol/fixture/parser validation only
- **P**: protocol prepared but source execution pending/unavailable

Tier A is the primary source-backed external layer. Tier C/P support implementation or readiness claims only.

## Reproducing the benchmark

The executed environment recorded in `results/environment.json` used Python 3.13.5, SciMLRigor 1.5.2, scikit-learn 1.8.0, NumPy 2.3.5, and pandas 2.2.3.

A validated SciMLRigor v1.5.2 wheel is included under `artifacts/` for exact reproduction.

Example setup:

```bash
python -m venv .venv
# Windows: .venv\\Scripts\\activate
# Linux/macOS: source .venv/bin/activate
pip install artifacts/scimlrigor-1.5.2-py3-none-any.whl
pip install -r requirements.txt
python src/run_all.py
```

The frozen thresholds and benchmark rules are documented in `METHODS_PROTOCOL.md`. They were fixed before comparative outcomes were inspected.

## Repository structure

```text
SciMLRigorBench/
├── README.md
├── LICENSE
├── CITATION.cff
├── .zenodo.json
├── .gitignore
├── VERSION
├── METHODS_PROTOCOL.md
├── RELEASE_MANIFEST.json
├── SHA256SUMS.txt
├── requirements.txt
├── src/
├── results/
├── metadata/
├── figures/
├── reports/
├── context_v3/
└── artifacts/
```

## Relationship to SciMLRigor

SciMLRigorBench calls the released SciMLRigor audit engine rather than duplicating R01-R10 logic. The benchmarked core software release is:

**SciMLRigor v1.5.2** — Zenodo version DOI: **10.5281/zenodo.23001654**.

## Claim boundaries

The benchmark does not support claims that SciMLRigor is universally more accurate than all competing tools, that unexecuted third-party packages failed the benchmark, or that 60 controlled scenarios establish universal real-world sensitivity and specificity. Non-injected control alerts are reported as benchmark-ground-truth disagreements because natural datasets may contain genuine methodological sensitivity not introduced by the benchmark.

## Citation

Use `CITATION.cff` for citation metadata. A Zenodo DOI will be added for SciMLRigorBench v4.0.0 after archival of the GitHub release.

## License

MIT License.
