Metadata-Version: 2.4
Name: sma-audit
Version: 0.1.1
Summary: Sports Model Integrity Auditor — catches common ML failure modes (leakage, imbalance, drift, small-sample instability) and produces a plain-language report card.
Author: Mmopiemang Mmopiemang
License: MIT License
        
        Copyright (c) 2026 Mmopiemang Mmopiemang
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Project-URL: Homepage, https://github.com/kopommops/sports_model_auditor
Project-URL: Repository, https://github.com/kopommops/sports_model_auditor
Project-URL: Demo, https://sports-model-auditor.onrender.com/
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=1.24
Requires-Dist: pandas>=2.0
Requires-Dist: scipy>=1.10
Provides-Extra: api
Requires-Dist: fastapi>=0.110; extra == "api"
Requires-Dist: uvicorn>=0.29; extra == "api"
Requires-Dist: python-multipart>=0.0.9; extra == "api"
Requires-Dist: scikit-learn>=1.3; extra == "api"
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: scikit-learn>=1.3; extra == "dev"
Dynamic: license-file

<div align="center">
  <img src="https://raw.githubusercontent.com/kopommops/sports_model_auditor/main/docs/banner.svg" alt="sma terminal demo" width="100%">
</div>

<div align="center">

  ![tests](https://img.shields.io/badge/tests-32%20passing-e4e4e4?style=flat-square&labelColor=0a0a0a)
  ![python](https://img.shields.io/badge/python-3.9%2B-e4e4e4?style=flat-square&labelColor=0a0a0a)
  ![license](https://img.shields.io/badge/license-MIT-e4e4e4?style=flat-square&labelColor=0a0a0a)
  [![demo](https://img.shields.io/badge/demo-live-e4e4e4?style=flat-square&labelColor=0a0a0a)](https://sports-model-auditor.onrender.com/)

</div>

# sma — Sports Model Integrity Auditor

A model can look great in a notebook — high accuracy, clean metrics — and still fall over in production, because the number was never the problem. **sma** audits a fitted model plus its dataset for five of the most common ways that happens, and returns a plain-language report card instead of a wall of statistics.

Built framework-agnostic: works with anything exposing `.predict` — scikit-learn, XGBoost, LightGBM, or a hand-rolled class. See [`sma/core/checks/base.py`](sma/core/checks/base.py).

**[Live demo →](https://sports-model-auditor.onrender.com/)** *(free-tier hosting — first request after inactivity takes ~30–50s to wake up)*

---

## What it checks

| Check | Catches |
|---|---|
| `class_imbalance` | A skewed target that makes accuracy alone misleading |
| `small_sample_instability` | Too little data — overall, or in a minority class — for the reported metric to be trustworthy. Bootstraps the metric to quantify how noisy it actually is |
| `target_leakage` | A feature suspiciously correlated with the label — often a sign it encodes the answer (a "cancellation_notice_sent" column when predicting churn) |
| `temporal_leakage` | A non-chronological train/test split, or a feature that predicts *tomorrow's* label better than today's — the classic look-ahead bug in feature engineering |
| `data_drift` | The current data's distribution has shifted from what the model was trained on (PSI + KS-test), so its assumptions may no longer hold |

Every check returns `PASS` / `WARN` / `FAIL` / `NOT_APPLICABLE` with structured evidence — never a bare pass/fail with no explanation.

## Real-world validation

Built and tested against synthetic failure cases (a deliberate 9:1 class imbalance, a leaky churn feature, a genuine look-ahead bug, a simulated "price increase" distribution shift) — all five checks fired exactly as designed, with 32 passing unit tests.

Then dogfooded against [**PulseConnect**](https://github.com/kopommops/pulseconnect), an F1 telemetry and driver-compatibility ML platform, using its real `GradientBoostingRegressor` and 8 seasons of real race data (2019–2026):

- **Leakage checks came back clean** — `target_leakage` and `temporal_leakage` both passed, which is itself evidence the leakage fix already documented in PulseConnect's own code comments (recency-windowed features, season-indexed constructor form) actually worked.
- **`data_drift` caught something real**: comparing early seasons (2019–2022) against recent ones (2023–2026), driver pace and tyre-degradation features showed major distribution shift (PSI 1.3–1.4) — a legitimate signal of F1's 2022 regulation overhaul, not a bug. It's concrete evidence that pooling all seasons as one training set treats different regulation eras as statistically the same when they aren't.

## Architecture

<img src="https://raw.githubusercontent.com/kopommops/sports_model_auditor/main/docs/architecture.svg" alt="architecture diagram" width="100%">

```
sma/
├── sma/
│   ├── core/
│   │   ├── checks/           # one file per check, all sharing base.py's contract
│   │   │   ├── base.py       # CheckResult, Status, ModelAdapter
│   │   │   ├── class_imbalance.py
│   │   │   ├── small_sample.py
│   │   │   ├── target_leakage.py
│   │   │   ├── temporal_leakage.py
│   │   │   └── data_drift.py
│   │   ├── auditor.py        # orchestrates: run(model, X, y, ...) -> Report
│   │   └── report.py         # aggregates CheckResults, computes overall status
│   ├── report_renderers/     # Report -> markdown / html
│   └── cli.py
├── api/
│   ├── main.py                # FastAPI wrapper — imports sma.core directly
│   └── static/index.html      # terminal-styled demo frontend
└── tests/                      # 32 tests, one file per check
```

Every check shares one contract:

```python
def run(model, X, y, **kwargs) -> CheckResult:
    ...
```

registered in `sma/core/auditor.py::DEFAULT_CHECKS`. The CLI, the API, and both report renderers all consume the same `Auditor.run()` — nothing is duplicated between them, so what a `pip install`er gets is exactly what the hosted demo runs. A check that fails to run (missing optional input, not-yet-implemented) is isolated and reported as `NOT_APPLICABLE`, never crashes the whole audit.

## Install

```bash
pip install sma-audit          # once published — see below
# or, for local development:
git clone https://github.com/kopommops/sma-audit
cd sma-audit
pip install -e ".[dev,api]"
```

## Use as a package

```python
from sma import Auditor

report = Auditor().run(model, X, y)
print(report.overall_status)      # Status.PASS / WARN / FAIL
print(report.to_dict())
```

With the optional inputs each check can use:

```python
report = Auditor().run(
    model, X, y,
    timestamps=df["date"],           # enables temporal_leakage
    reference_X=training_data,       # enables data_drift
)
```

## Use as a CLI

```bash
sma audit model.pkl --data train.csv --target churned --format html --out report.html

# with timestamps + a chronological split check:
sma audit model.pkl --data train.csv --target churned \
  --timestamp-col date --split-date 2026-01-01

# with a reference dataset for drift detection:
sma audit model.pkl --data current.csv --target churned \
  --reference-data training_data.csv
```

## Run the API + demo frontend locally

```bash
uvicorn api.main:app --reload
# visit http://127.0.0.1:8000 for the terminal UI,
# or POST directly to /audit (JSON) or /audit/html (rendered report)
```

## Run tests

```bash
pytest -q
# 32 passed
```

## Design notes

- **Framework-agnostic by construction** — `ModelAdapter` (`sma/core/checks/base.py`) wraps any object exposing `.predict`; checks never call the model directly, so swapping in a future model type (a raw PyTorch wrapper, say) means changing one adapter, not five checks.
- **Isolated check failures** — the orchestrator catches exceptions per-check and reports them as a `FAIL` with the error message, rather than letting one broken check take down the whole audit.
- **Evidence over verdicts** — every result carries structured `evidence` (counts, scores, thresholds), not just a status label, so a report card is something you can actually investigate, not just trust.

## Roadmap

- [x] All 5 v1 checks implemented and tested
- [x] CLI, package API, and FastAPI demo, all sharing one core
- [x] Dogfooded against a real production model (PulseConnect)
- [x] Hosted demo deployed
- [ ] Publish to PyPI
- [ ] Config object for per-check thresholds (currently hardcoded per-module constants)
- [ ] Additional checks (e.g. feature importance stability, calibration)

## License

MIT — see [LICENSE](LICENSE).
