Metadata-Version: 2.4
Name: model-gym
Version: 0.8.0
Summary: HELM-style multi-leaderboard AI benchmarking platform — modernization, healthcare, frontier safety, agentic, code patches, compliance
Project-URL: Homepage, https://github.com/yadavilli-solutions/benchmark
Project-URL: Documentation, https://github.com/yadavilli-solutions/benchmark/blob/main/architecture.md
Project-URL: Repository, https://github.com/yadavilli-solutions/benchmark
Project-URL: Issues, https://github.com/yadavilli-solutions/benchmark/issues
Project-URL: Changelog, https://github.com/yadavilli-solutions/benchmark/blob/main/docs/WAVE_LOG.md
Author: Model Gym contributors
License: # Model Gym Educational Use License
        
        Copyright (c) 2026 Yadavilli Solutions
        
        Permission is granted to use, copy, modify, and share this repository for educational, research, personal learning, classroom, workshop, demonstration, and non-commercial evaluation purposes, subject to the terms below.
        
        ## Allowed Educational Uses
        
        You may use this software to:
        
        - Learn how AI agent benchmarking and certification systems are structured.
        - Teach, present, or demonstrate legacy modernization benchmarking concepts.
        - Run non-commercial experiments, prototypes, or internal learning exercises.
        - Modify the code for personal study, research, or classroom work.
        - Share non-commercial forks or derivatives that preserve this license notice.
        
        ## Commercial Use Requires Permission
        
        Commercial use is not granted by this license.
        
        You must contact Yadavilli Solutions before using this repository, any derivative work, or substantial portions of this repository for commercial purposes, including but not limited to:
        
        - Paid products, services, SaaS offerings, consulting engagements, or client delivery.
        - Internal production use by a for-profit company.
        - Commercial benchmarking, certification, modernization, migration, or AI engineering services.
        - Selling, licensing, hosting, or packaging this software or a derivative of it.
        - Using this work to support a paid customer engagement.
        
        For commercial licensing, contact:
        
        Yadavilli Solutions  
        https://github.com/yadavilli-solutions
        
        ## Attribution
        
        Any permitted use must preserve:
        
        - This license file.
        - Copyright notices.
        - Clear attribution to Yadavilli Solutions when the software or derivative work is shared publicly.
        
        ## No Warranty
        
        This software is provided "as is", without warranty of any kind, express or implied, including but not limited to warranties of merchantability, fitness for a particular purpose, and non-infringement.
        
        In no event shall the authors or copyright holders be liable for any claim, damages, or other liability arising from use of the software.
License-File: LICENSE.md
Keywords: ai,ai-act,benchmarking,compliance,gdpr,helm,hipaa,hti-1,inspect-ai,medhelm,rbac,swe-bench,tau-bench,wmdp
Classifier: Development Status :: 4 - Beta
Classifier: Framework :: FastAPI
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: <3.14,>=3.13
Requires-Dist: alembic>=1.13.0
Requires-Dist: asyncpg>=0.30.0
Requires-Dist: cryptography>=42.0.0
Requires-Dist: datasets>=3.0.0
Requires-Dist: falkordb>=1.0.0
Requires-Dist: fastapi>=0.115.0
Requires-Dist: httpx>=0.27.0
Requires-Dist: huggingface-hub>=0.30.0
Requires-Dist: jinja2>=3.1.0
Requires-Dist: pandas>=2.0.0
Requires-Dist: psutil>=6.0.0
Requires-Dist: psycopg2-binary>=2.9.0
Requires-Dist: pydantic>=2.10.0
Requires-Dist: pyjwt[cryptography]>=2.8.0
Requires-Dist: pypdf>=5.1.0
Requires-Dist: python-json-logger>=2.0.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: slowapi>=0.1.9
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: uvicorn[standard]>=0.34.0
Provides-Extra: connectors
Requires-Dist: azure-storage-blob>=12.19.0; extra == 'connectors'
Requires-Dist: boto3>=1.34.0; extra == 'connectors'
Requires-Dist: boxsdk>=3.9.0; extra == 'connectors'
Requires-Dist: google-api-python-client>=2.110.0; extra == 'connectors'
Requires-Dist: google-auth-oauthlib>=1.2.0; extra == 'connectors'
Provides-Extra: deepeval
Requires-Dist: deepeval>=0.21.0; extra == 'deepeval'
Provides-Extra: dev
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: playwright>=1.59.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.24.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.6.0; extra == 'dev'
Provides-Extra: graph
Requires-Dist: falkordb>=1.0.0; extra == 'graph'
Provides-Extra: ml
Requires-Dist: accelerate>=1.2.0; extra == 'ml'
Requires-Dist: bert-score>=0.3.13; extra == 'ml'
Requires-Dist: presidio-analyzer>=2.2.0; extra == 'ml'
Requires-Dist: presidio-anonymizer>=2.2.0; extra == 'ml'
Requires-Dist: sentence-transformers>=3.0.0; extra == 'ml'
Requires-Dist: spacy>=3.7.0; extra == 'ml'
Requires-Dist: torch>=2.5.0; extra == 'ml'
Requires-Dist: transformers>=4.48.0; extra == 'ml'
Provides-Extra: office
Requires-Dist: openpyxl>=3.1.0; extra == 'office'
Provides-Extra: swebench
Requires-Dist: swebench>=1.1.0; extra == 'swebench'
Description-Content-Type: text/markdown

# Model Gym

HELM-style, multi-leaderboard AI benchmarking for enterprise — legacy
modernization, healthcare AI, frontier safety, agentic tool-use, code patch
generation, and per-vertical / per-regulation evaluation, all under one
FastAPI service with shared persistence, an HMAC-chained audit ledger,
evidence-bundle export, and an Inspect-AI-compatible eval-log exporter.

> **40 leaderboards. 19+ metrics. 780+ tests green.**
> Last sync: 2026-05-17 (Wave 9 Track A/B).

---

## What you can do today

> Source: [`docs/diagrams/request-flow.mmd`](docs/diagrams/request-flow.mmd) — render via `npx -y @mermaid-js/mermaid-cli -i docs/diagrams/request-flow.mmd -o request-flow.svg`

```mermaid
flowchart LR
    User["User / CI / dashboard"] --> API["FastAPI<br/>app/main.py"]

    API --> Core["HELM-style<br/>eval-core<br/>app/eval/"]
    API --> Legacy["Legacy<br/>GymSpec runtime<br/>app/engine.py"]

    subgraph Boards["40 leaderboards (auto-discovered)"]
      Modern["Modernization × 5<br/>classic / evidence / robustness<br/>agentic v1.2 / safety"]
      SWE["SWE-Bench Verified<br/>(struct + Docker exec)"]
      Med["MedHELM (10 RunSpecs)"]
      Vert["Per-vertical × 17<br/>HCLS · retail · fin-svc<br/>hi-tech · industry · edu-K12"]
      Comp["Compliance × 9<br/>HIPAA · PCI-DSS · SOX · GDPR<br/>CMS · HTI-1 · 3 state AI<br/>OWASP × 2 · MITRE ATLAS"]
      Safe["Frontier safety × 2<br/>WMDP · medical_safety<br/>+ agent_threat_safety"]
    end

    Core --> Boards

    Boards --> Snap["Postgres :25432<br/>leaderboard_snapshots"]
    Boards --> Traces["traces/<run_id>/*.json"]
    Boards --> Ledger["audit_ledger.jsonl<br/>(HMAC chained)"]

    Snap --> Bundle["Evidence bundle ZIP<br/>/evidence_bundle"]
    Traces --> Bundle
    Ledger --> Bundle

    Snap --> Inspect["Inspect AI EvalLog v2<br/>/inspect_log"]
    Traces --> Inspect
```

- Run any of 30 leaderboards against any model with hash-keyed request caching
- Stream HuggingFace datasets with `access_tier: public | gated | private`
  (set `HF_TOKEN` for gated, `MODELGYM_HF_OFFLINE=1` for airgap / CI)
- Score with 19+ metrics across 7 groups (core, evidence, HCLS, compliance,
  agentic, safety, code)
- Export an Inspect-AI-compatible EvalLog v2 JSON for any run
- Export an evidence bundle (snapshot + traces + audit slice + scenario
  fingerprints) per run
- Drive the legacy GymSpec onboarding/certification path on the same store

---

## Leaderboard catalog

| Category             | ID prefix              | Boards | Headline metric         |
|----------------------|------------------------|--------|-------------------------|
| Modernization (core) | `modernization_*`      | 5      | varies per board        |
|   · classic          | `modernization_classic`         | 1 | exact_match + judge   |
|   · evidence         | `modernization_evidence`        | 1 | citation_f1 + hallucination_rate |
|   · robustness       | `modernization_robustness`      | 1 | robustness_delta      |
|   · agentic          | `modernization_agentic` v1.2    | 1 | step_success_rate + action_set_jaccard + task_completed_partial (TAU-Bench live mocks) |
|   · safety           | `modernization_safety`          | 1 | wmdp_safety (inverted) |
| SWE-Bench Verified   | `swe_bench_verified` v1.1  | 1      | patch_files_match_gold + swe_bench_resolved (opt-in Docker)  |
| MedHELM              | `medhelm_modernization` | 1     | jury_score + bertscore  |
| Vertical             | `vertical_*`           | 17     | per-vertical jury_score |
| Compliance           | `compliance_*`         | 9      | control_match (HIPAA, PCI-DSS, SOX, GDPR, CMS-Interop, HTI-1, NYC LL 144, CA AI laws, IL AI laws + OWASP LLM Top 10, OWASP Agentic, MITRE ATLAS) |
| Frontier safety      | `modernization_safety`, `medical_safety` | 2 | wmdp_safety, red_team_safety (inverted) |
| Agent threat         | `agent_threat_safety`  | 1 | atr_safety (ATR rule pattern coverage) |

For the full registry: `uv run python scripts/export_registry.py`.

---

## Quick start

```bash
# install (uv is the toolchain)
uv sync

# start Postgres on 25432 + apply migrations
docker compose up -d db
uv run alembic upgrade head

# run the full eval test suite (offline by default)
MODELGYM_HF_OFFLINE=1 uv run pytest tests/eval/ -q

# boot the API + dashboard
uv run uvicorn app.main:app --reload --port 8000
# → http://localhost:8000/leaderboards.html
# → http://localhost:8000/dashboard         (legacy GymSpec UI)
```

### Optional live data

```bash
# pull real HF datasets (MedHELM, WMDP, SWE-Bench Verified, etc.)
unset MODELGYM_HF_OFFLINE
export HF_TOKEN=hf_xxx      # only needed for gated-tier scenarios
uv run pytest tests/eval/test_medhelm_real_data.py -q
```

---

## Environment variables

| Var                              | Purpose                                                    | Required in prod |
|----------------------------------|------------------------------------------------------------|------------------|
| `DATABASE_URL`                   | Postgres DSN (default: `:25432/modelgym`)                  | yes              |
| `AUDIT_HMAC_SECRET`              | HMAC key for audit ledger chain                            | yes              |
| `MODELGYM_AUDIT_LEDGER`          | Path to append-only ledger JSONL                           | yes              |
| `MODELGYM_TRACE_ROOT`            | Root dir for per-instance trace JSON                       | yes              |
| `MODELGYM_REQUEST_CACHE_PATH`    | SQLite path for hash-keyed model-call cache                | yes              |
| `MODELGYM_HF_OFFLINE`            | `1` to forbid HF network calls during tests/airgap         | no               |
| `HF_TOKEN`                       | HuggingFace token — required for `access_tier=gated`       | when gated used  |

---

## Adding a new benchmark

1. Drop a scenario class in `app/eval/specs/<your_thing>.py` (subclass
   `HuggingFaceDatasetScenario` for HF, or implement `Scenario` directly).
2. Drop a leaderboard registration in `app/eval/leaderboards/<your_thing>.py`
   (calls `REGISTRY.register(Leaderboard(...))` at import time).
3. FastAPI auto-discovers it on next boot. Add a test under
   `tests/eval/test_<your_thing>.py`.

For a worked example, see `app/eval/specs/wmdp_scenarios.py` +
`app/eval/leaderboards/modernization_safety.py` + `tests/eval/test_wmdp.py`.

---

## Documentation map

- [`architecture.md`](architecture.md) — full architectural truth (read this
  first when extending)
- [`ROADMAP.md`](ROADMAP.md) — what shipped, what's next
- [`docs/WAVE_LOG.md`](docs/WAVE_LOG.md) — Plans 1-7 vs ship reality + Waves 1-5
- [`docs/superpowers/plans/`](docs/superpowers/plans/) — original written plans
  (superseded; WAVE_LOG is canonical)
- [`PATH_TO_PROD.md`](PATH_TO_PROD.md) — known prod-readiness boundaries

---

## Boundaries (not yet production-safe)

- `X-Gym-Role` header is **dev-mode only**. Set `MODELGYM_ENV=production`
  to fail-close — Keycloak is then required (or `MODELGYM_ALLOW_DEV_AUTH=1`
  for airgap deploys that issue their own local JWTs).
- SWE-Bench Docker execution (FAIL_TO_PASS / PASS_TO_PASS) is **opt-in**
  via `MODELGYM_SWE_BENCH_EXEC=1` + a docker daemon + the upstream
  `swebench/sweb.eval.*` per-instance images. Without it, scoring is
  structural-only.
- WMDP scoring remains structural (frontier-safety MCQ). TAU-Bench has
  live retail + airline mocks (Waves 7.4 + 7.5); 4 of 12 vendored tasks
  have `gold_final_state` for `TaskCompletionMetric` — remaining authoring
  is rolling content work.
- BERTScore falls back to `rouge_L` when the `bert_score` package isn't
  installed (set `MODELGYM_BERTSCORE_REQUIRE_REAL=1` to fail-close).
- Inspect AI export emits both `.json` and a real in-tree `.eval` zipfile
  (Wave 7.1) — no `inspect-ai` dep required.

---

## License

See [`LICENSE.md`](LICENSE.md).
