Metadata-Version: 2.4
Name: agi-evals
Version: 0.1.0
Summary: Plug any model into any major AGI eval and actually run it.
Project-URL: Homepage, https://agi-eval.studio
Project-URL: Documentation, https://agi-eval.studio/evals
Project-URL: Leaderboard, https://agi-eval.studio/leaderboard
Author: iso-ai
License: Apache-2.0
License-File: LICENSE
Keywords: agent,agi,benchmark,evals,evaluation,llm
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Requires-Dist: httpx>=0.27
Requires-Dist: posthog>=3.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: alfworld
Requires-Dist: alfworld>=0.4; extra == 'alfworld'
Provides-Extra: all
Requires-Dist: anthropic>=0.40; extra == 'all'
Requires-Dist: openai>=1.0; extra == 'all'
Provides-Extra: anthropic
Requires-Dist: anthropic>=0.40; extra == 'anthropic'
Provides-Extra: dev
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: hf
Requires-Dist: torch>=2.0; extra == 'hf'
Requires-Dist: transformers>=4.40; extra == 'hf'
Provides-Extra: mlx
Requires-Dist: mlx-lm>=0.10; extra == 'mlx'
Provides-Extra: openai
Requires-Dist: openai>=1.0; extra == 'openai'
Provides-Extra: scienceworld
Requires-Dist: scienceworld>=1.2; extra == 'scienceworld'
Description-Content-Type: text/markdown

# agi-evals

**Plug any model into any major AGI eval and actually run it.**

Open source. Six categories. 45 evals catalogued, deeply implemented over time —
not a directory pretending to be a platform. The runner code is Apache-2.0; each
eval's dataset keeps its original upstream license (documented per entry).

Everything runs **on your machine, no account or API key required** — bring your
own model credentials (or none, for local models via Ollama/vLLM/MLX). A free
account at **[agi-eval.studio](https://agi-eval.studio)** adds the hosted layer:
score-over-time dashboards, variant-vs-base comparison cards, a public
leaderboard, and challenges.

→ Catalog, per-eval docs, and scoreboard: **[agi-eval.studio](https://agi-eval.studio)**

---

## The idea

Two protocols decouple *what we run* from *what we run it on*:

- **`PatientAdapter`** — a model endpoint. Takes a prompt plus any eval-specific
  scenario, returns a response. Adapters ship for OpenAI, Anthropic, Grok, Ollama,
  vLLM, Hugging Face Transformers, MLX (Apple Silicon), and a custom-callable shim.
- **`EvalRunner`** — an eval. Takes a patient and a case, returns a scored result
  with a typed failure tag.

Get these two right and adding eval #2 through #50 is incremental. Every eval and
every model hangs off them — an eval never imports an adapter, an adapter never
imports an eval.

```
catalog/evals.yaml ──┬──► website (agi-eval.studio)
                     └──► registry ──► EvalRunner ──┐
                                                    ├──► harness ──► EvalReport ──► push to scoreboard
                              PatientAdapter ───────┘
```

## Install

```bash
pip install agi-evals                 # core + custom/ollama/vllm/grok/openai-compat
pip install 'agi-evals[openai]'       # + OpenAI SDK
pip install 'agi-evals[anthropic]'    # + Anthropic SDK
pip install 'agi-evals[hf]'           # + Transformers/torch
pip install 'agi-evals[mlx]'          # + MLX (Apple Silicon)
```

## Quickstart — CLI

```bash
agi-evals list --status live                 # browse the catalog
agi-evals info gpqa-diamond                  # inspect one eval
agi-evals run gpqa-diamond --model echo      # offline smoke test, no keys
agi-evals download --all                     # fetch + cache full datasets
agi-evals run gpqa-diamond --model openai:gpt-4o-mini --limit 50
agi-evals run humaneval-plus --model ollama:llama3.1:8b --concurrency 4
agi-evals run math --model anthropic:claude-opus-4-8 --push   # submit to scoreboard
```

Every shipped eval bundles a small real-schema sample so it runs offline out of the
box. `agi-evals download <eval>` fetches the full upstream dataset (HF datasets-server
or GitHub, no heavy deps) into `~/.cache/agi-evals/`, and runs use it automatically.
GPQA is gated upstream: set `HF_TOKEN` after accepting its terms, or the runner falls
back to the GPQA repo's published-password zip. An explicit `data_path=` always wins.

## Quickstart — SDK

```python
from agi_evals import load_runner, run_eval
from agi_evals.adapters import OpenAIAdapter, CustomAdapter

# Any of the built-in adapters...
patient = OpenAIAdapter("gpt-4o-mini")

# ...or wrap your own endpoint as a callable:
patient = CustomAdapter(lambda req: my_model(req.prompt), name="my-model")

report = run_eval(load_runner("gpqa-diamond"), patient, limit=100, concurrency=8)
print(report.score, report.pass_rate, report.failure_counts)

# Save it to your scoreboard at agi-eval.studio
from agi_evals.client import push_report
push_report(report, model="my-model")        # needs AGI_EVALS_API_KEY
```

## Track your scores at agi-eval.studio

Local runs print a report and exit — nothing leaves your machine. To keep a
history, add `--push`:

1. Sign in at [agi-eval.studio](https://agi-eval.studio) (GitHub OAuth).
2. Mint a key under **Settings → API keys** (shown once, stored hashed).
3. `export AGI_EVALS_API_KEY=ae_...`
4. Add `--push` to any `run` or `compare`.

Your [dashboard](https://agi-eval.studio/dashboard) charts every eval over time,
groups variant-vs-base comparisons into vs-cards, and lets you submit a run to a
[challenge](https://agi-eval.studio/challenges) or the public
[leaderboard](https://agi-eval.studio/leaderboard) — attach your GitHub repo or
an endpoint so others can see what the score belongs to.

## Live evals (runnable today)

| Eval | Category | Grading | Full dataset |
|------|----------|---------|--------------|
| GPQA Diamond | reasoning | single-letter MCQ | 198 |
| MMLU-Pro | reasoning | 10-choice MCQ | ~12k |
| MATH | reasoning | `\boxed{}` answer, math-aware match | 500 (MATH-500) |
| AIME 2024 | reasoning | integer exact-match | 30 |
| HumanEval+ | code | sandboxed test execution | 164 |
| BIG-Bench Hard | reasoning | normalized exact-match, 27 tasks | ~6.5k |
| MuSR | reasoning | narrative MCQ | 756 |
| BFCL (simple) | agent | function-call AST match | 400 |
| ZebraLogic | reasoning | full-grid JSON, puzzle-level | gated (HF_TOKEN) |
| JailbreakBench | safety | refusal rate, LLM-judged | 100 |
| LiveCodeBench | code | contest tests, pass@k, contamination-free | recent releases (~340) |
| HarmBench | safety | behavior classifier, score = 1 − ASR | 300 |
| τ-bench | agent | episode reward: DB-state × outputs | 165 (retail+airline) |
| ALFWorld | embodied | task success in the real TextWorld engine* | 134 unseen games |
| ScienceWorld | embodied | engine score 0–100, partial credit* | 30 tasks, test variations |
| AILuminate | safety | judged safe-response rate (practice set) | 1,200 |

τ-bench is a faithful port of the Sierra Research benchmark: the original tools,
databases, policy wikis, tasks, and reward function, vendored 1:1 (MIT). The
simulated user is any `PatientAdapter` (`TauBenchRunner(user=OpenAIAdapter("gpt-4o"))`).
Port verified by a gold-replay oracle scoring 165/165 on the real test sets.

\* ALFWorld and ScienceWorld drive their original engines and need extras:
`pip install 'agi-evals[alfworld]'` / `'agi-evals[scienceworld]'` (Java required
for ScienceWorld). Every other live eval runs with zero optional dependencies.

The other 29 catalogued evals across agent/tool-use, code, robotics, and safety
carry status `building` or `roadmap` — browse them all, with per-eval docs
(how it works, scoring, troubleshooting), at
[agi-eval.studio/evals](https://agi-eval.studio/evals).

## pass@k for code evals

```python
from agi_evals.evals import HumanEvalPlusRunner, LiveCodeBenchRunner

runner = LiveCodeBenchRunner(n_samples=10, k=5)   # 10 samples, report pass@5
```

Sampling uses the unbiased Chen et al. (2021) estimator; the default
`n_samples=1, k=1` is plain greedy pass@1.

## Compare a variant against its base

```bash
agi-evals compare gpqa-diamond --model openai:my-finetune \
    --baseline openai:gpt-4o-mini --push
```

Paired per-case comparison on identical cases: improvements (cases the variant
newly solves), regressions (cases it newly fails, listed by id), score delta, and
McNemar's exact test on the discordant pairs. Infra errors on either side are
excluded from pairing so endpoint flakes never read as regressions. `--push`
lands both runs on your dashboard as a vs-card.

## Typed failure taxonomy

Every result carries at most one `FailureTag`: `WRONG_ANSWER`, `NO_ANSWER`,
`REFUSED`, `MALFORMED_OUTPUT`, `TOOL_ERROR`, `TIMEOUT`, `CONTEXT_OVERFLOW`,
`ADAPTER_ERROR`, `HARNESS_ERROR`. Infrastructure errors (adapter/harness) are
excluded from the aggregate score so a flaky endpoint never silently penalizes a
model — they stay visible in `failure_counts`.

## Safety note

`HumanEval+` executes model-generated code locally in a subprocess with a timeout.
Run only models and datasets you trust, or wrap it in an OS-level sandbox.

## Contributing

The SDK source repository opens to contributions on **July 1, 2026**. Until
then: bug reports, eval requests, and questions →
[agi-eval.studio](https://agi-eval.studio). Adding an eval is deliberately
small — implement an `EvalRunner`, bundle a sample, add a catalog entry — and
the installed package is the reference: every live eval ships its source in
`agi_evals/evals/`.

## License

Runner code: **Apache-2.0** (see [LICENSE](LICENSE)). Eval datasets retain their
upstream licenses, documented per entry in the catalog.
