Metadata-Version: 2.4
Name: any-to-bench
Version: 0.1.1
Summary: Convert any exam materials (photos, PDFs, answer keys, rubrics) into a machine-gradable benchmark
Keywords: benchmark,exam,llm,evaluation,grading,dataset
Author: JacobLinCool
Author-email: JacobLinCool <jacoblincool@gmail.com>
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Dist: pydantic>=2.13
Requires-Dist: pydantic-ai-slim[openai,google]>=2.30
Requires-Dist: typer>=0.27
Requires-Dist: pypdfium2>=5.13
Requires-Dist: pillow>=12
Requires-Dist: jsonschema>=4.26
Requires-Dist: python-dotenv>=1.2
Requires-Dist: datasets>=5.0
Requires-Python: >=3.12
Project-URL: Repository, https://github.com/JacobLinCool/any-to-bench
Project-URL: Issues, https://github.com/JacobLinCool/any-to-bench/issues
Description-Content-Type: text/markdown

# any-to-bench

Convert **any exam materials** — photos of exam papers, exam PDFs, solution
PDFs/photos, official answer keys, scoring rubrics — into a **machine-gradable
benchmark**.

Give it everything you have for one exam; it produces an [exam bundle](docs/bundle.md):
the structured exam, a strict answer-sheet JSON Schema for any LLM harness, and a
grading spec. Fixed-answer questions grade deterministically; open-ended questions are
graded by multimodal LLM judges with rubrics extracted from your materials. All common
paper-exam question types are supported, including nested sub-questions, figures,
tables, and LaTeX math.

## Install

Requires Python ≥ 3.12.

```bash
uv tool install any-to-bench   # or: pip install any-to-bench
a2b --help
```

Set the API keys for the providers you use as environment variables (or in a
`.env` file in your working directory): `OPENAI_API_KEY`, `GOOGLE_API_KEY`,
`HF_TOKEN`, ...

For development, clone the repo and:

```bash
uv sync
cp .env.example .env   # fill in the API keys for the providers you use
```

## Usage

```bash
# 1. Ingest: any mix of PDFs and photos for ONE exam -> a bundle
a2b ingest exam.pdf answer-key.jpg rubric.pdf -o out/bundle --model openai:gpt-5.6-sol

# 2. Check the bundle
a2b validate out/bundle

# 3. Have an LLM take the exam (any provider — this is the benchmark part)
a2b solve out/bundle --model google:gemini-3.7-flash -o out/answers.json

# 4. Grade the answer sheet
a2b grade out/bundle out/answers.json -o out/report.json
# override judge model(s): --judge-model openai:gpt-5.6-sol --judge-model codex:gpt-5.6-sol

# Or benchmark several models at once: solve + grade each, compare in one table
a2b bench out/bundle -o out/bench \
    --model openai:gpt-5.6-terra --model google:gemini-3.7-flash

# Share bundles via Hugging Face datasets (viewer-friendly, byte-faithful round trip)
a2b upload out/bundle user/my-exams --name matha
a2b download user/my-exams --name matha -o local/bundle
```

`a2b` is a shorthand alias for `any-to-bench` — every command works with both. In a
cloned repo without installing, prefix commands with `uv run` (e.g. `uv run a2b ...`).

Ingest, solve, and judge models are independent. Use a `codex:` or `claude:` model
string (e.g. `codex:gpt-5.6-sol`, `claude:opus`) to run a phase **agentically** via
that CLI instead of direct LLM calls — same commands, same outputs. All commands accept `--effort` and
report token usage.

## Design principles

The three phases have deliberately **asymmetric goals**:

- **Ingest: spend freely, be exact.** A bundle is a dataset — built once, reused by
  everyone who ever benchmarks against it. Extraction accuracy is worth almost any
  model cost and wall time; this is why ingestion supports the expensive agentic mode,
  gap-repair rounds, and validate-and-fix loops. Intelligence spent here is amortized
  across every future run.
- **Solve: no constraints.** The taker is the thing being measured — anything from a
  cheap LLM call to a full agent belongs here.
- **Grade: require as little intelligence as possible.** The same answer sheet must
  earn the same score every time. Fixed-answer questions grade as pure scripts — zero
  model calls, bit-for-bit reproducible. Where an LLM judge is unavoidable
  (open-ended questions), it is *constrained*, not creative: precise rubrics with
  defined point levels, reference answers, and level snapping mean the judge follows
  the rubric mechanically instead of improvising — so even a non-frontier judge model
  grades accurately and consistently.

Put differently: ingest converts intelligence into structure (keys, rubrics, schemas)
exactly once, so that grading needs almost none, forever.

## Documentation

- [The exam bundle](docs/bundle.md) — output format, question model, validation
- [How ingestion works](docs/ingestion.md) — the LLM-mode extraction pipeline
- [Agentic mode](docs/agentic-mode.md) — `codex:`/`claude:` models, workspaces, the fix loop
- [Grading semantics](docs/grading.md) — deterministic rules and LLM judges
- [Benchmarking](docs/bench.md) — the `bench` model matrix and its metrics
- [Publishing](docs/publish.md) — sharing bundles as Hugging Face datasets
- [Models, effort, usage](docs/models.md) — model strings, `--effort`, token reporting

## Development

```bash
uv run pytest -q        # fully offline — model requests are forbidden in tests
uv run ruff check .
uv run ruff format .    # CI enforces this with --check
```

The test suite fakes the LLM layer (`any_to_bench.llm.build_agent`) and the agentic
subprocess layer (`any_to_bench.agentic.runner.run_codex` / `run_claude`), so the
entire ingest → solve → grade pipeline runs end-to-end in every mode without network
access or either CLI binary installed.
