Metadata-Version: 2.5
Name: sightread-bench
Version: 0.1.0
Summary: SightRead - can vision-language models read the board at first sight? Exact game-state extraction from screenshots, benchmarked field-by-field with cost capture.
Author: ansteinhuynh
License: MIT
License-File: LICENSE
Keywords: benchmark,chess,computer-vision,evaluation,games,go,leaderboard,mahjong,vlm
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx>=0.27
Requires-Dist: pyyaml>=6.0
Provides-Extra: analysis
Requires-Dist: matplotlib>=3.8; extra == 'analysis'
Requires-Dist: pandas>=2.0; extra == 'analysis'
Description-Content-Type: text/markdown

# SightRead

> **Can your vision-language model read the board at first sight?**

SightRead is a benchmark for **exact game-state extraction from screenshots**.
Not whether a model can *play* chess or mahjong — whether it can *see the
board*: every seat, every tile, every piece, every number, exactly.

Existing game benchmarks (GameWorld, LMGame-Bench, MET-Bench) score agents
**playing** games, confounding perception with policy. SightRead removes
policy entirely and audits **perception**: field-by-field, run-by-run,
dollar-by-dollar.

---

## Why this matters

In the original mahjong benchmark run that seeded this project
(6 models × 6 screenshots × 3 runs, 22 fields, hand-verified ground truth):

| Model | Field accuracy | Hand-tile F1 | $/call |
|---|---|---|---|
| qwen3.8-27b (API) | **89.4% ± 7.3%** | 90.3% | $0.035 |
| qwen3.8-27b (local Q5_K_M, free) | 84.8% ± 9.6% | **95.2%** | **free** |
| glm-5.3-flash (API) | 74.7% ± 13.8% | 85.1% | $0.008 |

The headline findings are more interesting than the leaderboard:

- **Exact hands are the hardest field by 2×** (25.7% exact) — but per-tile F1
  shows models are *close*; all-or-nothing scoring hides it.
- **Seat winds flip between runs while seat points score 100%** — a clean
  dissociation between reading numbers (OCR) and spatial relational reasoning
  (which badge belongs to which seat). This is SightRead's central question.
- **Populated discard piles: ~19% correct for every model.** Unsolved.
- **A decoy example in the prompt catches models that echo the prompt instead
  of looking at the image** — now a first-class metric: *grounding fidelity*.
- A **free local model** beats every paid API on the hardest component metric.

## What SightRead measures

| Metric | What it tells you |
|---|---|
| **Field accuracy** (mean ± sd over k runs) | exact-match rate per state field |
| **Component F1** | closeness on multisets (hands, piece lists) — not all-or-nothing |
| **Grounding fidelity** | decoy-echo rate: did the model read the image or the prompt? |
| **Flip rate** | which fields *change between identical runs* — instability, per field |
| **Cost & latency** | $/call and accuracy-per-dollar, cached-token aware, local = $0 |

## Architecture

A **core** (game-agnostic) + **packs** (per-game contributions):

```
sightread-bench/
├── src/sightread/        # core: comparators, scorer, runner, cost, reports
├── packs/
│   ├── mahjong/          # complete: parser ported from the original benchmark
│   │   ├── pack.yaml     #    field spec + grounding traps
│   │   ├── extract.py    #    format-agnostic response parser
│   │   ├── prompt.md     #    the extraction prompt
│   │   ├── gt/           #    hand-verified ground truth (JSON)
│   │   └── images/       #    screenshots stay LOCAL (gitignored - copyrighted)
│   └── chess/            # scaffold: FEN-driven GT generation, contamination notes
├── models.example.yaml   # OpenAI-compatible endpoints; llama-server works as-is
└── results/              # calls.jsonl -> results.csv + leaderboard.md
```

**Adding a game = adding a directory.** See [docs/adding-a-game.md](docs/adding-a-game.md).
Scoring runs identically across games; the leaderboard is generated, not hand-edited.

## Quickstart

```bash
pip install -e .

# validate a pack (schema, GT completeness, enum values)
sightread check --pack packs/mahjong

# run models (resume-safe; safe to kill and re-run)
cp models.example.yaml models.yaml   # add API keys via env vars
sightread run --pack packs/mahjong --models models.yaml --runs 3 --out results

# score + leaderboard
sightread score --pack packs/mahjong --calls results/calls.jsonl --out results
```

Local models: `llama-server --port 5006` and point `base_url` at
`http://127.0.0.1:5006/v1` — the same OpenAI-compatible path, $0/call.

## Packs

| Pack | Status | Ground truth | Visual difficulty |
|---|---|---|---|
| mahjong (riichi/MahjongSoul) | **ready** | hand-verified, 22 fields | tiny tiles, spatial seat rotation |
| chess | scaffold | programmatic (FEN) | requires non-standard skins: models memorize standard diagrams |
| card games | planned | programmatic (deck state) | occlusion, deduced hidden state |
| go | planned | programmatic (SGF) | counting; near-empty boards need careful design |

## Roadmap

- [ ] port remaining parser unit tests from the original harness (68 total)
- [ ] chess pack: contamination-resistant renderer (random piece sets/skins)
- [ ] held-out private packs per game (submission-time evaluation)
- [ ] HuggingFace dataset + leaderboard Space
- [ ] prompt-scaffolding ablation: does describing the seat layout fix wind flips?

## Citation

```bibtex
@software{sightread2026,
  title  = {SightRead: Exact Game-State Extraction Benchmark for Vision-Language Models},
  author = {ansteinhuynh},
  year   = {2026},
  url    = {https://github.com/ansteinhuynh/sightread-bench}
}
```

## License

MIT. Screenshots from game platforms are **not** included in this repository —
packs ship ground truth + prompts; capture images locally per the pack README.
