# `eval/` — `eval-jss` precision harness

This directory holds the Python package and supporting data files for
`eval-jss`, the companion CLI that measures per-rule precision of
`jss-lint` against a real-world corpus of JSS manuscripts. Spec and
design artifacts live under
[`specs/002-eval-jss-harness/`](../specs/002-eval-jss-harness/).

The harness implements Constitution §VI (≥90% precision per rule) as an
enforceable, empirical gate, and §XII (reproducible corpus) via a
pinned `corpus-manifest.csv`.

## Layout

```text
eval/
├── __init__.py                 # package marker
├── api.py                      # shared dataclasses + enums
├── db.py                       # SQLite connection + PRAGMAs + init()
├── cli.py                      # click entry; console script eval-jss
├── scan.py                     # walk corpus, shell out to jss-lint
├── human_review.py             # interactive TUI for labelling
├── review.py                   # AI-assisted review (Phase B)
├── report.py                   # precision table + CSV history + --by-source
├── corpus.py                   # manifest parse + fetch + SHA256 verify
├── schema.sql                  # authoritative DDL (loaded by db.init)
│
├── corpus-manifest.csv         # pinned corpus definition (checked in)
├── corpus-manifest-gaps.csv    # generated by `corpus fetch`; checked in
├── review-skip-list.toml       # rules that bypass AI review
├── report.csv                  # precision history (generated; checked in)
└── eval.db                     # runtime DB (gitignored)
```

## Runtime state vs. checked-in state

- **Gitignored**: `eval.db`, `eval.db-shm`, `eval.db-wal`, and any
  `examples/**/.sha256` marker files dropped by `corpus fetch`.
- **Checked in**: `corpus-manifest.csv`, `review-skip-list.toml`,
  `report.csv`, and `corpus-manifest-gaps.csv` — these are the durable,
  diffable records that make precision claims reproducible.

## Adding a paper to the corpus (MVP procedure)

1. Pick a CRAN package whose vignette uses `\documentclass{jss}` or
   `\usepackage{jss}` (e.g., `lme4`, `MASS`, `limma`, `zoo`).
2. Download the current tarball from CRAN `Archive/`. Extract and copy
   the `.tex` + `.bib` into `examples/cran_<pkg>_<version>/` along
   with a one-paragraph `README.md` recording the URL, version, and
   (if available) JSS DOI from `inst/CITATION`.
3. Add a manifest row pointing at the tarball URL with its SHA256.
4. Run `eval-jss scan && eval-jss corpus status` — both should succeed.

The current `examples/` directory contains three placeholder
vignettes that exist solely to exercise the plumbing end-to-end; see
[`examples/README.md`](../examples/README.md) for the handoff policy.

## Running the harness

Full walkthrough: [`specs/002-eval-jss-harness/quickstart.md`](../specs/002-eval-jss-harness/quickstart.md).

Short form:

```sh
eval-jss init                                    # idempotent schema
eval-jss scan --corpus examples/                 # persist violations
eval-jss human-review                            # interactive labelling
eval-jss review --confidence-threshold 0.8       # AI-assisted (Phase B)
eval-jss report --csv eval/report.csv            # print + append history
eval-jss report --by-source                      # per-source breakdown
eval-jss corpus fetch                            # reproduce from manifest
eval-jss corpus status                           # report corpus state
```

Exit codes: `0` all rules at ≥90% precision · `1` at least one rule
below threshold, or `scan` newly persisted violations, or `corpus fetch`
logged a gap · `2` the harness itself could not run.

## The eval-improve loop

One iteration grows the corpus, surfaces rule defects, and shrinks them.
Slash commands live under `.claude/commands/`:

1. **Add manuscripts** — `/eval-add-corpus` (edits
   `eval/corpus-manifest.csv`, then `eval-jss corpus fetch`).
2. **Scan** — `/eval-scan` (`eval-jss scan --corpus examples/ --force`).
3. **AI classifier** — `/eval-review`.
4. **Human review** — `/eval-human-review` (uncertain + skip-listed).
5. **Record baseline** — `/eval-record <label>` snapshots full +
   pinned precision into `eval/precision-history.db` and appends to
   `eval/improvement-log.md`.
6. **Investigate failing rules** — `/eval-investigate <JSS-XXX-NNN>`
   delegates the FP-pattern discovery to a subagent (clean context),
   then implements the fix in the main thread, runs `pytest` + `ruff`,
   and finishes with `eval-jss iterate refresh && eval-jss report
   --diff` to surface the precision delta. Replaces the earlier
   `/eval-suggest` + `/eval-plan` + `/eval-implement` trio.
7. **Re-scan + record** — `/eval-rescan <label>` (or directly
   `eval-jss iterate record post-<label>`). The post-snapshot's
   "Delta" block diffs against the previous iteration.

Two harness commands the loop relies on:

- `eval-jss iterate refresh` — wipe, rescan, and restore labels onto
  the new violation rows. Reports orphans (labels whose violation no
  longer fires); `--save-orphans <path>` dumps them to JSON.
- `eval-jss report --diff` — per-rule TP/FP/precision deltas between
  the two latest iterations in `precision-history.db`. Add
  `--against <id>` to compare against an older iteration.

`--pinned-only` restricts precision stats to violations from the
vignette named in `corpus-manifest.csv::vignette_file` (plus every
sibling `.bib` in the vignette's directory, since `\bibliography{...}`
resolves there). This isolates rule quality on canonical inputs from
noise introduced by other vignettes in the tarball.

`eval/precision-history.db` is append-only and independent of
`eval/eval.db` — wiping the raw-violations DB does not erase the
precision trend.
