Metadata-Version: 2.5
Name: verdict-gate
Version: 0.1.0
Summary: VERDICT: evidence-governed admission of self-evolved skills for small LLM agents
Author: Kunal, Sahil Jain
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.12
Requires-Dist: jsonschema>=4.23
Requires-Dist: numpy>=2.1
Requires-Dist: pydantic>=2.13.5
Requires-Dist: pyyaml>=6.0.3
Requires-Dist: typer>=0.15
Description-Content-Type: text/markdown

# VERDICT

**Evidence-governed admission of self-evolved skills for small LLM agents.**

Self-improving agents keep an edited skill when it scores higher than its parent. With small models, repeated runs
of an *unchanged* skill already differ by several points, so many of those "improvements" are noise: in our
measurements keep-if-better admits a large share of skills that did not change at all (noise floors and gate
simulations in [results/tables/noisegate/](results/tables/noisegate/)). VERDICT is the package
that decides, from paired evidence, whether an evolved skill is really better, and records each decision as an
admission card.

> **Status:** pre-release. The `verdict` package (paired evidence, policies, evidence cards, replay, budget guard,
> freeze manifests; CLI `verdict`) is working and tested; the distribution name is not final (see "Release status").

## Repository layout

```
src/verdict/      the VERDICT package: core/ (statistics, policies, cards), adapters/, runners/, budget/, cli.py
tests/            offline tests for verdict (synthetic fixtures in tests/fixtures/)
schemas/          evidence_card.schema.json, the JSON Schema every card is validated against
examples/         long-format CSV import format and an offline quickstart
configs/          policies, frozen splits, experiment configs, noise floors (frozen at prereg-v1)
prereg/           pre-registration and freeze manifests (checked by scripts/freeze_check.py)
scripts/          experiment runners (paid, budget-guarded) and offline analyses
harness/          experiment harness that produced the results (own pyproject, see harness/README.md)
  metaskill/      from-scratch replication of MetaSkill-Evolve (arXiv:2607.05297): SealQA, OfficeQA, ALFWorld, ALE, GSM8K
  noisegate/      NoiseGate, the stdlib-only admission gate prototype (noise floor, paired tests, admission card)
  scripts/        run wrappers and analyses (index in harness/scripts/README.md)
  configs/        one YAML per model configuration; gate settings; ALE presets
  data/           frozen splits (SHA256SUMS) and a 10-question OfficeQA sample
  tests/          offline tests for metaskill and the scripts
results/          every result, as small files: run summaries, per-item outcomes, tables, ledger (see results/README.md)
docs/             replication notes, research records (plans, decisions, run log), release checklist (see docs/README.md)
```

## Install

```bash
uv sync                  # verdict + its dev group (pytest, ruff, the harness workspace member the tests need)
cp .env.example .env     # only for commands that call a model or a gated dataset; nothing below needs it
```

From PyPI: `pip install verdict-gate` (the import name and the CLI stay `verdict`).

Requires Python 3.12+ ([uv](https://docs.astral.sh/uv/) recommended; `.python-version` pins 3.13). Without uv,
`pip install .` (or `pip install -e .`) in a checkout installs the `verdict` package alone (numpy, pydantic, pyyaml,
jsonschema, typer), without the harness. A wheel carries the card schema and the policy, validity, strata and floor
configs it needs (`verdict/_data`); inside a checkout the package reads `schemas/` and `configs/` from the checkout.

## Quickstart (offline)

```bash
uv run python examples/quickstart.py
```

It imports `examples/long_format.csv` (the generic import format, [examples/README.md](examples/README.md)), then
decides three synthetic candidates against their parent under keep-if-higher and under the moderate policy, and
writes one evidence card (`card.json`, valid against [schemas/evidence_card.schema.json](schemas/evidence_card.schema.json),
and `card.md`). The same decision from the CLI:

```bash
uv run verdict compare --baseline tests/fixtures/replay/s0.json --candidate tests/fixtures/replay/c_good.json --policy configs/policies/moderate.yaml --fnoise tests/fixtures/replay/fnoise_m.yaml --out card.json
uv run verdict card render card.json
```

`verdict compare` exits with a code per decision (0 = PASS_GLOBAL; see `src/verdict/cli.py`), `verdict replay`
recomputes decisions for many arms, `verdict freeze` writes a freeze manifest, `verdict budget status` reads the
spend ledger.

## Tests

```bash
uv run pytest            # all offline tests: no network (blocked by conftest.py), no API keys, no raw runs needed
uv run ruff check .
```

## Reproducing the results

None of the commands below makes a model call or needs a key; they read tracked files only.

| What | Command | Check |
|---|---|---|
| Frozen splits, configs and H48 artifacts are the ones pre-registered | `uv run python scripts/freeze_check.py` | prints `all hashes match` |
| E2 replication summary by provenance tier | `uv run python scripts/analyze_e2.py` | `git diff --exit-code results/outcomes/p3_e2/` |
| The decision core on synthetic evidence | `uv run python examples/quickstart.py` | three decisions printed, cards schema-valid |
| Paper tables and figure data from committed artifacts | `uv run python scripts/make_tables.py` (`make paper-tables`) | `git diff --exit-code reports/paper/` |

Every number in [results/](results/README.md) cites the run it came from;
[docs/plan/RUNLOG.md](docs/plan/RUNLOG.md) (thesis experiments) and [docs/research/RUNLOG.md](docs/research/RUNLOG.md)
(earlier NoiseGate work) give the exact command of each logged run. The paid runners (`scripts/run_*.py`) all have a
`--dry-run` that prints the planned runs and the estimated cost; model calls go through OpenRouter and cost money.

Not yet reproducible from tracked data: the E3 decision analysis (`scripts/analyze_e3.py`), H48 Event 1, the E8
evolution and H48 Event 2 read raw arm files under `runs/` (not tracked). Harness commands run from `harness/`; see
[harness/README.md](harness/README.md).

## Data and licenses

Code is licensed under [Apache-2.0](LICENSE). Datasets keep their own licenses. What is in this repository:

| Benchmark | License | What is tracked here |
|---|---|---|
| SealQA (`vtllms/sealqa`) | Apache-2.0 | Question ids in frozen splits, per-question scores, evolved skills; no question text |
| FRAMES (`google/frames-benchmark`) | Apache-2.0 | Question ids in frozen splits; a synthetic test fixture (no real rows) |
| FinSearchComp (`ByteSeedXpert/FinSearchComp`) | CC-BY-4.0 (attribution required) | Task ids in frozen splits and summary scores; raw rows are downloaded at the pinned revision, never committed |
| τ²-bench / τ³ retail (`sierra-research/tau2-bench`) | MIT | Task ids, per-task rewards, self-proposed retail skills; the environment is checked out separately at the pinned commit |
| GSM8K | MIT | Ids and scores |
| OfficeQA | CC-BY-SA-4.0 | A 10-question sample, see [harness/data/officeqa_sample/LICENSE.md](harness/data/officeqa_sample/LICENSE.md) |
| Agents' Last Exam | (task data contains hidden reference answers) | **Never included**: task ids and scores only |

All of the above can be redistributed in this form; the CC-BY-4.0 and CC-BY-SA-4.0 rows need the attribution kept.

## Release status

The distribution name is still open (RUNLOG D10: `verdict` is taken on PyPI); it lives only in `pyproject.toml` and
`src/verdict/`, so the rename is one change. The implementation spec's `revision-eval` CLI is the `verdict` command
here.

## Citation

A CITATION.cff will be added with the thesis title. Until then, cite MetaSkill-Evolve (arXiv:2607.05297) for the
method the harness replicates.
