Metadata-Version: 2.4
Name: runreceipt
Version: 0.2.0
Summary: Pin the workload before an eval run; recompute the denominator afterwards. Zero dependencies.
Author-email: Igor Voytyuk <vovchuklena@gmail.com>
License: MIT
Project-URL: Homepage, https://runreceipt.surge.sh
Project-URL: Worked example, https://green-that-lies.surge.sh
Keywords: evaluation,llm,testing,audit,denominator,ci
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.8
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# runreceipt

**A test run that reports `100%` over three of ten tests is green in every CI
you own.** `runreceipt` pins how many items a run is supposed to cover, then
recomputes the number afterwards and says out loud what went missing.

Zero dependencies. It is the thing that checks the other tools, so it imports
none of them.

```console
$ runreceipt fix --tool langsmith --inputs data/questions.jsonl -o run.lock
pinned 10 items from 1 input file(s); tool langsmith==0.10.18 (installed)
lock   run.lock  sha256 68d1e39f23d9a54a9d239ee1e0c796ec

$ pytest --junitxml=results.xml          # something silently drops 7 items
$ runreceipt verify results.xml --lock run.lock
runreceipt  RED
artefact    results.xml  (adapter: junit)

  pinned before run   10
  present in artefact 3
  reached a verdict   3   (passed 3, failed 0)
  lost                0
  skipped             0
  honest denominator  3

  reported pass_rate    1.0000   over denominator 3
  recomputed           1.0000   over denominator 3

  [RED] SUITE_SHRANK
      10 items were pinned before the run, 3 appear in the artefact, and the
      run surfaces no loss; the missing 7 are invisible to anyone reading the
      figure

  instrument   7 rules, each probed on a known-bad and a known-good input
  expires      langsmith==0.10.18 over input 92099c6974199bac
  receipt      sha256 54e156047a740dd5072477254c2ccc0f

$ echo $?
2
```

Note what the reported and recomputed figures do here: they **agree**. Both
say `1.0000`. Recomputing the artefact against itself can never find this
class of defect — the artefact no longer contains the missing items. Only the
number pinned *before* the run does. That is why `fix` exists.

## The three properties

Every receipt carries what the artefacts we examined did not:

1. **The denominator is stated, and it was fixed before the run.** Not derived
   from whatever survived.
2. **Losses are named as losses.** An item that raised is a non-pass, not an
   absent question. It stays in the denominator.
3. **The green has an expiry.** A receipt is valid for one release over one
   input set: `runreceipt check --lock run.lock` re-derives both and prints
   `VALID` or `EXPIRED`. A verdict with no expiry is a verdict about the past.

```console
$ runreceipt check --lock run.lock
VALID    pinned at 2026-08-14T20:17:20Z; langsmith==0.10.18 and input 92099c697419 are unchanged

$ echo '{"id": "q10", "question": "one more case"}' >> data/questions.jsonl
$ runreceipt check --lock run.lock
EXPIRED  a result derived under this lock no longer applies:
  - the input set changed (pinned 92099c697419, now 87b22cfe46d5)
$ echo $?
2
```

## Install

```console
pip install runreceipt
```

## Commands

| | |
|---|---|
| `runreceipt fix` | pin the item count, the input hashes and the installed tool version, before the run |
| `runreceipt verify` | recompute from the artefact; `--json`/`--out` for CI |
| `runreceipt check` | has the green expired? |
| `runreceipt recheck` | re-verify a receipt someone handed you, without trusting them |
| `runreceipt selftest` | probe every rule both ways, then replay the whole regression set |

**Exit codes:** `0` consistent · `2` findings · `3` artefact unreadable ·
`4` the instrument failed its own probe. There is no code that means
*probably fine*. An artefact that cannot be parsed is `RED-BLIND`, never a
pass.

`fix` takes nothing on trust. The item count is **counted** from the input set
rather than read from a field, and the tool version is resolved from the
installed distribution rather than from what you typed — if those disagree with
your claim, the receipt says so. A pinned number that was never measured would
just move the defect one file earlier.

## A receipt you can check without trusting who issued it

`verify --out receipt.json` writes a portable receipt: the verdict, the counts,
the findings, what it is valid for, and a `receipt_sha256` over its own body.
Whoever relies on that number — an insurer, a certifier, a buyer in
diligence — does not have to take the issuer's word for it:

```console
$ runreceipt recheck receipt.json
receipt     receipt.json
integrity   INTACT   the hash matches the contents
corroborate CORROBORATED  an independent recomputation of results.xml
            reproduces the receipt's verdict (RED), counts and findings
$ echo $?
0
```

Two independent checks, either enough to reject. **Integrity** re-derives the
receipt's own hash, so a receipt edited after issue — a `RED` quietly turned
`GREEN` — is caught with nothing but the file. **Corroboration** re-runs the
rules on the artefact the receipt names and reports any substance field where a
fresh computation disagrees, naming exactly what diverged rather than a bare
mismatch. Two honest runs differ only in their timestamp, which corroboration
ignores; a forged one does not.

```console
$ runreceipt recheck tampered.json
integrity   ALTERED  the hash does not match the contents - edited after it was issued
corroborate DIVERGES  a fresh recomputation disagrees on: verdict
$ echo $?
2
```

This is the point of a receipt over a screenshot: a vendor cannot certify
itself, because anyone downstream can re-derive the verdict from the bytes.

## What it checks

| code | the shape of the defect |
|---|---|
| `EMPTY_DENOMINATOR` | a rate printed over zero items; `NaN < threshold` is `false`, and `false` reads as *threshold not breached* |
| `SUITE_SHRANK` | fewer items in the artefact than were pinned, and nothing reports the difference |
| `LOSS_UNREPORTED` | items reached no verdict, and the run publishes the count it kept instead of the count it lost |
| `DENOMINATOR_OVER_SURVIVORS` | the divisor is exactly the number of survivors, so the score *rises* as more items fail |
| `COUNTER_MODULO_BUFFER` | the reported "processed" count is a flush remainder, so a full buffer counts as nothing |
| `VERDICT_CONTRADICTS_ERROR` | an item carries recorded error evidence and is still counted as passed |
| `FIGURE_MISMATCH` | the printed figure is not what the items add up to |
| `LOCK_TAMPERED` | the lock's own hash does not match its contents |

## The regression set is real releases

Each fixture reproduces a defect found in a shipped version of a tool people
run in CI today, and carries a `_reproduces` field naming it. `selftest`
replays all of them on every commit:

```console
$ runreceipt selftest
  promptfoo-zero-tests       expect RED   got RED       ok  EMPTY_DENOMINATOR
  deepeval-judge-raised      expect RED   got RED       ok  VERDICT_CONTRADICTS_ERROR
  langsmith-3-of-10          expect RED   got RED       ok  SUITE_SHRANK
  weave-mean-survivors       expect RED   got RED       ok  LOSS_UNREPORTED,DENOMINATOR_OVER_SURVIVORS,FIGURE_MISMATCH
  galileo-buffer-counter     expect RED   got RED       ok  COUNTER_MODULO_BUFFER
  langfuse-8-of-100          expect RED   got RED       ok  LOSS_UNREPORTED,DENOMINATOR_OVER_SURVIVORS,FIGURE_MISMATCH
  swebench-dropped-row       expect RED   got RED       ok  SUITE_SHRANK
  honest-all-pass            expect GREEN got GREEN     ok
  honest-some-fail           expect GREEN got GREEN     ok
  honest-losses-named        expect GREEN got GREEN     ok
  honest-skips-excluded      expect GREEN got GREEN     ok
  unreadable-artefact        expect BLIND got BLIND     ok
SELFTEST OK  7 rules probed both ways, 12 fixtures verified
```

The four `honest-*` fixtures are the half that matters. `honest-losses-named`
has two items that raised and reports `lost: 2` — it must stay **green**.
Errors are not the defect; hiding them is. A detector that reddens on any
imperfect run is a constant, and a constant detects nothing.

## Why the rules probe themselves

Each rule ships two inputs: one it **must** fire on and one it **must** stay
silent on. If any rule fails either probe, the whole receipt goes RED and the
exit code is `4` — even when no finding fired. Silence from a broken
instrument is not a pass, and that is precisely the failure being sold against
here: every defect in the table above is a green produced by a number nobody
could recompute.

The test suite applies the same standard to itself. One test disables every
rule and asserts the seven defects then stop being detected — if they still
came out RED, the RED came from somewhere other than the rules and the
regression set proves nothing. Another asserts that the same neutered build
still refuses to print GREEN, and exits `4` rather than `2`, so *"nothing
found"* and *"nothing looked"* never collapse into the same signal.

## Adapters

`native` · `junit` (pytest, and anything that writes JUnit XML) · `promptfoo`
· `deepeval` · `weave` · `galileo` · `langfuse` · `swebench`. Sniffed
automatically; override with `--adapter`.

Adding a vendor costs one adapter and zero rules: adapters lower an artefact
into a normalized `Run`, and the rules only ever see a `Run`.

## What this tool does not claim

- Not that any tool's authors acted in bad faith.
- Not that any number was falsified. Every defect above is an ordinary bug —
  a `bare except`, a buffer remainder, a mean over a filtered list.
- Not that a green receipt means the model is good. It means the figure
  survives recomputation against a denominator fixed before the run.
- Out of scope: how a leaderboard *ranks* or *badges* rows. `runreceipt` reads
  one run's artefact and says whether its number holds.

## License

MIT.
