Metadata-Version: 2.4
Name: szl-crosscheck
Version: 0.3.0
Summary: Cross-implementation receipt verification - two harness chains in, CONSISTENT/DIVERGENT/INCOMPARABLE out
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Dynamic: license-file

# szl-crosscheck

Two independent implementations of the same measurement disagree quietly more
often than teams admit. This package ends that: feed it the receipt chains from
**both** harnesses and get one verdict — `CONSISTENT`, `DIVERGENT` (with the
metric and max relative delta named), `INCOMPARABLE` (no shared MEASURED lanes),
or `INVALID` (a chain failed verification — fail closed).

Built to settle the duplicated bench layer in this org: the stdlib repos
(`szl-retrieval-bench`, `szl-engine-bench`, `szl-quant-bench`) and the FastAPI
planes (`retrieval-bench`, `frontier-bench`, `quant-curve`). Their receipt
formats differ: the retrieval HTTP runner returns an **unchained** RunReceipt,
while the stdlib runner uses a `self_hash` chain. Explicit adapters preserve
these native records inside an adapter-owned integrity envelope. This does
not claim that the HTTP service emitted a chain or attested the caller's context.

## Doctrine

- Both chains verify before anything is compared. Tampering voids the run.
- Measured results require matching machine identity, CPU, OS, Python version,
  dataset/model revision, parameters, and corpus/query/qrels hashes. Source
  repository commits are recorded separately so independent implementations
  can differ. These fields are declared evidence, not hardware attestation.
- Only shared MEASURED lanes with shared metric keys are compared.
- Divergence is always named: lane, metric, max relative delta.
- The dual receipt commits to both terminal hashes and every lane verdict —
  same chains, same tolerance, same receipt, any machine.

## Usage

```bash
pip install -e . pytest && python -m pytest tests/ -q
```

```python
import json
from szl_crosscheck import crosscheck

a = json.loads(open("stdlib-chain.json").read())
b = json.loads(open("plane-chain.json").read())
print(crosscheck(a, b, rel_tol=0.01)["verdict"])
```

## Scope

Python 3.11+, standard library only. Never re-measures; verifies and compares
what the harnesses already receipted. Numbers from different hardware remain
incomparable by design.

## Native BM25 adapters

`szl_crosscheck.adapters.stdlib_retrieval(chain, context)` validates the native
stdlib chain and requires its final `run.context` to equal the supplied context.
Its final `run.result` must be a measured BM25 result at the declared cutoff.
`fastapi_retrieval(response, context, captured_inputs)` checks the HTTP response's
dataset, qrels, model/config and result hashes, then binds all three captured
input hashes. When `query_hash` is supplied, it must also match the captured
queries using the native producer's SHA-256 over UTF-8 JSON with sorted keys,
default ASCII escaping and default (noncompact) separators. Null, malformed or
mismatching supplied hashes are rejected. This matches the
[pinned retrieval-bench producer](https://github.com/szl-holdings/retrieval-bench/blob/d16b1c8945725f266af25dc0f28ff65314f6100d/retrieval/runner.py#L30-L65).

The normalized envelope records `native_query_hash_state = MATCHED` when that
hash matches. Historical responses that omit the field remain supported with
`native_query_hash_state = UNVERIFIED_LEGACY` and an explicit capture-only limitation;
an absent field is different from a supplied null or invalid hash. In both cases
request capture and context remain caller-declared. Matching hashes establish
input integrity, not proof of server execution, semantic accuracy or signer identity.
Both adapters compare only the shared nDCG@10 and Recall@10 by default; they
retain the complete native records, including non-compared metrics.

Context is an object containing `machine` (`identity`, `cpu`, `os`, `python`),
`dataset_revision`, `model_revision`, `parameters` (`top_k`, `k1`, `b`),
`input_hashes` (full SHA-256 values for `corpus`, `queries`, `qrels`) and
`source` (`repository` and full 40-character `commit`). Input hashes use UTF-8
JSON with sorted keys, compact separators, default ASCII escaping and no NaN.
Do not relabel fixture inputs as a real benchmark. Source commits and input
hashes support reproducibility; unsigned receipts do not prove signer identity.

## Recorded native run

`evidence/2026-09-05/crosscheck-native-scifact-20260905.json` records an actual
same-machine run over all 5,183 cached public SciFact documents and 300 test
queries. The FastAPI side was called using an actual loopback HTTP POST and
the ephemeral server was stopped afterward. With identical inputs and BM25
parameters, nDCG@10 was 0.6380 (stdlib) and 0.6643898053153046 (FastAPI).
The 3.972% relative difference is **DIVERGENT** at the declared 1% tolerance.
The native tokenizers differ; this receipt demonstrates the disagreement,
not interchangeability or a performance ranking. Full normalized input chains,
native records, source pins and input hashes are retained. CI recomputes both
the complete bundle hash and dual report; it does not rerun the corpus benchmark.

## Independent Lean-build comparison

`schemas/lean-build-comparison-v1.json` and
`szl_crosscheck.validate_lean_build_receipt` define a second, deliberately
narrow comparison plane for large Lean builds. The target is bound to an exact
artifact digest, Lean version, entrypoint, theorem name, reference-statement
digest and declared axiom surface. Each harness must record its own implementation
digest, runner identity, exact source revision, rebuild and kernel outcomes,
statement observation, axiom observation, final-chain `sorry` count, dependency
cone and log digest.

The semantic validator recomputes every comparison field and the final verdict:

- `CONSISTENT` means all required bounded observations agree across independently
  identified harness implementations and runners;
- `DIVERGENT` names a complete but disagreeing statement, axiom, dependency or
  kernel observation;
- `INCOMPARABLE` is mandatory when independence, target identity, execution,
  statement fidelity, dependency evidence or final-chain evidence is incomplete.

A caller cannot write `CONSISTENT` over disagreeing or incomplete evidence: the
receipt fails validation unless its comparison, verdict and ordered reasons are
exactly derivable from the observations. This contract never emits `BREAK` or
`PROOF_VALID`. Comparison consistency is not proof validity, theorem correctness,
security, or production readiness; the Lean kernel's own check remains the
relevant formal check.

```python
from szl_crosscheck import validate_lean_build_receipt

validated = validate_lean_build_receipt(receipt)
print(validated["verdict"], validated["verdict_reasons"])
```

## License

Apache-2.0 — see LICENSE (same text as the org's other bench repos).
