Metadata-Version: 2.4
Name: szl-crosscheck
Version: 0.3.2
Summary: Cross-implementation receipt verification - two harness chains in, CONSISTENT/DIVERGENT/INCOMPARABLE out
License: Apache-2.0
Project-URL: Homepage, https://github.com/szl-holdings/szl-crosscheck
Project-URL: Source, https://github.com/szl-holdings/szl-crosscheck
Project-URL: Issues, https://github.com/szl-holdings/szl-crosscheck/issues
Project-URL: Changelog, https://github.com/szl-holdings/szl-crosscheck/releases
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Dynamic: license-file

# szl-crosscheck

<!-- szl:header v1 -->
<!-- Native identity: szl-brand@4309f6fce89179331b0b3ed8c99494e76c5be256; Orbit v2 icon. -->
<p>
  <a href="https://github.com/szl-holdings">
    <img src="https://raw.githubusercontent.com/szl-holdings/szl-brand/4309f6fce89179331b0b3ed8c99494e76c5be256/kit/logos/orbit-v2/suite/szl-icon-32.svg" alt="SZL Holdings — Orbit v2" width="32" height="32" />
  </a>
</p>

**Control before action. Evidence after.**

[Product](https://a-11-oy.com) · [Proof](https://a11oy.net) · [GitHub organization](https://github.com/szl-holdings)
<!-- /szl:header -->

[![PyPI](https://img.shields.io/pypi/v/szl-crosscheck)](https://pypi.org/project/szl-crosscheck/) [![Python](https://img.shields.io/pypi/pyversions/szl-crosscheck)](https://pypi.org/project/szl-crosscheck/) [![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/szl-holdings/szl-crosscheck/badge)](https://scorecard.dev/viewer/?uri=github.com/szl-holdings/szl-crosscheck)

Two independent implementations of the same measurement disagree quietly more
often than teams admit. This package ends that: feed it the receipt chains from
**both** harnesses and get one verdict — `CONSISTENT`, `DIVERGENT` (with the
metric and max relative delta named), `INCOMPARABLE` (no shared MEASURED lanes),
or `INVALID` (a chain failed verification — fail closed).

Built to settle the duplicated bench layer in this org: the stdlib repos
(`szl-retrieval-bench`, `szl-engine-bench`, `szl-quant-bench`) and the FastAPI
planes (`retrieval-bench`, `frontier-bench`, `quant-curve`). Their receipt
formats differ: the retrieval HTTP runner returns an **unchained** RunReceipt,
while the stdlib runner uses a `self_hash` chain. Explicit adapters preserve
these native records inside an adapter-owned integrity envelope. This does
not claim that the HTTP service emitted a chain or attested the caller's context.

## Doctrine

- Both chains verify before anything is compared. Tampering voids the run.
- Measured results require matching machine identity, CPU, OS, Python version,
  dataset/model revision, parameters, and corpus/query/qrels hashes. Source
  repository commits are recorded separately so independent implementations
  can differ. These fields are declared evidence, not hardware attestation.
- Only shared MEASURED lanes with shared metric keys are compared.
- Divergence is always named: lane, metric, max relative delta.
- The dual receipt commits to both terminal hashes and every lane verdict —
  same chains, same tolerance, same receipt, any machine.

## Usage

```bash
pip install -e . pytest && python -m pytest tests/ -q
```

```python
import json
from szl_crosscheck import crosscheck

a = json.loads(open("stdlib-chain.json").read())
b = json.loads(open("plane-chain.json").read())
print(crosscheck(a, b, rel_tol=0.01)["verdict"])
```

## Scope

Python 3.11+, standard library only. Never re-measures; verifies and compares
what the harnesses already receipted. Numbers from different hardware remain
incomparable by design.

When either measurement is an integer, the comparator preserves its exact
value through subtraction and relative normalization before rounding the
reported delta to a float. Distinct counts above the float exact-integer range
therefore remain distinguishable at zero tolerance, including mixed integer /
float inputs. Existing floating-only calculations and recorded SciFact receipt
bytes are unchanged. Tolerance decisions still use the reported floating delta;
this is not an arbitrary-precision receipt format. Existing numeric admission
and the small-value scale floor remain unchanged.

## Frozen full-population comparisons

The default API intentionally compares shared measured lanes and metric keys.
It cannot establish that omitted measurements were checked. Existing v2 report
bytes, historical evidence and the default behavior remain unchanged.

For an exact eligible population frozen **before** obtaining the two result sets,
supply the new keyword-only `required_metrics` argument:

```python
report = crosscheck(chain_a, chain_b, rel_tol=0.01, required_metrics={
    "bm25": ["ndcg@10", "recall@10"],
})
```

This opt-in returns `szl.crosscheck.v3`, with an integrity-bound
`szl.crosscheck-population/v1` record. Both sides must contain exactly the frozen
case-sensitive `(lane, metric)` identities as **MEASURED** entries. Counts alone
are insufficient. Missing and unexpected identities are listed separately for
each side; the full non-compared producer records remain in the input chains.
An explicit empty policy is `INCOMPARABLE`, not vacuous success. A nonempty lane
needs a nonempty unique list of metric names. Policy limits are 256 lanes, 4096
lane/metric pairs and 256 characters per trimmed name; malformed policy is
`INVALID`. These are implementation bounds, not a scientific sufficiency rule.

`population.complete` states only exact measured-identity coverage.
`population.comparison_complete` additionally requires compatible recorded
contexts and comparison of every expected pair. Removing a required measurement
from either side therefore cannot yield full-population `CONSISTENT` even when
the remaining shared subset agrees. A discrepancy in a present required metric
remains `DIVERGENT` while incomplete coverage stays explicit. An unexpected,
unrequested metric is reported as a population mismatch, not used to invent a
divergence among requested measurements. Signature/chain failures remain
`INVALID`; incomparable contexts never become complete comparisons.

The normalized expectation, its SHA-256, coverage observations and verdict are
all committed into the existing dual-report digest. Ordering the same metric
list differently does not change policy identity. This does **not** prove the
caller preregistered the population: policy and producer context remain
caller-declared, hashes are not signatures, and equal comparisons do not prove
accuracy, source independence or trusted execution. In v3,
`execution_authorized` is always false. Do not derive the required set from the
intersection being tested or pool v2 shared-only and v3 full-population claims.

The existing native BM25 adapters feed this mode without a new receipt format
or publisher. Source tests exercise all 256 pairs of subsets of a four-measurement
fixture against an independent test-owned set-equality oracle. That is one
exhaustive synthetic test family, not 256 independent real experiments.

## Native BM25 adapters

`szl_crosscheck.adapters.stdlib_retrieval(chain, context)` validates the native
stdlib chain and requires its final `run.context` to equal the supplied context.
Its final `run.result` must be a measured BM25 result at the declared cutoff.
`fastapi_retrieval(response, context, captured_inputs)` checks the HTTP response's
dataset, qrels, model/config and result hashes, then binds all three captured
input hashes. When `query_hash` is supplied, it must also match the captured
queries using the native producer's SHA-256 over UTF-8 JSON with sorted keys,
default ASCII escaping and default (noncompact) separators. Null, malformed or
mismatching supplied hashes are rejected. This matches the
[pinned retrieval-bench producer](https://github.com/szl-holdings/retrieval-bench/blob/d16b1c8945725f266af25dc0f28ff65314f6100d/retrieval/runner.py#L30-L65).

The normalized envelope records `native_query_hash_state = MATCHED` when that
hash matches. Historical responses that omit the field remain supported with
`native_query_hash_state = UNVERIFIED_LEGACY` and an explicit capture-only limitation;
an absent field is different from a supplied null or invalid hash. In both cases
request capture and context remain caller-declared. Matching hashes establish
input integrity, not proof of server execution, semantic accuracy or signer identity.
Both adapters compare only the shared nDCG@10 and Recall@10 by default; they
retain the complete native records, including non-compared metrics.

Context is an object containing `machine` (`identity`, `cpu`, `os`, `python`),
`dataset_revision`, `model_revision`, `parameters` (`top_k`, `k1`, `b`),
`input_hashes` (full SHA-256 values for `corpus`, `queries`, `qrels`) and
`source` (`repository` and full 40-character `commit`). Input hashes use UTF-8
JSON with sorted keys, compact separators, default ASCII escaping and no NaN.
Do not relabel fixture inputs as a real benchmark. Source commits and input
hashes support reproducibility; unsigned receipts do not prove signer identity.

Machine identity fields, dataset/model revisions, and source repository names
must be nonblank strings. Comparison contexts use canonical JSON equality,
excluding the independent implementation's `source` record: Boolean, integer,
and floating parameter values remain distinct, including inside nested values.
Different contexts across harnesses are `INCOMPARABLE`; mixed contexts within a
single chain are `INVALID`. JSON key ordering does not affect compatibility.
Native adapter context and cutoff bindings preserve the same JSON type
distinctions. The stdlib adapter binds the full native context, including source;
source is excluded only when comparing independent implementations.

## Recorded native run

`evidence/2026-09-05/crosscheck-native-scifact-20260905.json` records an actual
same-machine run over all 5,183 cached public SciFact documents and 300 test
queries. The FastAPI side was called using an actual loopback HTTP POST and
the ephemeral server was stopped afterward. With identical inputs and BM25
parameters, nDCG@10 was 0.6380 (stdlib) and 0.6643898053153046 (FastAPI).
The 3.972% relative difference is **DIVERGENT** at the declared 1% tolerance.
The native tokenizers differ; this receipt demonstrates the disagreement,
not interchangeability or a performance ranking. Full normalized input chains,
native records, source pins and input hashes are retained. CI recomputes both
the complete bundle hash and dual report; it does not rerun the corpus benchmark.

## Independent Lean-build comparison

`schemas/lean-build-comparison-v1.json` and
`szl_crosscheck.validate_lean_build_receipt` define a second, deliberately
narrow comparison plane for large Lean builds. The target is bound to an exact
artifact digest, Lean version, entrypoint, theorem name, reference-statement
digest and declared axiom surface. Each harness must record its own implementation
digest, runner identity, exact source revision, rebuild and kernel outcomes,
statement observation, axiom observation, final-chain `sorry` count, dependency
cone and log digest.

The semantic validator recomputes every comparison field and the final verdict:

- `CONSISTENT` means all required bounded observations agree across independently
  identified harness implementations and runners;
- `DIVERGENT` names a complete but disagreeing statement, axiom, dependency or
  kernel observation;
- `INCOMPARABLE` is mandatory when independence, target identity, execution,
  statement fidelity, dependency evidence or final-chain evidence is incomplete.

A caller cannot write `CONSISTENT` over disagreeing or incomplete evidence: the
receipt fails validation unless its comparison, verdict and ordered reasons are
exactly derivable from the observations. This contract never emits `BREAK` or
`PROOF_VALID`. Comparison consistency is not proof validity, theorem correctness,
security, or production readiness; the Lean kernel's own check remains the
relevant formal check.

```python
from szl_crosscheck import validate_lean_build_receipt

validated = validate_lean_build_receipt(receipt)
print(validated["verdict"], validated["verdict_reasons"])
```

## License

Apache-2.0 — see LICENSE (same text as the org's other bench repos).

Estate on Hugging Face: models, datasets and Spaces are published under the [SZLHOLDINGS](https://huggingface.co/SZLHOLDINGS) org.
