Metadata-Version: 2.4
Name: synapsek-claims
Version: 0.1.1
Summary: SVC — Synapsek Verified Claims: neutral, cryptographic verification of measured claims about AI artifacts and pipelines
Author: Synapsek Labs
License: Apache-2.0
Keywords: ai,attestation,verification,reproducibility,llm,supply-chain
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: cryptography>=41
Provides-Extra: test
Requires-Dist: pytest>=7; extra == "test"
Dynamic: license-file

<div align="center">

# synapsek-claims

**Portable, verifiable receipts for AI evaluation results.**

*Verify who issued a result, which pinned evidence supports it, and whether you can reproduce it — in your own environment, with one command.*

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)]
[![CI](https://github.com/Synapsek-Labs/synapsek-verified-claims/actions/workflows/ci.yml/badge.svg)]
[![Tests](https://img.shields.io/badge/tests-29_passing-brightgreen.svg)](tests)

</div>

---

## The problem

The AI industry runs on vendor-reported numbers. The uncomfortable,
well-documented facts:

- A large share of published model benchmarks is **vendor-reported, never
  independently re-run** — public analyses have found the majority of
  leaderboard entries with no third-party reproduction.
- Reproduction failures are not hypothetical: high-profile releases have
  shipped with benchmark scores that could not be reproduced when
  re-measured (claims of ~57% vs ~33% measured on real suites).
- Integrity signing (Sigstore/OpenSSF-style, Hugging Face model signing)
  solved one narrow problem: *the bytes didn't change*. It says nothing
  about whether a **measured quality claim** re-derives from evidence.

`svc` (Synapsek Verified Claims) is the referee for the second problem. It
is a framework-agnostic, cryptographically verifiable layer that signs
**measured quality claims** about any AI artifact or pipeline — and lets a
buyer **re-derive every number with one command**.

## What one command does

```
$ svc verify claim.json --trusted-pubkey trusted-issuer.pub --base-dir . --rerun
=== SVC VERIFY synapsek.claim/knowledge-encoder-dedup/v3b-seed1 ===
  [PASS] L1 authenticity: valid
  [PASS] L2 integrity: 10 files match
  [PASS] L3 evidence: 2 traces match
  [PASS] L4 soundness: 1/1 values recomputed
  [PASS] L4b rerun: 1 claims re-executed
  [PASS] L5 claims: statuses + recommendations re-derived
  claim dedup_spearman: value=0.4172424238214698 recomputed=0.4172424238214698 status=proven [OK]
RESULT: PASS — all layers verified; claims re-derived
```

That is the entire buyer workflow. Exit code `0` = every layer you asked for
passed. What you have actually established depends on which layers ran —
a `PASS` with `--rerun` is a different (much stronger) statement than a
`PASS` without it. The table below says exactly what each layer does and
does not establish.

## The five layers — and what each one establishes

| Layer | What the buyer checks | What a PASS establishes | A PASS does NOT establish |
|---|---|---|---|
| **L1 authenticity** | Ed25519 signature under *your* trusted key; issuer key == trusted key | Who signed this claim, and that nothing changed after signing | That the numbers are good or even true |
| **L2 integrity** | SHA-256 of every pinned file (weights, code, data, docs) | The pinned bundle is exactly what the issuer signed | That the evaluation was run the way the docs say |
| **L3 evidence** | SHA-256 + item count + Merkle root of every evidence trace | The per-item evidence is complete and unedited (one swapped row fails this) | That the evidence came from a real run |
| **L4 soundness** | Every claimed value recomputed from the pinned evidence via its documented derivation rule | The claimed number is **supported by the pinned evidence** under the documented rule. For `field:`/`file:`/`attested:` rules this means *read and re-aggregated from verified inputs* — it is **not** a regeneration of the underlying model predictions | That the predictions themselves were generated by the pinned code |
| **L4b re-execution** (opt-in `--rerun`) | Re-runs the issuer's pinned evaluation command in your environment; compares its output to the claimed value within tolerance | The pinned eval **actually reproduces** the claimed number when you run it | That the eval measures what it claims to (the script is visible and hash-pinned — you read it; that judgment stays with you) |
| **L5 claims** | Every `proven`/`not_proven` status re-derived from the **recomputed** value + documented threshold; recommendation sets re-derived from proven labels | The claim's pass/fail labels and "recommended for" lists follow from the numbers; **a claim that over-claims FAILS** | That the threshold itself was a reasonable bar |

The read / recompute / re-execute distinction matters and is never elided:
reading a number from a signed summary (L4 via `file:`/`attested:`) is
weaker than recomputing it from row-level evidence (L4 via `field:`), which
is weaker than regenerating the predictions by running the pinned eval
(L4b). Each example's disclosures state which level applies to each of its
numbers.

If the issuer claimed more than the numbers support, **verification
fails** — at a specific layer, with the specific reason. That last
property, anti-overclaiming by construction, is what makes this different
from a signed eval dashboard.

| You worry about… | The layer that stops it |
|---|---|
| claim edited after signing (value, status, threshold) | L1 |
| rogue issuer self-attesting under its own new key | L1 |
| weights / code / data swapped in the claim dir | L2 |
| evidence trace edited, reordered, truncated | L3 |
| claimed value not supported by its own evidence | L4 |
| a claimed value that only *looks* supported because the threshold was checked against the claimed number instead of the recomputed one | L4 + L5 |
| number not reproducible by re-running the pinned eval | L4b |
| "proven" claimed below the documented threshold | L5 |
| a not-proven label quietly added to "recommended for" | L5 |
| claim circulated that was never registered | `--registry` |

## Quickstart

```bash
pip install synapsek-claims

# get the shipped examples
git clone https://github.com/Synapsek-Labs/synapsek-verified-claims.git
cd synapsek-verified-claims

# the dedup example pins a 160 MB weights file as a release asset — download it once:
mkdir -p examples/pipeline-dedup/artifact
curl -L -o examples/pipeline-dedup/artifact/encoder.pt \
  https://github.com/Synapsek-Labs/synapsek-verified-claims/releases/download/v0.1.0/encoder.pt

# buyer — verify the light example (no dependencies beyond the package):
svc verify examples/cartridge-v11/claim.json \
    --trusted-pubkey examples/trusted-issuer.pub \
    --base-dir examples/cartridge-v11 \
    --registry registry/registry.json

# buyer — verify with full re-execution (needs the pinned env: torch, transformers):
pip install torch transformers
svc verify examples/pipeline-dedup/claim.json \
    --trusted-pubkey examples/trusted-issuer.pub \
    --base-dir examples/pipeline-dedup --rerun
```

Notes:

- `--rerun` is **explicit consent to execute code** (the issuer's pinned
  evaluation, hash-pinned and readable before it runs). Without it, any
  claim with a `script:` derivation rule fails L4 with a clear "not
  executed" message — nothing runs silently, ever.
- Re-execution never goes through a shell: the pinned command is split into
  an argv list and executed directly. It runs in the claim directory with a
  timeout and with proxy/network env vars stripped (`--no-env-scrub` to
  disable), and only after L1+L2+L3 all pass.
- The pinned rerun command is exactly what the issuer pinned (here:
  `python3 …`). On systems where `python3` is not the real interpreter
  (e.g. the Windows Store alias), use WSL or provide a real `python3` on
  PATH — the command is deliberately not rewritten per platform.

Machine-readable reports: `svc verify ... --json`. Exit code `0` = PASS,
`1` = FAIL.

## Issuing a claim

```bash
svc keygen --out-dir ./keys
svc attest skeleton.json --key ./keys/issuer_ed25519 --base-dir ./claimdir
svc registry add registry.json --claim ./claimdir/claim.json --issuer-id you
```

## What a claim is

A claim is a signed JSON document:

- **subject** — what it's about: files (hashed) + evidence traces
  (hashed + Merkle-rooted, one JSON item per line);
- **pinned** — the environment, dataset, and config the measurement used;
- **claims** — each with a metric, a measured `value`, a documented
  `threshold`, a derived `status` (`proven` / `not_proven`), and a
  `derivation` rule telling the buyer how to recompute it;
- **recommendation** — labels the numbers support, and labels they don't;
- **disclosures** — honest limitations, in writing, including which
  verification level (read / recompute / re-execute) applies to each
  number;
- **signature** — Ed25519 over the canonical form of all of the above.

Derivation rules are a tiny grammar with **no hidden code**:

| rule | recomputation | input constraint |
|---|---|---|
| `field:<name>\|mean` / `field:<name>\|last` | aggregate a field over the evidence trace | the trace must pass L3 |
| `file:<path>#<field>` | read a field from a pinned JSON file | the file must be a manifest entry that passed L2/L3, and the path must stay inside the claim directory |
| `script:<path>#<id>` | run the issuer's pinned eval; compare `id` from its final JSON stdout | runs **only** with `--rerun` AND after L1+L2+L3 all pass; executed without a shell |
| `attested:<json.path>` | read a value from a pinned attestation document | the document must be a manifest entry that passed L2 |

Every file a derivation rule can read must be pinned in the manifest and
hash-verified before it is read. Non-finite values (NaN/Inf) — in claimed
values, recomputed values, or tolerances — always fail; they can never
"prove" anything.

## Two working examples (real, measured, verify PASS as shipped)

**`examples/pipeline-dedup`** — a knowledge encoder (2-layer transformer +
Perceiver resampler, K=128 latents) claiming Spearman **0.4172** between
latent similarity and token Jaccard over a pinned 100-doc holdout (4,950
pairs, row-by-row evidence). L4 reads the value from the Merkle-pinned
summary (a third party has independently recomputed the Spearman from all
4,950 rows and matched it). `--rerun` goes one level further: it
re-executes the pinned eval on CPU and reproduces the value
**bit-exactly** (tolerance 1e-6, observed drift 0).

**`examples/cartridge-v11`** — the Synapsek knowledge cartridge v1.1 wrapped
through the same pipeline: 13 claims. The 4 v0 gist dimensions are recorded
as **`not_proven`** — below the pre-registered bars (topic 22.67 vs ≥50,
dedup 0.122 vs ≥0.4, retrieval 9.33 vs ≥25, summarize 0.123 vs ≥0.2). An
honest claim is allowed to say *no*; the system enforces that it actually
means it. The 9 QA-fidelity claims are `proven`: 6 are re-derived from
pinned per-item QA traces (L4 via `field:`); the 3 cross-family 7B scores
are carried from a pinned attestation document (L4 via `attested:` — the
per-item traces for those live in that attestation's own run, disclosed in
the claim).

```
$ svc verify examples/cartridge-v11/claim.json --trusted-pubkey ... --base-dir .
  [PASS] L1 authenticity: valid
  [PASS] L2 integrity: 1 files match
  [PASS] L3 evidence: 6 traces match
  [PASS] L4 soundness: 13/13 values recomputed
  [PASS] L5 claims: statuses + recommendations re-derived
  claim gist_topic: value=22.67 recomputed=22.67 status=not_proven [OK]
  claim qa_qwen_s1: value=0.097809 recomputed=0.0978094028382634 status=proven [OK]
  claim qa_7b_s3: value=7.55 recomputed=7.55 status=proven [OK]
RESULT: PASS — all layers verified; claims re-derived
```

## Trust model

- **Ed25519** over deterministic canonical bytes (sorted keys, compact
  JSON, UTF-8). Same byte format as the Synapsek v1.1 attestation engine —
  the two ecosystems interoperate.
- Only the **public** key ships with claims. Buyers trust a public key held
  **out-of-band** (`examples/trusted-issuer.pub`, or the v1.1 attestation
  bundle); a claim cannot choose its own trusted key.
- The **registry** (`registry/registry.json`) is a plain-JSON trust anchor:
  claim id → issuer key → claim hash. Portable today, a hosted endpoint
  later. See `examples/TRUST.md`.
- Every layer reports independently: a failure names the layer, the file,
  the claim, and the reason.

## What this is — and is not

| | |
|---|---|
| ✅ Signs **quality claims** and proves they re-derive, at a stated level (read / recompute / re-execute) | ❌ Not a model-signing replacement — integrity signing is a commodity and a good one; SVC is complementary |
| ✅ Pins *your* evaluation; structurally hostile to fudging | ❌ Not a benchmarking service — it cannot make a bad number look good |
| ✅ `proven` = the recomputed value meets the documented threshold, and the claim says so | ❌ Not a quality certifier — the threshold is the issuer's disclosed commitment; the buyer judges it |
| ✅ Execution is consent-gated, manifest-pinned, and shell-free | ❌ Not a sandbox — a `script:`/`rerun` command you run on your machine can do what any command can; read it first, run it where you'd run that eval |

## Roadmap

- [x] claim schema v0.1, 5-layer verify, pinned re-run engine, registry, CLI
- [x] execution consent gates, no-shell executor, manifest-pinned derivation inputs, finite-value guards (2026-10-17 audit fixes)
- [x] two attested real-world examples, 29-test suite, Apache-2.0, CI
- [ ] public registry endpoint + claim URI scheme
- [ ] verifiable agent-execution records (tool calls, approvals, audit logs as evidence)
- [ ] batch verify; GPU/CPU cross-device tolerance policies

## Security & contributing

Found a verification bypass? Read **[SECURITY.md](SECURITY.md)** before
reporting. Contribution guidelines in **[CONTRIBUTING.md](CONTRIBUTING.md)**.

Design rationale and threat model: **[docs/DESIGN.md](docs/DESIGN.md)**.
Release/publish runbook: **[RELEASE-CHECKLIST.md](RELEASE-CHECKLIST.md)**.

## Tests

```bash
python3 -m pytest tests/ -q    # 29 tests: roundtrip, tamper L1/L2/L3/L4b/L5,
                                # execution-consent gates, overclaim, NaN guards,
                                # manifest/path-escape guards, UTF-8, registry, canonical
```

## License

[Apache-2.0](LICENSE) © Synapsek Labs
