Metadata-Version: 2.4
Name: modelcertify
Version: 0.2.0
Summary: Supply-chain certification for trained models
License: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
Requires-Dist: ruff>=0.5; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Dynamic: license-file

# modelcertify

Supply-chain certification for trained models.

A model is only as trustworthy as the run that produced it. modelcertify
makes that run auditable and certifies it against a written policy.

* A **run proof** hashes every artefact the run produced (config, metrics,
  checkpoints, weights) and combines the hashes into a Merkle **root**. Change
  one byte of one artefact and the root changes.
* An **attestation** is a signed statement by an issuer that they vouch for a
  proof root.
* A **certificate** is a policy evaluated against a run's evidence: it
  records every gate, how each one came out, the proof root it is bound to,
  and the issuer who signed it.
* An **evaluation suite** (P4.x) is a private set of scored cases. Running it
  produces a report whose **suite digest** is committed to the proof, so a
  policy can gate on exactly which suite was used without shipping the suite.
  The runner supports two modes: `quick` (deterministic subsample, for
  iteration) and `full` (every case, for the attestable result). Scoring is a
  dependency-free port of the decision-index primitives (accuracy, macro/binary
  F1, NDCG@10, MRR, chance baselines).

The certificate never overstates. It carries the gate results, so a reader can
see *why* something was certified, not just that it was.

## Install

```bash
pip install modelcertify
pip install -e ".[dev]"           # from a checkout
```

modelcertify has **no runtime dependencies**: it is stdlib-only on purpose, so
it can be dropped into an air-gapped review environment.

## Relationship to iroh-tinker

modelcertify is a standalone product. It does not import iroh-tinker, and it
does not need to: it certifies a run's *artefacts*, wherever they were
produced.

The natural pairing is that an iroh-tinker run writes its artefacts (the
launch spec, the metrics, the checkpoints) to a directory, and
`modelcertify proof --dir <that directory>` turns them into a proof.

## CLI

```bash
modelcertify proof --run-id run-1 --dir ./artifacts --out proof.json
modelcertify policy --file policy.json --json
modelcertify certify --evidence evidence.json --policy policy.json \
    --issuer scai --key "$MODELCERTIFY_KEY" --out certificate.json
modelcertify verify --certificate certificate.json --key "$MODELCERTIFY_KEY"
modelcertify eval --suite suite.json --mode full --run-id run-1 --out certificate.json
```

Every command accepts `--json`.

Exit codes: `0` success, `1` the run was not certified (or the certificate did
not verify), `2` a usage or input error.

The signing key is hex. Pass it with `--key` or set `MODELCERTIFY_KEY`.

## Evidence and policies

An **evidence** document is what a policy is allowed to look at:

```json
{
  "proof": {"run_id": "run-1", "artifacts": [{"name": "model.safetensors", "digest": "..."}]},
  "metrics": {"eval_loss": 0.21, "eval_accuracy": 0.94},
  "provenance": {"dataset_id": "ds-1", "licence": "cc-by-4.0"}
}
```

A **policy** is a set of gates over that evidence:

```json
{
  "policy_id": "baseline",
  "name": "Baseline release policy",
  "required_artifacts": ["config.json", "model.safetensors"],
  "gates": [
    {"name": "loss", "kind": "metric", "field": "eval_loss", "op": "<=", "threshold": 0.5},
    {"name": "dataset", "kind": "provenance", "field": "dataset_id", "op": "!=", "threshold": null}
  ]
}
```

Gate kinds:

| kind | what it checks |
|---|---|
| `metric` | a measured metric against a threshold |
| `artifact` | that a named artefact is in the proof |
| `provenance` | a field of the run's provenance record |

Operators: `>=`, `<=`, `>`, `<`, `==`, `!=`, `in`.

## What it does not do

* It does not prove the *code* was correct, only that the artefacts are the
  ones that were certified.
* The signature is an HMAC, not a public-key signature: the verifier needs
  the issuer's key. That is deliberate for v1 (no key ceremony before there
  is a customer), and the interface is narrow enough to swap in Ed25519 later.
* A failed policy still produces a certificate. The point of a certificate is
  to record the outcome, including a refusal.

## Tests

```bash
pip install -e ".[dev]"
pytest
```

## Layout

```
src/modelcertify/
  proof.py       run proofs, artefacts, the Merkle root
  attestation.py signed statements about a proof root
  certify.py     policies, gates, evidence, certificates
  cli.py         modelcertify proof | policy | certify | verify
```

## Licence

Apache-2.0.
