Metadata-Version: 2.5
Name: etalon
Version: 0.0.1
Summary: Qualification evidence for private AI deployments
Project-URL: Homepage, https://github.com/coyos-ai/etalon
Project-URL: Website, https://github.com/coyos-ai/etalon-www
Project-URL: Packs, https://github.com/coyos-ai/etalon-packs
Author: Coyos d.o.o.
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: evidence,llm,qualification
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.11
Requires-Dist: httpx==0.28.1
Requires-Dist: jinja2==3.1.6
Requires-Dist: jsonschema==4.23.0
Requires-Dist: pydantic>=2.10.6
Requires-Dist: pyyaml==6.0.2
Provides-Extra: dev
Requires-Dist: pytest==8.3.5; extra == 'dev'
Provides-Extra: typesafe
Requires-Dist: typesafe-sdk==0.7.1; extra == 'typesafe'
Description-Content-Type: text/markdown

# Etalon

Qualification evidence for a specific self-hosted model deployment.

Etalon runs a versioned pack against an OpenAI-compatible endpoint on the
customer's network and writes an evidence bundle: fingerprint, raw inputs and
outputs, case results, coverage, a three-state decision, and a self-contained
HTML report. Nothing in the runner phones home.

This repository is the runner (Apache-2.0):
[coyos-ai/etalon](https://github.com/coyos-ai/etalon). The product site is
[coyos-ai/etalon-www](https://github.com/coyos-ai/etalon-www). Commercial pack
content stays in
[coyos-ai/etalon-packs](https://github.com/coyos-ai/etalon-packs). The packs in
this tree are public fixtures.

Etalon does not determine that a regulation, contract, or internal control has
been met. It records what was evaluated, under which configuration, and how the
pack's own policy read the scores.

## Documentation

The operator guide is [docs/README.md](docs/README.md): a localhost
qualification, how a bundle and a decision are produced, the CLI, packs,
judges, offline install, and signing.

## Open the prebuilt report

No install required. Open either file in a browser:

- [examples/sample-run/report.html](examples/sample-run/report.html) — `QUALIFIED` on the local fixture
- [examples/sample-run-broken/report.html](examples/sample-run-broken/report.html) — `NOT_QUALIFIED`, with failure excerpts and case redaction

Those checked-in reports keep the coverage sentence recorded when the bundle
was generated. v0.1 realised coverage is value-presence: a declared value
counts when it appears at least once. The number is not a cross-product of
coverage cells. `etalon inspect` and `etalon compare` state that limit when you
read the bundle, and a newly rendered report states it in the coverage section.

`etalon verify examples/sample-run` recomputes the bundle hashes after install.

## Demo, copy-paste

```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"

# terminal A — localhost only
python3 examples/mock_endpoint.py --port 8000

# terminal B
source .venv/bin/activate
etalon pack validate examples/demo-pack

etalon qualify \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator" \
  --output runs/

# use the printed run directory
etalon inspect runs/<run-id>
etalon verify runs/<run-id>
etalon report runs/<run-id>

# re-supply every recorded input. Exit 0 when nothing captured changed.
etalon check runs/<run-id> \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator"

# bare check does not exit 0: recorded triggers were not re-observed
etalon check runs/<run-id> --endpoint http://127.0.0.1:8000/v1

# sampling drift is review_required (exit 1) once other recorded triggers are re-supplied
etalon check runs/<run-id> \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/demo-pack/serving.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator" \
  --temperature 0.7

# quantization drift is always_requalify (exit 2)
etalon check runs/<run-id> \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/demo-pack \
  --serving-config examples/scenarios/serving-int4.yaml \
  --system-prompt examples/demo-pack/system_prompt.txt \
  --model etalon-demo-baseline \
  --operator "Demo Operator"

etalon compare examples/sample-run examples/sample-run-broken

# contact-routing fixture: reference metrics plus a judge overlay
python3 examples/mock_endpoint.py --port 8000 --pack examples/contact-routing

# live judge: export TYPESAFE_API_KEY and pip install 'etalon[typesafe]'
etalon qualify \
  --endpoint http://127.0.0.1:8000/v1 \
  --pack examples/contact-routing \
  --serving-config examples/contact-routing/serving.yaml \
  --system-prompt examples/contact-routing/system_prompt.txt \
  --model etalon-contact-baseline \
  --operator "Demo Operator" \
  --output runs/

etalon inspect runs/<run-id>
```

The fixture models are scripted, not weights:

| Model id | What it does |
| --- | --- |
| `etalon-demo-baseline` | Reference labels. Decision `QUALIFIED`. |
| `etalon-demo-quantized` | Scripted label misses. `NOT_QUALIFIED` (`requirement_miss:label_accuracy`). |
| `etalon-demo-degraded-prompt` | Ignores abstain and escalate. `NOT_QUALIFIED` (`requirement_miss:abstention_correctness`). |
| `etalon-demo-small` | Misroutes complaint labels. `NOT_QUALIFIED` (`critical`). |
| `etalon-demo-sampling` | Two label flips when temperature is above 0. Still `QUALIFIED`; label accuracy moves. |
| `etalon-demo-broken` | Non-JSON output. `NOT_QUALIFIED` (`schema` and `critical`). |
| `etalon-demo-error` | HTTP 500. `INDETERMINATE` (`endpoint_errors`), including when scored cases already miss. |

`examples/contact-routing` (`coyos.fs.contact-routing` @ 0.3.0) is the
**public synthetic fixture** for demos and tests. The **commercial pack** of
the same id and version lives in
[coyos-ai/etalon-packs](https://github.com/coyos-ai/etalon-packs)
(`packs/coyos.fs.contact-routing`). The commercial licence applies there;
this example stays under Apache-2.0. The fixture uses the same mock process
with `--pack examples/contact-routing`:

| Model id | What it does |
| --- | --- |
| `etalon-contact-baseline` | Referee labels on the system under test. Reference metrics pass. The run is `QUALIFIED` only when the judge returns usable verdicts inside the uncertainty maximum and calibration agreement holds. |
| `etalon-contact-security-leak` | Security tickets go to billing. `NOT_QUALIFIED` (critical misroute) when the capture completes. |
| `etalon-contact-abstain-miss` | Abstain becomes classify. `NOT_QUALIFIED` (abstention) when the capture completes. |
| `etalon-contact-soft` | Ambiguous tickets take the next public label. `NOT_QUALIFIED` when the capture completes. |

The pack judge is TypeSafe Jev (`transport: typesafe`), not a fixture chat model.
Set `TYPESAFE_API_KEY` and install `etalon[typesafe]` for a live judge. Without
the key the judge fails and the run is `INDETERMINATE`. CI mocks the SDK.
`offline-mock` remains for taxonomy tests. The mock endpoint still has
`etalon-judge-uncertain` and `etalon-judge-fault` for an `endpoint` judge transport;
this pack does not select that transport. See `docs/typesafe-judge.md`.

`python3 examples/run_degradation.py` prints the demo profile table and exits non-zero if a
row's status or reason class drifts. `pytest` runs the same gate. The harness
does not measure a real quantized model. The int4 serving file records a
quantization name on the fingerprint; the quantized fixture's decision change
is the scripted label-accuracy miss. If these profiles stop separating, the
pack thresholds are what to change.

## Commands

```text
etalon pack validate <pack>
etalon qualify --endpoint URL --pack DIR --operator NAME --output DIR
etalon report <run>
etalon report <run> --output report-copy.html
etalon report <run> --show-sensitive --output report-sensitive.html
etalon inspect <run>
etalon verify <run>
etalon verify <run> --public-key etalon-public.json --require-signature
etalon keygen --private etalon-private.json --public etalon-public.json
etalon sign <run> --key etalon-private.json
etalon export <run> --output auditor/
etalon check <run> --endpoint URL
etalon compare <run-a> <run-b>
```

`verify` answers whether the bundle is intact (exit 1 otherwise). The expected
evidence file set is authoritative. Verification fails when an expected file
is missing, an expected file is absent from `hashes.json`, an extra file is
in the run directory, a digest is malformed or uses another algorithm, the
manifest decision does not match `decision.json`, the fingerprint is
ambiguous, or file bytes do not match. `signature.json` is optional and is
not hashed. Unsigned bundles still verify. A signature that does not match
the bytes of `hashes.json` fails verification. Pass `--public-key` to require
the signature from that key. See [docs/signing.md](docs/signing.md).

`report` with no `--output` prints the existing `report.html` path and does
not rewrite `report.html`, `report.json`, or `hashes.json`. Showing sensitive
case content requires `--output`, which writes a derivative and leaves the
hashed bundle unchanged.

`check` answers whether the recorded fingerprint is still current:

- exit 0 — every recorded trigger value was re-supplied and matches
  (`no_material_impact` may still be listed; a stored `unavailable` gap is
  listed at its trigger severity and does not by itself change the exit code)
- exit 1 — a `review_required` trigger fired or its recorded value was not
  re-supplied, and nothing forces 2
- exit 2 — an `always_requalify` trigger fired or its recorded value was not
  re-supplied

`etalon check <run> --endpoint URL` alone does not exit 0. Human output for every
fired or not-reobserved trigger prints the severity, the trigger name, the
stored value, and the live value (`not_observed` when it was not re-supplied).
Fields that were not supplied at qualification are `unavailable` with a reason.
`check` lists those at their trigger severity rather than inventing a hash.
Weight files are never hashed as one blob; pass `--model-index` with per-file
hashes if you have them.

`etalon check` separates three kinds of row. Fired means the re-supplied value
differs. Not re-observed means a captured value was not re-supplied; that
changes the exit by the trigger's severity. Unverified gaps are stored
`unavailable` fields, printed at their severity, and they do not by themselves
change the exit. An ambiguous fingerprint (a mapped field is missing, or an
envelope contradicts itself) is printed on its own. It is not an unavailable
gap, and the check cannot exit 0.

`etalon compare` prints a table of decision status, metric point estimates with
confidence intervals, critical-event counts, and realised coverage. The
coverage line is value-presence, not a cross-product of coverage cells. It
then lists the fingerprint fields that differ and the trigger severity of each
(`always_requalify`, `review_required`, or `no_material_impact`). It also says
whether qualification evidence changed. That line is the decision, the
intervals, the critical-event count, and realised coverage. `--json` prints
the same comparison as structured objects, including each interval's lower and
upper bounds and `coverage_model`.

`etalon qualify` prints the realised coverage figure with the same value-presence
limit. `etalon inspect` does too, including for a bundle generated by an earlier
runner.

`qualify` exits 0 when a bundle was written, including `NOT_QUALIFIED` and
`INDETERMINATE`. Exit 2 means the endpoint did not complete the capture.

## Air gap

Direct dependencies are pinned in `pyproject.toml`. Transitive dependencies,
including the dev extra, are locked with hashes in `requirements.txt`. The
lock was generated with:

```bash
uv pip compile pyproject.toml --extra dev --generate-hashes --universal -o requirements.txt
```

`uv` is only the lock generator. Installation stays on `pip`. Regenerate the
lock with that command when a direct pin changes.

`--no-index` and `--find-links` install from a local wheelhouse. Those flags
do not disable the machine's network.

`examples/offline_install.sh` fetches the wheelhouse with the network on,
then installs in a new network namespace. It refuses to start `pip install`
unless a fresh sysfs lists only `lo` and a TCP connect to `1.1.1.1:443`
returns `ENETUNREACH` (errno 101). The procedure and the observed output are
in [docs/offline-install.md](docs/offline-install.md).

```bash
bash examples/offline_install.sh
```

That script succeeded on 2026-09-24 on Linux x86_64 with CPython 3.12.3.
The fetch environment upgraded pip to 26.2.1. `pip download` and `pip wheel`
used the network, including build isolation for the project wheel. The
install environment was a new virtual environment whose pip was 24.0 (not
upgraded). Isolation was `sudo unshare --propagation private --mount --net`.
Inside it, sysfs listed only `lo`, the connect returned errno 101, and
`pypi.org` did not resolve. Both install commands passed `--no-index
--find-links` and read only `/tmp/etalon-wheels`. The requirements install
also passed `--require-hashes`. After the script exited, the host still
reached `1.1.1.1:443`.

`etalon pack validate examples/demo-pack` printed `result: pack matches the schema and pack rules`.
`etalon verify examples/sample-run` printed `bundle hashes match`.
On this Linux host, pip ignored the Windows-only `colorama` line in the lock.

That is evidence for this install. It does not determine that a regulation
or an internal control has been met.

The HTTP client sets `trust_env=False` and talks only to `--endpoint`.
Credentials come from `ETALON_API_KEY` or `OPENAI_API_KEY`, never from a pack file.
A test asserts that qualify does not open a connection to any other host.

## Report redaction

Cases with `sensitive: true` are quoted as `[redacted]` in `report.html` and
`report.json`. The rest of the bundle keeps the captured text. The report says,
in each section, whether redaction was applied and to how many items. Tokens and
credentials are removed from the bundle entirely (`[secret-excluded]`), not kept
beside a mask.

## Language

Say "qualified under pack X", "evaluated against methodology Y", "evidence
generated by runner Z". Do not write that a run is a legal determination or
that a regulation has been met.

## Sign a bundle

`etalon keygen` writes an Ed25519 seed and a public key. The seed file is mode
`0600`. It is not encrypted. Keep it offline, out of the bundle, and out of
CI. The public key is what a verifier or a deploy job holds. There is no key
server. Details are in [docs/signing.md](docs/signing.md).

```bash
etalon keygen --private etalon-private.json --public etalon-public.json
etalon sign runs/<run-id> --key etalon-private.json
etalon verify runs/<run-id> --public-key etalon-public.json --require-signature

python3 examples/verify_bundle.py runs/<run-id> \
  --public-key etalon-public.json \
  --require-signature

etalon export runs/<run-id> \
  --public-key etalon-public.json \
  --output auditor/
```

The signature covers the raw bytes of `hashes.json`, so it covers every
recorded evidence digest, including the pack content hash. Changing an
evidence file and rewriting `hashes.json` without the seed fails verification.
`examples/verify_bundle.py` uses the standard library plus
`etalon/evidence/ed25519.py`. It does not install the runner and it does not
open a network connection.

A deploy workflow can fail closed on that result:

```yaml
- uses: coyos-ai/etalon/.github/actions/etalon-verify@main
  with:
    bundle: evidence/run
    public-key: etalon-public.json
```

The job fails when the bundle is missing, the hashes do not match, the
signature is missing, or the signature was not made with that public key.
The action checks qualification evidence. It does not decide that a control
has been met. `etalon export` writes `auditor-report.html` and `manifest.json`
for a person to read. That package is evidence for review.

## Hash a file without installing Etalon

`examples/hash_file.py` uses only the Python standard library. Bundle files in
`hashes.json` are the SHA-256 of the raw bytes. Prompt hashes on a fingerprint
use newline normalisation; pass `--text` for that.

```bash
python3 examples/hash_file.py examples/sample-run/manifest.json
python3 examples/hash_file.py --text examples/demo-pack/system_prompt.txt
```

Each line is `sha256:<64 hex>` and the path. Compare the digest with
`hashes.json` or with `prompting.system_prompt_sha256` on a fingerprint. This
does not verify a bundle. `etalon verify` checks a bundle when the runner is
installed. `examples/verify_bundle.py` checks a signed bundle with the
standard library and a public key, without installing the runner.

## Container

The image runs as the non-root user `etalon` (uid 10001). The working directory
is `/opt/etalon`, owned by that user, so a run directory can be written there.

```bash
docker build -t etalon .
docker run --rm etalon pack validate examples/demo-pack
```

Mount a pack or a run directory when you need to read or write files outside
the image.

## Layout

```text
etalon/                  Python package (cli, execution, evaluators, scoring,
                       fingerprint, triggers, qualification, evidence,
                       redaction, reporting, schema)
examples/demo-pack/    synthetic pack, no judge
examples/contact-routing/  contact-routing fixture with a judge overlay
examples/mock_endpoint.py
examples/sample-run/   prebuilt QUALIFIED bundle
examples/verify_bundle.py  air-gapped signature check
schema                 symlink to etalon/schema
```

A pack may declare a `rubric_judge`. `offline-mock` scores on the route
taxonomy and does not open a socket. `endpoint` sends the rubric prompt to an
OpenAI-compatible chat model (`--judge-endpoint`, defaulting to `--endpoint`,
and `--judge-model`). `typesafe` (alias `jev`) calls TypeSafe Jev with
`typesafe-sdk` and `TYPESAFE_API_KEY` (`pip install 'etalon[typesafe]'`).
Evidence records the judge model, prompt or question hash, prompt version,
rubric version, and uncertainty. On the typesafe transport the model id is
the id Jev returns, uncertainty is the Noul probability that the verdict is
uncertain, and choice probabilities and confidence are stored beside it.
Missing uncertainty is a judge failure: the run is `INDETERMINATE`, not a
model miss. Uncertainty at or above the pack maximum is `INDETERMINATE` when
the pack lists `judge_uncertainty`, and it is still stated when a hard miss
is already `NOT_QUALIFIED`.

Calibration cases are not sent to the system under test. The judge scores the
candidate outputs a person already scored. The run reports judge–referee
agreement, provenance (`expert-authored` or `synthetic`), and known-failure
cases. Agreement below the declared record is `calibration_invalid`.

Reference metrics stay the qualification decision. Inspect, compare, and the
HTML report show the reference route beside the judge where a judge ran.
A critical dimension scored only by a judge is still rejected. The demo pack
does not declare a judge. There is no calibration UI.

See [docs/README.md](docs/README.md) for the operator guide. Decision rules
are in `docs/overview.md`.
