Research & Frontier

The proof artifacts no competitor publishes: a savings-vs-quality curve where every point carries a certification verdict, a never-regressing self-distilling loop, and savings telemetry you can verify yourself.

Certified compression frontier

Module: distil/eval.py · Command: distil eval [--corpus --runner --out]

Most compression tools publish a headline savings number. Distil publishes the full curve: a sweep of compression aggressiveness levels where every point is tagged with (token savings, decision-equivalence, certified/not). This locates the cliff past which lossy compression starts dropping agent decisions — and proves that distil sits safely inside the certified region.

How the sweep works

distil/eval.py defines frontier(), which measures two reference operating points (tier-0 lossless and distil (cache-aware)) and then sweeps a range of truncation limits ([4000, 2000, 1200, 700, 450, 300, 220, 160, 120, 90] chars) across the corpus. For each point it measures:

• Token savings — average reduction fraction over the corpus.
• Decision equivalence — average match rate between compressed and uncompressed decisions.
• Certified — both TOST non-inferiority passes and equivalence equals 100%. A single trajectory failure uncertifies the point.

Results are sorted by savings ascending and printed as the frontier table. Run offline with the deterministic runner (structural decision-equivalence). For real task-accuracy on a benchmark (tau-bench, SWE-bench, GSM8K) ingest its traces and run with --runner anthropic.

copy$ distil eval
certified compression frontier  (runner=deterministic)

level                   savings   equiv  certified  curve
--------------------------------------------------------------------------
distil (cache-aware)       8.4%    100%    ✔ PASS   ██
truncate@1200              7.2%     79%       ✘ —    ██
truncate@700              20.0%     36%       ✘ —    ████
truncate@300              41.3%      0%       ✘ —    █████████
--------------------------------------------------------------------------

distil: 8.4% savings @ 100% decision-equivalence — certified.
certified ceiling: 8.4% savings (beyond this, lossy compression drops decisions and the gate rejects it).

The raw curve can be written to JSONL with --out <dir> for CI integration or archiving.

Flags

Self-distilling keep-model

Module: distil/online.py · Command: distil online [--corpus --promote-to]

Most learned compressors train a keep-model on a generic judge applied to synthetic data. Distil trains on causal labels — certified-safe drops discovered by the ablation engine running on your own real traffic. The label source is the moat: a model that learns what actually changed a decision, not what a judge guessed.

The never-regressing loop

The loop has four steps, each grounded in the source:

1. Collect causal labels (collect_causal_labels()) — run counterfactual ablation (discover()) on every trajectory. Blocks that never changed a decision are PRUNABLE (label=0). Blocks that changed at least one decision are KEPT (label=1). Only VOLATILE blocks are scanned; the stable cacheable prefix is out of scope. Conflict resolution: keep wins (label=1 is never overridden to 0).
2. Retrain (retrain()) — featurize the labeled lines (9 features, same as the built-in logistic model) and train logistic regression with full-batch gradient descent, L2 regularization, and a deterministic SHA-256 train/test split. No random seed dependency.
3. Certify promotion (certify_promotion()) — wrap the new weights as a compression strategy, run the TOST non-inferiority gate on every single corpus trajectory. A single regression anywhere blocks the promotion.
4. Persist iff certified (online_round()) — write the new weights to --promote-to only if step 3 passed. A cycle that would degrade quality is silently discarded.

Observed performance on the bundled corpus

copy$ distil online --corpus ./mycorpus --promote-to ./weights/keep_model.json
self-distilling round — keep-model learns from causal labels, gated by non-inferiority

  n_labels: 286
  accuracy: 0.873
  f1: 0.932
  certified: True
  promoted: True

If the gate fails (a regression anywhere in the corpus), promotion is blocked and the output will show certified: False and promoted: False with a message that the candidate failed the non-inferiority gate.

Flags

Verifiable federated telemetry

Module: distil/telemetry.py · Command: distil federated-leaderboard --dir <dir> [--keys <keys.json> --html <out.html>]

Each opt-in instance contributes a signed, content-free savings aggregate with its certification verdict attached. The leaderboard aggregates only verified submissions — every number on the board is tamper-evident. No prompt text, no response content, no tool outputs ever leave the machine: only (instance_id, tokens_saved, dollars_saved, runs, certified, timestamp).

How signing works

Signing uses HMAC-SHA256 (symmetric: both sides share a per-instance key). The canonical form of the aggregate is json.dumps(fields, sort_keys=True, separators=(',', ':')) — deterministic across Python versions and dict orderings. Verification uses hmac.compare_digest for constant-time comparison.

Leaderboard rules

• Submissions whose instance_id has no entry in the keys map are rejected.
• Submissions whose signature does not verify are rejected.
• For each instance, the latest (highest timestamp) verified submission wins.
• Only certified submissions (where certified == True) contribute to headline totals. Uncertified instances are shown but their numbers do not roll into the aggregate.

copy$ distil federated-leaderboard \
    --dir ./submissions \
    --keys ./instance-keys.json \
    --html ./leaderboard.html
verifiable savings — 3 verified instance(s), 0 rejected

  totals (certified only): {'tokens_saved': 142816, 'dollars_saved': 0.071408, 'runs': 47, 'instances': 3}

The --html flag writes a self-contained dark HTML page (the same style as the gateway dashboard) showing per-instance verified savings. The page is served inline — no CDN, no external assets.

Flags