Metadata-Version: 2.4
Name: deltacert
Version: 1.0.4
Summary: Calibrated divergence certification for LLM serving systems
License-Expression: Apache-2.0
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy
Requires-Dist: torch>=2.0
Requires-Dist: transformers>=4.38
Requires-Dist: accelerate>=0.30
Provides-Extra: bnb
Requires-Dist: bitsandbytes>=0.43; extra == "bnb"
Provides-Extra: peft
Requires-Dist: peft>=0.9; extra == "peft"
Provides-Extra: vllm
Requires-Dist: vllm>=0.4; extra == "vllm"
Provides-Extra: gptq
Requires-Dist: gptqmodel>=1.0; extra == "gptq"
Requires-Dist: optimum>=1.20; extra == "gptq"
Provides-Extra: validation
Requires-Dist: datasets>=2.20; extra == "validation"
Requires-Dist: lm-eval>=0.4; extra == "validation"
Requires-Dist: matplotlib; extra == "validation"
Requires-Dist: human-eval>=1.0; extra == "validation"
Dynamic: license-file

# DeltaCert

**Certify any change to your LLM serving stack — quantization, engine upgrades, batch size, model updates — with a mathematical bound, before you deploy.**

[PyPI](https://pypi.org/project/deltacert/) · [SPEC.md](SPEC.md) · [LICENSE](LICENSE)

```bash
pip install deltacert
```

---

## The catch

Standard benchmarks said this 4-bit quantized model was fine:

| Config | Short eval said | d_COMM | safe_until_token | failure_after_token | Verdict |
|---|---|---|---|---|---|
| Llama-3.1-8B fp16 → nf4 | GSM8K 5-shot exact-match +1.0% (looks safe) | 0.00 | 31 | 14 | **unsafe** |

GSM8K said +1.0% — looks safe. DeltaCert flagged it unsafe. The generated text actually forks from the fp16 model at **token 14**, on long-form coding generations, well before the math bound (token 31) would even flag it. Same nf4 config, three independent signals, one real finding: a standard short-form benchmark missed a regression that shows up on longer generations.

Reproduce it yourself:

```bash
deltacert generate-cases --model meta-llama/Llama-3.1-8B-Instruct --output cases.jsonl
deltacert certify --model meta-llama/Llama-3.1-8B-Instruct --quantization int4 \
    --checks trajectory --trajectory-cases cases.jsonl
```

Every number above traces to a real certificate in `validation_results/`; every row in the table below reproduces with one script.

## The proof

Seven real flagship tests, each a full before/after comparison on a real model, real GPU, real downstream benchmark:

| Change | Business gain | d_COMM | Downstream effect | Verdict |
|---|---|---|---|---|
| Llama-3.1-8B batch=1 → batch=64 | 64 concurrent requests, same GPU | 6.21 | GSM8K -1.0 pt | ✅ Safe |
| vLLM 0.8.5 → vLLM 0.9.0 | take the upgrade same-week, not months later | 16.09 | GSM8K -1.0 pt | ✅ Safe |
| KV cache default → fp8 (vLLM native) | 2x concurrent capacity | 4.83 | GSM8K -1.0 pt | ✅ Safe |
| gpt-4o-mini pinned snapshot → current alias | same-day provider-drift check | 6.65 | canary acc +0.0 pt | ✅ Safe |
| Standard decode → speculative decode (k=5) | claimed ~2x throughput | 15.38 | GSM8K +0.0 pt, **measured 0.28x** (slower) | ✅ Safe on quality, not on speed |
| Llama-3.1-8B fp16 → nf4 (W4) | +60% VRAM reduction | 0.00 | forks at token 14 on long generations | ❌ Unsafe |
| Llama-3.1-8B fp16 → GPTQ int4 | +75% VRAM reduction | 1.16 | GSM8K -8.0 pts | ❌ Unsafe |

Five safe, two unsafe. A tool that only ever says "safe" isn't measuring anything — the two unsafe rows above are DeltaCert doing its job.

## How it works

DeltaCert compares output distributions before and after a change. From the cosine similarity `c` between two runs, it computes:

```
Δ = 4c√(1-c²)        (commutator magnitude)
d = -log(Δ/2)        (algebraic distance)
divergence_bound = 2·exp(-d)
```

`d` is an algebraic distance; `2e⁻ᵈ` is the certified bound on output divergence — deterministic, minutes to compute, checkable at every token position, no eval harness or labeled data required.

> **"Certified" throughout this document means:** measured against a calibrated threshold with a stated bound — not a guarantee of downstream quality.

- **Full derivation, clamp behavior, per-method calibration (with sample sizes disclosed), and the top-k logprobs caveat:** see `SPEC.md`.
- **This README asserts. The spec defends. Nothing here is a proof.**

## Getting started

```bash
pip install deltacert

deltacert certify --model meta-llama/Llama-3.1-8B-Instruct --quantization int8
```

That uses DeltaCert's shipped reference calibration (from the 7-test suite above). For a threshold tuned to **your own model and workload**, run the sweep yourself:

```bash
deltacert capture --model your-model --output baseline.npz
deltacert capture --model your-model --quantization int8 --output candidate.npz
deltacert calibrate --baseline baseline.npz --candidates candidate.npz \
    --names int8 --downstream-file your_evals.json
```

`deltacert certify` always tells you when it's using the shipped calibration instead of your own.

## What it certifies

DeltaCert ships all 20 collectors described in the design — the code is real, tested, and doesn't get deleted just because a given check hasn't been run in a full end-to-end validation yet. Only 7 have real flagship validation results behind them so far (the table above); the rest are working code with the same math, not yet run through that process.

| # | Check | CLI-drivable | Status |
|---|---|---|---|
| 1 | `weight_quant` | ✅ | ✅ validated |
| 2 | `kv_cache_quant` | ✅ | ✅ validated |
| 3 | `batch_divergence` | ✅ (needs vLLM) | ✅ validated |
| 4 | `spec_decoding` | ✅ (needs vLLM) | ✅ validated |
| 5 | `engine_swap` | ✅ | ✅ validated |
| 6 | `provider_drift` | ✅ | ✅ validated |
| 7 | `trajectory` | ✅ | ✅ validated |
| 8 | `activation_quant` | ✅ | 🔬 code exists, not yet validated |
| 9 | `prefix_cache` | ✅ | 🔬 code exists, not yet validated |
| 10 | `lora` | ✅ | 🔬 code exists, not yet validated |
| 11 | `model_swap` | ✅ | 🔬 code exists, not yet validated |
| 12 | `prompt_swap` | ✅ | 🔬 code exists, not yet validated |
| 13 | `sparse_attention` | Python API only | 🔬 code exists, not yet validated |
| 14 | `moe_token_dropping` | Python API only | 🔬 code exists, not yet validated |
| 15 | `neuron_skipping` | Python API only | 🔬 code exists, not yet validated |
| 16 | `allreduce_tp` | Python API only | 🔬 code exists, not yet validated |
| 17 | `alltoall_ep` | Python API only | 🔬 code exists, not yet validated |
| 18 | `pipeline_parallel` | Python API only | 🔬 code exists, not yet validated |
| 19 | `kv_transfer` | Python API only | 🔬 code exists, not yet validated |
| 20 | `gradient_compress` | Python API only | 🔬 code exists, not yet validated |

"Python API only" means the check needs code you supply (a custom `compress_fn`, attention mask, etc.) — the CLI can't conjure that for you; see `import deltacert as dc; dc.certify_system(...)`.

## Limitations (stated up front, not discovered by you later)

- The 7 validated results above all use one model family (Llama-3.1-8B-Instruct) plus one hosted API (gpt-4o-mini). Cross-model generalization isn't proven yet.
- Per-method calibration (e.g. the bnb/GPTQ thresholds) is an initial calibration from n=5 configs on one model — not a settled constant. Run `deltacert calibrate` on your own model/workload rather than trusting the shipped default for anything production-critical.
- The provider_drift result above is a same-day proxy (pinned snapshot vs. current alias), not the real weekly-cadence drift measurement, which needs two runs across real time.
- `d_comm` is a reliable within-method damage indicator but is not directly comparable across different compression methods — see `SPEC.md` for the bnb-vs-GPTQ false-negative this caused and how it's handled.

## Roadmap

- More models, more downstream tasks — firming up per-method calibration beyond n=5
- Full validation pass on the remaining 13 collectors
- Real weekly-cadence provider_drift run (beyond the same-day proxy)

## Citation

If you use DeltaCert, cite it as:

```bibtex
@software{deltacert2026,
  title  = {DeltaCert: Calibrated Divergence Certification for LLM Serving Systems},
  author = {Shorya},
  year   = {2026},
  url    = {https://pypi.org/project/deltacert/}
}
```

## License

Apache-2.0. See [LICENSE](LICENSE).

## Contact

- Issues and questions: open a GitHub issue
- Full validation data: `validation_results/` in this repo
