Observability catches crashes.
dedrift catches character changes.
LLM systems fail differently: outputs stay well-formed while length, refusals, tool use, latency tails, and meaning quietly shift. Nothing throws — so nothing pages you.
Silent model updates
The API alias stays the same. The model behind it doesn't. Nobody emails you.
Prompt & tool-schema edits
A colleague "improves" the system prompt. Refusal behavior shifts overnight. No test fails.
RAG & pipeline refreshes
New embeddings, new chunking, new retrieval order — same code, different agent.
Complementary tools, different jobs
Observability platforms are the right tool for crashes, latency, and spend. Drift monitors ship useful shift scores at telemetry scale. dedrift answers a different question: did my agent's behavior change, and can I trust the alert?
| Capability | Observability dashboards | Raw PSI/ADP monitors | dedrift |
|---|---|---|---|
| Crashes, exceptions, latency, spend | ✓ core strength | — | — not an APM |
| Behavioral distributions on frozen canaries | — | partial raw shift scores | ✓ one primary test per channel |
| Calibrated false-alarm rate at canary scale | — | ✗ flagged 100% of stable checks in our harness | ✓ 3.2% measured, enforced in CI |
| Multiplicity control (BH FDR) + materiality gates | — | — | ✓ |
| Anytime-valid mode for continuous watch | — | — | ✓ lifetime error budget |
| Config-fingerprint attribution | partial deploy markers | — | ✓ model / prompt / tools |
Factual, not tribal: use all three. dedrift deliberately does not do uptime, traces, or billing.
A designed experiment, not a dashboard
Frozen canary prompts, run N times per cycle, compared as distributions — because a single LLM run is noise.
Freeze the canaries
A versioned suite across six behavioral families, happy-path to adversarial.
Run N reps per cycle
Every record carries a config fingerprint of your model, prompt, and tools.
Compare distributions
One primary test per channel → BH FDR adjustment → observed-effect materiality gate.
Dual baselines
Sudden, vs a rolling window; cumulative, vs a frozen golden baseline.
Attribute
Alerts matched to fingerprint changes — "consistent with", never "caused by".
v0.4.0 — the audited release
The independent audit's own harness re-confirms this build — every finding locked behind a regression test. Then there's the new reach.
Tool-order drift
A new tool_order_inversions signature detects tool-call-order drift — agents calling the
right tools in the wrong order, a channel most monitors never test. It flags the drift on the first
post-drift check (5/10 harness runs; a battery without the channel measures 0/30, blind by construction).
The battery grows to m≈336 primaries.
Pooled SAFE rate processes
Anytime mode runs two-sample SAFE rate processes, plus one pooled e-process per rate signature (battery 24 → 28). Broad drift detection roughly doubles: a +20pp refusal shift is caught in 100/100 runs, median 17 cycles; +10pp in 89/100, median 50.
Cycle-effect correction + persistence
Hosted models wobble cycle to cycle. cycle_effect="auto" and alert_persistence=2
reduce the false-alert rate — honestly, they don't eliminate it:
| wobble σ | record-level battery (default) | cluster-aware (auto) | auto + persist=2 |
|---|---|---|---|
| 0.00 | 3.0% | 3.3% | 0.0% |
| 0.05 | 36.5% | 34.0% | 6.0% |
| 0.10 | 71.0% | 64.0% | 31.0% |
| 0.15 | 89.0% | 81.3% | 47.0% |
| 0.25 | 98.5% | 94.0% | 66.0% |
Per-check false-alert rate, stable agent + latent per-cycle offset. The correction reduces but does not restore calibration; persistence halves it again; neither reaches nominal at canary scale — at these sample sizes a per-cycle offset is a real distributional change. Pinned/self-hosted: leave it off. Hosted with wobble: anytime mode (valid there) or auto + persistence, with these numbers in front of you. The full ladder and intervals →
Every audit finding, regression-tested
Streams mask to observed cycles, attribution events are labeled relative to onset, degradation can never mask drift — each a regression-tested answer to a finding from the independent audit, re-confirmed by the auditors' own harness. The battery's m-dependence is published wherever the headline null rate appears.
We swapped the model.
Nobody told the detector.
A scripted silent swap you can regenerate in four commands — 30 canaries × 7 reps, golden frozen at cycle 3, the swap at cycle 8 behind a changed model fingerprint.
- First post-swap check: DRIFT DETECTED on both baselines
- 21 signature groups co-shift, attributed to the fingerprint change at onset
- Same pipeline pages you on prompt edits and RAG refreshes
Measured, not asserted
Monitoring tools love p-values and hate publishing their false-alarm rates. We enforce ours in CI — a release cannot ship if the statistics fail their own audit.
- Every p-valued detector calibration-tested against a documented band, on every commit
- Pipeline null: 16/500 stable checks alerted (3.2%, Wilson upper 5.1%), family-wise over m≈336 primaries
- Anytime-valid mode: 2/500 stable runs ever alerted over 2,000 cycles (Wilson upper 1.5%)
- The edge of the guarantee, published: σ=0.25 wobble with persistent AR(1) offsets → 7.2% ever-alert vs the 5% budget (Wilson upper 9.8%) — the measured boundary, in the docs, not the fine print
- Vendor cross-check: raw PSI flags 92.5% of stable windows at the conventional threshold — dedrift's guard emits none
No unmeasured claims
If a number carries a p-value, the simulation proving its false-alarm rate runs on every commit. A release that fails its own audit doesn't ship.
Alerts must earn it twice
Multiplicity-adjusted equality-test evidence and an observed effect worth acting on. Statistically real but practically trivial doesn't page you — alert fatigue kills monitoring.
We publish what we can't detect
Small suites have limited power, and the docs show that math instead of hiding it — including the shifts this battery will likely miss. Overclaiming is just drift of another kind.
A guarantee with a measured edge
Anytime-valid mode bounds the whole watch horizon, not each check: 2 of 500 stable runs over 2,000 cycles. Where the assumptions break — σ=0.25 wobble with persistent AR(1) offsets — the measured 7.2% ever-alert rate against the 5% budget is published as the boundary of the guarantee. The trade-off, quantified.
Don't take our word for any of it — audit the statistics.
Open core, honestly priced
The statistics are free forever. The advanced tier funds the work.
Open source
- Full detector battery, BH FDR adjustment + observed-effect gating
- Anytime mode — lifetime-valid rate e-processes, assumptions and boundary reported
- Dual baselines, attribution, reports
- Calibration suite — audit us
- Runs on a laptop: JSONL + SQLite, zero ML dependencies
Design partner
- Hands-on setup on your agent
- Canary suite designed for your failure modes
- A statistician reading your drift reports
- Priority input on the roadmap
Every statistical guarantee — including anytime-valid mode — is in the open core, forever. A commercial hosted tier (managed monitoring, production-traffic drift) is in development — write to us to hear when it ships.