Benchmark · v1 · null calibration

Measured false-alarm rates of common drift-detection configurations at canary scale

How often does each configuration alert when nothing happened — and how does that compare with what it states or implies? 500 seeded stable-agent histories per configuration, two scales, Wilson 95% intervals on every rate, one command to regenerate every number on this page.

measured 2026-08-09 dedrift 0.4.0 seeds 0–499 shipped

The question, and what this is not

Every drift tool implicitly promises: when I alert, something happened. Agent behavioral monitoring runs at canary scale — tens of rows per comparison window, by economic necessity — while most general-purpose drift machinery was designed and calibrated for production tables with thousands of rows. This page measures what six common configurations actually do on data with no drift and no config change, ever: the probability of at least one false alarm per check, and per 50-cycle monitored history, because that is how monitoring is used.

This is a validity-scale study, not a tool ranking. Tools built for thousand-row production tables are not wrong for being miscalibrated at 60-row windows — they are being used outside their domain of validity, and nothing warns the user. dedrift is included under identical conditions and reported with the same scrutiny, including its own uncalibrated diagnostic channel and the fixed-sample cumulative number that motivated its anytime-valid path.

Results — default suite scale (18 canaries × 7 repetitions)

126 records per cycle; reference window 378 rows (3 golden cycles), current window 126 rows. Rates are binomial over 500 independent seeded runs; brackets are Wilson 95% intervals.

Configuration	Default setting used	Nominal / stated	≥ 1 false alarm, per check	≥ 1 false alarm, 50-cycle history
General-purpose configurations
PSI, folk threshold 0.1	10 reference-quantile bins; flag at PSI ≥ 0.1	none stated	500/500 = 100.0% [99.2, 100.0]	—
PSI, folk threshold 0.25	same metric; flag at PSI ≥ 0.25 (“major”)	none stated	500/500 = 100.0% [99.2, 100.0]	—
Naive two-sample KS battery	α = 0.05 per (family × signature), no control	5% per test only	250/500 = 50.0% [45.6, 54.4]	—
dedrift, measured under the same protocol
dedrift PSI + validity guard	identical metric; refused where E[PSI] > 0.05	none stated (diagnostic)	0/500 = 0.0% [0.0, 0.8]	—
dedrift fixed-sample path (alerts)	BH-FDR q = 0.05 + materiality, dual baselines	FDR q = 0.05 per check	8/500 = 1.6% [0.8, 3.1]	358/500 = 71.6% [67.5, 75.4]
dedrift flag channel (Page–Hinkley / guarded PSI)	diagnostic flags, deliberately uncalibrated	none stated — published limitation	372/500 = 74.4% [70.4, 78.0]	—
dedrift anytime-valid path	e-processes + e-BH, golden baseline, twosample rates	≤ 0.05 lifetime per epoch	—	0/500 = 0.0% [0.0, 0.8]

k/n = runs with at least one false alarm over 500 seeded stable histories; brackets are Wilson 95% intervals. “—”: the configuration produces no such estimand. The dedrift flag channel and the PSI rows are diagnostics, not alerts; they are shown because hiding them would break the symmetry this table exists for. The anytime-valid path has no per-check cell: it makes no per-check claim, and its per-fold rate is over 25,000 folds against every other row's 500 runs, so printing it in that column would compare different estimands. Per-run and per-comparison raw rows: benchmark/results/.

Results — small scale (12 canaries × 5 repetitions)

60 records per cycle; reference 180 rows, current 60. This is the scale of dedrift's CI null-calibration gate, included so the scale-dependence is visible rather than asserted.

Configuration	Default setting used	Nominal / stated	≥ 1 false alarm, per check	≥ 1 false alarm, 50-cycle history
General-purpose configurations
PSI, folk threshold 0.1	10 reference-quantile bins; flag at PSI ≥ 0.1	none stated	500/500 = 100.0% [99.2, 100.0]	—
PSI, folk threshold 0.25	same metric; flag at PSI ≥ 0.25 (“major”)	none stated	500/500 = 100.0% [99.2, 100.0]	—
Naive two-sample KS battery	α = 0.05 per (family × signature), no control	5% per test only	268/500 = 53.6% [49.2, 57.9]	—
dedrift, measured under the same protocol
dedrift PSI + validity guard	identical metric; refused where E[PSI] > 0.05	none stated (diagnostic)	0/500 = 0.0% [0.0, 0.8]	—
dedrift fixed-sample path (alerts)	BH-FDR q = 0.05 + materiality, dual baselines	FDR q = 0.05 per check	16/500 = 3.2% [2.0, 5.1]	420/500 = 84.0% [80.5, 87.0]
dedrift flag channel (Page–Hinkley / guarded PSI)	diagnostic flags, deliberately uncalibrated	none stated — published limitation	349/500 = 69.8% [65.6, 73.7]	—
dedrift anytime-valid path	e-processes + e-BH, golden baseline, twosample rates	≤ 0.05 lifetime per epoch	—	1/500 = 0.2% [0.0, 1.1]

Why this happens — the arithmetic

None of the failure modes above are mysterious once the sampling arithmetic is written down. Each measured row has a closed-form prediction beside it.

PSI cannot say "stable" at small n

E[PSI] ≈ (B − 1) · (1/n + 1/m) — B bins, window sizes n, m (Yurdakul & Naranjo-style first-order null)

With the standard 10 bins at per-family canary scale (n = 63 reference rows, m = 21 current rows at suite scale; 30 vs 10 at small scale) the null expectation is 0.57 and 1.2 respectively — both multiples of the 0.25 "major shift" folk threshold, from sampling noise alone, on unchanged data. PSI is a large-sample index; below its validity scale the thresholds are tripwires laid inside the noise floor.

Uncontrolled per-column testing multiplies

P(≥ 1 false flag) = 1 − (1 − α)^k — k independent per-column tests at level α

At α = 0.05 and k = 12 columns the independence prediction is 1 − 0.95¹² ≈ 46% per report; at k = 72 (family × signature) it is 97.5%. Independence overstates or understates once tests correlate or go conservative on discrete columns — which is why we measure rather than extrapolate. Measured, the naive battery flags 50.0% of stable runs at suite scale: multiplicity did not disappear, per-test exactness did.

Per-check guarantees are not trajectory guarantees

P(≥ 1 alert by cycle T) = 1 − (1 − p)^T ≈ 1 − e^−pT — p per-check, T cycles

A perfectly calibrated per-check rate of 3% compounds to ≈ 78% over 50 checks. Any procedure whose control is per check — including dedrift's own fixed-sample path — accumulates false alerts without bound as monitoring continues. That arithmetic is the entire motivation for the anytime-valid path, whose claim is about the whole trajectory (ever-alert ≤ α), and why its row in the table is the only one whose nominal rate is a lifetime bound.

Methodology

Complete enough to reproduce without reading the code; the code is linked below for everything else.

Data generation. dedrift.sim stable-agent histories: no injected drift, no config change, at any cycle. Seeds 0–499, shipped verbatim in benchmark/seeds.txt; the first six cycles of every 53-cycle history are bit-identical to the per-check history at the same seed, so per-check and per-history estimands share data.
Scales. Suite: 18 canaries × 7 repetitions per cycle (the documented default suite). Small: 12 × 5 (dedrift's CI gate scale). Reference = first 3 cycles (golden); per-check methods adjudicate cycle 5; monitoring methods fold cycles 3–52 (50 cycles).
Tables under test. The per-record behavioral signature frame: 8 scalar channels (output_chars, output_words, tool_call_count, tool_order_inversions, steps, retries, latency_ms, tokens_out) and 4 rate channels (refusal, format_valid, args_schema_ok_all, had_error); one all-NaN column (exact_match, no expected answers in the simulator) is dropped for every method alike.
Defaults only. Each method runs at its documented default configuration; no tuning against any tool. Ambiguous defaults are resolved by the tool's own documentation and recorded in benchmark/methods.py.
dedrift (0.4.0). The shipped fixed-sample path (BH-FDR q = 0.05 + materiality gates, dual baselines) and the anytime-valid path (e-processes + e-BH on the golden baseline, rate_model="twosample", lifetime α = 0.05), both at ProjectConfig() defaults, run cycle-by-cycle as in deployment. The uncalibrated diagnostic flag channel (validity-guarded PSI + per-stream Page–Hinkley) is recorded and published.
Intervals. Every headline rate is a binomial proportion over independent seeded runs with a Wilson 95% interval. Per-comparison rates pool correlated comparisons within a run; their intervals treat comparisons as the unit and are anti-conservative — the run-level interval is primary wherever both appear.
Methods considered and excluded. Driftbase 0.15.1 (public write path disconnected from the read path compute_drift uses), Nautilus Compass 2.3.0 (within-session persona drift — our histories have no sessions), UpTrain 0.7.1 (drift operator fails under every installable pydantic), whylogs 1.6.4 (no programmatic drift-alarm API at defaults), llm-drift and drift_orchestrator (not pip-installable). Full detail, each checkable: METHODS_CONSIDERED.md.

Reproduction

One command: make benchmark in the repository regenerates every number on this page (several hours on a 14-core machine; make benchmark-quick for a 20-seed smoke run).
Scripts and results: benchmark/ — runner, method adapters, the 500-seed list, and the raw per-run JSON this page is rendered from.
Pinned versions: benchmark/requirements-benchmark.txt; every results file also embeds the exact package versions it was produced with.
Paper: the same table appears in the preprint (validity-scale section), rendered from the same results files.

Right of reply

Everything needed to check us is public: the scripts, the raw per-run results, the seed list, and the pinned versions (linked above). If we misconfigured something, the table gets a correction entry in the changelog — not a quiet edit.

Changelog

v1.0 · 2026-08-09 Initial study: 500 runs × 5 configurations × 2 scales, measured with dedrift 0.4.0. This page is versioned and re-run when the measured configurations change; every revision is dated.