Measured false-alarm rates of common drift-detection configurations at canary scale

The question, and what this is not

Every drift tool implicitly promises: when I alert, something happened. Agent behavioral monitoring runs at canary scale — tens of rows per comparison window, by economic necessity — while most general-purpose drift machinery was designed and calibrated for production tables with thousands of rows. This page measures what six common configurations actually do on data with no drift and no config change, ever: the probability of at least one false alarm per check, and per 50-cycle monitored history, because that is how monitoring is used.

Results — default suite scale (18 canaries × 7 repetitions)

126 records per cycle; reference window 378 rows (3 golden cycles), current window 126 rows. Rates are binomial over 500 independent seeded runs; brackets are Wilson 95% intervals.

k/n = runs with at least one false alarm over 500 seeded stable histories; brackets are Wilson 95% intervals. “—”: the configuration produces no such estimand. The dedrift flag channel and the PSI rows are diagnostics, not alerts; they are shown because hiding them would break the symmetry this table exists for. The anytime-valid path has no per-check cell: it makes no per-check claim, and its per-fold rate is over 25,000 folds against every other row's 500 runs, so printing it in that column would compare different estimands. Per-run and per-comparison raw rows: benchmark/results/.

Results — small scale (12 canaries × 5 repetitions)

60 records per cycle; reference 180 rows, current 60. This is the scale of dedrift's CI null-calibration gate, included so the scale-dependence is visible rather than asserted.

k/n = runs with at least one false alarm over 500 seeded stable histories; brackets are Wilson 95% intervals. “—”: the configuration produces no such estimand. The dedrift flag channel and the PSI rows are diagnostics, not alerts; they are shown because hiding them would break the symmetry this table exists for. The anytime-valid path has no per-check cell: it makes no per-check claim, and its per-fold rate is over 25,000 folds against every other row's 500 runs, so printing it in that column would compare different estimands. Per-run and per-comparison raw rows: benchmark/results/.

Why this happens — the arithmetic

None of the failure modes above are mysterious once the sampling arithmetic is written down. Each measured row has a closed-form prediction beside it.

PSI cannot say "stable" at small n

With the standard 10 bins at per-family canary scale (n = 63 reference rows, m = 21 current rows at suite scale; 30 vs 10 at small scale) the null expectation is 0.57 and 1.2 respectively — both multiples of the 0.25 "major shift" folk threshold, from sampling noise alone, on unchanged data. PSI is a large-sample index; below its validity scale the thresholds are tripwires laid inside the noise floor.

Uncontrolled per-column testing multiplies

At α = 0.05 and k = 12 columns the independence prediction is 1 − 0.95¹² ≈ 46% per report; at k = 72 (family × signature) it is 97.5%. Independence overstates or understates once tests correlate or go conservative on discrete columns — which is why we measure rather than extrapolate. The per-column breakdown identifies exactly which columns break the arithmetic: low-cardinality integer channels (tool_call_count, steps, tool_order_inversions) are routed by the auto-selection heuristic to a chi-square test whose own small-cell validity condition is violated at these window sizes (the 0.7.21 implementation is scipy.stats.chisquare on the raw contingency counts — no merging, no exact fallback), and reject at 28–47% each on stable data while the well-behaved columns sit at 2–5%. Measured per column, pooled granularity:

Per-column false-flag rate over 500 stable runs, pooled granularity, Evidently DataDriftPreset 0.7.21 defaults. The three chi-square-routed columns are the low-cardinality integer channels; had_error and retries are structurally all-zero on the null, so their 0% is degenerate, not calibrated.

Per-check guarantees are not trajectory guarantees

A perfectly calibrated per-check rate of 3% compounds to ≈ 78% over 50 checks. Any procedure whose control is per check — including dedrift's own fixed-sample path — accumulates false alerts without bound as monitoring continues. That arithmetic is the entire motivation for the anytime-valid path, whose claim is about the whole trajectory (ever-alert ≤ α), and why its row in the table is the only one whose nominal rate is a lifetime bound.

Methodology

Complete enough to reproduce without reading the code; the code is linked below for everything else.

• Data generation. dedrift.sim stable-agent histories: no injected drift, no config change, at any cycle. Seeds 0–499, shipped verbatim in benchmark/seeds.txt; the first six cycles of every 53-cycle history are bit-identical to the per-check history at the same seed, so per-check and per-history estimands share data.
• Scales. Suite: 18 canaries × 7 repetitions per cycle (the documented default suite). Small: 12 × 5 (dedrift's CI gate scale). Reference = first 3 cycles (golden); per-check methods adjudicate cycle 5; monitoring methods fold cycles 3–52 (50 cycles).
• Tables under test. The per-record behavioral signature frame: 8 scalar channels (output_chars, output_words, tool_call_count, tool_order_inversions, steps, retries, latency_ms, tokens_out) and 4 rate channels (refusal, format_valid, args_schema_ok_all, had_error); one all-NaN column (exact_match, no expected answers in the simulator) is dropped for every method alike.
• Defaults only. Each method runs at its documented default configuration; no tuning against any tool. Ambiguous defaults are resolved by the tool's own documentation and recorded in benchmark/methods.py.
• Evidently (0.7.21). DataDriftPreset, all defaults, on the signature table — the documented route for drift over per-output descriptor tables — run pooled over the whole suite. We record the per-column flag rate (its auto-selected per-column tests at the default 0.05 threshold), the any-column rate, and its dataset-level verdict (drifted-column share ≥ 0.5, its own default). A per-canary-family arm was also run and is in the results JSON, but it is not tabled: splitting one report into six and taking any-of is a user's choice rather than a documented default, so its rate would partly be our own multiplicity rather than the tool's calibration.
• dedrift (0.4.0). The shipped fixed-sample path (BH-FDR q = 0.05 + materiality gates, dual baselines) and the anytime-valid path (e-processes + e-BH on the golden baseline, rate_model="twosample", lifetime α = 0.05), both at ProjectConfig() defaults, run cycle-by-cycle as in deployment. The uncalibrated diagnostic flag channel (validity-guarded PSI + per-stream Page–Hinkley) is recorded and published.
• Intervals. Every headline rate is a binomial proportion over independent seeded runs with a Wilson 95% interval. Per-comparison rates pool correlated comparisons within a run; their intervals treat comparisons as the unit and are anti-conservative — the run-level interval is primary wherever both appear.
• Methods considered and excluded. Driftbase 0.15.1 (public write path disconnected from the read path compute_drift uses), Nautilus Compass 2.3.0 (within-session persona drift — our histories have no sessions), UpTrain 0.7.1 (drift operator fails under every installable pydantic), whylogs 1.6.4 (no programmatic drift-alarm API at defaults), llm-drift and drift_orchestrator (not pip-installable). Full detail, each checkable: METHODS_CONSIDERED.md.

Reproduction

• One command: make benchmark in the repository regenerates every number on this page (several hours on a 14-core machine; make benchmark-quick for a 20-seed smoke run).
• Scripts and results: benchmark/ — runner, method adapters, the 500-seed list, and the raw per-run JSON this page is rendered from.
• Pinned versions: benchmark/requirements-benchmark.txt; every results file also embeds the exact package versions it was produced with.
• Paper: the same table appears in the preprint (validity-scale section), rendered from the same results files.

Right of reply

Maintainers of measured tools are invited to respond; responses will be linked here, unedited, beside the table they address. Everything needed to check us first is public: the scripts, the raw per-run results, the seed list, and the pinned versions (linked above). If we misconfigured something, the table gets a correction entry in the changelog — not a quiet edit.

Changelog

• v1.0 · 2026-08-09 Initial study: 500 runs × 7 configurations × 2 scales, measured with dedrift 0.4.0 and Evidently 0.7.21. This page is versioned and re-run when measured tools update; every revision is dated.