Metadata-Version: 2.4
Name: dedrift
Version: 0.2.1
Summary: Statistically rigorous behavioral drift detection for AI agents. Agents don't throw errors when they degrade — dedrift catches it.
Project-URL: Homepage, https://dedrift.ai
Project-URL: Repository, https://github.com/dedrift/dedrift
Project-URL: Issues, https://github.com/dedrift/dedrift/issues
Author-email: Ali Mahmoudi <alimahmoudidev26@gmail.com>
License-Expression: AGPL-3.0-only
License-File: LICENSE
Keywords: ai-agents,drift-detection,llm,monitoring,statistics
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: jinja2>=3.1
Requires-Dist: numpy>=1.26
Requires-Dist: pandas>=2.1
Requires-Dist: pydantic>=2.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: scipy>=1.11
Requires-Dist: typer>=0.9
Provides-Extra: dev
Requires-Dist: mypy>=1.10; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Requires-Dist: types-pyyaml; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: sentence-transformers>=2.3; extra == 'embeddings'
Provides-Extra: judge
Requires-Dist: anthropic>=0.40; extra == 'judge'
Requires-Dist: openai>=1.50; extra == 'judge'
Description-Content-Type: text/markdown

# dedrift

**Agents don't throw errors when they degrade. They keep confidently producing worse outputs. dedrift catches it.**

dedrift is an open-source Python package that detects silent behavioral drift in AI agents. It logs agent interactions, runs a frozen canary suite repeatedly, extracts behavioral signatures, and applies statistically rigorous drift detection with config-change attribution ("behavior shifted within 6h of the model version change").

## Why dedrift

Model updates, prompt edits, tool-schema changes, RAG index refreshes, and provider-side silent updates all shift agent behavior without throwing a single error. Existing monitoring catches crashes, not character changes.

dedrift's differentiation is statistical correctness:

- Every p-valued detector's false-alarm rate is *measured* by simulation tests in CI
  against stated acceptance bands — and the full pipeline's null alert rate is bounded
  (Wilson 95% upper bound < 5%) over 500 seeded stable-agent runs at a stated scale
  (12 canaries × 5 repetitions; see [the statistics page](https://dedrift.ai/statistics/)).
- All alerting passes through FDR control (Benjamini–Hochberg) over one primary test per
  channel; redundant tests run as corroboration outside the pool. No raw per-test
  p-values dressed up as alerts.
- Every alert requires both statistical significance and a configurable effect-size (materiality) threshold. Fewer, higher-confidence alerts.
- LLM outputs are stochastic: canaries run N times per cycle and we compare distributions, never single outputs.
- Dual baselines: every check runs against a rolling recent window (sudden breaks) and a frozen golden baseline (boiling-frog drift).
- Honest about power: small N means low detection power, and the docs show you the math instead of hiding it.

## Status

Pre-alpha, under active development. Working today: logging schema + store, canary runner
(N repetitions per cycle), Tier-1 structural signatures, Tier-2 semantic signatures
(pinned embedder, semantic displacement, MMD-RBF with a seeded permutation null and an
auto-calibrated materiality floor), the full detector battery
(KS/Levene/permutation-P95/two-proportion z/MMD as primaries; AD and Welch as
corroboration; PSI and Page–Hinkley as labeled diagnostics) with BH-FDR over primaries
and materiality gating, dual baselines, config-change attribution, and deterministic
markdown reports — all with calibration and power tests enforced in CI.

## Install

```bash
pip install dedrift              # core: zero ML dependencies
pip install "dedrift[embeddings]"  # + semantic signatures (sentence-transformers)
pip install "dedrift[judge]"       # + LLM-judge tier
```

For development: `pip install -e ".[dev]"`.

## Quickstart — a full simulated drift incident in five commands

No API keys needed: `dedrift sim` ships a seeded synthetic agent whose "model
version" is swapped mid-history, shifting output length, refusal rate, and
format validity — the classic silent degradation.

```bash
pip install dedrift
mkdir drift-demo && cd drift-demo

dedrift init                                   # create the project
dedrift embedder pin hash                      # optional: enable Tier-2 semantic signatures
dedrift sim --cycles 8 --change-cycle 7        # 8 canary cycles; model swap at cycle 7
dedrift baseline set cycle-0000 cycle-0001 cycle-0002   # freeze known-good cycles
dedrift check                                  # exits 2: DRIFT DETECTED (both baselines)
dedrift report --out report.md                 # deterministic markdown report
```

The report shows what shifted in plain units (e.g. refusal +21 pp, output
variance ratio ~9x), BH-adjusted p-values, and attribution: "nearest config
event: model fingerprint change, 0.0 h before onset." With your own agent,
replace `sim` with `dedrift canary run --suite canaries.yaml --agent
yourmodule:agent_fn --model 'provider/model@version'` on a schedule.

## Detection power: the honest table

Statistical power depends on sample size, and canary suites are small. For a
rate signature (e.g. refusal) at a 5% baseline in a family of 30 canaries,
two-sided α=0.05 per test (before FDR, which reduces power further), simulated
power to detect a shift of the given size:

| Repetitions N | n per window | +2 pp | +5 pp | +10 pp | +15 pp |
|---|---|---|---|---|---|
| 5  | 150 | 0.07 | 0.28 | 0.78 | 0.97 |
| 7 (default) | 210 | 0.11 | 0.42 | 0.91 | 1.00 |
| 10 | 300 | 0.13 | 0.60 | 0.99 | 1.00 |

Read the first column honestly: **a 2 pp refusal shift is essentially
undetectable at this scale.** dedrift's default materiality gate (2 pp) is a
floor on what may alert, not a promise of what will be detected. If small rate
shifts matter to you, grow the refusal-boundary family or raise N — power
comes from samples, not from wishful thresholds.

## dedrift Pro

A commercial tier with advanced inference is in development and lives outside
this repository. Email [support@dedrift.ai](mailto:support@dedrift.ai) to hear
when it ships.

## Contact

Questions, bug reports, or interest in being a design partner:
[open an issue](https://github.com/dedrift/dedrift/issues) or email
[support@dedrift.ai](mailto:support@dedrift.ai).

## License

AGPL-3.0-only. See `LICENSE`.
