Metadata-Version: 2.4
Name: nunchi-drift
Version: 0.1.1
Summary: Decision-stability measurement for LLMs: run the same decision N times, get a flip rate with a bootstrap CI, and compare models against their own noise floor.
Author: Hada Kang
License: MIT
Project-URL: Homepage, https://github.com/hadakang/nunchi-drift
Project-URL: Repository, https://github.com/hadakang/nunchi-drift
Project-URL: Leaderboard, https://hadakang.github.io/nunchi-drift/
Keywords: llm,evaluation,consistency,stability,reliability,nondeterminism,agents,tool-calling,drift
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Dynamic: license-file

# nunchi-drift

**Cassette tests can't see a model that flips its decision 20% of the time.**

Most LLM testing makes variance *disappear* — retries until green, `temperature=0`,
VCR cassettes that freeze one lucky sample. nunchi-drift does the opposite: it treats
run-to-run variance as the thing to **measure**. Give a model the exact same decision
N times and get an answer you can defend statistically:

- **flip rate** — P(two runs disagree), with a **bootstrap 95% CI**
- **cross-model distance vs. each model's own noise floor** — "these two models
  behave differently" only counts as a signal when it exceeds both models'
  self-disagreement, and a **permutation test** puts a p-value on it
- **pass^k** (τ-bench vocabulary) when you do have a success oracle

Zero dependencies. Stdlib only. Works with Anthropic, OpenAI, GLM, Qwen, DeepSeek,
or any OpenAI-compatible endpoint.

**Live demo**: a [7-model leaderboard](https://hadakang.github.io/nunchi-drift/)
(Claude / GPT / GLM families) measured with exactly these metrics — flip rates,
CIs, statistical ties, and cross-model signals over byte-identical inputs.

## Quickstart

```python
from nunchi_drift import DecisionTask, measure, compare

task = DecisionTask(
    system="You are a refund-processing agent for an online store.",
    prompt="Customer requests a refund for a product delivered 15 days ago. "
           "Policy window is 14 days. The customer is polite and a longtime user.",
    actions=("approve", "reject", "escalate"),
)

r = measure(task, model="claude-haiku-4-5", runs=10)
print(r.summary())
# model=claude-haiku-4-5 n=10/10 flip_rate=0.356 [0.178, 0.489] majority=escalate (60%)

r.stable(threshold=0.1)          # False — this decision point is flaky
r.distribution                   # {'escalate': 0.6, 'approve': 0.3, 'reject': 0.1}

c = compare(task, "claude-haiku-4-5", "gpt-4o-mini", runs=10)
print(c.summary())
# claude-haiku-4-5 vs gpt-4o-mini: cross=0.640 floor=0.356 SIGNAL (p=0.014, permutation n=1000)
```

The action enum is enforced through tool/function calling — every sample is
directly countable, no output parsing.

## Why the statistics are not optional

These metrics were hardened in the measurement experiments behind the
[leaderboard](https://hadakang.github.io/nunchi-drift/), where every naive
shortcut failed in a documented way:

| Naive approach | What went wrong | What this library does |
|---|---|---|
| Baseline-first drift (compare run 1 vs rest) | A single outlier first run inflated drift **5×** (0.40 vs 0.08) | mean-pairwise: no run is privileged |
| Aggregate distributions only | Two agents swapping actions → aggregate unchanged, drift 0 | `per_agent_flip_rates` sees every flip |
| Point estimates, no CI | Leaderboard gaps of 0.003 read as rankings; with n=10 they were statistical ties | bootstrap CI on everything, ties reported as ties |
| "Model A ≠ model B" by eyeballing | Cross distance below either model's own noise floor means nothing | noise-floor gate + stratified permutation test |

## Cost, honestly

Repeated sampling is inherently uncacheable — that's the measurement.
Mitigations built in or recommended:

- **Opt-in, not blanket**: measure decision *points* (a router, a guardrail
  verdict, a trade action), not entire suites. One task × 10 runs on a small
  model costs about a cent.
- `workers=2` by default — bursts of identical calls are the worst case for
  rate limiters, and a failed run is lost signal (failures reduce n and are
  reported, never silently ignored).
- Run nightly, not per-commit, for regression tracking.

## API sketch

| Call | Returns |
|---|---|
| `measure(task, model, runs=10)` | `StabilityResult`: `.flip_rate` `.ci` `.distribution` `.majority` `.entropy` `.stable(threshold)` |
| `compare(task, model_a, model_b)` | `CompareResult`: `.cross` `.noise_floor` `.signal` `.p_value` |
| `metrics` module | `drift_score`, `mean_pairwise_drift`, `bootstrap_within_ci`, `permutation_test_cross`, `per_agent_flip_rates`, `pass_k`, … all pure functions over your own data |
| `caller=` parameter | swap the provider layer (custom endpoints, replays, tests) |

Multi-agent / multi-scenario batteries (the full leaderboard workflow with
per-scenario stratification) currently live in the parent project and are
the v0.2 extraction target, together with a scheduled model-change canary.

## Related work

pass^k is from [τ-bench](https://arxiv.org/abs/2406.12045). The CI agenda
follows ["Adding Error Bars to Evals"](https://arxiv.org/abs/2411.00640).
For continuous behavior-change auditing see
["An Auditing Test to Detect Behavioral Shift in Language Models"](https://arxiv.org/abs/2410.19406) —
the statistical backbone we build toward in the canary. Unlike semantic-entropy
methods, nunchi-drift needs no logits: enum-forced decisions make the
distribution directly observable, which is what keeps it cheap and provider-agnostic.

*nunchi (눈치): the Korean art of reading a room — noticing what shifted
without being told.*

MIT license.
