# streamsieve v0.2.0 — Full Machine-Readable Reference (`llms-full.txt`)

> SIMD C++20/AVX2 runtime (8-wide cache-tiled batch kernel) + `<4 ms` native CLI (`--jsonl` / CSV / `--season` / `--detrend`) + Python SDK with zero runtime dependencies + drop-in `river` & `scikit-learn` adapters + 7-tool JSON-RPC 2.0 stdio MCP server for strictly causal, anytime-valid multi-agent streaming inference, zero-echo CRDT belief fusion, budgeted hierarchical VOI attention, and delayed-label Byzantine-safe agent trust calibration.
> License: Apache-2.0 | Zero runtime dependencies | Verified on GCC `-O3`, `clang++ -Werror`, MSVC C++20, and ASan/UBSan/`float-cast-overflow` (79/79 tests passing, 0 compiler warnings).

---

## 1. Executive Summary & Decision Guide

`streamsieve` solves four coupled failure modes of AI agent fleets operating on live numeric streams:

| Problem | `streamsieve` Engine | Mechanism | Verified Headline Result (`benchmarks/results.json`) |
|---|---|---|---|
| Context blowup + continuous-peeking p-hacking + diurnal/trending false alarms + loud-channel traps | `EStream` (`ss_estream_t`) | 8-wide cache-tiled AVX2 SoA dual-resolution e-process (optional causal per-phase median deseasonalizer + first-diff/5-median-EWMA detrender + overflow-free Winsorized-MAD + clamped z + bounded squash + JZS + Beta-Binomial + dyadic checkpoints + exact online e-BH + lossless raw ring) | **255.67 M ch-ticks/s** native AVX2 (`26.9×` faster than `river.ADWIN`, `92.3×` faster than `river.PageHinkley`, `10.55×` vs NumPy/SciPy); **99.99% (`11,829×`)** token compression (`3.84M -> 325` tokens) with `0.0` exact raw drill-down error; **0.004–0.010** stationary peeking null FAR; **0.010–0.050** non-stationary null FAR (diurnal/trend/RW vs `83.5–100%` Welch/ADWIN/PH); **0.65–0.70** zero-mean `±3.5σ` burst recall (ADWIN/Welch = `0.00`); `40.8 ms` even at 55× peeking (faster than ADWIN `49.3 ms`) |
| Echo-chamber double counting + LWW overwrite churn + JSON wire bloat | `EvidenceLattice` (`ss_lattice_t`) | State-based Join-Semilattice CRDT keyed by 64-bit `provenance_id` + Vovk–Wang admissible cross-agent e-merging + `SSLT` v1 binary/Base85 wire codec | **0.0% overwrite churn**, **1.00× inflation** at $W=8$ (vs `96.5%` LWW churn / `10.39×` naive log-odds); **10.7× smaller** wire (`10,428 B` vs `111,143 B` JSON debate); **290,141** 100-item CRDT merges/s |
| Firehose attention under tight read/token budgets across $W$ agents | `VOISifter` (`ss_sifter_t`) | Hierarchical `site × channel × tick` mean-field Bayesian posterior + causal noise-scale normalization + sparse-burst mixture LLR + greedy Expected Brier-Risk Reduction with immediate discounting | **+0.702 Brier skill**, **78.3% Recall@Top**, **0.11%** inter-agent duplicate reads on 2,048-ch firehose ($W=4$) vs `-0.010` round-robin / `-32.42` naive top-raw-mean |
| Delayed/out-of-order labels + bloviators + Byzantine agents + regime flips + asymmetric cost | `TrustCalibrator` (`ss_trust_t`) | Causal pending-prediction store + non-negative online Platt shrinkage + fixed-share BOA + leave-one-out Ville e-value quarantine & raw-loss shadow recovery + 4-rate delayed AgACI + Bayes-optimal quantile action | **100.0% Byzantine quarantine**, post-flip Brier **0.1840** (vs `0.2179` ExpWeights, `0.2539` mean); **30.39%** cost reduction (`3.889` vs `5.587`) & **90.01%** empirical coverage for 90% AgACI intervals on real NOAA Seattle `temp_max` |

### When to Choose Incumbents Instead
- Need online supervised ML estimators (Hoeffding trees, online regression/classification): use `river`.
- Need single fixed-horizon offline statistical tests on static tables (no continuous peeking): use `scipy.stats` / `statsmodels`.
- Steep linear ramps: use `detrend_alpha=1.0` (causal first difference, 0 transient), NOT `0 < detrend_alpha < 1` (EWMA high-pass intended for slow drift). Non-integer or rapidly drifting seasonal periods should be residualized externally before feeding `EStream`.

---

## 2. Build, Install & CLI Reference

The CLI flags and adapters below refer to v0.2.0 on `main`. For the shared API and a checked result on both versions, see the [Python example](README.md#try-it).

```bash
# v0.1.0 wheel, CPython 3.9–3.13 on Linux x86-64 or Apple Silicon macOS
python -m pip install --no-index --only-binary=:all: --find-links https://github.com/pgg898200-ship-it/streamsieve/releases/expanded_assets/v0.1.0 streamsieve==0.1.0

# v0.2.0 Python SDK and CLI, from a main checkout with a C++20 compiler
python -m pip install .

# Or build from source: shared lib + native CLI + C++20/ASan/UBSan + 79 Python tests
make all test-cpp test-san
PYTHONPATH=python python3 -m unittest discover -s tests -p "test_*.py" -v
```

### Native CLI (`./build/streamsieve`, `3.87 ms` cold start) & Python CLI (`streamsieve` / `sieve`)
- `streamsieve sift [--alpha 0.05] [--top 10] [--warmup 32] [--season 0] [--detrend 0.0] [--jsonl] [--json] [FILE|-]`
  - Sifts CSV, TSV, whitespace, or **auto-detected strict RFC-8259 JSONL/NDJSON** streams from file or stdin (`-`).
  - Emits compact text brief or strict RFC-8259 JSON brief (`allow_nan=False`, UTF-8-validated string escaping) with effective sanitized parameters and channel handles (`ch:<idx>@t=<cp>..<tick>`).
- `streamsieve bench [--channels 1024] [--ticks 4000] [--json]`
  - Native 8-wide AVX2 throughput self-benchmark with exact e-BH verification.
- `streamsieve verify [--json]`
  - End-to-end self-verification of bit-exact zero-lookahead causality, CRDT axioms, VOI zero-overlap, and Ville quarantine.
- `streamsieve mcp`
  - Launches the zero-dependency JSON-RPC 2.0 stdio MCP server.
- `streamsieve version`
  - Prints version (`0.2.0`) and compile-time SIMD capability (`AVX2+FMA` or `scalar`).

---

## 3. Zero-Dependency Python API Reference (`import streamsieve`)

Cold import latency: `25.30 ms` subprocess median (`<1 ms` in-process, `70.8×` faster than `numpy+scipy+river`). Zero third-party runtime imports (`numpy`, `scipy`, `river`, `sklearn`, `torch`, `pandas` are never imported). Accepts Python sequences, `array('d')`, `memoryview`, or zero-copy NumPy `float64` contiguous arrays via `__array_interface__`.

### 3.1 `EStream`, `sift_matrix` & Drop-In Ecosystem Adapters
```python
from streamsieve import (
    EStream,
    EStreamDrift,          # Drop-in river.drift.ADWIN/PageHinkley replacement
    SklearnSieveDetector,  # Drop-in scikit-learn OutlierMixin/BaseEstimator detector
    sift_matrix,
    parse_channel_handle,
)

est = EStream(
    n_channels: int,
    *,
    warmup: int = 32,           # >= 12 finite warmup ticks per channel
    ring_capacity: int = 256,   # 0 disables lossless ring for max throughput
    alpha: float = 0.05,        # nominal e-BH FDR level in [1e-9, 0.5]
    burst_tail_p0: float = 0.025,
    two_sided: bool = True,
    season_period: int = 0,     # 0 or [2, 4096]; freezes 3-cycle per-phase medians
    detrend_alpha: float = 0.0, # 1.0 = causal 1st diff; (0,1) = 5-median EWMA high-pass
    names: list[str] | None = None,
)
est.update_row(values)                     # 1D sequence/buffer of length n_channels
est.update_batch(matrix)                   # 2D row-major sequence/buffer (T x n_channels)
est.update_one(channel: int, value: float) # sparse per-channel update
est.ebh_reject(alpha: float | None = None) -> list[int]
est.channel_status(channel: int) -> ChannelStatus  # exact cached e-BH rejected_ebh
est.topk(k: int = 10) -> list[ChannelStatus]
est.fetch_raw(channel_or_handle, *, limit=None, start_tick=None, end_tick=None) -> list[tuple[int, float]]
est.brief(max_channels: int = 10, names=None, alpha=None) -> StreamBrief
est.reset()
est.close()  # also supports `with EStream(...) as est:`

# One-liner helper:
est, brief = sift_matrix(
    matrix, names=None, alpha=0.05, warmup=32, season_period=0, detrend_alpha=0.0, max_channels=10, ring_capacity=256
)

# Drop-in river.drift detector (zero river dep):
det = EStreamDrift(alpha=0.05, warmup=32, season_period=0, detrend_alpha=0.0, auto_reset=True)
det.update(x)  # -> self; read det.drift_detected, det.warning_detected, det.e_value, det.mode, det.clone()

# Drop-in scikit-learn OutlierMixin detector (zero sklearn/numpy dep):
sk = SklearnSieveDetector(alpha=0.05, warmup=32, season_period=0, detrend_alpha=0.0, check_every=1)
sk.fit(X_base)                # atomic fit on (T, D) or (T,)
sk.partial_fit(X_more)        # incremental causal ingestion
scores = sk.decision_function(X_live)  # batch-size-invariant signed e-BH log-margins (<=0 anomaly)
labels = sk.predict(X_live)            # +1 calm, -1 anytime-valid e-BH anomaly
labels = sk.fit_predict(X_all)         # reset + strictly causal +1/-1 labels
```

### 3.2 `EvidenceLattice`
```python
from streamsieve import EvidenceLattice, hash_provenance

lat = EvidenceLattice(merge_mode: str = "strict_origin")  # or "max_reliability"
lat.set_prior(hyp_id: str | int, prior_log_odds: float)
lat.set_agent_weight(agent_id: int, weight: float)  # 0.0 strictly mutes both LLR and Vovk-Wang merged_e_value
was_echo: bool = lat.observe(
    hyp_id: str | int,
    provenance_id: str | int,
    llr: float,
    *,
    log_e: float | None = None,  # defaults to llr
    agent_id: int = 0,
    reliability: float = 1.0,    # [0, 1] quantized to Q16
    timestamp_ns: int = 0,
)
lat.merge(other: EvidenceLattice)            # in-place CRDT join
joined = lat.merged(other: EvidenceLattice)  # non-mutating CRDT join
summary: HypSummary = lat.query(hyp_id)      # posterior_prob, total_log_odds, merged_e_value, unique_evidence, echoes_suppressed, distinct_agents
lat.list_hyps() -> list[int]
wire: bytes = lat.to_bytes()                 # SSLT v1 binary
b85: str = lat.to_base85()                   # ASCII-safe wire format
lat2 = EvidenceLattice.from_bytes(wire)      # strict finiteness & bounds validation
lat3 = EvidenceLattice.from_base85(b85)
```

### 3.3 `VOISifter`
```python
from streamsieve import VOISifter

sf = VOISifter(
    n_channels: int,
    n_sites: int = 1,
    *,
    ticks_per_epoch: int = 8,
    prior_active: float = 0.02,
    hot_site_prior: float = 0.10,
    hot_site_boost: float = 0.25,
    shift_snr: float = 1.5,
    burst_prob: float = 0.15,
    burst_snr: float = 2.5,
)
sf.set_noise_scales(scales)                  # per-channel stds for loud-channel immunity
sf.observe_site_agg(site_id: int, agg_value: float)
sf.observe_mean(channel: int, mean_value: float)
sf.observe_ticks(channel: int, ticks)        # catches sparse bursts via exact mixture LLR
sf.posteriors() -> list[float]
sf.step_epoch(stay_prob: float = 0.99)
actions: list[VOIAction] = sf.allocate(
    budget_per_agent: int,
    n_agents: int = 1,
    *,
    allow_site_agg: bool = True,
    allow_mean: bool = True,
    allow_ticks: bool = True,
)  # VOIAction(agent_slot, tool_kind, target_id, cost, expected_voi, target_prob)
```

### 3.4 `TrustCalibrator`
```python
from streamsieve import TrustCalibrator

tc = TrustCalibrator(
    n_agents: int,
    *,
    fixed_share_alpha: float = 0.04,
    eta: float = 0.8,
    quarantine_e_thresh: float = 20.0,
    recovery_e_thresh: float = 5.0,
    platt_lr: float = 0.15,
    platt_l2: float = 0.02,
    conformal_alpha: float = 0.10,
)
dec: TrustDecision = tc.predict_prob(
    decision_id: str | int,
    agent_probs: Sequence[float | None],  # None or NaN = agent abstains
    *,
    underage_cost: float = 1.0,
    overage_cost: float = 1.0,
)
dec: TrustDecision = tc.predict_scalar(
    decision_id: str | int,
    agent_means: Sequence[float | None],
    agent_stds: Sequence[float | None] | None = None,
    *,
    underage_cost: float = 1.0,
    overage_cost: float = 1.0,
)
tc.resolve(decision_id: str | int, outcome: float)  # arbitrary delay / out-of-order
agents: list[AgentTrust] = tc.agents()  # agent_id, weight, platt_slope, platt_bias, ema_brier, quarantine_e, quarantined, n_resolved
```

### 3.5 Causality Verification (`assert_causal`)
```python
from streamsieve import assert_causal, verify_estream_causality, CausalityViolationError

assert_causal(fn, stream, probe_ticks=None)  # poisons s > t with +1e12, -1e12, NaN, +Inf -> enforces 0.0 diff at <= t
verify_estream_causality(matrix, probe_ticks=None, warmup=32, ring_capacity=256, season_period=0, detrend_alpha=0.0)
```

---

## 4. MCP Stdio Server (`streamsieve mcp` / `python3 -m streamsieve.mcp_server`)

Supports both newline-delimited JSON (NDJSON) and LSP-style `Content-Length` framing over stdio (JSON-RPC 2.0, MCP protocol `2024-11-05`). Hardened against malformed JSON, deeply nested payloads (`RecursionError`), `Infinity`/`NaN`/`10**400` coercion overflows, negative `Content-Length`, atomic session replacement on failed `append`/`reset`, and JSONL key-order alignment across `append=True` calls (always emits strict RFC-8259 JSON with `allow_nan=False`).

Exposed tools (7):
1. `sift_stream` — Ingest `rows`, `csv_text`, `jsonl_text`, or `json_data` (with optional `season_period`, `detrend_alpha`, `append`) into stateful `EStream` -> compact e-BH brief, drill-down handles, and token savings stats.
2. `fetch_raw_ticks` — Lossless raw `(tick, value)` drill-down by handle (`ch:3@t=180..600`) or channel index.
3. `lattice_observe` — Submit provenance-tagged evidence -> `was_echo` + updated `HypSummary`.
4. `lattice_merge` — CRDT-merge two session lattices or a Base85 `SSLT` payload -> converged summaries + wire size + Base85.
5. `lattice_query` — Query deduplicated posterior, Vovk–Wang merged e-value, unique count, and echoes suppressed.
6. `voi_allocate` — Feed site/channel/tick observations into stateful `VOISifter` -> non-overlapping multi-agent read plan + top posteriors.
7. `trust_step` — Run `predict_prob`, `predict_scalar`, `resolve`, or `status` on stateful `TrustCalibrator`.

---

## 5. Exception-Free C ABI (`csrc/streamsieve_capi.h`)

Cross-platform `SS_API` exports (`__declspec(dllexport)` on Windows MSVC, `visibility("default")` on GCC/Clang). Status codes: `SS_OK (0)`, `SS_ERR_NULL (-1)`, `SS_ERR_INVALID_ARG (-2)`, `SS_ERR_OOM (-3)`, `SS_ERR_BOUNDS (-4)`, `SS_ERR_PARSE (-5)`, `SS_ERR_BUFFER_SMALL (-6)`.
Core entry points:
- Utility: `ss_version`, `ss_has_avx2`, `ss_hash_provenance`.
- `EStreamEngine`: `ss_estream_default_config`, `ss_estream_create`, `ss_estream_destroy`, `ss_estream_reset`, `ss_estream_update_row`, `ss_estream_update_batch`, `ss_estream_update_one`, `ss_estream_ebh_reject`, `ss_estream_get_channel`, `ss_estream_topk`, `ss_estream_fetch_raw`, `ss_estream_n_channels`, `ss_estream_tick_count`.
- `EvidenceLattice`: `ss_lattice_create`, `ss_lattice_destroy`, `ss_lattice_set_prior`, `ss_lattice_set_agent_weight`, `ss_lattice_observe`, `ss_lattice_merge`, `ss_lattice_query`, `ss_lattice_list_hyps`, `ss_lattice_serialize`, `ss_lattice_deserialize`.
- `VOISifter`: `ss_sifter_default_config`, `ss_sifter_create`, `ss_sifter_destroy`, `ss_sifter_set_noise_scales`, `ss_sifter_observe_site_agg`, `ss_sifter_observe_mean`, `ss_sifter_observe_ticks`, `ss_sifter_posteriors`, `ss_sifter_step_epoch`, `ss_sifter_allocate`.
- `TrustCalibrator`: `ss_trust_default_config`, `ss_trust_create`, `ss_trust_destroy`, `ss_trust_predict_prob`, `ss_trust_predict_scalar`, `ss_trust_resolve`, `ss_trust_get_agents`.

---

## 6. Complete Verified Benchmarks Snapshot (2026-11-22, v0.2.0, Intel Xeon 2.30 GHz 2 vCPU)

- **Native 8-Wide AVX2 CLI Throughput:** `255.67 M ch-ticks/s` (`256×16k`, `16.02 ms`), `191.20 M/s` (`1024×4k`, `21.42 ms`), `146.69 M/s` (`4096×2k`, `55.85 ms`).
- **Incumbent Speedups:** `26.95×` faster than `river.drift.ADWIN(0.002)` (`1.21 ms` vs `32.62 ms`), `92.26×` faster than `river.drift.PageHinkley()` (`111.67 ms`), `10.55×` faster than vectorized NumPy/SciPy causal rolling z+BH (`7.58 ms` vs `79.92 ms`), `70.78×` faster Python cold import (`25.30 ms` vs `1,791.01 ms`), `3.87 ms` native CLI cold start (`463×` faster).
- **Stationary Continuous Peeking Null FAR (500 trials × T=600, peek/10, α=0.05):** Gaussian `0.010`, Student-$t(3)$ `0.008`, LogNormal `0.004`, `1e10_DC+1000xLoud` `0.006` (vs `0.196–0.834` rolling `|z|>3`, `0.476–0.542` peeking Welch, `0.108–1.000` PageHinkley).
- **Non-Stationary Null FAR (200 trials × T=600, peek/10, α=0.05):** `Diurnal_P24_6sigma` `0.010`, `LinearTrend_0.25/tick` `0.035`, `RandomWalk_UnitStep` `0.025`, `Diurnal_P24+LinearTrend` `0.050` (vs `0.835–1.000` peeking Welch, `1.000` ADWIN/PH on trends & random walks).
- **64-Ch Diurnal (`P=24`, `6σ`) Firehose:** `streamsieve_season24` `1.00` shift / `0.35` burst / `0.000` FDR / `1.000` Prec / `0.799` F1 in `1.72 ms` (vs `0.667` F1 / `125.49 ms` Welch; `0.618` F1 / `11.57 ms` ADWIN).
- **256-Ch Mixed Firehose (one-sided bursts):** `streamsieve_ebh` `0.000` FDR / `1.000` Prec / `0.883` F1 / `2.81 ms`; `streamsieve_ebh_peeking` `0.040` FDR / `0.909` F1 / `40.85 ms` (**faster than ADWIN** `49.33 ms`, `0.054` FDR, `0.862` F1; peeking Welch+BH `0.321` FDR / `0.799` F1 / `183.03 ms`).
- **256-Ch Zero-Mean Symmetric `±3.5σ` Bursts:** `streamsieve` `0.65–0.70` burst recall / `0.897–0.900` F1 / `0.000–0.022` FDR vs `0.00` burst recall for BOTH `river.ADWIN` and `peeking_welch_bh`.
- **Token Compression:** `64×500`: `99.47%` (`189×`, `1.22 ms`); `256×1000`: `99.93%` (`1,492×`, `5.13 ms`); `1024×2000`: `99.99%` (`11,829×`, `53.42 ms`, `3,844,353 -> 325` tokens, `1.00/1.00` Prec/Rec, `0` loud FPs, `0.0` exact raw drill-down error across 10,565 verified ticks).
- **Multi-Agent Probes:**
  - `echo` ($W=8$): `0.0%` churn, `1.00×` inflation, `+0.686` skill (vs `96.5%` LWW churn, `10.39×` naive log-odds inflation).
  - `sift`+`voi` ($W=4$, 2,048 ch): `+0.702` skill, `78.3%` Recall@Top, `0.11%` dup rate (vs `-0.010` RR, `-32.42` raw-mean).
  - `trust`+`delay` (4-ep delay, flip @ 60): `100.0%` Byzantine quarantine, `0.1848` overall / `0.1840` post-flip Brier (vs `0.2179` ExpWeights, `0.2539` mean).
  - `act` (NOAA Seattle `temp_max`, 4:1 cost, 1-day delay): **30.39%** cost reduction (`3.889` vs `5.587`), **90.01%** empirical coverage for 90% AgACI intervals.
  - `bits` (256 findings): `10,428 B` SSLT / `13,035` Base85 vs `111,143 B` JSON debate (`10.7×` / `8.5×` smaller).
  - `stop` ($T=400$, $\alpha=0.05$): `0.015` $H_0$ false-stop, `1.000` $H_1$ power, `120.0` mean stop tick (`70.0%` sample savings vs fixed horizon).
