---
name: streaming-writer-exclusion-spike
description: >-
  Self-modifying spike/POC scaffold for eliminating overlapping bars caused by
  kintsugi's trailing-edge fill writing to streaming symbols that the sidecar owns.
  Contains root cause analysis, multiple spike approaches, POC verification scripts,
  and a findings log that updates as each approach is tested.
  Use when investigating or fixing persistent Stathera overlaps on BTCUSDT/ETHUSDT
  that grow over time despite a stable sidecar session.
allowed-tools: Read, Write, Edit, Bash, Grep, Glob, Agent
---

# Streaming Writer Exclusion Spike

ultrathink

## Self-Modification Protocol

This skill is a **living document**. As each spike is tested:

1. Update the [Findings Log](#findings-log) with results
2. Mark spike status: UNTESTED → TESTING → PASSED / FAILED / PARTIAL
3. If FAILED, record WHY and add to [Anti-Patterns](#anti-patterns-from-this-spike)
4. If PASSED, promote to [Winning Approach](#winning-approach) and draft implementation

---

## Root Cause (Definitive)

### The Problem

Kintsugi's `_trailing_edge_fill()` calls `backfill_gap()` for **ALL** symbols including
BTCUSDT and ETHUSDT (streaming symbols owned by the sidecar). This produces `+backfill`
bars with a **fresh stateless processor** that have different `first_agg_trade_id` values
than the sidecar's `+sidecar` bars. ClickHouse keeps both (different ORDER BY keys) →
persistent overlaps that grow with every kintsugi pass.

### Evidence

```
# Kintsugi trailing-edge fill writing bars for streaming symbols (2026-03-19):
15:07:46 — kintsugi: ETHUSDT @ 1000 dbps: trailing edge 39.2 min — backfilling via REST
15:25:17 — kintsugi: BTCUSDT @ 1000 dbps: trailing edge 37.8 min — backfilling via REST
15:25:55 — kintsugi: ETHUSDT @ 750 dbps: trailing edge 34.0 min — backfilling via REST
```

Meanwhile, the sidecar was actively streaming for these same symbols:

```
Sidecar up 7.1h, writing +sidecar bars for BTCUSDT and ETHUSDT
```

### Exact Code Paths

| Component             | File                 | Line      | What It Does                                                               |
| --------------------- | -------------------- | --------- | -------------------------------------------------------------------------- |
| Trailing-edge fill    | `kintsugi.py`        | 2021-2054 | `_trailing_edge_fill()` calls `backfill_gap()` or `backfill_all_gaps()`    |
| Caller in daemon loop | `kintsugi.py`        | 2226      | `trailing_bars = _trailing_edge_fill(symbol=None, ...)` — NO symbol filter |
| backfill_all_gaps     | `backfill.py`        | 1026-1049 | Iterates ALL (symbol, threshold) pairs from CH, including BTC/ETH          |
| backfill_gap          | `backfill.py`        | 365-530   | Fills trailing edge with fresh processor, writes `+backfill` bars          |
| Day-recompute guard   | `kintsugi.py`        | ~1870     | `_repair()` returns -1 for today's shards — but ONLY covers day-recompute  |
| Streaming symbols env | `deploy/sidecar.env` | 9         | `OPENDEVIATIONBAR_STREAMING_SYMBOLS=BTCUSDT,ETHUSDT`                       |

### Why the Today-Guard Doesn't Help

The today-guard (Writer Ownership #280) only covers kintsugi's **day-recompute** path
(`_repair()`). The **trailing-edge fill** is a completely separate code path that calls
`backfill_gap()` directly — it has NO guard for streaming symbols or today's date.

---

## Spike Approaches

### Spike A: Skip streaming symbols in trailing-edge fill

**Status**: PASSED (2026-03-19, empirically validated on bigblack)

**Hypothesis**: If `_trailing_edge_fill()` skips symbols that are actively streamed by the
sidecar, there will be zero overlaps between kintsugi and sidecar bars.

**Implementation** (minimal change, ~5 lines):

```python
# In _trailing_edge_fill() at kintsugi.py:2032, before calling backfill:
import os
streaming_symbols = set(
    s.strip() for s in
    os.environ.get("OPENDEVIATIONBAR_STREAMING_SYMBOLS", "").split(",")
    if s.strip()
)
# Filter out streaming symbols from backfill_all_gaps
```

**Pros**: Simplest possible fix. One env var check.
**Cons**: Relies on env var being set correctly. If sidecar isn't running, streaming symbols get no trailing-edge fill at all.
**Risk level**: LOW

**POC verification**: Run `scripts/poc_spike_a.py` on bigblack.

### Spike B: Symbol registry `streaming` flag

**Status**: UNTESTED

**Hypothesis**: Add a `streaming = true` field to `symbols.toml` for actively-streamed symbols.
`backfill_all_gaps()` filters out `streaming=true` symbols. More durable than env var.

**Implementation**: Edit `symbols.toml` + modify `backfill_all_gaps()`.

**Pros**: Registry is SSoT for symbol metadata. No env var coupling.
**Cons**: More files to change. Need maturin develop after symbols.toml edit.
**Risk level**: LOW

### Spike C: Sidecar health check before trailing-edge fill

**Status**: UNTESTED

**Hypothesis**: Before calling `backfill_gap()` for a symbol, kintsugi checks the sidecar's
`/health` endpoint. If the sidecar reports the symbol as active (WS connected, bars emitting),
skip the trailing-edge fill for that symbol.

**Implementation**: HTTP call to `localhost:8081/health` in `_trailing_edge_fill()`.

**Pros**: Dynamic — doesn't need env var or registry. Handles sidecar down/up transitions.
**Cons**: Adds network dependency. If sidecar is slow to respond, kintsugi blocks.
**Risk level**: MEDIUM (failure mode: what if health check fails?)

### Spike D: Unified writer lock (Redis)

**Status**: UNTESTED

**Hypothesis**: Before any writer (sidecar or kintsugi) writes bars for a (symbol, threshold),
acquire a Redis lock. Only one writer at a time per pair.

**Pros**: Most robust — prevents ALL concurrent writer scenarios.
**Cons**: Redis dependency. Lock acquisition adds latency. Complex failure modes (stale locks).
**Risk level**: HIGH (over-engineered for the problem)

---

## POC Test Protocol

Each spike is tested by:

1. **Deploy the change** to bigblack (rsync for Python changes)
2. **Restart kintsugi** (picks up new code)
3. **Wait 30 min** (at least 1 kintsugi pass)
4. **Query Stathera** for new overlaps:

   ```sql
   WITH bars AS (
       SELECT symbol, threshold_decimal_bps,
              first_agg_trade_id,
              lagInFrame(last_agg_trade_id, 1) OVER w AS prev_last_tid,
              first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta
       FROM opendeviationbar_cache.open_deviation_bars FINAL
       WHERE symbol IN ('BTCUSDT', 'ETHUSDT')
       WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode
                    ORDER BY first_agg_trade_id)
   )
   SELECT countIf(tid_delta < 0) AS overlaps, countIf(tid_delta > 0 AND tid_delta < 500000) AS gaps
   FROM bars WHERE prev_last_tid > 0
   ```

5. **Check kintsugi logs**: no `trailing edge` lines for BTC/ETH
6. **Check non-streaming symbols**: trailing-edge fill still works for FILUSDT, NEARUSDT, etc.

**Success criteria**: overlaps = 0 for BTC/ETH, trailing-edge fill active for non-streaming symbols.

---

## POC Scripts

### scripts/poc_spike_a.py

Minimal test: monkey-patch `_trailing_edge_fill` to skip streaming symbols, run one
kintsugi pass, verify no new overlaps.

```bash
# Run on bigblack:
~/opendeviationbar-py/.venv/bin/python3 \
  /path/to/poc_spike_a.py
```

### scripts/verify_stathera.py

Query ClickHouse for current overlap/gap counts. Reusable across all spikes.

### scripts/check_writer_versions.py

Show which writers (versions) produced bars for today, to confirm kintsugi is no longer
writing +backfill bars for streaming symbols.

---

## Anti-Patterns (From This Spike)

_Updated as spikes are tested. Each entry records what was tried and why it failed._

(none yet)

---

## Findings Log

_Updated after each spike test. Most recent first._

| Date       | Spike             | Status       | Overlaps Before | Overlaps After                | Notes                                                                                                                                                                                                                |
| ---------- | ----------------- | ------------ | --------------- | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 2026-03-19 | A: env var filter | **PASSED**   | 39 overlaps     | 0 overlaps                    | 8 streaming pairs skipped, 34 non-streaming filled                                                                                                                                                                   |
| 2026-03-19 | Post-A audit      | INFO         | n/a             | 3 intra-day + 21 day-boundary | BTC@250 has 3 residual intra-day from restarts. 21 day-boundary gaps on NEAR/SOL/SUI/WIF/WLD.                                                                                                                        |
| 2026-03-19 | Problem 2 RCA     | **RESOLVED** | n/a             | n/a                           | Day-boundary gaps are progressive backfill gaps — missing days, NOT orphan flush bug. SUIUSDT@250 missing Mar 17-18 entirely. Self-resolves as kintsugi fills. Spike E/G not needed. Spike F (heartbeat) is the fix. |

---

## Problem 2: Day-Boundary Gaps on Non-Streaming Symbols

### Evidence (2026-03-19 full inventory)

21 day-boundary gaps across 6 non-streaming symbols:

| Symbol   | Thresholds Affected | Gap Sizes (trades) | Dates            |
| -------- | ------------------- | ------------------ | ---------------- |
| SUIUSDT  | 250, 500, 750, 1000 | 789-8650           | Mar 17→18, 18→19 |
| WLDUSDT  | 250, 500, 750       | 321-2632           | Mar 17→18, 18→19 |
| SOLUSDT  | 500, 750, 1000      | 6306-30416         | Mar 18→19        |
| NEARUSDT | 1000                | 2264-11049         | Mar 17→18, 18→19 |
| WIFUSDT  | 1000                | 30-188             | Mar 17→18, 18→19 |

Plus 3 residual intra-day anomalies on BTCUSDT@250 from prior restart sessions.

### Root Cause (Confirmed 2026-03-19)

**NOT an orphan flush bug.** These are **progressive backfill gaps** from kintsugi's
recency-first ordering. Example: SUIUSDT@250 has Mar 16 and Mar 19 data but Mar 17-18
are completely missing (kintsugi hasn't reached those days yet). The Stathera chain
shows a gap between Mar 16's last bar and Mar 19's first bar.

Verified empirically:

```
SUIUSDT@250: 415/417 days filled. Mar 17 and Mar 18 completely absent.
```

These gaps **self-resolve** as kintsugi progressively fills the missing days. They are
NOT anomalies requiring intervention — they're expected behavior during catch-up.

**Spike E and G: NOT NEEDED.** The correct fix is Spike F: improve heartbeat to not
alarm on progressive backfill gaps.

### Proposed Spikes

**Spike E: Carry processor state across day boundaries in kintsugi**

Instead of flushing an orphan at midnight and starting fresh, save the processor
checkpoint at midnight and restore it for the next day's repair. The forming bar
crosses the boundary seamlessly.

**Spike F: Heartbeat excludes day-boundary gaps from health reporting**

If day-boundary gaps are structurally unavoidable (the orphan flush is correct),
the heartbeat should classify them differently from intra-day anomalies. The
current heartbeat already has day-boundary detection but still reports them as
"bar-boundary gaps" with warning severity.

**Spike G: Kintsugi repairs adjacent days atomically**

When repairing day N, also check if day N+1's first bar leaves a gap with day N's
last bar. If so, extend the repair to cover the junction. Requires fetching a few
extra trades past midnight.

---

## Winning Approach

_Promoted here when a spike passes all POC criteria._

**Spike A: Env var streaming symbol filter** — empirically validated 2026-03-19.

- 39 overlaps → 0 overlaps for BTC/ETH
- 106 `+backfill` bars → 0 for streaming symbols
- Non-streaming symbols unaffected (34 pairs still filled)
- Implementation: ~5 lines in `_trailing_edge_fill()`, reads `OPENDEVIATIONBAR_STREAMING_SYMBOLS` env var
- Deployed live on bigblack via hotpatch, running stable

**Remaining problem**: Day-boundary structural gaps on NON-streaming symbols (NEARUSDT, SOLUSDT). These are gaps between yesterday's last bar and today's first bar, caused by kintsugi's orphan bar flush at Ouroboros midnight reset. See Spike E below.

---

## Related

| Resource                                                                                                                                    | What                                                                 |
| ------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| [sidecar-restart-zero-overlap-zero-gap-recovery-playbook](/.claude/skills/sidecar-restart-zero-overlap-zero-gap-recovery-playbook/SKILL.md) | Restart recovery patterns (different problem — this is steady-state) |
| [/CLAUDE.md § Writer Ownership](/CLAUDE.md)                                                                                                 | Sidecar owns today, kintsugi owns yesterday+                         |
| [/scripts/CLAUDE.md](/scripts/CLAUDE.md)                                                                                                    | Kintsugi daemon, today-guard details                                 |
| `kintsugi.py:2021`                                                                                                                          | `_trailing_edge_fill()` — the function to gate                       |
| `backfill.py:1026`                                                                                                                          | `backfill_all_gaps()` — iterates all pairs                           |
| `deploy/sidecar.env:9`                                                                                                                      | `OPENDEVIATIONBAR_STREAMING_SYMBOLS` env var                         |
