# Heartbeat Blind Spot Analysis

**Date**: 2026-03-21
**Context**: 495 overlaps accumulated for ~1 hour before manual detection. The heartbeat reported them but did not prevent the cascade.

## TGI-1: Gate Key Map (CAN detect vs CANNOT detect)

### Full Gate Key Inventory (15 keys across 3 tiers)

| Gate Key                  | Tier     | CAN Detect                                         | CANNOT Detect                                                                         |
| ------------------------- | -------- | -------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `clickhouse`              | CRITICAL | CH process down                                    | CH returning stale data due to unmerged parts                                         |
| `sidecar`                 | CRITICAL | systemd unit inactive                              | Sidecar running but silently broken (WS connected, no trades arriving)                |
| `kintsugi`                | CRITICAL | systemd unit inactive                              | Kintsugi running but stuck in a non-progressing loop                                  |
| `disk`                    | CRITICAL | Disk <10 GB free                                   | Disk I/O latency (NVMe degradation)                                                   |
| `stathera_overlaps`       | CRITICAL | Overlap count increasing when kintsugi not healing | **Rate of overlap growth** (495 in 1h = ~8/min is actionable but treated same as 1)   |
| `stathera_gaps`           | CRITICAL | Gap count increasing when kintsugi not healing     | Gap within the 5-min polling window (gap appears and is healed before next heartbeat) |
| `circuit_breaker_healthy` | CRITICAL | Circuit breaker in OPEN state                      | Circuit breaker flapping (CLOSED->OPEN->HALF_OPEN->CLOSED within one heartbeat cycle) |
| `kintsugi_stuck`          | WARNING  | Same shard being repaired for >N min               | Kintsugi repeatedly healing the same shards (Sisyphean cycle)                         |
| `kintsugi_failures`       | WARNING  | Failed repairs in last 1h                          | Repairs that "succeed" but produce wrong data                                         |
| `kintsugi_memory`         | WARNING  | Memory >95% of MemoryHigh                          | Gradual memory leak below 95%                                                         |
| `cross_midnight`          | WARNING  | Cross-midnight bars in DB                          | Cross-midnight bars being produced faster than kintsugi repairs them                  |
| `version_drift`           | WARNING  | Installed != GitHub release                        | Hotfix .py files that break runtime but pass mtime check                              |
| `seeder_ok`               | WARNING  | Seeder timer stale, failed count >0                | Seeder running but producing corrupt Parquet files                                    |
| `kintsugi_telemetry`      | DEGRADED | NDJSON file stale >2h                              | Telemetry file being written but containing misleading data                           |
| `yesterday_completion`    | DEGRADED | Yesterday's orphan bar TID mismatch                | Only checks BTCUSDT and ETHUSDT (priority symbols), not all 18                        |

---

## TGI-2: Five Specific Blind Spots

### Blind Spot 1: Sync-flush bars vanishing after write (async DELETE race)

**Can the heartbeat detect it?** NO.

The `_delete_today_bars()` function uses `mutations_sync=1` which blocks until the DELETE completes. This was the fix for #N48. However, `store_bars_replacing()` in `maintenance.py` also runs targeted single-row DELETEs for straddle bars -- and those use `mutations_sync=1` too.

**The actual blind spot**: If `mutations_sync=1` fails silently (ClickHouse returns success but the mutation is queued), the subsequent INSERT races with the pending DELETE. The heartbeat checks `pending_mutations` count but **does not correlate it with recent INSERTs**. It reports `stathera_pending_mutations` as a string but this is NOT a health gate key -- mutations can be non-zero and the system reports HEALTHY.

**Detection gap**: Between the INSERT and the async DELETE completing, bars exist in CH. The next heartbeat (5 min later) sees them via Stathera. But if the DELETE then fires and removes them, the heartbeat after THAT would see them as gaps. This is a minimum 10-minute detection delay for a write-then-delete race, and it manifests as a "gap that appears from nowhere" which the heartbeat attributes to kintsugi needing to heal -- not as a bug in the mutation pipeline.

**Recommendation**: Add a `mutations_healthy` key to CRITICAL_KEYS (currently exists as a result but is NOT gated). Any pending mutation on `open_deviation_bars` while the sidecar is writing = CRITICAL.

### Blind Spot 2: fill_from_rest producing bars for wrong dates

**Can the heartbeat detect it?** PARTIALLY -- only after the damage is done.

The `cross_midnight` oracle detects bars where `toDate(open_time_ms) != toDate(close_time_ms)`. But `fill_from_rest` could produce bars for the CORRECT UTC date that are simply WRONG (e.g., bars for today built from stale REST data, or bars anchored from an inflated seed TID that skip a range of trades).

The L1 filter in `_sync_flush_rest_bars()` catches cross-midnight bars, but there is no check that:

- The bars' trade-ID range is contiguous with yesterday's last bar
- The bars' timestamps fall within today's expected range
- The `fill_from_rest()` return count is plausible (e.g., >0 for a symbol that has been trading)

**Detection gap**: fill_from_rest returns 0 bars (because seed TID is too far in the future), \_sync_flush_rest_bars logs "ring buffer empty", and the heartbeat sees this as normal. The gap only becomes visible when Stathera audit runs 5 min later and detects the missing range.

**Recommendation**: Add a `fill_from_rest_bars_count` junction telemetry field. If fill_from_rest returns 0 bars for any symbol during startup, flag it as WARNING.

### Blind Spot 3: Mutation cascade detection BEFORE it becomes critical

**Can the heartbeat detect it?** NO -- it detects the RESULT (overlaps/gaps) not the CAUSE (cascade).

The 495-overlap incident is the proof. The heartbeat's Stathera check uses delta tracking:

```python
anomalies_growing = overlap_delta > 0 or gap_delta > 0
```

But the health logic has a crucial escape hatch (stathera.py:99):

```python
elif anomalies_growing and not kintsugi_healing:
    results["stathera_overlaps"] = overlap_count == 0
    results["stathera_gaps"] = gap_count == 0
else:
    results["stathera_overlaps"] = overlap_count == 0 or kintsugi_healing
    results["stathera_gaps"] = gap_count == 0 or kintsugi_healing
```

**If kintsugi is running and not stuck, growing overlaps are considered OK.** This means:

- Overlaps grow from 0 to 50: CRITICAL (delta >0, kintsugi not healing yet -- kintsugi might not have started its cycle)
- Overlaps grow from 50 to 495: HEALTHY (kintsugi is "healing" = service is active and not stuck)

The heartbeat trusts that kintsugi will fix the problem. But kintsugi's today-guard prevents it from touching today's data. If the overlaps are in today's bars (sidecar's domain), kintsugi literally cannot help, yet the heartbeat reports it as "healing."

**The cascade blind spot**: There is no check for the RATE of anomaly growth. 495 overlaps in 1 hour = ~8/min. This rate of growth means something is actively broken and producing overlaps faster than any repair can fix. The heartbeat treats "1 new overlap" and "495 new overlaps" identically.

**Recommendation**: Add overlap/gap velocity thresholds. If `abs(overlap_delta) > 10` in a single 5-min cycle, escalate to CRITICAL regardless of kintsugi state. Also: if `stathera_overlaps` or `stathera_gaps` are in today's date range AND kintsugi_healing=True, that is a lie -- kintsugi CANNOT heal today (#280).

### Blind Spot 4: Sidecar WS stream dropping trades

**Can the heartbeat detect it?** NO.

The heartbeat does NOT query the sidecar's `/health` or `/status` endpoints for `dropped_bars`, `backpressure_events`, or any ring buffer metrics. The junction telemetry check (`junction.py`) reads `rest_last_tid`, `ws_first_tid`, `forming_bar`, and `committed_floors` -- but these are informational only (`junction_telemetry` is NOT in any gate key set).

Specifically missing from heartbeat monitoring:

- `dropped_bars` counter from LiveBarEngine metrics (Lethe)
- `backpressure_events` counter
- `max_queue_depth` (ring buffer high-water mark)
- Per-symbol trade-ID freshness (Argus timeout detection)
- WS reconnection count (Antaeus)

The sidecar has its own Lethe alerting that sends Telegram messages on ring buffer overflow. But:

1. Lethe fires in the sidecar process, which might be the one crashing
2. If the sidecar process dies, Lethe dies with it -- no independent observer
3. The heartbeat (independent process) does not check ANY of these metrics

**Detection gap**: WS silently drops trades (e.g., due to network buffer overflow at the OS level, not ring buffer level). No Lethe alert fires because the ring buffer isn't full -- trades simply never arrive. The heartbeat sees the sidecar systemd unit as "active" and reports HEALTHY. The Stathera audit eventually catches gaps, but only after bars are written with missing trade ranges -- minimum 5-10 min delay.

**Recommendation**: Add a `sidecar_metrics` CRITICAL check that queries `localhost:8081/health`, reads `dropped_bars` and `backpressure_events`, and flags any non-zero value as CRITICAL. Also add per-symbol trade freshness (if any symbol has no trades for >10 min during market hours, flag WARNING).

### Blind Spot 5: CH seed TID inflation

**Can the heartbeat detect it?** NO.

The `_seed_trade_ids_from_clickhouse()` queries `max(last_agg_trade_id)` from ClickHouse for each (symbol, threshold). The `seed_trade_id()` Rust function keeps the MAX value -- it never decreases. If CH is seeded BEFORE `_delete_today_bars()`, the seed captures today's high-water mark. Then `_delete_today_bars()` removes those bars. Then `_run_midnight_preflight()` tries to override with `crossover_tid - 1`, but `seed_trade_id(MAX)` ignores it because the stale seed is higher.

The code currently enforces the ordering (delete THEN seed, per #294). But the heartbeat has NO check to verify:

1. That the seed TID for each pair is plausible (e.g., close to the crossover TID)
2. That fill_from_rest actually started from the expected TID
3. That the seed was not inflated by a concurrent writer

The junction telemetry shows `committed_floors` from the sidecar, but:

- It is NOT a health gate key
- It does not compare floors against expected values
- It does not detect inflation (floor too high = trades silently skipped)

**Detection gap**: Seed inflation causes `fill_from_rest` to start from a future TID, producing 0 bars. This leaves a gap between yesterday's last bar and today's first WS bar. The heartbeat eventually detects this via Stathera, but attributes it to "normal startup gap" rather than a seed inflation bug.

**Recommendation**: Add seed-vs-crossover validation to the junction telemetry check. After startup, compare `committed_floors[symbol]` against the crossover TID. If `floor > crossover_tid`, flag CRITICAL (seed inflation detected).

---

## TGI-3: Root Cause of the 495-Overlap Incident

The heartbeat REPORTED the overlaps (they showed up in `stathera_overlap_count`) but did NOT escalate to CRITICAL because:

1. **Kintsugi was running** (`kintsugi=True`, `kintsugi_stuck` not False)
2. Therefore `kintsugi_healing=True`
3. Therefore the Stathera health logic took the `else` branch: `stathera_overlaps = overlap_count == 0 or kintsugi_healing` = `False or True` = `True` (HEALTHY)
4. The overlaps were in today's bars (sidecar domain), but kintsugi's today-guard (#280) prevents repair
5. **The heartbeat assumed kintsugi would fix it. Kintsugi was forbidden from fixing it.**

This is the fundamental design flaw: the heartbeat conflates "kintsugi is running" with "kintsugi can fix the current problem." For today's data, only the sidecar (Puck) or manual intervention can fix it. The heartbeat has no concept of writer ownership boundaries.

---

## Summary: Unmonitored Failure Modes

| Failure Mode                                          | Heartbeat Detection            | Delay                         | Severity |
| ----------------------------------------------------- | ------------------------------ | ----------------------------- | -------- |
| Async DELETE race (bars vanish)                       | Indirect via Stathera          | 10+ min                       | HIGH     |
| fill_from_rest wrong date/empty                       | Indirect via Stathera          | 5-10 min                      | HIGH     |
| Mutation cascade (overlaps growing in today)          | SUPPRESSED by kintsugi_healing | NEVER (until manual)          | CRITICAL |
| WS trade drops (network level)                        | NONE                           | 5-10 min via Stathera         | HIGH     |
| Ring buffer overflow (dropped_bars)                   | NONE                           | NEVER (only Lethe in-process) | HIGH     |
| CH seed TID inflation                                 | NONE                           | 5-10 min via Stathera         | HIGH     |
| Kintsugi Sisyphean cycle (heal same shard repeatedly) | Partial (stuck detection)      | N/A                           | MEDIUM   |
| Per-symbol trade silence (Argus)                      | NONE in heartbeat              | NEVER                         | MEDIUM   |
| Sidecar WS reconnection storm                         | NONE                           | NEVER                         | MEDIUM   |

## Recommendations (Priority Order)

1. **FIX NOW**: Split Stathera health logic by writer ownership. If overlaps/gaps are in today's bars, `kintsugi_healing` MUST NOT suppress the alert. Only Puck (sidecar) can heal today.

2. **FIX NOW**: Add overlap/gap velocity threshold. `abs(delta) > 10` per 5-min cycle = CRITICAL regardless of kintsugi state.

3. **ADD**: `sidecar_metrics` CRITICAL check -- query `/health` for `dropped_bars > 0`, `backpressure_events > 0`.

4. **ADD**: `mutations_healthy` to CRITICAL_KEYS (key already exists, just not gated).

5. **ADD**: Post-startup seed validation in junction telemetry (compare committed_floors vs crossover TID).

6. **ADD**: fill_from_rest bar count plausibility check (0 bars for an active symbol = WARNING).

7. **ADD**: Per-symbol trade freshness from sidecar `/status` endpoint (silence >10 min = WARNING).
