---
phase: quick
plan: 260321-n48
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar.py
  - crates/opendeviationbar-streaming/src/live_engine.rs
  - scripts/heartbeat/formatter.py
autonomous: true
requirements: [N48-01, N48-02, N48-03, N48-04]

must_haves:
  truths:
    - "_delete_today_bars DELETE completes synchronously before any INSERT can run"
    - "fill_from_rest ring buffer overflows are logged with symbol/threshold context and counted in dropped_bars"
    - "_sync_flush_rest_bars logs raw drain count and filter breakdown before batching"
    - "Heartbeat shows restart diagnostic hint when sidecar uptime is under 30 minutes"
  artifacts:
    - path: "python/opendeviationbar/sidecar.py"
      provides: "Synchronous DELETE via mutations_sync=1, diagnostic logging in _sync_flush_rest_bars"
      contains: "mutations_sync"
    - path: "crates/opendeviationbar-streaming/src/live_engine.rs"
      provides: "Overflow logging and metrics in fill_from_rest push sites"
      contains: "ring buffer full"
    - path: "scripts/heartbeat/formatter.py"
      provides: "Restart diagnostic reference when sidecar uptime < 30min"
      contains: "restart"
  key_links:
    - from: "python/opendeviationbar/sidecar.py:_delete_today_bars"
      to: "ClickHouse lightweight DELETE"
      via: "SETTINGS mutations_sync=1"
      pattern: "mutations_sync.*=.*1"
    - from: "crates/opendeviationbar-streaming/src/live_engine.rs:fill_from_rest"
      to: "LiveEngineMetrics.dropped_bars"
      via: "fetch_add on push failure"
      pattern: "dropped_bars\\.fetch_add"
---

<objective>
Fix async DELETE race in _delete_today_bars that causes fill_from_rest bars to vanish, and add diagnostics for restart debugging.

Purpose: The lightweight DELETE in \_delete_today_bars is async by default (mutations_sync=0). It returns immediately but the background mutation can delete subsequently-INSERTed bars that match the same predicate (toDate(open_time_ms/1000) = today()). This creates Stathera gaps from midnight until WS streaming catches up. Secondary: fill_from_rest silently discards ring buffer overflows without logging or metrics.

Output: Patched sidecar.py, live_engine.rs, formatter.py
</objective>

<execution_context>
@~/.claude/get-shit-done/workflows/execute-plan.md
@~/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@python/opendeviationbar/sidecar.py
@crates/opendeviationbar-streaming/src/live_engine.rs
@scripts/heartbeat/formatter.py
</context>

<tasks>

<task type="auto">
  <name>Task 1: Make _delete_today_bars DELETE synchronous</name>
  <files>python/opendeviationbar/sidecar.py</files>
  <action>
In `_delete_today_bars()` (around line 734), add `mutations_sync = '1'` to the SETTINGS clause of the DELETE command. The current SETTINGS line is:

```
"SETTINGS lightweight_mutation_projection_mode = 'drop'"
```

Change it to:

```
"SETTINGS lightweight_mutation_projection_mode = 'drop', mutations_sync = 1"
```

This makes the DELETE block until the mutation is fully applied, preventing the background mutation from racing with subsequent INSERTs in `_sync_flush_rest_bars`.

Also update the docstring of `_delete_today_bars` to replace "Uses lightweight DELETE (milliseconds, non-blocking)" with "Uses lightweight DELETE with mutations_sync=1 (blocks until complete). Without sync, the async mutation races with subsequent INSERTs — bars matching today() get deleted after being written (#N48)."
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "mutations_sync" python/opendeviationbar/sidecar.py</automated>
</verify>
<done>DELETE in \_delete_today_bars blocks until mutation completes; docstring explains the race condition</done>
</task>

<task type="auto">
  <name>Task 2: Add diagnostic logging to _sync_flush_rest_bars</name>
  <files>python/opendeviationbar/sidecar.py</files>
  <action>
In `_sync_flush_rest_bars()`, add diagnostic logging after the drain and before the batching loop. After line ~776 where `all_bars` is obtained (either from `drain_bars()` or the fallback loop), add:

```python
raw_count = len(all_bars)
```

Then after the for loop that populates `batches` and `drained` (around line 807 after the continue statements), and before the `if drained == 0:` check, add a log line:

```python
filtered_invalid_pairs = raw_count - drained - cross_midnight_count
logger.info(
    "sync_flush: drain diagnostics: raw=%d, valid=%d, cross_midnight=%d, invalid_pair=%d",
    raw_count, drained, cross_midnight_count, filtered_invalid_pairs,
)
```

To track cross_midnight_count, add `cross_midnight_count = 0` before the for loop and increment it in the cross-midnight filter branch (the `if open_ms > 0 and close_ms > 0 and (open_ms // day_ms) != (close_ms // day_ms):` block) before the `continue`.

This gives visibility into how many bars are drained vs filtered, critical for diagnosing bar vanishing issues.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "drain diagnostics\|cross_midnight_count" python/opendeviationbar/sidecar.py</automated>
</verify>
<done>\_sync_flush_rest_bars logs raw drain count, valid count, cross-midnight filtered count, and invalid-pair filtered count</done>
</task>

<task type="auto">
  <name>Task 3: Add overflow logging to fill_from_rest ring buffer pushes</name>
  <files>crates/opendeviationbar-streaming/src/live_engine.rs</files>
  <action>
In `fill_from_rest()` (the method starting around line 530), there are two `let _ = bar_buffer.push(completed);` calls at lines 583 and 597. These silently discard the push result. The WS processing path (lines 1087-1091, 1123-1128) already does this correctly — use the same pattern.

Replace line 583:

```rust
let _ = bar_buffer.push(completed);
```

with:

```rust
if !bar_buffer.push(completed) {
    metrics.dropped_bars.fetch_add(1, Ordering::Relaxed);
    tracing::warn!(%symbol, threshold = *threshold, "fill_from_rest: ring buffer full, bar dropped (midnight orphan)");
}
```

Replace line 597:

```rust
let _ = bar_buffer.push(completed);
```

with:

```rust
if !bar_buffer.push(completed) {
    metrics.dropped_bars.fetch_add(1, Ordering::Relaxed);
    tracing::warn!(%symbol, threshold = *threshold, "fill_from_rest: ring buffer full, bar dropped");
}
```

This ensures fill_from_rest overflow is visible in logs and counted in the dropped_bars metric, matching the existing WS path behavior.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && cargo check -p opendeviationbar-streaming 2>&1 | tail -5</automated>
</verify>
<done>Both fill_from_rest push sites log warnings and increment dropped_bars on overflow, matching WS path pattern</done>
</task>

<task type="auto">
  <name>Task 4: Add restart diagnostic hint in heartbeat formatter</name>
  <files>scripts/heartbeat/formatter.py</files>
  <action>
In the Services section of `format_message()` (around line 67-76), after the sidecar line is appended, add a restart diagnostic hint when sidecar uptime is under 30 minutes.

After line 76 (`lines.append(f"  {_icon(svc)} {svc}{suffix}")`), inside the `for svc in ("sidecar", "kintsugi"):` loop, add a check specifically for sidecar:

```python
        # Restart diagnostic: hint to check fill_from_rest logs when sidecar recently restarted
        if svc == "sidecar" and uptime:
            # Parse uptime like "5m", "29m", "1h 5m" — extract minutes
            import re
            m = re.match(r"^(\d+)m$", uptime)
            if m and int(m.group(1)) < 30:
                lines.append(
                    "    \u21b3 recently restarted \u2014 check fill_from_rest drain: "
                    "journalctl --user -u opendeviationbar-sidecar --since '30 min ago' | grep sync_flush"
                )
```

Move the `import re` to the top of the file (next to existing imports) to avoid repeated imports. This provides actionable guidance when investigating post-restart Stathera gaps.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "from scripts.heartbeat.formatter import format_message; print('import ok')" 2>&1 || python -c "import sys; sys.path.insert(0, 'scripts'); from heartbeat.formatter import format_message; print('import ok')"</automated>
</verify>
<done>Heartbeat shows restart diagnostic hint with journalctl grep command when sidecar uptime is under 30 minutes</done>
</task>

</tasks>

<verification>
1. `grep -n "mutations_sync" python/opendeviationbar/sidecar.py` shows the sync setting
2. `grep -n "drain diagnostics" python/opendeviationbar/sidecar.py` shows diagnostic logging
3. `cargo check -p opendeviationbar-streaming` compiles without errors
4. `grep -n "fill_from_rest.*ring buffer" crates/opendeviationbar-streaming/src/live_engine.rs` shows overflow logging
5. `grep -n "recently restarted" scripts/heartbeat/formatter.py` shows restart hint
</verification>

<success_criteria>

- \_delete_today_bars uses mutations_sync=1 to block until DELETE completes
- \_sync_flush_rest_bars logs raw/valid/filtered bar counts for drain diagnostics
- fill_from_rest overflow increments dropped_bars and logs warning (matching WS path)
- Heartbeat formatter shows restart diagnostic hint when sidecar uptime < 30min
- cargo check passes for opendeviationbar-streaming crate
  </success_criteria>

<output>
After completion, create `.planning/quick/260321-n48-fix-async-delete-race-in-delete-today-ba/260321-n48-SUMMARY.md`
</output>
