# py-syncflush-findings: Data Loss Paths in `_sync_flush_rest_bars()`

**Date**: 2026-03-21
**Scope**: `python/opendeviationbar/sidecar.py` lines 749-863
**Method**: Trace every bar from `drain_bars()` through to `store_bars_replacing()` / fallback

---

## Function Flow Summary

```
drain_bars()
  → loop: pop _symbol/_threshold from each bar dict
    → filter 1: valid_pairs gate (silent drop)
    → filter 2: L1 cross-midnight gate (logged drop)
  → batch by (symbol, threshold)
  → for each batch:
      → store_bars_replacing() [primary]
        → _do_replace(): sync DELETE + INSERT
      → _write_batch_with_retry() [fallback on exception]
        → _write_bars_batch_to_clickhouse() → store_bars_batch()
          → _run_write_guards() → store_bars_batch INSERT
        → dead-letter on exhaustion
```

---

## Finding 1: `valid_pairs` Silent Drop (KNOWN)

**Location**: sidecar.py:796
**Severity**: Medium (already identified in prior investigation)

Bars whose `(symbol, threshold)` is not in `valid_pairs` are silently discarded. The diagnostic log on line 814-816 counts these as `filtered_invalid_pairs`, but the bars are permanently lost — they are never dead-lettered, never retried.

**Impact**: Bars for thresholds below `min_threshold` in the symbol registry are silently dropped. If `valid_pairs` is misconfigured, all bars for a pair vanish without Telegram alert.

---

## Finding 2: DELETE-then-INSERT Non-Atomicity in `_do_replace()`

**Location**: maintenance.py:395-414, sidecar.py:848-857
**Severity**: HIGH

`_do_replace()` performs:

1. Synchronous DELETE (`mutations_sync=1`) — removes all overlapping bars
2. INSERT via `store_bars_batch()`

If step 2 raises an exception (e.g., `_run_write_guards` ValueError from timestamp scale check or negative volume), the bars are **deleted from CH but never re-inserted**. The exception propagates to `_sync_flush_rest_bars` line 848 `except Exception`, which falls back to `_write_batch_with_retry`.

However, `_write_batch_with_retry` calls `_write_bars_batch_to_clickhouse` which calls `store_bars_batch` with `overlap_strategy="off"`. If the same guard that rejected the INSERT in `_do_replace` rejects it again in the fallback, the bars end up in dead-letter — but the original bars were already DELETED.

**Data loss window**: Between sync DELETE completing and INSERT failing, CH has no bars for that trade-ID range. If the fallback also fails, old bars are gone and new bars are in dead-letter.

**Mitigating factor**: `_run_write_guards` raises for genuinely corrupt data (timestamps in wrong scale, negative volumes), which shouldn't occur for bars produced by Rust `fill_from_rest`. But any transient ClickHouse error during INSERT (network, timeout) also triggers this path.

---

## Finding 3: Fallback `written` Count Overestimates

**Location**: sidecar.py:857
**Severity**: Low

When the primary `store_bars_replacing` fails and `_write_batch_with_retry` succeeds, the code adds `len(bars)` to `written` (line 857). But `store_bars_replacing` returns the actual rows inserted (from CH), while `len(bars)` counts input dicts. If `store_bars_batch` internally filters or drops bars (e.g., empty DataFrame guard at bulk_operations.py:748, or `_check_overlap_strategy` returning False for "skip"), the count diverges.

In practice, the fallback uses `overlap_strategy="off"` so the overlap check is skipped, and bars from Rust shouldn't be empty. But the inconsistency in counting semantics (CH-confirmed vs input count) is a code smell.

---

## Finding 4: Destructive `.pop()` on Shared Dict References

**Location**: sidecar.py:792-793
**Severity**: Low (no current exposure)

`bar.pop("_symbol", "UNKNOWN")` destructively removes metadata from the bar dict. The dicts come from `drain_bars()` which returns owned Python dicts (no shared references in current code). But if any future code retains a reference to `all_bars` after the loop, the dicts will be missing `_symbol`/`_threshold`.

Not a current bug, but a fragility point.

---

## Finding 5: Fallback Path Lacks `_skip_lock=False` Parity

**Location**: sidecar.py:856
**Severity**: Low-Medium

The primary path (`store_bars_replacing`) acquires a Redis write lock. The fallback path (`_write_batch_with_retry` -> `_write_bars_batch_to_clickhouse` -> `store_bars_batch`) uses `overlap_strategy="off"` which skips the overlap check, and Redis lock is acquired by `store_bars_batch` (default `_skip_lock=False`). So the fallback DOES acquire the lock — this is correct. No issue here.

---

## Finding 6: `fill_from_rest` Failure = 0 Bars, No sync_flush

**Location**: sidecar.py:1028-1030, 1036
**Severity**: Info (not a loss path)

If `fill_from_rest()` raises, `rest_bars = 0`, and the `if rest_bars > 0` guard on line 1036 skips `_sync_flush_rest_bars` entirely. This is correct — no bars to flush. But it means the ring buffer might have partial data from a failed fill_from_rest that never gets drained. The `drain_bars()` call only happens inside `_sync_flush_rest_bars`, so bars stuck in the ring buffer after a failed fill_from_rest are abandoned.

**Impact**: These bars would be from an incomplete REST fill and are likely better discarded. WS + kintsugi will cover the gap. Not a silent loss — the failure is logged.

---

## Summary: Actionable Findings

| #   | Finding                                                | Severity | Action                                                                                                      |
| --- | ------------------------------------------------------ | -------- | ----------------------------------------------------------------------------------------------------------- |
| 1   | `valid_pairs` silent drop                              | Medium   | Already known. Consider dead-letter or Telegram alert.                                                      |
| 2   | DELETE succeeds, INSERT fails → bars gone              | **HIGH** | INSERT-first strategy would eliminate this. Or wrap DELETE+INSERT in a try that restores on INSERT failure. |
| 3   | Fallback `written` count uses `len(bars)` not CH count | Low      | Use `store_bars_batch` return value instead.                                                                |
| 4   | Destructive `.pop()`                                   | Low      | Cosmetic. No current exposure.                                                                              |
| 5   | Lock parity                                            | N/A      | Verified correct.                                                                                           |
| 6   | Ring buffer not drained on fill_from_rest failure      | Info     | Acceptable — incomplete bars should be discarded.                                                           |

---

## Recommendation for Finding 2 (HIGH)

The root fix is to reverse the order: **INSERT first, then DELETE stale**. This is the INSERT-only replace strategy already documented in the `store_bars_replacing` docstring ("INSERT-first strategy means crash at any point = no data loss"). However, `_do_replace()` does the opposite: DELETE-first with `mutations_sync=1`.

The `_do_replace()` comment on line 389-390 says `mutations_sync=1` was the fix for the async DELETE race in v13.39-v13.40 (#293). But it introduced a new risk: sync DELETE + failed INSERT = data loss.

**Proposed fix**: In `_do_replace()`, wrap the DELETE in a nested try. If INSERT fails, log a critical alert but do NOT retry the DELETE. The fallback path already handles re-insertion. Alternatively, switch to INSERT-first + post-INSERT targeted DELETE (the strategy described in the `store_bars_replacing` docstring but not implemented in `_do_replace`).
