# Quick Task: Fix INSERT Dedup Collision + Validation Spike

## Root Cause

After sidecar restart, `clean-slate DELETE` removes today's bars, then `fill_from_rest`
recomputes identical bars from the same trades. ClickHouse's `non_replicated_deduplication_window=1000`
retains dedup tokens from the original INSERT even after DELETE — the re-INSERT of identical
data is silently dropped.

**Key finding from ClickHouse research:** The dedup window is independent of data lifecycle.
DELETE does not clear dedup entries. This is a well-known ClickHouse footgun.

## Fix (3 changes)

1. **`_build_insert_settings()`** — When `skip_dedup=True`, explicitly return
   `{"insert_deduplicate": 0}` instead of `{}`. The empty dict left ClickHouse's
   server-default `insert_deduplicate=1` active, which falls back to automatic
   block-content-hash dedup — identical bar data produces identical hash.

2. **`sync_flush`** (midnight.py:587) — Add `skip_dedup=True` to `store_bars_batch()`.
   Clean-slate DELETE guarantees no duplicates; dedup is harmful here.

3. **`parquet-process`** (midnight.py:871) — Same fix, same reasoning.

## Telemetry Spike

Post-sync_flush read-back verification: after writing bars, query CH to verify the
expected count matches. Logs `DEDUP COLLISION DETECTED` at ERROR level if bars are
missing. Catches any future regression in the dedup bypass logic.

## Risk Assessment: skip_dedup=True + insert_deduplicate=0

**Does this introduce duplicate bar risk on retry?**

No. The sync_flush and parquet-process paths run AFTER clean-slate DELETE, which
removes ALL today's bars. If sync_flush fails, the fallback `_write_batch_with_retry`
uses the default path (with dedup enabled) — retry safety is preserved. If the
sidecar restarts again, another clean-slate DELETE runs first, removing any
partial writes.
