# Phase 2: Midnight Preflight Verification - Context

**Gathered:** 2026-03-21
**Status:** Ready for planning

<domain>
## Phase Boundary

Add midnight preflight verification to the sidecar startup flow. Before `fill_from_rest()`, query Binance REST API for each streaming symbol's midnight crossover trade ID, verify yesterday's orphan bar in ClickHouse closes at that boundary, and anchor `fill_from_rest` from the verified crossover ID.

</domain>

<decisions>
## Implementation Decisions

### Crossover Trade ID Query (PREFLIGHT-01)

- New Python function `_get_midnight_crossover_tid(symbol)` in `sidecar.py`
- Uses Binance REST API: `GET /api/v3/aggTrades?symbol=X&startTime=midnight_ms&limit=1` to get the first trade at or after UTC midnight
- `midnight_ms` = start of today UTC in milliseconds
- Returns `Optional[int]` — the `agg_trade_id` of the first trade of today
- Uses existing `fetch_aggtrades_rest()` from Rust bindings (already supports `startTime`)
- Called for each streaming symbol in `_create_engine()`, between `_delete_today_bars()` and `fill_from_rest()`

### Yesterday Orphan Verification (PREFLIGHT-02)

- Query ClickHouse for yesterday's last bar per symbol: `SELECT last_agg_trade_id FROM ... WHERE toDate(open_time_ms/1000) = yesterday() AND is_orphan = 1 ORDER BY last_agg_trade_id DESC LIMIT 1`
- If `orphan.last_agg_trade_id == crossover_tid - 1`: perfect continuity, log INFO
- If mismatch: log WARNING with details (expected vs actual), defer to kintsugi — no crash, no data corruption
- If no orphan bar exists (fresh deploy, gap): log WARNING, fall back to current behavior

### fill_from_rest Anchoring (PREFLIGHT-03)

- Pass verified crossover trade IDs to `engine.fill_from_rest()` as override seeds
- In `_create_engine()`, after preflight: call `engine.seed_trade_id(symbol, threshold, crossover_tid - 1)` to override the CH seed
- This makes `fill_from_rest` start from `crossover_tid` (not the CH max which may include bars from multiple thresholds)
- Falls back to current CH-seeded behavior if preflight returns None

### Claude's Discretion

- Error handling details (network failures, API rate limits during preflight)
- Log message formatting and structured logging fields
- Whether to run preflight queries in parallel or sequentially
- Test design and mocking strategy
- Exact placement of the new function within sidecar.py

</decisions>

<code_context>

## Existing Code Insights

### Reusable Assets

- `sidecar.py:447-503` — `_seed_trade_ids_from_clickhouse()` pattern for per-symbol seeding
- `sidecar.py:615-645` — `_delete_today_bars()` placement in startup flow
- `sidecar.py:809-904` — `_create_engine()` startup sequence
- `clickhouse/query_operations.py:348-402` — `get_latest_bar_with_trade_id()` query pattern
- `binance_bindings.rs:707-829` — `fetch_aggtrades_rest()` with `startTime` support

### Established Patterns

- `engine.seed_trade_id(symbol, threshold, tid)` — existing per-(symbol,threshold) seed mechanism
- `_skip_lock=False` — sidecar doesn't hold external locks
- stamina retry for network calls
- `valid_pairs` filtering for streaming symbols only

### Integration Points

- `_create_engine()` line ~846 — insertion point between `_delete_today_bars()` and `fill_from_rest()`
- `engine.seed_trade_id()` — existing Rust→Python seed injection API
- `fill_from_rest()` line 466-475 — uses `max(checkpoint_tid, ch_seed_tid)` for start point

</code_context>

<specifics>
## Specific Ideas

### User's Key Insight (from PROJECT.md)

"We can always use REST API to pinpoint the crossover time of day to get a precise read of the exact crossover aggregator trade ID." The orphan bar should always close at the very last aggregated trade ID of the day.

### Production Evidence

BTCUSDT/ETHUSDT (streamed continuously) have ZERO midnight gaps. Non-streamed symbols have gaps from the processor seam. The midnight crossover verification validates this for ALL streaming symbols.

</specifics>

<deferred>
## Deferred Ideas

None — discussion stayed within phase scope.

</deferred>
