# Phase 6: Fix SEED-001 Parquet-First Bar Processing - Context

**Gathered:** 2026-03-25
**Status:** Ready for planning
**Source:** Conversation audit of session 6692463b + production log verification

<domain>
## Phase Boundary

Replace the broken SEED-001 "skip Parquet range" logic in the sidecar startup sequence with "process Parquet ticks into bars, then REST-fill remaining gap."

**Root cause (verified from v13.54.0 restart logs at 08:52 UTC):**

1. Midnight preflight correctly sets seed to crossover TID (e.g., BTCUSDT@3915404668)
2. Clean-slate DELETE removes all today's bars (43.2s, 72 pairs)
3. Parquet-first **overwrites** seed with Parquet max_tid (e.g., BTCUSDT@3915679334, age=386s)
4. fill_from_rest starts from Parquet max_tid, only filling 2,766 trades
5. **Result**: 274,666 trades (00:00-08:47 UTC) never processed into bars

**Affected symbols in current production**: BTCUSDT (274K gap), ETHUSDT (208K gap), AAVEUSDT (14K gap), NEARUSDT (7K gap), FILUSDT (6K gap), plus 7 symbols with no today data at all (ADAUSDT, AVAXUSDT, BCHUSDT, BNBUSDT, DOGEUSDT, LINKUSDT, LTCUSDT — their Parquet-first advanced seeds past the point where any bars would form in the fill window).

**What this phase delivers**: On every sidecar restart, Parquet ticks are processed through the Rust processor into bars (written to ClickHouse), then fill_from_rest handles only the small remaining gap from Parquet-end to WS-start.

</domain>

<decisions>
## Implementation Decisions

### Processing approach

- Read today's Parquet ticks via `TickStorage.read_ticks()` (returns Polars LazyFrame)
- Convert to Arrow and process through `OpenDeviationBarProcessor.process_trades_arrow_native()` (microsecond timestamps)
- Process each symbol × threshold combination independently
- Write resulting bars to ClickHouse via existing `cache_write()` bulk operations
- Seed engine with `last_completed_bar_tid` (NOT Parquet max_tid) so fill_from_rest picks up forming bar correctly

### Parquet naming

- TickStorage uses `BINANCE_SPOT_BTCUSDT` directory names, sidecar uses `BTCUSDT`
- Must map between the two naming conventions (prefix `BINANCE_SPOT_`)

### Forming bar handling

- Parquet processing produces completed bars + leaves a forming bar in progress
- We seed fill_from_rest with the last COMPLETED bar's `last_agg_trade_id`
- fill_from_rest creates fresh processors starting from that TID, naturally re-processing the forming bar's tail trades
- This means a few trades at the Parquet boundary are processed twice — acceptable, produces correct bars

### Staleness check

- Keep the 600s staleness threshold — if Parquet is stale, fall back to full REST fill from crossover TID (existing behavior, proven safe)
- When Parquet IS fresh: process ticks → bars instead of skipping

### Error handling

- Parquet processing failure is non-fatal: fall back to crossover TID seed (same as current "stale" path)
- Log clearly: how many bars produced from Parquet, how many trades processed, time taken

### Claude's Discretion

- Chunking strategy for large Parquet files (500K batch size matches existing `process_trades_polars`)
- Whether to parallelize across symbols or process sequentially
- Exact log format and telemetry fields

</decisions>

<canonical_refs>

## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Sidecar startup sequence

- `python/opendeviationbar/sidecar/__init__.py` — Lines 253-391: `_create_engine()` method with the full startup sequence
- `python/opendeviationbar/sidecar/midnight.py` — `_read_parquet_max_tids()` (current broken SEED-001), `_run_midnight_preflight()` (crossover TID seeding)

### Tick storage

- `python/opendeviationbar/storage/parquet.py` — `TickStorage` class: `read_ticks()`, directory structure (`BINANCE_SPOT_` prefix)
- `python/opendeviationbar/storage/CLAUDE.md` — Tier-1 Parquet ticks cache documentation

### Rust processor API

- `python/opendeviationbar/processors/core.py` — `OpenDeviationBarProcessor.process_trades_arrow_native()` (µs Arrow input)
- `python/opendeviationbar/processors/api.py` — `process_trades_polars()` (reference for chunking pattern)
- `src/core_bindings.rs` — PyO3 bindings for Arrow processing

### ClickHouse operations

- `python/opendeviationbar/clickhouse/bulk_operations.py` — `cache_write()` for bar insertion

### Architecture

- `CLAUDE.md` — Sidecar Startup Sequence Invariant (#294, #295), SEED-001 context
- `python/opendeviationbar/sidecar/CLAUDE.md` — Sidecar architecture details

</canonical_refs>

<specifics>
## Specific Ideas

- The v13.54.0 restart at 08:52 UTC is the reference case: 12/16 Binance symbols had Parquet-first advance their seeds, 4 were stale (SOLUSDT, SUIUSDT, UNIUSDT, XRPUSDT skipped — these are the ones with small/no gaps)
- Processing 274K BTCUSDT trades through Rust takes ~27ms (100ms/1M benchmark) — far faster than 8.5s REST fetch
- The existing `_read_parquet_max_tids` function already handles the freshness check and Parquet reading — modify it to return ticks data instead of just max_tid
- `process_trades_arrow_native()` expects columns: `timestamp` (int64 µs), `price` (float64), `quantity` (float64)

</specifics>

<deferred>
## Deferred Ideas

- Feeding Parquet ticks directly into LiveBarEngine's Rust-side `fill_from_rest()` (would require Rust API changes)
- Using checkpoints to carry forming bar state from Parquet processing to fill_from_rest (complex, unnecessary since re-processing a few trades is cheap)

</deferred>

---

_Phase: 06-fix-seed-001-process-parquet-ticks_
_Context gathered: 2026-03-25 via conversation audit + production log verification_
