---
phase: 06-fix-seed-001-process-parquet-ticks
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/midnight.py
  - python/opendeviationbar/sidecar/__init__.py
  - tests/test_parquet_bar_processing.py
autonomous: true
requirements: [SEED-FIX-01, SEED-FIX-02]

must_haves:
  truths:
    - "On sidecar restart with fresh Parquet, ticks are processed into bars via Rust processor and written to ClickHouse"
    - "Engine is seeded with last_completed_bar_tid (not Parquet max_tid) so fill_from_rest picks up the forming bar correctly"
    - "Stale or missing Parquet falls back to crossover TID seeds (no regression from current behavior)"
  artifacts:
    - path: "python/opendeviationbar/sidecar/midnight.py"
      provides: "_process_parquet_ticks() function"
      contains: "def _process_parquet_ticks"
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "Updated parquet-first block calling _process_parquet_ticks"
      contains: "_process_parquet_ticks"
    - path: "tests/test_parquet_bar_processing.py"
      provides: "Tests for Parquet tick processing and seeding"
      contains: "test_seed_uses_completed_bar_tid"
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/midnight.py"
      via: "from .midnight import _process_parquet_ticks"
      pattern: "_process_parquet_ticks"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "TickStorage().read_ticks(symbol)"
      pattern: "read_ticks"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/processors/core.py"
      via: "OpenDeviationBarProcessor.process_trades_arrow_native()"
      pattern: "process_trades_arrow_native"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/clickhouse/bulk_operations.py"
      via: "cache.store_bars_batch()"
      pattern: "store_bars_batch"
---

<objective>
Replace the broken SEED-001 "skip Parquet range" logic with "process Parquet ticks into bars via Rust processor." On sidecar restart, today's Parquet ticks are read, processed per (symbol, threshold) into bars, written to ClickHouse, and the engine is seeded with `last_completed_bar_tid` so `fill_from_rest` only handles the small remaining gap.

Purpose: Eliminates 274K+ trade gaps caused by SEED-001 advancing the REST seed past deleted bars without processing them.
Output: Modified `midnight.py` with `_process_parquet_ticks()`, updated `__init__.py` call site, and unit tests.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/06-fix-seed-001-process-parquet-ticks/06-CONTEXT.md
@.planning/phases/06-fix-seed-001-process-parquet-ticks/06-RESEARCH.md

@python/opendeviationbar/sidecar/**init**.py
@python/opendeviationbar/sidecar/midnight.py
@python/opendeviationbar/processors/core.py
@python/opendeviationbar/clickhouse/bulk_operations.py
@python/opendeviationbar/storage/parquet.py
@tests/test_parquet_first_catchup.py

<interfaces>
<!-- Key types and contracts the executor needs. Extracted from codebase. -->

From python/opendeviationbar/storage/parquet.py:

```python
class TickStorage:
    @property
    def ticks_dir(self) -> Path: ...

    def read_ticks(
        self,
        symbol: str,
        start_ts: int | None = None,
        end_ts: int | None = None,
        *,
        timestamp_col: str = "timestamp",
        max_memory_mb: int | None = None,
    ) -> pl.DataFrame:
        """Read tick data from Parquet files.

        symbol: bare name e.g. "BTCUSDT" (NOT "BINANCE_SPOT_BTCUSDT")
        start_ts/end_ts: microseconds (milliseconds auto-detected and converted)
        Returns materialized pl.DataFrame (lazy scan + collect internally).
        Returns empty DataFrame if symbol dir or files don't exist.
        """
        ...
```

Note: `_get_symbol_dir()` returns `ticks_dir / symbol` (bare symbol name).
The seeder writes to bare symbol dirs for spot. Symlinks (`BINANCE_SPOT_BTCUSDT/ -> BTCUSDT/`)
exist at the filesystem level but `read_ticks("BTCUSDT", ...)` works directly — no prefix mapping needed.

From python/opendeviationbar/processors/core.py:

```python
class OpenDeviationBarProcessor:
    def __init__(self, threshold_decimal_bps: int, *, symbol: str = "", ...): ...
    def process_trades_arrow_native(self, arrow_table: pa.Table) -> pa.RecordBatch: ...
    def last_completed_bar_tid(self) -> int | None: ...
```

From python/opendeviationbar/clickhouse/bulk_operations.py:

```python
def store_bars_batch(
    self,
    symbol: str,
    threshold_decimal_bps: int,
    bars: pl.DataFrame,
    version: str | None = None,
    ouroboros_mode: str | None = None,
    *,
    skip_dedup: bool = False,
    overlap_strategy: OverlapStrategy = "warn",
    _skip_lock: bool = False,
) -> int: ...
```

From python/opendeviationbar/sidecar/midnight.py:

```python
def _read_parquet_max_tids(
    symbols: list[str],
    *,
    ticks_dir: Path | None = None,
) -> dict[str, int]: ...
```

From python/opendeviationbar/sidecar/**init**.py (lines 366-391):

```python
# Current broken parquet-first block:
try:
    from .midnight import _read_parquet_max_tids
    parquet_tids = _read_parquet_max_tids(config.symbols)
    if parquet_tids:
        for symbol, max_tid in parquet_tids.items():
            for threshold in config.thresholds:
                engine.seed_trade_id(symbol, threshold, max_tid)
        logger.info(...)
except Exception:
    logger.warning("parquet-first: failed ...", exc_info=True)
```

Parquet schema columns (\_CANONICAL_COLS in parquet.py):

```python
["agg_trade_id", "price", "quantity", "first_trade_id", "last_trade_id", "timestamp", "is_buyer_maker"]
```

- `timestamp` column is Int64 microseconds
- Arrow processing expects: `timestamp` (Int64), `price` (Float64), `quantity` (Float64)
  </interfaces>

</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add _process_parquet_ticks() to midnight.py with tests</name>
  <files>python/opendeviationbar/sidecar/midnight.py, tests/test_parquet_bar_processing.py</files>
  <read_first>
    python/opendeviationbar/sidecar/midnight.py
    python/opendeviationbar/processors/core.py
    python/opendeviationbar/clickhouse/bulk_operations.py
    python/opendeviationbar/storage/parquet.py
    tests/test_parquet_first_catchup.py
    .planning/phases/06-fix-seed-001-process-parquet-ticks/06-RESEARCH.md
  </read_first>
  <behavior>
    - Test 1 (test_fresh_parquet_processes_bars): Given a fresh Parquet file with known trades for 2 symbols, _process_parquet_ticks() returns {symbol: last_completed_bar_tid} for symbols that produced bars. Mock OpenDeviationBarProcessor and store_bars_batch to verify they are called per (symbol, threshold). Mock TickStorage.read_ticks to return the test DataFrame.
    - Test 2 (test_seed_uses_completed_bar_tid): Verify returned TIDs come from processor.last_completed_bar_tid() NOT from max(agg_trade_id) in Parquet. Mock processor where last_completed_bar_tid() returns 5000 but max agg_trade_id in Parquet is 5500 -- result must be 5000.
    - Test 3 (test_stale_parquet_skipped): Parquet file mtime older than 600s is skipped, returns empty dict for that symbol. Write the Parquet file with old mtime so the freshness check fails before read_ticks is called.
    - Test 4 (test_missing_parquet_returns_empty): No Parquet file exists, TickStorage.read_ticks returns empty DataFrame, returns empty dict.
    - Test 5 (test_processing_failure_skips_symbol): If process_trades_arrow_native raises, that symbol is skipped (non-fatal), other symbols still processed.
    - Test 6 (test_no_completed_bars_omits_symbol): If processor.last_completed_bar_tid() returns None (no complete bars produced), symbol is omitted from result.
    - Test 7 (test_all_thresholds_processed): Given 2 thresholds [250, 500], verify a processor is created and process_trades_arrow_native called for each threshold.
  </behavior>
  <action>
    1. Create `tests/test_parquet_bar_processing.py` following the pattern in `tests/test_parquet_first_catchup.py`:
       - Same fixtures: `today_str`, `ticks_dir`, `_write_parquet` helper
       - Extend `_write_parquet` to include all required columns: `agg_trade_id` (Int64), `price` (Float64), `quantity` (Float64), `first_trade_id` (Int64), `last_trade_id` (Int64), `timestamp` (Int64 microseconds), `is_buyer_maker` (Boolean)
       - Mock `OpenDeviationBarProcessor` and `OpenDeviationBarCache` to avoid real Rust/ClickHouse calls
       - **Per CONTEXT.md locked decision**: Mock `TickStorage.read_ticks` via `unittest.mock.patch` on `opendeviationbar.sidecar.midnight.TickStorage` — the function uses `TickStorage().read_ticks(symbol)` to load ticks, NOT `pl.scan_parquet()` directly
       - Use `unittest.mock.patch` on `opendeviationbar.sidecar.midnight.OpenDeviationBarProcessor` and `opendeviationbar.sidecar.midnight.OpenDeviationBarCache`
       - For staleness tests (test 3): still write a real Parquet file to the ticks_dir so the `parquet_path.exists()` and `parquet_path.stat().st_mtime` freshness checks work — the mock on TickStorage.read_ticks is only reached if freshness passes

    2. Add `_process_parquet_ticks()` function to `midnight.py` AFTER the existing `_read_parquet_max_tids()` function (after line 756). The function signature:

       ```python
       def _process_parquet_ticks(
           symbols: list[str],
           thresholds: list[int],
           *,
           ticks_dir: Path | None = None,
       ) -> dict[str, int]:
       ```

       Implementation:
       - Import `TickStorage` from `opendeviationbar.storage.parquet` (lazy, inside function)
       - Import `OpenDeviationBarProcessor` from `opendeviationbar.processors.core` (lazy, inside function)
       - Import `OpenDeviationBarCache` from `opendeviationbar.clickhouse.cache` (lazy, inside function)
       - Import `__version__` from `opendeviationbar._core`
       - Reuse the same staleness_threshold_s = 600 and ticks_dir resolution pattern from `_read_parquet_max_tids()`
       - For each symbol: check Parquet file exists at `ticks_dir / symbol / f"{today_str}.parquet"` and freshness (<600s via `st_mtime`)
       - **Per CONTEXT.md locked decision**: Read ticks via `TickStorage().read_ticks(symbol)` which returns a `pl.DataFrame`. Note: `read_ticks()` uses bare symbol names ("BTCUSDT") internally — no `BINANCE_SPOT_` prefix needed (research confirmed seeder and TickStorage agree on bare names for spot). Pass `start_ts=None, end_ts=None` to get all ticks for the symbol (read_ticks internally discovers daily files).
       - **IMPORTANT**: Since `read_ticks()` scans ALL daily files for the symbol (not just today), and we only want today's ticks, either: (a) compute today's start/end timestamps in microseconds and pass as `start_ts`/`end_ts` to `read_ticks()`, or (b) construct the TickStorage with the ticks_dir override if provided. Option (a) is cleaner — compute `today_start_us` and `today_end_us` from `today_str`. Example: `today_start_us = int(datetime.strptime(today_str, "%Y-%m-%d").replace(tzinfo=UTC).timestamp() * 1_000_000)` and `today_end_us = today_start_us + 86_400 * 1_000_000 - 1`.
       - Convert DataFrame to Arrow: `arrow_table = ticks.to_arrow()`
       - For each threshold: create `OpenDeviationBarProcessor(threshold, symbol=symbol)`, call `processor.process_trades_arrow_native(arrow_table)`, convert result to Polars `pl.from_arrow(bars_arrow)`, write non-empty results via `cache.store_bars_batch(symbol=symbol, threshold_decimal_bps=threshold, bars=bars_pl, version=f"{__version__}+parquet", overlap_strategy="off")`
       - Track best (highest) `last_completed_bar_tid` across all thresholds per symbol
       - If best_tid is not None, add to result dict
       - Wrap per-symbol processing in try/except (non-fatal: log warning, continue to next symbol)
       - Log summary per symbol: bars count, thresholds processed, seed_tid, age

    3. Run tests: `uv run --python 3.13 pytest tests/test_parquet_bar_processing.py -x -q`

  </action>
  <verify>
    <automated>uv run --python 3.13 pytest tests/test_parquet_bar_processing.py -x -q</automated>
  </verify>
  <acceptance_criteria>
    - midnight.py contains `def _process_parquet_ticks(symbols: list[str], thresholds: list[int]`
    - midnight.py contains `TickStorage` import and `read_ticks(` call (NOT `pl.scan_parquet()` directly)
    - midnight.py contains `processor.last_completed_bar_tid()` (NOT `max(agg_trade_id)`)
    - midnight.py contains `process_trades_arrow_native(arrow_table)`
    - midnight.py contains `store_bars_batch(` with `overlap_strategy="off"`
    - midnight.py contains `staleness_threshold_s = 600` in the new function
    - tests/test_parquet_bar_processing.py mocks `TickStorage` (NOT `pl.scan_parquet`)
    - tests/test_parquet_bar_processing.py contains `test_seed_uses_completed_bar_tid`
    - tests/test_parquet_bar_processing.py contains `test_fresh_parquet_processes_bars`
    - tests/test_parquet_bar_processing.py contains `test_all_thresholds_processed`
    - All 7 tests pass
  </acceptance_criteria>
  <done>_process_parquet_ticks() function exists in midnight.py, reads ticks via TickStorage.read_ticks() per CONTEXT.md locked decision, processes into bars via Rust processor, writes to ClickHouse, returns {symbol: last_completed_bar_tid}. All 7 tests pass.</done>
</task>

<task type="auto">
  <name>Task 2: Replace parquet-first block in sidecar __init__.py</name>
  <files>python/opendeviationbar/sidecar/__init__.py</files>
  <read_first>
    python/opendeviationbar/sidecar/__init__.py
    python/opendeviationbar/sidecar/midnight.py
    .planning/phases/06-fix-seed-001-process-parquet-ticks/06-CONTEXT.md
  </read_first>
  <action>
    Replace lines 366-391 in `sidecar/__init__.py` (the current parquet-first block) with the new logic that calls `_process_parquet_ticks()` instead of `_read_parquet_max_tids()`.

    The current block (lines 366-391):
    ```python
    # Parquet-first catchup (PQFIRST-01, PQFIRST-02, PQFIRST-03):
    # Read max(agg_trade_id) from today's seeder Parquet cache per symbol.
    # Advances REST start point past cached data, reducing API weight.
    # Non-fatal: falls back to midnight preflight seeds on any failure.
    try:
        from .midnight import _read_parquet_max_tids
        parquet_tids = _read_parquet_max_tids(config.symbols)
        if parquet_tids:
            for symbol, max_tid in parquet_tids.items():
                for threshold in config.thresholds:
                    engine.seed_trade_id(symbol, threshold, max_tid)
            logger.info(
                "parquet-first: advanced seeds for %d/%d symbols: %s",
                len(parquet_tids),
                len(config.symbols),
                ", ".join(f"{s}@{tid}" for s, tid in sorted(parquet_tids.items())),
            )
    except Exception:
        logger.warning(
            "parquet-first: failed (non-fatal, falling back to preflight seeds)",
            exc_info=True,
        )
    ```

    Replace with:
    ```python
    # SEED-FIX-01, SEED-FIX-02: Process Parquet ticks into bars via Rust,
    # then seed fill_from_rest with last_completed_bar_tid (not max_tid).
    # Non-fatal: falls back to midnight preflight seeds on any failure.
    try:
        from .midnight import _process_parquet_ticks

        parquet_seeds = _process_parquet_ticks(config.symbols, config.thresholds)
        if parquet_seeds:
            for symbol, seed_tid in parquet_seeds.items():
                for threshold in config.thresholds:
                    engine.seed_trade_id(symbol, threshold, seed_tid)
            logger.info(
                "parquet-process: seeded %d/%d symbols: %s",
                len(parquet_seeds),
                len(config.symbols),
                ", ".join(
                    f"{s}@{tid}"
                    for s, tid in sorted(parquet_seeds.items())
                ),
            )
    except Exception:
        logger.warning(
            "parquet-process: failed (non-fatal, falling back to preflight seeds)",
            exc_info=True,
        )
    ```

    Key changes:
    - Import `_process_parquet_ticks` instead of `_read_parquet_max_tids`
    - Pass both `config.symbols` and `config.thresholds` (new function needs thresholds for per-threshold processing)
    - Variable renamed from `parquet_tids`/`max_tid` to `parquet_seeds`/`seed_tid` (semantic: these are now completed-bar TIDs, not raw max TIDs)
    - Log prefix changed from "parquet-first" to "parquet-process"
    - Comment updated to reference SEED-FIX-01, SEED-FIX-02

    After modification, run the full existing test suite to ensure no regressions:
    `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py tests/test_parquet_bar_processing.py -x -q`

  </action>
  <verify>
    <automated>uv run --python 3.13 pytest tests/test_parquet_first_catchup.py tests/test_parquet_bar_processing.py -x -q</automated>
  </verify>
  <acceptance_criteria>
    - sidecar/__init__.py contains `from .midnight import _process_parquet_ticks` (NOT `_read_parquet_max_tids`)
    - sidecar/__init__.py contains `_process_parquet_ticks(config.symbols, config.thresholds)`
    - sidecar/__init__.py contains `parquet_seeds` variable (NOT `parquet_tids`)
    - sidecar/__init__.py contains `"parquet-process:"` log prefix (NOT `"parquet-first:"`)
    - sidecar/__init__.py contains `# SEED-FIX-01, SEED-FIX-02`
    - sidecar/__init__.py does NOT contain `_read_parquet_max_tids` (old function no longer imported)
    - Both test files pass: test_parquet_first_catchup.py and test_parquet_bar_processing.py
  </acceptance_criteria>
  <done>The sidecar startup sequence processes Parquet ticks into bars and seeds with last_completed_bar_tid. Old _read_parquet_max_tids import replaced. All tests pass.</done>
</task>

</tasks>

<verification>
1. `uv run --python 3.13 pytest tests/test_parquet_bar_processing.py -x -q` -- all 7 new tests pass
2. `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py -x -q` -- existing tests still pass (no regression)
3. `grep -n "_process_parquet_ticks" python/opendeviationbar/sidecar/midnight.py` -- function exists
4. `grep -n "_process_parquet_ticks" python/opendeviationbar/sidecar/__init__.py` -- called from startup
5. `grep -n "last_completed_bar_tid" python/opendeviationbar/sidecar/midnight.py` -- correct seeding API used
6. `grep -n "read_ticks" python/opendeviationbar/sidecar/midnight.py` -- uses TickStorage.read_ticks() (NOT pl.scan_parquet)
7. `grep -c "_read_parquet_max_tids" python/opendeviationbar/sidecar/__init__.py` -- returns 0 (old import removed)
</verification>

<success_criteria>

- \_process_parquet_ticks() in midnight.py reads ticks via TickStorage.read_ticks() (per CONTEXT.md locked decision), processes through Rust per (symbol, threshold), writes bars to CH, returns {symbol: last_completed_bar_tid}
- Sidecar **init**.py calls \_process_parquet_ticks() instead of \_read_parquet_max_tids()
- Stale/missing Parquet correctly falls back (same as before)
- 7 new tests + existing parquet_first tests all green
  </success_criteria>

<output>
After completion, create `.planning/phases/06-fix-seed-001-process-parquet-ticks/06-01-SUMMARY.md`
</output>
