---
phase: 03-parquet-first-catchup
plan: 01
type: tdd
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/midnight.py
  - python/opendeviationbar/sidecar/__init__.py
  - tests/test_parquet_first_catchup.py
autonomous: true
requirements: [PQFIRST-01, PQFIRST-02, PQFIRST-03]

must_haves:
  truths:
    - "Sidecar startup reads max(agg_trade_id) from today's Parquet tick cache per symbol"
    - "REST API is called only for the gap between Parquet's last trade ID and WS buffer start"
    - "When Parquet data is stale (>10 min) or missing, sidecar falls back to full fill_from_rest with no behavior change"
  artifacts:
    - path: "python/opendeviationbar/sidecar/midnight.py"
      provides: "_read_parquet_max_tids() function"
      contains: "_read_parquet_max_tids"
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "Parquet-first seed override in _create_engine between DELETE today and fill_from_rest"
      contains: "parquet_tids"
    - path: "tests/test_parquet_first_catchup.py"
      provides: "Unit tests for all three PQFIRST requirements"
      min_lines: 80
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/midnight.py"
      via: "import _read_parquet_max_tids"
      pattern: "_read_parquet_max_tids"
    - from: "python/opendeviationbar/sidecar/midnight.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "TickStorage._get_parquet_path() + pl.scan_parquet"
      pattern: "scan_parquet"
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "engine.seed_trade_id"
      via: "Parquet max TID overrides preflight seed"
      pattern: "seed_trade_id.*parquet"
---

<objective>
Implement Parquet-first catchup: sidecar startup reads max(agg_trade_id) from today's Parquet tick cache per symbol, advances REST fill start point past cached data, and falls back gracefully when Parquet is stale or missing.

Purpose: Reduce sidecar startup REST API weight from ~500 to ~50 when fresh Parquet data exists (seeder writes every 5 min). Eliminates redundant re-fetching of trades the seeder already cached locally.

Output: `_read_parquet_max_tids()` function in midnight.py, integration in sidecar `_create_engine()`, and comprehensive tests.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/03-parquet-first-catchup/03-RESEARCH.md

@python/opendeviationbar/sidecar/**init**.py
@python/opendeviationbar/sidecar/midnight.py
@python/opendeviationbar/storage/parquet.py
@python/opendeviationbar/storage/parquet_query.py

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/storage/parquet.py:

```python
class TickStorage:
    def __init__(self, cache_dir: Path | str | None = None): ...
    def _get_parquet_path(self, symbol: str, day_key: str) -> Path: ...
    # day_key format: "YYYY-MM-DD"
    # Returns: ticks_dir / symbol / f"{day_key}.parquet"
```

From python/opendeviationbar/sidecar/**init**.py (integration point, lines ~340-368):

```python
# Step 3: DELETE today (clean slate) -- lines 340-358
# ... partition-scoped DELETE loop ...

# << NEW: Parquet-first seed override goes HERE >>

# Step 5: fill_from_rest -- line 368
rest_bars = engine.fill_from_rest()
```

From python/opendeviationbar/sidecar/midnight.py (existing seed pattern):

```python
# Line 880: engine.seed_trade_id(symbol, threshold, crossover_tid - 1)
# This is how midnight preflight seeds the engine.
# Parquet override calls the same method with a higher TID.
```

Engine API (Rust, exposed via PyO3):

```python
engine.seed_trade_id(symbol: str, threshold: int, trade_id: int) -> None
# Sets the gap detector seed. fill_from_rest starts from max(checkpoint_tid, ch_seed_tid) + 1.
# Calling again with a higher TID overwrites the previous seed.
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: _read_parquet_max_tids() with TDD tests</name>
  <files>python/opendeviationbar/sidecar/midnight.py, tests/test_parquet_first_catchup.py</files>
  <read_first>
    - python/opendeviationbar/sidecar/midnight.py (full file -- understand existing patterns and imports)
    - python/opendeviationbar/storage/parquet.py lines 68-140 (TickStorage class, _get_parquet_path)
    - python/opendeviationbar/storage/parquet_query.py lines 40-70 (_get_parquet_path, _get_symbol_dir)
    - .planning/phases/03-parquet-first-catchup/03-RESEARCH.md (pattern code, anti-patterns, pitfalls)
  </read_first>
  <behavior>
    - Test 1 (PQFIRST-01): Given a temp dir with fresh Parquet files containing agg_trade_id column for BTCUSDT and ETHUSDT, _read_parquet_max_tids returns {BTCUSDT: max_tid_btc, ETHUSDT: max_tid_eth}
    - Test 2 (PQFIRST-03 stale): Given a Parquet file with mtime >600s ago, _read_parquet_max_tids omits that symbol from results
    - Test 3 (PQFIRST-03 missing): Given symbols where no today Parquet file exists, _read_parquet_max_tids returns empty dict
    - Test 4 (PQFIRST-03 corrupt): Given a Parquet file that raises on read, _read_parquet_max_tids skips that symbol (logs debug, no exception)
    - Test 5 (PQFIRST-02): Given _read_parquet_max_tids returns {BTCUSDT: 5000}, seed_trade_id is called with tid=5000 for each threshold, advancing REST start past Parquet data
  </behavior>
  <action>
    RED phase: Create tests/test_parquet_first_catchup.py with all 5 tests. Tests use tmp_path fixtures to create mock Parquet files with polars. For staleness tests, use os.utime() to backdate mtime. For Test 5, mock engine.seed_trade_id and verify calls.

    GREEN phase: Add `_read_parquet_max_tids(symbols: list[str], ticks_dir: Path | None = None) -> dict[str, int]` to midnight.py. Implementation:
    1. Import TickStorage, get ticks_dir from TickStorage().ticks_dir if not provided
    2. today_str = datetime.now(UTC).strftime("%Y-%m-%d")
    3. For each symbol, resolve path via ticks_dir / symbol / f"{today_str}.parquet"
    4. Check path.exists() -- skip if missing
    5. Check staleness: time.time() - path.stat().st_mtime > 600 -- skip if stale, log info
    6. Use pl.scan_parquet(path).select(pl.col("agg_trade_id").max()).collect().item() -- lazy scan, minimal I/O
    7. If max_tid valid (not None, > 0), add to result dict
    8. Wrap per-symbol in try/except: log debug on failure, continue to next symbol
    9. Return result dict

    IMPORTANT anti-patterns from research:
    - Do NOT read full Parquet into memory -- use pl.scan_parquet().select(max()).collect()
    - Do NOT process Parquet trades through Rust -- only need max TID as a cursor
    - Do NOT use TickStorage.read_ticks() -- too heavy, we only need one integer per symbol
    - Use bare symbol names (BTCUSDT not BINANCE_SPOT_BTCUSDT) -- matches config.symbols convention

  </action>
  <verify>
    <automated>uv run --python 3.13 pytest tests/test_parquet_first_catchup.py -x -q</automated>
  </verify>
  <done>_read_parquet_max_tids returns correct max TID for fresh symbols, omits stale/missing/corrupt symbols, all 5 tests pass</done>
  <acceptance_criteria>
    - _read_parquet_max_tids exists in midnight.py with correct signature
    - Fresh Parquet files return max(agg_trade_id) per symbol
    - Stale files (>10 min) are skipped with info log
    - Missing files are silently skipped
    - Corrupt files are skipped with debug log (no exception propagation)
    - All 5 tests in test_parquet_first_catchup.py pass
  </acceptance_criteria>
</task>

<task type="auto">
  <name>Task 2: Integrate Parquet-first seed override in sidecar startup</name>
  <files>python/opendeviationbar/sidecar/__init__.py, tests/test_parquet_first_catchup.py</files>
  <read_first>
    - python/opendeviationbar/sidecar/__init__.py lines 240-400 (_create_engine startup sequence)
    - tests/test_parquet_first_catchup.py (tests from Task 1 -- understand test patterns)
    - .planning/phases/03-parquet-first-catchup/03-RESEARCH.md "Integration in sidecar/__init__.py" section
  </read_first>
  <action>
    In python/opendeviationbar/sidecar/__init__.py, add Parquet-first seed override between the DELETE today block (ends ~line 362) and the fill_from_rest call (line 368). The exact insertion point is AFTER the logger.info("pre-fill: killed mutations + DELETE today...") block and BEFORE "rest_bars = engine.fill_from_rest()".

    Add this block (wrapped in try/except for non-fatal fallback per PQFIRST-03):

    ```python
    # Parquet-first catchup (PQFIRST-01, PQFIRST-02, PQFIRST-03):
    # Read max(agg_trade_id) from today's seeder Parquet cache per symbol.
    # Advances REST start point past cached data, reducing API weight.
    # Non-fatal: falls back to midnight preflight seeds on any failure.
    try:
        from .midnight import _read_parquet_max_tids
        parquet_tids = _read_parquet_max_tids(config.symbols)
        if parquet_tids:
            for symbol, max_tid in parquet_tids.items():
                for threshold in config.thresholds:
                    engine.seed_trade_id(symbol, threshold, max_tid)
            logger.info(
                "parquet-first: advanced seeds for %d/%d symbols: %s",
                len(parquet_tids),
                len(config.symbols),
                ", ".join(f"{s}@{tid}" for s, tid in sorted(parquet_tids.items())),
            )
    except Exception:
        logger.warning(
            "parquet-first: failed (non-fatal, falling back to preflight seeds)",
            exc_info=True,
        )
    ```

    Also add a test (Test 6) to test_parquet_first_catchup.py that validates the integration: mock _read_parquet_max_tids to return a dict, mock engine.seed_trade_id, and verify that seed_trade_id is called for each (symbol, threshold) pair with the Parquet max TID.

    IMPORTANT: The Parquet seed override is ADDITIVE on top of midnight preflight. If _read_parquet_max_tids returns empty or throws, the preflight seeds remain unchanged -- zero regression (PQFIRST-03).

  </action>
  <verify>
    <automated>uv run --python 3.13 pytest tests/test_parquet_first_catchup.py -x -q</automated>
  </verify>
  <done>Sidecar startup calls _read_parquet_max_tids between DELETE today and fill_from_rest, advances engine seed per (symbol, threshold) when fresh Parquet exists, falls back silently on failure</done>
  <acceptance_criteria>
    - Parquet-first block inserted at correct position in _create_engine (after DELETE, before fill_from_rest)
    - engine.seed_trade_id called for each (symbol, threshold) pair when Parquet data is fresh
    - Entire block wrapped in try/except with WARNING log on failure (non-fatal)
    - When _read_parquet_max_tids returns empty dict, no seed_trade_id calls (preflight seeds unchanged)
    - All tests pass including new integration test
  </acceptance_criteria>
</task>

</tasks>

<verification>
1. All tests pass: `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py -x -v`
2. No regressions: `uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x -q` (sidecar tests from Phase 02)
3. Full suite: `mise run test-py` (should pass -- no breaking changes)
4. PQFIRST-01: _read_parquet_max_tids reads max(agg_trade_id) from today's Parquet per symbol
5. PQFIRST-02: engine.seed_trade_id called with Parquet max TID, advancing REST start point
6. PQFIRST-03: Stale/missing/corrupt Parquet handled gracefully with fallback to preflight seeds
</verification>

<success_criteria>

- \_read_parquet_max_tids() exists in midnight.py and correctly extracts max TID from fresh Parquet files
- Sidecar \_create_engine() calls Parquet-first seed override at the correct point in startup sequence
- All failure modes (stale, missing, corrupt, exception) fall back to preflight seeds with no behavior change
- 6+ tests covering all three PQFIRST requirements pass
- No regressions in existing sidecar tests
  </success_criteria>

<output>
After completion, create `.planning/phases/03-parquet-first-catchup/03-01-SUMMARY.md`
</output>
