---
phase: 03-parquet-first-catchup
plan: 01
subsystem: sidecar
tags: [parquet, polars, sidecar-startup, rest-optimization, tick-cache]

# Dependency graph
requires:
  - phase: 02-sidecar-fill-pipeline
    provides: fill_from_rest bar pipeline with Stathera validation
provides:
  - "_read_parquet_max_tids() function for reading max trade ID from Parquet cache"
  - "Parquet-first seed override in sidecar startup between DELETE today and fill_from_rest"
  - "Graceful fallback when Parquet is stale/missing/corrupt"
affects: [sidecar-startup, seeder-parquet-cache]

# Tech tracking
tech-stack:
  added: []
  patterns:
    [
      "Parquet lazy scan for single-column max aggregation via pl.scan_parquet().select(max()).collect()",
    ]

key-files:
  created:
    - tests/test_parquet_first_catchup.py
  modified:
    - python/opendeviationbar/sidecar/midnight.py
    - python/opendeviationbar/sidecar/__init__.py

key-decisions:
  - "Use bare ticks_dir/symbol/today.parquet path resolution instead of TickStorage.read_ticks() for minimal I/O"
  - "Parquet seed override is additive on top of midnight preflight -- never replaces, only advances"
  - "10-minute staleness threshold matches seeder's 5-min write cycle (2x margin)"

patterns-established:
  - "Parquet-first pattern: read cursor from Parquet, advance engine seed, let Rust fill residual gap"
  - "Non-fatal integration blocks: wrap optional optimization in try/except with WARNING log"

requirements-completed: [PQFIRST-01, PQFIRST-02, PQFIRST-03]

# Metrics
duration: 3min
completed: 2026-03-24
---

# Phase 03 Plan 01: Parquet-First Catchup Summary

**Sidecar startup reads max(agg_trade_id) from today's Parquet tick cache per symbol, advancing REST fill start point past cached data to reduce API weight from ~500 to ~50**

## Performance

- **Duration:** 3 min
- **Started:** 2026-03-24T20:05:39Z
- **Completed:** 2026-03-24T20:08:38Z
- **Tasks:** 2 (TDD: RED + GREEN + integration)
- **Files modified:** 3

## Accomplishments

- Implemented `_read_parquet_max_tids()` in midnight.py with lazy Polars scan for minimal I/O
- Integrated Parquet-first seed override in sidecar `_create_engine()` between DELETE today and fill_from_rest
- All failure modes (stale >10min, missing, corrupt) fall back gracefully to preflight seeds
- 8 tests covering all three PQFIRST requirements pass

## Task Commits

Each task was committed atomically:

1. **Task 1 RED: Failing tests** - `bfc5e77` (test)
2. **Task 1 GREEN: \_read_parquet_max_tids implementation** - `240216c` (feat)
3. **Task 2: Sidecar integration + integration tests** - `a2461cb` (feat)

## Files Created/Modified

- `python/opendeviationbar/sidecar/midnight.py` - Added `_read_parquet_max_tids()`: reads max(agg_trade_id) from today's Parquet per symbol with staleness check
- `python/opendeviationbar/sidecar/__init__.py` - Added Parquet-first seed override block between DELETE today and fill_from_rest
- `tests/test_parquet_first_catchup.py` - 8 tests: fresh/stale/missing/corrupt Parquet, seed override, integration, empty dict, exception non-fatal

## Decisions Made

- Used `pl.scan_parquet().select(max()).collect()` for minimal I/O -- reads only agg_trade_id column statistics, not full DataFrame
- Parquet seed override is additive: calls `engine.seed_trade_id()` again with higher TID, overwriting midnight preflight seed only when Parquet max > crossover TID
- ticks_dir parameter is keyword-only for testing; production path resolved from TickStorage().ticks_dir

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all data paths are fully wired.

## Issues Encountered

- Pre-commit ruff hook removed unused `patch` import on first commit; re-added when integration tests needed it. No impact.
- Pre-existing test failure in `test_sync_flush_atomic_replace.py::TestSyncFlushCrossMidnightFilter::test_cross_midnight_bars_filtered` -- confirmed pre-existing (not caused by this plan's changes).

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Phase 03 (Parquet-First Catchup) is complete -- all 3 PQFIRST requirements satisfied
- Phase 04 (Combined WebSocket Streams) and Phase 05 (Dead Code Cleanup) are independent and can proceed

## Self-Check: PASSED

- All 3 created/modified files exist on disk
- All 3 task commits (bfc5e77, 240216c, a2461cb) found in git log
- 8/8 tests pass

---

_Phase: 03-parquet-first-catchup_
_Completed: 2026-03-24_
