---
phase: 06-fix-seed-001-process-parquet-ticks
plan: 01
subsystem: sidecar
tags: [parquet, rust-processor, clickhouse, seed-tid, sidecar-startup]

# Dependency graph
requires:
  - phase: none
    provides: existing sidecar startup sequence with _read_parquet_max_tids
provides:
  - "_process_parquet_ticks() function that reads Parquet ticks, processes into bars, writes to CH"
  - "Sidecar startup seeding with last_completed_bar_tid instead of max agg_trade_id"
affects: [sidecar-startup, kintsugi-gap-repair, deploy]

# Tech tracking
tech-stack:
  added: []
  patterns:
    [
      "Parquet ticks -> Arrow -> Rust processor -> Polars -> ClickHouse pipeline",
    ]

key-files:
  created:
    - tests/test_parquet_bar_processing.py
  modified:
    - python/opendeviationbar/sidecar/midnight.py
    - python/opendeviationbar/sidecar/__init__.py

key-decisions:
  - "Lazy imports inside _process_parquet_ticks() to avoid circular deps; patches target source modules"
  - "overlap_strategy='off' for store_bars_batch since clean-slate DELETE precedes this step"

patterns-established:
  - "Parquet tick processing: TickStorage.read_ticks() -> .to_arrow() -> process_trades_arrow_native() -> pl.from_arrow() -> store_bars_batch()"

requirements-completed: [SEED-FIX-01, SEED-FIX-02]

# Metrics
duration: 4min
completed: 2026-03-25
---

# Phase 06 Plan 01: Fix SEED-001 Parquet-First Bar Processing Summary

**Replaced broken SEED-001 "skip Parquet range" with "process Parquet ticks into bars via Rust processor, seed with last_completed_bar_tid"**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-25T09:52:05Z
- **Completed:** 2026-03-25T09:55:51Z
- **Tasks:** 2
- **Files modified:** 3

## Accomplishments

- Added `_process_parquet_ticks()` to midnight.py: reads today's Parquet via TickStorage.read_ticks(), processes per (symbol, threshold) via Rust `process_trades_arrow_native()`, writes bars to ClickHouse, returns `{symbol: last_completed_bar_tid}`
- Replaced broken parquet-first block in sidecar `__init__.py` with new function call
- 7 new tests covering all edge cases (fresh, stale, missing, failure, no-bars, multi-threshold, correct TID source)

## Task Commits

Each task was committed atomically:

1. **Task 1: Add \_process_parquet_ticks() to midnight.py with tests (TDD)**
   - `b07fe97` (test: failing tests for Parquet tick bar processing)
   - `23e6284` (feat: implement \_process_parquet_ticks() in midnight.py)
2. **Task 2: Replace parquet-first block in sidecar **init**.py** - `1e572da` (feat)

## Files Created/Modified

- `python/opendeviationbar/sidecar/midnight.py` - Added `_process_parquet_ticks()` function (100 lines)
- `python/opendeviationbar/sidecar/__init__.py` - Replaced parquet-first block to call new function
- `tests/test_parquet_bar_processing.py` - 7 unit tests for SEED-FIX-01/02

## Decisions Made

- Lazy imports inside `_process_parquet_ticks()` (TickStorage, OpenDeviationBarProcessor, OpenDeviationBarCache) to match existing `_read_parquet_max_tids()` pattern and avoid circular imports
- Test patches target source modules (`opendeviationbar.storage.parquet.TickStorage` etc.) since lazy imports aren't module-level attributes
- `overlap_strategy="off"` is correct because clean-slate DELETE of today's bars happens in Step 3 before this runs

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 3 - Blocking] Fixed test mock patch targets for lazy imports**

- **Found during:** Task 1 (TDD GREEN phase)
- **Issue:** Plan specified patching `opendeviationbar.sidecar.midnight.TickStorage` but lazy imports inside the function don't create module-level attributes, causing `AttributeError`
- **Fix:** Changed patch targets to source modules: `opendeviationbar.storage.parquet.TickStorage`, `opendeviationbar.processors.core.OpenDeviationBarProcessor`, `opendeviationbar.clickhouse.cache.OpenDeviationBarCache`
- **Files modified:** tests/test_parquet_bar_processing.py
- **Verification:** All 7 tests pass
- **Committed in:** 23e6284 (Task 1 GREEN commit)

---

**Total deviations:** 1 auto-fixed (1 blocking)
**Impact on plan:** Necessary fix for test infrastructure to work with lazy import pattern. No scope creep.

## Issues Encountered

None beyond the deviation above.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- SEED-001 fix is complete and ready for deploy
- Existing `_read_parquet_max_tids()` function preserved for backward compatibility (old tests still pass)
- Production deploy will activate the fix on next sidecar restart

## Self-Check: PASSED

---

_Phase: 06-fix-seed-001-process-parquet-ticks_
_Completed: 2026-03-25_
