---
phase: 30-parquet-first-gap-fill-junction-telemetry
plan: 01
subsystem: sidecar
tags: [parquet, gap-fill, aion, performance]
dependency_graph:
  requires: [29-01, 29-02]
  provides: [RESUME-02]
  affects: [sidecar-startup, fill-from-rest]
tech_stack:
  added: []
  patterns: [parquet-first-gap-fill, tid-filtering, non-fatal-fallback]
key_files:
  created:
    - python/opendeviationbar/sidecar/parquet_fill.py
  modified:
    - python/opendeviationbar/sidecar/__init__.py
    - tests/test_sidecar_parquet_fill.py
decisions:
  - "Use min(last_completed_bar_tid) across thresholds for conservative seed TID"
  - "600s staleness threshold matching _process_parquet_ticks in midnight.py"
  - "Non-fatal wrapper: any failure returns empty dict, falls through to REST"
metrics:
  duration: 5m31s
  completed: "2026-03-28T08:14:21Z"
  tasks: 1/1
  files_created: 1
  files_modified: 2
  test_count: 6
---

# Phase 30 Plan 01: Parquet-First Aion Gap Fill Summary

Parquet-first gap fill for Aion sidecar startup: reads local tick cache before falling back to REST API, eliminating REST round-trips for ranges already cached by the seeder.

## One-liner

Parquet-first Aion gap fill via `_parquet_prefill_aion()` that reads cached ticks, processes through Rust per threshold, writes bars to CH, and advances engine seed TIDs so `fill_from_rest` only covers uncached ranges.

## What Was Built

### parquet_fill.py (new module)

`_parquet_prefill_aion(ch_cache, aion_symbols, resume_tids, thresholds)` function that:

1. Creates `TickStorage` and checks for fresh Parquet files (today, <10 min old)
2. For each Aion symbol with a resume TID: reads ticks via `read_ticks_streaming()`
3. Filters trades to only those with `agg_trade_id > resume_tid`
4. Processes filtered trades through `OpenDeviationBarProcessor` per threshold
5. Writes completed bars to ClickHouse via `store_bars_batch()` with `ouroboros_mode="aion"`
6. Returns `{symbol: min(last_completed_bar_tid)}` across thresholds (conservative seed)

Split into `_parquet_prefill_aion()` (try/except wrapper) and `_prefill_symbol()` (per-symbol logic) to keep under ruff's statement limit.

### **init**.py integration

Wired as Step 4.5 in the Aion startup path:

- After Step 4 (seed engine with CH resume TIDs)
- Before `fill_from_rest()` call
- If Parquet produces seed TIDs, `engine.seed_trade_id()` is called for all thresholds
- Non-fatal: `(OSError, RuntimeError, ConnectionError)` caught and logged

### Test coverage

6 tests in `tests/test_sidecar_parquet_fill.py`:

1. Returns seed TIDs when Parquet has ticks
2. Omits symbol when no Parquet files exist
3. Skips stale Parquet (>10 min old)
4. Returns empty dict on exception (non-fatal)
5. Filters trades with `agg_trade_id <= resume_tid`
6. Engine seeds updated correctly at call site

## Commits

| Hash    | Type | Description                                                      |
| ------- | ---- | ---------------------------------------------------------------- |
| 66f7587 | test | Add failing tests for \_parquet_prefill_aion (RED)               |
| 260b74a | feat | Implement \_parquet_prefill_aion + wire into **init**.py (GREEN) |

## Decisions Made

1. **Conservative seed TID**: Use `min(last_completed_bar_tid)` across thresholds, not max. This ensures `fill_from_rest` covers any threshold that's behind.
2. **Staleness = 600s**: Matches `_process_parquet_ticks` in midnight.py for consistency.
3. **Non-fatal wrapper**: Outer function catches all exceptions and returns `{}`, inner functions catch `(OSError, RuntimeError, ConnectionError)` per ruff rules.
4. **Single-file check**: Currently checks today's Parquet file only. Future enhancement could scan recent days for Aion symbols with multi-day gaps.

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 3 - Blocking] Function too long for ruff PLR0915**

- **Found during:** GREEN phase
- **Issue:** `_parquet_prefill_aion_inner` exceeded 50 statements
- **Fix:** Extracted `_prefill_symbol()` for per-symbol processing logic
- **Files modified:** `python/opendeviationbar/sidecar/parquet_fill.py`

**2. [Rule 3 - Blocking] Import sorting (I001)**

- **Found during:** GREEN phase
- **Issue:** `parquet_fill` import placed out of alphabetical order in `__init__.py`
- **Fix:** Moved import after `notifications` and before `recovery`
- **Files modified:** `python/opendeviationbar/sidecar/__init__.py`

## Known Stubs

None -- all data paths are wired to real implementations.

## Self-Check: PASSED
