---
phase: 30-parquet-first-gap-fill-junction-telemetry
plan: 01
type: tdd
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/sidecar/parquet_fill.py
  - python/opendeviationbar/sidecar/__init__.py
  - tests/test_sidecar_parquet_fill.py
autonomous: true
requirements: [RESUME-02]

must_haves:
  truths:
    - "On sidecar restart, Aion gap fill reads trades from Parquet tick cache first"
    - "Only REST API is used for TID ranges not covered by Parquet"
    - "When Parquet covers the full gap range, zero REST API calls during fill_from_rest"
    - "Parquet-first fill is non-fatal -- failures fall back to full REST fill"
  artifacts:
    - path: "python/opendeviationbar/sidecar/parquet_fill.py"
      provides: "_parquet_prefill_aion() function"
      exports: ["_parquet_prefill_aion"]
    - path: "tests/test_sidecar_parquet_fill.py"
      provides: "Unit tests for Parquet pre-fill"
      min_lines: 50
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/sidecar/parquet_fill.py"
      via: "_parquet_prefill_aion() call between seed step and fill_from_rest"
      pattern: "_parquet_prefill_aion"
    - from: "python/opendeviationbar/sidecar/parquet_fill.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "TickStorage().read_ticks() for loading cached trades"
      pattern: "TickStorage"
---

<objective>
Parquet-first gap fill for Aion sidecar startup: read local tick cache before falling back to REST API.

Purpose: Sidecar restart gap fill currently always uses Binance REST API (network-dependent, rate-limited). The seeder maintains local Parquet tick cache (~3ms/file read). Using Parquet first eliminates REST round-trips for ranges already cached, dramatically reducing restart latency.

Output: `parquet_fill.py` module with `_parquet_prefill_aion()` that processes Parquet ticks through Rust processors, writes bars to CH, and advances seed TIDs so `fill_from_rest()` only covers the remaining uncached range.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/29-delete-strategy-exchange-gap-floor/29-01-SUMMARY.md
@.planning/phases/29-delete-strategy-exchange-gap-floor/29-02-SUMMARY.md

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/sidecar/**init**.py (Aion startup path, lines 276-370):
The Parquet pre-fill must be inserted AFTER Step 4 (seed engine with resume TIDs)
and BEFORE the fill_from_rest() call (line 513). It must update seed TIDs via
engine.seed_trade_id(sym, thr, new_tid) for any symbol where Parquet advanced
the TID beyond the CH resume point.

```python
# Step 4: Seed engine with resume TIDs (no crossover override)
for sym in aion_symbols:
    resume_tid = resume_tids.get(sym)
    if resume_tid is not None:
        for thr in config.thresholds:
            engine.seed_trade_id(sym, thr, resume_tid)

# >>> NEW: Parquet pre-fill goes here <<<

# STREAM-01: fill_from_rest processes trades entirely in Rust.
rest_bars = engine.fill_from_rest()
```

From python/opendeviationbar/storage/parquet.py (TickStorage API):

```python
class TickStorage:
    def has_ticks(self, symbol, start_ts, end_ts, *, min_coverage=0.95) -> bool: ...
    def read_ticks(self, symbol, start_ts=None, end_ts=None) -> pl.DataFrame: ...
    def read_ticks_streaming(self, symbol, start_ts=None, end_ts=None, *, chunk_size=100_000) -> Iterator[pl.DataFrame]: ...
```

From python/opendeviationbar/processors/core.py:

```python
class OpenDeviationBarProcessor:
    def __init__(self, threshold_decimal_bps, *, symbol=None): ...
    def process_trades_arrow_native(self, arrow_table) -> PyArrowRecordBatch: ...
    def last_completed_bar_tid(self) -> int | None: ...
```

From python/opendeviationbar/sidecar/midnight.py (\_process_parquet_ticks, lines 847-991):
This is the EXACT pattern to follow for day-mode Parquet processing.
Key differences for Aion pre-fill:

1. No date filtering -- use TID range from resume_tids
2. Parquet files span from resume date to today
3. Filter trades by agg_trade_id > resume_tid (Parquet may contain trades before resume point)
4. Return dict of {symbol: max_completed_bar_tid} to re-seed engine
   </interfaces>

</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Parquet-first Aion gap fill with TDD</name>
  <files>python/opendeviationbar/sidecar/parquet_fill.py, python/opendeviationbar/sidecar/__init__.py, tests/test_sidecar_parquet_fill.py</files>
  <read_first>
    - python/opendeviationbar/sidecar/__init__.py (lines 269-370 for Aion startup path, lines 508-543 for fill_from_rest call)
    - python/opendeviationbar/sidecar/midnight.py (lines 847-991 for _process_parquet_ticks pattern)
    - python/opendeviationbar/storage/parquet.py (TickStorage class API)
    - python/opendeviationbar/sidecar/delete_strategy.py (module structure pattern)
  </read_first>
  <behavior>
    - Test 1: When Parquet has ticks covering the gap range, _parquet_prefill_aion returns {symbol: last_completed_bar_tid}
    - Test 2: When Parquet has no ticks for a symbol, that symbol is omitted from results (falls through to REST)
    - Test 3: When Parquet is stale (>10 min old), symbol is skipped
    - Test 4: When processing raises exception, function returns empty dict (non-fatal)
    - Test 5: Trades with agg_trade_id <= resume_tid are filtered out (only process gap trades)
    - Test 6: After _parquet_prefill_aion, engine seed TIDs are updated in __init__.py call site
  </behavior>
  <action>
    **RED phase**: Create tests/test_sidecar_parquet_fill.py with 6 tests for _parquet_prefill_aion().
    Mock TickStorage, OpenDeviationBarProcessor, and OpenDeviationBarCache. Tests must fail initially.

    **GREEN phase**: Create python/opendeviationbar/sidecar/parquet_fill.py:

    ```python
    def _parquet_prefill_aion(
        ch_cache,          # OpenDeviationBarCache (shared ALLOC-01 instance)
        aion_symbols,      # list[str]
        resume_tids,       # dict[str, int | None]
        thresholds,        # list[int]
    ) -> dict[str, int]:
    ```

    Implementation (follow _process_parquet_ticks pattern from midnight.py):
    1. Create TickStorage() instance
    2. For each aion_symbol with resume_tid not None:
       a. Convert resume_tid to approximate date (query CH for timestamp, or use current date - 7 days as safe window)
       b. Check has_ticks() for date range from resume date to today
       c. If no ticks available, skip symbol (log info, REST will cover it)
       d. Use read_ticks_streaming() for memory-efficient chunked reading
       e. Filter each chunk: keep only trades where agg_trade_id > resume_tid
       f. Create OpenDeviationBarProcessor per threshold, process trades via process_trades_arrow_native()
       g. Write bars to CH via ch_cache.store_bars_batch() with ouroboros_mode="aion"
       h. Track last_completed_bar_tid per threshold, take min across thresholds as seed TID
    3. Return {symbol: min_last_completed_bar_tid} for symbols that produced bars
    4. Wrap entire function in try/except -- return empty dict on any failure (non-fatal)
    5. Log telemetry: symbols processed, bars written, time elapsed, Parquet vs REST coverage

    Use TYPE_CHECKING guard for OpenDeviationBarCache import (same pattern as delete_strategy.py).
    Use staleness_threshold_s = 600 (10 min, same as _process_parquet_ticks).

    **Wire into __init__.py**: After Step 4 (seed engine with resume TIDs, line ~370) and BEFORE fill_from_rest (line ~513):
    ```python
    # Step 4.5: Parquet pre-fill (RESUME-02) -- read local tick cache before REST
    try:
        from .parquet_fill import _parquet_prefill_aion
        parquet_seeds = _parquet_prefill_aion(
            ch_cache, aion_symbols, resume_tids, config.thresholds,
        )
        if parquet_seeds:
            for sym, seed_tid in parquet_seeds.items():
                for thr in config.thresholds:
                    engine.seed_trade_id(sym, thr, seed_tid)
            logger.info(
                "parquet-prefill: advanced %d/%d aion symbols: %s",
                len(parquet_seeds), len(aion_symbols),
                ", ".join(f"{s}@{tid}" for s, tid in sorted(parquet_seeds.items())),
            )
    except Exception:
        logger.warning("parquet-prefill: failed (non-fatal)", exc_info=True)
    ```

    Add import to __init__.py's import block:
    ```python
    from .parquet_fill import (
        _parquet_prefill_aion as _parquet_prefill_aion,
    )
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_sidecar_parquet_fill.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "def _parquet_prefill_aion" python/opendeviationbar/sidecar/parquet_fill.py
    - grep -q "_parquet_prefill_aion" python/opendeviationbar/sidecar/__init__.py
    - grep -q "store_bars_batch" python/opendeviationbar/sidecar/parquet_fill.py
    - grep -q "read_ticks_streaming\|read_ticks" python/opendeviationbar/sidecar/parquet_fill.py
    - grep -q "agg_trade_id.*resume_tid\|resume_tid.*agg_trade_id" python/opendeviationbar/sidecar/parquet_fill.py
    - grep -q "last_completed_bar_tid" python/opendeviationbar/sidecar/parquet_fill.py
    - grep -q "def test_" tests/test_sidecar_parquet_fill.py
    - uv run --python 3.13 pytest tests/test_sidecar_parquet_fill.py -x passes
  </acceptance_criteria>
  <done>
    _parquet_prefill_aion() processes Parquet ticks for Aion symbols between resume_tid and now,
    writes bars to CH, and returns updated seed TIDs. Call site in __init__.py advances engine
    seeds so fill_from_rest only covers uncached TID ranges. All tests pass.
  </done>
</task>

</tasks>

<verification>
- tests/test_sidecar_parquet_fill.py passes with all 6+ tests
- grep confirms _parquet_prefill_aion is called in __init__.py between seed step and fill_from_rest
- grep confirms TickStorage usage in parquet_fill.py
- grep confirms agg_trade_id filtering (only process trades beyond resume_tid)
- ruff check passes on new files
</verification>

<success_criteria>

1. New parquet_fill.py module with_parquet_prefill_aion() function
2. Call site in **init**.py Aion startup path (after seed, before fill_from_rest)
3. Parquet ticks processed through OpenDeviationBarProcessor per threshold
4. Bars written to CH with ouroboros_mode="aion"
5. Engine seed TIDs advanced to last_completed_bar_tid from Parquet processing
6. Non-fatal: any failure falls through to full REST fill
7. All unit tests pass
   </success_criteria>

<output>
After completion, create `.planning/phases/30-parquet-first-gap-fill-junction-telemetry/30-01-SUMMARY.md`
</output>
