# Phase 03: Parquet-First Catchup - Research

**Researched:** 2026-03-24
**Domain:** Sidecar startup optimization — Parquet tick cache integration with fill_from_rest
**Confidence:** HIGH

## Summary

SEED-001 (commit d2dacc5) is a **documentation-only seed** -- it committed `.planning/seeds/SEED-001-sidecar-parquet-first-catchup.md` with a design sketch but **zero implementation code**. The sidecar's `fill_from_rest` path currently fetches all trades from midnight crossover TID via Binance REST API with no Parquet cache awareness. All three PQFIRST requirements are unimplemented.

The implementation is straightforward: the Python startup sequence in `sidecar/__init__.py` already has a clear integration point between midnight preflight (which determines the crossover TID) and `engine.fill_from_rest()` (which paginates REST from that TID). The new code reads today's Parquet file per symbol, extracts `max(agg_trade_id)`, and if fresh (<10 min), calls `engine.seed_trade_id()` to advance the REST start point past what Parquet already covers. The Rust `fill_from_rest` then fetches only the residual gap.

**Primary recommendation:** Insert Parquet-read logic in `sidecar/__init__.py` between midnight preflight seed override and `engine.fill_from_rest()`. Use `TickStorage.read_ticks()` to get today's max trade ID, then `engine.seed_trade_id()` to advance the fill start point. No Rust changes needed.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices are at Claude's discretion (pure infrastructure phase).

### Claude's Discretion

All implementation choices are at Claude's discretion. Key technical constraints:

- SEED-001 already committed (sidecar reads Parquet cache before REST) -- check if implementation exists
- `TickStorage` class in `python/opendeviationbar/storage/parquet.py` provides `has_ticks()` and `read_ticks()`
- Parquet tick cache path: `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet`
- Seeder populates Parquet ticks every 5 min (Binance) / 15 min (Exness)
- `fill_from_rest` in `sidecar/midnight.py` is the integration point
- Writer ownership: sidecar owns today, kintsugi owns yesterday+older

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID         | Description                                                                                                  | Research Support                                                                                                                                                |
| ---------- | ------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| PQFIRST-01 | Sidecar startup reads `max(agg_trade_id)` from today's Parquet tick cache per symbol before hitting REST API | TickStorage.read_ticks() returns Polars DataFrame with `agg_trade_id` column; today's file at `{SYMBOL}/{today}.parquet`; seeder writes every 5 min             |
| PQFIRST-02 | REST API called only for gap between Parquet's last trade and WS buffer start (incremental)                  | `engine.seed_trade_id(symbol, threshold, parquet_max_tid)` advances Rust fill start point; fill_from_rest paginates from `max(checkpoint_tid, ch_seed_tid) + 1` |
| PQFIRST-03 | Falls back to current `fill_from_rest` if Parquet stale (>10 min) or missing (no regression)                 | If Parquet file missing or mtime > 10 min old, skip seed override; existing midnight preflight seeds remain unchanged                                           |

</phase_requirements>

## Architecture Patterns

### Current Startup Flow (no Parquet)

```
1. _run_midnight_preflight()  -> crossover_tid per symbol
2. engine.seed_trade_id(sym, thr, crossover_tid - 1)  per (sym, thr)
3. DELETE today (clean slate)
4. engine.fill_from_rest()    -> REST from seed+1 to latest
5. _sync_flush_rest_bars()    -> bars to CH
```

### Target Flow (Parquet-first)

```
1. _run_midnight_preflight()  -> crossover_tid per symbol
2. engine.seed_trade_id(sym, thr, crossover_tid - 1)  per (sym, thr)
3. DELETE today (clean slate)
4. ** NEW: _read_parquet_max_tids()  -> {symbol: max_tid} from today's Parquet **
5. ** NEW: engine.seed_trade_id(sym, thr, parquet_max_tid) per (sym, thr) where fresh **
6. engine.fill_from_rest()    -> REST from max(preflight_seed, parquet_seed)+1 to latest
7. _sync_flush_rest_bars()    -> bars to CH
```

### Key Insight: Parquet Has Trades, Not Bars

The Parquet tick cache stores raw aggTrades, not bars. The sidecar does not need to process Parquet trades into bars -- it only needs `max(agg_trade_id)` to tell the Rust engine "start REST from here instead of midnight crossover." The Rust `fill_from_rest` then fetches a much smaller range and processes trades through its own processors.

**Why this works:** `engine.seed_trade_id(symbol, threshold, tid)` sets the gap detector seed. `fill_from_rest` starts REST pagination from `max(checkpoint_tid, ch_seed_tid) + 1`. By calling `seed_trade_id` with Parquet's max TID (which is >= crossover TID), we advance the REST start point.

### Integration Point (Confirmed)

File: `python/opendeviationbar/sidecar/__init__.py`, lines 364-377

The `engine.fill_from_rest()` call (line 368) is preceded by midnight preflight + DELETE today. The new Parquet read slots in between DELETE today (line 358) and `engine.fill_from_rest()` (line 368).

### Seed Override Ordering

The seed override must happen AFTER midnight preflight (which sets crossover_tid - 1) but BEFORE fill_from_rest. The Rust engine takes `max(checkpoint_tid, ch_seed_tid)` as the start point. Since checkpoints are skipped in fill_from_rest mode (line 226-233) and CH seeding is also skipped (line 243-244), the only seed is from `_run_midnight_preflight()`. Parquet override calls `seed_trade_id` again, which overwrites the preflight seed if parquet_max_tid > crossover_tid - 1.

**Safety:** If Parquet max TID < crossover TID (e.g., stale cache from yesterday), the override is still safe -- fill_from_rest would start from crossover and re-fetch some trades that overlap with yesterday. But the staleness check (>10 min) already guards this.

### Recommended Project Structure

No new files needed. All changes in existing files:

```
python/opendeviationbar/sidecar/
├── __init__.py         # NEW: _read_parquet_max_tids() call + seed override
├── midnight.py         # NEW: _read_parquet_max_tids() function
└── (rest unchanged)
```

### Pattern: Parquet Max TID Extraction

```python
# Source: SEED-001 design + TickStorage API (storage/CLAUDE.md)
def _read_parquet_max_tids(symbols: list[str]) -> dict[str, int]:
    """Read max(agg_trade_id) from today's Parquet tick cache per symbol.

    Returns dict of {symbol: max_trade_id} for symbols with fresh Parquet data.
    Symbols with missing or stale (>10 min) Parquet files are omitted.
    """
    from datetime import datetime, UTC
    from pathlib import Path
    import time
    from opendeviationbar.storage.parquet import TickStorage

    storage = TickStorage()
    today_str = datetime.now(UTC).strftime("%Y-%m-%d")
    staleness_threshold_s = 600  # 10 minutes
    result = {}

    for symbol in symbols:
        parquet_path = storage._get_parquet_path(symbol, today_str)
        if not parquet_path.exists():
            continue

        # Freshness check: file mtime vs now
        mtime = parquet_path.stat().st_mtime
        age_s = time.time() - mtime
        if age_s > staleness_threshold_s:
            logger.info("parquet-first: %s stale (%.0fs old > %ds), skipping",
                       symbol, age_s, staleness_threshold_s)
            continue

        # Read max agg_trade_id from Parquet (lazy scan, minimal I/O)
        import polars as pl
        try:
            df = pl.scan_parquet(parquet_path).select(
                pl.col("agg_trade_id").max()
            ).collect()
            max_tid = df.item()
            if max_tid and max_tid > 0:
                result[symbol] = int(max_tid)
                logger.info("parquet-first: %s max_tid=%d (age=%.0fs)",
                           symbol, max_tid, age_s)
        except Exception:
            logger.debug("parquet-first: failed to read %s", symbol, exc_info=True)

    return result
```

### Anti-Patterns to Avoid

- **Do not read full Parquet into memory:** Only need `max(agg_trade_id)`. Use `pl.scan_parquet().select(max()).collect()` -- lazy scan reads only the `agg_trade_id` column and computes max without materializing all rows.
- **Do not process Parquet trades through Rust processors:** The sidecar's fill_from_rest already processes trades. Parquet is only a hint for where REST should start.
- **Do not feed Parquet trades to the engine:** The Rust engine's `fill_from_rest` fetches and processes trades internally. Parquet just advances the start cursor.
- **Do not skip midnight preflight seeds:** Parquet max TID is an additional override on top of preflight. If Parquet is stale/missing, preflight seeds remain -- zero regression.

## Don't Hand-Roll

| Problem              | Don't Build                 | Use Instead                                             | Why                                                                               |
| -------------------- | --------------------------- | ------------------------------------------------------- | --------------------------------------------------------------------------------- |
| Parquet file reading | Custom Arrow/Parquet reader | `TickStorage._get_parquet_path()` + `pl.scan_parquet()` | Already handles symlinks, daily partitioning, path resolution                     |
| Staleness detection  | Custom mtime tracker        | `Path.stat().st_mtime`                                  | OS-level mtime is authoritative -- seeder atomically renames `.tmp` to `.parquet` |
| Trade ID seeding     | Custom REST offset logic    | `engine.seed_trade_id()`                                | Rust engine already has the plumbing to accept external seed overrides            |
| Today's date key     | Custom date formatting      | `datetime.now(UTC).strftime("%Y-%m-%d")`                | Consistent with TickStorage's `_timestamp_to_day()`                               |

## Common Pitfalls

### Pitfall 1: Symbol Name Mismatch (BTCUSDT vs BINANCE_SPOT_BTCUSDT)

**What goes wrong:** Seeder uses bare symbol names for spot ("BTCUSDT"), but checkpoint system uses prefixed names ("BINANCE_SPOT_BTCUSDT"). Symlinks bridge the gap.
**Why it happens:** Two naming conventions coexist -- seeder writes to `BTCUSDT/`, checkpoint code uses `BINANCE_SPOT_BTCUSDT/`.
**How to avoid:** Use the bare symbol name from `config.symbols` (same as sidecar uses everywhere). `TickStorage._get_symbol_dir()` resolves symlinks automatically via `ticks_dir / symbol`.
**Warning signs:** FileNotFoundError when reading Parquet, empty results for symbols that definitely have data.

### Pitfall 2: Parquet Timestamps Are Microseconds, REST Uses Milliseconds

**What goes wrong:** Parquet `agg_trade_id` is an integer (no unit issue), but if anyone adds timestamp-based filtering, the ms/us mismatch bites.
**Why it happens:** Seeder normalizes to microseconds since 2025-01-01 Vision format change. REST always returns milliseconds.
**How to avoid:** Only use `agg_trade_id` (integer, unit-free) for the Parquet-first optimization. Do not use timestamp-based filtering.
**Warning signs:** Trade IDs look correct but timestamp filtering returns wrong ranges.

### Pitfall 3: Parquet Max TID < Crossover TID

**What goes wrong:** If the seeder hasn't run since before midnight, today's Parquet file doesn't exist yet. The previous day's file has a max TID that's before today's crossover.
**Why it happens:** Seeder runs every 5 min. After a sidecar restart at, say, 00:02 UTC, the seeder may not have written today's file yet.
**How to avoid:** The staleness check (>10 min) already handles this -- if no today file exists, `parquet_path.exists()` returns False and the symbol is skipped. The midnight preflight seed remains.
**Warning signs:** Log shows "parquet-first: {symbol} not found" for all symbols shortly after midnight.

### Pitfall 4: Race Between Seeder Write and Sidecar Read

**What goes wrong:** Sidecar reads Parquet while seeder is writing (`.parquet.tmp` rename).
**Why it happens:** Seeder uses atomic rename (write to `.tmp`, then `os.rename()`). The read sees either the old complete file or the new complete file -- never a partial write.
**How to avoid:** No action needed. Atomic rename guarantees read consistency. At worst, sidecar sees slightly stale data (seeder's previous 5-min run), which is fine.
**Warning signs:** None expected -- this is safe by construction.

### Pitfall 5: Parquet Seed > WS Buffer Start

**What goes wrong:** If Parquet max TID is very recent (seeder just ran), it might exceed the WS buffer's first accumulated trade, creating an overlap.
**Why it happens:** WS buffer started accumulating when the engine was created (step 1). Parquet was written by the seeder independently.
**How to avoid:** This is NOT a problem. The Rust `fill_from_rest` fetches from seed+1 to latest REST TID. WS buffer starts from whenever WS connected. The junction between REST fill and WS buffer is handled by the existing committed_floors dedup mechanism (L8). Overlapping trade IDs are deduplicated.
**Warning signs:** None -- dedup handles this transparently.

## Code Examples

### Integration in sidecar/**init**.py (between DELETE and fill_from_rest)

```python
# Source: Derived from SEED-001 design + existing sidecar startup sequence

# After DELETE today (clean slate) and before fill_from_rest:
# Parquet-first: advance REST start point using seeder's cached trades
try:
    parquet_tids = _read_parquet_max_tids(config.symbols)
    if parquet_tids:
        for symbol, max_tid in parquet_tids.items():
            for threshold in config.thresholds:
                engine.seed_trade_id(symbol, threshold, max_tid)
        logger.info(
            "parquet-first: advanced seeds for %d/%d symbols: %s",
            len(parquet_tids),
            len(config.symbols),
            ", ".join(f"{s}@{tid}" for s, tid in sorted(parquet_tids.items())),
        )
except Exception:
    logger.warning("parquet-first: failed (non-fatal, falling back to preflight seeds)",
                  exc_info=True)
```

### Lazy Parquet Max TID (minimal I/O)

```python
# Source: Polars scan_parquet API + TickStorage path resolution

import polars as pl

# Lazy scan: reads only agg_trade_id column statistics from Parquet metadata
# For a 6.5 MB daily file (761K trades), this takes <1ms
max_tid = (
    pl.scan_parquet(parquet_path)
    .select(pl.col("agg_trade_id").max())
    .collect()
    .item()
)
```

### Staleness Check

```python
# Source: SEED-001 design spec (10 min threshold)

import time
from pathlib import Path

parquet_path = storage._get_parquet_path(symbol, today_str)
if not parquet_path.exists():
    continue  # No Parquet for today -- use preflight seed

age_s = time.time() - parquet_path.stat().st_mtime
if age_s > 600:  # 10 minutes
    logger.info("parquet-first: %s stale (%.0fs), skipping", symbol, age_s)
    continue  # Stale Parquet -- use preflight seed
```

## Validation Architecture

### Test Framework

| Property           | Value                                                     |
| ------------------ | --------------------------------------------------------- |
| Framework          | pytest                                                    |
| Config file        | `pyproject.toml` ([tool.pytest.ini_options])              |
| Quick run command  | `uv run --python 3.13 pytest tests/test_sidecar.py -x -q` |
| Full suite command | `mise run test-py`                                        |

### Phase Requirements to Test Map

| Req ID     | Behavior                                                | Test Type | Automated Command                                                                                   | File Exists? |
| ---------- | ------------------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------- | ------------ |
| PQFIRST-01 | Reads max(agg_trade_id) from today's Parquet per symbol | unit      | `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py::test_read_parquet_max_tids -x`    | Wave 0       |
| PQFIRST-02 | REST called only for Parquet-to-WS gap (seed advanced)  | unit      | `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py::test_seed_advanced_by_parquet -x` | Wave 0       |
| PQFIRST-03 | Falls back when Parquet stale/missing                   | unit      | `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py::test_fallback_stale_parquet -x`   | Wave 0       |

### Sampling Rate

- **Per task commit:** `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py -x -q`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_parquet_first_catchup.py` -- covers PQFIRST-01, PQFIRST-02, PQFIRST-03
- [ ] Test fixtures: mock TickStorage with temp Parquet files, mock engine with seed_trade_id tracking

## State of the Art

| Old Approach                      | Current Approach                  | When Changed       | Impact                                            |
| --------------------------------- | --------------------------------- | ------------------ | ------------------------------------------------- |
| Full REST from midnight crossover | Full REST from midnight crossover | Current (v13.51.0) | ~500 REST weight per restart, ~30s for 26 symbols |
| n/a                               | Parquet-first (this phase)        | Target             | ~50 REST weight, <5s startup                      |

## Open Questions

1. **Forex symbols (Exness) have no agg_trade_id**
   - What we know: Exness Parquet stores bid/ask quotes, not aggTrades. No `agg_trade_id` column.
   - What's unclear: Whether forex symbols go through fill_from_rest at all.
   - Recommendation: Filter to Binance symbols only (same as midnight preflight already does on line 854: `entries[s].exchange == "binance"`). Exness symbols have no REST fill path.

2. **Polars scan_parquet metadata-only max**
   - What we know: Polars can read Parquet column statistics from metadata without scanning rows.
   - What's unclear: Whether `pl.scan_parquet().select(max()).collect()` actually uses Parquet statistics or scans all rows.
   - Recommendation: Either way, for a 6.5 MB file this is <10ms. Not worth optimizing further. If Polars does use stats, it's <1ms.

## Sources

### Primary (HIGH confidence)

- `python/opendeviationbar/sidecar/__init__.py` lines 200-397 -- current startup sequence, seed override flow
- `crates/opendeviationbar-streaming/src/live_engine/mod.rs` lines 227-420 -- Rust fill_from_rest: reads `max(checkpoint_tid, ch_seed_tid)` then paginates REST
- `python/opendeviationbar/storage/parquet.py` -- TickStorage API, daily file layout
- `scripts/tick_cache_seeder.py` -- seeder writes to bare symbol dirs, 5-min timer
- `.planning/seeds/SEED-001-sidecar-parquet-first-catchup.md` -- design spec (documentation only, no code)

### Secondary (MEDIUM confidence)

- Storage CLAUDE.md -- symlink naming convention (BINANCE_SPOT_BTCUSDT -> BTCUSDT)

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - no new libraries needed, all existing APIs
- Architecture: HIGH - clear integration point confirmed by reading actual code
- Pitfalls: HIGH - naming conventions, timestamp units, and race conditions verified against codebase

**Research date:** 2026-03-24
**Valid until:** 2026-04-24 (stable infrastructure, no external dependencies)
