---
id: SEED-001
title: Sidecar Parquet-First Catchup
planted: 2026-03-24
trigger: next milestone after v2.0 stabilization
status: dormant
tags: [sidecar, parquet, REST, rate-limit, startup-performance]
dependencies:
  - seeder today-seeding (shipped 2026-03-24)
  - incremental REST resume from last agg_trade_id (shipped 2026-03-24)
  - stable production after v2.0 microsecond migration
---

# SEED-001: Sidecar Parquet-First Catchup

## The Idea

The sidecar's `fill_from_rest` on startup independently fetches trades from Binance REST API — the same trades the seeder already downloaded to local Parquet every 5 minutes. The sidecar should read the seeder's Parquet tick cache first, then REST only for the narrow gap between Parquet's last trade and the WS live stream.

## Why It Matters

- **Eliminates redundant REST calls**: seeder already has today's trades in `~/.cache/opendeviationbar/ticks/{SYMBOL}/{today}.parquet`
- **Faster sidecar restart**: local NVMe read (~1ms) vs REST API pagination (~30s for 16 symbols)
- **Lower rate limit consumption**: every sidecar restart currently burns ~500 REST weight; Parquet-first reduces to ~50
- **Unified data pipeline**: kintsugi already uses Parquet-first (Phase 2 migration); sidecar should match

## Target Data Flow

```
Seeder (5min timer) → REST API → Parquet ticks (incremental, today included)
                                        ↓
Sidecar (on restart) → read Parquet → REST from last_id+1 → WS junction
                                        ↓
Kintsugi (yesterday+) → read Parquet → bar construction → ClickHouse
```

## Implementation Sketch

1. Sidecar startup: for each symbol, read `max(agg_trade_id)` from today's Parquet file
2. If Parquet exists and is fresh (< 10 min old): load trades from Parquet, REST only from `last_id + 1` to fill gap to WS buffer
3. If Parquet stale or missing: fall back to current `fill_from_rest` behavior (no regression)
4. The WS buffer has been accumulating since step 1 — junction point is where REST trades meet WS trades

## When to Surface

- When planning the next milestone after v2.0 is stable in production
- When rate limit pressure becomes a concern (seeder + sidecar competing for REST budget)
- When sidecar restart time needs optimization (currently ~30s, target <5s)

## Breadcrumbs

- Seeder today-seeding: commit `a618336` (no T-1 cutoff)
- Incremental REST resume: commit `ef09de5` (resume from last agg_trade_id)
- Seeder auto-detect ms/us: commit `4424654` (Arrow path auto-detect)
- Kintsugi Parquet-first: Phase 2 of v2.0 milestone (KPAQ-01..03)
- Sidecar fill_from_rest: `python/opendeviationbar/sidecar/midnight.py`
- TickStorage read API: `python/opendeviationbar/storage/parquet.py` → `read_ticks()`
