# Phase 5: Seeder REST Integration & Rate Limit Coordination - Context

**Gathered:** 2026-03-22
**Status:** Ready for planning

<domain>
## Phase Boundary

Add REST API fallback to the tick cache seeder for recent days not yet available on Binance Vision, unify REST API rate limit coordination across all three consumers (sidecar, kintsugi, seeder) to prevent 429 errors and starvation, and harden Parquet integrity with completeness validation.

</domain>

<decisions>
## Implementation Decisions

### REST fallback strategy

- **D-01:** Seeder tries Vision CSV first (existing behavior), then falls back to REST API for days where Vision returns 404 (typically yesterday, since Vision publishes ~8h delayed)
- **D-02:** REST fallback uses the same `aggTrades` endpoint and Ariadne-style pagination as kintsugi (`fromId` pagination, 1000 trades/request) via existing Rust `fetch_aggtrades_parallel()` exposed through PyO3
- **D-03:** Seeder REST path writes to Parquet tick cache ONLY (no ClickHouse writes) — respects writer ownership model. The seeder's job is to pre-populate ticks so kintsugi's 64-worker fast Parquet path stays active even for recent days
- **D-04:** **Never seed today** — today is sidecar's domain per writer ownership model. REST seeding targets yesterday only (where Vision may be delayed). Older days wait for Vision

### Overlap prevention

- **D-05:** Add `Conflicts=opendeviationbar-seeder.service` to systemd service file (same pattern as kintsugi). Prevents concurrent seeder runs when previous run exceeds 5 minutes
- **D-06:** Future: when seeder entry point moves to Rust binary, add `flock`-based overlap guard with progress reporting to heartbeat

### Parquet completeness validation

- **D-07:** After every Parquet write (Vision or REST), validate trade-ID continuity: `trade_id[i+1] == trade_id[i] + 1` for all consecutive rows. Reject and delete files that fail
- **D-08:** REST-sourced Parquet files carry metadata flag `source=REST, complete=false`. When Vision later becomes available for that day, seeder overwrites with Vision-sourced file (self-healing)
- **D-09:** Route all Parquet writes through `storage.write_ticks()` (or Rust equivalent) which deduplicates on `agg_trade_id` before writing — seeder currently bypasses this
- **D-10:** Periodic Vision vs REST oracle check (weekly heartbeat): compare trade counts for random 5 days across 3 symbols. Alert on mismatch >0.1%

### Rate limit coordination mechanism

- **D-11:** Shared rate limit state via a lightweight file-based mechanism (atomic JSON file in `/tmp/opendeviationbar-ratelimit.json`) that all three consumers read/write. Each consumer reports its weight consumption; each consumer reads aggregate state before acquiring budget
- **D-12:** If Redis is available on bigblack, use Redis atomic counters as an upgrade path (graceful fallback to file-based if Redis unavailable — consistent with existing Redis graceful degradation pattern)
- **D-13:** Budget allocation: sidecar 2400 weight/min (P0 — real-time), kintsugi 2000 weight/min (P1 — gap repair), seeder 800 weight/min (P2 — proactive cache), reserve 800 weight/min (monitoring/heartbeat). Total: 6000
- **D-14:** Batch rotation: seed 6 symbols per 5-min run, prioritized by volume (BTCUSDT first). Full cycle = 15 min for all 18 symbols. Each run stays well within 800 weight budget

### Rust-first architecture

- **D-15:** Rust library via PyO3 for heavy lifting (Vision download, REST fetch, CSV parse, timestamp normalization, Parquet write with validation), Python orchestrator for scheduling/configuration
- **D-16:** Expose `AdaptiveRateLimiter` to Python via PyO3 bindings so kintsugi and seeder use the same Rust rate limiter as the sidecar
- **D-17:** Kintsugi wraps each REST call with `rate_limiter.acquire(weight=1)` before fetching. On budget exhaustion, sleeps until budget refills rather than retry-storming
- **D-18:** Reuse existing Rust `fetch_aggtrades_parallel()` (already exposed as `fetch_aggtrades_parallel_py` in PyO3 bindings) for seeder REST path
- **D-19:** Rust Vision download + CSV parse already exists in `historical.rs:694-810` — expose via PyO3 to replace Python `requests` + Polars CSV read

### Seeder scheduling optimization

- **D-20:** Cache "last fully-seeded date" per symbol in state file. `_find_missing_days()` scans only from that date forward, reducing 52K → ~360 `stat()` calls per run (18 symbols × 20 recent days)
- **D-21:** Newest-first ordering preserved (matches kintsugi shard priority)

### Telemetry enhancements

- **D-22:** Heartbeat reports per-consumer REST utilization (sidecar, kintsugi, seeder) and aggregate. LOUD alert if aggregate exceeds 90% sustained for 3 consecutive checks
- **D-23:** Seeder logs REST fallback events at INFO level: `"REST fallback for {symbol} {date}: {trade_count} trades fetched in {duration}s"`
- **D-24:** New heartbeat metrics: days-behind per symbol, Vision 404 count, Parquet-path activation rate (kintsugi reports), budget utilization vs 5-min window
- **D-25:** Days-behind is the primary actionable metric — surfaces symbols falling behind before they become kintsugi bottlenecks

### Claude's Discretion

- File format and locking strategy for `/tmp/opendeviationbar-ratelimit.json`
- Exact PyO3 binding API surface for AdaptiveRateLimiter
- Whether to use `fcntl.flock()` or atomic rename for file-based coordination
- Seeder REST retry policy (stamina config)
- ZSTD compression level tuning (currently hardcoded at 3 — may want env var override)

</decisions>

<canonical_refs>

## Canonical References

**Downstream agents MUST read these before planning or implementing.**

### Seeder

- `scripts/tick_cache_seeder.py` -- Current seeder implementation (Vision-only), entry point `_seed_symbol()`, config via pydantic
- `python/opendeviationbar/config/seeder.py` -- Seeder pydantic-settings configuration
- `python/opendeviationbar/storage/CLAUDE.md` -- Tier-1 Parquet tick cache architecture, daily partitions
- `scripts/heartbeat/checks/seeder.py` -- Seeder health monitoring
- `scripts/systemd/opendeviationbar-seeder.service` -- systemd service (missing Conflicts= directive)
- `scripts/systemd/opendeviationbar-seeder.timer` -- 5-min timer

### Parquet I/O

- `python/opendeviationbar/storage/parquet_io.py` lines 142-206 -- Atomic write (fsync + rename), ZSTD-3 compression
- `python/opendeviationbar/storage/parquet_io.py` lines 43-140 -- Validation (header/footer only, NO trade-ID continuity)
- `python/opendeviationbar/storage/parquet.py` -- `write_ticks()` with dedup (seeder bypasses this)

### Rate Limiting

- `crates/opendeviationbar-providers/src/binance/rate_limiter.rs` -- AdaptiveRateLimiter: server feedback, per-process ceiling, CAS loop
- `crates/opendeviationbar-providers/src/binance/parallel_fetch.rs` -- `fetch_aggtrades_parallel()`, DEFAULT_CONCURRENCY=3, PAGE_SIZE=1000
- `crates/opendeviationbar-streaming/src/stream_manager.rs` -- StreamManager integration with rate limiter
- `crates/opendeviationbar-streaming/src/live_engine.rs` lines 1900+ -- fill_from_rest and Puck gap-fill REST usage
- `src/binance_bindings.rs` -- PyO3 `fetch_aggtrades_parallel_py` binding (already exists)

### Kintsugi & Backfill

- `python/opendeviationbar/backfill.py` -- Ariadne pagination, REST fallback path, stamina retries
- `python/opendeviationbar/kintsugi.py` line 670-689 -- `_shard_is_parquet_cached()` (Path.exists only)
- `python/opendeviationbar/kintsugi.py` line 1142-1176 -- `_repair_from_parquet_cache()` (no freshness check)
- `scripts/CLAUDE.md` -- Kintsugi architecture, writer ownership model, parallelism tuning
- `python/opendeviationbar/config/resilience.py` -- Resilience retry configuration

### Rust Vision/REST

- `crates/opendeviationbar-providers/src/binance/historical.rs` lines 694-810 -- Rust Vision download + CSV parse (not exposed to seeder)
- `crates/opendeviationbar-providers/src/binance/historical.rs` line 1448 -- µs→ms timestamp normalization

### Writer Ownership

- `CLAUDE.md` section "Writer Ownership Model (#280)" -- Sidecar owns today, kintsugi owns yesterday+older

</canonical_refs>

<code_context>

## Existing Code Insights

### Reusable Assets

- `AdaptiveRateLimiter` (Rust): Full-featured rate limiter with server feedback, ceilings, atomic operations — needs PyO3 bindings
- `fetch_aggtrades_parallel()` (Rust): Parallel REST fetch with rate limiting — already has PyO3 binding `fetch_aggtrades_parallel_py`
- `historical.rs` Vision download + CSV parse (Rust): Complete Vision pipeline — needs new PyO3 binding for seeder use
- `_download_vision_day()` in seeder: Python Vision download — to be replaced by Rust equivalent
- `storage.write_ticks()`: Parquet write with dedup — seeder should route through this instead of direct `_atomic_write_parquet()`
- ZSTD-3 compression: Empirically validated best for tick data (6.50 MB for 761K trades, 5.37x compression)

### Established Patterns

- **Vision-first, REST-fallback**: `backfill.py` already does this for bar construction. Seeder extends the same pattern to tick caching
- **Pydantic-settings configuration**: Seeder config already uses `BaseSettings` — new REST params follow same pattern
- **Writer ownership model**: Seeder writes to Parquet (not ClickHouse), never seeds today (sidecar's domain)
- **Newest-first ordering**: Both kintsugi shards and seeder process newest days first
- **systemd Conflicts=**: Kintsugi already uses this pattern for overlap prevention

### Integration Points

- `src/lib.rs` PyO3 module: New `PyAdaptiveRateLimiter` class + Vision download binding registered here
- `scripts/tick_cache_seeder.py`: REST fallback + batch rotation + missing-days cache + write_ticks routing
- `scripts/systemd/opendeviationbar-seeder.service`: Add `Conflicts=` directive
- `python/opendeviationbar/backfill.py`: Replace stamina-only retries with `AdaptiveRateLimiter.acquire()` + stamina
- `scripts/heartbeat/checks/seeder.py`: 4 new metrics (days-behind, Vision 404, Parquet-path rate, budget utilization)

### Blind Spots Identified & Mitigated

| Blind Spot                                    | Severity | Mitigation (D-XX)                               |
| --------------------------------------------- | -------- | ----------------------------------------------- |
| Seeder overlap (no systemd guard)             | MEDIUM   | D-05: `Conflicts=` directive                    |
| Parquet completeness (no trade-ID validation) | HIGH     | D-07: continuity check after every write        |
| Seeder bypasses `write_ticks()` dedup         | MEDIUM   | D-09: route through `write_ticks()`             |
| Partial day via REST                          | HIGH     | D-04: never seed today                          |
| REST-sourced file assumed complete            | MEDIUM   | D-08: "partial" metadata flag, Vision overwrite |
| Vision vs REST mismatch                       | LOW      | D-10: periodic oracle check                     |
| 52K stat() calls per run at 18 symbols        | LOW      | D-20: cache last-seeded date                    |
| Missing heartbeat metrics                     | MEDIUM   | D-24: 4 new metrics                             |
| No cross-process rate limit coordination      | HIGH     | D-11/D-12: file-based + Redis upgrade           |
| Budget pressure at 18 symbols                 | MEDIUM   | D-14: batch rotation (6 symbols/run)            |

</code_context>

<specifics>
## Specific Ideas

- User wants seeder to run "as frequently as possible" to cover REST API data — bounded to yesterday-only + batch rotation to control budget
- Concern about "single source of bottleneck" — rate limit coordination is the core deliverable
- "Do as much heavy lifting in Rust as possible" — Rust library via PyO3, Python orchestrator
- "Make sure Parquet file will be completely the same in format with full sequences and no missing data" — trade-ID continuity check + dedup routing
- "Help me think of all the blind spots that disaster might happen" — 10 blind spots identified and mitigated
- Vision CSV ~8h delayed is the specific gap that REST fallback fills
- All three consumers (sidecar, kintsugi, seeder) share a single Binance IP — coordination is mandatory, not optional

</specifics>

<deferred>
## Deferred Ideas

- **Full 18-symbol streaming** (eliminates kintsugi REST dependency entirely) — separate milestone, already noted in REQUIREMENTS.md v2
- **WebSocket-based tick seeding** (subscribe to aggTrade WS for real-time tick caching) — more complex, higher priority if REST budget proves insufficient
- **Multi-IP rate limiting** (proxy pool for higher aggregate budget) — only if 6000 weight/min proves insufficient for 18 symbols
- **Full Rust binary for seeder** (replace Python orchestrator entirely) — after Phase 5 proves the PyO3 approach works
- **ZSTD compression level env var** — not urgent, level 3 is already optimal

</deferred>

---

_Phase: 05-seeder-rest-ratelimit_
_Context gathered: 2026-03-22_
