# Phase 5: Seeder REST Integration & Rate Limit Coordination - Research

**Researched:** 2026-03-22
**Domain:** Cross-process REST API rate limiting, Vision/REST tick seeding, Parquet integrity
**Confidence:** HIGH

## Summary

Phase 5 adds REST API fallback to the tick cache seeder for days not yet available on Binance Vision (~8h delay), unifies REST API rate limit coordination across three consumers (sidecar, kintsugi, seeder), and hardens Parquet tick files with trade-ID continuity validation. The existing codebase already has most building blocks: `AdaptiveRateLimiter` in Rust with per-process ceiling support, `fetch_aggtrades_parallel` with PyO3 bindings, `write_ticks()` with dedup, and the per-process singleton rate limiter pattern in `binance_bindings.rs`.

The primary new work is: (1) cross-process rate limit coordination via file-based state sharing, (2) REST fallback path in the seeder, (3) trade-ID continuity validation in Parquet I/O, (4) heartbeat telemetry for aggregate REST utilization, and (5) exposing `AdaptiveRateLimiter` status to Python for cross-process budget reporting.

**Primary recommendation:** Build the file-based rate limit coordination first (D-11/D-12), then add REST fallback to seeder, then Parquet validation, then heartbeat metrics. Each is independently testable and deployable.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

- **D-01:** Vision-first, REST fallback for 404 days (yesterday typically)
- **D-02:** REST uses same aggTrades endpoint + Ariadne-style fromId pagination via existing Rust `fetch_aggtrades_parallel()`
- **D-03:** Seeder REST writes Parquet ticks ONLY (no ClickHouse), respects writer ownership
- **D-04:** Never seed today — yesterday only for REST, older days wait for Vision
- **D-05:** Add `Conflicts=opendeviationbar-seeder.service` to systemd service
- **D-06:** Future: flock-based overlap guard when seeder moves to Rust binary
- **D-07:** Trade-ID continuity validation after every Parquet write: `trade_id[i+1] == trade_id[i] + 1`
- **D-08:** REST-sourced Parquet files carry `source=REST, complete=false` metadata; Vision overwrites when available
- **D-09:** Route all Parquet writes through `storage.write_ticks()` with dedup
- **D-10:** Weekly Vision vs REST oracle check (5 random days, 3 symbols, >0.1% mismatch = alert)
- **D-11:** File-based rate limit coordination via `/tmp/opendeviationbar-ratelimit.json`
- **D-12:** Redis upgrade path (graceful fallback to file if unavailable)
- **D-13:** Budget: sidecar 2400, kintsugi 2000, seeder 800, reserve 800 (total 6000 weight/min)
- **D-14:** Batch rotation: 6 symbols per 5-min run, BTCUSDT first
- **D-15:** Rust library via PyO3 for heavy lifting, Python orchestrator for scheduling
- **D-16:** Expose AdaptiveRateLimiter to Python via PyO3
- **D-17:** Kintsugi wraps REST calls with rate_limiter.acquire(weight=1)
- **D-18:** Reuse fetch_aggtrades_parallel (already has PyO3 binding)
- **D-19:** Expose Rust Vision download + CSV parse via PyO3 (replace Python requests)
- **D-20:** Cache "last fully-seeded date" per symbol to reduce stat() calls
- **D-21:** Newest-first ordering preserved
- **D-22:** Heartbeat per-consumer REST utilization + aggregate, LOUD alert at 90% sustained
- **D-23:** Seeder REST fallback events logged at INFO
- **D-24:** New heartbeat metrics: days-behind, Vision 404 count, Parquet-path rate, budget utilization
- **D-25:** Days-behind is primary actionable metric

### Claude's Discretion

- File format and locking strategy for `/tmp/opendeviationbar-ratelimit.json`
- Exact PyO3 binding API surface for AdaptiveRateLimiter
- Whether to use `fcntl.flock()` or atomic rename for file-based coordination
- Seeder REST retry policy (stamina config)
- ZSTD compression level tuning

### Deferred Ideas (OUT OF SCOPE)

- Full 18-symbol streaming (eliminates kintsugi REST dependency)
- WebSocket-based tick seeding
- Multi-IP rate limiting (proxy pool)
- Full Rust binary for seeder
- ZSTD compression level env var
  </user_constraints>

<phase_requirements>

## Phase Requirements

| ID        | Description                                                                                                  | Research Support                                                                                                                                                                            |
| --------- | ------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SEEDER-01 | Seeder attempts REST API fallback for days where Vision CSV is not yet available                             | Existing `fetch_aggtrades_parallel` PyO3 binding + Vision 404 detection in `_download_vision_day()`. REST fallback targets yesterday only (D-04).                                           |
| SEEDER-02 | All three REST consumers share rate limit state via common mechanism                                         | File-based `/tmp/opendeviationbar-ratelimit.json` with atomic rename (D-11). Existing per-process `AdaptiveRateLimiter` singleton in `binance_bindings.rs` provides the intra-process side. |
| SEEDER-03 | Total REST weight stays below 80% of 6000 weight/min IP-wide budget                                          | Budget allocation D-13 (2400+2000+800+800=6000). AdaptiveRateLimiter already has `safety_margin=0.20` (usable=4800). Per-process ceilings enforce allocation.                               |
| SEEDER-04 | Seeder REST path respects writer ownership: Parquet only, no ClickHouse, only days kintsugi hasn't processed | Seeder writes via `storage.write_ticks()` (D-09), never today (D-04). Kintsugi's today-guard is absolute (#280).                                                                            |

</phase_requirements>

## Project Constraints (from CLAUDE.md)

- **Python 3.13 ONLY** -- all `uv run` commands use `--python 3.13`
- **Local-First CI/CD** -- all quality gates via `mise run check-full`, never GitHub Actions for testing
- **Rust-first** -- heavy lifting in Rust via PyO3, Python handles I/O orchestration only
- **Spot-only** -- all Binance data uses spot market
- **Writer ownership** -- sidecar owns today, kintsugi owns yesterday+older
- **DST-correct** -- use `zoneinfo` with IANA timezone names, never fixed UTC hours
- **Thread-local connection reuse** -- never create `OpenDeviationBarCache()` in a loop

## Standard Stack

### Core (Already in Project)

| Library                           | Version        | Purpose                                        | Why Standard                                                    |
| --------------------------------- | -------------- | ---------------------------------------------- | --------------------------------------------------------------- |
| `AdaptiveRateLimiter` (Rust)      | In-tree        | Per-process rate limiting with server feedback | Already handles CAS acquire, ceiling, window reset, header sync |
| `fetch_aggtrades_parallel` (Rust) | In-tree        | Parallel REST aggTrades with early termination | Already has PyO3 binding (`fetch_aggtrades_parallel_py`)        |
| `TickStorage.write_ticks()`       | In-tree        | Parquet write with dedup on `agg_trade_id`     | Anti-join dedup, atomic write, timestamp normalization          |
| `pydantic-settings`               | Already pinned | Configuration via env vars                     | Project convention, `SeederConfig` already uses it              |
| `stamina`                         | Already pinned | Retry with backoff                             | Project convention for Python retries                           |
| `polars`                          | Already pinned | DataFrame operations                           | Project standard, used throughout seeder                        |

### Supporting (New)

| Library    | Version | Purpose                           | When to Use                                                          |
| ---------- | ------- | --------------------------------- | -------------------------------------------------------------------- |
| `fcntl`    | stdlib  | File locking for rate limit state | Cross-process coordination of `/tmp/opendeviationbar-ratelimit.json` |
| `json`     | stdlib  | Rate limit state serialization    | Simple, human-readable, debuggable                                   |
| `tempfile` | stdlib  | Atomic file writes                | Rate limit state file update via atomic rename                       |

### Alternatives Considered

| Instead of                   | Could Use            | Tradeoff                                                                                        |
| ---------------------------- | -------------------- | ----------------------------------------------------------------------------------------------- |
| File-based coordination      | Redis-only           | Redis may be unavailable; file-based is zero-dependency fallback                                |
| Atomic rename for state file | `fcntl.flock()`      | Atomic rename is simpler (no lock release needed), but flock allows readers to wait for writers |
| Per-process JSON file        | Shared memory (mmap) | Shared memory is faster but harder to debug; JSON is human-readable                             |

## Architecture Patterns

### Recommended Project Structure (New Files)

```
python/opendeviationbar/
├── ratelimit/                   # NEW: Cross-process rate limit coordination
│   ├── __init__.py
│   ├── file_coordinator.py      # File-based state sharing (/tmp/opendeviationbar-ratelimit.json)
│   └── redis_coordinator.py     # Redis upgrade path (graceful fallback)
scripts/
├── tick_cache_seeder.py         # MODIFIED: REST fallback + batch rotation + write_ticks routing
├── heartbeat/checks/
│   ├── seeder.py                # MODIFIED: days-behind, Vision 404, budget metrics
│   └── ratelimit.py             # NEW: aggregate REST utilization check
python/opendeviationbar/
├── config/
│   └── seeder.py                # MODIFIED: new REST config fields
├── storage/
│   └── parquet_io.py            # MODIFIED: trade-ID continuity validation
src/
└── binance_bindings.rs          # MODIFIED: PyAdaptiveRateLimiter class + Vision binding
scripts/systemd/
└── opendeviationbar-seeder.service  # MODIFIED: Conflicts= directive
```

### Pattern 1: File-Based Cross-Process Rate Limit Coordination

**What:** Each consumer writes its per-window weight usage to a shared JSON file; each consumer reads aggregate before acquiring budget.
**When to use:** Multiple processes on the same host sharing a single IP's Binance rate limit budget.

**Recommended approach: Atomic rename (not flock)**

```python
# Writer: report consumed weight
import json, os, tempfile, time
STATE_PATH = "/tmp/opendeviationbar-ratelimit.json"

def report_usage(consumer_id: str, weight_used: int, window_start: float) -> None:
    """Atomically update this consumer's weight report."""
    # Read current state (tolerate missing/corrupt file)
    try:
        with open(STATE_PATH) as f:
            state = json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        state = {"consumers": {}, "window_start": time.time()}

    # Reset if window expired (>60s)
    if time.time() - state.get("window_start", 0) > 60:
        state = {"consumers": {}, "window_start": time.time()}

    state["consumers"][consumer_id] = {
        "weight_used": weight_used,
        "updated": time.time(),
    }

    # Atomic write via tempfile + rename
    fd, tmp = tempfile.mkstemp(dir="/tmp", prefix=".ratelimit_", suffix=".json")
    try:
        os.write(fd, json.dumps(state).encode())
        os.fsync(fd)
        os.close(fd)
        os.rename(tmp, STATE_PATH)
    except Exception:
        os.close(fd)
        os.unlink(tmp)
        raise

def get_aggregate_usage() -> int:
    """Read total weight across all consumers in current window."""
    try:
        with open(STATE_PATH) as f:
            state = json.load(f)
    except (FileNotFoundError, json.JSONDecodeError):
        return 0
    # Only count entries from current window (stale entries expire)
    now = time.time()
    window_start = state.get("window_start", 0)
    if now - window_start > 60:
        return 0  # Window expired, all counts reset
    return sum(
        c["weight_used"]
        for c in state.get("consumers", {}).values()
        if now - c.get("updated", 0) < 65  # 5s grace for clock skew
    )
```

**Why atomic rename over flock:**

1. Readers never block -- they read whatever file exists (may be slightly stale, which is safe)
2. No risk of deadlock from process crash while holding lock
3. Simpler error handling -- partial reads impossible (file is either old complete or new complete)
4. Matches the existing `_atomic_write_parquet` pattern in the codebase

### Pattern 2: Seeder REST Fallback Logic

**What:** After Vision 404, attempt REST API for yesterday only.
**When to use:** Vision CSV not yet published (~8h delayed).

```python
def _seed_day(symbol: str, date_str: str, storage: TickStorage, market: str) -> bool:
    """Try Vision first, REST fallback for yesterday only."""
    # Try Vision
    df = _download_vision_day(symbol, date_str, timeout_s=cfg.vision_timeout_s, market=market)
    if df is not None:
        storage.write_ticks(symbol, df)  # Route through dedup path (D-09)
        return True

    # REST fallback: only for yesterday (D-04)
    today = datetime.now(UTC).date()
    target_date = date.fromisoformat(date_str)
    if target_date >= today:
        return False  # Never seed today
    if (today - target_date).days > 2:
        return False  # Only yesterday gets REST fallback; older waits for Vision

    # Use existing Rust parallel fetch
    from opendeviationbar._core import fetch_aggtrades_parallel
    day_start_ms = int(datetime.combine(target_date, time.min, UTC).timestamp() * 1000)
    day_end_ms = day_start_ms + 86_400_000 - 1

    # Get first trade ID for the day via REST
    from opendeviationbar._core import fetch_aggtrades_rest
    trades = fetch_aggtrades_rest(symbol, day_start_ms, day_end_ms)
    if not trades:
        return False

    df = pl.DataFrame(trades)
    # Validate trade-ID continuity (D-07)
    if not _validate_trade_id_continuity(df):
        logger.warning("%s %s: REST trades failed continuity check, rejecting", symbol, date_str)
        return False

    # Mark as REST-sourced (D-08) via Parquet metadata
    storage.write_ticks(symbol, df)  # Route through dedup path
    return True
```

### Pattern 3: Trade-ID Continuity Validation

**What:** After every Parquet write, verify `trade_id[i+1] == trade_id[i] + 1`.
**When to use:** All Parquet tick writes (Vision and REST).

```python
def _validate_trade_id_continuity(df: pl.DataFrame) -> bool:
    """Verify trade-ID continuity: each consecutive ID increments by exactly 1."""
    if "agg_trade_id" not in df.columns or df.height < 2:
        return True  # Can't validate, assume ok
    sorted_df = df.sort("agg_trade_id")
    deltas = sorted_df["agg_trade_id"].diff().drop_nulls()
    return (deltas == 1).all()
```

### Pattern 4: Per-Process Ceiling via Environment Variables

**What:** Each consumer sets its ceiling via env var, read by the existing `PYTHON_RATE_LIMITER` singleton.
**When to use:** Already exists in `binance_bindings.rs`.

The existing code already reads `OPENDEVIATIONBAR_REST_CEILING` env var. Each systemd service sets its own ceiling:

- Sidecar: `Environment=OPENDEVIATIONBAR_REST_CEILING=2400`
- Kintsugi: `Environment=OPENDEVIATIONBAR_REST_CEILING=2000`
- Seeder: `Environment=OPENDEVIATIONBAR_REST_CEILING=800`

### Anti-Patterns to Avoid

- **Single shared rate limiter across processes:** Each process has its own `OnceLock<Arc<AdaptiveRateLimiter>>`. Cross-process coordination MUST use external state (file/Redis), not Rust atomics.
- **Seeder writing to ClickHouse:** Violates writer ownership model. Seeder writes Parquet ticks ONLY.
- **Seeding today's data:** Today belongs to sidecar. REST seeding targets yesterday only.
- **Creating new `TickStorage()` per download:** Follow thread-local reuse pattern.
- **Trusting REST data without continuity check:** REST can return partial days if rate limited. Always validate before writing.

## Don't Hand-Roll

| Problem                   | Don't Build              | Use Instead                                                     | Why                                                                                     |
| ------------------------- | ------------------------ | --------------------------------------------------------------- | --------------------------------------------------------------------------------------- |
| REST aggTrades fetching   | Custom HTTP pagination   | `fetch_aggtrades_parallel` (Rust, via PyO3)                     | Already handles fromId pagination, rate limiting, early termination, retry with backoff |
| Per-process rate limiting | Python token bucket      | `AdaptiveRateLimiter` (Rust)                                    | Thread-safe CAS, server header feedback, dynamic ceiling                                |
| Atomic Parquet writes     | Manual tempfile + rename | `_atomic_write_parquet()`                                       | Already handles fsync, same-filesystem guarantee, cleanup on failure                    |
| Trade dedup on write      | Manual merge logic       | `TickStorage.write_ticks()`                                     | Anti-join dedup on `agg_trade_id`, handles schema alignment                             |
| Retry with backoff        | Manual sleep loops       | `stamina.retry()`                                               | Project convention, configurable attempts/timeout                                       |
| Vision CSV parsing        | Manual CSV → DataFrame   | Existing `_download_vision_day()` (or future Rust binding D-19) | Handles spot/futures, header detection, us/ms normalization                             |

## Common Pitfalls

### Pitfall 1: Cross-Process Window Desynchronization

**What goes wrong:** Each process independently resets its 60-second rate limit window. Process A resets at T=0, Process B at T=30. At T=45, A thinks budget is 50% used, B thinks 25% used. Aggregate exceeds limit.
**Why it happens:** `AdaptiveRateLimiter` uses `Instant::now()` per-process. No shared clock.
**How to avoid:** The file-based coordinator uses wall-clock time (`time.time()`) for window boundaries. All processes read the same file. The Binance `X-MBX-USED-WEIGHT-1m` header is the authoritative source -- it reflects IP-wide usage regardless of which process made the request. Each process should update `server_used_weight` from response headers, and this value should be reported to the shared state file.
**Warning signs:** Sporadic 429 responses despite each process reporting under-budget.

### Pitfall 2: Stale Rate Limit State File

**What goes wrong:** A process crashes or hangs. Its weight report remains in the state file. Other processes read a higher aggregate than reality, unnecessarily throttling themselves.
**Why it happens:** The crashed process never updated its entry to reflect window expiry.
**How to avoid:** Each entry has an `updated` timestamp. Entries older than 65 seconds (window + 5s grace) are treated as expired and ignored when computing aggregate. The 5s grace handles clock skew between processes.
**Warning signs:** Seeder throughput drops after kintsugi crash.

### Pitfall 3: REST Partial Day for Yesterday

**What goes wrong:** REST fetch for yesterday returns partial data (e.g., yesterday is still "today" in a different timezone interpretation, or the REST endpoint rate-limits mid-fetch).
**Why it happens:** Yesterday is recent enough that data may still be accumulating on Binance's side.
**How to avoid:** D-07 trade-ID continuity check catches gaps. D-08 marks REST files as `complete=false`. D-10 weekly oracle comparison catches systemic mismatches. Seeder should also compare REST trade count against a reasonable minimum (e.g., BTCUSDT averages ~750K trades/day; if REST returns <100K for a non-holiday weekday, flag as suspicious).
**Warning signs:** Kintsugi sees Stathera gaps at the REST/Vision file boundary.

### Pitfall 4: Seeder Bypassing write_ticks() Dedup

**What goes wrong:** Currently seeder calls `_atomic_write_parquet()` directly (line 360 of `tick_cache_seeder.py`). This bypasses the dedup logic in `write_ticks()`. If seeder runs twice for the same day (e.g., timer overlap before Conflicts= is added), duplicate trades accumulate.
**Why it happens:** Original seeder design assumed Vision downloads are idempotent (same file). REST fallback makes this assumption false.
**How to avoid:** D-09 routes all writes through `write_ticks()`. D-05 adds systemd Conflicts= to prevent concurrent runs.
**Warning signs:** Parquet file sizes for a day are 2x expected.

### Pitfall 5: Budget Exhaustion from Sidecar Puck Gap Fill

**What goes wrong:** Sidecar's Puck (inline gap fill) triggers a burst of parallel REST calls that consume the full sidecar budget (2400 weight). Simultaneously, kintsugi and seeder are also fetching. Total exceeds 6000.
**Why it happens:** Puck's `fill_from_rest` uses the sidecar's in-process rate limiter, which only knows about sidecar's usage. The file-based coordinator adds cross-process awareness, but the sidecar may not read it frequently enough during burst fills.
**How to avoid:** The per-process ceiling (2400 for sidecar) enforces the hard cap. Even if all three consumers hit their ceiling simultaneously: 2400+2000+800=5200 < 6000. The 800 reserve absorbs any overshoot from server header reporting latency.
**Warning signs:** Multiple 429 responses clustered within a 10-second window.

### Pitfall 6: REST Weight Per Request is 4, Not 1

**What goes wrong:** Binance Spot aggTrades endpoint costs **4 weight per request** (verified in `historical.rs:756`). If the code uses `acquire(weight=1)`, it underestimates by 4x.
**Why it happens:** Different Binance endpoints have different weights. aggTrades=4 is documented in Binance API docs.
**How to avoid:** D-17 says "acquire(weight=1)" but the actual weight for aggTrades is 4. The existing Rust code in `send_with_rate_limit_backoff()` already passes `weight=4`. Kintsugi's Python wrapper must use the same weight. Seeder uses `fetch_aggtrades_parallel` which already handles this internally.
**Warning signs:** Server header `X-MBX-USED-WEIGHT-1m` climbs 4x faster than local tracking predicts.

## Code Examples

### Existing: Per-Process Rate Limiter Singleton (binance_bindings.rs)

```rust
// Source: src/binance_bindings.rs lines 12-44
static PYTHON_RATE_LIMITER: OnceLock<Arc<AdaptiveRateLimiter>> = OnceLock::new();

fn get_python_rate_limiter() -> Arc<AdaptiveRateLimiter> {
    Arc::clone(PYTHON_RATE_LIMITER.get_or_init(|| {
        let max_weight = std::env::var("OPENDEVIATIONBAR_REST_MAX_WEIGHT_PER_MINUTE")
            .ok().and_then(|v| v.parse::<u32>().ok()).unwrap_or(6000);
        let ceiling = std::env::var("OPENDEVIATIONBAR_REST_CEILING")
            .ok().and_then(|v| v.parse::<u32>().ok()).unwrap_or(0);
        // ... creates AdaptiveRateLimiter with ceiling
    }))
}
```

### Existing: Rate Limiter Status PyO3 Binding (binance_bindings.rs)

```rust
// Source: src/binance_bindings.rs lines 50-59
#[pyfunction]
pub fn get_rest_rate_limiter_status(py: Python) -> PyResult<PyObject> {
    let limiter = get_python_rate_limiter();
    let dict = PyDict::new(py);
    dict.set_item("utilization", limiter.utilization())?;
    dict.set_item("remaining", limiter.remaining())?;
    dict.set_item("max_weight_per_minute", limiter.max_weight_per_minute())?;
    Ok(dict.into_any().unbind())
}
```

### Existing: Seeder Direct Write (BYPASSES dedup -- to be fixed by D-09)

```python
# Source: scripts/tick_cache_seeder.py line 360
day_path = storage._get_parquet_path(cache_sym, date_str)
_atomic_write_parquet(df, day_path)  # Direct write, no dedup
# Should become: storage.write_ticks(cache_sym, df)
```

### Existing: write_ticks() Dedup (parquet.py lines 266-292)

```python
# Source: python/opendeviationbar/storage/parquet.py
# Anti-join dedup: remove trades already in existing file
write_deduped = write_df.unique(subset=["agg_trade_id"], maintain_order=True)
existing_df = pl.read_parquet(parquet_path)
write_new = write_deduped.join(
    existing_df.select("agg_trade_id"), on="agg_trade_id", how="anti",
)
```

### Existing: Parallel Fetch PyO3 Binding

```python
# Source: src/binance_bindings.rs lines 357-398
# Already exposed as fetch_aggtrades_parallel(symbol, from_id, to_id, concurrency=5)
# Uses get_python_rate_limiter() singleton
# Returns List[Dict] with timestamp (ms), price, quantity, agg_trade_id, etc.
```

## State of the Art

| Old Approach                             | Current Approach                            | When Changed   | Impact                                             |
| ---------------------------------------- | ------------------------------------------- | -------------- | -------------------------------------------------- |
| No cross-process rate limiting           | Per-process ceiling via env var             | #291 (2026-03) | Prevents single process from starving others       |
| Monthly Parquet files                    | Daily Parquet files                         | #277 (2026-02) | O(1) has_ticks(), simpler write_ticks()            |
| Python requests for Vision               | Python requests (Rust binding planned D-19) | Current        | Rust binding will eliminate Python HTTP dependency |
| Direct `_atomic_write_parquet` in seeder | Route through `write_ticks()` (D-09)        | This phase     | Adds dedup, normalization, consistency             |

## Validation Architecture

### Test Framework

| Property           | Value                                      |
| ------------------ | ------------------------------------------ |
| Framework          | pytest (already configured)                |
| Config file        | `pyproject.toml` [tool.pytest.ini_options] |
| Quick run command  | `mise run test-py`                         |
| Full suite command | `mise run check-full`                      |

### Phase Requirements to Test Map

| Req ID    | Behavior                                  | Test Type | Automated Command                                                       | File Exists? |
| --------- | ----------------------------------------- | --------- | ----------------------------------------------------------------------- | ------------ |
| SEEDER-01 | REST fallback on Vision 404               | unit      | `pytest tests/test_seeder_rest_fallback.py -x`                          | No -- Wave 0 |
| SEEDER-02 | Cross-process rate limit coordination     | unit      | `pytest tests/test_ratelimit_coordinator.py -x`                         | No -- Wave 0 |
| SEEDER-03 | Aggregate stays below 80% of 6000         | unit      | `pytest tests/test_ratelimit_coordinator.py::test_budget_allocation -x` | No -- Wave 0 |
| SEEDER-04 | Writer ownership (Parquet only, no today) | unit      | `pytest tests/test_seeder_rest_fallback.py::test_never_seed_today -x`   | No -- Wave 0 |
| D-07      | Trade-ID continuity validation            | unit      | `pytest tests/test_parquet_validation.py::test_trade_id_continuity -x`  | No -- Wave 0 |
| D-20      | Last-seeded-date cache                    | unit      | `pytest tests/test_seeder_rest_fallback.py::test_last_seeded_cache -x`  | No -- Wave 0 |

### Sampling Rate

- **Per task commit:** `pytest tests/test_seeder_rest_fallback.py tests/test_ratelimit_coordinator.py tests/test_parquet_validation.py -x`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** `mise run check-full` green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_seeder_rest_fallback.py` -- covers SEEDER-01, SEEDER-04, D-04, D-08, D-20
- [ ] `tests/test_ratelimit_coordinator.py` -- covers SEEDER-02, SEEDER-03, D-11, D-13
- [ ] `tests/test_parquet_validation.py` -- covers D-07 (trade-ID continuity)
- [ ] `tests/conftest.py` -- may need fixtures for mock TickStorage, mock rate limit state file

## Environment Availability

| Dependency            | Required By                | Available                                    | Version | Fallback                      |
| --------------------- | -------------------------- | -------------------------------------------- | ------- | ----------------------------- |
| Python 3.13           | All Python code            | Verified (project policy)                    | 3.13    | --                            |
| Rust toolchain        | PyO3 bindings              | Verified (maturin develop works)             | --      | --                            |
| Redis                 | D-12 upgrade path          | Available on bigblack (graceful degradation) | --      | File-based coordinator (D-11) |
| `/tmp` filesystem     | D-11 rate limit state file | Always available                             | --      | --                            |
| systemd user services | D-05 Conflicts=            | Available on bigblack                        | --      | --                            |

**Missing dependencies with no fallback:** None.
**Missing dependencies with fallback:** Redis (file-based coordinator is the primary mechanism anyway).

## Open Questions

1. **aggTrades weight on different endpoints**
   - What we know: Spot aggTrades = 4 weight per request (verified in `historical.rs:756`)
   - What's unclear: D-17 says `acquire(weight=1)` -- is this a documentation error or intentional?
   - Recommendation: Use weight=4 consistently. The Rust `send_with_rate_limit_backoff` already handles this correctly. Python kintsugi wrapper should match.

2. **Parquet metadata for REST vs Vision source tracking (D-08)**
   - What we know: Polars `write_parquet()` does not natively support custom key-value metadata
   - What's unclear: Best way to embed `source=REST` in Parquet file
   - Recommendation: Use a sidecar `.meta` JSON file (e.g., `2026-03-21.parquet.meta`) or embed a `_source` column in the DataFrame. Sidecar file is simpler and doesn't pollute the data schema.

3. **Batch rotation interaction with newest-first ordering (D-14 vs D-21)**
   - What we know: D-14 says 6 symbols per 5-min run. D-21 says newest-first ordering within a symbol.
   - What's unclear: Does "6 symbols per run" mean first 6 alphabetically, or round-robin across runs?
   - Recommendation: Rotate through all 18 symbols in groups of 6, prioritized by volume (BTCUSDT/ETHUSDT always in first batch). Within each symbol, process yesterday first (only REST candidate), then missing Vision days newest-first.

## Sources

### Primary (HIGH confidence)

- `src/binance_bindings.rs` -- Existing PyO3 rate limiter singleton, `fetch_aggtrades_parallel` binding
- `crates/opendeviationbar-providers/src/binance/rate_limiter.rs` -- Full `AdaptiveRateLimiter` implementation
- `crates/opendeviationbar-providers/src/binance/parallel_fetch.rs` -- Parallel fetch with early termination
- `crates/opendeviationbar-providers/src/binance/historical.rs` -- REST fetch, Vision download, weight=4
- `scripts/tick_cache_seeder.py` -- Current seeder (Vision-only)
- `python/opendeviationbar/storage/parquet.py` -- `write_ticks()` with dedup
- `python/opendeviationbar/storage/parquet_io.py` -- `_atomic_write_parquet()`
- `python/opendeviationbar/config/seeder.py` -- `SeederConfig` pydantic-settings
- `scripts/heartbeat/checks/seeder.py` -- Current seeder health check
- `scripts/systemd/opendeviationbar-seeder.service` -- Current service file (missing Conflicts=)

### Secondary (MEDIUM confidence)

- CONTEXT.md decisions D-01 through D-25 -- User architecture decisions
- scripts/CLAUDE.md -- Writer ownership model, kintsugi parallelism, bigblack capacity

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- all libraries already in-tree, patterns well-established
- Architecture: HIGH -- extends existing patterns (atomic write, pydantic config, PyO3 bindings)
- Pitfalls: HIGH -- based on known production incidents (#295, #280, #291) and verified code paths

**Research date:** 2026-03-22
**Valid until:** 2026-04-22 (stable domain, no external library changes expected)
