---
phase: 05-seeder-rest-ratelimit
plan: 03
type: execute
wave: 2
depends_on: [05-01, 05-02]
files_modified:
  - scripts/tick_cache_seeder.py
  - python/opendeviationbar/config/seeder.py
  - tests/test_seeder_rest_fallback.py
autonomous: true
requirements: [SEEDER-01, SEEDER-04]

must_haves:
  truths:
    - "Seeder falls back to REST API when Vision returns 404 for yesterday"
    - "REST fallback targets yesterday only -- never seeds today, older days wait for Vision"
    - "REST-sourced files get a .meta sidecar marking source=REST for Vision overwrite"
    - "Seeder processes 6 symbols per run in batch rotation with BTCUSDT priority"
    - "Missing-days scan uses cached last-seeded-date to reduce stat() calls"
    - "Seeder skips REST seeding for days where Parquet file already exists without .meta sidecar (kintsugi already processed)"
  artifacts:
    - path: "scripts/tick_cache_seeder.py"
      provides: "REST fallback path, batch rotation, missing-days cache, kintsugi-processed guard"
      contains: "fetch_aggtrades_parallel"
    - path: "python/opendeviationbar/config/seeder.py"
      provides: "New REST config fields"
      contains: "rest_fallback_enabled"
    - path: "tests/test_seeder_rest_fallback.py"
      provides: "Tests for REST fallback, day guards, batch rotation, kintsugi guard"
  key_links:
    - from: "scripts/tick_cache_seeder.py"
      to: "opendeviationbar._core.fetch_aggtrades_parallel"
      via: "Two-step: fetch_aggtrades_rest for first trade ID, then fetch_aggtrades_parallel for bulk"
      pattern: "fetch_aggtrades_parallel"
    - from: "scripts/tick_cache_seeder.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "storage.write_ticks() for REST-fetched trades"
      pattern: "storage\\.write_ticks"
---

<objective>
Add REST API fallback to the seeder for yesterday (when Vision is not yet published), implement batch rotation for 18-symbol throughput, optimize missing-days scanning with cached last-seeded-date, and add kintsugi-processed guard to skip days already completed.

Purpose: Fills the ~8h Vision publication delay gap by using REST API for yesterday's trades (D-01, D-02, D-04). Batch rotation ensures all 18 symbols complete a full cycle in 15 minutes without exceeding the 800 weight/min budget (D-14). Missing-days cache eliminates 52K stat() calls per run (D-20). Kintsugi guard ensures seeder only seeds days kintsugi hasn't yet processed (SEEDER-04).

Output: Modified seeder with REST fallback, batch rotation, missing-days cache, kintsugi guard, REST source metadata, test suite.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-seeder-rest-ratelimit/05-CONTEXT.md
@.planning/phases/05-seeder-rest-ratelimit/05-RESEARCH.md
@.planning/phases/05-seeder-rest-ratelimit/05-01-SUMMARY.md
@.planning/phases/05-seeder-rest-ratelimit/05-02-SUMMARY.md

@scripts/tick_cache_seeder.py
@python/opendeviationbar/config/seeder.py
@src/binance_bindings.rs

<interfaces>
<!-- From Plan 01: validate_trade_id_continuity is now called inside write_ticks() -->
From python/opendeviationbar/storage/parquet_io.py:
```python
def validate_trade_id_continuity(df: pl.DataFrame) -> bool:
    """Returns True if agg_trade_id increments by exactly 1 for all consecutive rows."""
```

<!-- From Plan 02: RateLimitCoordinator for cross-process budget -->

From python/opendeviationbar/ratelimit/file_coordinator.py:

```python
class RateLimitCoordinator:
    def report_usage(self, consumer_id: str, weight_used: int) -> None: ...
    def get_aggregate_usage(self) -> int: ...
    def can_acquire(self, consumer_id: str, weight: int, budget: int) -> bool: ...
    def get_state(self) -> dict: ...

BUDGET_SEEDER = 800
```

<!-- CRITICAL: fetch_aggtrades_parallel binding signature -->
<!-- Uses from_id/to_id positional args, NOT start_time_ms/end_time_ms -->

From src/binance_bindings.rs:

```python
# fetch_aggtrades_parallel(symbol, from_id, to_id, concurrency=5)
# Returns List[Dict] with keys: timestamp (ms), price, quantity, agg_trade_id, etc.
# Uses get_python_rate_limiter() singleton with OPENDEVIATIONBAR_REST_CEILING
# Weight per request: 4 (handled internally by Rust)
```

```python
# fetch_aggtrades_rest(symbol, start_time_ms, end_time_ms, market_type=None)
# Returns List[Dict] -- use this to get the first trade of a day by time range
# Then use fetch_aggtrades_parallel with from_id for bulk fetch
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add REST config fields to SeederConfig and implement REST fallback + batch rotation in seeder</name>
  <files>python/opendeviationbar/config/seeder.py, scripts/tick_cache_seeder.py</files>
  <read_first>
    - python/opendeviationbar/config/seeder.py (full file -- current config fields)
    - scripts/tick_cache_seeder.py (full file -- understand main(), _seed_symbol(), _find_missing_days())
    - src/binance_bindings.rs (lines 195-230 -- fetch_aggtrades_rest signature; lines 355-400 -- fetch_aggtrades_parallel signature)
    - python/opendeviationbar/storage/parquet.py (lines 188-230 -- write_ticks() signature)
  </read_first>
  <action>
    **Step 1: Extend SeederConfig** (per D-01, D-04, D-14, D-20):

    Add to `python/opendeviationbar/config/seeder.py`:
    ```python
    rest_fallback_enabled: bool = True        # D-01: enable REST fallback for Vision 404
    rest_fallback_max_age_days: int = 2       # D-04: REST only for yesterday (1 day old max)
    batch_size: int = 6                       # D-14: symbols per 5-min run
    last_seeded_state_file: str = "/var/tmp/opendeviationbar-seeder-last-seeded.json"  # D-20
    ```
    Add docstring entries for each new env var:
    - `OPENDEVIATIONBAR_SEEDER_REST_FALLBACK_ENABLED` (default True)
    - `OPENDEVIATIONBAR_SEEDER_REST_FALLBACK_MAX_AGE_DAYS` (default 2, meaning only yesterday)
    - `OPENDEVIATIONBAR_SEEDER_BATCH_SIZE` (default 6)
    - `OPENDEVIATIONBAR_SEEDER_LAST_SEEDED_STATE_FILE` (default path)

    **Step 2: Add REST fallback to _seed_symbol() using two-step fetch pattern** (per D-01, D-02, D-03, D-04, D-18):

    IMPORTANT: `fetch_aggtrades_parallel` takes `(symbol, from_id, to_id)` -- NOT `start_time_ms`. Use a two-step pattern:

    1. Call `fetch_aggtrades_rest(symbol, day_start_ms, day_end_ms)` with `limit=1` to get the first trade of the day and its `agg_trade_id`.
    2. Call `fetch_aggtrades_rest(symbol, day_end_ms - 1000, day_end_ms)` to get the last trade and its `agg_trade_id`.
    3. Call `fetch_aggtrades_parallel(symbol, first_trade_id, last_trade_id)` for the bulk fetch.

    In `scripts/tick_cache_seeder.py`, modify `_seed_symbol()` to add REST fallback after Vision 404:

    After `_download_vision_day()` returns None (line ~356-357 in the download loop), add:
    ```python
    # REST fallback for recent days only (D-01, D-04)
    if df is None and cfg.rest_fallback_enabled:
        target_date = date.fromisoformat(date_str)
        today = datetime.now(UTC).date()
        age_days = (today - target_date).days
        if 1 <= age_days <= cfg.rest_fallback_max_age_days:
            df = _rest_fallback_day(symbol, date_str, market)
            if df is not None:
                rest_fallback_used = True
                logger.info("%s %s [%s]: REST fallback succeeded, %d trades",
                           symbol, date_str, market, df.height)
    ```

    Add new function `_rest_fallback_day()`:
    ```python
    def _rest_fallback_day(symbol: str, date_str: str, market: str = "spot") -> pl.DataFrame | None:
        """Fetch one day's trades via REST API (D-02). Uses two-step: time->ID, then parallel fetch."""
        if market != "spot":
            return None  # REST fallback only for spot (our primary market)

        from opendeviationbar._core import fetch_aggtrades_rest, fetch_aggtrades_parallel

        target_date = date.fromisoformat(date_str)
        day_start_ms = int(datetime.combine(target_date, datetime.min.time(), UTC).timestamp() * 1000)
        day_end_ms = day_start_ms + 86_400_000 - 1

        try:
            # Step 1: Get first trade ID for the day via time-based REST query
            first_trades = fetch_aggtrades_rest(symbol, day_start_ms, day_start_ms + 60_000)
            if not first_trades:
                logger.warning("%s %s: REST fallback found no trades at day start", symbol, date_str)
                return None
            first_trade_id = first_trades[0]["a"]  # agg_trade_id

            # Step 2: Get last trade ID for the day
            last_trades = fetch_aggtrades_rest(symbol, day_end_ms - 60_000, day_end_ms)
            if not last_trades:
                logger.warning("%s %s: REST fallback found no trades at day end", symbol, date_str)
                return None
            last_trade_id = last_trades[-1]["a"]  # agg_trade_id

            # Step 3: Bulk fetch using from_id/to_id parallel fetch
            # fetch_aggtrades_parallel(symbol, from_id, to_id, concurrency=5)
            trades = fetch_aggtrades_parallel(symbol, first_trade_id, last_trade_id)
        except Exception as exc:
            logger.warning("%s %s: REST fallback failed: %s", symbol, date_str, exc)
            return None

        if not trades:
            logger.warning("%s %s: REST fallback returned 0 trades", symbol, date_str)
            return None

        # Convert to polars DataFrame matching Vision schema
        df = pl.DataFrame(trades)
        # Rename REST single-letter keys to our column names
        rename_map = {
            "a": "agg_trade_id", "p": "price", "q": "quantity",
            "T": "timestamp", "m": "is_buyer_maker",
            "f": "first_trade_id", "l": "last_trade_id",
        }
        # Only rename columns that exist (some may already use full names)
        existing_renames = {k: v for k, v in rename_map.items() if k in df.columns}
        if existing_renames:
            df = df.rename(existing_renames)

        # Ensure correct types
        df = df.select([
            pl.col("agg_trade_id").cast(pl.Int64),
            pl.col("price").cast(pl.Float64),
            pl.col("quantity").cast(pl.Float64),
            pl.col("first_trade_id").cast(pl.Int64),
            pl.col("last_trade_id").cast(pl.Int64),
            pl.col("timestamp").cast(pl.Int64),
            pl.col("is_buyer_maker").cast(pl.Boolean),
        ])
        df = df.sort("agg_trade_id")
        return df
    ```

    After successful REST write, create a `.meta` sidecar file (per D-08):
    ```python
    if rest_fallback_used:
        meta_path = storage._get_parquet_path(cache_sym, date_str).with_suffix(".parquet.meta")
        try:
            meta_path.write_text(json.dumps({
                "source": "REST",
                "complete": False,
                "fetched_at": datetime.now(UTC).isoformat(),
                "trade_count": df.height,
            }))
        except OSError:
            pass  # non-critical metadata
    ```

    Add `import json` to the top of the file.

    **Step 3: Add kintsugi-processed guard** (per SEEDER-04 "only for days kintsugi hasn't yet processed"):

    In `_find_missing_days()`, when a Parquet file already exists WITHOUT a `.meta` sidecar, it means the file was written by Vision or by kintsugi (complete source). Skip these days -- REST seeding should only target truly missing days:
    ```python
    if not day_path.exists():
        missing.append(date_str)
    elif day_path.with_suffix(".parquet.meta").exists():
        # REST-sourced file -- try Vision overwrite (D-08)
        missing.append(date_str)
    # else: Parquet exists without .meta = kintsugi or Vision already processed (SEEDER-04), skip
    ```

    This ensures seeder only REST-seeds days that are truly missing. If kintsugi has already fetched trades and written a complete Parquet file for a day, the seeder will not re-seed it via REST.

    **Step 4: Add batch rotation to main()** (per D-14, D-21):

    Modify `main()` to process symbols in batches of `cfg.batch_size` (default 6). In each 5-min timer run, process the next batch:
    ```python
    # Batch rotation (D-14): process cfg.batch_size symbols per run
    # State file tracks which batch to process next
    batch_state_path = Path(cfg.last_seeded_state_file)
    batch_offset = _load_batch_offset(batch_state_path, len(symbols))

    # Prioritize by volume: BTCUSDT and ETHUSDT always first (D-14)
    priority = ["BTCUSDT", "ETHUSDT"]
    ordered = [s for s in priority if s in symbols] + [s for s in symbols if s not in priority]

    # Select this run's batch
    batch = ordered[batch_offset:batch_offset + cfg.batch_size]
    if not batch:
        batch = ordered[:cfg.batch_size]
        batch_offset = 0

    # Process only the batch
    for market in markets:
        for symbol in batch:
            # ... existing per-symbol processing ...
    ```

    Add helper functions:
    ```python
    def _load_batch_offset(state_path: Path, total_symbols: int) -> int:
        """Load and advance batch offset from state file."""
        try:
            data = json.loads(state_path.read_text())
            offset = data.get("batch_offset", 0)
            if offset >= total_symbols:
                return 0
            return offset
        except (FileNotFoundError, json.JSONDecodeError, OSError):
            return 0

    def _save_batch_offset(state_path: Path, offset: int) -> None:
        """Save next batch offset to state file (D-20)."""
        try:
            state_path.write_text(json.dumps({
                "batch_offset": offset,
                "updated": datetime.now(UTC).isoformat(),
            }))
        except OSError:
            pass
    ```

    At the end of `main()`, save the next batch offset:
    ```python
    next_offset = batch_offset + cfg.batch_size
    if next_offset >= len(ordered):
        next_offset = 0
    _save_batch_offset(batch_state_path, next_offset)
    ```

    **Step 5: Add rate limit reporting** (per D-11):

    At the start of `main()`, create coordinator and report initial state:
    ```python
    from opendeviationbar.ratelimit import RateLimitCoordinator
    coordinator = RateLimitCoordinator()
    ```

    After each REST fallback call, report weight usage:
    ```python
    # Report weight to cross-process coordinator
    try:
        from opendeviationbar._core import get_rest_rate_limiter_status
        status = get_rest_rate_limiter_status()
        used = int(status.get("max_weight_per_minute", 6000) * status.get("utilization", 0))
        coordinator.report_usage("seeder", used)
    except Exception:
        pass  # non-critical telemetry
    ```

    **Step 6: Vision overwrite for REST-sourced files** (per D-08):

    In `_seed_symbol()`, when Vision succeeds for a day that has a `.meta` file (previous REST source), overwrite:
    ```python
    # Check if this day was previously REST-sourced (D-08)
    meta_path = storage._get_parquet_path(cache_sym, date_str).with_suffix(".parquet.meta")
    if meta_path.exists():
        logger.info("%s %s [%s]: Vision overwriting REST-sourced file", symbol, date_str, market)
        # write_ticks will overwrite the existing file
        # Remove the .meta sidecar
        meta_path.unlink(missing_ok=True)
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "from opendeviationbar.config.seeder import SeederConfig; c = SeederConfig(); assert c.rest_fallback_enabled == True; assert c.batch_size == 6; print('OK')"</automated>
  </verify>
  <acceptance_criteria>
    - seeder.py SeederConfig contains `rest_fallback_enabled: bool = True`
    - seeder.py SeederConfig contains `batch_size: int = 6`
    - seeder.py SeederConfig contains `rest_fallback_max_age_days: int = 2`
    - seeder.py SeederConfig contains `last_seeded_state_file`
    - tick_cache_seeder.py contains `def _rest_fallback_day(`
    - tick_cache_seeder.py contains `fetch_aggtrades_parallel`
    - tick_cache_seeder.py contains `fetch_aggtrades_rest` (two-step: time-based first, then parallel bulk)
    - tick_cache_seeder.py contains `"source": "REST"` (meta sidecar)
    - tick_cache_seeder.py contains `_load_batch_offset`
    - tick_cache_seeder.py contains `coordinator.report_usage`
    - tick_cache_seeder.py main() uses batch rotation (processes cfg.batch_size symbols per run)
    - tick_cache_seeder.py _find_missing_days() checks for .meta files (REST overwrite path)
    - tick_cache_seeder.py _find_missing_days() skips days where Parquet exists without .meta (SEEDER-04 kintsugi guard)
  </acceptance_criteria>
  <done>Seeder falls back to REST for yesterday on Vision 404 using two-step fetch (time->ID, then parallel), processes 6 symbols per run with BTCUSDT priority, skips kintsugi-processed days, reports weight to cross-process coordinator.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Create tests for REST fallback, day guards, batch rotation, and kintsugi guard</name>
  <files>tests/test_seeder_rest_fallback.py</files>
  <read_first>
    - scripts/tick_cache_seeder.py (the modified file from Task 1 -- understand REST fallback and batch rotation)
    - python/opendeviationbar/config/seeder.py (new config fields)
  </read_first>
  <behavior>
    - Test 1 (test_never_seed_today): _rest_fallback_day returns None when target_date == today (D-04, SEEDER-04)
    - Test 2 (test_rest_fallback_yesterday): _rest_fallback_day attempts fetch for yesterday (age=1 day)
    - Test 3 (test_no_rest_for_old_days): REST fallback skipped for days older than rest_fallback_max_age_days
    - Test 4 (test_vision_overwrite_rest): When Vision succeeds for a REST-sourced day (.meta exists), the .meta file is removed
    - Test 5 (test_batch_rotation_offset): _load_batch_offset / _save_batch_offset round-trip correctly
    - Test 6 (test_batch_rotation_wraps): Batch offset wraps to 0 when exceeding total symbols
    - Test 7 (test_batch_priority_order): BTCUSDT and ETHUSDT always appear in the first batch
    - Test 8 (test_rest_fallback_disabled): When rest_fallback_enabled=False, REST path is never attempted
    - Test 9 (test_meta_file_created): After REST fallback write, .meta JSON file exists with source=REST
    - Test 10 (test_skip_kintsugi_processed): _find_missing_days() skips days where Parquet exists without .meta (SEEDER-04)
    - Test 11 (test_include_rest_sourced): _find_missing_days() includes days where Parquet exists WITH .meta (D-08 Vision overwrite)
  </behavior>
  <action>
    Create `tests/test_seeder_rest_fallback.py` with the 11 test cases from `<behavior>`.

    Use `monkeypatch` to mock:
    - `datetime.now(UTC).date()` for today/yesterday control
    - `opendeviationbar._core.fetch_aggtrades_parallel` and `fetch_aggtrades_rest` to return mock trade data without network calls
    - `TickStorage` with `tmp_path` fixture for isolated file system

    For test_never_seed_today (SEEDER-04): Create a date equal to today, verify REST fallback returns None or is not attempted. This validates the writer ownership model -- today belongs to sidecar.

    For test_skip_kintsugi_processed (SEEDER-04): Create a Parquet file at the expected path WITHOUT a .meta sidecar. Verify _find_missing_days() does NOT include this date. This ensures seeder only REST-seeds days that kintsugi hasn't yet processed.

    For test_include_rest_sourced (D-08): Create a Parquet file WITH a .meta sidecar (source=REST). Verify _find_missing_days() INCLUDES this date (so Vision can overwrite it).

    For test_batch_priority_order: Given 18 symbols, verify that the first batch of 6 includes BTCUSDT and ETHUSDT.

    Import the seeder functions directly:
    ```python
    sys.path.insert(0, str(Path(__file__).parent.parent / "scripts"))
    from tick_cache_seeder import _rest_fallback_day, _load_batch_offset, _save_batch_offset
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_seeder_rest_fallback.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - tests/test_seeder_rest_fallback.py exists with 11+ test functions
    - Test names include: test_never_seed_today, test_rest_fallback_yesterday, test_no_rest_for_old_days
    - Test names include: test_batch_rotation_offset, test_batch_rotation_wraps, test_batch_priority_order
    - Test names include: test_skip_kintsugi_processed, test_include_rest_sourced
    - All tests pass
  </acceptance_criteria>
  <done>Full test coverage for REST fallback day guards, batch rotation, meta file handling, and kintsugi-processed guard (SEEDER-04).</done>
</task>

</tasks>

<verification>
- `uv run --python 3.13 pytest tests/test_seeder_rest_fallback.py -x -v` passes
- `grep "fetch_aggtrades_parallel" scripts/tick_cache_seeder.py` confirms REST integration
- `grep "fetch_aggtrades_rest" scripts/tick_cache_seeder.py` confirms two-step fetch pattern
- `grep "rest_fallback_enabled" python/opendeviationbar/config/seeder.py` confirms config
- `grep "batch_size" python/opendeviationbar/config/seeder.py` confirms batch rotation config
- `grep "parquet.meta" scripts/tick_cache_seeder.py` confirms kintsugi guard and meta handling
</verification>

<success_criteria>

1. Seeder falls back to REST API when Vision returns 404 for days within rest_fallback_max_age_days (D-01, D-02)
2. REST fallback uses two-step pattern: fetch_aggtrades_rest for time-based ID lookup, then fetch_aggtrades_parallel for bulk fetch (Warning 1 fix)
3. REST fallback never seeds today (D-04, SEEDER-04 -- writer ownership)
4. Seeder skips days where Parquet exists without .meta sidecar (SEEDER-04 -- kintsugi already processed)
5. REST-sourced files get .meta sidecar with source=REST (D-08)
6. Vision overwrites REST-sourced files when available (D-08)
7. Batch rotation processes 6 symbols per run with BTCUSDT/ETHUSDT priority (D-14)
8. Rate limit usage reported to cross-process coordinator (D-11)
9. All tests pass
   </success_criteria>

<output>
After completion, create `.planning/phases/05-seeder-rest-ratelimit/05-03-SUMMARY.md`
</output>
