---
phase: 05-seeder-rest-ratelimit
plan: 04
type: execute
wave: 2
depends_on: [05-02]
files_modified:
  - scripts/heartbeat/checks/seeder.py
  - scripts/heartbeat/checks/ratelimit.py
  - scripts/heartbeat/checks/vision_oracle.py
  - scripts/health_heartbeat.py
  - tests/test_heartbeat_ratelimit.py
  - tests/test_heartbeat_seeder.py
autonomous: true
requirements: [SEEDER-02, SEEDER-03]

must_haves:
  truths:
    - "Heartbeat reports per-consumer REST utilization and aggregate across sidecar, kintsugi, seeder"
    - "LOUD Telegram alert fires when aggregate REST utilization exceeds 90% sustained for 3 consecutive checks"
    - "ratelimit_ok participates in the healthy gate (False when sustained 90%+)"
    - "Heartbeat reports days-behind per symbol as the primary actionable metric"
    - "Seeder heartbeat reports Vision 404 count and REST fallback count"
    - "Weekly Vision vs REST oracle check alerts on >0.1% mismatch (D-10)"
  artifacts:
    - path: "scripts/heartbeat/checks/ratelimit.py"
      provides: "Aggregate REST API utilization check"
      contains: "def check_ratelimit"
    - path: "scripts/heartbeat/checks/seeder.py"
      provides: "Enhanced seeder metrics: days-behind, Vision 404, REST fallback"
      contains: "days_behind"
    - path: "scripts/heartbeat/checks/vision_oracle.py"
      provides: "Weekly Vision vs REST trade count comparison (D-10)"
      contains: "def check_vision_oracle"
    - path: "tests/test_heartbeat_ratelimit.py"
      provides: "Tests for rate limit heartbeat check including healthy gate"
    - path: "tests/test_heartbeat_seeder.py"
      provides: "Tests for seeder heartbeat enhancements"
  key_links:
    - from: "scripts/heartbeat/checks/ratelimit.py"
      to: "python/opendeviationbar/ratelimit/file_coordinator.py"
      via: "RateLimitCoordinator.get_state()"
      pattern: "RateLimitCoordinator"
    - from: "scripts/heartbeat/checks/ratelimit.py"
      to: "opendeviationbar._core.get_rest_rate_limiter_status"
      via: "per-process utilization from Rust singleton"
      pattern: "get_rest_rate_limiter_status"
    - from: "scripts/heartbeat/checks/vision_oracle.py"
      to: "opendeviationbar._core.fetch_aggtrades_rest"
      via: "REST trade count for comparison"
      pattern: "fetch_aggtrades_rest"
---

<objective>
Add heartbeat telemetry for aggregate REST API utilization across all consumers, enhance seeder metrics with days-behind and REST fallback tracking, and implement weekly Vision vs REST oracle check (D-10).

Purpose: Provides operational visibility into REST budget health (D-22, D-24) and surfaces symbols falling behind before they become kintsugi bottlenecks (D-25). LOUD alert at 90% sustained prevents 429 ban cascades (D-22). Vision oracle catches systematic Vision/REST divergence (D-10).

Output: ratelimit.py heartbeat check, enhanced seeder.py check, vision_oracle.py check, test suites.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-seeder-rest-ratelimit/05-CONTEXT.md
@.planning/phases/05-seeder-rest-ratelimit/05-02-SUMMARY.md

@scripts/heartbeat/checks/seeder.py
@scripts/health_heartbeat.py

<interfaces>
<!-- From Plan 02: RateLimitCoordinator for reading cross-process state -->
From python/opendeviationbar/ratelimit/file_coordinator.py:
```python
class RateLimitCoordinator:
    def get_aggregate_usage(self) -> int: ...
    def get_consumer_usage(self, consumer_id: str) -> int: ...
    def get_state(self) -> dict: ...
    # state dict shape: {"consumers": {"sidecar": {"weight_used": N, "updated": T}, ...}, "window_start": T}

BUDGET_TOTAL = 6000

````

<!-- Existing Rust rate limiter status binding -->
From src/binance_bindings.rs:
```python
def get_rest_rate_limiter_status() -> dict:
    # Returns: {"utilization": 0.0-1.0, "remaining": int, "max_weight_per_minute": int, "safety_margin": float}
````

<!-- Existing heartbeat check pattern -->

From scripts/heartbeat/checks/seeder.py:

```python
def check_seeder(results: dict[str, bool | str]) -> None:
    # results["seeder_ok"] = bool (participates in healthy gate)
    # results["seeder_*"] = str (informational)
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Create ratelimit heartbeat check for aggregate REST utilization</name>
  <files>scripts/heartbeat/checks/ratelimit.py, tests/test_heartbeat_ratelimit.py</files>
  <read_first>
    - scripts/heartbeat/checks/seeder.py (full file -- existing heartbeat check pattern)
    - scripts/health_heartbeat.py (lines 1-50 -- how checks are called, results dict pattern, how ratelimit_ok feeds into healthy gate)
    - python/opendeviationbar/ratelimit/file_coordinator.py (get_state() return format)
  </read_first>
  <behavior>
    - Test 1: check_ratelimit() sets ratelimit_ok=True when aggregate is below 80%
    - Test 2: check_ratelimit() sets ratelimit_ok=True when aggregate is between 80-90% (warning only)
    - Test 3: check_ratelimit() sets ratelimit_ok=False when aggregate exceeds 90% for 3+ consecutive checks
    - Test 4: check_ratelimit() reports per-consumer breakdown (sidecar_weight, kintsugi_weight, seeder_weight)
    - Test 5: check_ratelimit() handles missing state file gracefully (ratelimit_ok=True, utilization=0)
    - Test 6: Sustained threshold counter resets when aggregate drops below 90%
    - Test 7: ratelimit_ok=False causes the results dict healthy gate to be False (verify wiring - Warning 3 fix)
  </behavior>
  <action>
    Create `scripts/heartbeat/checks/ratelimit.py` following the existing heartbeat check pattern (per D-22):

    ```python
    """REST API rate limit utilization check for heartbeat monitoring."""
    from __future__ import annotations
    import logging
    from pathlib import Path

    logger = logging.getLogger("health-heartbeat")

    # Track consecutive high-utilization checks for LOUD alert (D-22)
    _SUSTAINED_HIGH_COUNT_FILE = Path("/var/tmp/opendeviationbar-ratelimit-sustained.txt")
    _SUSTAINED_THRESHOLD = 3  # 3 consecutive checks at 90%+ = LOUD alert
    _ALERT_THRESHOLD = 0.90   # 90% of total budget


    def check_ratelimit(results: dict[str, bool | str]) -> None:
        """Check aggregate REST API utilization across all consumers (D-22, D-24).

        ``ratelimit_ok`` is a boolean and participates in the healthy gate.
        A sustained >90% utilization means 429 bans are imminent -- flag as UNHEALTHY.
        """
        try:
            from opendeviationbar.ratelimit import RateLimitCoordinator
            coord = RateLimitCoordinator()
            state = coord.get_state()
        except Exception:
            # Rate limit module not available -- not a health issue
            results["ratelimit_ok"] = True
            results["ratelimit_status"] = "unavailable"
            return

        consumers = state.get("consumers", {})
        aggregate = sum(c.get("weight_used", 0) for c in consumers.values())
        budget_total = 6000
        utilization = aggregate / budget_total if budget_total > 0 else 0.0

        # Per-consumer breakdown (D-22)
        for consumer_id in ("sidecar", "kintsugi", "seeder"):
            weight = consumers.get(consumer_id, {}).get("weight_used", 0)
            results[f"ratelimit_{consumer_id}_weight"] = str(weight)

        results["ratelimit_aggregate"] = str(aggregate)
        results["ratelimit_utilization"] = f"{utilization:.1%}"

        # Also read this process's own rate limiter status (Rust singleton)
        try:
            from opendeviationbar._core import get_rest_rate_limiter_status
            local_status = get_rest_rate_limiter_status()
            results["ratelimit_local_utilization"] = f"{local_status.get('utilization', 0):.1%}"
        except Exception:
            pass

        # Sustained high utilization check (D-22)
        if utilization >= _ALERT_THRESHOLD:
            count = _load_sustained_count() + 1
            _save_sustained_count(count)
            if count >= _SUSTAINED_THRESHOLD:
                results["ratelimit_ok"] = False
                results["ratelimit_alert"] = f"SUSTAINED {utilization:.0%} for {count} checks"
                return
        else:
            _save_sustained_count(0)  # reset on drop below threshold

        results["ratelimit_ok"] = True


    def _load_sustained_count() -> int:
        try:
            return int(_SUSTAINED_HIGH_COUNT_FILE.read_text().strip())
        except (FileNotFoundError, ValueError, OSError):
            return 0


    def _save_sustained_count(count: int) -> None:
        try:
            _SUSTAINED_HIGH_COUNT_FILE.write_text(str(count))
        except OSError:
            pass
    ```

    Create `tests/test_heartbeat_ratelimit.py` with the 7 test cases from `<behavior>`. Mock `RateLimitCoordinator.get_state()` to return controlled state dicts. Use `tmp_path` for sustained count file. For Test 7 (Warning 3 fix): create a results dict, call check_ratelimit() with mocked 95% utilization for 3 consecutive checks, then verify `results["ratelimit_ok"] is False` and that any code using `all(v for k, v in results.items() if k.endswith("_ok") and isinstance(v, bool))` would return False.

    Wire `check_ratelimit` into `scripts/health_heartbeat.py`: After the existing check calls, add:
    ```python
    from heartbeat.checks.ratelimit import check_ratelimit
    check_ratelimit(results)
    ```
    Verify that `ratelimit_ok` is included in the healthy gate computation (the existing pattern uses `all(v for k, v in results.items() if k.endswith("_ok"))` or similar).

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_heartbeat_ratelimit.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - scripts/heartbeat/checks/ratelimit.py contains `def check_ratelimit(results`
    - ratelimit.py contains `_ALERT_THRESHOLD = 0.90`
    - ratelimit.py contains `_SUSTAINED_THRESHOLD = 3`
    - ratelimit.py sets `ratelimit_ok` as bool in results
    - ratelimit.py reports `ratelimit_sidecar_weight`, `ratelimit_kintsugi_weight`, `ratelimit_seeder_weight`
    - ratelimit.py reports `ratelimit_aggregate` and `ratelimit_utilization`
    - tests/test_heartbeat_ratelimit.py has 7+ test functions including healthy gate wiring test
    - health_heartbeat.py contains `from scripts.heartbeat.checks.ratelimit import check_ratelimit` (or equivalent import)
    - health_heartbeat.py contains `check_ratelimit(results)` call
    - All tests pass
  </acceptance_criteria>
  <done>Heartbeat reports per-consumer and aggregate REST utilization, fires LOUD alert on sustained 90%+ usage, ratelimit_ok gates healthy status.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 2: Enhance seeder heartbeat check with days-behind, Vision 404 count, and budget metrics</name>
  <files>scripts/heartbeat/checks/seeder.py, tests/test_heartbeat_seeder.py</files>
  <read_first>
    - scripts/heartbeat/checks/seeder.py (full file -- existing check_seeder function)
    - scripts/tick_cache_seeder.py (state file format -- understand what fields are written)
  </read_first>
  <behavior>
    - Test 1 (test_days_behind_zero): When all Parquet files exist for last 7 days, seeder_days_behind = "0"
    - Test 2 (test_days_behind_nonzero): When 3 of last 7 Parquet files are missing, seeder_days_behind = "3"
    - Test 3 (test_vision_404_count): seeder_vision_404 parsed from state file kv
    - Test 4 (test_rest_fallback_count): seeder_rest_fallback parsed from state file kv
    - Test 5 (test_budget_used): seeder_budget_used reports from RateLimitCoordinator
    - Test 6 (test_days_behind_does_not_gate_ok): seeder_ok remains True even when days_behind > 0
  </behavior>
  <action>
    Enhance `check_seeder()` in `scripts/heartbeat/checks/seeder.py` to report new metrics (per D-24, D-25):

    1. **Days-behind per symbol** (D-25 -- primary actionable metric):
       After parsing the state file kv pairs, compute days-behind. The seeder state file already writes per-symbol seeded counts. Add a new section that reads the Parquet cache to compute how many days are missing for each symbol:
       ```python
       # Days-behind per symbol (D-25)
       try:
           from opendeviationbar.storage.parquet import TickStorage
           from opendeviationbar.symbol_registry import get_registered_symbols
           from datetime import date, timedelta, UTC, datetime

           storage = TickStorage()
           today = datetime.now(UTC).date()
           yesterday = today - timedelta(days=1)
           symbols = get_registered_symbols(enabled_only=True)

           max_behind = 0
           behind_symbols = []
           for sym in symbols[:6]:  # Check top-6 by volume to stay fast
               # Check last 7 days for gaps
               missing_count = 0
               for d in range(1, 8):
                   check_date = today - timedelta(days=d)
                   day_path = storage._get_parquet_path(sym, check_date.isoformat())
                   if not day_path.exists():
                       missing_count += 1
               if missing_count > 0:
                   behind_symbols.append(f"{sym}:{missing_count}d")
                   max_behind = max(max_behind, missing_count)

           results["seeder_days_behind"] = str(max_behind)
           if behind_symbols:
               results["seeder_behind_symbols"] = " | ".join(behind_symbols)
       except Exception:
           pass  # non-critical diagnostic
       ```

    2. **Vision 404 and REST fallback counts** (D-24):
       Parse from the seeder state file. The modified seeder (Plan 03) will write these:
       ```python
       vision_404 = kv.get("vision_404_count", "")
       rest_fallback = kv.get("rest_fallback_count", "")
       if vision_404:
           results["seeder_vision_404"] = vision_404
       if rest_fallback:
           results["seeder_rest_fallback"] = rest_fallback
       ```

    3. **Seeder REST budget utilization** (D-24):
       ```python
       # Budget utilization from rate limit coordinator
       try:
           from opendeviationbar.ratelimit import RateLimitCoordinator
           coord = RateLimitCoordinator()
           seeder_weight = coord.get_consumer_usage("seeder")
           results["seeder_budget_used"] = f"{seeder_weight}/800"
       except Exception:
           pass
       ```

    Place the days-behind calculation AFTER the existing state file parsing (after line 110) but before the final `results["seeder_ok"]` assignment. The days-behind metric is informational only -- it does NOT gate `seeder_ok` (which stays gated on timer freshness and failed count).

    Create `tests/test_heartbeat_seeder.py` with 6 test cases from `<behavior>`. Mock `TickStorage._get_parquet_path` to return controlled paths. Use `tmp_path` to create/omit Parquet files. Mock `RateLimitCoordinator.get_consumer_usage` for budget test. Verify seeder_ok is not gated by days_behind.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_heartbeat_seeder.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - seeder.py contains `results["seeder_days_behind"]`
    - seeder.py contains `results["seeder_behind_symbols"]`
    - seeder.py contains `results["seeder_vision_404"]`
    - seeder.py contains `results["seeder_rest_fallback"]`
    - seeder.py contains `results["seeder_budget_used"]`
    - days-behind check reads from TickStorage (not ClickHouse)
    - days-behind is informational only (does NOT gate seeder_ok)
    - tests/test_heartbeat_seeder.py has 6+ test functions
    - All tests pass
  </acceptance_criteria>
  <done>Seeder heartbeat reports days-behind (primary actionable metric per D-25), Vision 404 count, REST fallback count, and per-consumer budget utilization. All verified by automated tests.</done>
</task>

<task type="auto" tdd="true">
  <name>Task 3: Create weekly Vision vs REST oracle check (D-10)</name>
  <files>scripts/heartbeat/checks/vision_oracle.py, tests/test_heartbeat_ratelimit.py</files>
  <read_first>
    - scripts/heartbeat/checks/ratelimit.py (heartbeat check pattern from Task 1)
    - scripts/heartbeat/checks/seeder.py (existing heartbeat check pattern)
    - scripts/health_heartbeat.py (lines 1-50 -- how checks are registered in the heartbeat loop, must add check_vision_oracle call)
    - python/opendeviationbar/storage/parquet.py (read_ticks() for getting local Parquet trade counts)
  </read_first>
  <behavior>
    - Test 1: check_vision_oracle() skips if last check was < 7 days ago (weekly cadence)
    - Test 2: check_vision_oracle() compares trade counts and sets vision_oracle_ok=True when mismatch < 0.1%
    - Test 3: check_vision_oracle() sets vision_oracle_ok=False and vision_oracle_alert when mismatch > 0.1%
    - Test 4: check_vision_oracle() handles missing Parquet files gracefully (skip that day)
  </behavior>
  <action>
    Create `scripts/heartbeat/checks/vision_oracle.py` implementing D-10 (periodic Vision vs REST oracle check):

    ```python
    """Weekly Vision vs REST trade count oracle check (D-10).

    Compares trade counts from local Parquet (Vision-sourced) against REST API
    for a sample of days/symbols. Alerts if mismatch exceeds 0.1%.
    """
    from __future__ import annotations
    import json
    import logging
    import random
    import time
    from datetime import UTC, datetime, timedelta
    from pathlib import Path

    logger = logging.getLogger("health-heartbeat")

    _LAST_CHECK_FILE = Path("/var/tmp/opendeviationbar-vision-oracle-last.json")
    _CHECK_INTERVAL_DAYS = 7
    _MISMATCH_THRESHOLD = 0.001  # 0.1%
    _SAMPLE_DAYS = 5
    _SAMPLE_SYMBOLS = ["BTCUSDT", "ETHUSDT", "SOLUSDT"]


    def check_vision_oracle(results: dict[str, bool | str]) -> None:
        """Compare Vision (Parquet) vs REST trade counts weekly (D-10).

        ``vision_oracle_ok`` participates in the healthy gate only when a check runs.
        On non-check weeks, set True (no-op).
        """
        # Check if we've run recently
        try:
            last = json.loads(_LAST_CHECK_FILE.read_text())
            last_ts = last.get("checked_at", 0)
            if time.time() - last_ts < _CHECK_INTERVAL_DAYS * 86400:
                results["vision_oracle_ok"] = True
                results["vision_oracle_status"] = "skipped (checked recently)"
                return
        except (FileNotFoundError, json.JSONDecodeError, OSError):
            pass  # First run or corrupt state

        try:
            from opendeviationbar.storage.parquet import TickStorage
            from opendeviationbar._core import fetch_aggtrades_rest

            storage = TickStorage()
            today = datetime.now(UTC).date()

            # Sample random completed days (8-30 days ago, Vision-confirmed range)
            candidate_days = [today - timedelta(days=d) for d in range(8, 31)]
            sample_days = random.sample(candidate_days, min(_SAMPLE_DAYS, len(candidate_days)))

            mismatches = []
            checked = 0

            for sym in _SAMPLE_SYMBOLS:
                for check_date in sample_days:
                    date_str = check_date.isoformat()
                    parquet_path = storage._get_parquet_path(sym, date_str)
                    if not parquet_path.exists():
                        continue

                    # Get Vision (Parquet) trade count
                    import polars as pl
                    vision_count = pl.scan_parquet(parquet_path).select(pl.len()).collect().item()

                    # Get REST trade count (just first+last ID, compute count from range)
                    day_start_ms = int(datetime.combine(check_date, datetime.min.time(), tzinfo=UTC).timestamp() * 1000)
                    day_end_ms = day_start_ms + 86_400_000 - 1

                    first_trades = fetch_aggtrades_rest(sym, day_start_ms, day_start_ms + 60_000)
                    last_trades = fetch_aggtrades_rest(sym, day_end_ms - 60_000, day_end_ms)
                    if not first_trades or not last_trades:
                        continue

                    rest_count = last_trades[-1]["a"] - first_trades[0]["a"] + 1

                    # Compare
                    if rest_count > 0:
                        mismatch = abs(vision_count - rest_count) / rest_count
                        if mismatch > _MISMATCH_THRESHOLD:
                            mismatches.append(f"{sym}/{date_str}: vision={vision_count} rest={rest_count} ({mismatch:.2%})")
                    checked += 1

            # Save check timestamp
            try:
                _LAST_CHECK_FILE.write_text(json.dumps({
                    "checked_at": time.time(),
                    "checked_count": checked,
                    "mismatches": len(mismatches),
                }))
            except OSError:
                pass

            if mismatches:
                results["vision_oracle_ok"] = False
                results["vision_oracle_alert"] = f"{len(mismatches)} mismatches: " + "; ".join(mismatches[:3])
            else:
                results["vision_oracle_ok"] = True
                results["vision_oracle_status"] = f"checked {checked} pairs, all within 0.1%"

        except Exception as exc:
            results["vision_oracle_ok"] = True  # Don't block on oracle failure
            results["vision_oracle_status"] = f"error: {exc}"
    ```

    Add Vision oracle tests to `tests/test_heartbeat_ratelimit.py` (append to existing file since it's the heartbeat test file):
    - Use monkeypatch to control `_LAST_CHECK_FILE` path (use tmp_path)
    - Mock `TickStorage._get_parquet_path`, `fetch_aggtrades_rest`, and `polars.scan_parquet`

    Wire into `scripts/health_heartbeat.py`:
    ```python
    from heartbeat.checks.vision_oracle import check_vision_oracle
    check_vision_oracle(results)
    ```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_heartbeat_ratelimit.py -x -v -k "vision_oracle"</automated>
  </verify>
  <acceptance_criteria>
    - scripts/heartbeat/checks/vision_oracle.py contains `def check_vision_oracle(results`
    - vision_oracle.py contains `_MISMATCH_THRESHOLD = 0.001`
    - vision_oracle.py contains `_CHECK_INTERVAL_DAYS = 7`
    - vision_oracle.py compares Parquet trade count against REST trade count
    - vision_oracle.py sets `vision_oracle_ok` and `vision_oracle_alert` on mismatch
    - health_heartbeat.py contains `check_vision_oracle(results)` call
    - Tests for weekly skip, match, mismatch, and missing file handling pass
  </acceptance_criteria>
  <done>Weekly Vision vs REST oracle check implemented (D-10), alerts on >0.1% trade count mismatch across 5 sampled days and 3 symbols.</done>
</task>

</tasks>

<verification>
- `uv run --python 3.13 pytest tests/test_heartbeat_ratelimit.py -x -v` passes
- `uv run --python 3.13 pytest tests/test_heartbeat_seeder.py -x -v` passes
- `grep "check_ratelimit" scripts/heartbeat/checks/ratelimit.py` finds the function
- `grep "check_vision_oracle" scripts/heartbeat/checks/vision_oracle.py` finds the function
- `grep "days_behind" scripts/heartbeat/checks/seeder.py` confirms new metric
- `grep "budget_used" scripts/heartbeat/checks/seeder.py` confirms budget reporting
</verification>

<success_criteria>

1. Heartbeat reports aggregate REST utilization across all consumers (D-22)
2. LOUD alert fires on sustained 90%+ utilization for 3 consecutive checks (D-22)
3. ratelimit_ok participates in healthy gate (Warning 3 fix)
4. Per-consumer weight breakdown visible in heartbeat (D-22)
5. Days-behind per symbol reported as primary actionable metric (D-24, D-25)
6. Vision 404 and REST fallback counts reported (D-24)
7. Weekly Vision vs REST oracle check with >0.1% mismatch alert (D-10)
8. All tests pass with automated pytest verification
   </success_criteria>

<output>
After completion, create `.planning/phases/05-seeder-rest-ratelimit/05-04-SUMMARY.md`
</output>
