---
phase: 05-seeder-rest-ratelimit
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/ratelimit/__init__.py
  - python/opendeviationbar/ratelimit/file_coordinator.py
  - src/binance_bindings.rs
  - python/opendeviationbar/backfill.py
  - scripts/systemd/opendeviationbar-sidecar.service
  - scripts/systemd/opendeviationbar-kintsugi.service
  - tests/test_ratelimit_coordinator.py
autonomous: true
requirements: [SEEDER-02, SEEDER-03]

must_haves:
  truths:
    - "All three REST consumers can report and read aggregate weight usage via shared state file"
    - "Budget allocation enforces sidecar=2400, kintsugi=2000, seeder=800, reserve=800 totaling 6000"
    - "Stale consumer entries (process crash) expire after window+grace and do not block other consumers"
    - "Missing or corrupt state file is handled gracefully (default to 0 usage)"
    - "PyAdaptiveRateLimiter exposes acquire() and utilization() to Python callers (D-16)"
    - "Kintsugi wraps REST calls with rate limiter acquire before fetching (D-17)"
  artifacts:
    - path: "python/opendeviationbar/ratelimit/__init__.py"
      provides: "Package init with public exports"
      contains: "RateLimitCoordinator"
    - path: "python/opendeviationbar/ratelimit/file_coordinator.py"
      provides: "File-based cross-process rate limit coordinator"
      contains: "class RateLimitCoordinator"
    - path: "src/binance_bindings.rs"
      provides: "PyAdaptiveRateLimiter PyO3 class with acquire() and utilization()"
      contains: "PyAdaptiveRateLimiter"
    - path: "python/opendeviationbar/backfill.py"
      provides: "Kintsugi REST calls wrapped with rate limiter acquire"
      contains: "get_rest_rate_limiter_status"
    - path: "tests/test_ratelimit_coordinator.py"
      provides: "Tests for rate limit coordination"
  key_links:
    - from: "python/opendeviationbar/ratelimit/file_coordinator.py"
      to: "/tmp/opendeviationbar-ratelimit.json"
      via: "atomic rename write"
      pattern: "opendeviationbar-ratelimit\\.json"
    - from: "scripts/systemd/opendeviationbar-sidecar.service"
      to: "Binance rate limiter"
      via: "OPENDEVIATIONBAR_REST_CEILING=2400 env var"
      pattern: "REST_CEILING=2400"
    - from: "python/opendeviationbar/backfill.py"
      to: "src/binance_bindings.rs"
      via: "get_rest_rate_limiter_status() call before REST fetch"
      pattern: "get_rest_rate_limiter_status"
---

<objective>
Create cross-process REST API rate limit coordination via file-based state sharing, expose AdaptiveRateLimiter to Python via PyO3 (D-16), wrap kintsugi REST calls with rate limit acquire (D-17), and configure per-process budget ceilings in systemd services.

Purpose: Prevents sidecar, kintsugi, and seeder from collectively exceeding Binance's 6000 weight/min IP-wide budget (D-11, D-13). Each consumer reports its weight and reads aggregate state before acquiring budget. Per-process ceilings via env vars provide the hard cap (D-13). PyO3 binding exposes the Rust rate limiter to Python consumers (D-16), and kintsugi integrates it (D-17).

Output: RateLimitCoordinator class, PyAdaptiveRateLimiter PyO3 binding, kintsugi rate limit integration, systemd per-process ceiling env vars, test suite.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-seeder-rest-ratelimit/05-CONTEXT.md
@.planning/phases/05-seeder-rest-ratelimit/05-RESEARCH.md

@src/binance_bindings.rs
@python/opendeviationbar/backfill.py
@scripts/systemd/opendeviationbar-sidecar.service
@scripts/systemd/opendeviationbar-kintsugi.service

<interfaces>
<!-- Existing rate limiter singleton in binance_bindings.rs -->
From src/binance_bindings.rs:
```rust
static PYTHON_RATE_LIMITER: OnceLock<Arc<AdaptiveRateLimiter>> = OnceLock::new();
// Configured via:
// - OPENDEVIATIONBAR_REST_MAX_WEIGHT_PER_MINUTE (default 6000)
// - OPENDEVIATIONBAR_REST_SAFETY_MARGIN (default 0.20)
// - OPENDEVIATIONBAR_REST_CEILING (default 0 = no ceiling)

# [pyfunction]

pub fn get_rest_rate_limiter_status(py: Python) -> PyResult<PyObject>;
// Returns dict: {utilization, remaining, max_weight_per_minute, safety_margin}

````

<!-- AdaptiveRateLimiter Rust API (from rate_limiter.rs) -->
Key methods to expose via PyO3:
- `acquire(weight: u32)` -> blocks until budget available (async, needs tokio runtime)
- `utilization() -> f32` -> 0.0 to 1.0
- `remaining() -> u32` -> weight units remaining in current window
- `update_from_server(used_weight: u32)` -> sync with X-MBX-USED-WEIGHT-1m header

<!-- Budget allocation per D-13 -->
Budget: sidecar=2400, kintsugi=2000, seeder=800, reserve=800 (total=6000)
Each process sets OPENDEVIATIONBAR_REST_CEILING in its systemd service file.
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Create RateLimitCoordinator with file-based cross-process state sharing</name>
  <files>python/opendeviationbar/ratelimit/__init__.py, python/opendeviationbar/ratelimit/file_coordinator.py, tests/test_ratelimit_coordinator.py</files>
  <read_first>
    - src/binance_bindings.rs (lines 1-60, existing rate limiter singleton and get_rest_rate_limiter_status())
    - .planning/phases/05-seeder-rest-ratelimit/05-RESEARCH.md (Pattern 1: file-based coordination, lines 142-209)
  </read_first>
  <behavior>
    - Test 1: report_usage() creates state file at configured path with consumer entry
    - Test 2: get_aggregate_usage() returns sum of all consumer weights in current window
    - Test 3: Stale entries (updated >65s ago) are excluded from aggregate
    - Test 4: Window expiry (>60s since window_start) resets all counts to 0
    - Test 5: Corrupt/missing state file returns 0 aggregate (no crash)
    - Test 6: can_acquire(consumer_id, weight, budget) returns False when aggregate + weight > budget
    - Test 7: can_acquire() returns True when within budget
    - Test 8: Budget constants match D-13: SIDECAR=2400, KINTSUGI=2000, SEEDER=800, RESERVE=800
    - Test 9: Atomic rename write does not produce partial reads (simulate concurrent access)
  </behavior>
  <action>
    1. Create directory `python/opendeviationbar/ratelimit/` and `__init__.py`:
       ```python
       from .file_coordinator import RateLimitCoordinator, BUDGET_SIDECAR, BUDGET_KINTSUGI, BUDGET_SEEDER, BUDGET_RESERVE
       __all__ = ["RateLimitCoordinator", "BUDGET_SIDECAR", "BUDGET_KINTSUGI", "BUDGET_SEEDER", "BUDGET_RESERVE"]
       ```

    2. Create `python/opendeviationbar/ratelimit/file_coordinator.py` with class `RateLimitCoordinator` (per D-11):
       - Class attributes / module constants:
         ```python
         STATE_PATH = Path("/tmp/opendeviationbar-ratelimit.json")
         WINDOW_SECONDS = 60
         STALE_GRACE_SECONDS = 5  # entries older than WINDOW + GRACE are ignored
         BUDGET_SIDECAR = 2400     # per D-13
         BUDGET_KINTSUGI = 2000    # per D-13
         BUDGET_SEEDER = 800       # per D-13
         BUDGET_RESERVE = 800      # per D-13
         BUDGET_TOTAL = 6000       # per D-13
         ```
       - Constructor: `__init__(self, state_path: Path | None = None)` -- allows override for testing
       - Method `report_usage(consumer_id: str, weight_used: int) -> None`:
         - Read current state file (tolerate FileNotFoundError, json.JSONDecodeError -> fresh state)
         - If `time.time() - state["window_start"] > WINDOW_SECONDS`, reset state with new window_start
         - Update `state["consumers"][consumer_id] = {"weight_used": weight_used, "updated": time.time()}`
         - Atomic write via `tempfile.mkstemp(dir=state_path.parent)` + `os.write` + `os.fsync` + `os.rename`
         - Wrap in try/except, log warning on failure (never crash caller)
       - Method `get_aggregate_usage() -> int`:
         - Read state file (tolerate missing/corrupt -> return 0)
         - If window expired (>WINDOW_SECONDS), return 0
         - Sum `weight_used` for entries where `time.time() - entry["updated"] < WINDOW_SECONDS + STALE_GRACE_SECONDS`
       - Method `can_acquire(consumer_id: str, weight: int, budget: int) -> bool`:
         - Return `get_aggregate_usage() + weight <= budget`
       - Method `get_consumer_usage(consumer_id: str) -> int`:
         - Read state, return this consumer's weight_used (0 if not found or expired)
       - Method `get_state() -> dict`:
         - Return parsed state dict (for heartbeat telemetry). Tolerate missing/corrupt -> empty dict.

    3. Create `tests/test_ratelimit_coordinator.py` with the 9 test cases from `<behavior>`. Use `tmp_path` fixture for isolated state files. Use `time.time()` mocking where needed for stale/window tests (monkeypatch `time.time`).

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_ratelimit_coordinator.py -x -v</automated>
  </verify>
  <acceptance_criteria>
    - python/opendeviationbar/ratelimit/__init__.py exports RateLimitCoordinator
    - file_coordinator.py contains `class RateLimitCoordinator`
    - file_coordinator.py contains `BUDGET_SIDECAR = 2400`
    - file_coordinator.py contains `BUDGET_KINTSUGI = 2000`
    - file_coordinator.py contains `BUDGET_SEEDER = 800`
    - file_coordinator.py contains `BUDGET_RESERVE = 800`
    - file_coordinator.py contains `os.rename` (atomic write pattern)
    - file_coordinator.py contains `STALE_GRACE_SECONDS = 5`
    - tests/test_ratelimit_coordinator.py has 9+ test functions
    - All tests pass
  </acceptance_criteria>
  <done>RateLimitCoordinator provides cross-process rate limit state sharing via atomic JSON file with budget constants matching D-13.</done>
</task>

<task type="auto">
  <name>Task 2: Configure per-process REST ceiling env vars in sidecar and kintsugi systemd services</name>
  <files>scripts/systemd/opendeviationbar-sidecar.service, scripts/systemd/opendeviationbar-kintsugi.service</files>
  <read_first>
    - scripts/systemd/opendeviationbar-sidecar.service (current service file)
    - scripts/systemd/opendeviationbar-kintsugi.service (current service file)
    - scripts/systemd/opendeviationbar-kintsugi-catchup.service (reference, may also need ceiling)
  </read_first>
  <action>
    Per D-13 budget allocation, add per-process REST ceiling env vars to systemd service files. The existing Rust `PYTHON_RATE_LIMITER` singleton in `binance_bindings.rs` already reads `OPENDEVIATIONBAR_REST_CEILING` -- these env vars activate it.

    1. In `scripts/systemd/opendeviationbar-sidecar.service`, add in the `[Service]` section:
       ```ini
       Environment=OPENDEVIATIONBAR_REST_CEILING=2400
       ```

    2. In `scripts/systemd/opendeviationbar-kintsugi.service`, add in the `[Service]` section:
       ```ini
       Environment=OPENDEVIATIONBAR_REST_CEILING=2000
       ```

    3. In `scripts/systemd/opendeviationbar-kintsugi-catchup.service`, add in the `[Service]` section:
       ```ini
       Environment=OPENDEVIATIONBAR_REST_CEILING=2000
       ```
       (Same as kintsugi -- catchup is the same role with Conflicts= to prevent concurrent run.)

    Note: seeder ceiling (800) was already added in Plan 01, Task 2.
  </action>
  <verify>
    <automated>grep -c "OPENDEVIATIONBAR_REST_CEILING=2400" /Users/terryli/eon/opendeviationbar-py/scripts/systemd/opendeviationbar-sidecar.service && grep -c "OPENDEVIATIONBAR_REST_CEILING=2000" /Users/terryli/eon/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi.service</automated>
  </verify>
  <acceptance_criteria>
    - opendeviationbar-sidecar.service contains `OPENDEVIATIONBAR_REST_CEILING=2400`
    - opendeviationbar-kintsugi.service contains `OPENDEVIATIONBAR_REST_CEILING=2000`
    - opendeviationbar-kintsugi-catchup.service contains `OPENDEVIATIONBAR_REST_CEILING=2000`
  </acceptance_criteria>
  <done>All three REST consumer processes have per-process ceiling env vars: sidecar=2400, kintsugi=2000, seeder=800 (Plan 01). Combined 5200 < 6000 with 800 reserve (D-13).</done>
</task>

<task type="auto">
  <name>Task 3: Expose PyAdaptiveRateLimiter via PyO3 and wrap kintsugi REST calls (D-16, D-17)</name>
  <files>src/binance_bindings.rs, python/opendeviationbar/backfill.py</files>
  <read_first>
    - src/binance_bindings.rs (full file -- existing rate limiter singleton, get_rest_rate_limiter_status)
    - crates/opendeviationbar-providers/src/binance/rate_limiter.rs (AdaptiveRateLimiter API: acquire, utilization, remaining, update_from_server)
    - python/opendeviationbar/backfill.py (full file -- find REST fetch call sites that need rate limit wrapping)
    - src/lib.rs (PyO3 module registration -- where to add PyAdaptiveRateLimiter)
  </read_first>
  <action>
    **Part 1: PyO3 binding for AdaptiveRateLimiter (D-16)**

    In `src/binance_bindings.rs`, add a new `#[pyclass]` wrapper:

    ```rust
    #[pyclass]
    pub struct PyAdaptiveRateLimiter {
        inner: Arc<AdaptiveRateLimiter>,
    }

    #[pymethods]
    impl PyAdaptiveRateLimiter {
        /// Get the per-process singleton rate limiter.
        #[staticmethod]
        fn get_instance() -> Self {
            Self { inner: get_python_rate_limiter() }
        }

        /// Block until `weight` units of budget are available, then consume them.
        /// Uses a tokio runtime internally since acquire() is async in Rust.
        fn acquire(&self, py: Python, weight: u32) -> PyResult<()> {
            py.allow_threads(|| {
                let rt = tokio::runtime::Runtime::new()
                    .map_err(|e| PyRuntimeError::new_err(format!("Runtime error: {e}")))?;
                rt.block_on(self.inner.acquire(weight))
                    .map_err(|e| PyRuntimeError::new_err(format!("Acquire error: {e}")))?;
                Ok(())
            })
        }

        /// Current utilization as a fraction 0.0-1.0.
        fn utilization(&self) -> f32 {
            self.inner.utilization()
        }

        /// Remaining weight units in the current window.
        fn remaining(&self) -> u32 {
            self.inner.remaining()
        }
    }
    ```

    Register `PyAdaptiveRateLimiter` in `src/lib.rs` module init:
    ```rust
    m.add_class::<binance_bindings::PyAdaptiveRateLimiter>()?;
    ```

    **Part 2: Wrap kintsugi REST calls with rate limiter (D-17)**

    In `python/opendeviationbar/backfill.py`, locate the REST fetch call sites (where `fetch_aggtrades_rest` or `fetch_aggtrades_parallel` are called). Before each REST call, add rate limit checking:

    ```python
    # Before REST fetch (D-17: kintsugi wraps REST calls with rate limiter)
    try:
        from opendeviationbar._core import get_rest_rate_limiter_status
        status = get_rest_rate_limiter_status()
        utilization = status.get("utilization", 0.0)
        if utilization > 0.85:
            # Near budget ceiling -- sleep until window refills rather than retry-storm
            import time
            remaining = status.get("remaining", 0)
            logger.info("Rate limiter at %.0f%% utilization (%d remaining), waiting for refill",
                       utilization * 100, remaining)
            time.sleep(max(5, 60 - int(60 * (1 - utilization))))
    except ImportError:
        pass  # Pre-build, rate limiter not available
    ```

    This is a lightweight integration: kintsugi checks utilization before each REST call and sleeps if near budget ceiling. The actual acquire() is handled internally by the Rust `fetch_aggtrades_parallel` / `fetch_aggtrades_rest` functions which already use the singleton rate limiter. The Python-side check adds cross-process awareness via the ceiling env var.

    Note: The actual weight per aggTrades request is 4 (not 1 as D-17 states -- see Pitfall 6 in RESEARCH.md). The Rust side already handles this correctly. The Python wrapper just needs to respect the ceiling.

    Run `maturin develop` after Rust changes to rebuild the PyO3 bindings.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && maturin develop 2>&1 | tail -3 && uv run --python 3.13 python -c "from opendeviationbar._core import PyAdaptiveRateLimiter; rl = PyAdaptiveRateLimiter.get_instance(); print(f'utilization={rl.utilization()}, remaining={rl.remaining()}')"</automated>
  </verify>
  <acceptance_criteria>
    - src/binance_bindings.rs contains `pub struct PyAdaptiveRateLimiter`
    - src/binance_bindings.rs contains `fn acquire(&self`
    - src/binance_bindings.rs contains `fn utilization(&self)`
    - src/lib.rs registers PyAdaptiveRateLimiter class
    - python/opendeviationbar/backfill.py contains `get_rest_rate_limiter_status` call before REST fetch
    - `maturin develop` builds successfully
    - `PyAdaptiveRateLimiter.get_instance()` returns a usable object from Python
  </acceptance_criteria>
  <done>PyAdaptiveRateLimiter exposed to Python via PyO3 (D-16), kintsugi backfill checks utilization before REST calls (D-17).</done>
</task>

</tasks>

<verification>
- `uv run --python 3.13 pytest tests/test_ratelimit_coordinator.py -x -v` passes
- `grep "OPENDEVIATIONBAR_REST_CEILING" scripts/systemd/*.service` shows correct values for all services
- `python -c "from opendeviationbar.ratelimit import RateLimitCoordinator; print('OK')"` imports successfully
- `python -c "from opendeviationbar._core import PyAdaptiveRateLimiter; print('OK')"` imports successfully
- `grep "get_rest_rate_limiter_status" python/opendeviationbar/backfill.py` confirms kintsugi integration
</verification>

<success_criteria>
1. RateLimitCoordinator class exists with report_usage(), get_aggregate_usage(), can_acquire(), get_state() methods
2. Budget constants: SIDECAR=2400, KINTSUGI=2000, SEEDER=800, RESERVE=800 (D-13)
3. Atomic rename write pattern (no partial reads)
4. Stale entry expiry (>65s ignored)
5. All systemd services have per-process OPENDEVIATIONBAR_REST_CEILING configured
6. PyAdaptiveRateLimiter PyO3 class exposes acquire(), utilization(), remaining() (D-16)
7. Kintsugi backfill checks utilization before REST calls (D-17)
8. All tests pass
</success_criteria>

<output>
After completion, create `.planning/phases/05-seeder-rest-ratelimit/05-02-SUMMARY.md`
</output>
````
