---
status: investigating
trigger: "Investigate REST API rate limiting on bigblack — historical telemetry, root causes of Binance API budget exhaustion during fill_from_rest, and options for smarter rate management"
created: 2026-03-22T00:00:00Z
updated: 2026-03-22T00:00:00Z
---

## Current Focus

hypothesis: Confirmed — fill_from_rest budget math makes exhaustion inevitable with 16 symbols
test: Code analysis of rate limiter, parallel_fetch, and fill_from_rest complete
expecting: N/A — root cause analysis confirmed from code
next_action: Compile analysis report with options for improvement

## Symptoms

expected: fill_from_rest for 16 symbols should complete within a single Binance rate limit window (4800 weight/min), taking ~2-3 minutes total
actual: fill_from_rest exhausts the 4800-weight budget within seconds, hitting "rate limiter: budget exhausted, waiting for window reset" repeatedly. Multiple rate limit windows needed (5-8 minutes total).
errors: WARN logs: "rate limiter: budget exhausted, waiting for window reset weight=4 used=4800 budget=4800 wait_ms=51xxx"
reproduction: Every sidecar restart triggers fill_from_rest for all 16 symbols. Each parallel_fetch chunk fires 50 pages regardless of data density. With 4 symbols filling concurrently (semaphore=4), budget exhausts in ~10 seconds.
started: Present since fill_from_rest was introduced. Worse with 16 symbols (was 2 originally).

## Eliminated

## Evidence

- timestamp: 2026-03-22
  checked: rate_limiter.rs — budget model
  found: |
  Budget: 6000 weight/min \* (1 - 0.20 safety) = 4800 usable.
  Sidecar ceiling: OPENDEVIATIONBAR_REST_CEILING=3000 (sidecar.env), but StreamManager
  constructor creates AdaptiveRateLimiter::new(6000, 0.20) WITHOUT ceiling parameter.
  fill_from_rest() dynamically sets ceiling to 4800 during fill, then 3000 after.
  Window reset: 60-second local timer. acquire() sleeps for remainder of window when budget exhausted.
  Server feedback: X-MBX-USED-WEIGHT-1m header synced via update_from_headers(), uses MAX(server, local).
  implication: The StreamManager rate limiter is NOT the Python OnceLock limiter — it's a separate instance created per-engine without the env var ceiling. But fill_from_rest sets ceiling=4800 explicitly.

- timestamp: 2026-03-22
  checked: parallel_fetch.rs — weight per request
  found: |
  Each aggTrades REST call costs 4 weight (Spot endpoint).
  PAGE_SIZE=1000, DEFAULT_CONCURRENCY=5.
  Early termination (Issue #297): AtomicBool flags when a page returns <1000 trades.
  Tasks check flag before and after acquiring semaphore permit.
  Weight saved = pages_skipped \* 4 per chunk.
  implication: Early termination helps but only within a single 50K-trade chunk. The outer loop in fill_from_rest fires multiple chunks per symbol.

- timestamp: 2026-03-22
  checked: live_engine.rs — fill_from_rest budget math
  found: |
  fill_from_rest() architecture:
  - Semaphore(4): up to 4 symbols fill concurrently
  - Per-symbol: chunked REST pagination with CHUNK_SIZE=50,000 trades
  - Each chunk = 50 pages of 1000 trades = 50 REST calls = 200 weight
  - With 4 symbols concurrent: 4 \* 200 = 800 weight per chunk-round
  - Budget 4800 / 800 = 6 chunk-rounds before exhaustion
  - At concurrency 5 per parallel_fetch: ~1-2 seconds per chunk
  - So budget exhausts in ~6-12 seconds → 60s wait → repeat

    Termination: 3 consecutive empty chunks or 3 consecutive errors.
    After fill: ceiling lowered from 4800 to 3000.
    implication: Budget exhaustion is EXPECTED with this architecture. 4 concurrent symbols \* 50 pages each = 200 REST calls = 800 weight in ~2 seconds. 6 rounds exhausts 4800 weight.

- timestamp: 2026-03-22
  checked: Consumer analysis — who uses REST API weight
  found: |
  1. fill_from_rest (sidecar startup): HEAVIEST. 16 symbols _multiple chunks_ 50 pages each.
     Ceiling temporarily raised to 4800 during fill.
  2. puck_fill (inline gap fill): Uses parallel_fetch with range [last_known_id, first_ws_id).
     Typically small gaps (5-10K trades) = 5-10 pages = 20-40 weight. Runs AFTER fill completes.
  3. kintsugi: SEPARATE PROCESS with own rate limiter (ceiling=2400 via env var).
     Actually uses Vision ZIP (zero API weight) or local Parquet as primary source.
     REST only as fallback. Does NOT share sidecar's rate limiter instance.
  4. shadow_validation: 1 REST call per symbol per interval (weight=1 each, limit=1).
     Adaptive: skips when utilization > 60%.
  5. midnight_preflight: 1 REST call per symbol (fetch_latest_aggtrade, weight=1).
     implication: fill_from_rest is the sole heavy consumer during startup. kintsugi is separate process/limiter. puck_fill shares the same limiter but runs after fill completes.

- timestamp: 2026-03-22
  checked: Rate limiter architecture — cross-process coordination
  found: |
  TWO separate rate limiters exist:
  1. Python OnceLock: binance_bindings.rs PYTHON_RATE_LIMITER — used by all Python→Rust REST calls.
     Configured via OPENDEVIATIONBAR_REST_CEILING env var (kintsugi=2400).
  2. StreamManager: stream_manager_bindings.rs creates new AdaptiveRateLimiter::new(6000, 0.20)
     per engine creation — NO ceiling from env var. fill_from_rest dynamically adjusts.
     These two limiters are INDEPENDENT — they don't share state.
     Binance's X-MBX-USED-WEIGHT-1m header provides cross-process ground truth via update_from_headers().
     But window resets are local (each limiter tracks its own 60s window).
     implication: If kintsugi and sidecar run simultaneously and both make REST calls, they each track their own budget independently. Server header feedback provides partial coordination but window timing differs.

## Resolution

root_cause: |
Budget exhaustion is an expected consequence of the fill_from_rest architecture:

1. 16 symbols _50-page chunks_ 4 weight/page = theoretical max 3200 weight per chunk-round
2. Semaphore(4) limits to 4 concurrent symbols, but each symbol's parallel_fetch has concurrency=5
3. Effective: 4 symbols _~50 pages_ 4 weight = ~800 weight bursts in 1-2 seconds
4. Budget (4800) exhausts in ~6 rounds = ~6-12 seconds
5. Then waits 48-54 seconds for window reset → repeat
6. With early termination saving some pages, actual throughput is better but still multi-window

The rate limiter is WORKING CORRECTLY — it prevents 429 errors from Binance.
The issue is that fill_from_rest's demand exceeds single-window capacity.
This is a throughput optimization opportunity, not a bug.

fix:
verification:
files_changed: []
