---
phase: 05-seeder-rest-ratelimit
plan: 03
subsystem: scripts
tags:
  [
    seeder,
    rest-fallback,
    batch-rotation,
    kintsugi-guard,
    parquet,
    rate-limiting,
  ]

# Dependency graph
requires:
  - phase: 05-01
    provides: "write_ticks() dedup gate, trade-ID continuity validation"
  - phase: 05-02
    provides: "RateLimitCoordinator for cross-process REST weight reporting"
provides:
  - "REST API fallback for yesterday when Vision 404s (two-step fetch pattern)"
  - "Batch rotation processing 6 symbols per run with BTCUSDT/ETHUSDT priority"
  - "Kintsugi-processed guard skipping days with Parquet but no .meta sidecar"
  - ".meta sidecar marking REST-sourced files for Vision overwrite"
  - "Rate limit usage reporting to cross-process coordinator"
affects: [05-04]

# Tech tracking
tech-stack:
  added: []
  patterns:
    [
      "Two-step REST fetch: time-based lookup for trade ID boundaries then parallel bulk",
      "Batch rotation with persistent state file for 5-min timer runs",
      ".meta sidecar pattern for REST-vs-Vision source tracking",
    ]

key-files:
  created:
    - tests/test_seeder_rest_fallback.py
  modified:
    - scripts/tick_cache_seeder.py
    - python/opendeviationbar/config/seeder.py

key-decisions:
  - "REST fallback uses Rust bindings (full column names) not raw REST single-letter keys -- avoids rename mapping entirely"
  - "Kintsugi guard uses .meta sidecar presence/absence rather than querying ClickHouse -- O(1) file check per day"
  - "Batch rotation state stored in JSON file, not in-memory -- survives systemd timer restarts"

patterns-established:
  - ".meta sidecar files mark REST-sourced Parquet for future Vision overwrite"
  - "Batch rotation with priority symbols front-loaded in ordered list"

requirements-completed: [SEEDER-01, SEEDER-04]

# Metrics
duration: 6min
completed: 2026-03-22
---

# Phase 05 Plan 03: REST Fallback + Batch Rotation Summary

**REST API fallback for yesterday's Vision gaps using two-step fetch, 6-symbol batch rotation with BTCUSDT priority, and kintsugi-processed day guard**

## Performance

- **Duration:** 6 min
- **Started:** 2026-03-22T23:18:17Z
- **Completed:** 2026-03-22T23:24:22Z
- **Tasks:** 2
- **Files modified:** 3

## Accomplishments

- Seeder falls back to REST API when Vision returns 404 for days within rest_fallback_max_age_days (default 2)
- Two-step fetch pattern: fetch_aggtrades_rest for time-based ID lookup, then fetch_aggtrades_parallel for bulk
- Batch rotation processes 6 symbols per 5-min timer run with BTCUSDT/ETHUSDT always first
- Kintsugi-processed guard skips days where Parquet exists without .meta (already processed by kintsugi/Vision)
- 15 test cases covering all REST fallback paths, batch rotation, and kintsugi guard

## Task Commits

1. **Task 1: REST config + fallback + batch rotation** - `92e3e9f` (feat)
2. **Task 2: Test suite for REST fallback** - `ba2641d` (test)

## Files Created/Modified

- `python/opendeviationbar/config/seeder.py` - Added rest_fallback_enabled, batch_size, rest_fallback_max_age_days, last_seeded_state_file
- `scripts/tick_cache_seeder.py` - REST fallback, batch rotation, kintsugi guard, .meta sidecar, rate limit reporting
- `tests/test_seeder_rest_fallback.py` - 15 tests for all new functionality

## Decisions Made

- REST fallback uses Rust binding output directly (full column names like `agg_trade_id`) rather than mapping from single-letter REST keys -- the plan suggested a rename map from `a`/`p`/`q` but the Rust bindings already normalize to full names
- Kintsugi guard uses `.meta` sidecar file presence rather than ClickHouse queries -- keeps the guard O(1) per day with zero network cost
- Batch rotation state persisted to JSON file on disk so it survives systemd timer restarts

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 1 - Bug] Fixed REST trade column name mapping**

- **Found during:** Task 1 (REST fallback implementation)
- **Issue:** Plan specified renaming single-letter keys (a, p, q, T, m) from REST response, but Rust bindings already return full column names (agg_trade_id, price, quantity, timestamp, is_buyer_maker)
- **Fix:** Removed the rename mapping; used Rust binding output directly with type casting only
- **Files modified:** scripts/tick_cache_seeder.py
- **Verification:** Tests pass with mock data using full column names
- **Committed in:** 92e3e9f (Task 1 commit)

---

**Total deviations:** 1 auto-fixed (1 bug fix)
**Impact on plan:** Essential correctness fix. Plan was based on raw REST API response format, but the seeder calls Rust bindings which normalize column names.

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Known Stubs

None - all functionality is fully wired.

## Next Phase Readiness

- REST fallback and batch rotation ready for integration testing
- Plan 04 (systemd integration) can configure the new env vars

---

_Phase: 05-seeder-rest-ratelimit_
_Completed: 2026-03-22_
