# Phase 5: Seeder REST Integration & Rate Limit Coordination - Discussion Log

> **Audit trail only.** Do not use as input to planning, research, or execution agents.
> Decisions are captured in CONTEXT.md -- this log preserves the alternatives considered.

**Date:** 2026-03-22
**Phase:** 05-seeder-rest-ratelimit
**Areas discussed:** REST fallback strategy, Rate limit coordination, Seeder scheduling, AdaptiveRateLimiter adoption, Telemetry, Overlap prevention, Parquet integrity, Rust-first architecture, Blind spots
**Mode:** Round 1 --auto, Round 2 interactive (blind spot analysis + user confirmation)

---

## Round 1: Initial Gray Areas (--auto)

### REST Fallback Strategy

| Option                      | Description                                              | Selected |
| --------------------------- | -------------------------------------------------------- | -------- |
| Vision-first, REST-fallback | Try Vision CSV, fall back to REST for 404s (recent days) | ✓        |
| REST-only for recent days   | Skip Vision for today/yesterday, always use REST         |          |
| Parallel Vision+REST        | Download both, use whichever completes first             |          |

**User's choice:** [auto] Vision-first, REST-fallback (recommended default)

### Rate Limit Coordination Mechanism

| Option                       | Description                                        | Selected     |
| ---------------------------- | -------------------------------------------------- | ------------ |
| File-based shared state      | Atomic JSON file in /tmp, all consumers read/write | ✓            |
| Redis atomic counters        | Shared Redis counter per consumer                  | upgrade path |
| Per-process ceilings only    | Each process assumes fixed budget, no coordination |              |
| Centralized rate limit proxy | HTTP proxy that enforces budget                    |              |

**User's choice:** [auto] File-based with Redis upgrade path (recommended default)

### Budget Allocation

| Option                  | Description                                 | Selected |
| ----------------------- | ------------------------------------------- | -------- |
| 2400/2000/800/800       | Sidecar P0, Kintsugi P1, Seeder P2, Reserve | ✓        |
| Equal split (1500 each) | Treats all consumers equally                |          |
| Dynamic priority        | Highest-priority consumer gets remainder    |          |

**User's choice:** [auto] Priority-weighted allocation (recommended default)

### Seeder Scheduling

| Option                            | Description                                | Selected |
| --------------------------------- | ------------------------------------------ | -------- |
| Same 5-min timer, dual path       | Single process does Vision + REST fallback | ✓        |
| Separate REST timer               | Independent timer for REST seeding         |          |
| On-demand (triggered by kintsugi) | Kintsugi triggers seeder when Parquet miss |          |

**User's choice:** [auto] Same 5-min timer, dual path (recommended default)

### AdaptiveRateLimiter Adoption

| Option                         | Description                                      | Selected |
| ------------------------------ | ------------------------------------------------ | -------- |
| PyO3 bindings for Rust limiter | Expose AdaptiveRateLimiter to Python via PyO3    | ✓        |
| Python reimplementation        | Rewrite rate limiter in Python                   |          |
| Keep stamina, add budget check | Add file-based budget check before stamina retry |          |

**User's choice:** [auto] PyO3 bindings (recommended default)

---

## Round 2: Blind Spot Analysis (interactive)

User requested: "Help me think of all the blind spots that disaster might happen and give me more options that I can choose from."

### Blind Spot 1: Seeder Overlap

| Option                         | Description                         | Selected |
| ------------------------------ | ----------------------------------- | -------- |
| A. `flock` in Python           | Lock file at script entry           |          |
| B. systemd `Conflicts=`        | OS-level guard (same as kintsugi)   | ✓        |
| C. Rust `flock` + orchestrator | Rust binary with process-level lock | future   |

**User's choice:** B now, C later (accepted recommendation)

### Blind Spot 2: Parquet Completeness

| Option                            | Description                                    | Selected |
| --------------------------------- | ---------------------------------------------- | -------- |
| A. Trade-ID continuity check      | Verify trade_id[i+1] == trade_id[i] + 1        | ✓        |
| B. Expected count from REST       | Compare against REST total                     |          |
| C. Mark REST as "partial"         | Metadata flag, Vision overwrite when available | ✓        |
| D. Rust Stathera post-write audit | Full validation in Rust                        | future   |

**User's choice:** A + C combined (accepted recommendation)

### Blind Spot 3: Partial Day Seeding

| Option                      | Description                               | Selected |
| --------------------------- | ----------------------------------------- | -------- |
| A. Never seed today         | Yesterday only, today is sidecar's domain | ✓        |
| B. Seed today, mark partial | Re-seed each run, append/replace          |          |
| C. Seed today at 23:55 UTC  | Single near-midnight seed                 |          |
| D. Don't seed today at all  | Kintsugi doesn't need today's ticks       |          |

**User's choice:** A (accepted recommendation)

### Blind Spot 4: Rust-First Architecture

| Option                               | Description                                   | Selected |
| ------------------------------------ | --------------------------------------------- | -------- |
| A. Rust library, Python orchestrator | PyO3 for heavy lifting, Python for scheduling | ✓        |
| B. Full Rust binary                  | New opendeviationbar-seeder binary            | deferred |
| C. Incremental migration             | Move fetch+write to Rust first                |          |

**User's choice:** A (accepted recommendation)

### Blind Spot 5: Vision vs REST Mismatch

| Option                    | Description                         | Selected |
| ------------------------- | ----------------------------------- | -------- |
| A. Accept the risk        | Trust empirical verification        |          |
| B. Periodic oracle check  | Weekly cron comparing row counts    | ✓        |
| C. On-seed cross-validate | Compare when both sources available |          |

**User's choice:** B (accepted recommendation)

### Blind Spot 6: Rate Limit Budget Math

| Option                               | Description                        | Selected |
| ------------------------------------ | ---------------------------------- | -------- |
| A. 800 weight/min, accept slow       | May take 4-5 min per run           |          |
| B. Increase to 1200, reduce kintsugi | Faster seeding                     |          |
| C. Dynamic budget borrowing          | Borrow from idle kintsugi          |          |
| D. Batch rotation (6 symbols/run)    | Full cycle = 15 min, within budget | ✓        |

**User's choice:** D (accepted recommendation)

---

## Round 3: Additional Insights

Three additional optimizations surfaced during deep research:

1. Route Parquet writes through `write_ticks()` for dedup (D-09)
2. Cache last-seeded date to reduce 52K stat() → 360 (D-20)
3. Four new heartbeat metrics (D-22-D-25)

**User's choice:** Accepted all three

---

## Claude's Discretion

- File locking strategy for shared state file
- Exact PyO3 API surface for AdaptiveRateLimiter
- Seeder REST retry policy details
- ZSTD compression level env var

## Deferred Ideas

- Full 18-symbol streaming (separate milestone)
- WebSocket-based tick seeding (if REST budget insufficient)
- Multi-IP rate limiting (if 6000 weight/min insufficient)
- Full Rust binary for seeder (after Phase 5 proves PyO3 approach)
