---
phase: 05-seeder-rest-ratelimit
plan: 04
subsystem: monitoring
tags: [heartbeat, ratelimit, vision-oracle, telemetry, telegram]

# Dependency graph
requires:
  - phase: 05-02
    provides: RateLimitCoordinator file-based cross-process coordination
provides:
  - Aggregate REST API utilization heartbeat check with LOUD alert on sustained 90%+
  - Enhanced seeder metrics: days-behind, Vision 404 count, REST fallback count, budget usage
  - Weekly Vision vs REST oracle check (D-10) with 0.1% mismatch threshold
affects: [deploy, scripts/systemd]

# Tech tracking
tech-stack:
  added: []
  patterns:
    [
      heartbeat check module pattern,
      sustained threshold counter via file,
      weekly cadence oracle check,
    ]

key-files:
  created:
    - scripts/heartbeat/checks/ratelimit.py
    - scripts/heartbeat/checks/vision_oracle.py
    - tests/test_heartbeat_ratelimit.py
    - tests/test_heartbeat_seeder.py
  modified:
    - scripts/heartbeat/checks/__init__.py
    - scripts/heartbeat/checks/seeder.py
    - scripts/heartbeat/config.py

key-decisions:
  - "ratelimit_ok added to WARNING_KEYS (not CRITICAL) -- 429 risk is operational, not data integrity"
  - "days_behind is informational only, does NOT gate seeder_ok -- seeder health gated by timer freshness and failures"
  - "Vision oracle uses weekly cadence with 0.1% mismatch threshold to minimize REST API budget impact"

patterns-established:
  - "Sustained threshold counter: file-based counter that increments per check cycle and resets on recovery"
  - "Weekly oracle check: last-check-file JSON with checked_at timestamp for cadence control"

requirements-completed: [SEEDER-02, SEEDER-03]

# Metrics
duration: 8min
completed: 2026-03-22
---

# Phase 05 Plan 04: Heartbeat Telemetry Summary

**Aggregate REST utilization monitoring with sustained-90% LOUD alert, seeder days-behind metric, and weekly Vision vs REST oracle check**

## Performance

- **Duration:** 8 min
- **Started:** 2026-03-22T23:26:34Z
- **Completed:** 2026-03-22T23:34:59Z
- **Tasks:** 3
- **Files modified:** 7

## Accomplishments

- Heartbeat reports per-consumer and aggregate REST utilization (sidecar/kintsugi/seeder weight breakdown)
- LOUD alert fires on sustained 90%+ utilization for 3 consecutive checks, preventing 429 ban cascades
- Seeder heartbeat reports days-behind as primary actionable metric (D-25)
- Weekly Vision vs REST oracle check compares trade counts across 5 sampled days and 3 symbols

## Task Commits

Each task was committed atomically:

1. **Task 1: Create ratelimit heartbeat check** - `5043c3a` (feat)
2. **Task 2: Enhance seeder heartbeat** - `b15d1ad` (feat)
3. **Task 3: Create Vision oracle check** - `c01e5a5` (feat)

## Files Created/Modified

- `scripts/heartbeat/checks/ratelimit.py` - Aggregate REST utilization check with sustained threshold
- `scripts/heartbeat/checks/vision_oracle.py` - Weekly Vision vs REST trade count comparison
- `scripts/heartbeat/checks/seeder.py` - Enhanced with days-behind, Vision 404, REST fallback, budget
- `scripts/heartbeat/checks/__init__.py` - Wired new checks into orchestrator
- `scripts/heartbeat/config.py` - Added ratelimit_ok to WARNING_KEYS
- `tests/test_heartbeat_ratelimit.py` - 11 tests (7 ratelimit + 4 vision oracle)
- `tests/test_heartbeat_seeder.py` - 6 tests for enhanced seeder metrics

## Decisions Made

- ratelimit_ok added to WARNING_KEYS (not CRITICAL) -- 429 risk is operational, not data integrity
- days_behind is informational only, does NOT gate seeder_ok -- seeder health gated by timer freshness and failures
- Vision oracle uses weekly cadence with 0.1% mismatch threshold to minimize REST API budget impact

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all metrics are wired to live data sources (RateLimitCoordinator, TickStorage, seeder state file).

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Phase 05 complete: all 4 plans executed
- REST rate limiting, tick cache seeder, kintsugi guard, and monitoring telemetry all implemented
- Ready for production deployment and v1.4 milestone verification

## Self-Check: PASSED

All 7 files verified present. All 3 commit hashes found in git log.

---

_Phase: 05-seeder-rest-ratelimit_
_Completed: 2026-03-22_
