---
phase: 260321-ql9
plan: 01
subsystem: reliability
tags: [guard-hardening, data-integrity, heartbeat, dead-letter, ring-buffer]

provides:
  - "Ring buffer capacity 100K prevents silent bar drops during fill_from_rest"
  - "Fatal _delete_today_bars crashes sidecar on failure instead of corrupting state"
  - "Dead-letter on INSERT failure in _do_replace prevents permanent data loss"
  - "Heartbeat today-gap awareness never suppressed by kintsugi_healing"
  - "stathera_today_gaps and stathera_today_overlaps gate health to CRITICAL"
affects: [sidecar, kintsugi, heartbeat, maintenance]

key-files:
  modified:
    - deploy/opendeviationbar.env
    - python/opendeviationbar/sidecar.py
    - python/opendeviationbar/clickhouse/maintenance.py
    - scripts/heartbeat/checks/stathera.py
    - scripts/heartbeat/config.py
    - scripts/heartbeat/formatter.py
    - tests/test_heartbeat_stathera.py
    - tests/test_heartbeat_health_levels.py

key-decisions:
  - "Ring buffer 100K capacity is ~50MB on bigblack's 64GB, negligible tradeoff for preventing silent bar drops"
  - "_delete_today_bars failure is FATAL: continuing with stale bars corrupts seed TIDs"
  - "Today's anomalies NEVER suppressed by kintsugi_healing because kintsugi can't touch today (#280)"
  - "Updated existing test_today_gap_suppressed to expect LOUD alert fires (was incorrectly testing old suppression behavior)"

requirements-completed: [P0-1a, P0-4, P0-2, P0-3]

duration: 4min
completed: 2026-03-22
---

# Quick Task 260321-ql9: Sprint 1 Guard Hardening P0 Fixes Summary

**4 P0 data-loss prevention fixes: ring buffer 100K capacity, fatal delete-today, dead-letter on INSERT failure, and writer-ownership-aware heartbeat today-gap alerts**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-22T02:15:17Z
- **Completed:** 2026-03-22T02:19:13Z
- **Tasks:** 4
- **Files modified:** 8

## Accomplishments

- Ring buffer capacity increased to 100K via env var, preventing silent bar drops during fill_from_rest (P0-1a)
- \_delete_today_bars failure now crashes sidecar instead of continuing with corrupted state (P0-4)
- INSERT failure in \_do_replace saves bars to dead-letter Parquet before re-raising, preventing permanent data loss (P0-2)
- Heartbeat distinguishes today's Stathera anomalies from historical ones; today-gap LOUD alert fires regardless of kintsugi_healing state (P0-3)
- 29 heartbeat tests passing (5 new + 1 updated + 23 existing)

## Task Commits

1. **Task 1: P0-1a ring buffer capacity + P0-4 delete crash-on-failure** - `7dd29b4` (fix)
2. **Task 2: P0-2 dead-letter on INSERT failure in \_do_replace** - `abf3a94` (fix)
3. **Task 3: P0-3 heartbeat today-gap awareness with writer-ownership health logic** - `affe37a` (feat)
4. **Task 4: Tests for P0-3 heartbeat today-gap awareness** - `ca76a6b` (test)

## Files Created/Modified

- `deploy/opendeviationbar.env` - Added OPENDEVIATIONBAR_MAX_PENDING_BARS=100000
- `python/opendeviationbar/sidecar.py` - Removed try/except from \_delete_today_bars (fatal on failure)
- `python/opendeviationbar/clickhouse/maintenance.py` - Added \_dead_letter_replace_failure + try/except in \_do_replace
- `scripts/heartbeat/checks/stathera.py` - Added today-only anomaly query + removed kintsugi_healing suppression from_check_today_gaps
- `scripts/heartbeat/config.py` - Added stathera_today_gaps and stathera_today_overlaps to CRITICAL_KEYS
- `scripts/heartbeat/formatter.py` - Added remediation hints for today keys
- `tests/test_heartbeat_stathera.py` - 4 new tests (TestTodayAnomalyAwareness + LOUD alert fires with kintsugi_healing)
- `tests/test_heartbeat_health_levels.py` - 2 new tests + updated critical keys assertion

## Decisions Made

- Ring buffer 100K is ~50MB on bigblack's 64GB: negligible memory for preventing silent bar drops
- \_delete_today_bars failure is FATAL because continuing with stale bars corrupts seed TIDs and causes overlapping bars
- Today's anomalies are NEVER suppressed by kintsugi_healing because kintsugi cannot touch today (#280 today-guard)
- Updated existing `test_today_gap_suppressed_when_kintsugi_healing` to expect LOUD alert to fire (old behavior was incorrect per #280)

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 1 - Bug] Updated existing test that asserted incorrect suppression behavior**

- **Found during:** Task 4
- **Issue:** `test_today_gap_suppressed_when_kintsugi_healing` asserted that LOUD alert is suppressed when kintsugi is healing, but P0-3 fix makes this incorrect (kintsugi can't fix today)
- **Fix:** Renamed to `test_today_gap_not_suppressed_when_kintsugi_healing` and updated assertions to expect LOUD alert fires
- **Files modified:** tests/test_heartbeat_stathera.py
- **Committed in:** ca76a6b (Task 4 commit)

---

**Total deviations:** 1 auto-fixed (1 bug - stale test assertion)
**Impact on plan:** Essential fix for test correctness. No scope creep.

## Issues Encountered

None.

## Known Stubs

None - all code paths are fully wired.

## User Setup Required

None - no external service configuration required.

## Next Steps

- Deploy to bigblack via `mise run deploy:bigblack`
- Monitor heartbeat for correct today-gap alerting behavior
- P0-1b (future): Add inline per-symbol drain in Rust for bounded memory regardless of fill size

---

_Quick Task: 260321-ql9_
_Completed: 2026-03-22_
