---
phase: 22-monitor-hardening-schema
plan: 02
subsystem: monitoring
tags: [heartbeat, systemd, sidecar, restart-telemetry]

requires:
  - phase: 18-monitor-validation
    provides: Heartbeat service checks framework
provides:
  - Sidecar PID age and restart count in heartbeat telemetry
  - Root cause analysis: operator-induced restarts, not a bug
affects: [heartbeat-formatter, sidecar-operations]

tech-stack:
  added: []
  patterns: [systemd NRestarts property query for restart telemetry]

key-files:
  created: []
  modified:
    - scripts/heartbeat/checks/services.py

key-decisions:
  - "No sidecar code fix needed: 5 PIDs/hour was operator-induced during v7.0 deploy"
  - "Added telemetry for ongoing visibility rather than fixing a non-bug"

patterns-established:
  - "MON-05 pattern: PID age + restart count telemetry for detecting future instability"

requirements-completed: [MON-05]

duration: 3min
completed: 2026-03-27
---

# Phase 22 Plan 02: Sidecar Restart Investigation + Telemetry Summary

**Sidecar restart root cause identified as operator-induced (v7.0 deploy); PID age + restart count telemetry added to heartbeat**

## Performance

- **Duration:** 3 min (including checkpoint wait)
- **Started:** 2026-03-27T19:59:24Z
- **Completed:** 2026-03-27T20:05:00Z
- **Tasks:** 2 (1 checkpoint:human-action, 1 auto)
- **Files modified:** 1

## Accomplishments

- Root cause identified: 5 PIDs/hour was from repeated deploy/restart cycles during v7.0 audit session, not organic instability
- Sidecar memory healthy at 1.5G/4G (37%), zero OOM kills
- PID age and restart count now reported in heartbeat for ongoing monitoring
- Frequent restart flag (`sidecar_restart_freq=frequent`) triggers when PID age <15min with NRestarts >0

## Task Commits

1. **Task 1: Investigate sidecar restart root cause** - checkpoint:human-action (no commit, diagnostic only)
2. **Task 2: Add sidecar restart telemetry to heartbeat** - `09dbd9f` (feat)

## Files Created/Modified

- `scripts/heartbeat/checks/services.py` - Added NRestarts query, sidecar_pid_age_min, sidecar_restarts_total, sidecar_restart_freq to collect_service_stats()

## Decisions Made

- No sidecar code fix needed: root cause was operator-induced restarts during v7.0 deploy + fleet recompute + hotfixes
- Added telemetry-only solution: PID age and restart count provide ongoing visibility for detecting future genuine instability

## Deviations from Plan

None - plan executed as written. Root cause investigation revealed no bug, so the "apply fix based on root cause" portion of Task 2 was correctly skipped.

## Issues Encountered

None

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- All MON-05 telemetry in place
- Heartbeat will surface any future restart instability automatically

## Self-Check: PASSED

---

_Phase: 22-monitor-hardening-schema_
_Completed: 2026-03-27_
