---
phase: 22-monitor-hardening-schema
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/heartbeat/checks/services.py
  - python/opendeviationbar/sidecar/__init__.py
  - scripts/systemd/opendeviationbar-sidecar.service
autonomous: false
requirements: [MON-05]

must_haves:
  truths:
    - "Sidecar restart root cause is identified with evidence from bigblack logs"
    - "Fix is deployed and sidecar holds a stable PID for >1 hour under normal operation"
  artifacts:
    - path: "scripts/heartbeat/checks/services.py"
      provides: "Sidecar PID age and restart frequency reporting"
      contains: "sidecar.*pid.*age\\|restart.*count"
  key_links:
    - from: "scripts/heartbeat/checks/services.py"
      to: "systemctl --user show opendeviationbar-sidecar"
      via: "systemd property query"
      pattern: "NRestarts|MainPID"
---

<objective>
Investigate and fix sidecar restart frequency (5 PIDs/hour observed by A2 audit).

Purpose: Frequent restarts cause clean-slate DELETE + fill_from_rest on every startup, wasting REST API budget and creating transient Stathera gaps. Target: stable PID for >1 hour under normal operation.
Output: Root cause analysis, fix deployed, heartbeat restart telemetry added.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

@python/opendeviationbar/sidecar/**init**.py
@scripts/systemd/opendeviationbar-sidecar.service
@scripts/heartbeat/checks/services.py

<interfaces>
From scripts/systemd/opendeviationbar-sidecar.service:
```ini
StartLimitIntervalSec=600
StartLimitBurst=5
Restart=always
RestartSec=10
MemoryMax=4G
```

Sidecar watchdog config (from .mise.toml):

```
OPENDEVIATIONBAR_STREAMING_WATCHDOG_TIMEOUT_S = "600"   # 10 min stale = restart
OPENDEVIATIONBAR_STREAMING_MAX_WATCHDOG_RESTARTS = "3"  # 3 internal restarts then exit
```

Possible restart causes to investigate:

1. OOM kill (MemoryMax=4G, peak during Puck gap-fill ~2G, but 18 symbols may be higher)
2. Watchdog timeout (600s no trades = engine restart, 3 restarts = process exit)
3. Unhandled exception in main loop
4. Python memory watermark abort (90% of 4G = 3.6G)
   </interfaces>

</context>

<tasks>

<task type="checkpoint:human-action" gate="blocking">
  <name>Task 1: Investigate sidecar restart root cause on bigblack</name>
  <files>scripts/heartbeat/checks/services.py</files>
  <action>
Run these diagnostic commands on bigblack via SSH and collect the output:

1. Check recent sidecar restarts and exit codes:
   journalctl --user -u opendeviationbar-sidecar --since "6 hours ago" | grep -E "Started|Stopped|Main process exited|oom-kill|memory|WATCHDOG|max restarts" | tail -30

2. Check systemd restart count and current PID age:
   systemctl --user show opendeviationbar-sidecar --property=NRestarts,MainPID,ActiveEnterTimestamp,Result

3. Check for OOM kills:
   journalctl --user -u opendeviationbar-sidecar --since "6 hours ago" | grep -iE "oom|killed|memory|cgroup" | tail -10

4. Current memory usage:
   systemctl --user status opendeviationbar-sidecar | head -15

5. Check if watchdog triggers are frequent:
   journalctl --user -u opendeviationbar-sidecar --since "6 hours ago" | grep -c "WATCHDOG"

Paste all output so the executor can determine the root cause and apply the appropriate fix.
</action>
<verify>
<automated>echo "checkpoint: manual investigation required"</automated>
</verify>
<done>Root cause of sidecar restarts identified with supporting evidence from bigblack logs</done>
</task>

<task type="auto">
  <name>Task 2: Add sidecar restart telemetry to heartbeat and apply fix (MON-05)</name>
  <files>scripts/heartbeat/checks/services.py</files>
  <read_first>scripts/heartbeat/checks/services.py</read_first>
  <action>
Add sidecar PID age and restart count reporting to the heartbeat service check. This provides ongoing visibility regardless of root cause.

In `collect_service_stats()` (or equivalent function in services.py), add:

1. Query `systemctl --user show opendeviationbar-sidecar --property=NRestarts,MainPID,ActiveEnterTimestamp` via subprocess
2. Compute PID age = now - ActiveEnterTimestamp
3. Store in results dict:
   - `results["sidecar_pid_age_min"]` = PID age in minutes (string, for display)
   - `results["sidecar_restarts_total"]` = NRestarts count (string, for display)
4. If PID age < 15 minutes AND NRestarts > 0, set `results["sidecar_restart_freq"]` = "frequent" (informational, not health-gating yet)

This provides the telemetry baseline needed to validate any fix and detect future regressions.

Note: The actual root cause fix depends on Task 1 findings. If the root cause is:

- **OOM**: Raise MemoryMax in the service file (e.g., 4G -> 6G for 18-symbol streaming)
- **Watchdog**: Increase watchdog_timeout_s or investigate WS connectivity
- **Exception**: Fix the unhandled exception path
- **Memory watermark**: Adjust OPENDEVIATIONBAR_MEMORY_MAX_BYTES to match new MemoryMax

The executor should apply the appropriate fix based on Task 1 findings before completing this task.
</action>
<verify>
<automated>grep -n "sidecar_pid_age\|NRestarts\|sidecar_restarts" scripts/heartbeat/checks/services.py | head -5</automated>
</verify>
<done>Heartbeat reports sidecar PID age and restart count. Root cause fix applied based on Task 1 investigation.</done>
</task>

</tasks>

<verification>
1. `grep -n "NRestarts\|pid_age" scripts/heartbeat/checks/services.py` shows telemetry collection
2. After fix deployed: sidecar PID stable for >1 hour on bigblack
3. Heartbeat Telegram message shows sidecar PID age in minutes
</verification>

<success_criteria>

- Root cause of 5 PIDs/hour identified with evidence from bigblack logs
- Fix applied (appropriate to root cause)
- Heartbeat reports sidecar PID age and restart count for ongoing monitoring
- Sidecar holds stable PID for >1 hour post-fix
  </success_criteria>

<output>
After completion, create `.planning/phases/22-monitor-hardening-schema/22-02-SUMMARY.md`
</output>
