---
id: SEED-002
title: Migrate ODB to System-Level Services + Production Hardening
planted: 2026-03-24
trigger: next milestone or immediately if another outage occurs
status: dormant
tags: [systemd, isolation, blast-radius, crash-loop, production]
dependencies:
  - sudo access on bigblack (for /etc/systemd/system/)
  - dedicated odb user account (useradd odb)
---

# SEED-002: System-Level Service Migration + Production Hardening

## The Incident (2026-03-24)

A broken `whisper-api-health-monitor.service` (not ODB-related) crash-looped every 10s → systemd killed the entire user@1001 slice → ALL ODB services died for 1h47m. No OOM, no reboot — just a rogue service in the same user slice.

## What Needs to Change

### 1. System-Level Services (Blast Radius Isolation)
Move all ODB services from `~/.config/systemd/user/` to `/etc/systemd/system/`:
- `opendeviationbar-sidecar.service`
- `opendeviationbar-kintsugi.service`
- `opendeviationbar-kintsugi-catchup.service`
- `opendeviationbar-heartbeat.service` + `.timer`
- `opendeviationbar-seeder.service` + `.timer`
- `opendeviationbar-exness-seeder.service` + `.timer`

Run as `User=odb` (dedicated user). System services survive user slice crashes.

### 2. Heartbeat Service Audit
New heartbeat check: scan `systemctl --user list-unit-files` for non-ODB services with `Restart=always`. Alert on Telegram if a rogue crash-looping service is detected. Prevents silent cascade.

### 3. Crash-Loop Circuit Breaker
Add to ALL ODB service files:
```ini
StartLimitBurst=5
StartLimitIntervalSec=60
StartLimitAction=none
```
This means: if ODB itself crash-loops 5 times in 60s, systemd stops restarting it — but does NOT cascade-kill other services.

### 4. cc-skill: systemd-production-hardening
Universal skill covering: isolation patterns, crash-loop prevention, service inventory hygiene, migration checklist from --user to system-level.

## When to Surface
- Immediately as v3.0 Phase 0 (pre-requisite before feature work)
- Or on next outage caused by user slice cascade
