# systemd Services & Timers

**Parent**: [/scripts/CLAUDE.md](/scripts/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

All services run as `--user` units on bigblack. Managed via `mise run deploy:bigblack`.

---

## Services (4 units)

| Service                             | Type            | Purpose                                     | Memory (High/Max) |
| ----------------------------------- | --------------- | ------------------------------------------- | ----------------- |
| `opendeviationbar-sidecar`          | simple (daemon) | Real-time streaming via WebSocket           | 2G / 2.5G         |
| `opendeviationbar-kintsugi`         | notify (daemon) | Unified gap reconciler + trailing-edge fill | 16G / 20G         |
| `opendeviationbar-kintsugi-catchup` | notify (daemon) | Aggressive historical gap repair            | 16G / 20G         |
| `opendeviationbar-heartbeat`        | oneshot (timer) | Unified health + Stathera audit to Telegram | — / 256M          |

## Timers (1)

| Timer                              | Interval    | Service   |
| ---------------------------------- | ----------- | --------- |
| `opendeviationbar-heartbeat.timer` | Every 5 min | heartbeat |

---

## Dependency Chain

```
sidecar → kintsugi
```

Kintsugi uses `After=opendeviationbar-sidecar.service`. The heartbeat timer is independent (runs regardless of daemon state — that's what it monitors).

**Conflicts**: `kintsugi-catchup` conflicts with `kintsugi` — only one runs at a time. The `Conflicts=` directive in the unit file ensures systemd stops one when the other starts.

---

## Environment Files

All services load TWO env files:

1. `deploy/opendeviationbar.env` — git-tracked, non-secret config
2. `deploy/opendeviationbar.local.env` — **NOT** git-tracked, secrets (Telegram token, etc.)

The `-` prefix in `EnvironmentFile=-` means "ignore if missing" (used for `local.env` so services start even without secrets).

---

## mimalloc

Long-running services (sidecar, kintsugi) use:

```ini
Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmimalloc.so.2
Environment=MIMALLOC_PURGE_DELAY=1000
```

Prevents glibc ptmalloc2 RSS creep by returning freed memory to the OS. Without this, long-running Python+Rust processes accumulate unreclaimable RSS over days.

---

## Timezone (Issue #206)

All services set `Environment=TZ=UTC` to prevent naive-datetime bugs regardless of the host's system timezone. This is defense-in-depth — bigblack's system TZ is also UTC (`timedatectl set-timezone UTC`) and ClickHouse has an explicit `<timezone>UTC</timezone>` in `/etc/clickhouse-server/config.d/timezone.xml`.

---

## OOM Protection

All long-running services use `MemorySwapMax=0` (no swap escape) and `ManagedOOMMemoryPressure=kill` for graceful systemd-oomd degradation via PSI instead of hard kernel OOM kill.

Priority order via `OOMScoreAdjust`:

1. sidecar (-500) — highest priority, latency-critical
2. kintsugi / kintsugi-catchup (-200) — background repair + trailing-edge fill
3. oneshot services (0, default) — heartbeat

---

## Deploy

`mise run deploy:bigblack` syncs service files, enables timers, and restarts all services.

---

## Related

- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) - Operations scripts, gap detection, kintsugi
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache, circuit breaker
- [/CLAUDE.md](/CLAUDE.md) - Project hub
