# Deploy Configuration

**Parent**: [/CLAUDE.md](/CLAUDE.md) | **Related**: [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md)

Environment files, monit watchdog configuration, and deployment scripts for bigblack.

---

## Environment Files (SSoT)

| File                         | Git-tracked | Purpose                               | Deployed To                     |
| ---------------------------- | ----------- | ------------------------------------- | ------------------------------- |
| `opendeviationbar.env`       | Yes         | Non-secret config (SSoT for bigblack) | `~/opendeviationbar-py/deploy/` |
| `opendeviationbar.local.env` | **No**      | Secrets: Telegram TOKEN, Redis pw     | `~/opendeviationbar-py/deploy/` |
| `sidecar.env`                | Yes         | Streaming-specific vars               | `~/opendeviationbar-py/deploy/` |

Services load both env files via `EnvironmentFile=` in each `.service` unit. The `-` prefix (`EnvironmentFile=-path`) means "ignore if missing" — used for `local.env` so services start even without secrets.

**To edit bigblack config**: Edit `deploy/opendeviationbar.env` locally, then `mise run deploy:bigblack` to sync. Never edit on bigblack directly — it will be overwritten on next deploy.

---

## opendeviationbar.env — Key Variables

```ini
# Connection mode
OPENDEVIATIONBAR_MODE=local
OPENDEVIATIONBAR_CH_HOSTS=localhost

# Ouroboros mode (Aion = continuous stream, no resets)
OPENDEVIATIONBAR_OUROBOROS_MODE=aion

# Redis write lock (Issue #266: short timeout, fall through to L2 dedup)
OPENDEVIATIONBAR_REDIS_HOST=localhost
OPENDEVIATIONBAR_REDIS_LOCK_BLOCKING_TIMEOUT=2

# Telegram alert destination
OPENDEVIATIONBAR_TELEGRAM_CHAT_ID=90417581

# Kintsugi parallelism (see scripts/CLAUDE.md for details)
OPENDEVIATIONBAR_KINTSUGI_MAX_REPAIRS_PER_PASS=500
OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REPAIRS=24  # empirical sweet spot; 32 = cliff (0 repairs/30s)
OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_PARQUET=64
OPENDEVIATIONBAR_KINTSUGI_BATCH_WORKERS=1
OPENDEVIATIONBAR_KINTSUGI_BATCH_DAYS=365

# Parquet seeder (Issue #270)
OPENDEVIATIONBAR_SEEDER_HISTORICAL_START=2018-01-01
OPENDEVIATIONBAR_SEEDER_MAX_CONCURRENT_DOWNLOADS=12
OPENDEVIATIONBAR_SEEDER_REST_FALLBACK_MAX_AGE_DAYS=7  # covers full week of Vision lag
OPENDEVIATIONBAR_SEEDER_REST_FIRST_T1=true  # REST-seed yesterday immediately for kintsugi
```

**Kintsugi parallelism SSoT**: All kintsugi parallelism values live in this file. See [/scripts/CLAUDE.md](/scripts/CLAUDE.md#kintsugi-parallelism-tuning-bigblack) for the meaning of each value and tuning guidance.

---

## Monit Configuration (`monit/`)

| File                             | Deployed To                                         | Purpose                                              |
| -------------------------------- | --------------------------------------------------- | ---------------------------------------------------- |
| `opendeviationbar.conf`          | `/etc/monit/conf.d/opendeviationbar.conf`           | 10 monitors: processes, ports, filesystem, system    |
| `monit-override.conf`            | `/etc/systemd/system/monit.service.d/override.conf` | systemd drop-in: allow uid/gid exec (CAP_SETUID etc) |
| `obdpy-monit-alert.sh`           | `/usr/local/bin/obdpy-monit-alert.sh`               | Telegram alert on monit events                       |
| `obdpy-monit-heartbeat.sh`       | `/usr/local/bin/obdpy-monit-heartbeat.sh`           | Self-report every 10 monit cycles                    |
| `obdpy-check-heartbeat-timer.sh` | `/usr/local/bin/obdpy-check-heartbeat-timer.sh`     | Exit-code check: did timer run on schedule?          |

### Syncing Monit Config (Requires sudo)

```bash
scp deploy/monit/opendeviationbar.conf bigblack:/tmp/
ssh bigblack 'sudo cp /tmp/opendeviationbar.conf /etc/monit/conf.d/ && sudo monit reload'
```

### Why the systemd Drop-in (`monit-override.conf`)

Monit runs as root but needs to exec commands `as uid "tca"`. systemd 247+ hardening flags block this by default. The drop-in sets:

- `NoNewPrivileges=false`
- `CapabilityBoundingSet` includes `CAP_SETUID`, `CAP_SETGID`, `CAP_SYS_PTRACE`
- `ProtectHome=no`
- `PrivateTmp=no`

Without these:

- **Missing `NoNewPrivileges=false` + caps**: monit alert scripts silently fail (no Telegram message).
- **Missing `PrivateTmp=no`**: monit runs with a private `/tmp`+`/var/tmp` namespace, making files written by services to the real `/var/tmp/` invisible to `check file` monitors. Symptom: `seeder-state` permanently shows "Does not exist" even though the file exists on disk. Diagnosed 2026-03-13 via `sudo nsenter -t $(monit PID) -m -- ls /var/tmp/`.

### Alert Script Pattern

`obdpy-monit-alert.sh` sources both env files to get `TOKEN` (from `local.env`) and `TELEGRAM_CHAT_ID` (from `opendeviationbar.env`), then calls the Telegram API. This keeps secrets out of the monit config file (which is root-readable).

---

## Deploy Workflow

Full pipeline: `mise run deploy:bigblack`

What it does:

1. Syncs git repo on bigblack (`git fetch && git reset --hard origin/main`)
2. Installs new wheel into `.venv` (from PyPI or local wheel SCP fallback)
3. Copies systemd units to `~/.config/systemd/user/`
4. `systemctl --user daemon-reload && enable --now` timers
5. Restarts all services (monit unmonitor first, start sidecar then kintsugi)
6. `sudo monit monitor sidecar && sudo monit monitor kintsugi`

For Python-only hotfixes (no Rust rebuild): use `/odb-ship hotfix` skill — rsyncs Python files to site-packages directly.

**Full deploy procedure**: [/.claude/skills/odb-ship/SKILL.md](/.claude/skills/odb-ship/SKILL.md)

---

## local.env (Secrets — Not in Git)

```ini
OPENDEVIATIONBAR_TELEGRAM_TOKEN=<bot token>
OPENDEVIATIONBAR_REDIS_PASSWORD=<redis pw>
```

This file lives at `~/opendeviationbar-py/deploy/opendeviationbar.local.env` on bigblack. It survives `git reset --hard` (not tracked). Never commit it.

---

## Monitoring & Data Integrity

Automated monitoring prevents silent data loss in the ClickHouse cache. Born from the Feb 2026 incident where 5.5 days of BTCUSDT@500 bars were lost due to `try_cache_write()` silently swallowing connection errors.

### Root Cause (Resolved)

`try_cache_write()` in `orchestration/open_deviation_bars_cache.py` was fire-and-forget. When SSH tunnel dropped, writes failed silently for days while the checkpoint advanced. Fixed: `_fatal_cache_write()` raises on ClickHouse failure.

### Defense-in-Depth (Issue #158 Phase 7)

**Write integrity**: Four layers prevent silent data loss. **Overlap prevention**: Four layers prevent duplicate bars from concurrent writers.

| Layer                 | Mechanism                                                              | Details                                                             |
| --------------------- | ---------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Bounded flush**     | `OPENDEVIATIONBAR_FLUSH_THRESHOLD` (default 10K) in `_process_day()`   | Caps peak bars-in-flight during populate; prevents memory spikes    |
| **Fatal writes**      | `_fatal_cache_write()` in populate path                                | Raises on ClickHouse failure instead of warning                     |
| **Circuit breaker**   | `fatal_write_breaker` (3 failures → OPEN, 60s recovery)                | Prevents cascade during ClickHouse outages                          |
| **T-1 guard**         | `populate_cache_resumable()` clamps `end_date`                         | Prevents crash on unavailable Binance Vision data                   |
| **Gap detection**     | `scripts/detect_gaps.py` + heartbeat timer                             | Every 5 min via systemd, alerts via @obdpy_bot                      |
| **L1 Redis lock**     | `cache_write_lock()` per (symbol, threshold)                           | Prevents concurrent writes; graceful degradation if Redis down      |
| **L2 Schema dedup**   | ORDER BY `first_agg_trade_id` in ReplacingMergeTree                    | Native dedup on merge — same trades from different writers collapse |
| **L3 Heartbeat**      | Service liveness + Stathera audit + disk + circuit breaker every 5 min | `health_heartbeat.py` (sole Telegram telemetry checkpoint)          |
| **L4 Invariant gate** | Reject bars with inverted trade IDs or timestamps before CH insert     | Catches corruption from any source — last line of defense (#280)    |
| **L5 Checksum audit** | `ChecksumRegistry` + JSONL telemetry via heartbeat (v6.0)              | SHA-256 Vision ZIP verification, LOUD alert on failure, local log   |

### Telegram Notifications (@obdpy_bot)

| Event                      | Behavior                                                                           |
| -------------------------- | ---------------------------------------------------------------------------------- |
| All checks pass            | Ephemeral green heartbeat, auto-deletes after 15 min (~2-3 visible)                |
| Gap or regression detected | **Persistent** red alert, LOUD notification, stays forever                         |
| Cache write failure        | Circuit breaker alert via `opendeviationbar.notify.telegram`                       |
| Checksum failure (24h)     | **Persistent** red alert, LOUD notification, `checksum_ok` in CRITICAL_KEYS (v6.0) |

**Config**: `OPENDEVIATIONBAR_TELEGRAM_TOKEN` in `.mise.local.toml` (local) or `deploy/opendeviationbar.local.env` (bigblack). Chat ID in `.mise.toml`.

**Heartbeat**: `scripts/health_heartbeat.py` via systemd timer (`opendeviationbar-heartbeat.timer`, every 5 min). Checks: ClickHouse, sidecar, kintsugi, Stathera audit (gaps + overlaps), disk, circuit breaker. **Sole Telegram telemetry checkpoint** for all ODB monitoring (#238 Phase 3). Systemd overrides (drop-ins) are **informational** — they display as `⚠️ overrides active` but do NOT cause UNHEALTHY (v13.15.4+).

### Monit Monitors (11 total)

| Monitor             | Type    | Check                                                         | Debounce      |
| ------------------- | ------- | ------------------------------------------------------------- | ------------- |
| clickhouse          | host    | HTTP `/ping` on :8123                                         | 3 cycles      |
| sidecar             | process | PID exists, mem < 38500 MB, cpu < 80%                         | immediate/3/5 |
| sidecar-health      | host    | HTTP `/health` on :8081                                       | 3 cycles      |
| kintsugi            | process | PID exists, mem < 42700 MB                                    | immediate/3   |
| heartbeat-timer     | program | `obdpy-check-heartbeat-timer.sh` exit code                    | 3 cycles      |
| seeder-state        | program | `/var/tmp/opendeviationbar-seeder-state.txt` mtime <15 min    | 3 cycles      |
| exness-seeder-state | file    | `/var/tmp/opendeviationbar-exness-seeder-state.txt` mtime <2h | 3 cycles      |
| system ($HOST)      | system  | loadavg < 8, mem < 85%, swap < 25%                            | 3/5 cycles    |
| redis               | host    | Redis protocol on :6379                                       | 3 cycles      |
| rootfs              | fs      | Disk < 80%/90%, inodes < 85%                                  | immediate     |
| monit-heartbeat     | program | `obdpy-monit-heartbeat.sh` (every 10 cycles)                  | 3 cycles      |

**Note (v7.2)**: Monit sidecar threshold (38.5GB) is a legacy value -- systemd MemoryMax=4G is the effective guard. v7.2 Arrow FFI reduces peak sidecar RSS to ~0.9G.

### External Monitoring Ports (bigblack, not ODB-managed)

| Service       | Port  | Purpose             |
| ------------- | ----- | ------------------- |
| Prometheus    | 9090  | Metrics collection  |
| node_exporter | 9100  | Host metrics        |
| Netdata       | 19999 | Real-time dashboard |
| Cockpit       | 9091  | System admin web UI |

---

## Related

- [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md) — systemd service units, dependency chain, OOM protection
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) — kintsugi parallelism tuning, monitoring
- [/CLAUDE.md](/CLAUDE.md) — project hub
