---
name: odb-ship
description: >-
  Unified release-publish-deploy pipeline for opendeviationbar-py.
  Covers semantic-release versioning, Rust+Python wheel builds, PyPI and crates.io publishing,
  bigblack production deployment (atomic venv swap, systemd sync, service orchestration),
  monit alignment, and 73-point post-deploy verification.
  Use when shipping a new version, deploying hotfixes, verifying production health,
  or recovering from partial release/deploy failures.
  Orchestrates mise release/deploy tasks with adaptive failure handling.
  mise tasks are building blocks; odb-ship is the SSoT orchestrator.
allowed-tools: Read, Bash, Grep, Glob, Agent, WebFetch, TeamCreate, SendMessage, TaskCreate, TaskUpdate, TaskList, TaskGet
---

# odb-ship: Release, Publish & Deploy

ultrathink

Unified pipeline from "code ready on main" to "production verified healthy". Claude Code is the orchestrator — dynamically adapting to failures instead of rigid bash scripts that break on SSH quoting edge cases.

## When to Use

- `/odb-ship` — Full pipeline: version bump through production verification
- `/odb-ship --agents` — Full pipeline with agent team (parallel test+build, parallel publish)
- `/odb-ship deploy` — Deploy only (version already published)
- `/odb-ship verify` — Verify production health without deploying
- `/odb-ship hotfix` — Pure Python-only deploy (no Rust rebuild needed)
- `/odb-ship rollback` — Recover from a bad deploy

---

## Architecture: Skill as SSoT Orchestrator

**odb-ship is the SSoT for release+deploy.** Mise tasks are building blocks it invokes:

| Mise Task           | odb-ship Phase | Role                                 |
| ------------------- | -------------- | ------------------------------------ |
| `release:version`   | 1.2            | Semantic-release version bump        |
| `release:test`      | 1.3            | Rust + Python test gate              |
| `release:build-all` | 1.4            | Wheel builds (macOS + Linux + sdist) |
| `release:pypi`      | 2.1            | PyPI upload                          |
| `release:crates`    | 2.2            | crates.io workspace publish          |

**Do NOT use `mise run release:full` directly** — it duplicates odb-ship's orchestration without adaptive failure handling or deploy phases. Use odb-ship as the single entry point.

**Why a skill, not a script**: The `mise run deploy:bigblack` bash script fails on SSH quoting, network transients, partial failures requiring judgment, and service timing. Claude Code adapts dynamically.

---

## Agent Team Mode (`--agents`)

When invoked with `--agents`, odb-ship creates an agent team to parallelize the two slowest segments of the pipeline. Without `--agents`, the default single-agent sequential mode is used (simpler, lower token cost).

### Why Teams Help

The pipeline has two naturally parallel segments that account for ~7 of ~11 minutes:

| Segment                                | Sequential | With Team | Savings  |
| -------------------------------------- | ---------- | --------- | -------- |
| Phase 1: Tests (~2.5m) + Build (~2.8m) | ~5.3 min   | ~2.8 min  | ~2.5 min |
| Phase 2: PyPI (~2m) + crates.io (~2m)  | ~4 min     | ~2 min    | ~2 min   |

Phases 3-5 (deploy, service orchestration, verification) are sequential SSH commands to one host — no parallelism benefit.

### Team Structure

```
Team: odb-ship-v{VERSION}
Lead (you): Phase 0 → Phase 1.1-1.2 → coordinate → Phase 3-5

Teammates:
  tester     — Phase 1.3: mise run release:test
  builder    — Phase 1.4: mise run release:build-all
  pypi-pub   — Phase 2.1: mise run release:pypi
  crates-pub — Phase 2.2: mise run release:crates
```

### Team Flow

**Step 1: Pre-flight + Version** (Lead, sequential)

```
Phase 0: Pre-flight checks
Phase 1.1: git pull --rebase
Phase 1.2: semantic-release (must complete before tests/builds)
```

**Step 2: Test + Build** (Parallel teammates)

```python
# After semantic-release succeeds, create team and spawn:
TeamCreate(team_name="odb-ship-v{VERSION}")

TaskCreate(title="Run tests", description="mise run release:test — gate on Rust + Python passing")
TaskCreate(title="Build wheels", description="mise run release:build-all — macOS ARM64 + Linux x86_64 + sdist")

# Spawn teammates
Agent(name="tester", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:test. Report PASS/FAIL with test counts.",
      mode="bypassPermissions")

Agent(name="builder", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:build-all. Report wheel filenames and sizes from dist/.",
      mode="bypassPermissions")

# Wait for both to complete — messages arrive automatically
```

**Step 3: Publish** (Parallel teammates)

```python
# After both test + build pass:
TaskCreate(title="Publish PyPI", description="mise run release:pypi")
TaskCreate(title="Publish crates.io", description="mise run release:crates")

Agent(name="pypi-pub", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:pypi. Then verify: curl -sf 'https://pypi.org/pypi/opendeviationbar/{VERSION}/json' | jq '.info.version'. Report the confirmed version.",
      mode="bypassPermissions")

Agent(name="crates-pub", team_name="odb-ship-v{VERSION}",
      prompt="Run: mise run release:crates. Then verify: cargo search opendeviationbar-core --limit 1. Report the confirmed version.",
      mode="bypassPermissions")

# Wait for both to complete
```

**Step 4: Deploy + Verify** (Lead, sequential)

```
Phase 2.3: Verify publications (quick confirmation)
Phase 3: Deploy to bigblack
Phase 4: Service orchestration
Phase 5: Verification
```

**Step 5: Cleanup**

```python
# Shut down teammates
SendMessage(to="tester", message={"type": "shutdown_request"})
SendMessage(to="builder", message={"type": "shutdown_request"})
SendMessage(to="pypi-pub", message={"type": "shutdown_request"})
SendMessage(to="crates-pub", message={"type": "shutdown_request"})
TeamDelete(team_name="odb-ship-v{VERSION}")
```

### Failure Handling in Team Mode

| Failure                         | Action                                                                                |
| ------------------------------- | ------------------------------------------------------------------------------------- |
| **tester fails**                | Stop builder (no point publishing broken code). Report failure.                       |
| **builder fails**               | Wait for tester. If tests pass, retry build. If tests also fail, report both.         |
| **pypi-pub fails**              | Wait for crates-pub. Retry PyPI. If still fails, fall back to SCP wheel.              |
| **crates-pub fails**            | Non-blocking for deploy. Log warning, continue to Phase 3 (PyPI wheel is sufficient). |
| **Teammate stops unexpectedly** | Lead receives idle notification. Check task status. Spawn replacement if needed.      |

### When NOT to Use `--agents`

- **`/odb-ship deploy`** — No test/build/publish, just SSH commands. Single agent is faster.
- **`/odb-ship verify`** — Just SSH checks. No parallelism benefit.
- **`/odb-ship hotfix`** — No wheel build, just rsync. Single agent is faster.
- **`/odb-ship rollback`** — Emergency path. Keep simple.

### Prerequisites

Agent teams require:

- `CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS=1` in settings.json or environment
- Claude Code v2.1.32+

---

## Phase Overview

| Phase | Name                                                    | Purpose                                | Skippable         |
| ----- | ------------------------------------------------------- | -------------------------------------- | ----------------- |
| 0     | [Pre-Flight](#phase-0-pre-flight)                       | Connectivity, credentials, clean state | No                |
| 1     | [Version & Build](#phase-1-version--build)              | Semantic-release + wheel builds        | Yes (deploy-only) |
| 2     | [Publish](#phase-2-publish)                             | PyPI + crates.io + GitHub release      | Yes (deploy-only) |
| 3     | [Deploy](#phase-3-deploy-to-bigblack)                   | Atomic venv swap on bigblack           | No                |
| 4     | [Service Orchestration](#phase-4-service-orchestration) | Stop/start services, monit alignment   | No                |
| 5     | [Verification](#phase-5-verification)                   | 73-point production health check       | No                |

---

## Phase 0: Pre-Flight

### 0.1 Network Connectivity (Run First)

```bash
# GitHub SSH (may fail — HTTPS fallback available)
ssh -o ConnectTimeout=5 -T git@github.com 2>&1 | head -3

# Bigblack SSH (REQUIRED — no fallback)
ssh -o ConnectTimeout=5 bigblack 'echo OK'

# GitHub API via HTTPS (works even when SSH is blocked)
gh api repos/terrylica/opendeviationbar-py --jq '.pushed_at'
```

**If GitHub SSH fails**: Swap remote to HTTPS before any git/release operations:

```bash
git remote set-url origin "https://github.com/terrylica/opendeviationbar-py.git"
# Restore after: git remote set-url origin "git@github.com-terrylica:terrylica/opendeviationbar-py.git"
```

### 0.2 Clean Working Directory

```bash
git status --porcelain
git log --oneline @{u}..HEAD  # unpushed commits
```

- Unpushed commits: `git push origin main` first
- Dirty files: commit or stash before proceeding
- Lockfile drift (uv.lock): `git checkout -- uv.lock`

### 0.3 Checksum Bindings (v6.0+)

Verify Rust checksum PyO3 bindings compiled into the wheel. If these are missing, the seeder will crash on first run after deploy.

```bash
python3 -c "from opendeviationbar._core import compute_sha256_hex, parse_checksum_file_content, verify_checksum_hex; print('✓ Checksum bindings present')"
```

**If this fails**: `maturin develop` or `maturin build --release` did not include the `data-providers` feature gate, or `src/binance_bindings.rs` changes were not compiled. Rebuild before proceeding.

### 0.4 Version State

```bash
VERSION=$(grep -A5 '\[workspace.package\]' Cargo.toml | grep '^version' | head -1 | sed 's/.*= "\(.*\)"/\1/')
echo "Current Cargo.toml version: $VERSION"
```

---

## Phase 1: Version & Build

### 1.1 Sync with Remote

```bash
git pull --rebase origin main
```

If SSH fails, use HTTPS push:

```bash
GH_TOKEN=$(gh auth token) git \
  -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
  push origin main
```

### 1.2 Semantic Release

```bash
mise run release:version
```

This runs `semantic-release --no-ci` which:

- Analyzes commits since last tag (feat: → minor, fix: → patch)
- Bumps `Cargo.toml` + creates git tag + GitHub release
- **CRITICAL**: Must use `--no-ci` flag (otherwise dry-run only)
- **CRITICAL**: Needs SSH or HTTPS connectivity to `git ls-remote`

**If no releasable commits**: semantic-release exits cleanly, creates `.release-skip` file. All downstream phases skip gracefully.

### 1.3 Run Tests

```bash
mise run release:test
```

Gates on both Rust (`cargo nextest`) and Python (`pytest`) passing.

### 1.4 Build Wheels

```bash
mise run release:build-all
```

Builds:

- macOS ARM64 wheel (native maturin)
- Linux x86_64 wheel (zig cross-compile, ~55s)
- Source distribution (sdist)

Output: `dist/opendeviationbar-{VERSION}-*.whl`

> **Team mode (`--agents`)**: After semantic-release (1.2), spawn `tester` and `builder` teammates in parallel instead of running 1.3 and 1.4 sequentially. Wait for both to report back. If `tester` fails, shut down `builder` immediately — no point publishing broken code. See [Agent Team Mode](#agent-team-mode---agents) for full flow.

---

## Phase 2: Publish

### 2.1 PyPI

```bash
mise run release:pypi
```

Uses 1Password credentials from `~/.claude/.secrets/`. After upload, PyPI CDN may take 30-60s to propagate. The deploy phase polls for availability.

### 2.2 Crates.io

```bash
mise run release:crates
```

Uses `cargo publish --workspace` (Rust 1.90+ native topological ordering). Publishes 5 crates: core, providers, streaming, client, hurst.

**Token resolution** (automatic, 3-source fallback in `release-crates.toml`):

1. `~/.claude/.secrets/crates-io-token` (local file, fastest)
2. Doppler `claude-config/prd` → `CRATES_IO_CLAUDE_CODE` (auto-caches to #1 on success)
3. Manual: `echo TOKEN > ~/.claude/.secrets/crates-io-token && chmod 600 ~/.claude/.secrets/crates-io-token`

**Known issue**: If crates.io publish races with version bump, you get "already exists" errors. Fix: wait for `Cargo.toml` to have the new version, then re-run.

> **Team mode (`--agents`)**: Spawn `pypi-pub` and `crates-pub` teammates in parallel instead of running 2.1 and 2.2 sequentially. Each teammate verifies its own publication before reporting back. If `crates-pub` fails, continue anyway — PyPI wheel is sufficient for deploy. See [Agent Team Mode](#agent-team-mode---agents) for full flow.

### 2.3 Verify Publications

```bash
# PyPI
curl -sf "https://pypi.org/pypi/opendeviationbar/$VERSION/json" | jq '.info.version'

# Crates.io (check one representative crate)
cargo search opendeviationbar-core --limit 1
```

---

## Phase 3: Deploy to Bigblack

### Critical Context

| Fact                    | Detail                                                               |
| ----------------------- | -------------------------------------------------------------------- |
| **Deploy target**       | `bigblack` (remote GPU host, SSH alias)                              |
| **Remote dir**          | `/home/tca/opendeviationbar-py`                                      |
| **Python**              | mise-managed 3.13 (uv discovers via `--python 3.13`)                 |
| **Package manager**     | `uv` at `~/.local/bin/uv` (NOT in venv)                              |
| **Services managed by** | systemd `--user` (monit watches and auto-restarts)                   |
| **Secrets**             | `deploy/opendeviationbar.local.env` (Telegram TOKEN, Redis password) |

### 3.1 Git Sync on bigblack

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main --tags && git reset --hard origin/main && git clean -fd'
```

### 3.2 SCP Wheel to bigblack

**Primary (SCP — faster and more reliable than PyPI CDN polling)**:

```bash
LINUX_WHEEL=$(ls dist/opendeviationbar-${VERSION}-*manylinux*.whl | head -1)
scp "$LINUX_WHEEL" bigblack:/tmp/
```

**CRITICAL**: SCP to `/tmp/`, NOT to `~/opendeviationbar-py/`. Installing a wheel from inside a directory that contains `pyproject.toml` causes uv to create an editable `.pth` file that shadows the wheel — the root cause of the v13.56.1 version drift incident.

SCP delivers the wheel in ~1s over LAN. No CDN propagation delay, no polling loop. The wheel was already built locally by `mise run release:build-all`.

**Fallback (PyPI — if local wheel unavailable, e.g., deploy-only mode)**:

```bash
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv-staging/bin/python3 opendeviationbar==$VERSION"
```

### 3.3 Build Staging Venv + Install

Combined into one SSH call to eliminate round-trip overhead:

```bash
ssh bigblack "rm -rf ~/opendeviationbar-py/.venv-staging && uv venv --python 3.13 ~/opendeviationbar-py/.venv-staging && uv pip install --python ~/opendeviationbar-py/.venv-staging/bin/python3 /tmp/$(basename $LINUX_WHEEL)"
```

uv 0.11+ discovers mise-managed Python natively via `--python 3.13` — no glob needed (spike-validated 2026-03-26, 41ms). Install resolves deps from uv's 17GB local cache (~200ms, spike-validated).

**Anti-pattern (NEVER DO)**: `cd ~/opendeviationbar-py && uv pip install ./wheel.whl` — uv detects the `pyproject.toml` and creates an editable `.pth` instead of installing the wheel. Always install from `/tmp/` or an absolute path outside the project directory.

**Note on `uv sync`**: Spike-tested `uv sync --frozen --no-install-project` on bigblack (2026-03-26) — installs 0 packages for maturin projects. Not viable. `uv pip install` with warm cache is already optimal (~200ms).

### 3.4 Verify Staging Venv

All verification in one SSH call (saves ~4s of SSH handshake overhead vs 3 separate calls):

```bash
ssh bigblack '
~/opendeviationbar-py/.venv-staging/bin/python3 -c "
import opendeviationbar; print(opendeviationbar.__version__)
from opendeviationbar import OpenDeviationBarProcessor
from pathlib import Path
import opendeviationbar.clickhouse.cache as ch
assert (Path(ch.__file__).parent / \"schema.sql\").exists()
import opendeviationbar.symbol_registry as sr
assert (Path(sr.__file__).parent / \"data\" / \"symbols.toml\").exists()
from opendeviationbar._core import compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content
from opendeviationbar.checksum_registry import ChecksumRegistry
from opendeviationbar.checksum_telemetry import log_checksum_event
print(\"OK: version + .so + schema.sql + symbols.toml + checksum\")
"
# Verify NO .pth shadowing (must fail — if .pth exists, the install was editable)
SITE_PKG=$(~/opendeviationbar-py/.venv-staging/bin/python3 -c "import site; print(site.getsitepackages()[0])")
if [ -f "$SITE_PKG/opendeviationbar.pth" ]; then
  echo "FATAL: .pth shadowing detected — wheel installed as editable!"
  echo "Fix: install from /tmp/ not from inside the project directory"
  cat "$SITE_PKG/opendeviationbar.pth"
  exit 1
fi
echo "OK: no .pth shadowing"
# Cleanup /tmp wheel
rm -f /tmp/opendeviationbar-*.whl
'
```

**If `.pth` detected**: The wheel was installed from inside the project directory. Fix: re-run step 3.3 with the wheel path pointing to `/tmp/`.

**If checksum imports fail**: Wheel lacks v6.0 checksum modules. Seeder will crash on next timer trigger — rollback and rebuild.

---

## Phase 4: Service Orchestration

### 4.1 Stop Services (BEFORE Venv Swap)

**CRITICAL**: Must unmonitor from monit first, otherwise monit auto-restarts within 120s:

```bash
ssh bigblack 'sudo monit unmonitor sidecar 2>/dev/null; sudo monit unmonitor kintsugi 2>/dev/null'
ssh bigblack 'systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-kintsugi-catchup 2>/dev/null; true'
```

No sleep needed — `systemctl stop` is synchronous (returns after the process exits).

### 4.1b Kill ALL ClickHouse Mutations (BEFORE Venv Swap)

**CRITICAL**: Kill ALL mutations — both pending and completed. Kintsugi's `_do_replace()` creates `DELETE IN PARTITION` mutations for historical repairs; any still-executing mutations can interfere with the sidecar's fresh start. Killing preemptively ensures a clean mutation queue.

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "KILL MUTATION WHERE database = '"'"'opendeviationbar_cache'"'"'" && echo "All mutations killed"'
# Verify zero remaining
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "SELECT count() FROM system.mutations WHERE database = '"'"'opendeviationbar_cache'"'"'"'
```

**Why kill ALL**: Executing mutations from the previous kintsugi session can race with the new sidecar's sync_flush writes. The only safe path is killing every mutation before restart.

### 4.2 OPTIMIZE TABLE FINAL (Force RMT Dedup)

**CRITICAL**: Before swapping venvs, force ReplacingMergeTree to deduplicate all parts. This resolves any pre-existing overlaps from previous deploys, mutation storms, or concurrent writer races. Much faster than ALTER TABLE DELETE (no mutations created).

```bash
ssh bigblack 'curl -s "http://localhost:8123/?wait_end_of_query=1" --data-binary "OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars FINAL" && echo "OPTIMIZE FINAL done"'
```

**Why every deploy**: Overlaps can accumulate silently between deploys (sidecar restarts, kintsugi races). OPTIMIZE FINAL resolves **same-key duplicates** (RMT deduplicates by `ORDER BY first_agg_trade_id`, keeping the latest version). It does **NOT** resolve overlapping bars with different `first_agg_trade_id` (different TID boundaries from different processor sessions) — those require kintsugi repair or Pattern 2. Cost: 5-30s depending on parts count.

### 4.2b Run Pending Schema Migrations (AFTER OPTIMIZE, BEFORE Venv Swap)

Check for and run any pending ClickHouse migration scripts. Services are already stopped, so migrations are safe.

```bash
# Check for pending migrations
MIGRATION=$(ssh bigblack 'ls ~/opendeviationbar-py/scripts/migrate_*.py 2>/dev/null | head -1')
if [ -n "$MIGRATION" ]; then
  echo "Found migration: $(basename $MIGRATION)"
  ssh bigblack "~/opendeviationbar-py/.venv/bin/python3 $MIGRATION --apply"
  echo "Migration complete"
else
  echo "No pending migrations"
fi
```

**v4.0 migration**: `scripts/migrate_column_rename_us.py --apply` renames `close_time_ms` → `close_time_us` and `open_time_ms` → `open_time_us` via `ALTER TABLE RENAME COLUMN` (metadata-only, instant on 42M+ rows). Must run BEFORE venv swap because the new code expects `_us` column names.

**Why here (after OPTIMIZE, before venv swap)**: OPTIMIZE FINAL uses the old column names. The migration renames them. Then the new venv (with `_us` code) starts against the renamed columns. If migration fails, the old venv still works with old column names — safe rollback.

### 4.3 Clear Stale Streaming Checkpoints

Checkpoints from the previous sidecar session often point beyond ClickHouse data (especially after OPTIMIZE merges parts). Stale checkpoints are harmless (rejected + preserved) but cause noisy warnings on startup. Clear them for a clean start:

```bash
ssh bigblack 'rm -f ~/.cache/opendeviationbar/checkpoints/streaming_*.json && echo "Streaming checkpoints cleared"'
```

The sidecar's `_inline_startup_backfill()` will cover any gaps from the cleared checkpoints.

### 4.3b Parquet Tick Cache (No Synchronous Wait)

The seeder runs on a 5-min timer independently. **Do NOT wait synchronously for seeder completion during deploy** — the sidecar's SEED-001 Parquet-first catchup handles missing Parquet gracefully (falls back to REST). The seeder will populate the cache on its next timer trigger.

Previous versions of this skill documented a synchronous seeder wait (up to 5 min). This was eliminated in the 2026-03-26 pipeline audit — the `mise run deploy:bigblack` script already skips it, confirming it provides zero safety benefit.

### 4.4 Atomic Venv Swap

```bash
ssh bigblack 'cd ~/opendeviationbar-py && rm -rf .venv-old && mv .venv .venv-old 2>/dev/null; mv .venv-staging .venv && rm -rf .venv-old'
```

Same-filesystem `mv` is atomic. Old venv untouched until swap succeeds.

### 4.5 Sync Systemd Units

```bash
ssh bigblack '
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-sidecar.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-kintsugi-catchup.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-heartbeat.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-heartbeat.timer ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-seeder.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-seeder.timer ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-exness-seeder.service ~/.config/systemd/user/
cp ~/opendeviationbar-py/scripts/systemd/opendeviationbar-exness-seeder.timer ~/.config/systemd/user/
# Remove stale systemd override directories (prevents heartbeat "overrides active" warnings)
for d in ~/.config/systemd/user/opendeviationbar-*.service.d; do
  [ -d "$d" ] && rm -rf "$d" && echo "Removed override: $(basename $d)"
done
systemctl --user daemon-reload
systemctl --user enable --now opendeviationbar-heartbeat.timer
systemctl --user enable --now opendeviationbar-seeder.timer
systemctl --user enable --now opendeviationbar-exness-seeder.timer
echo "OK: 6 services + 3 timers synced"
'
```

### 4.6 Start Services (Order Matters)

```bash
# Sidecar first (provides health endpoint + real-time data)
ssh bigblack 'systemctl --user start opendeviationbar-sidecar'
sleep 8

# Verify sidecar health
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'

# CRITICAL (#295): Wait for sync_flush completion before starting kintsugi.
# Kintsugi's _do_replace mutations can delete sidecar's freshly-written bars
# if started before sync_flush completes.
#
# Stream journal instead of polling (single SSH connection, exits on match):
ssh bigblack 'timeout 300 journalctl -f --user -u opendeviationbar-sidecar --grep "sync_flush.*drained" --output cat 2>/dev/null | head -1 && echo "sync_flush complete"'
# If timeout exits with 124, sync_flush didn't complete — check sidecar logs.

# Verify zero Stathera anomalies on today BEFORE starting kintsugi.
# Must show overlaps=0 AND gaps=0 (intra-day only — day-boundary gaps are structural).
TODAY_ANOMALIES=$(ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT countIf(tid_delta < 0) AS overlaps, countIf(tid_delta > 0 AND day_gap = 0) AS intraday_gaps
FROM (
  SELECT
    first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta,
    toDate(close_time_us/1000000) - toDate(lagInFrame(close_time_us, 1) OVER w / 1000) AS day_gap,
    row_number() OVER w AS rn
  FROM opendeviationbar_cache.open_deviation_bars FINAL
  WHERE toDate(close_time_us/1000000) = today()
  WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode ORDER BY first_agg_trade_id)
) WHERE rn > 1
FORMAT TSV"')
echo "Today Stathera: $TODAY_ANOMALIES  (target: overlaps=0, intraday_gaps=0)"

# If non-zero, see "Failure Handling: Stathera Anomalies Before Kintsugi Start" below.
# DO NOT start kintsugi if overlaps or intraday_gaps > 0.

# Kintsugi (gap reconciliation — only after today is verified clean)
ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'
sleep 3
```

### 4.6c Clear Kintsugi Rate Limit (No Synchronous Wait)

Deploy kills the sidecar, creating a gap at the day boundary. Clear the rate-limit log so kintsugi can repair immediately, then let it self-heal asynchronously.

```bash
## WARNING (v7.3+): TRUNCATE TABLE kintsugi_log is BANNED — destroys exchange_gap permanent dismiss entries.
## Use safe cleanup instead (preserves exchange gap classifications):
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "ALTER TABLE opendeviationbar_cache.kintsugi_log DELETE WHERE result NOT IN ('"'"'exchange_gap'"'"') AND timestamp < now() - INTERVAL 7 DAY SETTINGS mutations_sync=1"'
ssh bigblack 'systemctl --user restart opendeviationbar-kintsugi'
```

**Do NOT wait synchronously for kintsugi repair completion during deploy.** Kintsugi is a self-healing daemon — it repairs yesterday's gaps on its own schedule (60s active interval). The previous 10-minute synchronous wait was eliminated in the 2026-03-26 pipeline audit. The heartbeat monitors kintsugi health every 5 min and alerts if repairs stall.

See also: [Kintsugi Log Table Rate-Limit Reset](#kintsugi-log-table-rate-limit-reset) in Lessons Learned.

### 4.6b Cross-Midnight Gap Note

**Context**: Killing the old sidecar loses its forming orphan bars at the day boundary. The new sidecar starts from midnight crossover TID, leaving a gap between yesterday's last committed bar and today's first bar.

**Resolution**: Kintsugi repairs these gaps automatically when they become "yesterday or older" at the next UTC midnight. This is the **only safe path**.

**Anti-pattern (BANNED): Running `backfill_gap()` while sidecar is streaming.** Tested v13.53.0 deploy — `backfill_gap(force=True)` while sidecar was running caused 13 overlaps. Different processor instances produce bars with different TID boundaries for the same trade range. `overlap_strategy="replace"` does NOT prevent this because the overlapping bars have different `first_agg_trade_id` values (RMT can't dedup them). Required Pattern 2 (stop sidecar → delete today → restart) to clean up.

**If cross-midnight gaps are unacceptable (rare)**: Run `backfill_gap` BEFORE starting the sidecar, as part of Pattern 2 from the restart recovery playbook. Never while the sidecar is running.

### 4.6a Failure Handling: Stathera Anomalies Before Kintsugi Start

If the Stathera check above shows `overlaps > 0` or `intraday_gaps > 0` before kintsugi starts, **do not start kintsugi**. The overlap/gap must be resolved first.

**Decision tree:**

```
overlaps > 0?
├── Run OPTIMIZE TABLE FINAL (fast RMT dedup, no mutations):
│     ssh bigblack 'curl -s "http://localhost:8123/?wait_end_of_query=1" \
│       --data-binary "OPTIMIZE TABLE opendeviationbar_cache.open_deviation_bars FINAL"'
│   Re-run Stathera check. If still > 0 → stop sidecar, run Pattern 2 from
│   sidecar-restart-zero-overlap-zero-gap-recovery-playbook.
│
intraday_gaps > 0 (no overlaps)?
├── Small gap (<1000 trades): Puck should fill on next WS trade — wait 60s and re-check.
├── Large gap (>1000 trades): stop sidecar, Pattern 2 from the restart playbook.
│
Both overlaps AND gaps?
└── Pattern 2 is the only safe fix. Do not start kintsugi until both are 0.
```

**Pattern 2 reference**: Stop sidecar → DELETE today's bars (`mutations_sync=1`) → `backfill_gap(force=True)` → start sidecar → wait for sync_flush → re-verify → THEN start kintsugi.

After any recovery loop, run the sync_flush gate (step 4.6) again before proceeding to 4.7.

### 4.7 Re-Monitor in Monit

```bash
ssh bigblack 'sudo monit monitor sidecar 2>/dev/null; sudo monit monitor kintsugi 2>/dev/null'
```

### 4.8 Post-Start Stathera Audit

**MANDATORY**: After services start, verify zero overlaps on today's data. If overlaps exist, run OPTIMIZE FINAL again (the restart may have created transient overlaps before Rust dedup initialized).

```bash
# Check today's overlaps
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "SELECT countIf(tid_delta < 0) AS overlaps FROM (SELECT first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta, row_number() OVER w AS rn FROM opendeviationbar_cache.open_deviation_bars FINAL WHERE first_agg_trade_id > 0 AND last_agg_trade_id > 0 AND toDate(close_time_us/1000000) >= today() WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps, ouroboros_mode ORDER BY first_agg_trade_id, close_time_us)) WHERE rn > 1 AND tid_delta < 0"'
```

**If overlaps > 0**: Stop sidecar, run OPTIMIZE FINAL again, clear checkpoints, restart sidecar, then repeat the sync_flush gate (4.6) and Stathera check before starting kintsugi. This loop converges in 1-2 iterations because OPTIMIZE FINAL is deterministic.

### 4.9 Verify Services Active

```bash
ssh bigblack '
for svc in opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer opendeviationbar-seeder.timer; do
  svc_state=$(systemctl --user is-active $svc 2>/dev/null || echo "inactive")
  echo "$svc: $svc_state"
done
'
```

---

## Phase 5: Verification

Full 73-point checklist in [references/verification-checklist.md](./references/verification-checklist.md). Below are the 5 critical quick-path checks.

### Quick Path (5 Commands)

**1. Version alignment** — all sources agree:

```bash
ssh bigblack '
VER=$(~/opendeviationbar-py/.venv/bin/python3 -c "import opendeviationbar; print(opendeviationbar.__version__)")
cd ~/opendeviationbar-py
TAG=$(git describe --tags --abbrev=0 2>/dev/null | sed "s/^v//")
CARGO=$(grep "^version" Cargo.toml | head -1 | sed "s/.*\"\(.*\)\"/\1/")
echo "Package=$VER Git=$TAG Cargo=$CARGO"
[ "$VER" = "$CARGO" ] && echo "ALIGNED" || echo "MISALIGNED"
'
```

**2. ClickHouse + Stathera gaps**:

```bash
ssh bigblack 'curl -sf "http://localhost:8123/ping" && echo " ClickHouse OK" || echo "ClickHouse DOWN"'
```

**2b. Yesterday-to-Today Stathera Boundary Check** -- verify zero gaps at the day boundary for all symbol+threshold combinations:

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "
SELECT
  symbol,
  threshold_decimal_bps,
  yesterday_last_tid,
  today_first_tid,
  today_first_tid - yesterday_last_tid - 1 AS tid_delta
FROM (
  SELECT
    symbol,
    threshold_decimal_bps,
    argMaxIf(last_agg_trade_id, close_time_us, toDate(close_time_us/1000000) = yesterday()) AS yesterday_last_tid,
    argMinIf(first_agg_trade_id, close_time_us, toDate(close_time_us/1000000) = today()) AS today_first_tid
  FROM opendeviationbar_cache.open_deviation_bars FINAL
  WHERE toDate(close_time_us/1000000) >= yesterday()
  GROUP BY symbol, threshold_decimal_bps
  HAVING yesterday_last_tid > 0 AND today_first_tid > 0
)
WHERE tid_delta != 0
FORMAT PrettyCompact"'
# Empty result = PASS (0 boundary gaps)
# Any rows = FAIL (gaps or overlaps at day boundary)
```

Empty result means zero Stathera gaps at the yesterday/today boundary. Any rows indicate gaps (positive `tid_delta`) or overlaps (negative `tid_delta`) that need investigation. If gaps exist, verify kintsugi completed its yesterday-repair pass (step 4.6c). If overlaps exist, run OPTIMIZE TABLE FINAL.

**3. Sidecar health + WS mode** (v3.0: /health exposes ws_mode and expected_ws_connections):

```bash
ssh bigblack 'curl -sf http://localhost:8081/health | python3 -m json.tool'
# Verify: ws_mode should be "combined" (default) or "per_symbol" (rollback)
# expected_ws_connections: 2 for combined, N for per_symbol
```

**3b. Junction telemetry** (v3.0: /status exposes REST→WS junction state):

```bash
ssh bigblack 'curl -sf http://localhost:8081/status | python3 -m json.tool'
# Verify per-symbol: rest_first_tid, rest_last_tid, ws_first_tid present
# Stathera check: ws_first_tid == rest_last_tid + 1 for each symbol
```

**4. Monit summary**:

```bash
ssh bigblack 'sudo monit summary 2>/dev/null'
```

**5. Checksum layer health** (v6.0+):

```bash
ssh bigblack 'cat ~/.cache/opendeviationbar/checksum-telemetry.jsonl 2>/dev/null | tail -3 || echo "No telemetry yet (expected on first deploy)"'
# Expected after first seeder run: recent verify_pass/verify_skip events
```

**6. Native extension + constants**:

```bash
ssh bigblack '~/opendeviationbar-py/.venv/bin/python3 -c "
from opendeviationbar._core import PyOpenDeviationBarProcessor
from opendeviationbar.constants import DEFAULT_THRESHOLD
from opendeviationbar.symbol_registry import get_thresholds_for_symbol
assert DEFAULT_THRESHOLD == 250
assert get_thresholds_for_symbol(\"BTCUSDT\") == (100, 250, 500, 750)
print(\"All imports + constants OK\")
"'
```

### Post-Deploy: Batch Repair Note

If running `repair_direct_parquet.py` after deploy, use `--workers 24` (or omit for auto-detected default). Thread-local CH cache (Issue #279) makes this fd-safe — ~4N+50 fds for N workers, well under `ulimit -n 1024`. See [scripts/CLAUDE.md Bigblack Capacity Envelope](/scripts/CLAUDE.md#bigblack-capacity-envelope) for full hardware budget.

### Verdict Table

After running all sections, summarize:

| Section                  | Status | Notes                                                                  |
| ------------------------ | ------ | ---------------------------------------------------------------------- |
| Network & Connectivity   |        |                                                                        |
| Package & Version        |        |                                                                        |
| Native Extension         |        |                                                                        |
| Process Management       |        |                                                                        |
| ClickHouse Health        |        |                                                                        |
| Stathera Audit           |        |                                                                        |
| Yesterday-Today Boundary |        | Step 2b: cross-day Stathera boundary check (0 gaps required)           |
| Sidecar /health          |        | v3.0: ws_mode + expected_ws_connections                                |
| Sidecar /status          |        | v3.0: junction telemetry (rest_first_tid, rest_last_tid, ws_first_tid) |
| Checksum Layer           |        | v6.0: bindings + registry + telemetry imports OK, JSONL events present |
| Infrastructure           |        |                                                                        |
| Monitoring               |        |                                                                        |
| Configuration            |        |                                                                        |
| Release Pipeline         |        |                                                                        |

Mark each PASS / WARN / FAIL / SKIP.

---

## Hotfix Path (Python-Only Changes)

When the change is pure Python (no Rust modifications), skip the full build pipeline:

1. Push changes to `origin/main`
2. Git sync on bigblack: `ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main --tags && git reset --hard origin/main'`
3. Rsync Python files to site-packages (excludes Rust .so):

   ```bash
   ssh bigblack '
   SITE_PKG=$(~/opendeviationbar-py/.venv/bin/python3 -c "import site; print(site.getsitepackages()[0])")
   rsync -av --exclude="__pycache__" --exclude="*.pyc" --exclude="_core*.so" \
     ~/opendeviationbar-py/python/opendeviationbar/ \
     $SITE_PKG/opendeviationbar/
   find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null; true
   '
   ```

4. **Clean-slate restart** (identical to Phase 4 full deploy — hotfix is NOT exempt):

   ```bash
   ssh bigblack 'sudo monit unmonitor sidecar 2>/dev/null; sudo monit unmonitor kintsugi 2>/dev/null'
   ssh bigblack 'systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-kintsugi-catchup 2>/dev/null; true'
   # Kill mutations (kintsugi may have pending DELETEs that race with new sidecar)
   ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "KILL MUTATION WHERE database = '"'"'opendeviationbar_cache'"'"'" > /dev/null'
   # Delete today's bars (prevents overlaps from different processor TID boundaries)
   ssh bigblack 'curl -s "http://localhost:8123/?mutations_sync=1" --data-binary "ALTER TABLE opendeviationbar_cache.open_deviation_bars DELETE WHERE toDate(close_time_us/1000000) = today()"'
   # Clear checkpoints (prevents crash-before-DELETE from stale schema)
   ssh bigblack 'rm -f ~/.cache/opendeviationbar/checkpoints/streaming_*.json'
   ssh bigblack 'systemctl --user start opendeviationbar-sidecar'
   sleep 8
   ```

   **Why all 3 steps matter** (learned 2026-03-28, 22 overlaps from skipping):
   - Kill mutations: pending kintsugi DELETEs can race with new sidecar writes
   - Delete today: different processor instances produce bars with different `first_agg_trade_id` — RMT can't dedup them
   - Clear checkpoints: stale schema crashes `_inject_checkpoints()` before the sidecar's own DELETE runs

   Running processes have the OLD code in memory; rsync only updates files on disk.

5. **Wait for sync_flush before starting kintsugi** (same as Phase 4.6 — applies to hotfixes too):

   ```bash
   # Use mise run deploy:verify-restart OR manual poll:
   for i in $(seq 1 30); do
     if ssh bigblack 'journalctl --user -u opendeviationbar-sidecar --since "5 min ago" --no-pager 2>/dev/null | grep -q "sync_flush.*drained and wrote"'; then
       echo "sync_flush complete"; break
     fi
     echo "Waiting... ($i/30)"; sleep 10
   done
   ssh bigblack 'systemctl --user start opendeviationbar-kintsugi'
   ssh bigblack 'sudo monit monitor sidecar 2>/dev/null; sudo monit monitor kintsugi 2>/dev/null'
   ```

6. Quick-path verification (Phase 5)

**NEVER** run `maturin develop` or `uv pip install -e .` on bigblack — it creates `.pth` files that shadow the wheel install and cause version drift on next deploy.

### Hotfix Gate: Always Clear Checkpoints (learned 2026-03-27, hardened 2026-03-28)

**Anti-pattern**: Python-only rsync without clearing checkpoints. Stale checkpoint files on disk have old schema; new code crashes with `AttributeError: 'NoneType' object has no attribute 'get'`. Crash happens in `_inject_checkpoints()` BEFORE the clean-slate DELETE runs, leaving old bars that cause overlaps when the sidecar eventually restarts successfully (#sidecar-restart-overlaps).

**Rule**: ALL hotfix deploys MUST clear checkpoints before restarting (step 4 above now includes this). This is a safe no-op when checkpoints are compatible — the sidecar's clean-slate DELETE + fill_from_rest rebuilds today regardless.

**If sidecar crash-looped before you cleared checkpoints**:

1. `systemctl --user reset-failed opendeviationbar-sidecar`
2. Check for overlaps: run Stathera verification query (Phase 5, step 2)
3. If overlaps exist: OPTIMIZE TABLE FINAL may resolve same-key duplicates; if not, use Pattern 2 from the restart playbook

---

## Rollback

If a deploy goes wrong:

### Option A: Restore Previous Venv (if .venv-old exists)

```bash
ssh bigblack '
systemctl --user stop opendeviationbar-sidecar opendeviationbar-kintsugi 2>/dev/null; true
cd ~/opendeviationbar-py
if [ -d .venv-old ]; then
  mv .venv .venv-bad && mv .venv-old .venv
  echo "Restored .venv-old"
fi
systemctl --user start opendeviationbar-sidecar opendeviationbar-kintsugi
'
```

### Option B: Install Specific Version

```bash
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv/bin/python3 --refresh-package opendeviationbar opendeviationbar==13.13.0"
```

Then restart services.

---

## Auto-Heal Procedures

When checks fail, execute these in order (dependencies flow top-to-bottom):

### Priority 1: Unpushed Commits

```bash
UNPUSHED=$(git log origin/main..HEAD --oneline 2>/dev/null)
if [ -n "$UNPUSHED" ]; then
  git push origin main || GH_TOKEN=$(gh auth token) git \
    -c "url.https://x-access-token:$(gh auth token)@github.com/.insteadOf=git@github.com-terrylica:" \
    push origin main
fi
```

### Priority 2: Git SHA Diverged on bigblack

```bash
ssh bigblack 'cd ~/opendeviationbar-py && git fetch origin main && git reset --hard origin/main'
```

### Priority 3: Version Mismatch

```bash
EXPECTED=$(grep '^version' Cargo.toml | head -1 | sed 's/.*"\(.*\)"/\1/')
ssh bigblack "uv pip install --python ~/opendeviationbar-py/.venv/bin/python3 --upgrade --refresh-package opendeviationbar opendeviationbar==$EXPECTED"
```

### Priority 4: Stale .so in Running Processes

```bash
ssh bigblack '
SO_EPOCH=$(stat -c %Y ~/opendeviationbar-py/.venv/lib/python*/site-packages/opendeviationbar/_core*.so 2>/dev/null | head -1)
for proc in "streaming_sidecar" "scripts/kintsugi"; do
  PID=$(pgrep -f "$proc" | head -1)
  if [ -n "$PID" ]; then
    START_EPOCH=$(date -d "$(ps -o lstart= -p $PID)" +%s 2>/dev/null)
    if [ -n "$START_EPOCH" ] && [ -n "$SO_EPOCH" ] && [ "$START_EPOCH" -lt "$SO_EPOCH" ]; then
      echo "STALE: $proc PID $PID — killing for monit respawn"
      sudo kill $PID
    fi
  fi
done
echo "Waiting 10s for monit respawn..."
sleep 10
pgrep -fa "streaming_sidecar|kintsugi"
'
```

### Priority 5: Dead Letters

```bash
ssh bigblack '
DL_COUNT=$(ls /tmp/opendeviationbar-dead-letter/ 2>/dev/null | wc -l)
if [ "$DL_COUNT" -gt 0 ]; then
  echo "$DL_COUNT dead letters — Charon replays every 5 min"
fi
'
```

---

## Lessons Learned (Hard-Won)

Full list in [references/lessons-learned.md](./references/lessons-learned.md). Critical ones:

### SSH Quoting with ClickHouse Queries

Complex SQL with single quotes (`'UTC'`) breaks when nested in SSH commands. **Solution**: Write query to temp file, SCP, execute remotely:

```bash
cat > /tmp/ch_query.sql << 'SQL'
SELECT count() FROM opendeviationbar_cache.open_deviation_bars FINAL
WHERE toDate(open_time_ms / 1000, 'UTC') != toDate(close_time_ms / 1000, 'UTC')
SQL
scp /tmp/ch_query.sql bigblack:/tmp/
ssh bigblack 'curl -s "http://localhost:8123/" -d @/tmp/ch_query.sql'
```

### Two Git Checkouts on bigblack

bigblack has `~/opendeviationbar-py` (where services run) and `~/eon/opendeviationbar-py` (another checkout). **Always** deploy to `~/opendeviationbar-py`.

### Venv vs Git Checkout Divergence

After `git pull`, the `.venv/lib/python3.13/site-packages/opendeviationbar/` still has the OLD wheel code. Services import from site-packages, not the git checkout. For Python-only hotfixes, rsync to site-packages. For Rust changes, full wheel install.

### Monit Must Be Unmonitored Before Service Stop

```bash
sudo monit unmonitor sidecar  # THEN systemctl stop
```

Otherwise monit restarts the service within 120s (daemon cycle).

### Heartbeat False Alarms During Restart

When kintsugi restarts (brief 0m-uptime window), heartbeat may report UNHEALTHY. This is timing — next cycle (5 min) shows healthy. Check `sudo monit summary` to confirm.

### zsh `status` Variable is Read-Only

On bigblack (zsh shell), never use `status` as a variable name in SSH commands. Use `svc_state` or similar.

### **pycache** Must Be Cleared After rsync

After rsyncing Python files, stale `.pyc` bytecode can cause import issues:

```bash
find $SITE_PKG -type d -name __pycache__ -exec rm -rf {} +
```

### Kintsugi Log Table Cleanup (v7.3+: NEVER TRUNCATE)

**Anti-pattern (BANNED since v7.3):** `TRUNCATE TABLE kintsugi_log` — destroys `result='exchange_gap'` permanent dismiss entries, causing kintsugi to re-attempt 96K+ unhealable gaps.

**Safe cleanup** (preserves exchange gap classifications):

```bash
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "ALTER TABLE opendeviationbar_cache.kintsugi_log DELETE WHERE result NOT IN ('"'"'exchange_gap'"'"') AND timestamp < now() - INTERVAL 7 DAY SETTINGS mutations_sync=1"'
```

**If you need to clear rate-limits for a specific repair run** (e.g., after deploying new shard discovery logic):

```bash
# Delete only clean/healed entries from last 2 hours (preserves exchange_gap + older entries)
ssh bigblack 'curl -s "http://localhost:8123/" --data-binary "ALTER TABLE opendeviationbar_cache.kintsugi_log DELETE WHERE result IN ('"'"'clean'"'"', '"'"'healed'"'"') AND timestamp > now() - INTERVAL 2 HOUR SETTINGS mutations_sync=1"'
```

### Data Integrity Verification: TID-Sequential Check (learned 2026-03-28)

**PATTERN (SSoT)**: Verify Parquet tick data completeness via TID-sequential check using Polars column-projected read. ~3.5ms/file, no network needed.

```python
import polars as pl
tids = pl.read_parquet(path, columns=["agg_trade_id"])["agg_trade_id"]
is_complete = bool((tids.diff().drop_nulls() <= 1).all())
```

**Anti-pattern (BANNED)**: Downloading entire Vision ZIP files to compute SHA-256 for integrity verification. ~200ms/file + network bandwidth. 57x slower than TID check.

**When to use**:

- Exchange gap classification: if TIDs sequential + 0 bars in TID range → exchange gap (permanent)
- Parquet cache audit: verify all files have sequential TIDs (no corruption)
- If TIDs are NON-sequential: download Vision CHECKSUM for THAT date only to verify (rare fallback)

**Replaces**: `scripts/integrity_sweep.py` (ZIP-download approach). The integrity sweep systemd timer (`opendeviationbar-integrity-sweep.timer`) is disabled. The seeder's on-download checksum verification (v6.0) remains for new downloads.

### Partition Migration Checklist (learned 2026-03-23)

If the PARTITION BY key changes (requires table recreation), follow this **exact** order:

1. **Stop ALL services** (sidecar, kintsugi, heartbeat)
2. **Create new table** with new partition key
3. **Copy data**: `INSERT INTO new SELECT * FROM old SETTINGS max_partitions_per_insert_block = 0`
4. **Verify counts**: `SELECT count() FROM old` vs `SELECT count() FROM new`
5. **Atomic rename**: `RENAME TABLE old TO backup, new TO production`
6. **OPTIMIZE TABLE FINAL** — **MANDATORY, wait for completion** (10-30 min for 19M+ bars). Without this, the table has 50K+ unmerged parts and every mutation times out at 300s.
7. **Verify parts**: `SELECT count() FROM system.parts WHERE active AND table = 'open_deviation_bars'` — target <10K
8. **Only then** restart services

**Lesson**: Skipping step 6 caused 5+ hours of cascading failures — kintsugi mutations timing out, phantom Stathera gaps, stuck repair queues. The parts count is the root cause, not the partition count.

### ClickHouse DELETE Strategy (#295 — 11 Iterations, Resolved)

The sidecar uses **pre-fill clean-slate `DELETE IN PARTITION`** (v13.49.0). Delete ALL today's bars per (symbol, threshold) partition BEFORE `fill_from_rest`. Scans only the target partition.

| Context          | Strategy              | Syntax                                              | Speed   |
| ---------------- | --------------------- | --------------------------------------------------- | ------- |
| Bulk cleanup     | `DROP PARTITION`      | `ALTER TABLE t DROP PARTITION (sym, thr)`           | Instant |
| Sidecar pre-fill | `DELETE IN PARTITION` | `ALTER TABLE t DELETE IN PARTITION (...) WHERE ...` | Fast    |
| Kintsugi replace | `DELETE IN PARTITION` | Same, with `mutations_sync=1`                       | Fast    |
| Manual recovery  | `DELETE IN PARTITION` | Same, with `mutations_sync=1`                       | Fast    |
| Fallback only    | `ALTER TABLE DELETE`  | Full table scan                                     | Slow    |

**Anti-pattern (BANNED):** `DELETE FROM` (lightweight DELETE) — creates `_row_exists=0` ghost rows causing false Stathera anomalies. Banned since v13.49.0.

**Current approach**: KILL mutations → OPTIMIZE FINAL → `DELETE IN PARTITION` today → midnight preflight → fill_from_rest → sync_flush.

### Start Kintsugi AFTER Sidecar sync_flush Completes (#295)

Kintsugi's `_do_replace()` creates DELETE mutations that scan all parts. If kintsugi starts before the sidecar's sync_flush writes today's bars, kintsugi mutations can delete freshly-written bars as collateral damage.

**Rule**: After starting sidecar, poll logs for `sync_flush: drained and wrote`, verify zero Stathera anomalies on today, THEN start kintsugi.

### Sidecar Startup: Skip Checkpoints + Skip CH Seeding (#295)

Stale checkpoints and CH seeds both override midnight preflight via `max(existing, override)`. The sidecar now:

1. Skips checkpoint injection entirely when `fill_from_rest` is available
2. Skips `_seed_trade_ids_from_clickhouse` — midnight preflight is the sole seed source
3. Uses fresh processors that start from the midnight crossover TID

---

## Monit Configuration Alignment

### 11 Monitors (SSoT: `deploy/monit/opendeviationbar.conf`)

| Monitor             | Type       | Check                                                             | Alert       |
| ------------------- | ---------- | ----------------------------------------------------------------- | ----------- |
| clickhouse          | host:8123  | HTTP /ping                                                        | 3 cycles    |
| sidecar             | process    | PID, mem<38.5GB, cpu<80%                                          | immediate   |
| sidecar-health      | host:8081  | HTTP /health                                                      | 3 cycles    |
| kintsugi            | process    | PID, mem<42.7GB                                                   | immediate   |
| heartbeat-timer     | program    | exit code check                                                   | 3×10 cycles |
| system              | system     | load<8, mem<70%, swap<25%                                         | 3/5 cycles  |
| redis               | host:6379  | Redis protocol                                                    | 3 cycles    |
| rootfs              | filesystem | space<80%/90%, inodes<85%                                         | immediate   |
| monit-heartbeat     | program    | self-report every 10 cycles                                       | 3 cycles    |
| seeder-state        | file       | `/var/tmp/opendeviationbar-seeder-state.txt` mtime <15 min        | 3 cycles    |
| exness-seeder-state | file       | `/var/tmp/opendeviationbar-exness-seeder-state.txt` mtime <30 min | 3 cycles    |

### Monit Files on bigblack

| Local Source                                  | Remote Target                                       |
| --------------------------------------------- | --------------------------------------------------- |
| `deploy/monit/opendeviationbar.conf`          | `/etc/monit/conf.d/opendeviationbar.conf`           |
| `deploy/monit/monit-override.conf`            | `/etc/systemd/system/monit.service.d/override.conf` |
| `deploy/monit/obdpy-monit-alert.sh`           | `/usr/local/bin/obdpy-monit-alert.sh`               |
| `deploy/monit/obdpy-monit-heartbeat.sh`       | `/usr/local/bin/obdpy-monit-heartbeat.sh`           |
| `deploy/monit/obdpy-check-heartbeat-timer.sh` | `/usr/local/bin/obdpy-check-heartbeat-timer.sh`     |

### Syncing Monit Config (Requires sudo)

```bash
scp deploy/monit/opendeviationbar.conf bigblack:/tmp/
ssh bigblack 'sudo cp /tmp/opendeviationbar.conf /etc/monit/conf.d/ && sudo monit reload'
```

### Monit Drop-in Override (for uid/gid exec + namespace isolation)

`deploy/monit/monit-override.conf` fixes systemd hardening that blocks monit's `as uid "tca"`:

- `NoNewPrivileges=false`
- `CapabilityBoundingSet` includes `CAP_SETUID`, `CAP_SETGID`, `CAP_SYS_PTRACE`
- `ProtectHome=no`
- `PrivateTmp=no` — **critical for seeder-state monitor**: monit upstream ships with `PrivateTmp=true`, giving it a private `/var/tmp/` mount namespace. Files written by services to the real `/var/tmp/` are invisible to monit's `check file` monitor. Without `PrivateTmp=no`, `seeder-state` permanently shows "Does not exist" even though the file exists. Diagnosed via `sudo nsenter -t $(systemctl show monit --property=MainPID --value) -m -- ls /var/tmp/`.

---

## Environment Files (SSoT)

| File                                | Tracked   | Purpose                | Deployed To                     |
| ----------------------------------- | --------- | ---------------------- | ------------------------------- |
| `deploy/opendeviationbar.env`       | git       | Non-secret config      | `~/opendeviationbar-py/deploy/` |
| `deploy/opendeviationbar.local.env` | gitignore | Secrets (TOKEN, Redis) | `~/opendeviationbar-py/deploy/` |
| `deploy/sidecar.env`                | git       | Streaming vars         | `~/opendeviationbar-py/deploy/` |

Services load both via `EnvironmentFile=` directives. `local.env` uses `EnvironmentFile=-` (optional, no error if missing).

### v3.0 Environment Variables

| Variable                             | Default    | Purpose                                                                                                                                | Set In               |
| ------------------------------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------------- | -------------------- |
| `OPENDEVIATIONBAR_STREAMING_WS_MODE` | `combined` | WS connection mode: `combined` (2 connections, 2x8 split for 16 crypto symbols) or `per_symbol` (rollback to N per-symbol connections) | `deploy/sidecar.env` |

**Rollback**: Set `OPENDEVIATIONBAR_STREAMING_WS_MODE=per_symbol` in `deploy/sidecar.env` and restart sidecar to revert to pre-v3.0 per-symbol WS connections. No code change needed.

**Parquet tick cache**: Sidecar startup (v3.0) reads `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet` to minimize REST API calls. Seeder timer (5 min) populates this cache. If cache is missing/stale, sidecar falls back to full REST fill (no regression).

---

## Systemd Service Reference

| Service          | Type    | MemoryMax | OOMScore | Purpose                                                                                       |
| ---------------- | ------- | --------- | -------- | --------------------------------------------------------------------------------------------- |
| sidecar          | simple  | 4G        | -500     | Real-time WS streaming (v3.0+: combined 2x8 for 16 crypto symbols, or per-symbol via WS_MODE) |
| kintsugi         | notify  | 20G       | -200     | Gap reconciliation + trailing-edge                                                            |
| kintsugi-catchup | notify  | 20G       | -200     | Historical backfill (Conflicts= with kintsugi)                                                |
| heartbeat        | oneshot | 256M      | 0        | Health telemetry every 5 min                                                                  |
| seeder           | oneshot | 6G        | 0        | Proactive Binance Parquet tick cache seeder                                                   |
| exness-seeder    | oneshot | 6G        | 0        | Proactive Exness forex tick cache seeder                                                      |

Timers: `heartbeat.timer` (every 5 min), `seeder.timer` (every 5 min), `exness-seeder.timer` (every 15 min).

All services: `TZ=UTC`, `MemorySwapMax=0`, `LD_PRELOAD=libmimalloc.so.2` (sidecar, kintsugi, seeder — all Polars DataFrame processors).

---

## Post-Execution Reflection

After this skill completes, reflect before closing the task:

0. **Locate yourself.** — Find this SKILL.md's canonical path (Glob for this skill's name) before editing. All corrections target THIS file and its sibling references/ — never other documentation.
1. **What failed?** — Fix the instruction that caused it. If it could recur, add it as an anti-pattern.
2. **What worked better than expected?** — Promote it to recommended practice. Document why.
3. **What drifted?** — Any script, reference, or external dependency that no longer matches reality gets fixed now.
4. **Log it.** — Every change gets an evolution-log entry with trigger, fix, and evidence.

## Do NOT defer. The next invocation inherits whatever you leave behind

## Post-Change Checklist

After modifying this skill:

- [ ] Phase ordering still respects dependencies (monit unmonitor → stop → deploy → start → monitor)
- [ ] SSH commands use correct quoting (no nested single-quote breaks)
- [ ] Anti-patterns table in Lessons Learned covers new failure modes
- [ ] Verification checklist (73 points) updated if new checks added
- [ ] Monit configuration table matches current `deploy/monit/opendeviationbar.conf`
- [ ] Systemd service table matches current unit files
- [ ] Environment variable table matches current `.env` files
- [ ] Agent team mode documentation consistent with sequential mode

---

## References

| File                                                                | Content                                                   |
| ------------------------------------------------------------------- | --------------------------------------------------------- |
| [verification-checklist.md](./references/verification-checklist.md) | Full 73-point checklist (11 sections)                     |
| [lessons-learned.md](./references/lessons-learned.md)               | All hard-won deployment lessons from production incidents |

## Related Skills

| Skill                                                      | Use When                                                              |
| ---------------------------------------------------------- | --------------------------------------------------------------------- |
| `/sidecar-restart-zero-overlap-zero-gap-recovery-playbook` | Restart recovery patterns, anti-patterns, today's data repair (SSoT)  |
| `/odb-telemetry`                                           | Reading production logs, ClickHouse queries, heartbeat interpretation |
| `/ops-monitoring`                                          | On-demand monitoring scripts, system resource checks                  |
| `/mise:run-full-release`                                   | Generic mise release pipeline (odb-ship is the ODB-specific superset) |
