# External Integrations

## Overview

**opendeviationbar-py** integrates with multiple external systems for data fetching, caching, real-time streaming, monitoring, and notifications. All integrations use spot market data exclusively (no futures).

---

## Data Sources

### Binance REST API

**Endpoint**: `https://api.binance.com/api/v3`

**Purpose**: Historical aggTrade fetch, real-time trade stream fallback

**Integration Points**:

- **Crate**: `crates/opendeviationbar-providers/src/binance/` (Rust)
- **Python Wrapper**: `python/opendeviationbar/__init__.py` (via Rust bindings)
- **HTTP Client**: `reqwest` with rustls-tls (cross-compilation compatible)

**Data Format**:

```json
{
  "a": 123456789, // Aggregate trade ID
  "p": "42100.50", // Price
  "q": "0.5", // Quantity
  "f": 123456788, // First trade ID
  "l": 123456789, // Last trade ID
  "T": 1609459200000, // Trade time (milliseconds)
  "m": false, // Is buyer maker
  "M": false // Is best match (ignored)
}
```

**Rate Limiting**:

- Official limit: 6,000 weight/min per IP
- Env config: `OPENDEVIATIONBAR_REST_MAX_WEIGHT_PER_MINUTE=6000`
- Per-process ceiling: `OPENDEVIATIONBAR_REST_CEILING` (kintsugi=2400, sidecar=3000)
- Safety margin: 20% buffer via `OPENDEVIATIONBAR_REST_SAFETY_MARGIN=0.20`

**Key Features**:

- Puck (inline gap fill): Detects trade-ID discontinuity, fills via parallel REST
- Adaptive rate limiting with feedback from `X-MBX-USED-WEIGHT-1M` header
- Configurable retry with exponential backoff (backon crate)

**Files**:

- `crates/opendeviationbar-providers/src/binance/rest.rs`
- `python/opendeviationbar/config/resilience.py` (rate limit config)

---

### Binance Vision API (Historical Data)

**Endpoint**: `https://data.binance.vision/data/spot/daily/aggTrades/`

**Purpose**: Daily aggregated trade history (deterministic, precomputed by Binance)

**Archive Format**:

- Zipped CSV files: `{SYMBOL}/aggTrades/{SYMBOL}-aggTrades-YYYY-MM-DD.zip`
- **8 columns** (no header): `agg_trade_id`, `price`, `quantity`, `first_trade_id`, `last_trade_id`, `timestamp_ms`, `is_buyer_maker`, `is_best_match`
- **Constant schema** since 2017 (no breaking changes)

**Integration**:

- **Crate**: `crates/opendeviationbar-providers/src/binance/vision.rs`
- **HTTP Client**: `reqwest` streaming (zip decompression inline)
- **Data Guarantee**: Bit-exact match with REST API for same day boundary

**Data Integrity**:

- REST & Vision produce **identical day boundaries** (verified 6/6 pairs, including cross-year)
- Timestamps normalized to milliseconds (Vision CSV uses milliseconds, consistent with REST)
- All spot market only (no futures)

**Key Features**:

- Tier 1 local Parquet cache: `~/.cache/opendeviationbar/ticks/{SYMBOL}/YYYY-MM-DD.parquet`
- Multi-threshold efficiency: Download once, process at 250/500/750/1000 dbps
- ZIP download with automatic retry and checksum validation

**Files**:

- `crates/opendeviationbar-providers/src/binance/vision.rs`
- `python/opendeviationbar/storage/parquet.py` (local cache)

---

### Binance WebSocket Stream API

**Endpoints**:

- Combined streams: `wss://stream.binance.com:9443/stream`
- Format: `{"method":"SUBSCRIBE","params":["btcusdt@aggTrade"]}`

**Purpose**: Real-time trade updates for streaming sidecar

**Stream Format** (aggTrade):

```json
{
  "e": "aggTrade", // Event type
  "E": 1609459200123, // Event time (ms)
  "s": "BTCUSDT", // Symbol
  "a": 123456789, // Aggregate trade ID
  "p": "42100.50", // Price
  "q": "0.5", // Quantity
  "f": 123456788, // First trade ID
  "l": 123456789, // Last trade ID
  "T": 1609459200000, // Trade time (ms)
  "m": false // Is buyer maker
}
```

**Integration**:

- **Crate**: `crates/opendeviationbar-streaming/src/binance_websocket.rs`
- **WebSocket Client**: `tokio-tungstenite` (rustls, cross-compile compatible)
- **Async Runtime**: Tokio
- **Resilience**: Antaeus (exponential backoff with jitter via backon crate)

**Reliability Features**:

- **Puck**: Inline gap-fill on trade-ID discontinuity
- **Niffler**: Immediate dead-letter replay for failed writes
- **Charon**: Periodic dead-letter ferry (5-min cadence)
- **Argus**: Per-symbol liveness monitoring
- **Lethe**: Ring buffer overflow detection
- **Antaeus**: Resilient WS reconnection

**Watchdog**:

- Trade flow staleness detection (300s default)
- Auto-restart on zero trade increment for 5 min
- Max 3 restarts before sidecar exit (systemd restart)

**Files**:

- `crates/opendeviationbar-streaming/src/binance_websocket.rs`
- `python/opendeviationbar/sidecar.py` (sidecar orchestrator)

---

### Exness REST API (Forex)

**Purpose**: FX pairs (EURUSD, GBPUSD, etc.) via Exness Raw Spread data

**Integration**:

- **Crate**: `crates/opendeviationbar-providers/src/exness/`
- **HTTP Client**: `reqwest`
- **Feature Flag**: `data-providers` (optional)

**Usage**:

```python
from opendeviationbar import get_open_deviation_bars
df = get_open_deviation_bars("EURUSD", "2024-01-01", "2024-06-30")
```

**Note**: Exness integration is optional and feature-gated.

**Files**:

- `crates/opendeviationbar-providers/src/exness/`
- `python/opendeviationbar/exness.py`

---

## Databases

### ClickHouse (Tier 2 Cache)

**Host**: `bigblack` (remote GPU host, primary deployment target)

**Purpose**: Precomputed bar storage (42M+ bars, multi-threshold)

**Database**: `opendeviationbar_cache`

**Tables**:

| Table                    | Columns                                                                  | Purpose                                       |
| ------------------------ | ------------------------------------------------------------------------ | --------------------------------------------- |
| `open_deviation_bars`    | 50+ (OHLCV + 15 microstructure + 16 inter-bar + 22 intra-bar + metadata) | Computed bars (Tier 2 cache)                  |
| `population_checkpoints` | symbol, threshold, date_utc, bars_written, computed_at                   | Resume state for `populate_cache_resumable()` |
| `kintsugi_log`           | symbol, threshold, gap_start, gap_end, gap_duration_hours, repair_status | Gap repair history (Issue #115)               |

**Schema**:

- **Engine**: ReplacingMergeTree (deduplication on merge)
- **ORDER BY**: `(symbol, threshold_decimal_bps, ouroboros_mode, first_agg_trade_id)`
- **Partitioning**: By `toYYYYMMDD(close_time_ms)` (daily)
- **Compression**: ZSTD (default)

**Schema File**: `python/opendeviationbar/clickhouse/schema.sql`

**Connection**:

- **Local (dev)**: `localhost:8123`
- **Remote (prod)**: SSH tunnel `localhost:18123 → bigblack:8123`
- **Mode**: `OPENDEVIATIONBAR_MODE=remote` (current, no local ClickHouse)

**Client Library**: `clickhouse-connect` (Python) via `python/opendeviationbar/clickhouse/cache.py`

**Key Operations**:

- `populate_cache_resumable()` — Incremental population (day-by-day with checkpoints)
- `get_open_deviation_bars()` — Fetch from cache
- `get_n_open_deviation_bars()` — Count-bounded query
- `precompute_open_deviation_bars()` — Batch precompute (Tier 1 validation + write)
- `deduplicate_bars()` — Post-write OPTIMIZE (dedup Layer 3)

**Write Protection** (Issue #158, 4 layers):

- L1: Redis write lock per (symbol, threshold)
- L2: ReplacingMergeTree native dedup on merge
- L3: INSERT dedup token (MD5 of cache_key)
- L4: Pre-write gate (reject inverted OHLC, cross-midnight bars)

**Files**:

- `python/opendeviationbar/clickhouse/cache.py` (main API)
- `python/opendeviationbar/clickhouse/bulk_operations.py` (write paths)
- `python/opendeviationbar/clickhouse/write_lock.py` (Redis lock integration)
- `python/opendeviationbar/clickhouse/schema.sql` (table definitions)
- `python/opendeviationbar/clickhouse/stathera_query.py` (gap detection SQL builder)

---

### Redis (Distributed Write Lock)

**Host**: `localhost` (co-located on bigblack with ClickHouse)

**Purpose**: Prevent concurrent writes to same (symbol, threshold) from different processes

**Integration**:

- **Client**: `redis>=5.0` (Python)
- **Lock Pattern**: `opendeviationbar:write_lock:{symbol}:{threshold}`

**Configuration**:

- **Host**: `OPENDEVIATIONBAR_REDIS_HOST=localhost`
- **Port**: `OPENDEVIATIONBAR_REDIS_PORT=6379` (default)
- **Blocking Timeout**: `OPENDEVIATIONBAR_REDIS_LOCK_BLOCKING_TIMEOUT=2` (short timeout, fall through to L2 dedup)
- **DB**: 0 (default)

**Behavior**:

- Graceful degradation if Redis unavailable (L2 ClickHouse dedup catches overlaps)
- L1 protection prevents concurrent writers to same bar range
- Timeout prevents head-of-line blocking (kintsugi + sidecar + gap-backfill can proceed concurrently if targeting different ranges)

**Files**:

- `python/opendeviationbar/clickhouse/write_lock.py`
- `python/opendeviationbar/config/redis.py` (config)

---

## Monitoring & Notifications

### Telegram Bot API

**Bot**: `@obdpy_bot` (OpenDeviationBar Operations bot)

**Purpose**: Real-time alerts for cache operations, gaps, failures, heartbeats

**Notification Types**:

| Event                       | Behavior                                      | Example                                               |
| --------------------------- | --------------------------------------------- | ----------------------------------------------------- |
| Heartbeat (all checks pass) | Green ephemeral ✅ (auto-delete after 15 min) | "healthy: 0 gaps, 0 overlaps, 42M bars"               |
| Gap detected                | Red persistent 🔴 (stays until acknowledged)  | "STATHERA ALERT: 567 gaps found, kintsugi healing..." |
| Cache write failure         | Orange ⚠️ (circuit breaker triggered)         | "Circuit breaker OPEN: 3 consecutive CH failures"     |
| Sidecar crash               | Red 🔴 (service exit detected)                | "Sidecar PID 1234 exited unexpectedly"                |

**Configuration**:

- **Token**: `OPENDEVIATIONBAR_TELEGRAM_TOKEN` (deploy/opendeviationbar.local.env, not git-tracked)
- **Chat ID**: `OPENDEVIATIONBAR_TELEGRAM_CHAT_ID=90417581` (Terry Li @EonLabsOperations)

**Integration Points**:

- `python/opendeviationbar/notify/telegram.py` — Core notification functions
- `python/opendeviationbar/hooks.py` — Hook system for populate/cache events
- `scripts/health_heartbeat.py` — Every 5 min via systemd timer
- `deploy/monit/obdpy-monit-alert.sh` — Monit events
- `python/opendeviationbar/clickhouse/cache.py` — Write failure alerts

**Message Context** (Issue #117-119):

- Hostname (which server)
- Version (opendeviationbar release)
- Uptime (service stability indicator)
- Ouroboros mode (day/chunk, operational context)

**Files**:

- `python/opendeviationbar/notify/telegram.py`
- `deploy/monit/obdpy-monit-alert.sh`
- `scripts/health_heartbeat.py`

---

### monit Watchdog

**Purpose**: Process & system resource monitoring, automated alerts & restarts

**Config**: `deploy/monit/opendeviationbar.conf` (10 monitors)

**Monitors** (systemd services on bigblack):

| Monitor         | Type    | Check                                                      | Alert             | Action                    |
| --------------- | ------- | ---------------------------------------------------------- | ----------------- | ------------------------- |
| clickhouse      | host    | HTTP `/ping` on :8123                                      | Red ⚠️            | Telegram via alert script |
| sidecar         | process | PID, mem < 2048 MB, cpu < 80%                              | Rebuild container | Restart service           |
| sidecar-health  | host    | HTTP `/health` on :8081                                    | Red 🔴            | Restart watchdog checks   |
| kintsugi        | process | PID, mem < 5120 MB                                         | Rebuild container | Restart service           |
| heartbeat-timer | program | Exit code from `obdpy-check-heartbeat-timer.sh`            | Timer stalled     | Alert                     |
| seeder-state    | program | `/var/tmp/opendeviationbar-seeder-state.txt` mtime <15 min | Stale             | Alert                     |
| system          | system  | loadavg <8, mem <85%, swap <25%                            | Overload warning  | Alert                     |
| redis           | host    | Redis protocol on :6379                                    | Connection failed | Alert                     |
| rootfs          | fs      | Disk <80%/90%, inodes <85%                                 | Disk pressure     | Alert                     |
| monit-heartbeat | program | `obdpy-monit-heartbeat.sh` (every 10 cycles)               | —                 | Self-report               |

**Integration**:

- **Alert Script**: `deploy/monit/obdpy-monit-alert.sh` (Telegram)
- **Systemd Drop-in**: `deploy/monit/monit-override.conf` (CAP_SETUID for process supervision)
- **Cycle Time**: 60 seconds (default, configurable)

**Features**:

- Graceful degradation: continues operating if Telegram unavailable
- Debouncing: 3 cycles for transients, immediate for critical
- State file checks: validates timer execution (cross-systemd verification)

**Files**:

- `deploy/monit/opendeviationbar.conf`
- `deploy/monit/obdpy-monit-alert.sh`
- `deploy/monit/monit-override.conf`

---

### systemd Services & Timers

**Purpose**: Long-running daemons (sidecar, kintsugi, heartbeat) on bigblack

**Services** (5 on bigblack):

| Service                                     | Type    | Purpose                   | Env File                           |
| ------------------------------------------- | ------- | ------------------------- | ---------------------------------- |
| `opendeviationbar-sidecar.service`          | Simple  | Real-time streaming       | sidecar.env + opendeviationbar.env |
| `opendeviationbar-kintsugi.service`         | Simple  | Gap reconciliation daemon | opendeviationbar.env               |
| `opendeviationbar-kintsugi-catchup.service` | Oneshot | Historical backfill       | opendeviationbar.env               |
| `opendeviationbar-heartbeat.service`        | Oneshot | Health check              | opendeviationbar.env               |
| `opendeviationbar-seeder.service`           | Simple  | Tick cache seeder         | opendeviationbar.env               |

**Timers** (3 on bigblack):

| Timer                              | Interval | Purpose                                             |
| ---------------------------------- | -------- | --------------------------------------------------- |
| `opendeviationbar-heartbeat.timer` | 5 min    | Health monitoring (Telegram alerts)                 |
| `opendeviationbar-seeder.timer`    | 1 hour   | Proactive Parquet tick cache population             |
| Cron-like jobs                     | —        | Backfill coordination (via systemctl --on-calendar) |

**Configuration**:

- **Socket limits**: `LimitNOFILE=65536` (prevent fd exhaustion)
- **Memory limits**: `MemoryMax` + soft abort at 90% (graceful exit before SIGKILL)
- **OOM handling**: systemd-oomd integration (last-resort killer)
- **Environment**: `EnvironmentFile=` loads both .env files
- **Timezone**: `TZ=UTC` (enforced, all timestamps UTC)

**Dependency Chain**:

```
opendeviationbar-heartbeat.timer
    ↓
    opendeviationbar-heartbeat.service
        ↓ (depends on)
        opendeviationbar-sidecar.service
        opendeviationbar-kintsugi.service
        clickhouse.service (external)
```

**Restart Behavior**:

- Service crash: Restart with exponential backoff (30s → 1min → 2min...)
- Config reload: `SIGHUP` triggers `Settings.reload_and_clear()`
- Graceful shutdown: `SIGTERM` (60s timeout before SIGKILL)

**Files**:

- `scripts/systemd/*.service`
- `scripts/systemd/*.timer`
- `scripts/systemd/CLAUDE.md` (full configuration reference)

---

### Application Monitoring Ports (External, Not ODB-Managed)

| Service             | Port  | Purpose                                                   |
| ------------------- | ----- | --------------------------------------------------------- |
| ClickHouse HTTP API | 8123  | Query interface (health checks, data reads)               |
| ClickHouse Native   | 9000  | Binary protocol (clickhouse-connect uses this internally) |
| Prometheus          | 9090  | Metrics collection (bigblack infrastructure)              |
| node_exporter       | 9100  | Host metrics (CPU, memory, disk)                          |
| Netdata             | 19999 | Real-time dashboard                                       |
| Cockpit             | 9091  | System admin web UI                                       |

---

## API Integrations

### Sidecar Consumer API (HTTP)

**Purpose**: External consumers subscribe to live bar updates

**Endpoints**:

| Endpoint                               | Method | Purpose                                       | Example                                           |
| -------------------------------------- | ------ | --------------------------------------------- | ------------------------------------------------- |
| `/health`                              | GET    | Service liveness                              | Returns JSON: `{"status": "healthy"}`             |
| `/bars/forming`                        | GET    | SSE stream of forming bars                    | Real-time OHLCV updates                           |
| `/bars/checkpoint`                     | GET    | SSE stream of closed bars                     | Async checkpoint push                             |
| `/trades/ariadne/{symbol}/{threshold}` | GET    | Trade lookup (trade-ID based)                 | Pagination: `?from_agg_id=123456&max_trades=1000` |
| `/trades/gap-fill`                     | GET    | Gap-fill trades                               | `?symbol=BTCUSDT&from_agg_id=X&to_agg_id=Y`       |
| `/catchup/{symbol}/{threshold}`        | GET    | Single-call gap-fill from last ClickHouse bar | `?max_trades=50000`                               |

**Request/Response Format**:

```json
// /trades/gap-fill response (v12.62+)
{
  "trades": [
    {"a": 123456789, "p": "42100", "q": "0.5", "T": 1609459200000, "m": false},
    ...
  ],
  "partial": false,
  "count": 1234
}

// /ariadne/{symbol}/{threshold} response (v12.62+)
{
  "source": "parquet",
  "trades": [...],
  "sources_tried": ["live_processor", "checkpoint_file", "parquet_cache"],
  "first_agg_trade_id": 123456700,
  "last_agg_trade_id": 123457000
}
```

**Circuit Breaker**:

- State: CLOSED (normal) → OPEN (too many failures) → HALF_OPEN → CLOSED
- Threshold: 3 consecutive failures
- Recovery: 60s timeout

**Integration**:

- **Port**: 8081 (sidecar HTTP server)
- **File**: `python/opendeviationbar/sidecar_api.py`

---

## Data Validation & Serialization

### Arrow Streaming

**Purpose**: Zero-copy columnar data exchange between Rust and Python

**Integration**:

- **Crate**: `pyo3-arrow` 0.17 (buffer protocol)
- **Format**: Apache Arrow 58 RecordBatch
- **Use Case**: Large dataset processing (>10M trades)

**Example**:

```python
from opendeviationbar import process_trades_arrow
import pyarrow as pa

# Create Arrow RecordBatch
trades = pa.RecordBatch.from_pydict({
    "agg_trade_id": [1, 2, 3],
    "price": [42100, 42101, 42102],
    "quantity": [0.5, 0.3, 0.2],
    "timestamp_ms": [1609459200000, 1609459201000, 1609459202000],
    "is_buyer_maker": [False, True, False]
})

# Process with Rust backend
bars = process_trades_arrow(trades, threshold_decimal_bps=250)
```

**Files**:

- `src/arrow_bindings.rs`
- `crates/opendeviationbar-core/src/arrow.rs`

---

### JSON Configuration (1Password)

**Purpose**: Secret management for PyPI credentials

**Integration**:

- **Vault**: 1Password (OP_PYPI_VAULT)
- **Item**: OP_PYPI_ITEM (credentials for PyPI uploads)
- **CLI**: `op` command (mise task integration)

**Usage** (via mise task):

```bash
mise run release:pypi    # Automatically fetches credentials
```

---

## Summary Matrix

| System                | Protocol              | Purpose                      | Status   | File                                                         |
| --------------------- | --------------------- | ---------------------------- | -------- | ------------------------------------------------------------ |
| **Binance REST**      | HTTPS                 | Historical trades (fallback) | Active   | `crates/opendeviationbar-providers/src/binance/rest.rs`      |
| **Binance Vision**    | HTTPS (ZIP)           | Daily trade archives         | Active   | `crates/opendeviationbar-providers/src/binance/vision.rs`    |
| **Binance WebSocket** | WSS                   | Real-time trade stream       | Active   | `crates/opendeviationbar-streaming/src/binance_websocket.rs` |
| **Exness REST**       | HTTPS                 | FX pairs                     | Optional | `crates/opendeviationbar-providers/src/exness/`              |
| **ClickHouse**        | HTTP + SSH tunnel     | Tier 2 cache (42M bars)      | Active   | `python/opendeviationbar/clickhouse/cache.py`                |
| **Redis**             | TCP                   | Write lock                   | Active   | `python/opendeviationbar/clickhouse/write_lock.py`           |
| **Telegram**          | HTTPS API             | Notifications                | Active   | `python/opendeviationbar/notify/telegram.py`                 |
| **monit**             | TCP (config)          | Process supervision          | Active   | `deploy/monit/opendeviationbar.conf`                         |
| **systemd**           | IPC                   | Service management           | Active   | `scripts/systemd/*.{service,timer}`                          |
| **Arrow**             | IPC (buffer protocol) | Zero-copy data exchange      | Active   | `src/arrow_bindings.rs`                                      |

---

## Related Documentation

- **Architecture**: [/docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)
- **Data Sources**: [/crates/opendeviationbar-providers/CLAUDE.md](/crates/opendeviationbar-providers/CLAUDE.md)
- **Deploy Config**: [/deploy/CLAUDE.md](/deploy/CLAUDE.md)
- **Streaming**: [/docs/adr/2026-01-31-realtime-streaming-api.md](/docs/adr/2026-01-31-realtime-streaming-api.md)
- **Monitoring**: [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md)
