# Streaming Sidecar Package

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Long-running sidecar that streams live trades from Binance WebSocket via Rust `StreamManager`, constructs open deviation bars in real-time, and writes them to ClickHouse. Modularized from a single `sidecar.py` (Phase 01 v1.5).

---

## Files

| File                  | Purpose                                                                                                                                            |
| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| `__init__.py`         | Main orchestrator: `run_sidecar()`, `_create_engine()`, `_main_loop()`, watchdog, Argus liveness, Lethe overflow detection, signal handlers        |
| `__init__.pyi`        | Type stubs for `run_sidecar`, `SidecarConfig` re-export, and all public names                                                                      |
| `sync_flush.py`         | `_sync_flush_rest_bars()` dispatch (Arrow vs dict drain), `_sync_flush_rest_bars_dict()` legacy fallback                                           |
| `http_server.py`      | HTTP health endpoint (`/health`), SSE forming bar push, consumer API routes (`/ariadne`, `/catchup`, `/trades/gap-fill`), junction state tracking  |
| `checkpoint.py`       | Streaming checkpoint persistence: `_save_checkpoint()`, `_load_checkpoint()`, `_inject_checkpoints()`, `_extract_checkpoints()` to/from JSON files |
| `clickhouse_write.py` | `_write_bars_batch_to_clickhouse()` with retry, Niffler dead-letter fallback on write failure, batch accumulation                                  |
| `config_helpers.py`   | Lazy-loaded config accessors: `_resilience_cfg()`, `_sidecar_cfg()`, `_backfill_cfg()` -- avoids circular imports at module level                  |
| `dead_letter.py`      | Niffler dead-letter serialization (`_niffler_dead_letter()`), immediate replay (`_niffler_immediate_replay()`), Charon periodic ferry              |
| `notifications.py`    | Telegram alert functions: `_notify_watchdog_trigger()`, `_notify_charon_replay()`, `_notify_startup()`, `_notify_shutdown()`                       |
| `recovery.py`         | `_restart_engine_with_recovery()` -- graceful engine shutdown with timeout guards, checkpoint extraction, fresh engine creation                    |
| `arrow_flush.py`      | Arrow RecordBatch drain + Polars zero-copy flush (#313, v7.2)                                                                                      |

---

## Startup Sequence

The sidecar startup invariant (#294, #295) is documented in root [/CLAUDE.md](/CLAUDE.md). Here is which files implement each step:

**Startup sequence** (Aion, all symbols):

| Step | Action                                 | Implemented In                            |
| ---- | -------------------------------------- | ----------------------------------------- |
| 1    | Start WS (buffered mode)               | `__init__.py` (`_create_engine()`)        |
| 2    | Kill mutations                         | `__init__.py` (`_create_engine()`)        |
| 3    | Find resume TID per symbol from CH     | `__init__.py` (`_create_engine()`)        |
| 4    | DELETE bars after resume_tid           | `__init__.py` (`_create_engine()`)        |
| 5    | Seed engine with resume TIDs           | `__init__.py` (`_create_engine()`)        |
| 6    | `fill_from_rest` (from resume_tid + 1) | `__init__.py` (`_create_engine()`)        |
| 7    | `sync_flush` (INSERT bars to CH)       | `sync_flush.py` (`_sync_flush_rest_bars()`) |

**Aion resume design**: Resume TID = `max(max(last_agg_trade_id))` across all thresholds for a symbol (switches to `min()` if repair checkpoints are available). `fill_from_rest` processes trades from `resume_tid + 1` to now, filling any gap between the last committed bar and the present. All thresholds share the same trade stream, so one fill covers all.

After startup, `_main_loop()` in `__init__.py` polls `engine.next_bar()` and writes via `clickhouse_write.py`.

## Writer Ownership

The sidecar owns everything from `resume_tid + 1` forward. Kintsugi handles historical gaps older than the resume point (#280).

## Self-Healing Layers

| Concept  | File                 | Behavior                                                                |
| -------- | -------------------- | ----------------------------------------------------------------------- |
| Niffler  | `dead_letter.py`     | Serialize failed bars to Parquet, attempt immediate replay              |
| Charon   | `dead_letter.py`     | Periodic ferry of dead-letter files back to ClickHouse                  |
| Argus    | `__init__.py`        | Per-symbol liveness monitoring via phi-accrual failure detector         |
| Lethe    | `__init__.py`        | Ring buffer overflow detection -- alerts when bars are dropped          |
| Watchdog | `__init__.py`        | Stale trade flow detection -- triggers engine restart via `recovery.py` |
| Antaeus  | Rust `StreamManager` | Resilient WS reconnection with exponential backoff                      |

---

## v7.2 Memory Efficiency

Issue #313 eliminates ~200K Python dict allocations per `sync_flush` cycle.

### Arrow drain path (`arrow_flush.py`)

`drain_bars_arrow()` returns an Arrow RecordBatch via FFI with `_symbol` and `_threshold_decimal_bps` metadata columns baked in. `_sync_flush_rest_bars_arrow()` uses `pl.DataFrame.filter()` for zero-copy per-pair slicing (no `partition_by` data duplication).

### Dispatch (`sync_flush.py`)

`_sync_flush_rest_bars()` dispatches at runtime: `hasattr(engine, "drain_bars_arrow")` selects the Arrow path, otherwise falls back to the dict-based `_sync_flush_rest_bars_dict()`.

### Allocation tags

| Tag      | Change                                                                                                                                                      |
| -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| ALLOC-01 | Single `OpenDeviationBarCache` instance shared across all startup phases and main loop (5-tuple return from `_create_engine`). Not used as context manager. |
| ALLOC-02 | `_write_bars_batch_to_clickhouse` accepts `pl.DataFrame` or `list[dict]` -- Arrow path passes DataFrame directly, no conversion to dicts.                   |
| ALLOC-03 | `symbol_modes` dict built once in `_create_engine` and threaded through all write paths (sync_flush, main loop).                                            |

---

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) -- Parent: Python API layer
- [/CLAUDE.md](/CLAUDE.md) -- Hub: startup sequence invariant (#294, #295), writer ownership (#280)
- [/scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md) -- systemd service config for `opendeviationbar-sidecar.service`
- [/deploy/CLAUDE.md](/deploy/CLAUDE.md) -- Monit watchdog, env files, defense-in-depth monitoring
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) -- ClickHouse write operations used by `clickhouse_write.py`
