---
name: full-symbol-streaming-spike
description: "COMPLETED (v8.0). Spike validated sidecar streaming all symbols via combined WebSocket streams. Now 16 crypto symbols via 2x8 combined WS (forex disabled). Eliminates persistent midnight-boundary Stathera gaps caused by kintsugi/backfill join at day boundaries. Sidecar startup is Aion-native: resume from min(max(last_agg_trade_id)) across thresholds, TID-based DELETE, no midnight crossover TID for crypto."
triggers:
  - full symbol streaming
  - combined websocket
  - all symbols streaming
  - sidecar all symbols
  - midnight gap fix
  - eliminate stathera gaps
---

# Full-Symbol Streaming Spike

> **STATUS: COMPLETED (v8.0 Aion-Native Pipeline).** All phases shipped. 16 crypto symbols stream via 2x8 combined WS connections. Forex symbols disabled (`enabled=false` in symbols.toml). Sidecar startup is Aion-native for crypto: resume from `min(max(last_agg_trade_id))` across thresholds per symbol, TID-based DELETE (not date-based), no midnight crossover TID, no checkpoints for Aion symbols.

## Problem Statement (Historical)

The sidecar only streamed BTCUSDT and ETHUSDT (`deploy/sidecar.env`). The other symbols
got data from kintsugi + trailing-edge backfill using independent processor instances.
At day boundaries, the two processors didn't share state — creating persistent Stathera gaps.

**Root cause proven 2026-03-19**: `opendeviationbar_version` tags confirm Mar 18 bars are
`+kintsugi-parquet` and Mar 19 bars are `+backfill` — two independent writers with a join gap.
BTCUSDT/ETHUSDT had **zero** midnight gaps because Puck handles them in real-time.

## Current State (v8.0)

### WebSocket Architecture

- **Combined streams SHIPPED**: 2 WS connections with 8+8 symbol split (16 crypto symbols total)
- Combined stream URL format: `wss://stream.binance.com:9443/stream?streams=s1@aggTrade/s2@aggTrade/...`
- **Binance limits**: 1024 streams/connection, 300 connections/5min/IP, 5 control msgs/sec, 24h max lifetime
- **Forex symbols**: disabled in `symbols.toml` (`enabled=false`). Day-mode path still exists but guarded behind `if day_symbols:`

### Aion-Native Startup (v8.0)

- Resume TID: `min(max(last_agg_trade_id))` across thresholds per symbol (NOT midnight crossover TID)
- DELETE: TID-based `WHERE first_agg_trade_id > resume_tid` (NOT date-based)
- No checkpoints for Aion symbols
- Exchange gap floor: resume TID clamped above highest known exchange gap from kintsugi_log
- Parquet-first gap fill: sidecar reads Parquet tick cache BEFORE REST API
- Junction telemetry: logged at REST→WS handoff (tid_delta, buffer_depth, skipped_trades)

## Spike Phases

### Phase 1: Quick Win — Add SOLUSDT to sidecar (per-symbol WS, no Rust changes)

**Goal**: Prove that adding one more symbol eliminates its midnight gaps.

1. Edit `deploy/sidecar.env`:

   ```
   OPENDEVIATIONBAR_STREAMING_SYMBOLS=BTCUSDT,ETHUSDT,SOLUSDT
   ```

2. Restart sidecar on bigblack:

   ```bash
   ssh bigblack 'systemctl --user restart opendeviationbar-sidecar'
   ```

3. Wait for one UTC midnight cycle (or at least several hours).

4. **Verification**: After midnight, check SOLUSDT has zero Stathera gaps at the day boundary:

   ```sql
   SELECT first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) OVER w - 1 AS tid_delta
   FROM open_deviation_bars FINAL
   WHERE symbol = 'SOLUSDT'
   WINDOW w AS (PARTITION BY symbol, threshold_decimal_bps ORDER BY first_agg_trade_id)
   HAVING tid_delta > 0 AND toDate(close_time_ms/1000) >= today() - 1
   ```

5. **Success criteria**: SOLUSDT midnight gaps = 0 (same as BTCUSDT/ETHUSDT).
   Stathera gap count drops from 13 to 10 (the 3 SOLUSDT@500/750/1000 gaps disappear).

**Risk**: Adding a 3rd per-symbol WS connection is trivial. Binance allows 300 connections/5min.
This phase requires ZERO code changes — just a config change.

### Phase 2: Combined WebSocket Streams in Rust (SHIPPED)

Combined streams implemented. 2 WS connections with 8+8 symbol split (16 crypto symbols).

### Phase 3: Enable All Crypto Symbols (SHIPPED)

16 crypto symbols streaming via 2x8 combined WS connections. Forex symbols disabled (`enabled=false` in symbols.toml).

## Anti-Patterns

- **DO NOT** use 16 individual WS connections — causes recv timeouts on low-volume symbols (#279)
- **DO NOT** increase REST ceiling beyond 3000 weight/min — Puck gap-fill needs burst capacity
- **DO NOT** remove kintsugi — it still handles historical backfill and yesterday's repairs
- **DO NOT** disable the today-guard — sidecar ownership of today is the right model
- **DO NOT** use midnight crossover TID for crypto symbols — v8.0 Aion-native uses `min(max(last_agg_trade_id))` resume
- **DO NOT** use date-based DELETE for crypto — v8.0 uses TID-based `WHERE first_agg_trade_id > resume_tid`

## Success Criteria (Full Spike) — ALL MET

- [x] All 16 crypto symbols streaming via 2 combined WS connections
- [x] Zero Stathera midnight-boundary gaps after one full UTC day cycle
- [x] Heartbeat shows clean bar-level continuity
- [x] Sidecar RSS < 1GB steady state
- [x] Ring buffer drops = 0
- [x] Puck gap-fill working for all 16 symbols
- [x] No Binance rate limit errors in sidecar journal

## Files to Modify

| File                                                         | Change                                              |
| ------------------------------------------------------------ | --------------------------------------------------- |
| `deploy/sidecar.env`                                         | Add symbols to `OPENDEVIATIONBAR_STREAMING_SYMBOLS` |
| `crates/opendeviationbar-providers/src/binance/websocket.rs` | Add `connect_combined_stream()`                     |
| `crates/opendeviationbar-streaming/src/live_engine.rs`       | Replace `symbol_task` with `combined_stream_task`   |
| `src/streaming_bindings.rs`                                  | Expose combined stream config to Python             |
