# opendeviationbar-providers

**Parent**: [/crates/CLAUDE.md](/crates/CLAUDE.md)

Rust crate implementing data provider adapters for Binance (crypto) and Exness (forex), normalizing into the unified `AggTrade` format.

---

## Binance Vision CSV Format History

**CRITICAL KNOWLEDGE**: Binance Vision spot aggTrades CSV format has CHANGED between 2024 and 2025.

### Format by Year

| Year Range   | Timestamp Column (field 6) | Format | Example Value      |
| ------------ | -------------------------- | ------ | ------------------ |
| 2019–2024    | 13-digit milliseconds      | ms     | `1655251200000`    |
| 2025–present | 16-digit microseconds      | μs     | `1749945600183572` |

### Column Layout (All Years, No Header)

All Binance Spot Vision ZIPs use a **headerless 8-column CSV**:

```
agg_trade_id, price, quantity, first_trade_id, last_trade_id, timestamp, is_buyer_maker, is_best_match
```

Columns 1–5, 7–8 are unchanged across all years. **Only column 6 (timestamp) changed.**

### How the Rust Code Handles Both

`crates/opendeviationbar-core/src/timestamp.rs` — `normalize_timestamp()`:

```rust
const MICROSECOND_THRESHOLD: u64 = 10_000_000_000_000;

pub fn normalize_timestamp(raw_timestamp: u64) -> i64 {
    if raw_timestamp < MICROSECOND_THRESHOLD {
        (raw_timestamp * 1_000) as i64  // ms → μs
    } else {
        raw_timestamp as i64            // already μs
    }
}
```

**Both formats are handled transparently** — no year-based branching needed.

### Python Downstream Handling

`_stream_bars_with_processor()` in `orchestration/helpers.py`:

```python
if use_native_arrow and "timestamp" in chunk.columns:
    # stream_binance_trades_arrow returns μs (after Rust normalize_timestamp).
    # process_trades_arrow() expects ms. Convert before processing.
    chunk = chunk.with_columns((pl.col("timestamp") // 1000).alias("timestamp"))
```

This `// 1000` (μs → ms) is always applied:

- **2019–2024 data**: Rust normalized 13-digit ms → 16-digit μs. Python divides back → 13-digit ms. ✓
- **2025+ data**: Rust passed 16-digit μs through unchanged. Python divides → 13-digit ms. ✓

### UM Futures vs Spot Difference

The README claims "Spot = 16-digit μs". This is accurate only for **2025+ data**. Historical spot data (2019–2024) used 13-digit ms. The `normalize_timestamp()` threshold handles the difference automatically.

| Market     | Column Layout     | Header | Timestamp (Current) | Timestamp (Pre-2025) |
| ---------- | ----------------- | ------ | ------------------- | -------------------- |
| Spot       | `a,p,q,f,l,T,m,b` | No     | 16-digit μs         | 13-digit ms          |
| UM Futures | Descriptive       | Yes    | 13-digit ms         | 13-digit ms          |

### Key Files

| File                                                          | Role                                             |
| ------------------------------------------------------------- | ------------------------------------------------ |
| `crates/opendeviationbar-core/src/timestamp.rs`               | `normalize_timestamp()` — ms/μs detection + conv |
| `crates/opendeviationbar-providers/src/binance/historical.rs` | Vision ZIP download + CSV parsing                |
| `python/opendeviationbar/orchestration/helpers.py`            | `_stream_bars_with_processor()` — μs → ms step   |

### Bug History: Vision ZIP "width 32 vs 31" (2026-03-12)

**Symptom**: `pl.concat(bar_frames)` in `_stream_bars_with_processor()` failed with "unable to append a DataFrame of width 32 with a DataFrame of width 31".

**Root cause**: When `include_incomplete=True`, the incomplete bar from `processor.get_incomplete_bar()` returns a Python dict (31 unconditional columns including `lookback_*` features), which has a different schema than the Arrow-format frames (32 columns: OHLCV + microstructure + trade IDs, NO lookback features). Mixing the two in `bar_frames` before `pl.concat` caused the mismatch.

**Fix** (`orchestration/helpers.py`): Two-part normalization before appending the incomplete bar to `bar_frames`:

1. **Width fix**: null-fill missing Arrow columns (e.g. `turnover`, `buy_trade_count`), drop dict-only columns (e.g. `lookback_*`)
2. **Type fix**: cast existing columns to match Arrow dtype (dict gives Int64 for `individual_trade_count`, `agg_record_count`, etc.; Arrow exports them as UInt32). Without this cast, ClickHouse INSERT fails with "type Int64 is incompatible with expected type UInt32".

**Confirmed**: This bug affected ALL years (not year-specific). The Vision timestamp format change (ms → μs) was a red herring; Rust's `normalize_timestamp()` already handled both correctly. Vision now writes ~100–700 bars/day at ~3 sec/day instead of REST fallback ~4 min/day.

---

## Spot-Only Data Source Policy

**All Binance data uses spot market exclusively. No futures.**

| Source                          | Endpoint                                                 | Notes                                 |
| ------------------------------- | -------------------------------------------------------- | ------------------------------------- |
| REST API                        | `https://api.binance.com/api/v3/aggTrades`               | Spot only. NOT `fapi.binance.com`     |
| Vision archives                 | `https://data.binance.vision/data/spot/daily/aggTrades/` | Spot, daily only. NOT `futures/um/`   |
| WebSocket (per-symbol, legacy)  | `wss://stream.binance.com:9443/ws/{symbol}@aggTrade`     | Spot, one connection per symbol       |
| WebSocket (combined, preferred) | `wss://stream.binance.com:9443/stream?streams=...`       | Spot, multiple symbols per connection |

**REST and Vision produce identical trade data** — verified empirically (6/6 date pairs including cross-year, all tid_gap=0, exact trade-ID match). Vision is preferred for efficiency (bulk ZIP), not correctness.

### WebSocket Architecture (validated 2026-03-16, Issue #279)

- **Use combined streams** (`wss://stream.binance.com:9443/stream?streams=...`), NOT per-symbol individual connections
- **2x9 hybrid**: 2 connections x 9 symbols each. Empirically best: 138.7 trades/s, zero errors, fault-isolated
- **Anti-pattern**: 18 individual connections caused recv timeouts on low-volume symbols (wifusdt, filusdt) and are susceptible to rate limiting under load
- **No rate limit feedback on WebSocket**: Binance provides no `X-MBX-USED-WEIGHT`. Limits enforced silently via drops. Design for zero-feedback.
- **Message format**: combined wraps in `{"stream":"...","data":{...}}`, individual sends data directly
- **Limits**: 1024 streams/connection, 300 connections/5min/IP, 5 control msgs/sec, 24h max lifetime

---

## Related

- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Workspace overview
- [/CLAUDE.md](/CLAUDE.md) - Project hub
- `crates/opendeviationbar-core/src/timestamp.rs` — `normalize_timestamp()`
- `crates/opendeviationbar-cli/src/bin/data_structure_validator.rs` — cross-year validation binary
