# Technology Stack

## Overview

**opendeviationbar-py** is a Rust workspace with Python bindings via PyO3/maturin. The codebase implements a specialized bar construction algorithm for cryptocurrency trading, published to both PyPI (`opendeviationbar`) and crates.io.

---

## Languages & Runtime Versions

### Rust

- **Edition**: 2021 (workspace in `Cargo.toml`)
- **Minimum Rust Version**: 1.93
- **Profile**: Release (fat LTO for native, thin LTO for cross-compile)
  - Fat LTO: macOS ARM64 native builds
  - Thin LTO: Zig cross-compile (Linux x86_64)
  - No `panic = "abort"` — PyO3 uses `catch_unwind` to convert panics to Python exceptions

**File**: `Cargo.toml` (workspace root)

### Python

- **Version Required**: 3.13 only (enforced in `pyproject.toml` via `requires-python = ">=3.13"`)
- **Build System**: maturin 1.7–1.9 (uses Rust to compile Python extension)
- **Distribution**: Per-Python-version wheels (not universal ABI3 due to `pyo3-arrow` buffer protocol requirements)

**File**: `pyproject.toml`

**Platform Support**:

- macOS ARM64 (native maturin build, ~10 sec)
- Linux x86_64 (Zig cross-compile from macOS, ~55 sec)

---

## Core Frameworks & Libraries

### Rust Dependencies (from `Cargo.toml` workspace)

#### Data Processing

- **polars** 0.51.0 — DataFrame manipulation (lazy + temporal + strings + parquet + csv + IPC + rolling window)
- **csv** 1.3 — CSV parsing
- **zip** 2.2 — ZIP file handling
- **arrow** 58 — Apache Arrow columnar format (streaming architecture)
- **arrow-array** 58
- **arrow-schema** 58
- **pyo3-arrow** 0.17 — Arrow↔PyO3 zero-copy bridge

#### Serialization

- **serde** 1.0 — Serialization framework
- **serde_json** 1.0 — JSON
- **toml** 0.8 — TOML config parsing
- **chrono** 0.4 — Date/time with serde support

#### Concurrency & Async

- **tokio** 1.0 (features: rt-multi-thread, sync, time, macros, net, io-util, signal)
- **tokio-stream** 0.1
- **tokio-util** 0.7 (rt)
- **async-trait** 0.1 — Async trait definitions
- **futures** 0.3
- **futures-util** 0.3
- **rayon** 1.11 — Data parallelism (par_iter)

#### Network & HTTP

- **reqwest** 0.12 (rustls-tls, json, stream) — HTTP client, cross-compile compatible via rustls
- **tokio-tungstenite** 0.23 (rustls-tls-native-roots) — WebSocket, Binance integration
- **eventsource-client** 0.17 — SSE client

#### Analytics & Statistics

- **rolling-stats** 0.1 — Streaming statistics
- **tdigests** 1.0 — T-Digest approximation
- **linreg** 0.2.0 — Linear regression (Hurst)
- **rustfft** 6.4 — FFT (Hurst)

#### Performance & Utilities

- **parking_lot** 0.12 — Faster mutexes & RwLock
- **foldhash** 0.2 — Hash table
- **smallvec** 1.13 — Small vector optimization
- **libm** 0.2 — Math functions
- **quick_cache** 0.6 — In-memory cache
- **wide** 1.1 — SIMD utilities
- **num_cpus** 1.17 — CPU count detection

#### Error Handling & Observability

- **thiserror** 2.0 — Error derive macro
- **tracing** 0.1 — Structured logging
- **tracing-subscriber** 0.3 (env-filter, json) — Logging output to stderr for systemd journal

#### Memory Allocation

- **mimalloc** 0.1 — High-performance allocator (optional)

#### CLI & Web

- **clap** 4.0 (derive, env) — CLI argument parsing
- **config** 0.14 — Config file parsing
- **utoipa** 5.4 — OpenAPI schema generation

#### Resilience

- **backon** 1.6 — Exponential backoff with jitter (Antaeus reconnection pattern)

#### PyO3

- **pyo3** 0.28 (extension-module) — Python bindings
- **pyo3-build-config** 0.28 — Build configuration

#### Benchmarking

- **criterion** 0.5 (html_reports) — Benchmarks

#### Development

- **version-sync** 0.9 — Version consistency
- **rstest** 0.25 — Parameterized tests

#### Internal Crates

- **opendeviationbar-hurst** (path) — MIT-licensed fork of evrom/hurst (GPL-3.0 → MIT for PyPI compatibility)
- **opendeviationbar-core** (path) — Core algorithm & microstructure features
- **opendeviationbar-providers** (path) — Binance Vision/REST, Exness data sources
- **opendeviationbar-streaming** (path) — Real-time streaming engine
- **opendeviationbar-config** (path, internal)
- **opendeviationbar-io** (path, internal)
- **opendeviationbar-batch** (path, internal)
- **opendeviationbar-cli** (path, internal)
- **opendeviationbar** (path, internal)
- **opendeviationbar-client** (path) — HTTP client library

**File**: `Cargo.toml`, lines 41–127 (workspace.dependencies)

### Python Dependencies (from `pyproject.toml`)

#### Core Data Processing

- **pandas** ≥2.0 — DataFrames (backtesting.py compatibility)
- **numpy** ≥1.24 — Numerical arrays
- **polars** ≥1.0 — 2–3x faster alternative to pandas, with lazy execution
- **pyarrow** ≥14.0 — Arrow bridge for polars
- **arro3-core** ≥0.6.5 — Arrow integration

#### Configuration & CLI

- **pydantic-settings** ≥2.7 — Unified config from env/CLI/TOML
- **click** ≥8.0 — CLI framework
- **platformdirs** ≥4.0 — Cross-platform cache directories

#### Database & Cache

- **clickhouse-connect** ≥0.8 — ClickHouse client (Tier 2 remote cache on bigblack)
- **redis** ≥5.0 — Distributed write lock (Issue #158 Phase 7)

#### Notifications & Monitoring

- **requests** ≥2.28 — HTTP for Telegram API calls
- **loguru** ≥0.7.0 — Structured NDJSON logging
- **psutil** ≥5.9 — System monitoring (memory, disk, processes)

#### Utilities & Progress

- **tqdm** ≥4.66 — Progress bars
- **httpx** ≥0.28.1 — Async HTTP client
- **stamina** ≥24.3.0 — Production-grade retry decorator (Charon dead-letter replay)

#### Optional Dependencies

- **backtesting** ≥0.3 — Backtesting framework integration
- **atr-adaptive-laguerre** ≥2.5.2 — Laguerre feature plugin

#### Development

- **pytest** ≥7.0 — Testing framework
- **pytest-benchmark** ≥4.0 — Performance benchmarks
- **mypy** ≥1.0 — Static type checking (strict mode)
- **black** ≥23.0 — Code formatting (88 char line length)
- **ruff** ≥0.1 — Linting (comprehensive, 50+ rule categories)
- **hypothesis** ≥6.100 — Property-based testing
- **pandera[polars]** ≥0.20 — Schema validation
- **persim** ≥0.3.8 — Persistence diagrams (TDA research)
- **pytest-asyncio** ≥0.24 — Async test support
- **pytest-cov** ≥7.0.0 — Coverage reporting
- **ripser** ≥0.6.14 — Topological data analysis
- **river** ≥0.23.0 — Online ML
- **scipy** ≥1.17.0 — Scientific computing

**File**: `pyproject.toml`, lines 26–43, 48–58

---

## Build System

### Maturin

- **Version**: 1.7–1.9 (specified in `pyproject.toml`)
- **Role**: Builds PyO3 Rust extension modules into Python wheels
- **Python Source**: `python/` directory (layout: pure Python package + compiled `_core.so`)
- **Output Module**: `opendeviationbar._core` (compiled extension)

**Configuration**: `pyproject.toml` [tool.maturin]

- `python-source = "python"`
- `module-name = "opendeviationbar._core"`
- `features = ["pyo3/extension-module", "data-providers"]`
- `profile = "wheel"` (fat LTO stripped, optimized for distribution)

**Files**:

- `pyproject.toml` (lines 70–84)
- `Cargo.toml` [profile.wheel] (lines 300–304)

### Zig Cross-Compilation

- **Purpose**: Native Linux x86_64 wheels from macOS ARM64 without SSH/remote builds
- **Tool**: Zig (installed by mise)
- **rustflags**: `.cargo/config.toml` sets Zig cross-compile targets

**Command**: `mise run release:linux` (~55 sec)

**Files**: `.cargo/config.toml`

### Task Runner: mise

- **Purpose**: Dev environment + task automation (replacing `make` + environment scripts)
- **Configuration**: `.mise.toml` (hub) + `.mise/tasks/*.toml` (domain-specific tasks)
- **SSoT for**:
  - Environment variables (Rust min version, Python version, Binance thresholds)
  - Tool versions (Python 3.13, Zig, Node.js for semantic-release)
  - ClickHouse host configuration

**Key Task Files**:

- `.mise/tasks/dev.toml` — fmt, lint, test, build
- `.mise/tasks/release.toml` — 4-phase release workflow
- `.mise/tasks/bench.toml` — Benchmarking suite (17 criterion benches)
- `.mise/tasks/cache.toml` — ClickHouse operations
- `.mise/tasks/validate.toml` — Production validation

**File**: `.mise.toml` (lines 22–150)

---

## Configuration Management

### Environment-First Design

**Priority**: CLI > Environment Variables > TOML Config > Defaults

**SSoT for Thresholds & Runtime Behavior**: `.mise.toml`

| Category                 | Env Prefix                         | Examples                                                | SSoT File                     |
| ------------------------ | ---------------------------------- | ------------------------------------------------------- | ----------------------------- |
| Ouroboros (day-mode)     | —                                  | `OPENDEVIATIONBAR_OUROBOROS_MODE=day`                   | `.mise.toml`                  |
| Per-symbol min threshold | `OPENDEVIATIONBAR_MIN_THRESHOLD_*` | `OPENDEVIATIONBAR_MIN_THRESHOLD_BTCUSDT=250`            | `.mise.toml`                  |
| Microstructure features  | `OPENDEVIATIONBAR_COMPUTE_*`       | `COMPUTE_TIER2=false`, `COMPUTE_HURST=false`            | `.mise.toml`                  |
| Streaming                | `OPENDEVIATIONBAR_STREAMING_*`     | `STREAMING_GAP_FILL=true`, `WATCHDOG_TIMEOUT_S=600`     | `.mise.toml`                  |
| ClickHouse cache         | `OPENDEVIATIONBAR_CH_*`            | `CH_HOSTS=bigblack`, `CH_PRIMARY=bigblack`              | `.mise.toml`                  |
| Telegram notifications   | `OPENDEVIATIONBAR_TELEGRAM_*`      | `TELEGRAM_CHAT_ID=90417581`                             | `.mise.toml`                  |
| Redis write lock         | `OPENDEVIATIONBAR_REDIS_*`         | `REDIS_HOST=localhost`, `REDIS_LOCK_BLOCKING_TIMEOUT=2` | `deploy/opendeviationbar.env` |
| Kintsugi parallelism     | `OPENDEVIATIONBAR_KINTSUGI_*`      | `MAX_CONCURRENT_REPAIRS=24` (tuned for bigblack)        | `deploy/opendeviationbar.env` |

**Key Files**:

- `.mise.toml` (repo root) — Development defaults
- `deploy/opendeviationbar.env` — Production bigblack config (git-tracked, non-secret)
- `deploy/opendeviationbar.local.env` — Secrets (Telegram TOKEN, Redis password) — **NOT git-tracked**
- `deploy/sidecar.env` — Streaming-specific overrides
- `python/opendeviationbar/config/` — Pydantic BaseSettings for programmatic access

**Python Config Classes** (`python/opendeviationbar/config/`):

- `PopulationConfig` — Cache population knobs
- `MonitoringConfig` — Telegram, heartbeat thresholds
- `ClickHouseConfig` — DB connection, SSH tunnel
- `SidecarConfig` — Streaming watchdog, backpressure
- `AlgorithmConfig` — Batch size, memory optimization
- `StreamingConfig` — Channel limits, circuit breaker
- `RedisConfig` — Write lock configuration
- `Settings` — Singleton that composes all above

---

## Data Format & Serialization

### Input Formats

- **CSV**: Binance Vision daily archives (zipped, 8-column aggTrade format)
- **Parquet**: Local tick cache (Tier 1), normalized daily
- **Arrow RecordBatch**: Streaming input (zero-copy via pyo3-arrow)

### Output Formats

- **Pandas DataFrame**: OHLCV (backtesting.py compatible)
- **Polars DataFrame**: Lazy + eager variants (2–3x faster)
- **Arrow RecordBatch**: Streaming export
- **ClickHouse**: Remote cache (Tier 2)

### Internal: Microstructure Features

- **Count**: 53 total (15 microstructure + 16 inter-bar + 22 intra-bar)
- **SSoT**: `crates/opendeviationbar-core/data/feature_manifest.toml`
- **Format**: TOML with full documentation (unit, range, formula, tier)

**File**: `crates/opendeviationbar-core/data/feature_manifest.toml`

---

## Key Crate Architecture (10 Crates)

### Published to crates.io

1. **opendeviationbar-core** — Core bar algorithm + microstructure features
2. **opendeviationbar-providers** — Binance Vision, Binance REST, Exness data sources
3. **opendeviationbar-streaming** — Real-time streaming engine
4. **opendeviationbar-client** — HTTP client library
5. **opendeviationbar-hurst** — Hurst exponent estimators (MIT-licensed fork)

### Internal (publish=false)

6. **opendeviationbar-config** — Configuration
7. **opendeviationbar-io** — I/O utilities (parquet read/write)
8. **opendeviationbar-batch** — Batch processing
9. **opendeviationbar-cli** — CLI tools
10. **opendeviationbar** (PyO3 Python bindings, also unpublished)

**Workspace Root**: `Cargo.toml` (resolver = "2", workspace members)

---

## PyO3 Bindings Structure

### Extension Module Name

- **Python**: `opendeviationbar._core`
- **Rust**: `src/lib.rs` (PyO3 module entry point)
- **Module Registration**: `#[pymodule] fn _core(_py: Python, m: &Bound<'_, PyModule>)`

### Domain Modules (Issue #94: Refactored from 2999-line monolith)

- `helpers.rs` — Type conversions
- `core_bindings.rs` — `PyOpenDeviationBarProcessor`, `PyAggTrade`, `PyPositionVerification`
- `binance_bindings.rs` — Binance data fetching (feature-gated)
- `exness_bindings.rs` — Exness integration (feature-gated)
- `arrow_bindings.rs` — Arrow export (feature-gated)
- `streaming_bindings.rs` — `LiveBarEngine` (feature-gated)
- `stream_manager_bindings.rs` — `StreamManager` (Issue #161)
- `odb_engine_bindings.rs` — Unified engine (Issue #178, feature-gated)
- `gap_bindings.rs` — Gap event watching (Issue #257, feature-gated)
- `thread_pool_init.rs` — Rayon thread pool init

### Rust→Python API Layers

| Layer             | Files                                        | Purpose                                            |
| ----------------- | -------------------------------------------- | -------------------------------------------------- |
| **Bindings**      | `src/*.rs`                                   | PyO3 pyclasses + functions                         |
| **Python API**    | `python/opendeviationbar/__init__.py`        | Public functions (`get_open_deviation_bars`, etc.) |
| **Type Stubs**    | `python/opendeviationbar/__init__.pyi`       | IDE/mypy support (re-exports only)                 |
| **Orchestration** | `python/opendeviationbar/orchestration/`     | Request flow, circuit breaker, count-bounded API   |
| **Cache**         | `python/opendeviationbar/clickhouse/`        | Tier 2 ClickHouse integration                      |
| **Storage**       | `python/opendeviationbar/storage/parquet.py` | Tier 1 local Parquet cache                         |

**File**: `src/lib.rs` (100 lines, thin orchestrator)

---

## Features & Feature Flags

### Rust Features (`Cargo.toml`)

| Feature          | Default | Components                   | Purpose                    |
| ---------------- | ------- | ---------------------------- | -------------------------- |
| `data-providers` | Yes     | Binance + Exness fetching    | Full API (auto-fetch data) |
| `arrow-export`   | Yes     | Arrow RecordBatch export     | Streaming architecture     |
| `streaming`      | Yes     | LiveBarEngine, StreamManager | Real-time processing       |

### Conditional Compilation

- `#[cfg(feature = "data-providers")]` — Binance/Exness modules
- `#[cfg(feature = "arrow-export")]` — Arrow output functions
- `#[cfg(feature = "streaming")]` — Streaming engine

**File**: `Cargo.toml` [features] (lines 180–191)

---

## Development Tools (via mise)

| Tool      | Version | Purpose                          |
| --------- | ------- | -------------------------------- |
| Python    | 3.13    | Runtime                          |
| Zig       | Latest  | Cross-compilation (Linux x86_64) |
| Rust      | 1.93+   | Compilation                      |
| Node.js   | Latest  | semantic-release (versioning)    |
| Maturin   | 1.7–1.9 | PyO3 wheel builder               |
| pytest    | 7.0+    | Python testing                   |
| mypy      | 1.0+    | Type checking                    |
| ruff      | 0.1+    | Linting                          |
| black     | 23.0+   | Formatting                       |
| criterion | 0.5+    | Benchmarking                     |

**Configuration**: `.mise.toml` (tool installation and environment setup)

---

## Performance Optimizations

### Build Profiles

| Profile         | LTO  | Codegen Units | Strip   | Use Case                                   |
| --------------- | ---- | ------------- | ------- | ------------------------------------------ |
| **release**     | fat  | 1             | —       | Native macOS builds (maximum optimization) |
| **wheel**       | thin | 1             | symbols | Cross-compile & distribution (balanced)    |
| **pgo-collect** | thin | 1             | —       | PGO data collection (Issue #96)            |
| **pgo-use**     | thin | 1             | —       | PGO optimized build (Issue #96)            |
| **dev**         | —    | —             | —       | Fast iteration                             |

**Panic Strategy**: `panic = "unwind"` (required for PyO3, never "abort")

**File**: `Cargo.toml` [profile.*] (lines 293–323)

### Compiler Flags (`.cargo/config.toml`)

- Stripped symbols for wheel size
- SIMD flags for performance (architecture-specific)
- Rustflags for cross-compile (Zig linker)

---

## Related Documentation

- **Architecture**: [/docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)
- **Rust Crates**: [/crates/CLAUDE.md](/crates/CLAUDE.md)
- **Python API**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md)
- **Build/Release**: [/docs/development/RELEASE.md](/docs/development/RELEASE.md)
- **Data Sources**: [/crates/opendeviationbar-providers/CLAUDE.md](/crates/opendeviationbar-providers/CLAUDE.md)
