Pipeline Visualization — End-to-End

Each box has a plain-English summary (yellow italic) and a technical detail block (IN / OUT / Why / When). Read the yellow lines first to understand the flow; drill into the technical detail when you want precision.

Project-invented terms are italicized and link to the glossary. Click any actor box for its source CLAUDE.md. Phase 4 brief: actors-and-edges.md.

What this pipeline does (in plain English)

Crypto and forex markets emit individual trades all day. This system bundles those trades into Open Deviation Bars — special candle-like structures that close when the price moves a fixed percentage from the bar's open price (instead of when a clock ticks). Every bar carries 50+ statistical features.

Two parallel pipelines exist because crypto and forex came online at different times with different broker constraints. They share the algorithm idea but not the code:

Crypto track (left column) — Binance trades flow through opendeviationbar-py into a database. Library users (alpha-forge, backtesting.py, live SSE clients) consume the bars. No live trading from this side today.
Forex track (right column) — FXView broker quotes flow through MetaTrader 5 + the mql5 repo into a separate database. opendeviationbar-patterns mines those bars for trading rules; promising rules port to backtesting.py for validation, then to NautilusTrader for live trading against the broker.

🪙 Concrete example — one BTCUSDT trade (illustrative)

A trader buys 0.05 BTC at $60,150.30. Binance assigns aggTrade #2,143,000,000. The current open bar (250 dbps threshold) opened at $60,000.00 — upper breach is $60,150.00.

The trade arrives at STEP 1 via WebSocket → reaches the sidecar at STEP 3 → processor sees $60,150.30 ≥ $60,150.00 = breach, finalizes the bar (open=60,000, close=60,150.30) → bar lands in STEP 5 ClickHouse → reachable by alpha-forge / backtesting / SSE consumers at STEP 6. Total time: sub-second on the live path.

💱 Concrete example — one EURUSD tick (illustrative)

FXView quotes bid 1.08234, ask 1.08236. Open bar (5 dbps threshold) opened at 1.08200 — upper breach ~1.08254.

Tick arrives at STEP 1 → MT5 EA captures it at STEP 2, applies UTC conversion (with 4-signal HALT validating the offset), writes to 3 destinations. Live engine at STEP 4 reads the ring buffer, accumulates ticks, breaches threshold around 1.08245 → bar finalizes via fxview-core's BarBuilder at STEP 5 → lands in ClickHouse at STEP 6 → opendeviationbar-patterns at STEP 7 mines patterns → validates at STEP 8 → NautilusTrader at STEP 9 sends a real order. Total tick → bar: ~10 sec. Tick → live trading decision: minutes-to-hours.

Legend & how to read

📐 Layout

Two columns by track (CRYPTO left, FOREX right). Five rows by function (Sources → Ingestion → Storage → Execution → Cross-cutting). Read top-to-bottom within each column.

📖 Inside each box

Yellow italic — plain-English summary (what this does)
IN: / OUT: — what data flows in & out
Why: / When: / etc. — technical specifics in monospace

Read the yellow lines first to understand the flow. Drill into the technical lines for precision.

🎨 Color codes (per function)

▮ blue — sources
▮ cyan — caches
▮ green — processing engines
▮ orange — storage
▮ purple — consumers + execution
▮ slate dashed — cross-cutting

↪ Arrow types

Solid bold — primary data flow on the hot path
Dashed — secondary or parallel path (live-vs-backfill, dashboard publish)
Dotted — library dependency (F4/F5 use F6 shared crate)

All arrows are 90°-orthogonal (Manhattan routing). No diagonals.

🔢 Step numbering

Each box has a STEP badge. Crypto reaches STEP 6, Forex reaches STEP 9 (deeper because of the live/backfill split + research/validation/execution chain). Cross-cutting boxes have CROSS · IDs instead of step numbers.

🔗 Hyperlinks

Every actor box links to its source CLAUDE.md. Project-invented terms (Kintsugi, Stathera, Puck, OuroborosMode, fxview-core, etc.) link to their canonical entries in the glossary.