opendeviationbar-py · cost-realism labels · implementation ACTIVE

Crypto Cost-Realism Labels

Give crypto bars the same execution-cost safety net the forex side already has — so a strategy can never look profitable on paper by ignoring slippage and fees. This page explains the plan in plain English and draws the full pipeline, end to end.

Crypto figures above (0 today · ~139M bars) are the investigation's read-only ClickHouse snapshot, re-verified at implementation time before any number is cited as authority.

The problem, in plain English

A backtest that assumes you buy at a bar's closing price and later sell at a closing price is lying to you twice:

• You can't trade instantly. In the seconds it takes your order to reach Binance and fill, the price moves against you. That gap is slippage.
• Binance charges a fee on every trade. Ignore both, and a strategy can look profitable on paper while quietly losing money live.

The forex side of this project already records, for every bar, the worst price a real market order would actually have filled at within 3 seconds of the bar closing (plus the exit leg 3 seconds later, plus fair benchmark prices). The crypto side records none of it — verified live: the crypto table has zero such columns. This work closes that gap.

What we're proposing — one architecture

A separate table open_deviation_bar_labels, keyed on (symbol, threshold, first_agg_trade_id), filled in afterwards (offline, over the Parquet tick cache), and joined to the bars at read time.

Why not just add columns to the existing bar table? Three hard reasons from the investigation:

• Inline columns would get wiped. The bar table de-duplicates by keeping the newest version of each row; a label written after the fact loses to any later re-inserted (label-less) row version — which happens at the live edge and wherever a repair rewrites a range.
• The live streaming engine can't compute these. At the moment a bar closes it hasn't seen the next few seconds yet — and adding a second "healing" writer to the bar table would break the writer-ownership rule (#280) the whole pipeline depends on.
• A separate table dissolves the conflict. The label job owns its own table outright and coexists with the live sidecar + kintsugi with zero coordination. It also eliminates the entire class of "columns defined but never wired" bug that shipped a 0%-populated batch before (#556): one writer, one column list, nothing to forget.

The decisions on the table

Five are locked by evidence (not up for debate). Four more are resolved into this proposal — Terry can override any, but nothing is left blocking.

End-to-end flowchart — how we'll do it

The project pipeline from Terry's handoff to a queryable label table. The violet star is the gate we're at right now: nothing below it starts until Terry gives the green light.

The implementation loop — and where it stops

The build runs as an autonomous loop: each label goes oracle → kernel → wiring → guards → PR, and every iteration appends one HTML page to the journal below. The loop owns everything up to the open PR (green) and then stops dead — the red step is a hard halt. Merge, table creation, backfill, and deploy are operator-owned (amber) and never happen on the loop's own initiative.

How one label actually gets computed

Zoom in on a single bar. This is the compute kernel the Oracle PR pins bit-for-bit, then the backfill runs it over every bar in the tick cache.

Forex → crypto: keep / adapt / new

Terry named the forex side (mql5 repo, fxview-core) as the reference to mirror. The forex table carries 32 label columns across the full {1,3,5,10}s ladder (29 label_fwd_* + 3 derived); we port its 3s slice — 7 measured columns + 1 derived round-trip cost = 8 new columns on crypto. Here's the full delta.

The horizon question — resolved to 3s (Q1)

Q2–Q4 — resolved (Terry can override)

Implementation journal — append-only

Each iteration of the loop above appends one HTML page to this folder (index_iter_NN_<verbose_slug>.html) plus one row to the machine-readable ledger LOOP_LEDGER.json. Pages are never edited or overwritten — a new iteration is always a new sibling, so the full build history stays intact and auditable (same convention as the batch-5 / batch-6 implementation journals).

Machine-readable ledger: LOOP_LEDGER.json — append-only; iterations[] grows by one per page.

What happens next

1. The loop implements, label by label — S1 oracle → S2–S3 kernel (≤1e-9 bit-exact) → S4 wiring + guards → S5 opens a PR, then stops. One bounded stage per firing; each appends a journal page above.
2. The operator reviews + merges each PR — the loop never merges.
3. Then the operator-gated production steps — CREATE table → 139M-row backfill on bigblack → coverage audit → deploy — each authorized one at a time. The loop never touches these.

The loop only writes code and opens PRs; it touches no production system. Merge, backfill, and deploy remain operator-owned.