probe · findings/evolution/shared_data/rotating_slice_orthogonality_probe.py

The auto-rotation coverage engine: testing features in every market condition

One engine that decides whether a new market feature is genuinely useful — by re-testing it in every kind of market (calm vs. wild, Asian hours vs. US/Europe hours, rising vs. falling) until no condition has been left under-tested. It keeps attacking whichever condition is least covered, so nothing slips through.

The problem it solves

A "feature" here is a number computed from market data — a candidate signal we might add to the production system. Before we trust it, we need to know two things: is it orthogonal (genuinely new information, not just a copy of something we already have) and non-redundant (not a near-duplicate of a sibling candidate)? The honest way to answer is to test the feature across every market condition it would have to survive in — not just one slice of the calendar. The two obvious approaches both fail:

• Fixed calendar slices (the old approach) — you test on, say, "January through June 2024." But whatever rare market conditions happened to fall in those dates get tested a lot, and the ones that didn't get tested barely or not at all. Coverage of unusual conditions is accidental.
• Pure random resampling — you draw random chunks of history. There's no guarantee every condition gets hit; rare ones (a high-volatility, Asian-session, down-trending market) get under-sampled by sheer bad luck.

The rotation mechanism replaces both with a coverage-ledger-driven, deficit-first loop: it lists out the full grid of conditions up front, then keeps attacking whichever condition is least-tested until every reachable one has been tested the target number of times.

The stratum grid

A stratum is one tiny, precisely-defined market situation — one cell of a big grid. The grid is the cross-product of these dimensions:

asset_class × symbol × threshold × vol(low/mid/high) × session(Asian/US-EU) × trend(up/flat/down)

Each dimension in plain terms:

• asset_class — forex (currencies/metals) or crypto. The same engine runs on both via a --asset switch.
• symbol — the specific instrument, e.g. BTCUSDT or EURUSD.
• threshold — how big a price move triggers a new bar, measured in dbps (decimal basis points; 1 dbps = 0.001%). Crypto uses 100/250/500/750; forex uses much smaller 1–50 because currency moves are tiny.
• vol (volatility) — how jumpy the market is, split into three buckets (low / mid / high) by ranking the rolling realized volatility. It is smoothed over 500 bars so each bucket is a long, coherent stretch of time rather than flickering bar-to-bar.
• session — which part of the trading day, by UTC clock: Asian = 00:00–08:00, US-EU = 12:00–20:00. Bars outside both bands are simply excluded (they aren't a target condition).
• trend — whether price sits above or below its own recent average (a 200-bar exponential moving average), normalized by its wobble: up if clearly above, down if clearly below, flat in between.

The base cells — every symbol × threshold combination — number 19 forex + 62 crypto = 81. Each base cell is then split into up to 3 vol × 2 session × 3 trend = 18 regime strata. (A regime is just one of those vol×session×trend combinations — a recognizable "mood" of the market.) So the engine is juggling on the order of hundreds of individual market situations.

Why it stratifies on value-rows, not bars

This is the clever bit that resolves an awkward tension. The candidate kernels (the formulas being tested) each need a contiguous causal window of 200 bars to compute their value — "causal" meaning the window only ever looks backward in time, never at future data, and respects gaps in the data. (A kernel is simply one candidate calculation in the registry.)

Here's the problem: if you first chopped the input bars down to a single regime (say, only the high-volatility Asian-session bars), you would shatter that 200-bar window into disconnected fragments — there'd be no clean trailing window to compute from. The fix is to stratify on the output values, not the input bars:

1. Fetch the full, continuous bar series for a base cell once.
2. Compute the candidate matrix — run every kernel at every STRIDE-th bar (stride = 20, so values overlap 10× across the 200-bar window). Each output value still sees a proper, unbroken trailing 200-bar window.
3. Attach the regime label (vol × session × trend) to each output value-row — a value-row being one computed result together with its market-condition tag.
4. A stratum's data is then just the subset of value-rows whose label matches that stratum.

The result: the kernels always see clean, look-ahead-free windows, while the sorting into market conditions happens afterward, on the outputs. The session-vs-window tension disappears.

The rotation loop

The engine works in batches, and it is fully resumable — it reads a saved ledger each time, so you can stop and restart. Each batch:

• Reads the coverage ledger and computes a deficit for every feasible stratum: deficit = target_N − paths_done (how many more tests it still needs).
• Picks the K strata with the largest deficit — the least-covered ones first.
• For each picked stratum, it pulls that stratum's real value-rows, then draws one resampled path using a stationary block bootstrap (Politis–Romano) — a resampling method that grabs short contiguous blocks rather than isolated points, so it preserves the local autocorrelation (the natural "stickiness" of market data) instead of scrambling it.
• It then runs the fused pass (below), records one per-path result, bumps paths_done, and persists the ledger.

It loops batch after batch until every feasible stratum has reached target_N = 20 tested paths.

The fused pass — two questions in one sweep

For each path, the engine answers both core questions at once:

One nuance: at the crypto 750 threshold (the spearman_only tier), each slice holds too few bars for the Chatterjee ξ to be reliable, so ξ is skipped there; the Spearman measure still runs.

The flowchart, verbatim from the source

  ┌──────────────────────────────────────────────────────────────────────┐
  │ BUILD STRATUM GRID  (once, on real bars)                              │
  │   asset × symbol × threshold × vol(L/M/H) × session(Asian/US-EU)       │
  │                                              × trend(up/flat/down)      │
  │   base cells: 19 forex + 62 crypto = 81  →  × up to 18 regime strata   │
  │   each tagged FEASIBLE (≥40 value-rows) or INFEASIBLE(reason) in ledger│
  └───────────────────────────────┬──────────────────────────────────────┘
                                  │
  ╔═══════════════════════════════▼══════════════════════════════════════╗
  ║ PER BATCH   (resumable · reads the persisted ledger)                   ║
  ║  1. deficit[s] = target_N − paths_done[s]   for every FEASIBLE s       ║
  ║  2. pick the K strata with the LARGEST deficit   (least-covered first) ║
  ║  3. group picks by cell; fetch + compute candidate matrix ONCE per cell║
  ║  4. for each picked stratum:                                          ║
  ║       a) take its REAL value-rows                                     ║
  ║       b) stationary block-bootstrap resample (Politis–Romano)         ║
  ║       c) FUSED PASS:  ρ + ξ vs production   +   LOO-R² + ρ vs siblings ║
  ║       d) paths_done[s] += 1 ; append result                           ║
  ║  5. persist ledger                                                    ║
  ╚═══════════════════════════════╤══════════════════════════════════════╝
                                  │
            ┌──────────────────────▼──────────────────────┐
            │  every FEASIBLE stratum reached target_N ?   │
            └───────┬───────────────────────────┬──────────┘
                NO  │                            │  YES
        ┌───────────▼──────────┐      ┌──────────▼─────────────────────────┐
        │ next batch loops back │      │ COVERAGE-COMPLETE                   │
        │ → auto-fills thinnest │      │  • verdict = worst-case over strata │
        │   (least-covered) cell│      │    + cross-asset replication        │
        └───────────────────────┘      │  • emit coverage map (N per cell;   │
                                       │    infeasible cells listed w/ reason)│
                                       └─────────────────────────────────────┘

Engine constants (from the source): WIN=200 (kernel window), STRIDE=20, FLOOR_BARS=1000 (min bars for reliable Chatterjee ξ), MIN_STRATUM_ROWS=40 (a stratum needs ≥40 value-rows to be testable), TARGET_N=20, VOL_SMOOTH=500, FETCH_BARS_DEFAULT=80000 recent bars pulled per cell.

What guarantees coverage

The guarantee comes from the ledger, not from randomness. Because every batch attacks the largest-deficit stratum, coverage fills in evenly and provably: the loop literally cannot terminate while any feasible stratum is still below target_N.

• Strata that simply cannot be reached — too few real bars exist — are written down as INFEASIBLE(reason). They are surfaced explicitly, never silently dropped, so there's never an unexplained gap.
• A feature is declared orthogonal / non-redundant only if it survives in the worst stratum (the hardest condition) and replicates across both asset classes (forex and crypto). Passing on average isn't enough — it has to pass where it's weakest.

Symbol & threshold plan

The exact instruments and thresholds the engine sweeps. This is the single source of truth (configurable in the ASSETS block at the top of the engine).

Forex — 4 symbols, 19 cells

Crypto — 16 symbols, 62 cells (thresholds 100/250/500/750)

All live in opendeviationbar_cache.open_deviation_bars (read from ClickHouse, the production columnar database, read-only). The 750-dbps threshold is the spearman-only tier (ξ skipped — too few bars per slice).

No symbol is dropped without a reason. The legacy 6-symbol set was a tractability choice under 2026-05-30 data limits (BTC/ETH only back to 2021, LTC@100 broken) that have since been repaired/backfilled — all 16 are now viable, so the engine uses all 16. The same 47 candidate kernels run on both asset classes (resolved from the shared mql5 campaign probe), which is exactly what makes the cross-asset comparison meaningful.

Column surface covered

Per slice, each candidate is compared against the usable production columns for that cell — the columns that pass a quality filter (≥20% non-null AND ≥5 distinct values):

• Forex ≈ 133 usable of 160 legal feature columns.
• Crypto ≈ 25 usable of 109 legal feature columns (the producer emits only trade/volume features; bid/ask/spread are impossible on Binance aggTrades, and the intra_/lookback_ columns are declared but not emitted). Crypto's breadth comes from 16-symbol cross-asset replication, not from column width.

Validation evidence

Two real-bars runs on bigblack confirm the engine works as intended — including a deliberate "we already know the answer" sanity check.