finding · findings/evolution/shared_data/README.md

Overview — the one shared data layer every test reads from

This is the front door to a folder that defines a single, frozen set of real market-data slices. Instead of each experiment grabbing its own data its own way, they all read from this one agreed-upon definition — so the answers stay comparable and reproducible.

What is this?

The folder is the hub-and-spoke entry point for the shared data layer: a short overview page (this one) that links out to deeper "spoke" documents. It holds a central, version-locked definition of real data slices that every orthogonality-related audit reads from. A "slice" here is one rectangle of data: a specific trading symbol, at a specific sensitivity setting, over a specific two-month calendar window.

Two different families of audits both feed off this same grid of slices:

• Metric evaluation — judging the measuring instruments themselves (do our orthogonality measurements work?).
• Candidate-feature evaluation — judging proposed new signals (does this new feature add anything?).

What lives in the folder:

• Pre-spec — the locked list of which symbols, sensitivity settings, and calendar windows are in scope. Immutable once locked
• Methodology & scope notes — the reasoning behind how the data is sliced, plus options for widening the scope later.
• The orthogonality probe — the one reusable program that reads each slice and measures whether candidate features are genuinely new information.
• Manifest — a leftover machine-readable index from an earlier design, kept only for reference.

Why it matters

The whole point of an orthogonality study is to trust the comparison. If every experiment pulled data in a slightly different way, you could never tell whether two results disagreed because of the idea being tested or because of mismatched inputs. By locking the data definition once and having every audit read from it, the folder removes that ambiguity. It is deliberately a "non-audit" folder: it holds the raw material and its provenance, but it issues no verdicts of its own — the judgments happen in the sibling audit folders that build on top of it.

How it works

The big architecture change (2026-05-30)

Originally the plan was fetch-to-file-then-read: pull each slice down into a local Parquet file (a compact columnar data file) and read from that. That fetcher (fetch_multi_slice.py) was never actually run — its index file manifest.json is empty — and it was removed. The replacement, orthogonality_probe.py, queries each slice live and directly from ClickHouse (the fast analytics database where the bar data is stored, running on the "bigblack" server). There is no intermediate file step anymore.

What the probe measures

For each slice it computes Axis-1 orthogonality using two fast statistics:

• Spearman ρ — a correlation score (between −1 and +1) that asks "do these two things move up and down together?" without assuming a straight-line relationship. A new feature that strongly correlates with what we already have is not orthogonal (not new).
• h_norm — a normalised information-overlap measure used alongside the correlation to judge how much genuinely independent signal a candidate adds.

It runs across all slices and reports, per candidate feature, how robust that "newness" looks from one slice to the next — i.e. is the feature consistently novel, or only in a lucky window?

Running it

python findings/evolution/shared_data/orthogonality_probe.py --smoke   # one-cell sanity check
python findings/evolution/shared_data/orthogonality_probe.py           # full 2021+ grid

The --smoke run is a quick one-cell sanity check; the full run sweeps the whole grid. Per-candidate results land in a results/ folder that is deliberately kept out of version control.

Key numbers

Reading the grid: 21 symbol/sensitivity combinations × 21 time windows = 441 cells. 3 cells are empty, leaving 438 with real data; of those, 24 cells (mostly at the most-sensitive 750 setting) hold too few bars to be trusted, so they're flagged and left out of the averages rather than silently included.

The locked scope (the "pantry inventory")

• Why only 21 slices, not 24: the three 2020 windows (slice22/23/24) are excluded because Bitcoin and Ethereum bars in this dataset only start on 2021-01-01 — there's no usable 2020 data to slice.
• The 3 empty cells: ETHUSDT@100 for slices 01/02/03 are empty because that data series ends 2025-06-17.
• Forex removed (v2, 2026-05-30): the layer is crypto-only for now. Forex (the EURUSD currency pair, from fxview_cache.forex_bars) can be reinstated later via the scope-expansion options; its availability findings are archived separately.

The spokes (where to go for depth)

Each spoke is a focused document you click into from the hub:

The house rules (how to use it)

• Want to know what data exists? Open pre-spec.md for the locked scope and data-quality-notes.md for which cells are actually populated.
• Want to measure orthogonality? Read orthogonality_probe.md, then run the probe on the bigblack server (read-only).
• Want to add symbols / thresholds / forex? That is a pre-spec scope change — it needs a new locked pre-spec, and the probe's symbol/threshold lists are updated to mirror it.
• Are you an audit author? Both sibling audits — the metric-evaluation audit (2026-05-16-orthogonality-measurement-instruments) and the candidate-feature audit (2026-05-08-orthogonal-features-btcusd) — consume the same slice grid. Run the probe live rather than reading pre-fetched files, and check purpose.md for the boundary rules.