Locked data scope: the pre-registration of what the probes may read

A frozen, version-controlled contract that pins down exactly which crypto data our research probes are allowed to look at — which coins, which settings, and which date windows — so that nobody can quietly cherry-pick the data that makes a result look good.

In plain English. Before running an experiment, serious scientists write down the rules in advance and lock them — that's "pre-registration." This document is that locked rule sheet for our crypto data. It says: read only these 6 coins, only at these price-sensitivity settings, only inside these 21 two-month time windows, and only by looking at the database, never changing it. Locking the rules first means a later result can't be accused of being tuned to flatter itself.

The headline numbers

Current locked version (crypto-only)

Crypto symbols

Two-month time windows

441

Total data cells (21×21)

438

Cells with real data (3 empty)

Independent market regimes

What is this?

A locked scope file. pre-spec.md defines the precise data the "shared_data" layer is allowed to read. It is the canonical scope — the single source of truth.
Version 2, crypto-only. Version 1 was locked on 2026-05-23; it was amended to v2 on 2026-05-30 and is ACTIVE.
Mirrored in code. A script, orthogonality_probe.py (the "probe" — the program that measures the data), reads the exact same scope. It now pulls each piece of data directly and read-only from ClickHouse — ClickHouse is the big database on the bigblack server that stores the bars. There is no longer a middle step of copying data out to files first.

Why it matters

If you let a researcher (or a tuning loop) freely choose which symbols, settings, and dates to include, you can accidentally — or deliberately — keep only the slices that happen to support your conclusion. That is "data dredging," and it makes a result look real when it isn't. Locking the scope before looking at outcomes removes that freedom. Any later verdict can cite the exact version it was computed against ("computed against shared_data pre-spec v2"), and the file's git history (its change log) is the permanent, tamper-evident record of what the rules were at each point in time.

What changed from v1 to v2 (2026-05-30)

The scope was deliberately narrowed. Each change below has a documented reason — nothing was dropped on a whim.

Change	Why (in plain words)
Forex removed — the EURUSD foreign-exchange data (at the 5 dbps setting) is gone	An operator directive narrowed the study to crypto only "for now." The forex findings weren't thrown away — they're archived in `forex-availability-probe.md` and a reintroduction plan sits in `scope-expansion-options.md`.
2020 dropped — the three 2020 time windows (slice22/23/24) were removed	Bitcoin and Ethereum bars only begin on 2021-01-01 in our database, so 2020 windows were empty or patchy. Window count fell from 24 → 21.
LTCUSDT @ 100 dbps dropped	Empirically broken: only 26 bars total exist in the database for that one combination — far too few to measure anything (re-confirmed 2026-05-30).
Threshold "tiers" added	The `750` setting is marked `spearman_only` (too few bars in a 2-month window for the heavier metrics); `100/250/500` are marked `full`. See the pool's robustness map.
Fetch step removed	The old file-copying script `fetch_multi_slice.py` was never actually run and has been deleted. The probe now reads slices live from the database.

A note on jargon used above

dbps ("decimal basis points") is the price-move setting that decides when a new bar is created. 1 dbps = 0.001%. So 100 dbps = 0.1%, 250 dbps = 0.25%, 750 dbps = 0.75%. A bigger number means a bar only forms after a bigger price move, so you get fewer, coarser bars.
bar — a summary of price action over a stretch of trading (open, high, low, close). These "open deviation bars" start a new bar whenever price moves a set percentage away from where the bar opened.
Spearman — a way of measuring whether two things move together in rank order (a correlation that doesn't assume a straight line). When a setting is "spearman_only," only this lighter measure is trustworthy because the data is too thin for the heavier ones.

How it works: the data substrate

"Substrate" just means the underlying raw material the probes stand on. In v2 it is purely crypto, read straight from the production database.

Setting	Value	What it means
ClickHouse host	`bigblack` (via Tailscale)	The remote server holding the data, reached over the team's private network.
Database	`opendeviationbar_cache`	The specific database name.
Table	`open_deviation_bars`	The specific table of price bars.
Access mode	read-only `SELECT`	The probe may only look, never add, edit, or delete.
Ordering key	`close_time_us`	Rows are ordered by each bar's close time, in UTC microseconds (exchange/venue time).

Forex (fxview_cache.forex_bars) was "Substrate 2" in v1 — removed in v2. It can be switched back on later via scope-expansion-options.md.

The six symbols and their settings

Each cell of the study is one combination of (symbol, threshold setting, time window). Below is which thresholds each coin is studied at. "Bar-layer first bar" is the real earliest date with bars (2021-01-01 for all defaults — note this differs from the trade-level start date listed in symbols.toml).

Symbol	Thresholds (dbps)	First bar	Covers all 21 windows?
BTCUSDT	100, 250, 500, 750	2021-01-01	yes — BTC@750 ends 2026-01-20, so the newest window (slice01) is slightly cut short
ETHUSDT	100, 250	2021-01-01	mostly — ETH@100 ends 2025-06-17, so windows 01/02/03 are empty
BNBUSDT	100, 250, 500, 750	2021-01-01	yes
LTCUSDT	250, 500, 750	2021-01-01	yes — LTC@100 dropped (only 26 bars)
ADAUSDT	100, 250, 500, 750	2021-01-01	yes
XRPUSDT	100, 250, 500, 750	2021-01-01	yes

Tier map: 100 / 250 / 500 = full (all metrics valid), 750 = spearman_only (only the lighter rank-correlation metric is reliable).
Cell count: the symbol-threshold combinations add up to (4+2+4+3+4+4) = 21 combos, times 21 windows = 441 cells.
3 cells are empty (ETHUSDT@100 in windows 01/02/03) → 438 populated cells.
~24 cells (mostly the 750 setting) have too few bars and are flagged meets_floor_1000=false, so they're left out of the heavier "conditional metric" calculations.
Every cell is read live from ClickHouse — there are no intermediate data files.

The 21 time windows (2021 onward)

Each window is exactly 2 calendar months, in UTC, non-overlapping, and listed newest-first. The regime column is a coarse label for what the market was doing — useful for checking whether a result holds across very different market conditions ("regime-stratified robustness"). The three 2020 windows are crossed out because there were no usable BTC/ETH bars that far back.

Slice	Window (UTC)	Market context (BTC-centric)
slice01	2026-01-01 → 2026-02-28	2026 H1 — current state
slice02	2025-10-01 → 2025-11-30	2025 Q4
slice03	2025-07-01 → 2025-08-31	2025 H2 mid
slice04	2025-04-01 → 2025-05-31	2025 Q2
slice05	2025-01-01 → 2025-02-28	2025 H1 start
slice06	2024-10-01 → 2024-11-30	2024 Q4
slice07	2024-07-01 → 2024-08-31	2024 H2 mid
slice08	2024-04-01 → 2024-05-31	2024 Q2 — post all-time-high
slice09	2024-01-01 → 2024-02-29	2024 H1 — ETF era
slice10	2023-10-01 → 2023-11-30	2023 Q4 recovery
slice11	2023-07-01 → 2023-08-31	2023 H2 mid
slice12	2023-04-01 → 2023-05-31	2023 Q2 mid
slice13	2023-01-01 → 2023-02-28	2023 H1 — post-FTX
slice14	2022-10-01 → 2022-11-30	2022 Q4 — FTX collapse
slice15	2022-07-01 → 2022-08-31	2022 H2 — 3AC aftermath
slice16	2022-04-01 → 2022-05-31	2022 Q2 — LUNA collapse
slice17	2022-01-01 → 2022-02-28	2022 H1 — top turning bear
slice18	2021-10-01 → 2021-11-30	2021 all-time-high region
slice19	2021-07-01 → 2021-08-31	2021 mid-bull
slice20	2021-04-01 → 2021-05-31	2021 mid-bull
slice21	2021-01-01 → 2021-02-28	2021 early-bull
~~slice22~~	~~2020-10-01 → 2020-11-30~~	EXCLUDED — 2020, no BTC/ETH bars
~~slice23~~	~~2020-07-01 → 2020-08-31~~	EXCLUDED — 2020, no BTC/ETH bars
~~slice24~~	~~2020-04-01 → 2020-05-31~~	EXCLUDED — 2020, no BTC/ETH bars

Why "~7 regimes," not 21 samples. Neighbouring windows from the same market era move together, so they aren't truly independent. The windows group into about seven distinct "regime epochs": bull_2021 (slices 18–21) · bear_2022 (slice 17) · stress_2022 (slices 14–16) · recovery_2023 (slices 10–13) · bull_2024 (slices 06–09) · range_2025 (slices 02–05) · current_2026 (slice 01). Treat the data as ~7 independent regime samples, not 21 — otherwise you'd overstate how much evidence you have.

The rules the probe must obey (invariants)

An "invariant" is a rule that must hold true every single time. These are baked into the probe code so the locked scope is actually enforced, not just promised.

Rule	How it's enforced
`read_only` — only look-ups (`SELECT`) ever touch the database	The probe issues `SELECT` only — never INSERT / ALTER / DELETE / DROP / OPTIMIZE (i.e. nothing that could change the data).
`boundary_inclusion` — every query stays inside its window	Each query hard-codes `close_time_us >= start AND close_time_us <= end`.
`min_year` — 2020 windows are excluded	A `MIN_YEAR = 2021` filter is applied to the schedule.
`cpu_budget` — keep it to a single CPU core by default	The math libraries are pinned to 1 thread (`OPENBLAS/OMP/MKL/NUMEXPR/NUMBA_NUM_THREADS=1`), overridable if needed.
`floor_guard` — too-small cells are skipped or flagged	Cells with fewer than 200 bars are skipped entirely; fewer than 1000 are flagged `meets_floor_1000=false` and left out of the heavier aggregation.
`numeric_only` — text columns can't corrupt the math	Everything is coerced to numbers, then any column that's more than 80% empty is dropped.

How you change the scope (versioning rules)

To change scope, bump the Version in the file header and add a change-log row (exactly as v1 → v2 did).
Then update the matching lists in the code — SYMBOL_THRESHOLDS, SLICES, THRESHOLD_TIERS in orthogonality_probe.py — so document and code stay in lockstep. The probe is the "executable mirror" of this scope.
Every later verdict cites the version it used ("computed against shared_data pre-spec v2"); the file's git history is the permanent version trail.
Bigger scope changes (Options B and C, plus the path to bring forex back) are pre-analyzed in scope-expansion-options.md — switching to a broader scope is itself a version bump.

Proving which rules were used: the hash check

When this file is committed to git, the exact content produces a unique fingerprint (a "content hash") that becomes the canonical identifier for that version. So any past result that says "computed on shared_data v1" can be re-verified by going back to the commit that first added v1 and comparing the fingerprint:

git log --diff-filter=A findings/evolution/shared_data/pre-spec.md
# Find the commit that first added v1; compare the file's hash

The same fingerprint is also recorded per-cell, so each cell of data carries proof of which version of the rules produced it.

Bottom line. This file is the frozen, version-controlled contract that says exactly which crypto data the probes may read — 6 symbols, four threshold settings, 21 two-month windows (438 of 441 cells populated), read-only from ClickHouse — so that every downstream result is anchored to rules that were locked in advance and can be re-verified by its version hash.

Crypto Shared Data Pool · finding · source: findings/evolution/shared_data/pre-spec.md · ← back to the pool · 2026-06-16