finding · findings/evolution/shared_data/slicing-methodology.md

How the data is sliced — and why

A guided tour of how the shared crypto market-data pool is cut into ~441 testable pieces — across time, across coins, and across "how big a price move counts as a bar" — and the plain-English reasoning behind every cut.

What is this?

The shared data pool is the central data layer that every research "probe" reads from. Rather than test a hypothesis on one blob of data, we slice the available history into a clean grid of cells. Each cell is one self-contained sample. Slicing along three independent ("orthogonal" — meaning they don't interfere with each other) directions lets us ask three separate reliability questions at once.

The three slicing dimensions

As of 2026-05-30 the pool is crypto-only (forex was removed — see below). The data lives in ClickHouse (a fast analytical database) on the bigblack server, and is read read-only, on demand — there is no intermediate file-export step.

• Time axis — 21 non-overlapping 2-month calendar windows, spanning Jan 2021 → Feb 2026. (2020 is excluded because BTC/ETH bars only begin on 2021-01-01.)
• Symbol axis — 6 crypto coins.
• Threshold axis — per coin, a set of "bar sensitivity" thresholds [100, 250, 500, 750] (ETH uses only [100, 250]; LTC drops the 100 level). Each threshold carries a tier — either full or spearman_only.

The cross-product is 21 (symbol, threshold) combinations × 21 time slices = 441 cells, of which 438 actually contain data.

Why three axes?

Each axis answers a different "would this verdict hold up?" question:

• Time axis → regime stability. Does the result survive across different market eras (bull runs, crashes, recoveries)? A regime epoch is a stretch of market behaving in one characteristic way.
• Symbol axis → cross-coin portability. Does the result hold for more than just Bitcoin?
• Threshold axis → density robustness. Does it hold whether each bar is built from many trades (dense) or few (sparse)? This is where the special tier boundary at 750 dbps lives — below the floor where the more demanding metrics stop being reliable.

Varying the verdict across this much real, varied data is our complete substitute for synthetic-data calibration — the project's policy is to never use made-up data, so instead we lean on the diversity of real data.

Why ~21 time windows?

The schedule originally defined 24 windows, but the three 2020 windows (slice22/23/24) were dropped because BTC/ETH bars don't exist before 2021 — those cells would be empty or ragged. So the active count is K = 21.

Where K=24 came from

The 24-fold number is inherited from Terry's walk-forward optimization (WFO — a backtesting method that repeatedly trains on a past window and tests on the next), which uses 24 folds as its working standard. His own note records:

"Doubled folds from 12 to 24 ... TAMRS-lite wins 5/6 metrics ... its win is robust across 10, 12, 24 folds on three different data sources."

Matching that K keeps this audit aligned with his cross-audit convention.

The statistics behind K=24

When you average many per-slice numbers into one summary, the uncertainty (standard error) shrinks roughly as 1/√K — i.e. more slices = a steadier average, with diminishing returns. The table shows that K=24 sits right at the "sweet spot" where adding more slices barely helps:

At the active K=21 the multiplier is ~0.69 vs K=24's 0.65 — a negligible precision loss. Going to K=30 buys only a marginal gain; K=50 buys almost nothing.

Does the calendar even fit?

24 windows × 2 months = 48 months of data needed. The available history is ~78 months (2020-04 → 2026-02 ≈ 6.5 years), so the schedule uses 62% of the calendar.

Why not push K higher (30, 50)?

K=24 is the upper bound before you'd have to sacrifice something:

• K=30 → adjacent-slice gap shrinks to ~0.7 months, so slices start sharing regimes (independence erodes).
• K=50 → forces either 1-month windows (too few bars at 750 dbps) or overlapping windows (independence lost).

Why 2-month windows (not 1, not 3+)?

This is a bar-density trade-off: each window must contain enough bars for the statistics to converge, even at the sparsest threshold and in the quietest months. Here's roughly how many bars a window yields for BTC at each threshold:

Now compare against the minimum sample size (N) each measurement tool needs to give a stable answer. These tools measure how strongly two things relate — from simple linear correlation up to multi-variable information measures:

Putting the two together at the sparsest threshold (750 dbps), the window choice falls out cleanly:

Why these 6 symbols?

The six coins were chosen to span market "caps" (sizes) and categories, while staying inside the symbols that have full data coverage:

Why not more symbols (e.g. 16)?

Going from 6 → 16 crypto symbols would add ~900 more database fetches (~15 min), ~3–7 GB more disk on bigblack, and stronger cross-symbol evidence (16 coins agreeing is more credible than 6). That's catalogued as Option C in the expansion options doc. The default is deliberately the minimal scope; expanding is a single pre-spec amendment away.

Why was forex removed?

Version 1 of the pool included EURUSD at 5 dbps as a cross-asset-class anchor. Version 2 removed forex entirely (operator directive, 2026-05-30) — the layer is crypto-only "for now." The forex availability probe and the reintroduction catalogue are preserved in the archived forex-availability-probe.md and scope-expansion-options.md for a later comeback.

Why threshold "tiers" (full vs spearman_only)?

Because bar density falls as the threshold rises, the sparsest threshold (750 dbps) routinely drops below the 1,000-bar floor the demanding metrics need. So thresholds are tiered by which metrics they can support:

So at 750 dbps, the simple Spearman rank correlation (which only needs ~100 points) stays valid, but the heavier conditional metrics (needing ≥1,000) are excluded wherever a cell misses the floor. The alternative — a special 3-month window just for 750 dbps — was rejected because it would break the clean, uniform 2-month / non-overlapping grid. Tiering keeps the grid tidy and is honest about which metrics each cell supports.

The rules that govern window selection

Each of the 24 scheduled windows was chosen to land in a distinct market-regime context. The governing principles:

• No two windows overlap — guarantees per-slice independence.
• Clean ISO boundaries — every window starts on the 1st of a month and ends on the last day of the next month, so it's trivial to verify.
• Uniform 2-month size — fair cross-slice comparison.
• Spread across regime epochs — COVID, 2021 bull, 2022 crash, FTX, 2023 recovery, 2024 ETF era, 2025 bull, 2026 current.
• Earliest slice fits all 6 symbols — same scope works across every default coin.

What we inherit from Terry's WFO conventions — and what we don't

Key numbers — the locked v2 scope

Total cells = 441 (438 populated). There is no Parquet export step — the probe reads each cell read-only from ClickHouse on demand. Expansion paths (including forex reintroduction) stay parked in scope-expansion-options.md; escalating scope is a pre-spec version bump plus a probe-constant update.