finding · findings/evolution/shared_data/purpose.md
One locked grid of real market data, defined a single time and reused by every research investigation that needs it — instead of each investigation re-defining and re-fetching its own copy.
The shared data pool is a central, version-locked definition — a fixed, frozen description that doesn't drift over time — of real slices of market data. "Real" matters: every slice comes live from ClickHouse (the high-speed database that stores the project's processed market bars), and the access is strictly read-only (the research can look but never change the data).
The core idea in one line: define the grid once, let many studies read from it many times. The data needed is the same shape for everyone — lots of features × lots of bars × lots of calendar windows × lots of price-move thresholds × lots of symbols. So instead of each study carving out its own windows, the shared layer locks that grid a single time (in a file called pre-spec.md), and a small program, orthogonality_probe.py, reads each cell of the grid live from the database whenever a study asks.
A big part of this finding is drawing a hard boundary: the pool holds the ingredients, never the conclusions. It is "pre-verdict" — it sits before any study reaches a judgement. The table below lists what deliberately does not belong here.
| Does NOT belong here | Why |
|---|---|
| Per-study verdicts (the final answers) | Each study's conclusions live inside that study's own folder, not in the shared pantry. |
| Each study's own detailed plan | A study's plan sits in its own folder. The pool's plan only defines which data is in scope — nothing about the study's logic. |
| The code that computes the candidate measures | That per-measure Python lives with each study, not in shared data. |
| Synthetic (made-up) data | Strict no-synthetic-data policy: only genuine ClickHouse data, ever. No simulated or fabricated number-series of any kind. |
| Write-ups of how a measure or feature behaves | That's study work. The pool only documents the raw material (the "substrate"). |
In short — the pool owns: the slice definitions, the read-only probe that fetches them, where the data came from (provenance), and notes on data quality. It owns none of the study-side number-crunching, verdict text, or measure implementations. If a file would belong in a study's verdict trail, it lives in that study's folder, not here.
An "audit" here just means a careful, documented investigation. Two are currently active, and both ask their question about orthogonality — a fancy word for "genuinely different." Two market indicators are orthogonal when one tells you something the other doesn't; if they always move together, the second adds no new information.
audits/2026-05-16-orthogonality-measurement-instruments/
Asks: "Does this candidate yardstick actually measure orthogonality correctly, and does its answer stay consistent across market conditions, symbols, and thresholds?" The candidate yardsticks have technical names — Chatterjee ξₙ, distance correlation (dCor), HSIC, CODEC, O-information, Marchenko–Pastur — but all of them are just different statistical rulers for "how related are these two things?"
It runs the same ruler on every cell of the shared grid, collects the answers into a distribution, then applies standard rules (the middle value, the spread, how often it clears a bar, and how stable it is across repeats) to reach a verdict: adopt, defer, or reject that ruler. Forex (cross-asset-class) is out of scope for now.
audits/2026-05-08-orthogonal-features-btcusd/
Asks: "Is this candidate feature actually different from the features we already use in production?" Candidate features again carry technical names (Persistence Landscape L²-norm, MFDFA-Δα, reduced transfer-entropy as a feature) — but the question is simple: does it add new information, or just echo what we already have?
It uses the shared grid to score each candidate feature against the existing production line-up, using whichever measuring ruler Audit 1 has blessed. Scoring it separately per time-slice, per symbol, and per threshold provides stability evidence — proof the answer isn't a fluke of one window.
They share the data: the same crypto slice grid, the same calendar windows, the same cleaned-up matrices (with missing values — "NaN" — already removed). Both read each cell live from the database via the same read-only probe.
They keep separate:
This is the heart of the finding. Letting every study fetch its own data seems simpler, but it quietly breaks reproducibility and wastes resources. The shared layer fixes six concrete problems at once:
| Concern | If each study fetched its own | With one shared layer |
|---|---|---|
| Reproducibility can you get the same result later? | Each study's snapshot drifts over time | One frozen snapshot both studies point at |
| Comparability do the studies mean the same thing? | "Slice 3" could mean different date windows in different studies | One fixed definition of every slice |
| Database cost | N studies × M experiments × repeated re-fetches | Fetch once, amortized across all studies and experiments forever |
| Disk usage | Multiple copies of the identical bars | One single source of truth |
| Data-quality digging | Must reproduce per study to investigate | One manifest to inspect |
| Version control | Each study re-locks its own scope | One plan governs every consumer |
The substrate IS the foundation. Audits build on top of the substrate's decisions.
"Substrate" is just the underlying raw material the studies stand on. Putting it in its own folder makes the boundary explicit, and it makes each study more honest about what it is — and isn't — testing. Each study tests the orthogonality question on this data. None of them claims to test whether the data itself is representative of all markets — that's a separate, bigger ("meta") question the pool deliberately leaves open.