Our production system uses Pearson correlation to decide if two market features carry the same information. We tested whether modern alternatives catch redundancy that Pearson misses. The short answer: yes, decisively โ and the gap is bigger than expected. This showcase walks through what we built, what we measured, and every commit, file, and number that backs it up.
What this research is about, in 30 seconds
A trading system uses dozens of "features" โ numbers computed from market data, like trade volume, price-change speed, or order imbalance. If two features carry the same information, keeping both wastes effort and confuses the model. The standard way to test "are these two features redundant?" is a 100-year-old method called Pearson correlation. It only catches relationships that look like a straight line. We tested two modern methods (Chatterjee ฮพโ and Aggregated HSIC) that can catch curved relationships too. We also tested a calibration trick (IAAFT surrogates) to make sure the results aren't artifacts of how financial data is naturally bumpy. We found that the modern methods catch real, substantial redundancy that the standard method completely misses โ across multiple cryptocurrencies and across every pair of features we tested.
We started from a catalog of ~95 candidate measurement instruments and spiked 3 directly, then ran 2 follow-up validations on the laptop. Spike 6 then re-ran all of them on bigblack against a fresh ClickHouse snapshot to confirm cross-platform robustness. Click any card to see the experiment's setup, data, and verdict.
Kernel-based method detects non-linear dependence, but its built-in significance thresholds over-reject on financial data by 2-3ร.
Gate 0.5 PASS ยท Gate 3 PARTIAL
Rank-based, parameter-free, well-calibrated on financial autocorrelated data. Detects what Spearman misses. The favoured candidate.
Gate 0.5 + Gate 3 + multi-symbol + broadening โ all PASS
Phase-randomised surrogate methodology โ not approved. Project policy: real ClickHouse data only.
See prohibition banner โ cascade impact on Spikes 2/4/5/6
Pair-8 finding holds on ETHUSDT and ADAUSDT, not just BTC. Magnitude shrinks with smaller-cap symbols but signal-to-null stays 4-11ร.
3 of 3 symbols IAAFT-REAL
100% of 171 BTC microstructure pairs are Chatterjee BH-FDR-significant. 69% are Pearson-blind. 8-pair finding is the rule, not the exception.
171 / 171 significant
All 5 prior spikes re-run on a different machine + fresh ClickHouse snapshot. 6/7 verdict checkpoints match exactly (one HSICAgg sub-test halted at the 3-min CPU cap). Findings are platform-and-substrate-robust.
6 / 7 checkpoints reproduced
All 6 candidates at a glance with verdicts at each gate. Chatterjee ฮพโ is the only metric that passes every rung; the validations (rows 4โ6) all confirm that finding rather than testing a new metric.
| # | Candidate | Type | Provenance | Gate 0.5 | Gate 3 (IAAFT) | Verdict | Detail |
|---|---|---|---|---|---|---|---|
| 1 | Aggregated HSIC | Kernel-based, p-value | Albert 2022 / Kim 2022 / Gretton 2007 | PASS | PARTIAL 2-3ร Type-I inflation | PARTIAL โ needs threshold adjustment | Open โ |
| 2 | Chatterjee ฮพโ | Rank-based, asymmetric | Chatterjee 2021 JASA | PASS | โ UNDER REVIEW Gate 3 = IAAFT (prohibited) | UNDER REVIEW ยท IAAFT dependency | Open โ |
| 3 | IAAFT surrogates ๐ซ PROHIBITED | Calibration tool | Schreiber & Schmitz 1996 PRL | n/a | BLOCKED phase-randomised surrogates | Methodology not approved | Open โ |
| 4 | Multi-symbol robustness | Validation of Chatterjee | (this audit) | n/a | โ UNDER REVIEW 3/3 verdict used IAAFT | UNDER REVIEW ยท IAAFT dependency | Open โ |
| 5 | Chatterjee broadening | Panel-wide validation | (this audit) | n/a | โ UNDER REVIEW BH-FDR used IAAFT p-values | UNDER REVIEW ยท IAAFT dependency | Open โ |
| 6 | Bigblack replication | Cross-platform replication | (this audit) | n/a | โ UNDER REVIEW Most checkpoints used IAAFT | UNDER REVIEW ยท IAAFT dependency | Open โ |
The original spike's most surprising finding was on pair 8: kyle_lambda_proxy โ buy_volume. The April 2026 3-axis audit called this pair "completely orthogonal" under Pearson. Our three candidate methods say:
| Method | Observed | Null distribution | Ratio / p-value | Verdict |
|---|---|---|---|---|
| Spearman ฯ (production baseline) | +0.049 | โ | p = 0.16 (NOT significant) | "Orthogonal" (Pearson view) |
| Chatterjee ฮพโ | 0.613 | median โ 0, max = 0.052 | 11.8ร null max; empirical p < 1/500 | REAL |
| HSICAgg p_min | 0.0050 | min = 0.0050, median = 0.189 | empirical p = 0.067 (200-perm floor) | PROBABLY REAL |
What "11.8ร null max" means
We took the original time series and generated 500 synthetic versions that have the same bumpy autocorrelation but no real cross-relationship with the partner column. On those 500 synthetic versions, the largest Chatterjee value we got was 0.052. The actual value is 0.613. So the real signal is almost 12 times stronger than what pure autocorrelation noise could produce. Decisively real.
Chatterjee ฮพโ has passed 4 of 6 possible validation rungs: Gate 0.5, Gate 3 (IAAFT), multi-symbol replication, panel broadening. The next move is one of two:
The biggest remaining uncertainty in the catalog is the ~20 candidates with no FOSS implementation at all. Lock the Chatterjee evidence and start spiking those (CODEC/FOCI, Reduced TE, O-information, etc.). ~2-4 hours per candidate.
The "100% of pairs flagged" broadening result could reflect (a) correct detection of pervasive structure, or (b) substrate degeneracy at 100 dbps. Re-run broadening at 250+ dbps + mutation tests + synthetic null sanity check.