finding · findings/evolution/shared_data/S1_GCM_OINFO_PIN_SPEC.md
A detailed blueprint for a self-test that deliberately breaks the analysis code in hundreds of tiny ways, then checks that each break gets caught. If even one break that matters slips through unnoticed, the whole gate is rejected. It exists to guarantee the code's most important yes/no decisions are genuinely protected.
This document is the single source of truth (the one authoritative blueprint) for building two small programs that guard a research script called ivs_loop_s3b.py:
gcm_oinfo_pins.py — the suite of "pins". A pin is one tiny, fixed test that nails down a specific behaviour of the code so it cannot drift. Each pin asserts something like "given these exact inputs, the answer must be exactly this."gcm_oinfo_oracle.py — the mutation oracle: a program that automatically rewrites the research code in thousands of tiny ways (each rewrite = a "mutant"), then runs the pins against every mutant to see whether the deliberate damage gets caught. ("Oracle" here just means the automated judge that decides pass/fail.)The whole thing is called S1 — it's the first hard quality gate in a research pipeline. It copies, line-for-line, a proven earlier gate (the cascade_pins.py / mutation_oracle.py pair, internally "#514"), so the pattern is battle-tested rather than invented from scratch.
The code it guards makes statistical decisions about whether crypto-market features are genuinely related or just coincidentally correlated. The two headline techniques it protects:
Earlier in this research program, a serious bug — a data leakage (where information from the "future" or the test set sneaks into training, producing fake-good results) — was caught only at the very last minute, at pull-request time. A whole walk-forward analysis looked brilliant (a near-perfect 0.926 score, passing 333 of 333 checks) precisely because it was secretly cheating.
The lesson baked into this spec: don't trust a measurement just because it looks good — try to break the thing that produced it first. A pin suite that doesn't actually trip on broken code gives false confidence, which is worse than no test at all. So this spec is unusually paranoid: every claim about what the tests catch was empirically verified on real frozen market data, and several "obvious" tests were thrown out because they would have wrongly flagged correct code or quietly missed real bugs.
The tests don't touch the live database or the network. Everything runs against a frozen fixture — a snapshot of real crypto data saved once to disk so every run is identical and reproducible. The primary snapshot is Bitcoin at a 100-dbps bar threshold (dbps = decimal basis points, the bar-size setting; 100 dbps = 0.1%), holding roughly 8,000 rows and 17 columns of real market features. A second Bitcoin snapshot (500 dbps, ~7,624 rows) is kept as a cross-check.
Before any testing, the oracle checks that the snapshot is rich enough to make the tests meaningful — and if it isn't, it refuses outright rather than quietly passing. "Fail-closed" is the recurring safety principle here: when in doubt, stop and reject, never wave it through. The snapshot must contain specific named columns (e.g. the arithmetic identity volume = buy_volume + sell_volume, plus a cluster of known-unrelated features), at least 5,000 rows, and at least 12 columns.
The core verdict is simple and strict. Over all the meaningful mutants (the genuinely-broken versions, excluding ones that are mathematically identical to the original):
Some code rewrites change nothing observable — e.g. a setting like "max 60 iterations" nudged to 59 — and shouldn't be counted against the pins. A signature function takes a behavioural fingerprint of the code; if a surviving mutant has the same fingerprint as the original, it's declared EQUIVALENT and excluded from the scoring. The spec is careful that this fingerprint is a superset of what the pins look at, so a genuine behaviour change can never disguise itself as "equivalent" and escape.
Before scoring anything real, the oracle runs two sanity checks on itself:
| Constant | Value | Plain meaning |
|---|---|---|
ALPHA | 0.05 | The significance cutoff — a relationship counts as "real" only if its p-value is below this. |
TAU | 0.30 | Correlation gate — feature pairs more correlated than this are too similar to both keep. |
XI_FLAG | 0.80 | A second redundancy gate (the "ξ" / xi sieve) catching duplicates that ordinary correlation misses. |
OMEGA_STRICT | 0.70 | The synergy floor — a triplet counts as synergistic only if its Ω is below −0.70. |
COND_CAP | (cap) | Max number of "control" variables used when testing a relationship; here the conditioning list comes out to 10 items. |
TRAIN / TEST / PURGE | 3000 / 1500 / 200 | The walk-forward windows: 3,000 training rows, 1,500 test rows, and a 200-row purge gap in between to stop the future leaking into the past. (Walk-forward = train on the past, test on the next unseen slice, repeat.) |
| Function | What it decides |
|---|---|
bh_reject | Which p-values pass the multiple-testing correction (Benjamini–Hochberg FDR). The literal yes/no line. |
folds | The walk-forward window geometry — the train/test/purge boundaries that prevent leakage. |
_hac_se | A noise-robust error bar (Newey–West HAC standard error) that feeds the GCM test. |
_resid + gcm_p | Strips out everything explainable, then computes the GCM p-value for "are these still related?" |
copnorm + oinfo_gc | Computes O-information (the Ω synergy/redundancy number). |
select_train | The orthogonality gate — picks feature pairs that are genuinely different from each other. |
cond_idx | Builds the list of control variables for each conditional test. |
gcm_flagset | The final list of relationships that survive multiple-testing correction. |
syn_set | The final list of synergistic triplets (the Ω verdict). |
The verdict is PASS only if all of these are true at once — otherwise REJECT:
| Rule | What it forbids / requires, and why |
|---|---|
| H1 — no self-reference | Every threshold test compares against a hard-coded number (0.30, 0.70, 0.05…), never the code's own variable. Otherwise the test would move along with the bug it's meant to catch and protect nothing. |
| H2 — independent reference | The O-information check is re-derived from scratch using a separate library, never by calling the very function it's testing. |
| H3 — banned clause | A tempting "correlation > 0.99" check is forbidden: the underlying model is intentionally lossy, so correct code only reaches ~0.97–0.99 and the check would wrongly fail honest code. |
| H4 — self-identity anchor | The "definitely related" test case compares a column against an exact copy of itself (guaranteed p≈0). An earlier buy/sell-volume anchor was empirically only p≈0.46 — not significant — so it was replaced. |
| H5 — fail-closed witnesses | If a boundary test case can't be constructed from real data, the build stops and rejects, rather than silently skipping the test. |
| H6 — explicit equivalent set | Only a short, documented list of rewrites may be excused as "equivalent". Everything else that survives counts as a real survivor. |
These came from actually measuring the real data, and each one killed a naïve test idea:
volume, buy_volume, sell_volume) has Ω ≈ −0.212 — it never reaches the −0.70 synergy floor. So the synergy test can't rely on it clearing the bar; instead it pins the relationship ("is membership exactly equal to: Ω ≤ −0.70?"), which catches threshold bugs even when no triplet actually clears the floor.OMEGA_STRICT as a calibration defect to be fixed in the next stage (S2), so the synergy test regains a valid positive control. The threshold is never silently accepted as "equivalent".At the end, the oracle emits a small machine-readable report: which fixture cell was used, how many mutants were generated, how many were equivalent vs killed vs survived, the final kill rate, whether the clean and lethal self-checks passed, the list of any boundary survivors, and the single PASS / REJECT verdict. The rule, stated once: kill rate ≥ 0.80 AND zero decision-boundary survivors AND all preconditions pass.