# Production Readiness Audit: Brute-Force Microstructure Pattern

**Date**: 2026-02-05 (Updated with Final Verification)
**Status**: **CORRECTED - CONDITIONAL GO**
**Auditors**: 6 adversarial sub-agents across 2 audit rounds

---

## Executive Summary

**VERDICT: Pattern is PRODUCTION-READY for use as NN feature (not standalone signal).**

The critical `lagInFrame` bug was fixed (Gen 111). Atomic verification test **PASSED** on 2026-02-05. Source data (`trade_intensity`, `kyle_lambda_proxy`) verified lookahead-free at computation level.

### Audit History

| Round | Date          | Finding                    | Resolution                                                                       |
| ----- | ------------- | -------------------------- | -------------------------------------------------------------------------------- |
| 1     | 2026-02-05 AM | `lagInFrame` lookahead bug | **FIXED** (Gen 111)                                                              |
| 2     | 2026-02-05 PM | Atomic verification        | **PASSED** ✅                                                                    |
| 3     | 2026-02-05 PM | Data deduplication         | **FIXED** ([opendeviationbar-py#77](https://github.com/terrylica/opendeviationbar-py/issues/77)) |

### Final Risk Assessment

| Finding                         | Severity     | Status                                |
| ------------------------------- | ------------ | ------------------------------------- |
| `lagInFrame` bug                | **CRITICAL** | ✅ FIXED (expanding window)           |
| ORDER BY timestamp ties         | HIGH         | ✅ NOT APPLICABLE (0 ties in SOLUSDT) |
| Source data lookahead           | MEDIUM       | ✅ VERIFIED CLEAN (Rust audit)        |
| 111 patterns = multiple testing | HIGH         | ✅ DSR = 1.000 (passes)               |
| 2024-2025 edge not significant  | HIGH         | ⚠️ USE AS FEATURE, not standalone     |

---

## Atomic Verification Test (2026-02-05)

**Test**: Compare manual p95 computation vs window function at bar 49,997.

| Test               | Result  | Meaning                           |
| ------------------ | ------- | --------------------------------- |
| **p95_match_test** | ✅ PASS | `manual_p95 = window_p95` (exact) |
| **exclusion_test** | ✅ PASS | 49,996 bars used for bar 49,997   |
| **p95_difference** | 0.0     | Zero floating-point error         |

**Conclusion**: `ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING` correctly excludes current bar.

---

## Source Data Verification (2026-02-05)

Audited `opendeviationbar-py/crates/opendeviationbar-core/src/intrabar/features.rs`:

| Feature             | Computation                                 | Lookahead Risk |
| ------------------- | ------------------------------------------- | -------------- |
| `trade_intensity`   | trades / duration_sec (within bar)          | ✅ NONE        |
| `kyle_lambda_proxy` | price_return / order_imbalance (within bar) | ✅ NONE        |

**All microstructure features** computed from trades strictly WITHIN each bar. No cross-bar lookback.

---

## Timestamp Tie Analysis (2026-02-05)

| Symbol  | Threshold | Total Bars | Ties          | Risk                                                                   |
| ------- | --------- | ---------- | ------------- | ---------------------------------------------------------------------- |
| SOLUSDT | 1000      | 90,752     | **0**         | ✅ NONE                                                                |
| BTCUSDT | 1000      | 31,403     | **0** (fixed) | ✅ CLEANED ([#77](https://github.com/terrylica/opendeviationbar-py/issues/77)) |
| ETHUSDT | 1000      | 47,980     | 0             | ✅ NONE                                                                |
| BNBUSDT | 1000      | 48,195     | 0             | ✅ NONE                                                                |

---

## Cross-Asset Validation (Post-Deduplication, 2026-02-05)

Pattern: `2 DOWN bars + trade_intensity > p95_expanding + kyle_lambda > 0 → LONG`

| Symbol  | Total Bars | Signals | Hits | Hit Rate   | Edge        | Z-Score | Status |
| ------- | ---------- | ------- | ---- | ---------- | ----------- | ------- | ------ |
| SOLUSDT | 89,752     | 1,017   | 640  | 62.93%     | +12.93%     | 8.25    | ✅ SIG |
| BTCUSDT | 30,403     | 196     | 131  | **66.84%** | **+16.84%** | 4.71    | ✅ SIG |
| ETHUSDT | 46,980     | 713     | 409  | 57.36%     | +7.36%      | 3.93    | ✅ SIG |
| BNBUSDT | 47,195     | 520     | 340  | 65.38%     | +15.38%     | 7.02    | ✅ SIG |

**All 4 assets show statistically significant edge (z > 1.96).** Pattern generalizes across major crypto pairs.

---

## SHORT Position Analysis (2026-02-05)

Pattern: `2 UP bars + trade_intensity > p95_expanding + kyle_lambda < 0 → SHORT`

| Symbol  | Signals | Hits | Hit Rate   | Edge        | Z-Score | Status |
| ------- | ------- | ---- | ---------- | ----------- | ------- | ------ |
| SOLUSDT | 246     | 145  | 58.94%     | +8.94%      | 2.81    | ✅ SIG |
| BTCUSDT | 78      | 50   | 64.10%     | +14.10%     | 2.49    | ✅ SIG |
| ETHUSDT | 199     | 135  | **67.84%** | **+17.84%** | 5.03    | ✅ SIG |
| BNBUSDT | 170     | 100  | 58.82%     | +8.82%      | 2.30    | ✅ SIG |

### LONG vs SHORT Aggregate

| Direction | Total Signals | Hit Rate | Edge    |
| --------- | ------------- | -------- | ------- |
| LONG      | 2,446         | 62.14%   | +12.14% |
| SHORT     | 693           | 62.05%   | +12.05% |

**Key Finding**: LONG and SHORT have **identical edge** (~12%), but LONG has 3.5x more signals. Pattern is symmetric.

---

## Original Findings (Pre-Fix)

The following documents the ORIGINAL bug that was discovered and fixed:

| Finding                              | Severity     | Impact                           |
| ------------------------------------ | ------------ | -------------------------------- |
| `lagInFrame` bug preserves lookahead | **CRITICAL** | +10-15pp inflated edge (FIXED)   |
| 111 patterns = multiple testing      | HIGH         | DSR = 1.000 (passes)             |
| 2024 edge = 2.38%, z=0.58            | **CRITICAL** | Pattern may be decaying          |
| 2021's 82% is regime artifact        | HIGH         | 28% of signals from outlier year |
| Slippage unmodeled                   | MEDIUM       | ~15bps during high-intensity     |

---

## 1. The Critical `lagInFrame` Bug

### What We Thought

```sql
-- "No-lookahead" fix
lagInFrame(ti_p95, 1) OVER w as ti_p95_prior  -- Uses PRIOR year's p95
```

### What Actually Happens

```sql
-- Step 1: Join bar to ITS OWN year's p95
JOIN yearly_percentiles yp ON b.year = yp.year

-- Step 2: Lag by ONE BAR (not one year!)
lagInFrame(ti_p95, 1) OVER w as ti_p95_prior
```

**Result**:

- Bar at 2021-06-15 is joined to 2021's p95 = 500.54 (computed from ALL of 2021)
- `lagInFrame(500.54, 1)` = 500.54 (previous bar's p95, same year)
- **LOOKAHEAD**: We're using December 2021 data to make June 2021 decisions

### Quantified Impact

| Bars Affected          | Percentage   | Bias                |
| ---------------------- | ------------ | ------------------- |
| First bar of each year | 6 bars       | Correct (0.007%)    |
| All other bars         | ~89,994 bars | Lookahead (99.993%) |

**Estimated edge inflation**: +10-15 percentage points

---

## 2. Statistical Issues

### Multiple Testing (111 Patterns)

| Method                | Threshold | Pattern's p-value     | Survives? |
| --------------------- | --------- | --------------------- | --------- |
| Bonferroni            | 0.00045   | 1.78 × 10⁻¹⁹ (biased) | YES       |
| After bias correction | 0.00045   | TBD (likely > 0.01)   | TBD       |

**DSR (Deflated Sharpe Ratio)**:

- Expected max under null (N=111): z ≈ 3.07
- Observed (biased): z = 8.95 → DSR ≈ 0.66
- After correction: z ≈ 1-3 → DSR ≈ 0.0-0.3 (NOT SIGNIFICANT)

### Confidence Intervals

With N=713, p=0.6676:

- Wilson 95% CI: [63.2%, 70.1%]
- **After bias correction** (p ≈ 0.51-0.53): CI includes 50%

### Temporal Autocorrelation

Signals are not independent (market regimes persist):

- Effective N after adjustment: ~200-350 (not 713)
- Adjusted z-score (biased): 4.5-6.0
- Adjusted z-score (corrected): likely 1-2

---

## 3. Pattern Decay (CRITICAL)

### Year-by-Year Performance

| Year | Hit Rate   | Edge        | Z-Score  | Status           |
| ---- | ---------- | ----------- | -------- | ---------------- |
| 2020 | 66.67%     | +16.67%     | 3.06     | Small N (84)     |
| 2021 | **82.09%** | **+32.09%** | **9.10** | **OUTLIER**      |
| 2022 | 66.92%     | +16.92%     | 3.86     | Post-peak        |
| 2023 | 59.79%     | +9.79%      | 1.93     | Degrading        |
| 2024 | **52.38%** | **+2.38%**  | **0.58** | **NEAR RANDOM**  |
| 2025 | 62.75%     | +12.75%     | 1.82     | Partial recovery |

### Key Observations

1. **2024's edge (2.38%) is NOT statistically significant** (z=0.58, p=0.56)
2. **Pattern decay rate**: ~4.3pp per year
3. **Structural break in 2024**: Chow test p < 0.0001
4. **2021 is an outlier**: Unique retail microstructure + lookahead bias amplification

### Extrapolation

At -4.3pp/year decay:

- 2026: Edge ≈ -2% (pattern may INVERT)
- Pattern likely already dead for production use

---

## 4. Economic Reality

### Why the Pattern Might Have Worked (2020-2022)

1. **Retail mania**: SOL attracted meme coin traders
2. **Predictable panic**: "2 DOWN bars + high intensity" = retail selling
3. **Kyle positive**: Smart money accumulating during panic
4. **Low sophistication**: Few algorithms exploiting this pattern

### Why It's Dying (2023-2025)

1. **Market maturation**: More sophisticated participants
2. **Pattern discovery**: Others may have found this
3. **Regime change**: ETF speculation, different microstructure
4. **Retail exodus**: Bear market reduced unsophisticated flow

### Slippage Concerns

Pattern fires during **high trade_intensity** = volatile moments:

- Expected slippage: 10-20 bps per trade
- If true edge is 2-3%, slippage consumes 30-100% of edge
- High-intensity regimes have WORST execution quality

---

## 5. Remediation Plan

### Immediate Actions

1. **Fix the SQL** (Task #191):

   ```sql
   -- CORRECT: Expanding window with strict PRECEDING
   SELECT timestamp_ms,
          quantile(0.95)(trade_intensity) OVER (
              ORDER BY timestamp_ms
              ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
          ) as ti_p95_expanding
   ```

2. **Re-run with corrected percentiles** (Task #192)

3. **Compute DSR** (Task #193)

4. **Make GO/NO-GO decision** (Task #194)

### Decision Framework

| True Edge (corrected) | DSR   | Decision                           |
| --------------------- | ----- | ---------------------------------- |
| > 5%                  | > 0.5 | GO - proceed to exp082             |
| 2-5%                  | > 0.5 | MARGINAL - paper trade first       |
| < 2%                  | any   | **NO-GO** - edge consumed by costs |
| any                   | < 0.5 | **NO-GO** - not significant        |

---

## 7. ACTUAL RESULTS: True No-Lookahead (Gen 111)

### Executed 2026-02-05

The fix was implemented using `quantileExactExclusive(0.95)` with `ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING`.

### Comparison: Biased vs True No-Lookahead

| Pattern                     | Gen 108 (biased) | Gen 111 (TRUE NLA)   | Δ Edge      | Δ Signals |
| --------------------------- | ---------------- | -------------------- | ----------- | --------- |
| **2DOWN + ti>p95 + kyle>0** | 66.76% (+16.76%) | **62.93% (+12.93%)** | **-3.83pp** | +304      |
| 2DOWN + kyle>0              | 51.76% (+1.76%)  | 51.76% (+1.76%)      | 0           | 0         |
| Pure 2DOWN                  | 51.58% (+1.58%)  | 51.58% (+1.58%)      | 0           | 0         |

### Key Findings

1. **Edge reduction was 3.83pp**, NOT the 10-15pp predicted by the audit
2. **Pattern is STILL statistically significant**: z=8.25 (vs biased z=8.95)
3. **More signals**: 1,017 vs 713 (expanding p95 is lower early in dataset)
4. **Simple patterns unchanged**: Confirms the fix only affects percentile-based patterns

### Revised Assessment

| Metric       | Biased  | Corrected | Change  |
| ------------ | ------- | --------- | ------- |
| Hit Rate     | 66.76%  | 62.93%    | -3.83pp |
| Edge         | +16.76% | +12.93%   | -3.83pp |
| Z-score      | 8.95    | 8.25      | -0.70   |
| Signal Count | 713     | 1,017     | +43%    |

**The corrected +12.93% edge is ABOVE the 5% threshold for GO decision.**

### DSR Analysis (Task #193 COMPLETE)

| Pattern                  | Observed Z | Expected Max (N=111) | DSR       | Significant? |
| ------------------------ | ---------- | -------------------- | --------- | ------------ |
| Full (2DOWN+ti>p95+kyle) | 8.25       | 3.07                 | **1.000** | **YES**      |
| Simple (2DOWN+kyle>0)    | 4.75       | 3.07                 | **0.985** | **YES**      |

**Both patterns pass DSR significance test after correction.**

### Temporal Stability (Gen 112 vs Gen 109)

| Year | Biased Edge | TRUE NLA Edge | Δ           | Comment                       |
| ---- | ----------- | ------------- | ----------- | ----------------------------- |
| 2020 | +16.67%     | **+18.97%**   | +2.3pp      | Slightly BETTER               |
| 2021 | +32.09%     | **+15.97%**   | **-16.1pp** | **MASSIVE inflation removed** |
| 2022 | +16.92%     | **+35.00%**   | +18.1pp     | More signals, higher edge     |
| 2023 | +9.79%      | **+12.03%**   | +2.2pp      | Slightly BETTER               |
| 2024 | +2.38%      | **+3.79%**    | +1.4pp      | Still weak (z=0.87)           |
| 2025 | +12.75%     | **+4.25%**    | **-8.5pp**  | Significant drop              |

**Critical Findings**:

1. **2021 was massively inflated**: 32.09% → 15.97% (lookahead removed 16pp!)
2. **2022 shows STRONGER edge**: 16.92% → 35.00% (more consistent data)
3. **2024-2025 are WEAK**: Both below 5% edge, z < 1.5
4. **Pattern decay confirmed**: 2020-2023 had ~15-35% edge, 2024-2025 have ~4%

### CRITICAL: 2024-2025 Performance

| Year | Hit Rate | Edge   | Z-Score | Significant?      |
| ---- | -------- | ------ | ------- | ----------------- |
| 2024 | 53.79%   | +3.79% | 0.87    | **NO** (z < 1.96) |
| 2025 | 54.25%   | +4.25% | 1.24    | **NO** (z < 1.96) |

**Neither 2024 nor 2025 shows statistically significant edge after lookahead correction.**

---

## 8. FINAL PRODUCTION READINESS DECISION

### Decision Matrix

| Criterion    | Threshold           | Actual | Pass?            |
| ------------ | ------------------- | ------ | ---------------- |
| Overall Edge | > 5%                | 12.93% | **YES**          |
| DSR          | > 0.5               | 1.000  | **YES**          |
| Z-score      | > 3.07 (Bonferroni) | 8.25   | **YES**          |
| 2024 Edge    | > 2%                | 3.79%  | **YES** (barely) |
| 2025 Edge    | > 2%                | 4.25%  | **YES** (barely) |
| 2024 Z-score | > 1.96              | 0.87   | **NO**           |
| 2025 Z-score | > 1.96              | 1.24   | **NO**           |

### VERDICT: **CONDITIONAL GO**

The pattern passes aggregate statistical tests (DSR, Bonferroni) but **FAILS recent-period significance tests**.

**Recommendation**:

1. **DO NOT deploy full pattern as standalone trading signal** (2024-2025 edge not significant)
2. **USE pattern as FEATURE for neural network** (exp082)
3. **Fallback to simple pattern** (2DOWN+kyle>0) which has:
   - 1.76% edge (small but consistent)
   - 18,176 signals (high sample size)
   - Stable across years (no percentile = no lookahead risk)

### Why Conditional Go for exp082

The NN can potentially:

1. Learn when the pattern works vs doesn't (regime detection)
2. Combine with other features for better selectivity
3. Use long-only gate to avoid losing shorts
4. Achieve 5%+ edge on subset of signals

### Monitoring Requirements

If proceeding with exp082:

- **Weekly edge monitoring**: Stop if 20-signal rolling hit rate < 52%
- **Monthly review**: Compare to simple pattern baseline
- **6-month expiration**: Re-evaluate pattern viability

---

## 9. UPDATED RECOMMENDATIONS FOR exp082

### Feature Set (Revised)

Given the temporal analysis, recommend using **TWO feature sets**:

**A. Champion Features** (for high-edge, low-frequency):

- direction_lag1, direction_lag2
- trade_intensity_z (z-scored, NO percentile threshold)
- kyle_lambda_z
- down_streak

**B. Simple Features** (for lower-edge, high-frequency):

- direction_lag1, direction_lag2
- kyle_positive (binary: kyle > 0)
- No trade_intensity (avoids percentile issues)

### Architecture Change

Instead of using `ti > p95_expanding` as a hard gate, let the NN **learn the threshold**:

- Input: `trade_intensity_z` (continuous)
- Let the network decide when intensity is "high enough"
- Avoids hand-crafted thresholds that may decay

### Expected Performance

| Approach               | Expected Edge | Frequency           | Risk          |
| ---------------------- | ------------- | ------------------- | ------------- |
| Fixed rule (2024-2025) | 3-4%          | ~100 signals/year   | High variance |
| NN (champion features) | 5-10%         | 50-200 signals/year | Model decay   |
| NN (simple features)   | 2-4%          | 1000+ signals/year  | Lower edge    |

### If NO-GO

Pivot options:

1. **Simple pattern only**: 2DOWN + kyle>0 has 1.76% edge with 18K signals (no percentile threshold = no lookahead issue)
2. **Different asset**: BNB showed 71.72% (but same lookahead bug)
3. **Different approach**: Use pattern as FEATURE for NN, not as standalone signal
4. **Abandon**: Accept that brute-force microstructure patterns are not production-viable

---

## 6. Lessons Learned

### Lookahead Bias is Insidious

The `lagInFrame(ti_p95, 1)` approach LOOKED correct but wasn't. Key insight: **lagging a value computed with lookahead doesn't remove the lookahead**.

### Correct Pattern for No-Lookahead Percentiles

```sql
-- WRONG: Lag a lookahead value
JOIN yearly_percentiles yp ON b.year = yp.year  -- Year Y's p95 uses all of Y
lagInFrame(ti_p95, 1) OVER w  -- Still uses Y's full p95

-- CORRECT: Expanding window with strict PRECEDING
quantile(0.95)(trade_intensity) OVER (
    ORDER BY timestamp_ms
    ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
)  -- Only uses data before current bar
```

### Multiple Testing Must Be Addressed

111 patterns is NOT 111 independent tests (they evolved from each other). True effective comparisons may be 200-500. Always compute DSR.

### Temporal Decay Must Be Monitored

Any pattern discovered on historical data is subject to:

1. **Alpha decay**: Market adaptation
2. **Regime change**: Structural shifts in microstructure
3. **Data snooping**: Fitting to historical quirks

### Year 2024 is the Canary

2024 showing z=0.58 (not significant) should have been a RED FLAG immediately. Always check the most recent period separately.

---

## Appendix: Corrected SQL Template

```sql
-- TRUE NO-LOOKAHEAD: Expanding window percentiles
-- Gen 111: Corrected champion pattern

INSERT INTO opendeviationbar_cache.feature_combinations
    (symbol, threshold_decimal_bps, combo_name, combo_description, n_features,
     feature_conditions, signal_type, lookback_bars,
     total_bars, signal_count, hits, hit_rate, edge_pct, z_score, p_value, ci_low, ci_high,
     generation)
WITH
-- Step 1: Compute running percentile using ONLY prior data
bars_with_running_pct AS (
    SELECT
        timestamp_ms,
        CASE WHEN close > open THEN 1 ELSE 0 END as direction,
        trade_intensity as ti,
        kyle_lambda_proxy as kyle,
        -- TRUE expanding window: strictly prior bars only
        quantileExactExclusive(0.95)(trade_intensity) OVER (
            ORDER BY timestamp_ms
            ROWS BETWEEN UNBOUNDED PRECEDING AND 1 PRECEDING
        ) as ti_p95_prior
    FROM opendeviationbar_cache.open_deviation_bars
    WHERE symbol = 'SOLUSDT' AND threshold_decimal_bps = 1000
),
-- Step 2: Lag direction and features
lagged AS (
    SELECT
        direction,
        lagInFrame(ti, 1) OVER w as ti_1,
        lagInFrame(kyle, 1) OVER w as kyle_1,
        lagInFrame(direction, 1) OVER w as dir_1,
        lagInFrame(direction, 2) OVER w as dir_2,
        ti_p95_prior  -- Already strictly prior
    FROM bars_with_running_pct
    WINDOW w AS (ORDER BY timestamp_ms)
)
SELECT
    'SOLUSDT', 1000,
    'true_nla_combo_2down_ti_p95_kyle_gt_0_long',
    'TRUE NO-LOOKAHEAD: 2DOWN + ti>p95_expanding + kyle>0 → LONG',
    4,
    '{"direction(t-2,t-1)": "DOWN,DOWN", "trade_intensity(t-1)": ">p95_expanding", "kyle_lambda(t-1)": ">0"}',
    'long', 2, count(*),
    countIf(dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0),
    countIf(direction = 1 AND dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0),
    countIf(direction = 1 AND dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0) /
        nullIf(countIf(dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0), 0),
    countIf(direction = 1 AND dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0) /
        nullIf(countIf(dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0), 0) - 0.5,
    (countIf(direction = 1 AND dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0) /
        nullIf(countIf(dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0), 0) - 0.5) /
        sqrt(0.25 / countIf(dir_2 = 0 AND dir_1 = 0 AND ti_1 > ti_p95_prior AND kyle_1 > 0)),
    0.5, 0.5, 0.5,
    111  -- Gen 111 = TRUE no-lookahead
FROM lagged
WHERE dir_2 IS NOT NULL
  AND ti_p95_prior IS NOT NULL
  AND ti_p95_prior > 0;  -- Skip warmup period where no prior data exists
```

---

## References

- Bailey et al. (2014) "The Deflated Sharpe Ratio" - Multiple testing correction
- Maddison et al. (2017) "The Concrete Distribution" - Binary Concrete gates
- arXiv:2512.12924v1 - Walk-Forward validation framework
