# Validation Framework

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Tiered validation framework for bar integrity, Stathera trade-ID continuity, gap classification, and schema contracts. Provides 4 validation tiers (Tier 0 through Tier 3) plus continuity checking, each targeting different speed/depth tradeoffs.

---

## Files

| File                    | Purpose                                                                                           |
| ----------------------- | ------------------------------------------------------------------------------------------------- |
| `__init__.py`           | Package exports: re-exports all public types and functions from submodules                        |
| `tier1.py`              | Tier 1 auto-validation (<30 sec): bounds checks, NaN/Inf detection, OFI correlation, basic smoke  |
| `tier2.py`              | Tier 2 statistical validation (~10 min): ADF stationarity, Spearman predictive power, mutual info |
| `continuity.py`         | Stathera trade-ID continuity validation with tiered gap classification integration                |
| `gap_classification.py` | Gap severity taxonomy: PRECISION, NOISE, MARKET_MOVE, MICROSTRUCTURE, SESSION_BOUNDARY            |
| `cache_staleness.py`    | Tier 0 staleness detection (<5ms): detects cached data computed with older schema versions        |
| `post_storage.py`       | Tier 0 post-storage validation (<1 sec): verifies OHLC invariants and data integrity after writes |
| `schemas.py`            | Pandera `BarOutputSchema` contract: column presence, types, timestamp scale (ms not us) bounds    |

## Validation Tiers

| Tier | Module            | Runtime | Trigger                                  | Checks                                                                   |
| ---- | ----------------- | ------- | ---------------------------------------- | ------------------------------------------------------------------------ |
| 0    | `cache_staleness` | <5 ms   | Every cache read (microstructure)        | Schema version, missing columns, stale data detection                    |
| 0    | `post_storage`    | <1 sec  | Every cache write                        | OHLC invariants, row count, checksum, data integrity                     |
| 1    | `tier1`           | <30 sec | Every `precompute_open_deviation_bars()` | Feature bounds (duration>=0, OFI in [-1,1]), NaN, correlations           |
| 2    | `tier2`           | ~10 min | Before production ML training            | ADF stationarity, Spearman with forward returns, mutual info, redundancy |

## Stathera Continuity

`continuity.py` implements the trade-ID alignment invariant (`bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`). Key functions:

- `validate_continuity_tiered()` -- full tiered validation with gap classification from `gap_classification.py`
- `validate_junction_continuity()` -- checks continuity between two DataFrames (e.g., cache chunk boundaries)
- `validate_continuity()` -- simple legacy validation (threshold-based)

Returns `TieredValidationResult` with per-tier gap summaries (`TierSummary`) and classified `GapInfo` records.

## Gap Classification

`gap_classification.py` provides a 5-tier severity taxonomy based on empirical analysis of 30-month BTC data (Issue #19), identifying 49 legitimate market microstructure events:

| Tier               | Range            | Meaning                                     |
| ------------------ | ---------------- | ------------------------------------------- |
| `PRECISION`        | < 0.001%         | Floating-point artifacts (always ignored)   |
| `NOISE`            | 0.001% - 0.01%   | Tick-level noise (logged, not flagged)      |
| `MARKET_MOVE`      | 0.01% - 0.1%     | Normal market movement (configurable)       |
| `MICROSTRUCTURE`   | > 0.1%           | Flash crashes, liquidations (warning/error) |
| `SESSION_BOUNDARY` | > threshold \* 2 | Definite session break (always error)       |

Asset-class-aware multipliers (`AssetClass.CRYPTO`, `AssetClass.FOREX`) adjust thresholds per instrument type.

## Schema Contract

`schemas.py` defines `BarOutputSchema` (Pandera) -- the data contract between Rust processor Arrow output and ClickHouse storage. Catches the three v12.50.0 hotfix cascade bugs: missing `_ms` suffix, microsecond vs millisecond scale, missing required columns.

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) -- Parent: Python API layer
- [/CLAUDE.md](/CLAUDE.md) -- Hub: project overview and architecture
- [/crates/CLAUDE.md](/crates/CLAUDE.md) -- Rust processor that produces the bars validated here
