# Phase 18: Monitor + Validation Gates - Research

**Researched:** 2026-03-27
**Domain:** Heartbeat monitoring, pre-write validation, Stathera auditing -- Aion mode compatibility
**Confidence:** HIGH

## Summary

Phase 18 makes three monitoring/validation subsystems Aion-aware: (1) the heartbeat's yesterday-completion check, (2) the `bulk_operations.py` pre-write day-mode gate, and (3) the Stathera audit queries. All three changes are purely conditional branches keyed on per-symbol `ouroboros_mode` from the registry -- no new dependencies, no schema changes.

The key insight is that `ouroboros_mode` is already available per-symbol via `SymbolEntry.ouroboros_mode` (Phase 13, REG-01/REG-02) and a utility `_get_mode_for_symbol()` already exists in `kintsugi/_helpers.py`. The Stathera audit is already mode-agnostic because it partitions by `ouroboros_mode` in its WINDOW clause. The pre-write gate and yesterday check need conditional logic based on per-symbol mode.

**Primary recommendation:** Add per-symbol mode branching to the yesterday check and pre-write gate; verify Stathera is already mode-agnostic; add Aion freshness check as the Aion equivalent of yesterday-completion.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all at Claude's discretion (infrastructure phase).

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

### Deferred Ideas (OUT OF SCOPE)

None -- infrastructure phase stays within phase scope.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID     | Description                                                                                                                    | Research Support                                                                                                                                                                                                                                                      |
| ------ | ------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| MON-01 | Yesterday completion check replaced with rolling-window freshness check for Aion symbols (last bar within configurable window) | `yesterday.py` currently checks `crossover_tid - 1` alignment for all symbols -- Aion symbols have no orphan bar at midnight and no daily boundary, so this check is meaningless for them. Replace with a rolling-window freshness check (last bar within N minutes). |
| MON-02 | Day-mode pre-write gate in `bulk_operations.py` is conditional -- skipped for Aion symbols                                     | Both `store_bars_batch()` (pandas path, line 625) and `store_bars_bulk()` (Polars path, line 955) currently call `get_operational_ouroboros_mode()` for a global mode check. This must become per-symbol using the symbol from the DataFrame.                         |
| MON-03 | Stathera audit query is mode-agnostic (trade-ID continuity checking works identically for both modes)                          | Already implemented -- `cross_pair_window()` in `stathera_query.py` partitions by `(symbol, threshold_decimal_bps, ouroboros_mode)`. Requires verification that no caller makes mode-specific assumptions.                                                            |

</phase_requirements>

## Architecture Patterns

### Current Code Topology (What Changes)

**MON-01: `scripts/heartbeat/checks/yesterday.py`**

- `collect_yesterday_completion()` iterates `get_heartbeat_symbols()` (all enabled Binance spot symbols)
- For each symbol: gets `crossover_tid`, queries yesterday's max `last_agg_trade_id`, compares against `crossover_tid - 1`
- Aggregates into `results["yesterday_completion"]` (boolean)
- Health tier: DEGRADED (from `config.py` line 101)

The Aion change: Aion symbols have no orphan bar and no midnight reset. The crossover TID concept is meaningless -- bars span midnight freely. For Aion symbols, the appropriate check is "freshness" -- has the symbol produced a bar within the last N minutes?

**MON-02: `python/opendeviationbar/clickhouse/bulk_operations.py`**

- Two write paths: `store_bars_batch()` (pandas, line 617-639) and `store_bars_bulk()` (Polars, line 947-972)
- Both call `get_operational_ouroboros_mode()` which returns a single global mode
- The check: `if _get_mode() == "day"` then filter cross-midnight bars
- Problem: Global mode check. When both Aion and Day symbols coexist, the global mode is ambiguous

The fix: Extract `symbol` from the DataFrame, look up per-symbol mode from registry, apply day-mode gate only for day-mode symbols. The DataFrame always has a `symbol` column.

**MON-03: `python/opendeviationbar/clickhouse/stathera_query.py`**

- `cross_pair_window()` already partitions by `(symbol, threshold_decimal_bps, ouroboros_mode)`
- The `lagInFrame()` window computes `tid_delta` independently within each partition
- **Already mode-agnostic by design** -- Aion and Day bars in the same table will have separate windows and never cross-contaminate

Additional Stathera audit sites to verify:

1. `stathera.py` lines 50-68: main per-symbol query uses `cross_pair_window()` -- OK
2. `stathera.py` lines 145-158: today-only query uses inline `PARTITION BY ... ouroboros_mode` -- OK
3. `stathera.py` lines 273-290: cross-midnight oracle checks ALL bars -- **needs Aion exclusion** (Aion bars intentionally cross midnight)
4. `stathera.py` lines 344-362: kintsugi queue uses `ouroboros_mode` partition -- OK
5. `stathera.py` lines 416-436: today-gap check uses `ouroboros_mode` partition -- OK

### Pattern: Per-Symbol Mode Lookup

Already established in `kintsugi/_helpers.py:_get_mode_for_symbol()`:

```python
def _get_mode_for_symbol(symbol: str) -> str:
    from opendeviationbar.symbol_registry import get_symbol_entries
    entries = get_symbol_entries()
    entry = entries.get(symbol)
    if entry is None:
        return "day"  # safe default
    return entry.ouroboros_mode.value
```

This pattern should be reused (or a shared utility extracted) rather than re-implemented in each module.

### Pattern: Rolling-Window Freshness for Aion

The freshness data is already collected by `freshness.py:collect_symbol_freshness()` which queries `max(close_time_us)` per symbol and computes `stale_min`. For MON-01, the yesterday check should:

1. For day-mode symbols: keep existing `crossover_tid - 1` alignment check
2. For aion-mode symbols: check if `stale_min < AION_FRESHNESS_WINDOW_MIN` (configurable, suggest 30 min default for live streaming symbols)

### Recommended Project Structure Changes

```
scripts/heartbeat/checks/
    yesterday.py        # MODIFY: branch on ouroboros_mode per symbol
python/opendeviationbar/clickhouse/
    bulk_operations.py  # MODIFY: per-symbol day-mode gate instead of global
scripts/heartbeat/checks/
    stathera.py         # MODIFY: exclude Aion bars from cross-midnight oracle
scripts/heartbeat/
    config.py           # MODIFY: add AION_FRESHNESS_WINDOW_MIN constant
```

## Don't Hand-Roll

| Problem                    | Don't Build         | Use Instead                                                                       | Why                                               |
| -------------------------- | ------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------- |
| Per-symbol mode resolution | New lookup function | `_get_mode_for_symbol()` from `kintsugi/_helpers.py` or extract to shared utility | Already exists, handles unknown symbols safely    |
| Symbol freshness data      | New CH query        | `freshness.py` already collects `stale_min` per symbol                            | Data is already queried; reuse or cross-reference |

## Common Pitfalls

### Pitfall 1: Global Mode Check in Pre-Write Gate

**What goes wrong:** `get_operational_ouroboros_mode()` returns a single global mode from Settings. When the global mode is "day" (default), Aion symbols still get their cross-midnight bars filtered out.
**Why it happens:** The pre-write gate was written before per-symbol modes existed.
**How to avoid:** Extract symbol from DataFrame, look up per-symbol mode from registry.
**Warning signs:** Aion bars being silently filtered with "Day-mode pre-write gate: filtered N cross-midnight bars" warnings.

### Pitfall 2: Cross-Midnight Oracle False Positives for Aion

**What goes wrong:** The cross-midnight oracle in `stathera.py` line 273-290 counts ALL bars where `toDate(open_time_us) != toDate(close_time_us)`. Aion bars deliberately cross midnight. This will report false violations.
**Why it happens:** The oracle was added for Day-mode integrity (#264) before Aion existed.
**How to avoid:** Add `AND ouroboros_mode = 'day'` to the cross-midnight query.
**Warning signs:** `cross_midnight` WARNING key firing in heartbeat, persistent orange alerts.

### Pitfall 3: Yesterday Check for Aion Produces "no_orphan" Noise

**What goes wrong:** `collect_yesterday_completion()` queries yesterday's max `last_agg_trade_id` and compares to `crossover_tid - 1`. Aion symbols don't have orphan bars at midnight; the query may return bars that happen to be near midnight but aren't boundary-aligned.
**Why it happens:** The concept of "yesterday completion" is day-mode specific.
**How to avoid:** Skip the `crossover_tid` check entirely for Aion symbols; use freshness instead.
**Warning signs:** `yesterday_{symbol}` = `"no_orphan"` or `False` for Aion crypto symbols.

### Pitfall 4: DataFrame May Contain Mixed Modes

**What goes wrong:** A single `store_bars_batch()` call could theoretically contain bars from both Day and Aion symbols. Applying a single mode check to the whole batch would be wrong.
**Why it happens:** Batch write paths aggregate bars across symbols.
**How to avoid:** Filter cross-midnight bars only for rows where the symbol's mode is "day". Build a mode lookup dict from the symbol column's unique values.
**Warning signs:** Aion bars being filtered or Day-mode cross-midnight bars leaking through.

## Code Examples

### MON-02: Per-Symbol Pre-Write Gate (pandas path)

```python
# Current (global mode check):
if _get_mode() == "day" and "open_time_us" in df.columns and "close_time_us" in df.columns:
    mask_cross = mask_valid & ((df["open_time_us"] // DAY_US) != (df["close_time_us"] // DAY_US))

# Proposed (per-symbol mode check):
if "open_time_us" in df.columns and "close_time_us" in df.columns and "symbol" in df.columns:
    from opendeviationbar.symbol_registry import get_symbol_entries
    entries = get_symbol_entries()
    # Build day-mode symbol set
    day_symbols = {
        s for s, e in entries.items()
        if e.ouroboros_mode.value == "day"
    }
    mask_day_mode = df["symbol"].isin(day_symbols)
    mask_valid = (df["open_time_us"] > 0) & (df["close_time_us"] > 0)
    mask_cross = mask_valid & mask_day_mode & (
        (df["open_time_us"] // DAY_US) != (df["close_time_us"] // DAY_US)
    )
```

### MON-01: Branching Yesterday Check by Mode

```python
# In collect_yesterday_completion():
from opendeviationbar.symbol_registry import get_symbol_entries
entries = get_symbol_entries()

for symbol in get_heartbeat_symbols():
    entry = entries.get(symbol)
    mode = entry.ouroboros_mode.value if entry else "day"

    if mode == "aion":
        # Rolling-window freshness: check last bar is recent
        _check_aion_freshness(client, symbol, results)
    else:
        # Existing day-mode orphan completion check
        _check_day_completion(client, symbol, results)
```

### MON-03: Cross-Midnight Oracle Aion Exclusion

```python
# Current:
cross_midnight = client.query(
    f"SELECT count() AS violations FROM {FULL_TABLE} FINAL "
    f"WHERE toDate(open_time_us / 1000000, 'UTC') != toDate(close_time_us / 1000000, 'UTC')"
)

# Proposed:
cross_midnight = client.query(
    f"SELECT count() AS violations FROM {FULL_TABLE} FINAL "
    f"WHERE toDate(open_time_us / 1000000, 'UTC') != toDate(close_time_us / 1000000, 'UTC') "
    f"AND ouroboros_mode = 'day'"
)
```

## State of the Art

| Old Approach                                      | Current Approach                              | When Changed      | Impact                                         |
| ------------------------------------------------- | --------------------------------------------- | ----------------- | ---------------------------------------------- |
| Global `OPENDEVIATIONBAR_OUROBOROS_MODE` env var  | Per-symbol `ouroboros_mode` in `symbols.toml` | Phase 13 (REG-01) | Mode resolution must be per-symbol, not global |
| Single mode in `get_operational_ouroboros_mode()` | Still global -- needs per-symbol branching    | This phase        | Pre-write gate and heartbeat need updating     |

## Open Questions

1. **Aion freshness window default**
   - What we know: Sidecar streams continuously; bars should appear within minutes for active symbols
   - What's unclear: Exact threshold for "stale" -- 15 min? 30 min? Depends on threshold (higher thresholds = fewer bars)
   - Recommendation: Default 30 min (configurable via `OPENDEVIATIONBAR_AION_FRESHNESS_WINDOW_MIN`). This is generous enough for high thresholds (1000 dbps) while still catching real outages.

2. **Health tier for Aion freshness**
   - What we know: `yesterday_completion` is DEGRADED tier. Aion freshness is arguably more important since it replaces the yesterday check.
   - Recommendation: Keep at DEGRADED tier for consistency. If no Aion bar in the freshness window, it's informational -- the sidecar check (CRITICAL) already catches if streaming is down.

## Validation Architecture

### Test Framework

| Property           | Value                                                                                                        |
| ------------------ | ------------------------------------------------------------------------------------------------------------ |
| Framework          | pytest                                                                                                       |
| Config file        | `pyproject.toml` ([tool.pytest.ini_options])                                                                 |
| Quick run command  | `uv run --python 3.13 pytest tests/test_heartbeat_stathera.py tests/test_heartbeat_yesterday_junction.py -x` |
| Full suite command | `mise run test-py`                                                                                           |

### Phase Requirements -> Test Map

| Req ID | Behavior                                                           | Test Type | Automated Command                                                           | File Exists?                 |
| ------ | ------------------------------------------------------------------ | --------- | --------------------------------------------------------------------------- | ---------------------------- |
| MON-01 | Aion symbols get freshness check, Day symbols get orphan check     | unit      | `uv run --python 3.13 pytest tests/test_heartbeat_yesterday_junction.py -x` | Exists, needs new test cases |
| MON-02 | Pre-write gate skips cross-midnight for Aion, enforces for Day     | unit      | `uv run --python 3.13 pytest tests/test_mon02_prewrite_gate.py -x`          | Wave 0                       |
| MON-03 | Stathera audit mode-agnostic + cross-midnight oracle excludes Aion | unit      | `uv run --python 3.13 pytest tests/test_heartbeat_stathera.py -x`           | Exists, needs new test cases |

### Sampling Rate

- **Per task commit:** `uv run --python 3.13 pytest tests/test_heartbeat_stathera.py tests/test_heartbeat_yesterday_junction.py tests/test_mon02_prewrite_gate.py -x`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_mon02_prewrite_gate.py` -- covers MON-02 (pandas + Polars paths)
- [ ] New test cases in `tests/test_heartbeat_yesterday_junction.py` for Aion freshness branch
- [ ] New test cases in `tests/test_heartbeat_stathera.py` for cross-midnight oracle Aion exclusion

## Sources

### Primary (HIGH confidence)

- Codebase inspection of `scripts/heartbeat/checks/stathera.py` (lines 273-290: cross-midnight oracle)
- Codebase inspection of `scripts/heartbeat/checks/yesterday.py` (full file: orphan completion)
- Codebase inspection of `python/opendeviationbar/clickhouse/bulk_operations.py` (lines 617-639, 947-972: pre-write gate)
- Codebase inspection of `python/opendeviationbar/clickhouse/stathera_query.py` (cross_pair_window: already partitions by ouroboros_mode)
- Codebase inspection of `python/opendeviationbar/kintsugi/_helpers.py` (lines 66-77: \_get_mode_for_symbol pattern)
- Codebase inspection of `python/opendeviationbar/ouroboros.py` (get_operational_ouroboros_mode: global mode)
- Codebase inspection of `python/opendeviationbar/symbol_registry.py` (SymbolEntry.ouroboros_mode: per-symbol)

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - no new dependencies, pure Python conditional logic
- Architecture: HIGH - all code paths traced, patterns already exist in codebase
- Pitfalls: HIGH - identified from direct code reading, each with specific line numbers

**Research date:** 2026-03-27
**Valid until:** 2026-04-27 (stable domain, no external dependency changes)
