# Phase 26: Exchange Gap Classification + Dismiss - Research

**Researched:** 2026-03-27
**Domain:** Kintsugi gap reconciliation -- exchange gap detection and permanent dismissal
**Confidence:** HIGH

## Summary

This phase adds exchange gap classification to kintsugi's Aion repair path. The core change is small: when `_repair_aion()` processes a shard using checksum-verified Parquet files and the TID-range filter produces 0 trades, the repair result is logged as `result='exchange_gap'` instead of `result='clean'`. The shard selection query (`_get_dismissed_set()`) is then updated to permanently skip exchange-gap shards (no 30-day expiry).

All the infrastructure exists. `_repair_aion()` already returns 0 for exchange gaps (line 438 of repair.py). `ChecksumRegistry.has_verified(symbol, date)` already checks per-date verification state. `kintsugi_log` already stores results. The changes are purely classification logic -- no new tables, no schema migrations, no new data sources.

**Primary recommendation:** Add exchange gap classification inside `_repair_aion()` (or `repair_shard()`) by checking ChecksumRegistry before returning 0 bars. Update `_get_dismissed_set()` to include permanent exchange-gap dismissal. Keep changes minimal -- 3 files touched.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices at Claude's discretion.

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

### Deferred Ideas (OUT OF SCOPE)

None -- discuss phase skipped.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID         | Description                                                                                                     | Research Support                                                                                                                                                  |
| ---------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| EXGAP-01   | Kintsugi `_repair_aion()` marks gap as `is_exchange_gap` when checksum-validated Parquet repair produces 0 bars | `_repair_aion()` returns 0 at line 438 when TID-filtered trades are empty. Add ChecksumRegistry check before this return to classify as exchange gap.             |
| EXGAP-02   | Classification requires both checksum-verified Parquet AND 0 trades in TID range                                | ChecksumRegistry.has_verified(symbol, date) provides per-date verification. Must check ALL dates in the shard's range.                                            |
| DISMISS-01 | Shard dismiss filter permanently skips exchange gaps (no 30-day expiry)                                         | `_get_dismissed_set()` currently queries `result = 'clean'` with 30-day window. Add separate query for `result = 'exchange_gap'` with NO time window (permanent). |
| DISMISS-02 | `kintsugi_log` records exchange gap classifications with `result='exchange_gap'`                                | `_log_repair()` in telemetry.py computes result string from `result.healed` and `result.error`. Need new field on KintsugiResult or new result computation logic. |

</phase_requirements>

## Architecture Patterns

### Current Flow (relevant excerpt)

```
repair_shard()                     # repair.py
  |
  +-- _repair()
  |     +-- _repair_aion()         # Aion shards: TID-range from Parquet
  |           +-- load Parquet for date range
  |           +-- filter to TID range [anchor+1, resume-1]
  |           +-- if trades_df.is_empty(): return 0   <-- EXCHANGE GAP POINT
  |           +-- process trades -> bars
  |           +-- write bars to CH
  |
  +-- return KintsugiResult(healed=False, bars_written=0, error=None)
  |     ^^ This is the "clean" result (0 bars, no error)
  |
  +-- _maybe_log_repair(result)    # telemetry.py
        +-- _log_repair(result)
              +-- result = "healed" if healed else "clean" if no error else "failed"
              +-- INSERT INTO kintsugi_log

_select_repair_candidates()        # discovery.py
  +-- _get_dismissed_set()
        +-- SELECT WHERE result = 'clean' AND bars_written = 0
        +--   AND timestamp > now() - INTERVAL 30 DAY
```

### Target Flow

```
_repair_aion()
  |
  +-- load Parquet, filter TID range
  +-- if trades_df.is_empty():
  |     +-- check ChecksumRegistry for ALL dates in range
  |     +-- if ALL verified: set thread-local flag "exchange_gap"
  |     +-- return 0
  |
repair_shard()
  +-- detect exchange_gap flag
  +-- KintsugiResult with is_exchange_gap=True (new field)
  |
_log_repair()
  +-- result = "exchange_gap" if is_exchange_gap else ...
  |
_get_dismissed_set()
  +-- existing: result='clean' AND 30-DAY window
  +-- new:      result='exchange_gap' (NO time window -- permanent)
  +-- union both into dismissed set
```

### Key Design Decisions

1. **Where to classify**: Inside `_repair_aion()` at the `trades_df.is_empty()` return point (line 437-438 of repair.py). This is the only place where we have both the Parquet source information and the 0-trade result.

2. **How to propagate**: Add `is_exchange_gap: bool = False` field to `KintsugiResult` dataclass. The repair_shard() wrapper reads a thread-local flag set by `_repair_aion()` (pattern already used for `last_source`). Alternative: return a sentinel value from `_repair_aion()` (e.g., -2 for exchange gap, similar to -1 for today-guard). The sentinel approach is simpler and consistent with existing patterns.

3. **ChecksumRegistry scope**: Must verify ALL dates in the shard's Parquet lookup range, not just one. A shard spanning 3 dates needs all 3 verified. Use `registry.has_verified(symbol, date_str)` for each date in `[start_date, end_date]`.

4. **Dismiss permanence**: Exchange gaps never expire because the underlying data (exchange TID sequence) is immutable. A TID gap from 2018 will never be filled by future data. The `_get_dismissed_set()` query should use no `timestamp >` filter for `result='exchange_gap'`.

5. **Log message fix**: The current log at line 107 says "day-start-fill" regardless of repair path. Should say "aion-repair" for Aion shards and distinguish exchange gap outcomes.

### Recommended Changes by File

| File           | Change                                                                                                                                                          | Lines     |
| -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `discovery.py` | Add `is_exchange_gap: bool = False` to `KintsugiResult`                                                                                                         | ~1 line   |
| `repair.py`    | In `_repair_aion()`: check ChecksumRegistry at empty-trades return, set thread-local flag. In `repair_shard()`: read flag, set `is_exchange_gap=True` on result | ~20 lines |
| `telemetry.py` | In `_log_repair()`: add `"exchange_gap"` result value when `result.is_exchange_gap`                                                                             | ~3 lines  |
| `discovery.py` | In `_get_dismissed_set()`: add permanent exchange-gap query to dismissed set                                                                                    | ~10 lines |

### Anti-Patterns to Avoid

- **Classifying without checksum verification**: An unverified Parquet file might be incomplete (missing trades). Only checksum-verified files guarantee the TID range is truly empty at the exchange level.
- **Using 30-day expiry for exchange gaps**: Exchange TID gaps are permanent -- the data will never appear. A 30-day dismiss window means kintsugi retries 96K shards every month for no reason.
- **Modifying Rust `is_exchange_gap` field**: Out of scope per REQUIREMENTS.md. Python-side classification is sufficient.
- **Checking ChecksumRegistry per shard in the hot path of `_get_dismissed_set()`**: The dismiss query runs once per pass for ALL shards. Keep it as a simple ClickHouse query on `kintsugi_log.result`, not a Python-side ChecksumRegistry check.

## Don't Hand-Roll

| Problem                     | Don't Build          | Use Instead                            | Why                                                                        |
| --------------------------- | -------------------- | -------------------------------------- | -------------------------------------------------------------------------- |
| Checksum verification state | Custom file tracking | `ChecksumRegistry.has_verified()`      | Already exists, per-date granularity, JSON-backed                          |
| Repair result logging       | New table or file    | `kintsugi_log` table + `_log_repair()` | Existing infrastructure with TTL, audit trail                              |
| Shard dismissal             | Custom state file    | `_get_dismissed_set()` query           | Batch ClickHouse query, already integrated with repair candidate selection |

## Common Pitfalls

### Pitfall 1: Parquet available but not checksum-verified

**What goes wrong:** Classifying a gap as exchange-level when the Parquet file is incomplete (downloaded partially, no Vision CHECKSUM match).
**Why it happens:** `_repair_aion()` checks `storage.has_ticks()` (file exists) but not checksum status.
**How to avoid:** EXGAP-02 requires BOTH conditions: `has_verified()` AND 0 trades. If Parquet exists but is unverified, treat as normal 0-bar clean result (30-day dismiss, retry later).
**Warning signs:** Exchange gap count suddenly spikes for recent dates where seeder hasn't run yet.

### Pitfall 2: Multi-day shard with partial verification

**What goes wrong:** A shard spans 2018-01-15 to 2018-01-17. Days 15 and 17 are verified but day 16 is not. Classifying as exchange gap is wrong.
**Why it happens:** Only checking the first or last date in the range.
**How to avoid:** Iterate ALL dates in `[start_date, end_date]` and require ALL to be verified.
**Warning signs:** Small number of exchange-gap classifications that later get re-discovered as real gaps.

### Pitfall 3: kintsugi_log TTL deleting exchange gap records

**What goes wrong:** The `kintsugi_log` table has `TTL timestamp + INTERVAL 90 DAY`. After 90 days, exchange gap records are deleted, causing kintsugi to rediscover and retry them.
**Why it happens:** TTL applies uniformly to all rows.
**How to avoid:** Either (a) exclude `result='exchange_gap'` from TTL (ALTER TABLE MODIFY TTL), or (b) accept 90-day re-classification cycles (they complete instantly since Parquet is still verified). Option (b) is simpler -- re-classification costs ~0s per shard since it just reads Parquet, filters, finds 0 trades, and re-logs.
**Warning signs:** Periodic spikes in kintsugi pass duration every 90 days.

### Pitfall 4: Column name mismatch in kintsugi_log

**What goes wrong:** The schema.sql defines columns as `gap_start_us` and `gap_end_us`, but the INSERT in `_log_repair()` uses `gap_start_ms` and `gap_end_ms`.
**Why it happens:** Legacy naming from pre-v4.0 when timestamps were milliseconds.
**How to avoid:** The live ClickHouse table likely has the `_ms` column names (schema.sql may be stale). Use the same column names as the existing INSERT. Do NOT try to "fix" the column names in this phase.
**Warning signs:** INSERT failures if you reference wrong column names.

## Code Examples

### Exchange gap classification in \_repair_aion()

```python
# In repair.py, _repair_aion(), after TID filtering produces empty result:

if trades_df.is_empty():
    # Check if ALL dates in range are checksum-verified.
    # Only classify as exchange gap when we can trust the Parquet data.
    from opendeviationbar.checksum_registry import ChecksumRegistry

    registry = ChecksumRegistry()
    all_verified = True
    current = start_date
    while current <= end_date:
        if not registry.has_verified(shard.symbol, current.isoformat()):
            all_verified = False
            break
        current += timedelta(days=1)

    if all_verified and parquet_available:
        _thread_local.is_exchange_gap = True
        logger.info(
            "kintsugi Aion: %s@%d -- exchange gap (0 trades in TID range "
            "[%s, %s], all %d Parquet days checksum-verified)",
            shard.symbol,
            shard.threshold_decimal_bps,
            shard.anchor_last_tid,
            shard.resume_first_tid,
            (end_date - start_date).days + 1,
        )

    return 0
```

### Updated result computation in \_log_repair()

```python
# In telemetry.py, _log_repair():
"result": (
    "exchange_gap"
    if getattr(result, "is_exchange_gap", False)
    else "healed"
    if result.healed
    else "clean"
    if not result.error
    else "failed"
),
```

### Updated \_get_dismissed_set()

```python
def _get_dismissed_set() -> set[tuple[str, int, int]]:
    _aletheia_purge()
    try:
        from opendeviationbar.clickhouse import OpenDeviationBarCache
        with OpenDeviationBarCache() as cache:
            # 30-day dismiss for clean (sub-threshold) shards
            clean_result = cache.client.query(
                "SELECT DISTINCT symbol, threshold_decimal_bps, gap_start_ms "
                "FROM opendeviationbar_cache.kintsugi_log "
                "WHERE result = 'clean' AND bars_written = 0 "
                "AND timestamp > now() - INTERVAL 30 DAY"
            )
            dismissed = {(row[0], row[1], row[2]) for row in clean_result.result_rows}

            # Permanent dismiss for exchange gaps (no time window)
            exgap_result = cache.client.query(
                "SELECT DISTINCT symbol, threshold_decimal_bps, gap_start_ms "
                "FROM opendeviationbar_cache.kintsugi_log "
                "WHERE result = 'exchange_gap'"
            )
            dismissed |= {(row[0], row[1], row[2]) for row in exgap_result.result_rows}

            return dismissed
    except (OSError, RuntimeError, ConnectionError):
        logger.debug("_get_dismissed_set query failed", exc_info=True)
        return set()
```

## State of the Art

| Old Approach                               | Current Approach                                      | When Changed | Impact                        |
| ------------------------------------------ | ----------------------------------------------------- | ------------ | ----------------------------- |
| 0-bar result = "clean" (30d dismiss)       | 0-bar + verified Parquet = "exchange_gap" (permanent) | This phase   | Eliminates 96K futile retries |
| Single dismiss window (30d)                | Two-tier: 30d for clean, permanent for exchange_gap   | This phase   | Exchange gaps never retry     |
| Log message "day-start-fill" for all paths | Path-specific labels (aion-repair, exchange_gap)      | This phase   | Accurate telemetry            |

## Open Questions

1. **kintsugi_log TTL and exchange gap records**
   - What we know: TTL is 90 days. Exchange gap records will be deleted after 90 days.
   - What's unclear: Whether we should modify TTL to exclude exchange_gap rows or accept periodic re-classification.
   - Recommendation: Accept re-classification. Cost is negligible (~0s per shard). Modifying TTL adds complexity and the 90-day cycle is harmless. The 96K shards will be re-classified in a single pass.

2. **Column naming: gap_start_ms vs gap_start_us**
   - What we know: Schema.sql says `gap_start_us` but live INSERT uses `gap_start_ms`. The live table likely has `_ms` names.
   - What's unclear: Whether the schema.sql is stale or the live table was altered.
   - Recommendation: Follow the existing INSERT code (use `gap_start_ms`). Do not investigate or fix this discrepancy in this phase.

3. **Thread-local vs sentinel for exchange gap signaling**
   - What we know: `_thread_local.last_source` pattern is used for source tracking. Return value -1 is used for today-guard.
   - What's unclear: Whether a sentinel (e.g., -2) or thread-local flag is cleaner.
   - Recommendation: Use `_thread_local.is_exchange_gap = True` (consistent with `last_source` pattern). Reset to False at the start of `_repair_aion()`. Alternative: add `is_exchange_gap` directly to `KintsugiResult` and set it in `repair_shard()` by reading the thread-local. Both work; thread-local + KintsugiResult field is the most explicit.

## Validation Architecture

### Test Framework

| Property           | Value                                           |
| ------------------ | ----------------------------------------------- |
| Framework          | pytest                                          |
| Config file        | `pyproject.toml` [tool.pytest.ini_options]      |
| Quick run command  | `pytest tests/test_kintsugi_exchange_gap.py -x` |
| Full suite command | `mise run test-py`                              |

### Phase Requirements to Test Map

| Req ID     | Behavior                                                                                                    | Test Type | Automated Command                                                                       | File Exists? |
| ---------- | ----------------------------------------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------------------- | ------------ |
| EXGAP-01   | \_repair_aion returns 0 and sets exchange_gap flag when checksum-verified Parquet has 0 trades in TID range | unit      | `pytest tests/test_kintsugi_exchange_gap.py::test_exchange_gap_classification -x`       | Wave 0       |
| EXGAP-02   | Classification requires both verified Parquet AND 0 trades; unverified = clean (not exchange_gap)           | unit      | `pytest tests/test_kintsugi_exchange_gap.py::test_unverified_parquet_not_classified -x` | Wave 0       |
| DISMISS-01 | \_get_dismissed_set includes exchange_gap results with no time window                                       | unit      | `pytest tests/test_kintsugi_exchange_gap.py::test_permanent_dismiss -x`                 | Wave 0       |
| DISMISS-02 | \_log_repair writes result='exchange_gap' for classified gaps                                               | unit      | `pytest tests/test_kintsugi_exchange_gap.py::test_log_exchange_gap_result -x`           | Wave 0       |

### Sampling Rate

- **Per task commit:** `pytest tests/test_kintsugi_exchange_gap.py -x`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_kintsugi_exchange_gap.py` -- covers EXGAP-01, EXGAP-02, DISMISS-01, DISMISS-02
- [ ] Mock fixtures for ChecksumRegistry, TickStorage, ClickHouse client

## Sources

### Primary (HIGH confidence)

- Codebase inspection: `python/opendeviationbar/kintsugi/repair.py` -- `_repair_aion()` lines 360-503
- Codebase inspection: `python/opendeviationbar/kintsugi/discovery.py` -- `_get_dismissed_set()` lines 464-487
- Codebase inspection: `python/opendeviationbar/kintsugi/telemetry.py` -- `_log_repair()` lines 98-144
- Codebase inspection: `python/opendeviationbar/checksum_registry.py` -- `ChecksumRegistry` class
- Codebase inspection: `python/opendeviationbar/clickhouse/schema.sql` -- `kintsugi_log` table definition

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- no new dependencies, pure Python changes to existing modules
- Architecture: HIGH -- all integration points inspected, patterns well-established
- Pitfalls: HIGH -- failure modes identified from code analysis (TTL, multi-day verification, column naming)

**Research date:** 2026-03-27
**Valid until:** 2026-04-27 (stable -- infrastructure changes to existing codebase)
