---
phase: 28-retroactive-sweep
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/retroactive_exchange_gap_sweep.py
autonomous: true
requirements:
  - RETRO-01
  - RETRO-02

must_haves:
  truths:
    - "Script classifies existing Stathera gaps as exchange_gap when checksum-verified Parquet has 0 trades in TID range"
    - "Classified gaps are logged to kintsugi_log with result='exchange_gap' for permanent dismiss"
    - "Script produces a summary report with total gaps, exchange classified, pipeline remaining, unclassifiable"
  artifacts:
    - path: "scripts/retroactive_exchange_gap_sweep.py"
      provides: "One-time retroactive sweep script"
      min_lines: 80
  key_links:
    - from: "scripts/retroactive_exchange_gap_sweep.py"
      to: "opendeviationbar.kintsugi.discovery.discover_shards"
      via: "import and call"
      pattern: "discover_shards"
    - from: "scripts/retroactive_exchange_gap_sweep.py"
      to: "opendeviationbar.checksum_registry.ChecksumRegistry"
      via: "import and call has_verified"
      pattern: "ChecksumRegistry"
    - from: "scripts/retroactive_exchange_gap_sweep.py"
      to: "opendeviationbar_cache.kintsugi_log"
      via: "INSERT with result='exchange_gap'"
      pattern: "exchange_gap"
---

<objective>
One-time retroactive sweep that classifies existing 96K Stathera gaps on bigblack as exchange-level when checksum-verified Parquet confirms 0 trades in the TID range.

Purpose: Permanently dismiss ~96K unhealable exchange gaps from kintsugi's repair queue, eliminating 146h of wasted repair cycles.
Output: `scripts/retroactive_exchange_gap_sweep.py` + summary report printed to stdout
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

@python/opendeviationbar/kintsugi/repair.py
@python/opendeviationbar/kintsugi/discovery.py
@python/opendeviationbar/kintsugi/telemetry.py
@python/opendeviationbar/checksum_registry.py

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/kintsugi/discovery.py:

```python
@dataclass
class Shard:
    symbol: str
    threshold_decimal_bps: int
    gap_start_ms: int
    gap_end_ms: int
    gap_hours: float
    anchor_last_tid: int | None = None
    resume_first_tid: int | None = None
    missing_trade_count: int = 0
    is_overlap: bool = False
    is_genesis: bool = False

def discover_shards(*, symbol=None, threshold=None) -> list[Shard]: ...
```

From python/opendeviationbar/checksum_registry.py:

```python
class ChecksumRegistry:
    def has_verified(self, symbol: str, date: str) -> bool: ...
```

From python/opendeviationbar/kintsugi/telemetry.py - \_log_repair INSERT format:

```python
# kintsugi_log columns: symbol, threshold_decimal_bps, priority,
#   gap_start_ms, gap_end_ms, gap_hours, action, result,
#   ariadne_from_id, ouroboros_splits, bars_written, duration_seconds,
#   error_message, ouroboros_mode
# result='exchange_gap' for exchange gaps (permanent dismiss)
```

From python/opendeviationbar/storage/parquet.py:

```python
class TickStorage:
    def has_ticks(self, symbol: str, start_ms: int, end_ms: int) -> bool: ...
    def read_ticks(self, symbol: str, start_ms: int, end_ms: int) -> pl.DataFrame: ...
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create retroactive exchange gap sweep script</name>
  <files>scripts/retroactive_exchange_gap_sweep.py</files>
  <read_first>
    - python/opendeviationbar/kintsugi/repair.py (lines 362-467 — _repair_aion exchange gap classification logic)
    - python/opendeviationbar/kintsugi/discovery.py (discover_shards, _get_dismissed_set)
    - python/opendeviationbar/kintsugi/telemetry.py (_log_repair INSERT format)
    - python/opendeviationbar/checksum_registry.py (ChecksumRegistry.has_verified)
    - python/opendeviationbar/storage/parquet.py (TickStorage.has_ticks, read_ticks)
    - python/opendeviationbar/orchestration/helpers.py (_select_trade_columns)
  </read_first>
  <action>
Create `scripts/retroactive_exchange_gap_sweep.py` — a one-time script (not a daemon) that retroactively classifies existing Stathera gaps as exchange gaps.

**Algorithm (reuses \_repair_aion classification logic from Phase 26):**

1. Call `discover_shards()` to get ALL Stathera gaps (no symbol/threshold filter)
2. Load `_get_dismissed_set()` to skip already-dismissed shards (both clean 30-day and permanent exchange_gap)
3. For each non-dismissed, non-overlap, non-genesis shard with `anchor_last_tid is not None`:
   a. Convert gap timestamps to date range (`start_date`, `end_date`)
   b. Check `TickStorage.has_ticks()` for ALL dates in range — if any missing, mark as "no_parquet" (unclassifiable)
   c. Load trades from Parquet via `TickStorage.read_ticks()` for each date, concatenate with `pl.concat`
   d. Filter trades to TID range: `agg_trade_id > anchor_last_tid` AND `agg_trade_id < resume_first_tid`
   e. Apply `_select_trade_columns()` to match_repair_aion's column projection
   f. If filtered trades are empty AND all dates are checksum-verified via `ChecksumRegistry.has_verified(symbol, date.isoformat())`: classify as EXCHANGE_GAP
   g. If filtered trades are empty but NOT all dates verified: mark as "unverified_parquet" (unclassifiable)
   h. If filtered trades are non-empty: mark as "pipeline_gap" (has trades, kintsugi should repair)
4. For each EXCHANGE_GAP shard: INSERT into `kintsugi_log` with `result='exchange_gap'` using the same INSERT format as `_log_repair()` in telemetry.py. Use `action='retroactive_sweep'` to distinguish from live classification.
5. Print summary report to stdout:
   - Total shards discovered
   - Already dismissed (skipped)
   - Exchange gaps classified (newly logged)
   - Pipeline gaps remaining (kintsugi will repair)
   - No Parquet available (unclassifiable — missing tick cache)
   - Unverified Parquet (unclassifiable — no checksum)
   - Overlaps skipped
   - Genesis shards skipped

**Implementation details:**

- Use `--dry-run` flag (argparse) that prints what WOULD be classified without INSERT
- Use `--symbol BTCUSDT` optional filter for targeted runs
- Use `--threshold 250` optional filter
- Batch INSERTs: collect all exchange_gap results, INSERT in a single loop (not transactional — kintsugi_log is append-only)
- Use thread-local `_get_worker_cache()` from `_helpers` for CH connection (consistent with kintsugi patterns)
- Log progress every 100 shards: "Processed 100/5000 shards: 42 exchange_gap, 18 pipeline, 40 no_parquet"
- Use Python 3.13 compatible syntax only
- Add `if __name__ == "__main__"` guard
- Import `_select_trade_columns` from `opendeviationbar.orchestration.helpers`
- The INSERT should use the exact same column list and parameter format as `_log_repair()` in telemetry.py, but with `action='retroactive_sweep'` and `ouroboros_mode` from `_get_mode_for_symbol()`
  </action>
  <verify>
  <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "import ast; ast.parse(open('scripts/retroactive_exchange_gap_sweep.py').read()); print('syntax OK')" && grep -c 'exchange_gap' scripts/retroactive_exchange_gap_sweep.py</automated>
  </verify>
  <acceptance_criteria>
  - grep -q "discover_shards" scripts/retroactive_exchange_gap_sweep.py (uses discovery)
  - grep -q "ChecksumRegistry" scripts/retroactive_exchange_gap_sweep.py (checksum verification)
  - grep -q "has_verified" scripts/retroactive_exchange_gap_sweep.py (per-date check)
  - grep -q "kintsugi_log" scripts/retroactive_exchange_gap_sweep.py (logs to CH)
  - grep -q "result.\*exchange_gap" scripts/retroactive_exchange_gap_sweep.py (correct result value)
  - grep -q "dry.run\|dry_run" scripts/retroactive_exchange_gap_sweep.py (dry-run flag)
  - grep -q "retroactive_sweep" scripts/retroactive_exchange_gap_sweep.py (distinguishing action)
  - grep -q "\_select_trade_columns" scripts/retroactive_exchange_gap_sweep.py (TID range filter)
  - grep -q "TickStorage" scripts/retroactive_exchange_gap_sweep.py (Parquet source)
  - grep -q "**main**" scripts/retroactive_exchange_gap_sweep.py (script guard)
    </acceptance_criteria>
    <done>
  - Script parses without errors
  - Discovers shards via discover_shards()
  - Classifies exchange gaps using same logic as \_repair_aion (checksum-verified Parquet + 0 trades in TID range)
  - Logs to kintsugi_log with result='exchange_gap' and action='retroactive_sweep'
  - Supports --dry-run, --symbol, --threshold flags
  - Prints summary report with all categories (exchange_gap, pipeline_gap, no_parquet, unverified, overlap, genesis, dismissed)
    </done>

</task>

</tasks>

<verification>
1. `python -c "import ast; ast.parse(open('scripts/retroactive_exchange_gap_sweep.py').read())"` -- syntax valid
2. Script imports resolve: `cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 python -c "from scripts.retroactive_exchange_gap_sweep import *"` or similar import check
3. All grep acceptance criteria pass
</verification>

<success_criteria>

- Script exists at scripts/retroactive_exchange_gap_sweep.py
- Reuses Phase 26 classification logic (checksum-verified Parquet + 0 trades in TID range)
- Logs exchange gaps to kintsugi_log with result='exchange_gap' for permanent dismiss via \_get_dismissed_set()
- Supports --dry-run for safe preview before production execution
- Summary report covers all gap categories
  </success_criteria>

<output>
After completion, create `.planning/phases/28-retroactive-sweep/28-01-SUMMARY.md`
</output>
