---
phase: 11-integrity-sweep
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/integrity_sweep.py
  - tests/test_integrity_sweep.py
autonomous: true
requirements:
  - SWEEP-01
  - SWEEP-02
  - SWEEP-03

must_haves:
  truths:
    - "Running the sweep audits existing Parquet cache files against Vision CHECKSUM files"
    - "Incremental mode skips already-verified files via ChecksumRegistry"
    - "Sweep produces a summary report with pass/fail/skip counts per symbol"
  artifacts:
    - path: "scripts/integrity_sweep.py"
      provides: "Integrity sweep CLI script"
      min_lines: 150
      contains: "argparse"
    - path: "tests/test_integrity_sweep.py"
      provides: "Unit tests for sweep logic"
      min_lines: 60
  key_links:
    - from: "scripts/integrity_sweep.py"
      to: "python/opendeviationbar/checksum_registry.py"
      via: "ChecksumRegistry import"
      pattern: "from opendeviationbar.checksum_registry import ChecksumRegistry"
    - from: "scripts/integrity_sweep.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "TickStorage.list_days for file discovery"
      pattern: "TickStorage"
    - from: "scripts/integrity_sweep.py"
      to: "opendeviationbar._core"
      via: "Rust SHA-256 and CHECKSUM parsing"
      pattern: "compute_sha256_hex|parse_checksum_file_content"
---

<objective>
Create an integrity sweep script that audits all existing Parquet cache files against Binance Vision CHECKSUM files. Supports full and incremental modes, records results in ChecksumRegistry, and produces a summary report.

Purpose: SWEEP-01/02/03 -- retroactive verification of all historical Parquet data that was downloaded before Phase 10 added checksum verification to the seeder pipeline.
Output: `scripts/integrity_sweep.py` with argparse CLI, concurrent verification, registry integration, and Telegram summary.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

@python/opendeviationbar/checksum_registry.py
@scripts/tick_cache_seeder.py
@python/opendeviationbar/storage/parquet.py

<interfaces>
<!-- Key types and contracts the executor needs. -->

From python/opendeviationbar/checksum_registry.py:

```python
class ChecksumRegistry:
    def __init__(self, path: Path | None = None) -> None: ...
    def add(self, symbol: str, date: str, sha256: str, source: str) -> None: ...
    def has_verified(self, symbol: str, date: str) -> bool: ...
    def save(self) -> None: ...
    def summary(self) -> dict: ...  # {"total_verified": int, "by_symbol": dict, "latest_verified_at": str|None}
```

From python/opendeviationbar/storage/parquet.py:

```python
class TickStorage:
    def __init__(self, cache_dir: Path | str | None = None) -> None: ...
    def list_days(self, symbol: str) -> list[str]: ...  # ["2024-01-01", "2024-01-02", ...]
    def _get_symbol_dir(self, symbol: str) -> Path: ...
    @property
    def ticks_dir(self) -> Path: ...
```

From opendeviationbar.\_core (Rust bindings, Phase 8):

```python
def compute_sha256_hex(data: bytes) -> str: ...
def parse_checksum_file_content(content: str) -> str: ...  # raises ValueError on invalid format
def verify_checksum_hex(data: bytes, expected_hex: str) -> None: ...  # raises RuntimeError on mismatch
```

From python/opendeviationbar/symbol_registry.py:

```python
def get_registered_symbols(enabled_only: bool = False) -> list[str]: ...
```

Vision CHECKSUM URL pattern:
<https://data.binance.vision/data/spot/daily/aggTrades/{SYMBOL}/{SYMBOL}-aggTrades-{DATE}.zip.CHECKSUM>

Vision ZIP URL pattern:
<https://data.binance.vision/data/spot/daily/aggTrades/{SYMBOL}/{SYMBOL}-aggTrades-{DATE}.zip>
</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Create integrity sweep script with full and incremental modes</name>
  <files>scripts/integrity_sweep.py, tests/test_integrity_sweep.py</files>
  <read_first>
    - scripts/tick_cache_seeder.py (lines 370-490 for _verify_vision_checksum and _download_and_verify_vision_day patterns)
    - python/opendeviationbar/checksum_registry.py (full file -- registry API)
    - python/opendeviationbar/storage/parquet.py (lines 68-130 and 553-580 for TickStorage.list_days)
    - python/opendeviationbar/symbol_registry.py (get_registered_symbols)
  </read_first>
  <behavior>
    - Test 1: _build_work_items() with empty registry returns all Parquet days as work items
    - Test 2: _build_work_items() with some verified entries skips those (incremental mode)
    - Test 3: _build_work_items() with --full flag returns all items regardless of registry
    - Test 4: _verify_one_day() with matching checksum returns {"status": "pass", "sha256": hash}
    - Test 5: _verify_one_day() with 404 CHECKSUM returns {"status": "skip", "reason": "404"}
    - Test 6: _verify_one_day() with mismatch returns {"status": "fail"} (no retry in sweep -- it's auditing, not downloading)
    - Test 7: _aggregate_results() produces correct per-symbol counts
  </behavior>
  <action>
    Create `scripts/integrity_sweep.py` following the established seeder pattern (argparse CLI, concurrent.futures thread pool, tqdm progress, Telegram summary).

    **Architecture:**

    1. **File discovery** -- For each enabled Binance spot symbol (from `get_registered_symbols(enabled_only=True)`, filtered to exchange="binance"), use `TickStorage().list_days(symbol)` to get all cached dates. The symbol key in TickStorage may be plain (e.g., "BTCUSDT") or prefixed (e.g., "BINANCE_SPOT_BTCUSDT"). Scan the `ticks_dir` for all subdirectories and iterate.

    2. **Work item filtering (per SWEEP-02):**
       - Default (incremental): skip days where `registry.has_verified(symbol, date)` is True
       - `--full` flag: verify all days regardless of registry state
       - `--symbol BTCUSDT` flag: restrict to one symbol
       - `--start-date` / `--end-date` flags: restrict date range

    3. **Verification per day (per SWEEP-01):**
       - Fetch `.CHECKSUM` file from Vision: `https://data.binance.vision/data/spot/daily/aggTrades/{SYMBOL}/{SYMBOL}-aggTrades-{DATE}.zip.CHECKSUM`
       - Parse expected hash via `parse_checksum_file_content(resp.text)`
       - Re-download the Vision ZIP (NOT the local Parquet -- the Parquet is processed data, the CHECKSUM is for the original ZIP)
       - Compute SHA-256 of ZIP bytes via `compute_sha256_hex(zip_bytes)`
       - Compare: match = pass, mismatch = fail, 404/timeout = skip
       - On pass: `registry.add(symbol, date, sha256, source="sweep")`
       - On fail: log WARNING with symbol, date, expected vs actual hash

    4. **Concurrency:** `concurrent.futures.ThreadPoolExecutor(max_workers=N)` where N defaults to 8 (network-bound, not CPU-bound; lower than seeder's 12 because sweep re-downloads full ZIPs which are larger). Configurable via `--workers`.

    5. **Summary report (per SWEEP-03):**
       - Print table with per-symbol pass/fail/skip counts
       - Print total elapsed time
       - If any failures, print list of failed (symbol, date) pairs
       - Optional `--telegram` flag to send summary via `send_telegram()`

    6. **Registry persistence:** `registry.save()` after all workers complete (batch save, not per-file -- same pattern as seeder).

    **argparse flags:**
    - `--full` -- verify all files (ignore registry)
    - `--symbol SYMBOL` -- restrict to one symbol
    - `--start-date YYYY-MM-DD` -- filter start
    - `--end-date YYYY-MM-DD` -- filter end
    - `--workers N` -- thread pool size (default 8)
    - `--timeout N` -- HTTP timeout seconds (default 30)
    - `--telegram` -- send summary to Telegram
    - `--dry-run` -- list work items without downloading

    **Tests** (`tests/test_integrity_sweep.py`):
    Write unit tests for the pure logic functions (_build_work_items, _aggregate_results) using mock registry and mock TickStorage. For _verify_one_day, mock requests.get to test the three outcomes (pass, skip, fail). Do NOT make real network calls in tests.

    **Important implementation details:**
    - The sweep downloads the original ZIP (not reads local Parquet) because the CHECKSUM is computed against the ZIP bytes, not the extracted/processed Parquet data
    - Use `requests.get()` for both CHECKSUM and ZIP downloads (same as seeder)
    - Handle the case where a symbol directory exists but uses a different naming convention (plain "BTCUSDT" vs "BINANCE_SPOT_BTCUSDT") -- scan ticks_dir subdirectories, extract the base symbol name
    - Exit code: 0 if no failures, 1 if any failures found

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_integrity_sweep.py -x -v 2>&1 | tail -30</automated>
  </verify>
  <acceptance_criteria>
    - grep -q "argparse" scripts/integrity_sweep.py
    - grep -q "ChecksumRegistry" scripts/integrity_sweep.py
    - grep -q "has_verified" scripts/integrity_sweep.py
    - grep -q "compute_sha256_hex\|verify_checksum_hex" scripts/integrity_sweep.py
    - grep -q "parse_checksum_file_content" scripts/integrity_sweep.py
    - grep -q "ThreadPoolExecutor" scripts/integrity_sweep.py
    - grep -q "source=\"sweep\"" scripts/integrity_sweep.py
    - grep -q "def test_" tests/test_integrity_sweep.py
    - grep -q "\-\-full" scripts/integrity_sweep.py
    - grep -q "list_days\|ticks_dir" scripts/integrity_sweep.py
  </acceptance_criteria>
  <done>
    - scripts/integrity_sweep.py exists with argparse CLI supporting --full, --symbol, --start-date, --end-date, --workers, --dry-run, --telegram flags
    - Script discovers Parquet cache files via TickStorage, checks registry for incremental skip, downloads Vision ZIPs and CHECKSUMs, verifies SHA-256
    - Results recorded in ChecksumRegistry with source="sweep"
    - Summary report printed with per-symbol pass/fail/skip counts
    - Tests pass covering work item filtering, verification outcomes, and result aggregation
  </done>
</task>

</tasks>

<verification>
1. `uv run --python 3.13 pytest tests/test_integrity_sweep.py -x -v` -- all tests pass
2. `python scripts/integrity_sweep.py --dry-run` -- lists work items without network calls
3. `grep -c "def test_" tests/test_integrity_sweep.py` -- at least 5 test functions
</verification>

<success_criteria>

- Integrity sweep script can audit Parquet cache files against Vision CHECKSUM files (SWEEP-01)
- Incremental mode skips already-verified files via ChecksumRegistry (SWEEP-02)
- Summary report shows pass/fail/skip counts per symbol (SWEEP-03)
- All unit tests pass
  </success_criteria>

<output>
After completion, create `.planning/phases/11-integrity-sweep/11-01-SUMMARY.md`
</output>
