---
phase: 10-seeder-checksum-integration
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - scripts/tick_cache_seeder.py
  - tests/test_seeder_checksum.py
autonomous: true
requirements: [SEEDER-01, SEEDER-02, SEEDER-03]

must_haves:
  truths:
    - "Every Vision ZIP download is SHA-256 verified against its .CHECKSUM file before the seeder writes Parquet"
    - "A checksum mismatch triggers up to 2 re-downloads; persistent failure sends a LOUD Telegram alert"
    - "Each successful verification is recorded in the ChecksumRegistry with symbol, date, checksum, source=seeder"
    - "The seeder state file includes checksum pass/fail/skip counts"
  artifacts:
    - path: "scripts/tick_cache_seeder.py"
      provides: "Checksum verification integrated into download pipeline"
      contains: "verify_checksum_hex"
    - path: "tests/test_seeder_checksum.py"
      provides: "Unit tests for checksum verification integration"
      min_lines: 80
  key_links:
    - from: "scripts/tick_cache_seeder.py"
      to: "opendeviationbar._core"
      via: "compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content"
      pattern: "from opendeviationbar._core import.*compute_sha256_hex"
    - from: "scripts/tick_cache_seeder.py"
      to: "python/opendeviationbar/checksum_registry.py"
      via: "ChecksumRegistry.add() + save()"
      pattern: "registry\\.add\\("
    - from: "scripts/tick_cache_seeder.py"
      to: "python/opendeviationbar/notify/telegram.py"
      via: "send_telegram for LOUD checksum failure alerts"
      pattern: "send_telegram\\("
---

<objective>
Wire SHA-256 checksum verification into the tick_cache_seeder download pipeline so every Vision ZIP is verified before Parquet write, with retry on mismatch, LOUD Telegram alerts on persistent failure, registry persistence, and summary reporting.

Purpose: Closes the data integrity gap where corrupted Vision downloads could silently poison the Parquet tick cache. Foundation for downstream sweep (Phase 11) and heartbeat telemetry (Phase 12).
Output: Modified tick_cache_seeder.py with checksum verification + tests
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/08-checksum-spike/08-01-SUMMARY.md
@.planning/phases/09-verification-registry/09-01-SUMMARY.md
@scripts/tick_cache_seeder.py
@python/opendeviationbar/checksum_registry.py

<interfaces>
<!-- Phase 8 PyO3 checksum bindings (from src/binance_bindings.rs, registered in _core module) -->

```python
from opendeviationbar._core import compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content

# compute_sha256_hex(data: bytes) -> str
#   Returns 64-char lowercase hex SHA-256 hash

# parse_checksum_file_content(content: str) -> str
#   Parses "abc123def...  filename.zip" format, returns 64-char hex hash
#   Raises ValueError on empty, wrong length, non-hex

# verify_checksum_hex(data: bytes, expected: str) -> dict
#   Returns {"verified": bool, "expected": str|None, "actual": str, "skipped": bool}
#   Raises RuntimeError("Checksum mismatch: expected X, got Y") on mismatch
```

<!-- Phase 9 ChecksumRegistry (from python/opendeviationbar/checksum_registry.py) -->

```python
from opendeviationbar.checksum_registry import ChecksumRegistry

registry = ChecksumRegistry()  # auto-loads from ~/.cache/opendeviationbar/checksum-registry.json
registry.add(symbol="BTCUSDT", date="2024-01-15", sha256="abc...", source="seeder")  # no auto-save
registry.save()  # atomic write via tmp+replace
registry.has_verified(symbol="BTCUSDT", date="2024-01-15") -> bool
```

<!-- Telegram notify (from python/opendeviationbar/notify/telegram.py) -->

```python
from opendeviationbar.notify.telegram import send_telegram
send_telegram("<b>LOUD:</b> Checksum mismatch", disable_notification=False)
```

<!-- Vision CHECKSUM URL pattern (from spike_checksum_validation.py) -->

# checksum_url = f"{zip_url}.CHECKSUM"

# e.g., <https://data.binance.vision/data/spot/daily/aggTrades/BTCUSDT/BTCUSDT-aggTrades-2024-01-15.zip.CHECKSUM>

# Content format: "abc123def456... BTCUSDT-aggTrades-2024-01-15.zip\n"

<!-- Edge case strategy (from Phase 8 spike validation) -->

# 404 CHECKSUM -> soft-skip (old data, no checksum available)

# timeout on CHECKSUM fetch -> soft-skip with warning

# mismatch -> hard fail (retry up to 2 times, then LOUD alert)

# invalid format -> hard fail (same as mismatch)

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Add checksum verification to Vision download pipeline with retry and registry</name>
  <files>scripts/tick_cache_seeder.py, tests/test_seeder_checksum.py</files>
  <read_first>
    - scripts/tick_cache_seeder.py (full file — integration target)
    - python/opendeviationbar/checksum_registry.py (registry API)
    - tests/test_checksum_spike.py (existing checksum test patterns)
    - python/opendeviationbar/notify/telegram.py (send_telegram signature)
  </read_first>
  <behavior>
    - Test 1: _verify_vision_checksum() returns {"verified": True, "sha256": "abc..."} when ZIP bytes match CHECKSUM
    - Test 2: _verify_vision_checksum() returns {"verified": False, "sha256": None, "reason": "404"} when .CHECKSUM URL returns 404 (soft-skip)
    - Test 3: _verify_vision_checksum() returns {"verified": False, "sha256": None, "reason": "timeout"} when .CHECKSUM fetch times out (soft-skip)
    - Test 4: _verify_vision_checksum() raises RuntimeError on hash mismatch (hard fail)
    - Test 5: _download_and_verify_vision_day() retries download up to 2 times on checksum mismatch before returning None
    - Test 6: _download_and_verify_vision_day() records successful verification in ChecksumRegistry
    - Test 7: _download_and_verify_vision_day() calls send_telegram on persistent checksum failure (all retries exhausted)
    - Test 8: Checksum stats (pass/fail/skip counters) accumulate correctly across multiple downloads
  </behavior>
  <action>
**Step 1: Create test file `tests/test_seeder_checksum.py`** with the 8 behaviors above. Use `unittest.mock.patch` to mock `requests.get` for CHECKSUM fetch, `opendeviationbar._core` checksum functions, `send_telegram`, and `ChecksumRegistry`. Tests must be fully offline (no network).

**Step 2: Add checksum verification helper to `scripts/tick_cache_seeder.py`:**

Add a new function `_verify_vision_checksum(zip_bytes, symbol, date_str, timeout_s, market)`:

- Build CHECKSUM URL: `_vision_url(symbol, date_str, market=market) + ".CHECKSUM"`
- `requests.get(checksum_url, timeout=timeout_s)` to fetch the CHECKSUM file
- If 404: return `{"verified": False, "sha256": None, "reason": "404"}` (soft-skip, per SPIKE-03)
- If timeout/network error: return `{"verified": False, "sha256": None, "reason": "timeout"}` (soft-skip, per SPIKE-03)
- Parse content via `parse_checksum_file_content(resp.text)` to extract expected hash
- If parse error (ValueError): return `{"verified": False, "sha256": None, "reason": "invalid_format"}`
- Call `verify_checksum_hex(zip_bytes, expected_hash)` — returns dict on match, raises RuntimeError on mismatch
- On match: return `{"verified": True, "sha256": expected_hash}`
- On mismatch: raise RuntimeError (caller handles retry)

**Step 3: Create wrapper `_download_and_verify_vision_day()`** that replaces `_download_vision_day()` calls in the hot path:

- Calls `_download_vision_day()` to get `(resp_content_bytes, df)` — modify `_download_vision_day` to ALSO return `resp.content` (the raw ZIP bytes needed for checksum). Use a new internal function `_download_vision_day_raw()` that returns `(bytes, DataFrame | None)` tuple, and have `_download_vision_day()` call it and return only the DataFrame (backward compat).
- Calls `_verify_vision_checksum(zip_bytes, ...)`
- On verified=True: record in ChecksumRegistry via `registry.add(symbol, date_str, sha256, source="seeder")`
- On soft-skip (404/timeout): proceed without verification (log warning, increment skip counter)
- On mismatch (RuntimeError): retry entire download+verify up to `max_retries=2` times
- After all retries exhausted: send LOUD Telegram alert via `send_telegram()` with symbol, date, expected/actual hash. Return None (skip this day).

**Step 4: Wire into `_seed_symbol()`:**

- Add `registry: ChecksumRegistry` parameter to `_seed_symbol()` (passed from `main()`)
- Add module-level `_checksum_stats` dict: `{"pass": 0, "fail": 0, "skip": 0}` reset in `main()`
- In the thread pool's download worker, replace `_download_day()` with a new `_download_and_verify_day()` that calls `_download_and_verify_vision_day()`
- After each successful verify: increment `_checksum_stats["pass"]`
- After each soft-skip: increment `_checksum_stats["skip"]`
- After each permanent failure: increment `_checksum_stats["fail"]`
- Call `registry.save()` after the thread pool completes (batch save, not per-file — per Phase 9 design decision)

**Step 5: Wire into `main()`:**

- Create `ChecksumRegistry()` instance at top of `main()` (single instance for entire run)
- Pass registry to `_seed_symbol()`
- Add checksum stats to state file output: `checksum_pass=N\nchecksum_fail=N\nchecksum_skip=N\n`
- Final `registry.save()` after all symbols complete

**Important implementation notes:**

- The `_download_day()` function used by thread pool workers calls `_download_vision_day()` which uses `requests.get()` internally. To get the raw bytes for checksum, modify `_download_vision_day` to accept an optional `return_raw_bytes=True` parameter. When True, return a tuple `(df, raw_bytes)`. Default False for backward compat with boundary backfill code that doesn't need checksums.
- Checksum verification ONLY applies to Vision downloads (not REST fallback). REST-sourced data has no CHECKSUM file.
- Thread safety: `_checksum_stats` uses a `threading.Lock` for counter increments. `ChecksumRegistry` operations are batched (add in worker, save after pool) — add() is dict upsert, thread-safe for disjoint keys (each worker processes different dates).
- LOUD Telegram format: `"<b>CHECKSUM FAIL</b>\n\nSymbol: {symbol}\nDate: {date_str}\nExpected: {expected}\nActual: {actual}\n\nAll {max_retries} retries exhausted. Day skipped — manual investigation required."`
  </action>
  <verify>
  <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_seeder_checksum.py -x -v</automated>
  </verify>
  <acceptance_criteria>
  - grep -q "verify_checksum_hex" scripts/tick_cache_seeder.py (SEEDER-01: SHA-256 verification wired in)
  - grep -q "parse_checksum_file_content" scripts/tick_cache_seeder.py (SEEDER-01: CHECKSUM file parsing)
  - grep -q "max_retries" scripts/tick_cache_seeder.py (SEEDER-02: retry logic present)
  - grep -q "send_telegram" scripts/tick_cache_seeder.py (SEEDER-02: LOUD alert on persistent failure)
  - grep -q "registry.add" scripts/tick_cache_seeder.py (SEEDER-03: registry write after verification)
  - grep -q "registry.save" scripts/tick_cache_seeder.py (SEEDER-03: registry persistence)
  - grep -q "checksum_pass" scripts/tick_cache_seeder.py (summary stats in state file)
  - grep -q "checksum_fail" scripts/tick_cache_seeder.py (summary stats in state file)
  - grep -q "checksum_skip" scripts/tick_cache_seeder.py (summary stats in state file)
  - grep -q "ChecksumRegistry" scripts/tick_cache_seeder.py (registry integration)
  - test -f tests/test_seeder_checksum.py (test file exists)
  - grep -c "def test\_" tests/test_seeder_checksum.py returns >= 8 (at least 8 test cases)
    </acceptance_criteria>
    <done>
  - Every Vision ZIP download goes through SHA-256 verification against .CHECKSUM file (SEEDER-01)
  - Mismatch triggers up to 2 re-downloads; persistent failure fires LOUD Telegram alert with details (SEEDER-02)
  - Each successful verification writes to ChecksumRegistry with symbol, date, sha256, source="seeder" (SEEDER-03)
  - State file includes checksum_pass/checksum_fail/checksum_skip counts
  - All 8+ tests pass offline with mocked network calls
    </done>

</task>

</tasks>

<verification>
1. `uv run --python 3.13 pytest tests/test_seeder_checksum.py -x -v` — all checksum integration tests pass
2. `uv run --python 3.13 pytest tests/test_checksum_spike.py tests/test_checksum_registry.py -x -v` — existing Phase 8+9 tests still pass (no regressions)
3. `grep -c "verify_checksum_hex\|parse_checksum_file_content\|compute_sha256_hex" scripts/tick_cache_seeder.py` — returns >= 1 (bindings wired)
4. `grep -c "ChecksumRegistry" scripts/tick_cache_seeder.py` — returns >= 1 (registry integrated)
5. `grep -c "send_telegram" scripts/tick_cache_seeder.py` — returns >= 1 (LOUD alert wired)
</verification>

<success_criteria>

- SEEDER-01: Every Vision ZIP download path includes SHA-256 verification call before Parquet write
- SEEDER-02: Mismatch triggers retry loop (2 retries) then LOUD Telegram alert with symbol/date/hash details
- SEEDER-03: Successful verifications persisted in ChecksumRegistry JSON file via add() + save()
- Existing seeder functionality (REST fallback, boundary backfill, batch rotation) unchanged
- All tests pass offline (no network dependency)
  </success_criteria>

<output>
After completion, create `.planning/phases/10-seeder-checksum-integration/10-01-SUMMARY.md`
</output>
