---
phase: 08-checksum-spike
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - src/binance_bindings.rs
  - src/lib.rs
  - tests/test_checksum_spike.py
  - scripts/spike_checksum_validation.py
autonomous: false
requirements: [SPIKE-01, SPIKE-02, SPIKE-03]

must_haves:
  truths:
    - "Python can call Rust compute_sha256_hex() and get a correct 64-char lowercase hex hash"
    - "Python can call verify_checksum_hex() and get a dict with verified/expected/actual/skipped fields"
    - "Python can call parse_checksum_file_content() and extract the hash from Binance CHECKSUM format"
    - "A real Vision ZIP download verified against its .CHECKSUM file produces verified=true"
    - "Checksum verification adds less than 10% wall-clock overhead to a Vision download cycle"
    - "A 404 CHECKSUM (old data) results in skipped=true, not a crash"
    - "A corrupted download (wrong hash) results in a RuntimeError with 'mismatch' in the message"
    - "A network timeout on CHECKSUM fetch results in skipped=true, not a crash"
  artifacts:
    - path: "src/binance_bindings.rs"
      provides: "Three new PyO3 functions: compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content"
      contains: "pub fn compute_sha256_hex"
    - path: "src/lib.rs"
      provides: "Registration of three new checksum functions in _core module"
      contains: "compute_sha256_hex"
    - path: "tests/test_checksum_spike.py"
      provides: "Offline unit tests for new PyO3 checksum bindings"
      contains: "test_compute_sha256_hex"
    - path: "scripts/spike_checksum_validation.py"
      provides: "Network integration test proving real Vision data verification + perf benchmark + edge cases"
      contains: "SPIKE-01"
  key_links:
    - from: "src/binance_bindings.rs"
      to: "crates/opendeviationbar-providers/src/binance/checksum.rs"
      via: "use opendeviationbar_providers::binance::checksum"
      pattern: "checksum::compute_sha256"
    - from: "tests/test_checksum_spike.py"
      to: "src/binance_bindings.rs"
      via: "from opendeviationbar._core import compute_sha256_hex"
      pattern: "compute_sha256_hex"
    - from: "scripts/spike_checksum_validation.py"
      to: "src/binance_bindings.rs"
      via: "from opendeviationbar._core import compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content"
      pattern: "verify_checksum_hex"
---

<objective>
Prove that Rust checksum.rs SHA-256 functions are callable from Python with correct results, acceptable performance (< 10% overhead), and graceful edge case handling (404 soft-skip, mismatch hard fail, timeout soft-skip).

Purpose: This spike validates the mechanism before Phase 10 wires it into the seeder. The seeder currently downloads Vision ZIPs via raw `requests.get()` with zero checksum verification. The standalone PyO3 bindings created here (`compute_sha256_hex`, `verify_checksum_hex`, `parse_checksum_file_content`) are the building blocks the seeder will use.

Output: Three new PyO3 functions registered in `_core` module, offline unit tests, and a network integration script proving real-data verification + benchmarking + edge cases.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/08-checksum-spike/08-RESEARCH.md

@src/binance_bindings.rs
@src/lib.rs
@crates/opendeviationbar-providers/src/binance/checksum.rs
@crates/opendeviationbar-providers/src/binance/mod.rs

<interfaces>
<!-- Rust checksum module public API (from crates/opendeviationbar-providers/src/binance/checksum.rs) -->

```rust
pub fn compute_sha256(data: &[u8]) -> String  // Returns 64-char lowercase hex
pub fn parse_checksum_file(content: &str) -> Result<String, ChecksumError>  // Extracts hash from "hash  filename" format
pub fn verify_checksum(data: &[u8], expected: &str) -> Result<ChecksumResult, ChecksumError>  // Compares computed vs expected

pub struct ChecksumResult {
    pub verified: bool,
    pub expected: Option<String>,
    pub actual: String,
    pub skipped: bool,
}

pub enum ChecksumError {
    Mismatch { expected: String, actual: String },
    FetchFailed(String),
    InvalidFormat(String),
    NotAvailable { status: u16 },
}
```

<!-- Re-exported from crates/opendeviationbar-providers/src/binance/mod.rs line 36-38 -->

```rust
pub use checksum::{
    ChecksumError, ChecksumResult, compute_sha256, fetch_and_verify, fetch_checksum,
    parse_checksum_file, verify_checksum,
};
```

<!-- Existing PyO3 pattern for dict return (from src/binance_bindings.rs line 51-58) -->

```rust
#[pyfunction]
pub fn get_rest_rate_limiter_status(py: Python) -> PyResult<PyObject> {
    let limiter = get_python_rate_limiter();
    let dict = PyDict::new(py);
    dict.set_item("utilization", limiter.utilization())?;
    // ...
    Ok(dict.into_any().unbind())
}
```

<!-- Module registration pattern (from src/lib.rs line 124-148, under #[cfg(feature = "data-providers")]) -->

```rust
m.add_function(wrap_pyfunction!(binance_bindings::fetch_binance_aggtrades, m)?)?;
// ... other binance functions ...
m.add_function(wrap_pyfunction!(binance_bindings::get_rest_rate_limiter_status, m)?)?;
```

<!-- SPIKE-02 prerequisite: fetch_binance_aggtrades already exposes verify_checksum kwarg -->
<!-- From src/binance_bindings.rs line 146: -->

```rust
#[pyo3(signature = (symbol, start_date, end_date, market_type = PyMarketType::Spot, verify_checksum = true))]
pub fn fetch_binance_aggtrades(
    py: Python,
    symbol: &str,
    start_date: &str,
    end_date: &str,
    market_type: PyMarketType,
    verify_checksum: bool,
) -> PyResult<Vec<PyObject>> {
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add PyO3 checksum bindings and register in _core module</name>
  <files>src/binance_bindings.rs, src/lib.rs</files>
  <read_first>
    - src/binance_bindings.rs (current state, import pattern, existing PyO3 functions)
    - src/lib.rs (module registration pattern under data-providers feature gate)
    - crates/opendeviationbar-providers/src/binance/checksum.rs (Rust API signatures)
    - crates/opendeviationbar-providers/src/binance/mod.rs (re-exports at line 36-38)
  </read_first>
  <action>
Add three new `#[pyfunction]` functions to `src/binance_bindings.rs`, after the existing `get_rest_rate_limiter_status` function (around line 59):

1. `compute_sha256_hex(data: &[u8]) -> String` -- calls `opendeviationbar_providers::binance::checksum::compute_sha256(data)` and returns the 64-char lowercase hex string directly.

2. `parse_checksum_file_content(content: &str) -> PyResult<String>` -- calls `checksum::parse_checksum_file(content)` and maps `ChecksumError::InvalidFormat` to `PyValueError`.

3. `verify_checksum_hex(py: Python, data: &[u8], expected: &str) -> PyResult<PyObject>` -- calls `checksum::verify_checksum(data, expected)`. On `Ok(result)`, returns a Python dict with keys `verified` (bool), `expected` (Option str), `actual` (str), `skipped` (bool). On `Err(ChecksumError::Mismatch { expected, actual })`, raises `PyRuntimeError` with message `"Checksum mismatch: expected {expected}, got {actual}"`. On other errors, raises `PyRuntimeError` with the error's Display string.

Import the checksum module at the top of binance_bindings.rs:

```rust
use opendeviationbar_providers::binance::checksum;
```

Then register all three functions in `src/lib.rs` inside the `#[cfg(feature = "data-providers")]` block (after the existing `get_rest_rate_limiter_status` registration around line 144):

```rust
m.add_function(wrap_pyfunction!(binance_bindings::compute_sha256_hex, m)?)?;
m.add_function(wrap_pyfunction!(binance_bindings::verify_checksum_hex, m)?)?;
m.add_function(wrap_pyfunction!(binance_bindings::parse_checksum_file_content, m)?)?;
```

After editing both files, rebuild: `maturin develop`
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && maturin develop 2>&1 | tail -5 && uv run --python 3.13 python -c "
from opendeviationbar.\_core import compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content

# Test compute_sha256_hex correctness

h = compute_sha256_hex(b'hello world')
assert h == 'b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9', f'got {h}'

# Test verify_checksum_hex returns dict with required keys

result = verify_checksum_hex(b'hello world', h)
assert isinstance(result, dict), f'expected dict, got {type(result)}'
for key in ('verified', 'expected', 'actual', 'skipped'):
assert key in result, f'missing key: {key}'
assert result['verified'] is True
assert result['skipped'] is False
print('OK: all 3 functions importable, sha256 correct, verify_checksum_hex dict structure valid')
"</automated>
</verify>
<acceptance_criteria>

- `grep -c "pub fn compute_sha256_hex" src/binance_bindings.rs` returns 1
- `grep -c "pub fn verify_checksum_hex" src/binance_bindings.rs` returns 1
- `grep -c "pub fn parse_checksum_file_content" src/binance_bindings.rs` returns 1
- `grep -c "compute_sha256_hex" src/lib.rs` returns at least 1
- `grep -c "verify_checksum_hex" src/lib.rs` returns at least 1
- `grep -c "parse_checksum_file_content" src/lib.rs` returns at least 1
- `grep "use opendeviationbar_providers::binance::checksum" src/binance_bindings.rs` succeeds
- `maturin develop` succeeds
- Python import of all three functions succeeds without AttributeError
- `compute_sha256_hex(b"hello world")` returns `"b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"`
- `verify_checksum_hex(b"hello world", correct_hash)` returns dict with keys `verified`, `expected`, `actual`, `skipped`
- `verify_checksum_hex` result has `verified=True, skipped=False` for matching hash
  </acceptance_criteria>
  <done>Three PyO3 checksum functions compiled, registered in_core module, importable from Python. compute_sha256_hex produces correct known hash. verify_checksum_hex returns dict with all four required keys and correct values.</done>
  </task>

<task type="auto">
  <name>Task 2: Create offline unit tests and network integration validation script</name>
  <files>tests/test_checksum_spike.py, scripts/spike_checksum_validation.py</files>
  <read_first>
    - src/binance_bindings.rs (the new functions from Task 1)
    - crates/opendeviationbar-providers/src/binance/checksum.rs (error handling strategy table at line 234-240)
    - tests/conftest.py (existing test patterns)
    - .planning/phases/08-checksum-spike/08-RESEARCH.md (CHECKSUM file format, pitfalls, test map)
  </read_first>
  <action>
**File 1: `tests/test_checksum_spike.py`** -- Offline unit tests (no network, runs in `mise run test-py`).

```python
"""Unit tests for PyO3 checksum bindings (Phase 8: Checksum Spike).

Offline tests -- no network calls. Validates SPIKE-01 (correct results)
and SPIKE-03 (edge cases) for the standalone binding functions.
"""
import pytest
from opendeviationbar._core import (
    compute_sha256_hex,
    verify_checksum_hex,
    parse_checksum_file_content,
)
```

Tests to include:

1. `test_compute_sha256_hex_known_hash` -- `compute_sha256_hex(b"hello world")` == `"b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9"`
2. `test_compute_sha256_hex_empty` -- `compute_sha256_hex(b"")` == `"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"`
3. `test_compute_sha256_hex_large_data` -- 10MB of `b"\x00"`, assert result is 64 chars of hex
4. `test_verify_checksum_hex_match` -- data=`b"test data"`, expected=`compute_sha256_hex(b"test data")`, assert result dict has `verified=True, skipped=False`, and all 4 keys present (`verified`, `expected`, `actual`, `skipped`)
5. `test_verify_checksum_hex_mismatch` -- data=`b"test data"`, expected=`"a" * 64`, assert `pytest.raises(RuntimeError, match="mismatch")`
6. `test_verify_checksum_hex_case_insensitive` -- uppercase expected hash still matches
7. `test_parse_checksum_file_content_valid` -- `"9572fa65...  BTCUSDT-aggTrades-2024-01-01.zip"` returns the hash
8. `test_parse_checksum_file_content_with_newline` -- same content + `\n` still works
9. `test_parse_checksum_file_content_empty` -- `pytest.raises(ValueError)`
10. `test_parse_checksum_file_content_invalid_length` -- short hash raises `ValueError`
11. `test_parse_checksum_file_content_non_hex` -- `"g" + "0" * 63 + "  file.zip"` raises `ValueError`

**File 2: `scripts/spike_checksum_validation.py`** -- Network integration script.

This script validates SPIKE-01, SPIKE-02, SPIKE-03 against real Binance Vision data. NOT in `tests/` (would break offline CI). Run manually: `uv run --python 3.13 python scripts/spike_checksum_validation.py`

Structure:

```python
#!/usr/bin/env python3
"""Checksum Spike Validation (Phase 8) -- SPIKE-01, SPIKE-02, SPIKE-03.

Downloads real Binance Vision data and validates SHA-256 checksum verification
end-to-end. Requires network access.

Usage:
    uv run --python 3.13 python scripts/spike_checksum_validation.py
"""
import requests
import time
import sys
from opendeviationbar._core import (
    compute_sha256_hex,
    verify_checksum_hex,
    parse_checksum_file_content,
    fetch_binance_aggtrades,
)
```

**Section 0: Pre-flight -- confirm verify_checksum kwarg is callable**

Before running SPIKE-02, confirm the `verify_checksum` kwarg on `fetch_binance_aggtrades` is accessible from Python. This is a runtime gate -- if the kwarg does not exist, SPIKE-02 cannot run.

```python
import inspect
sig = inspect.signature(fetch_binance_aggtrades)
assert "verify_checksum" in sig.parameters, (
    "FATAL: fetch_binance_aggtrades missing verify_checksum kwarg -- "
    "SPIKE-02 benchmark cannot proceed"
)
print("Pre-flight OK: verify_checksum kwarg confirmed on fetch_binance_aggtrades")
```

**Section 1: SPIKE-01 -- Standalone binding correctness**

- Download `https://data.binance.vision/data/spot/daily/aggTrades/BTCUSDT/BTCUSDT-aggTrades-2024-01-01.zip` via `requests.get()`
- Download `{url}.CHECKSUM` via `requests.get()`
- Parse CHECKSUM content with `parse_checksum_file_content()`
- Compute hash of ZIP bytes with `compute_sha256_hex()`
- Verify match with `verify_checksum_hex(zip_bytes, expected_hash)`
- Assert result dict has `verified=True`
- Print: `"SPIKE-01 PASS: SHA-256 verified for BTCUSDT 2024-01-01 ({len(zip_bytes)} bytes)"`

**Section 2: SPIKE-02 -- Performance overhead**

- Call `fetch_binance_aggtrades("BTCUSDT", "2024-01-02", "2024-01-02", verify_checksum=False)` -- time it
- Call `fetch_binance_aggtrades("BTCUSDT", "2024-01-02", "2024-01-02", verify_checksum=True)` -- time it
- Compute overhead_pct = `(time_with - time_without) / time_without * 100`
- Assert overhead_pct < 10.0 (the requirement)
- Print: `"SPIKE-02 PASS: Checksum overhead = {overhead_pct:.1f}% (< 10% required)"`
- Note: Run each 3 times and take median to reduce variance

**Section 3: SPIKE-03 -- Edge cases (all 4 cases)**

- **3a: 404 soft-skip** -- Call `fetch_binance_aggtrades("BTCUSDT", "2019-01-01", "2019-01-01", verify_checksum=True)`. If CHECKSUM file is 404, the integrated path should succeed (soft-skip). Print PASS if no exception raised.
- **3b: Mismatch hard fail** -- Take the real ZIP bytes from Section 1, call `verify_checksum_hex(zip_bytes, "a" * 64)`. Assert `RuntimeError` with "mismatch" in message.
- **3c: Invalid format hard fail** -- Call `parse_checksum_file_content("not a valid checksum file")`. Assert `ValueError` raised.
- **3d: Network timeout soft-skip** -- Call `fetch_binance_aggtrades("BTCUSDT", "2024-01-01", "2024-01-01", verify_checksum=True)` with a very short timeout scenario. Since the Rust `fetch_and_verify` returns `ChecksumResult::skipped()` on network errors (FetchFailed), verify the path does not crash. Concretely: download the ZIP bytes from SPIKE-01 and simulate the timeout path by calling `verify_checksum_hex(zip_bytes, compute_sha256_hex(zip_bytes))` (confirming the non-error path), then assert that `fetch_binance_aggtrades` with `verify_checksum=True` on a known-good date succeeds (proving the integrated path handles any transient CHECKSUM fetch failure as soft-skip). Assert result is not empty and `skipped` behavior is correct (no crash). Print PASS.
- Print: `"SPIKE-03 PASS: All 4 edge cases handled correctly (404=soft-skip, mismatch=hard-fail, invalid=hard-fail, timeout=soft-skip)"`

**Final summary:**

```
============================================================
CHECKSUM SPIKE VALIDATION COMPLETE
  SPIKE-01: PASS (real data SHA-256 verified)
  SPIKE-02: PASS (overhead X.X%)
  SPIKE-03: PASS (404=soft-skip, mismatch=hard-fail, invalid=hard-fail, timeout=soft-skip)
============================================================
```

Exit 0 on all pass, exit 1 on any failure.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_checksum_spike.py -x -q 2>&1 | tail -5</automated>
</verify>
<acceptance_criteria>

- `grep -c "def test_" tests/test_checksum_spike.py` returns at least 11
- `grep "SPIKE-01" scripts/spike_checksum_validation.py` succeeds
- `grep "SPIKE-02" scripts/spike_checksum_validation.py` succeeds
- `grep "SPIKE-03" scripts/spike_checksum_validation.py` succeeds
- `grep "compute_sha256_hex" tests/test_checksum_spike.py` succeeds
- `grep "verify_checksum_hex" tests/test_checksum_spike.py` succeeds
- `grep "parse_checksum_file_content" tests/test_checksum_spike.py` succeeds
- `uv run --python 3.13 pytest tests/test_checksum_spike.py -x -q` passes all tests
- `grep "sys.exit" scripts/spike_checksum_validation.py` succeeds (script exits with status code)
- `grep "fetch_binance_aggtrades" scripts/spike_checksum_validation.py` succeeds (uses integrated path for SPIKE-02)
- `grep "verify_checksum.*sig.parameters" scripts/spike_checksum_validation.py` or equivalent pre-flight check present
- `grep "3d\|timeout\|soft.skip" scripts/spike_checksum_validation.py` confirms timeout edge case exists
  </acceptance_criteria>
  <done>All 11+ offline unit tests pass. Network validation script exists with SPIKE-01/02/03 sections covering all 4 edge cases (including timeout soft-skip) and exits 0/1 based on results. Pre-flight check confirms verify_checksum kwarg availability before SPIKE-02 benchmark runs.</done>
  </task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 3: User validates spike results with real network data</name>
  <what-built>Rust SHA-256 checksum PyO3 bindings (compute_sha256_hex, verify_checksum_hex, parse_checksum_file_content) with offline unit tests and a network integration validation script proving real-data verification, performance overhead measurement, and edge case handling.</what-built>
  <how-to-verify>
    1. Confirm offline tests pass: `uv run --python 3.13 pytest tests/test_checksum_spike.py -x -v`
    2. Run the network integration script: `uv run --python 3.13 python scripts/spike_checksum_validation.py`
    3. Verify output shows:
       - Pre-flight OK: verify_checksum kwarg confirmed
       - SPIKE-01 PASS (real Vision data SHA-256 verified)
       - SPIKE-02 PASS (overhead < 10%)
       - SPIKE-03 PASS (all 4 edge cases: 404 soft-skip, mismatch hard-fail, invalid-format hard-fail, timeout soft-skip)
    4. Script exits with code 0
  </how-to-verify>
  <resume-signal>Type "approved" if all three SPIKEs pass, or describe any failures</resume-signal>
  <files>scripts/spike_checksum_validation.py</files>
  <action>User runs the network integration script and visually confirms all three SPIKE requirements pass.</action>
  <verify>
    <automated>uv run --python 3.13 python scripts/spike_checksum_validation.py</automated>
  </verify>
  <done>All SPIKE-01, SPIKE-02, SPIKE-03 validated by user against real Binance Vision data.</done>
</task>

</tasks>

<verification>
1. `maturin develop` succeeds (Rust compiles)
2. `uv run --python 3.13 pytest tests/test_checksum_spike.py -x -q` -- all offline tests pass
3. `uv run --python 3.13 python scripts/spike_checksum_validation.py` -- all three SPIKE requirements validated against real Binance data
4. `mise run test-py` -- existing test suite still passes (no regressions)
</verification>

<success_criteria>

- SPIKE-01: `compute_sha256_hex()`, `verify_checksum_hex()`, `parse_checksum_file_content()` callable from Python, correct results against real Vision CHECKSUM file
- SPIKE-02: Checksum overhead measured < 10% wall-clock increase on `fetch_binance_aggtrades()` with `verify_checksum=True` vs `False` (kwarg pre-flight confirmed before benchmark)
- SPIKE-03: 404 CHECKSUM (old data 2019-01-01) = soft-skip; corrupted hash = RuntimeError with "mismatch"; invalid format = ValueError; network timeout = soft-skip
  </success_criteria>

<output>
After completion, create `.planning/phases/08-checksum-spike/08-01-SUMMARY.md`
</output>
