# Phase 8: Checksum Spike - Research

**Researched:** 2026-03-25
**Domain:** Rust-to-Python SHA-256 checksum verification for Binance Vision data
**Confidence:** HIGH

## Summary

The Rust checksum module (`crates/opendeviationbar-providers/src/binance/checksum.rs`) is fully implemented (402 lines, 13 unit tests) and already integrated into the Rust-side Vision download pipeline. The critical finding is that **checksum verification is already exposed to Python** via the `verify_checksum` parameter on `fetch_binance_aggtrades()`, `stream_binance_trades()`, and `stream_binance_trades_arrow()` in `src/binance_bindings.rs`. The existing PyO3 bindings pass `verify_checksum=true` by default through `HistoricalDataLoader.load_single_day_trades_with_checksum()` and `IntraDayChunkIterator.with_checksum()`.

However, the Python tick cache seeder (`scripts/tick_cache_seeder.py`) downloads Vision ZIPs via raw `requests.get()` -- completely bypassing the Rust checksum pipeline. The spike must prove that the Rust checksum functions work correctly against real Binance data when called from Python, measure performance overhead, and validate edge case handling. This validates the mechanism before Phase 10 wires it into the seeder.

**Primary recommendation:** Write a standalone Python test script that calls the existing Rust-via-PyO3 checksum path (e.g., `fetch_binance_aggtrades("BTCUSDT", "2024-01-01", "2024-01-01", verify_checksum=True)`) against real data, then also expose `compute_sha256()` and `parse_checksum_file()` as thin PyO3 bindings for the seeder's standalone verification use case. Measure wall-clock overhead by comparing `verify_checksum=True` vs `verify_checksum=False`.

<phase_requirements>

## Phase Requirements

| ID       | Description                                                                                                                              | Research Support                                                                                                                                                                                                                                                                     |
| -------- | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| SPIKE-01 | Rust checksum.rs SHA-256 fetch+verify callable from Python, correct results against real Vision CHECKSUM files                           | Existing PyO3 bindings already expose checksum via `verify_checksum=True` parameter on `fetch_binance_aggtrades()`. Need new thin bindings for standalone `compute_sha256()` + `verify_checksum()` for seeder use case. Real CHECKSUM file format confirmed: `<64-hex>  <filename>`. |
| SPIKE-02 | Performance overhead measured (< 10% wall-clock increase acceptable)                                                                     | Benchmark `fetch_binance_aggtrades()` with `verify_checksum=True` vs `False`. Checksum compute is pure SHA-256 on ~50-200MB ZIP; fetch is one extra HTTP GET for the 98-byte CHECKSUM file.                                                                                          |
| SPIKE-03 | Edge cases validated: 404 CHECKSUM (soft-skip), network timeout (soft-skip), format mismatch (hard fail), corrupted download (hard fail) | All four cases already implemented in Rust `fetch_and_verify()`. Spike must exercise each from Python.                                                                                                                                                                               |

</phase_requirements>

## Standard Stack

### Core (Already in Project)

| Library          | Version           | Purpose                       | Why Standard                    |
| ---------------- | ----------------- | ----------------------------- | ------------------------------- |
| `sha2` (Rust)    | In workspace      | SHA-256 computation           | Already used by `checksum.rs`   |
| `reqwest` (Rust) | In workspace      | HTTP fetch of .CHECKSUM files | Already used by providers crate |
| PyO3/maturin     | In workspace      | Rust-to-Python bridge         | Project standard                |
| `pytest`         | In pyproject.toml | Python test framework         | Project standard                |

### Supporting

| Library                | Version    | Purpose                                  | When to Use                                 |
| ---------------------- | ---------- | ---------------------------------------- | ------------------------------------------- |
| `time` (Python stdlib) | --         | Wall-clock benchmarking                  | SPIKE-02 performance measurement            |
| `requests` (Python)    | In project | Standalone CHECKSUM fetch for comparison | Verify Rust results match Python-side fetch |

### Alternatives Considered

None -- this spike validates existing infrastructure, not new library choices.

## Architecture Patterns

### Current Rust Checksum Architecture

```
checksum.rs (402 lines)
  compute_sha256(data: &[u8]) -> String         # Pure hash, sync
  parse_checksum_file(content: &str) -> String   # Parse "hash  filename" format
  fetch_checksum(client, data_url) -> String      # HTTP GET {url}.CHECKSUM, async
  verify_checksum(data, expected) -> ChecksumResult  # Compare hash, sync
  fetch_and_verify(client, data_url, data) -> ChecksumResult  # All-in-one, async

ChecksumResult { verified: bool, expected: Option<String>, actual: String, skipped: bool }
ChecksumError { Mismatch, FetchFailed, InvalidFormat, NotAvailable }
```

### Current PyO3 Exposure Path

```
Python: fetch_binance_aggtrades(symbol, start, end, verify_checksum=True)
  -> Rust: HistoricalDataLoader.load_single_day_trades_with_checksum(date, verify_checksum)
    -> Rust: download_day_zip(date, verify_checksum)
      -> Rust: checksum::fetch_and_verify(client, url, raw_bytes)
```

The checksum verification happens inside the download pipeline. There are **no standalone PyO3 bindings** for `compute_sha256()`, `verify_checksum()`, or `fetch_checksum()`.

### What the Spike Needs

1. **Prove existing path works** -- call `fetch_binance_aggtrades()` with `verify_checksum=True` on real data, confirm tracing logs show checksum verified
2. **Add thin standalone bindings** -- expose `compute_sha256(bytes) -> str` and `verify_checksum_standalone(data: bytes, expected: str) -> dict` for the seeder's use case (seeder downloads via `requests.get`, needs to verify separately)
3. **Benchmark overhead** -- time with/without checksum on same data
4. **Exercise edge cases** -- 404, timeout, mismatch, format error

### Recommended Pattern for New Bindings

Add to `src/binance_bindings.rs`:

```rust
/// Compute SHA-256 hash of raw data (Issue #43 spike).
#[pyfunction]
pub fn compute_sha256_hex(data: &[u8]) -> String {
    checksum::compute_sha256(data)
}

/// Verify data against expected SHA-256 hash (Issue #43 spike).
#[pyfunction]
pub fn verify_checksum_hex(py: Python, data: &[u8], expected: &str) -> PyResult<PyObject> {
    match checksum::verify_checksum(data, expected) {
        Ok(result) => {
            let dict = PyDict::new(py);
            dict.set_item("verified", result.verified)?;
            dict.set_item("expected", result.expected)?;
            dict.set_item("actual", result.actual)?;
            dict.set_item("skipped", result.skipped)?;
            Ok(dict.into_any().unbind())
        }
        Err(ChecksumError::Mismatch { expected, actual }) => {
            Err(PyRuntimeError::new_err(format!(
                "Checksum mismatch: expected {expected}, got {actual}"
            )))
        }
        Err(e) => Err(PyRuntimeError::new_err(e.to_string())),
    }
}

/// Parse Binance .CHECKSUM file content, returning the hex hash string.
#[pyfunction]
pub fn parse_checksum_file_content(content: &str) -> PyResult<String> {
    checksum::parse_checksum_file(content)
        .map_err(|e| PyValueError::new_err(e.to_string()))
}
```

Register in `lib.rs` under the `data-providers` feature gate.

### Anti-Patterns to Avoid

- **Do NOT reimplement SHA-256 in Python** -- always use the Rust `compute_sha256()` via PyO3.
- **Do NOT add a Python `hashlib.sha256` fallback** -- single code path, Rust only.
- **Do NOT test against mocked data only** -- SPIKE-01 explicitly requires real Vision CHECKSUM files.

## Don't Hand-Roll

| Problem                    | Don't Build               | Use Instead                                                                                      | Why                                                    |
| -------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------ |
| SHA-256 hashing            | Python `hashlib` wrapper  | Rust `sha2::Sha256` via PyO3                                                                     | Single code path, Rust-first principle                 |
| CHECKSUM file parsing      | Python string splitting   | Rust `parse_checksum_file()` via PyO3                                                            | Format validation + hex validation already implemented |
| HTTP download for CHECKSUM | New Python download logic | Rust `fetch_checksum()` (for integrated path) or simple `requests.get()` (for seeder comparison) | Rust path already handles timeouts and 404 gracefully  |

## Common Pitfalls

### Pitfall 1: Assuming Checksum Files Exist for All Dates

**What goes wrong:** CHECKSUM files may not exist for very old Vision data (pre-2020).
**Why it happens:** Binance added CHECKSUM files at some point, not retroactively for all data.
**How to avoid:** `fetch_and_verify()` already returns `ChecksumResult::skipped()` on 404. The spike should test a known-old date AND a known-recent date.
**Warning signs:** Test only passes on recent data.

### Pitfall 2: Network-Dependent Tests in CI

**What goes wrong:** Tests that fetch real Vision data fail in offline/CI environments.
**Why it happens:** SPIKE-01 requires real data, but regular `pytest` suite must be fast and offline.
**How to avoid:** Keep the spike test as a separate script (not in `tests/`), or mark with a custom marker like `@pytest.mark.network`. The mise task `test:checksum:integration` already exists as a pattern.
**Warning signs:** `mise run test-py` starts failing due to network timeouts.

### Pitfall 3: Measuring Download Time Instead of Checksum Time

**What goes wrong:** SPIKE-02 benchmark includes the full download + parse time, not isolating checksum overhead.
**Why it happens:** `fetch_binance_aggtrades()` does download+checksum+parse as one unit.
**How to avoid:** Measure two runs of the same date: one with `verify_checksum=True`, one with `verify_checksum=False`. The delta is the checksum overhead (HTTP GET for .CHECKSUM + SHA-256 compute).
**Warning signs:** Reported overhead is >50% (would indicate measuring something else).

### Pitfall 4: Forgetting to Rebuild After Rust Changes

**What goes wrong:** Python test imports stale bindings without the new checksum functions.
**Why it happens:** PyO3 bindings require `maturin develop` after any `src/*.rs` change.
**How to avoid:** Run `maturin develop` before running the spike test script.
**Warning signs:** `AttributeError: module 'opendeviationbar._core' has no attribute 'compute_sha256_hex'`.

## Code Examples

### Existing Integration Test Pattern (from mise task)

```python
# Source: .mise/tasks/checksum.toml (test:checksum:integration)
from opendeviationbar import get_open_deviation_bars
import logging
logging.basicConfig(level=logging.DEBUG)

# Test with checksum verification (should see checksum events in logs)
df = get_open_deviation_bars("BTCUSDT", "2024-01-01", "2024-01-01", verify_checksum=True)
print(f"Downloaded {len(df)} bars with checksum verification")
```

### Binance CHECKSUM File Format (verified 2026-03-25)

```
# URL: https://data.binance.vision/data/spot/daily/aggTrades/BTCUSDT/BTCUSDT-aggTrades-2024-01-01.zip.CHECKSUM
# Content (98 bytes):
9572fa650e2a7850e528fc6d0c3ec7874211af47947416f75004f0f95751dca3  BTCUSDT-aggTrades-2024-01-01.zip
```

Format: `<64-char lowercase SHA-256 hex>  <filename>` (two spaces between hash and filename).

### Rust Error Handling Strategy (from checksum.rs docstring)

```
| Scenario          | Behavior                      | Rationale                           |
|-------------------|-------------------------------|-------------------------------------|
| Checksum matches  | Returns Ok (verified=true)    | Success case                        |
| Checksum mismatch | Returns Err (hard error)      | Data corruption detected            |
| Checksum file 404 | Returns Ok (skipped=true)     | Old data may not have checksums     |
| Network timeout   | Returns Ok (skipped=true)     | Network issues shouldn't block      |
| Invalid format    | Returns Err (hard error)      | Indicates API change                |
```

### Seeder Download Path (currently unverified)

```python
# Source: scripts/tick_cache_seeder.py line 130-167
def _download_vision_day(symbol, date_str, *, timeout_s, market="spot"):
    url = _vision_url(symbol, date_str, market=market)
    resp = requests.get(url, timeout=timeout_s, allow_redirects=True)
    # ... no checksum verification ...
    with zipfile.ZipFile(io.BytesIO(resp.content)) as zf:
        # parse CSV directly
```

## State of the Art

| Old Approach                          | Current Approach                                           | When Changed                | Impact                                 |
| ------------------------------------- | ---------------------------------------------------------- | --------------------------- | -------------------------------------- |
| No checksum verification              | Rust-side `fetch_and_verify()` in Vision download pipeline | Issue #43 (already shipped) | Integrated path verifies by default    |
| Python-only Vision downloads (seeder) | Still using `requests.get()` without checksum              | Not yet changed             | Seeder is the gap -- Phase 10 will fix |

## Open Questions

1. **How old is the oldest CHECKSUM file on Binance Vision?**
   - What we know: 2024-01-01 has a CHECKSUM file (verified). The Rust code handles 404 gracefully.
   - What's unclear: Exact cutoff date where CHECKSUM files start existing.
   - Recommendation: Test a date from 2019 or earlier in the spike to confirm 404 soft-skip works.

2. **Should standalone bindings return `ChecksumResult` as a dict or a pyclass?**
   - What we know: Other bindings (rate limiter status) use dicts. `ChecksumResult` has 4 fields.
   - Recommendation: Use dict for simplicity -- spike is validation, not production API design. Phase 10 can refine.

## Validation Architecture

### Test Framework

| Property           | Value                                      |
| ------------------ | ------------------------------------------ |
| Framework          | pytest (in pyproject.toml) + cargo nextest |
| Config file        | `pyproject.toml [tool.pytest.ini_options]` |
| Quick run command  | `uv run --python 3.13 pytest tests/ -x -q` |
| Full suite command | `mise run check-full`                      |

### Phase Requirements -> Test Map

| Req ID   | Behavior                                            | Test Type             | Automated Command                                                                                  | File Exists? |
| -------- | --------------------------------------------------- | --------------------- | -------------------------------------------------------------------------------------------------- | ------------ |
| SPIKE-01 | Rust checksum callable from Python, correct results | integration (network) | `uv run --python 3.13 python scripts/spike_checksum_validation.py`                                 | No -- Wave 0 |
| SPIKE-02 | Performance overhead < 10%                          | integration (network) | Same script, benchmark section                                                                     | No -- Wave 0 |
| SPIKE-03 | Edge cases: 404, timeout, mismatch, format error    | unit + integration    | `uv run --python 3.13 pytest tests/test_checksum_spike.py -x` (unit) + validation script (network) | No -- Wave 0 |

### Sampling Rate

- **Per task commit:** `uv run --python 3.13 pytest tests/ -x -q`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green + spike validation script passes

### Wave 0 Gaps

- [ ] `tests/test_checksum_spike.py` -- unit tests for new PyO3 bindings (offline, no network)
- [ ] `scripts/spike_checksum_validation.py` -- integration test with real Binance data (SPIKE-01, SPIKE-02, SPIKE-03)
- [ ] New PyO3 bindings in `src/binance_bindings.rs`: `compute_sha256_hex()`, `verify_checksum_hex()`, `parse_checksum_file_content()`
- [ ] Register new functions in `src/lib.rs`
- [ ] `maturin develop` rebuild

## Project Constraints (from CLAUDE.md)

- **Python 3.13 ONLY** -- all `uv run` commands must use `--python 3.13`
- **Rust-first** -- always use Rust crates for heavy lifting; Python handles I/O only
- **`maturin develop`** required after any `src/*.rs` or `crates/**/*.rs` change
- **Spot-only data** -- all Binance data uses spot market exclusively
- **Local-first CI/CD** -- no GitHub Actions for testing; all quality gates run locally

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-providers/src/binance/checksum.rs` -- Full Rust implementation (402 lines, 13 tests)
- `src/binance_bindings.rs` -- Existing PyO3 bindings with `verify_checksum` parameter
- `crates/opendeviationbar-providers/src/binance/historical.rs` -- Integration of `fetch_and_verify()` in download pipeline
- `scripts/tick_cache_seeder.py` -- Seeder's unverified download path via `requests.get()`
- `https://data.binance.vision/data/spot/daily/aggTrades/BTCUSDT/BTCUSDT-aggTrades-2024-01-01.zip.CHECKSUM` -- Real CHECKSUM file format verified

### Secondary (MEDIUM confidence)

- `.mise/tasks/checksum.toml` -- Existing mise task for checksum testing

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - all libraries already in project, no new dependencies
- Architecture: HIGH - existing Rust module is complete, PyO3 pattern well-established in codebase
- Pitfalls: HIGH - edge cases already implemented in Rust, just need Python-side validation

**Research date:** 2026-03-25
**Valid until:** 2026-04-25 (stable domain, no external dependency changes expected)
