---
phase: 11-integrity-sweep
plan: 01
subsystem: scripts
tags: [sha256, checksum, binance-vision, parquet, integrity, sweep]

requires:
  - phase: 08-checksum-spike
    provides: Rust SHA-256 compute_sha256_hex and parse_checksum_file_content bindings
  - phase: 09-checksum-registry
    provides: ChecksumRegistry for tracking verification state
provides:
  - Integrity sweep CLI script for retroactive Parquet cache auditing
  - Incremental and full verification modes
  - Per-symbol pass/fail/skip summary reporting
affects: [heartbeat, deploy]

tech-stack:
  added: []
  patterns:
    [
      concurrent ThreadPoolExecutor verification,
      ticks_dir scanning for file discovery,
    ]

key-files:
  created:
    - scripts/integrity_sweep.py
    - tests/test_integrity_sweep.py
  modified: []

key-decisions:
  - "Sweep downloads original Vision ZIPs (not local Parquet) because CHECKSUMs are computed against ZIP bytes"
  - "Default 8 workers (network-bound, lower than seeder's 12 since sweep re-downloads full ZIPs)"
  - "Batch registry.save() after all workers complete (same pattern as seeder)"

patterns-established:
  - "ticks_dir scanning pattern: iterate subdirectories, extract base symbol via prefix regex"
  - "_build_work_items / _verify_one_day / _aggregate_results separation for testability"

requirements-completed: [SWEEP-01, SWEEP-02, SWEEP-03]

duration: 4min
completed: 2026-03-26
---

# Phase 11 Plan 01: Integrity Sweep Summary

**Retroactive Parquet cache auditing via Vision ZIP SHA-256 verification with incremental registry tracking and concurrent downloads**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-26T05:46:07Z
- **Completed:** 2026-03-26T05:49:47Z
- **Tasks:** 1 (TDD: RED + GREEN)
- **Files modified:** 2

## Accomplishments

- Integrity sweep script audits all Parquet cache files against Binance Vision CHECKSUM files (SWEEP-01)
- Incremental mode skips already-verified files via ChecksumRegistry (SWEEP-02)
- Summary report with per-symbol pass/fail/skip counts and optional Telegram notification (SWEEP-03)
- 13 unit tests covering work item filtering, verification outcomes, and result aggregation

## Task Commits

Each task was committed atomically:

1. **Task 1 (RED): Failing tests** - `2c45ae7` (test)
2. **Task 1 (GREEN): Integrity sweep implementation** - `e694949` (feat)

## Files Created/Modified

- `scripts/integrity_sweep.py` - CLI script: argparse, concurrent verification, registry integration, Telegram summary (487 lines)
- `tests/test_integrity_sweep.py` - Unit tests: 13 tests covering build_work_items, verify_one_day, aggregate_results (319 lines)

## Decisions Made

- Sweep downloads original Vision ZIPs (not local Parquet) because CHECKSUMs are computed against ZIP bytes, not the extracted/processed Parquet data
- Default 8 workers for thread pool (network-bound, lower than seeder's 12 because sweep re-downloads full ZIPs which are larger)
- Batch registry.save() after all workers complete to reduce I/O (same pattern as seeder)
- ticks_dir scanning with prefix regex to handle both "BTCUSDT" and "BINANCE_SPOT_BTCUSDT" directory naming

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None.

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Sweep script ready for deployment to bigblack
- Can run `--dry-run` to see scope before committing to full verification
- Registry integration enables future heartbeat reporting on verification coverage

---

_Phase: 11-integrity-sweep_
_Completed: 2026-03-26_
