---
phase: 04-microsecond-data-layer
plan: 02
subsystem: storage
tags: [parquet, microsecond, migration, stathera, arrow]

requires:
  - phase: 04-microsecond-data-layer/01
    provides: "_US_MS_THRESHOLD constant and us-native TickStorage semantics"
provides:
  - "One-off Parquet reseed script (ms->us) for all pre-2025 tick cache files"
  - "Verification mode validating reseeded data produces correct bars with Stathera continuity"
affects: [04-microsecond-data-layer]

tech-stack:
  added: []
  patterns:
    [
      "Atomic Parquet file replacement via tempfile+fsync+os.replace",
      "us-native Arrow pipeline verification via process_trades_arrow_native",
    ]

key-files:
  created:
    - scripts/reseed_parquet_to_microseconds.py
  modified: []

key-decisions:
  - "Direct file-level operation (read->transform->write) instead of TickStorage.write_ticks() to avoid dedup/partitioning overhead"
  - "process_trades_arrow_native() for us-native verification with fallback to ms path for compatibility"

patterns-established:
  - "Parquet reseed: detect ms vs us via _US_MS_THRESHOLD (1e13), multiply by 1000, atomic write"
  - "Verification: us-native Arrow path -> Rust processor -> Stathera intra-day + cross-day check"

requirements-completed: [USEC-02]

duration: 3min
completed: 2026-03-24
---

# Phase 04 Plan 02: Parquet Tick Cache Reseed (ms to us) Summary

**One-off migration script reseeding all pre-2025 Parquet tick files from ms to us timestamps, with --verify mode validating Stathera continuity through the us-native Rust Arrow pipeline**

## Performance

- **Duration:** 3 min
- **Started:** 2026-03-24T05:28:21Z
- **Completed:** 2026-03-24T05:31:58Z
- **Tasks:** 2
- **Files modified:** 1

## Accomplishments

- Created `scripts/reseed_parquet_to_microseconds.py` with --dry-run, --symbol, --verify flags
- Script detects ms vs us timestamps via `_US_MS_THRESHOLD` (1e13) and upscales ms \* 1000
- Atomic file replacement via tempfile + fsync + os.replace (crash-safe)
- Verification mode uses `process_trades_arrow_native()` for us-native pipeline validation
- Cross-day Stathera continuity verification at the 2024-12-31/2025-01-01 boundary

## Task Commits

Each task was committed atomically:

1. **Task 1: Create reseed script and migrate all pre-2025 Parquet files to microseconds** - `e440fc1` (feat)
2. **Task 2: Verify reseeded data produces valid bars with Stathera continuity** - `02777f1` (feat)

## Files Created/Modified

- `scripts/reseed_parquet_to_microseconds.py` - One-off migration script: reseed ms->us, --dry-run, --verify with us-native Arrow path

## Decisions Made

- Used direct file I/O (read_parquet -> transform -> atomic_write) instead of TickStorage.write_ticks() to avoid triggering dedup logic, day partitioning, and timestamp normalization
- Used `process_trades_arrow_native()` for verification (accepts us timestamps directly) with fallback to ms-converting `process_trades_arrow()` for build compatibility
- Idempotent design: files already at us precision are skipped, safe to re-run

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all functionality is complete and wired.

## Issues Encountered

None

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Reseed script ready to execute on bigblack (`python scripts/reseed_parquet_to_microseconds.py`)
- Recommend --dry-run first, then execute, then --verify to confirm
- Depends on Plan 01 (TickStorage us-native changes) being deployed for full pipeline coherence

## Self-Check: PASSED

- FOUND: scripts/reseed_parquet_to_microseconds.py
- FOUND: commit e440fc1 (Task 1)
- FOUND: commit 02777f1 (Task 2)
- FOUND: 04-02-SUMMARY.md

---

_Phase: 04-microsecond-data-layer_
_Completed: 2026-03-24_
