---
phase: 04-microsecond-data-layer
plan: 02
type: execute
wave: 2
depends_on: [04-01]
files_modified:
  - scripts/reseed_parquet_to_microseconds.py
autonomous: true
requirements: [USEC-02]

must_haves:
  truths:
    - "All pre-2025 Parquet files contain 16-digit microsecond timestamps after reseeding"
    - "All post-2025 Parquet files already have 16-digit microsecond timestamps (verified, not modified)"
    - "Bars produced from reseeded Parquet data pass Stathera continuity validation for at least one symbol+threshold pair"
    - "Reseeded data produces identical bars to original ms data (same OHLCV, same trade IDs, same bar count)"
  artifacts:
    - path: "scripts/reseed_parquet_to_microseconds.py"
      provides: "One-off migration script for ms→us Parquet reseed"
      contains: "reseed_symbol"
  key_links:
    - from: "scripts/reseed_parquet_to_microseconds.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "TickStorage read/write for atomic file replacement"
      pattern: "TickStorage"
---

<objective>
Reseed all pre-2025 Parquet tick cache files from millisecond to microsecond timestamps, then verify bars produced from reseeded data pass Stathera continuity validation.

Purpose: After Plan 01, new data is written in microseconds. But existing pre-2025 Parquet files still have millisecond timestamps. This plan creates uniform 16-digit microsecond precision across ALL dates.

Output: Reseed script, all Parquet files verified at microsecond precision, Stathera validation passing.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-microsecond-data-layer/04-CONTEXT.md
@.planning/phases/04-microsecond-data-layer/04-01-SUMMARY.md

@python/opendeviationbar/storage/parquet.py
@python/opendeviationbar/storage/parquet_query.py
@scripts/repair_direct_parquet.py

<interfaces>
<!-- From Plan 01 — TickStorage is now us-native -->

From python/opendeviationbar/storage/parquet.py (after Plan 01):

```python
class TickStorage:
    _normalize_timestamps(ticks, timestamp_col="timestamp") -> pl.DataFrame  # Now normalizes TO microseconds
    write_ticks(symbol, ticks, *, timestamp_col="timestamp") -> int  # Now stores us timestamps
    read_ticks(symbol, start_ts=None, end_ts=None) -> pl.DataFrame  # Returns us timestamps, auto-detects filter unit

_US_MS_THRESHOLD = 10_000_000_000_000  # 1e13 — values below are ms, above are us
```

From python/opendeviationbar/storage/parquet_query.py (after Plan 01):

```python
def _timestamp_to_day(timestamp: int) -> str  # Auto-detects ms vs us
def _get_parquet_path(ticks_dir, symbol, day_key) -> Path
def _get_symbol_dir(ticks_dir, symbol) -> Path
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create reseed script and migrate all pre-2025 Parquet files to microseconds</name>
  <files>scripts/reseed_parquet_to_microseconds.py</files>
  <read_first>
    python/opendeviationbar/storage/parquet.py
    python/opendeviationbar/storage/parquet_query.py
    python/opendeviationbar/storage/parquet_io.py
    scripts/tick_cache_seeder.py (for pattern reference — file discovery, atomic writes)
    python/opendeviationbar/symbol_registry.py (for get_registered_symbols)
  </read_first>
  <action>
    Create `scripts/reseed_parquet_to_microseconds.py` — a one-off migration script that walks all Parquet tick files and upscales ms timestamps to us.

    **Script design:**

    1. **Discovery**: Walk `~/.cache/opendeviationbar/ticks/` (via TickStorage's `_ticks_dir`). For each symbol directory, list all `YYYY-MM-DD.parquet` files.

    2. **Detection**: For each file, read the `timestamp` column max value. If `max_val < _US_MS_THRESHOLD` (1e13), the file has ms timestamps and needs reseeding. If `>= _US_MS_THRESHOLD`, already us — skip with log message.

    3. **Migration per file**:
       - Read the full DataFrame with `pl.read_parquet(file_path)`
       - Multiply timestamp column by 1000: `df.with_columns((pl.col("timestamp") * 1000).alias("timestamp"))`
       - Write atomically: write to `{file_path}.tmp`, then `os.replace(tmp, file_path)` (atomic on same filesystem)
       - Log: `"{symbol} {date}: reseeded ms→us ({n_rows} rows)"`

    4. **Safety**:
       - `--dry-run` flag: report what would be reseeded without modifying files
       - `--symbol SYMBOL` flag: restrict to a single symbol (for targeted runs)
       - Verify row count after write matches original (corruption guard)
       - Skip `.parquet.tmp` files (leftover from interrupted writes)

    5. **Summary output**: Print table showing per-symbol counts: `{symbol}: {reseeded}/{total} files reseeded, {skipped} already us`

    6. **Entrypoint**:
       ```python
       if __name__ == "__main__":
           import argparse
           parser = argparse.ArgumentParser(description="Reseed Parquet tick files from ms to us timestamps")
           parser.add_argument("--dry-run", action="store_true", help="Report only, no modifications")
           parser.add_argument("--symbol", type=str, help="Restrict to single symbol")
           args = parser.parse_args()
           reseed_all(dry_run=args.dry_run, symbol_filter=args.symbol)
       ```

    **Key constants** (import from parquet.py):
    - `_US_MS_THRESHOLD = 10_000_000_000_000` — the detection threshold
    - Use `COMPRESSION = "zstd"` and `COMPRESSION_LEVEL = 3` from parquet_io.py for write consistency

    **Do NOT use TickStorage.write_ticks()** for the reseed — it would trigger dedup logic, day partitioning, etc. This is a direct file-level operation: read → transform → atomic write. The schema and file naming stay identical.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "import ast; ast.parse(open('scripts/reseed_parquet_to_microseconds.py').read()); print('SYNTAX OK')" && grep -c "dry.run\|dry_run" scripts/reseed_parquet_to_microseconds.py | xargs -I{} test {} -gt 0 && echo "PASS: dry-run flag exists" || echo "FAIL: no dry-run"; grep -c "_US_MS_THRESHOLD\|1e13\|10_000_000_000_000" scripts/reseed_parquet_to_microseconds.py | xargs -I{} test {} -gt 0 && echo "PASS: threshold detection exists" || echo "FAIL: no threshold detection"; grep -c "\* 1000" scripts/reseed_parquet_to_microseconds.py | xargs -I{} test {} -gt 0 && echo "PASS: ms→us upscale exists" || echo "FAIL: no upscale"</automated>
  </verify>
  <acceptance_criteria>
    - scripts/reseed_parquet_to_microseconds.py exists and parses without syntax errors
    - grep: `--dry-run` flag in argparse setup
    - grep: `_US_MS_THRESHOLD` or equivalent threshold for ms/us detection
    - grep: `* 1000` for ms→us multiplication
    - grep: `os.replace` or equivalent for atomic file replacement
    - grep: `--symbol` flag for single-symbol restriction
    - Script has `if __name__ == "__main__"` entrypoint
  </acceptance_criteria>
  <done>Reseed script created, capable of migrating all pre-2025 Parquet files from ms to us timestamps with dry-run safety and atomic writes</done>
</task>

<task type="auto">
  <name>Task 2: Verify reseeded data produces valid bars with Stathera continuity</name>
  <files>scripts/reseed_parquet_to_microseconds.py</files>
  <read_first>
    scripts/reseed_parquet_to_microseconds.py (just created in Task 1)
    scripts/repair_direct_parquet.py (for pattern: process_trades_arrow_native + Stathera check)
    python/opendeviationbar/storage/parquet.py (read_ticks API)
  </read_first>
  <action>
    Add a `--verify` mode to the reseed script that validates reseeded data produces correct bars.

    **Verification logic** (add as `verify_reseeded_data()` function):

    1. Pick one symbol that has both pre-2025 and post-2025 data (BTCUSDT is the best candidate — has data from 2018).

    2. Select 3 test days: one pre-2025 (e.g., 2024-06-15), one post-2025 (e.g., 2025-06-15), and the boundary day (2024-12-31 or 2025-01-01).

    3. For each test day:
       a. Read the Parquet file via `pl.read_parquet()`
       b. Verify ALL timestamp values are >= `_US_MS_THRESHOLD` (i.e., 16-digit microseconds)
       c. Verify timestamp column dtype is `pl.Int64`
       d. Process through Rust via `process_trades_arrow_native()`:
          - Create a `PyOpenDeviationBarProcessor(threshold_decimal_bps=250)`
          - Convert DataFrame to Arrow RecordBatch
          - Call `processor.process_trades_arrow_native(batch)`
          - Verify bars are produced (non-empty result)
       e. Check Stathera within the day's bars: for consecutive bars, verify `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`

    4. Cross-day continuity: read two consecutive days (e.g., 2024-12-31 + 2025-01-01), process each independently with Ouroboros reset between them. Verify the last bar of day 1 and first bar of day 2 maintain Stathera continuity (trade IDs are contiguous).

    5. Print verification report:
       ```
       Verification Report:
         2024-06-15: {n_rows} ticks, all us, {n_bars} bars, Stathera OK
         2024-12-31: {n_rows} ticks, all us, {n_bars} bars, Stathera OK
         2025-01-01: {n_rows} ticks, all us, {n_bars} bars, Stathera OK
         Cross-day (2024-12-31 → 2025-01-01): Stathera OK
       ```

    6. Exit code: 0 if all checks pass, 1 if any fail.

    **Usage**: `python scripts/reseed_parquet_to_microseconds.py --verify --symbol BTCUSDT`

    **Note**: This verification requires Parquet files to exist locally (it reads from the tick cache). If files don't exist, skip with a warning message. The verification is designed to run on bigblack where the full cache is populated.

    **Import pattern**: Use `from opendeviationbar._core import PyOpenDeviationBarProcessor` for the Rust processor. Use `import pyarrow as pa` for Arrow conversion. Follow the same pattern as `repair_direct_parquet.py`.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -c "verify\|Stathera\|last_agg_trade_id\|first_agg_trade_id" scripts/reseed_parquet_to_microseconds.py | xargs -I{} test {} -gt 3 && echo "PASS: verification logic present" || echo "FAIL: insufficient verification logic"; grep -c "process_trades_arrow_native" scripts/reseed_parquet_to_microseconds.py | xargs -I{} test {} -gt 0 && echo "PASS: uses us-native arrow path" || echo "FAIL: missing arrow native path"</automated>
  </verify>
  <acceptance_criteria>
    - grep: `--verify` flag in argparse setup
    - grep: `process_trades_arrow_native` used for bar generation in verify mode
    - grep: `last_agg_trade_id` and `first_agg_trade_id` used for Stathera check
    - grep: `_US_MS_THRESHOLD` used to verify all timestamps are microseconds
    - Script exits 0 on verification success, 1 on failure
    - Cross-day continuity check spans the 2024-12-31/2025-01-01 boundary
  </acceptance_criteria>
  <done>Reseed script has --verify mode that validates reseeded data produces correct bars with Stathera continuity across pre-2025/post-2025 boundary</done>
</task>

</tasks>

<verification>
1. Reseed script exists with --dry-run, --symbol, --verify flags
2. Script correctly detects ms vs us timestamps in existing files
3. Script upscales ms→us via *1000 with atomic file replacement
4. --verify mode processes reseeded data through Rust and validates Stathera
5. Cross-day verification spans the ms→us boundary (2024-12-31 → 2025-01-01)
</verification>

<success_criteria>

- Reseed script can migrate all pre-2025 Parquet files from ms to us timestamps
- --dry-run reports changes without modifying files
- --verify confirms reseeded data produces valid bars with Stathera continuity
- Cross-day boundary between pre-2025 (was ms, now us) and post-2025 (native us) produces contiguous bars
- Script is safe to run multiple times (idempotent — skips files already in us)
  </success_criteria>

<output>
After completion, create `.planning/phases/04-microsecond-data-layer/04-02-SUMMARY.md`
</output>
