---
phase: 04-microsecond-data-layer
verified: 2026-03-24T05:36:28Z
status: gaps_found
score: 5/6 must-haves verified
gaps:
  - truth: "All pre-2025 Parquet files contain 16-digit microsecond timestamps after reseeding"
    status: partial
    reason: "The reseed script (scripts/reseed_parquet_to_microseconds.py) is fully implemented and verified correct, but the actual migration on bigblack has not been executed. SUMMARY.md explicitly states 'Reseed script ready to execute on bigblack' under Next Phase Readiness. Pre-2025 Parquet files on bigblack still contain 13-digit ms timestamps."
    artifacts:
      - path: "scripts/reseed_parquet_to_microseconds.py"
        issue: "Script exists and is correct — but USEC-02 requires pre-2025 files to actually contain us timestamps, not just for the migration script to exist"
    missing:
      - "Run 'python scripts/reseed_parquet_to_microseconds.py --dry-run' on bigblack to confirm scope"
      - "Run 'python scripts/reseed_parquet_to_microseconds.py' on bigblack to execute migration"
      - "Run 'python scripts/reseed_parquet_to_microseconds.py --verify --symbol BTCUSDT' on bigblack to validate"
      - "Update REQUIREMENTS.md traceability table: USEC-02 row from 'Pending/TBD' to 'Complete/02'"
human_verification:
  - test: "Execute reseed migration on bigblack and confirm pre-2025 Parquet files show 16-digit timestamps"
    expected: "All YYYY-MM-DD.parquet files before 2025-01-01 have max(timestamp) >= 1e13 after migration"
    why_human: "Requires SSH access to bigblack and execution of the migration script against the live tick cache"
---

# Phase 4: Microsecond Data Layer Verification Report

**Phase Goal:** All tick data and bar timestamps use native microsecond precision from Rust through Parquet
**Verified:** 2026-03-24T05:36:28Z
**Status:** gaps_found
**Re-verification:** No — initial verification

## Goal Achievement

### Observable Truths

| #   | Truth                                                                                                  | Status     | Evidence                                                                                                                                                                                 |
| --- | ------------------------------------------------------------------------------------------------------ | ---------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1   | Binance seeder writes 16-digit microsecond timestamps to Parquet for post-2025-01-01 data              | ✓ VERIFIED | `tick_cache_seeder.py:230-232` — ms\*1000 upscale for pre-2025, pass-through for post-2025 us data (no `// 1000` truncation remains)                                                     |
| 2   | TickStorage write_ticks() preserves microsecond timestamps without truncating to milliseconds          | ✓ VERIFIED | `parquet.py:175` — `dt.epoch(time_unit="us")`, `parquet.py:238` — `Datetime(time_unit="us")` cast; `_normalize_timestamps` upscales ms\*1000                                             |
| 3   | TickStorage read_ticks() auto-detects caller timestamp unit and filters correctly for both ms and us   | ✓ VERIFIED | `parquet.py:377-380` — ms filter values auto-multiplied \*1000 before predicate; same in `parquet_query.py:179-182` for streaming path                                                   |
| 4   | repair_direct_parquet.py and orchestration helpers use process_trades_arrow_native for us timestamps   | ✓ VERIFIED | `repair_direct_parquet.py:296` — `process_trades_arrow_native`; `helpers.py:319-322` — `use_native_arrow=True` path calls native; `helpers.py:489` — called with `use_native_arrow=True` |
| 5   | Existing consumers (kintsugi, sidecar) continue to work with ms-based API calls against us-stored data | ✓ VERIFIED | Auto-detect in `read_ticks()`/`read_ticks_streaming()` converts ms filter values to us; `_timestamp_to_day` auto-detects both units                                                      |
| 6   | All pre-2025 Parquet files contain 16-digit microsecond timestamps after reseeding (USEC-02)           | ✗ FAILED   | Script `reseed_parquet_to_microseconds.py` exists and is correct, but migration has NOT been run on bigblack — SUMMARY confirms "script ready to execute on bigblack"                    |

**Score:** 5/6 truths verified

### Required Artifacts

| Artifact                                           | Expected                                               | Status     | Details                                                                                                                                      |
| -------------------------------------------------- | ------------------------------------------------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| `python/opendeviationbar/storage/parquet.py`       | TickStorage with us-native storage, auto-detect filter | ✓ VERIFIED | `_normalize_timestamps` targets us; `write_ticks` uses `Datetime(time_unit="us")`; read auto-normalizes ms→us                                |
| `python/opendeviationbar/storage/parquet_query.py` | Timestamp-to-day conversion handles both ms and us     | ✓ VERIFIED | `_timestamp_to_day` auto-detects at `_US_MS_THRESHOLD=1e13`; `_timestamp_to_year_month` same; streaming path patched                         |
| `scripts/tick_cache_seeder.py`                     | Seeder passes through native us timestamps from Vision | ✓ VERIFIED | `// 1000` truncation removed; replaced with `* 1000` upscale for ms data                                                                     |
| `python/opendeviationbar/orchestration/helpers.py` | Arrow processing with us-aware flag                    | ✓ VERIFIED | `use_native_arrow` param wired; `process_trades_arrow_native` called when `True`; `open_time_ms` CH conversion preserved                     |
| `scripts/repair_direct_parquet.py`                 | Repair script using process_trades_arrow_native        | ✓ VERIFIED | Line 296: `process_trades_arrow_native`; comment updated to reflect µs expectation                                                           |
| `scripts/reseed_parquet_to_microseconds.py`        | One-off migration script for ms→us Parquet reseed      | ✓ VERIFIED | File exists; `--dry-run`, `--symbol`, `--verify` flags present; `_US_MS_THRESHOLD` detection; `*1000` upscale; atomic write via `os.replace` |

### Key Link Verification

| From                                         | To                                                 | Via                                                | Status      | Details                                                                                                                                                                                                                                                                |
| -------------------------------------------- | -------------------------------------------------- | -------------------------------------------------- | ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `scripts/tick_cache_seeder.py`               | `python/opendeviationbar/storage/parquet.py`       | `write_ticks()` without ms truncation              | ✓ WIRED     | `storage.write_ticks(cache_sym, df)` at lines 469, 549; `_normalize_timestamps` handles us                                                                                                                                                                             |
| `python/opendeviationbar/storage/parquet.py` | `python/opendeviationbar/storage/parquet_query.py` | `_timestamp_to_day` auto-detecting ms vs us        | ✓ WIRED     | `parquet.py:40` imports `_timestamp_to_day`; `parquet.py:144` delegates to parquet_query                                                                                                                                                                               |
| `scripts/repair_direct_parquet.py`           | `src/core_bindings.rs`                             | `process_trades_arrow` with us-aware flag          | ✓ WIRED     | `repair_direct_parquet.py:296` calls `process_trades_arrow_native`; `core_bindings.rs:501-505` passes `timestamp_is_microseconds=true` to `process_from_arrow_columns`                                                                                                 |
| `scripts/reseed_parquet_to_microseconds.py`  | `python/opendeviationbar/storage/parquet.py`       | TickStorage read/write for atomic file replacement | ⚠️ DEVIATED | Script intentionally uses direct file I/O (not `TickStorage.write_ticks()`) to avoid dedup/partitioning overhead; imports `parquet_io.py` constants (`COMPRESSION`, `get_cache_dir`). This is a deliberate design decision documented in plan and summary — not a bug. |

### Data-Flow Trace (Level 4)

| Artifact                                           | Data Variable         | Source                                      | Produces Real Data | Status    |
| -------------------------------------------------- | --------------------- | ------------------------------------------- | ------------------ | --------- |
| `python/opendeviationbar/orchestration/helpers.py` | `bars_arrow` (output) | `process_trades_arrow_native(chunk_arrow)`  | Yes (Rust)         | ✓ FLOWING |
| `scripts/repair_direct_parquet.py`                 | `bars_arrow` (output) | `process_trades_arrow_native(trades_batch)` | Yes (Rust)         | ✓ FLOWING |
| `scripts/tick_cache_seeder.py`                     | `df["timestamp"]`     | Vision CSV (us passthrough, ms upscaled)    | Yes (file read)    | ✓ FLOWING |

### Behavioral Spot-Checks

| Behavior                                         | Command                                                                                | Result                                                       | Status |
| ------------------------------------------------ | -------------------------------------------------------------------------------------- | ------------------------------------------------------------ | ------ |
| `_timestamp_to_day` auto-detects ms input        | `uv run --python 3.13 python -c "_timestamp_to_day(1736899200000) == '2025-01-15'"`    | PASS                                                         | ✓ PASS |
| `_timestamp_to_day` auto-detects us input        | `uv run --python 3.13 python -c "_timestamp_to_day(1736899200000000) == '2025-01-15'"` | PASS                                                         | ✓ PASS |
| `_normalize_timestamps` upscales ms to us        | `uv run --python 3.13 python -c "result['timestamp'][0] > 1e13"`                       | PASS (1736899200000000)                                      | ✓ PASS |
| `_normalize_timestamps` preserves us passthrough | `uv run --python 3.13 python -c "result['timestamp'][0] == 1736899200000000"`          | PASS                                                         | ✓ PASS |
| Seeder `// 1000` truncation removed              | `grep -c "// 1000" tick_cache_seeder.py` for timestamp context                         | 0 occurrences near timestamp handling (only in day_start_ms) | ✓ PASS |
| `repair_direct_parquet.py` uses native path      | `grep -c "process_trades_arrow_native" scripts/repair_direct_parquet.py`               | 3 occurrences                                                | ✓ PASS |
| `reseed_parquet_to_microseconds.py` syntax valid | `python -c "ast.parse(open(...).read())"`                                              | SYNTAX OK                                                    | ✓ PASS |
| Commit hashes from SUMMARY.md all exist          | `git log --oneline 64f4f6d cf38be2 e440fc1 02777f1`                                    | All 4 commits verified                                       | ✓ PASS |

### Requirements Coverage

| Requirement | Source Plan | Description                                                                            | Status      | Evidence                                                                                                                                                                                                                |
| ----------- | ----------- | -------------------------------------------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| USEC-01     | 04-01       | Stop truncating µs to ms in seeder — Parquet tick cache stores native us for post-2025 | ✓ SATISFIED | `tick_cache_seeder.py:226-232` — old `// 1000` block removed, replaced with `* 1000` ms upscale; `write_ticks` stores us                                                                                                |
| USEC-02     | 04-02       | Reseed pre-2025 Parquet files — upscale ms timestamps to µs (×1000)                    | ✗ PARTIAL   | Script exists and is correct (commits `e440fc1`, `02777f1`), but actual migration NOT run on bigblack. Traceability table still shows "Pending/TBD" — needs update after migration.                                     |
| USEC-04     | 04-01       | Rust OpenDeviationBarProcessor accepts and outputs µs timestamps throughout            | ✓ SATISFIED | `core_bindings.rs:492-505` — `process_trades_arrow_native` passes `timestamp_is_microseconds=true`; `helpers.rs:172` — multiplier=1 when flag is true; `processor.rs:54` — `last_timestamp_us` native µs internal state |

**Note on USEC-02 status inconsistency:** `REQUIREMENTS.md` line 19 shows `[x] USEC-02` (checkbox marked), but the traceability table (line 48) shows `Pending/TBD`. The checkbox appears to have been marked when 04-02-SUMMARY.md was written, but the table was not updated. The actual state is: script complete, migration unexecuted.

**Orphaned requirements check:** No additional USEC requirements are mapped to Phase 4 in REQUIREMENTS.md beyond USEC-01, USEC-02, USEC-04. USEC-03, USEC-05, USEC-06 are correctly mapped to Phase 5.

### Anti-Patterns Found

| File                                               | Line    | Pattern                                                             | Severity | Impact                                                                                   |
| -------------------------------------------------- | ------- | ------------------------------------------------------------------- | -------- | ---------------------------------------------------------------------------------------- |
| `python/opendeviationbar/orchestration/helpers.py` | 734-735 | `TODO(Issue #59): inter-bar feature for Exness not yet implemented` | ℹ️ Info  | Pre-existing note, unrelated to phase 4 microsecond work. Does not affect us timestamps. |

No blocker anti-patterns found in phase 4 modified files.

### Human Verification Required

#### 1. Execute Parquet reseed on bigblack

**Test:** SSH to bigblack, activate venv, run dry-run then migration:

```bash
python scripts/reseed_parquet_to_microseconds.py --dry-run
python scripts/reseed_parquet_to_microseconds.py
python scripts/reseed_parquet_to_microseconds.py --verify --symbol BTCUSDT
```

**Expected:** Dry-run reports all pre-2025 files needing reseed; migration completes with "reseeded ms->us" for each pre-2025 file; verify mode reports all timestamps >= 1e13 and Stathera OK for BTCUSDT test days.
**Why human:** Requires SSH access to bigblack and execution against the live tick cache (local dev machine has no pre-2025 Parquet files to migrate).

#### 2. Update REQUIREMENTS.md traceability table

**Test:** After successful migration, update `REQUIREMENTS.md` line 48: `USEC-02 | 4 | TBD | Pending` → `USEC-02 | 4 | 02 | Complete`
**Expected:** Traceability table consistent with checkbox and SUMMARY state.
**Why human:** Minor clerical update, requires confirming migration succeeded first.

### Gaps Summary

One gap blocks full phase goal achievement:

**USEC-02 migration unexecuted.** The phase goal states "All tick data and bar timestamps use native microsecond precision from Rust through Parquet." For pre-2025 data, this is only true after running the reseed migration on bigblack. The tool exists and is correct — `scripts/reseed_parquet_to_microseconds.py` with all required capabilities (`--dry-run`, `--symbol`, `--verify`, `_US_MS_THRESHOLD` detection, atomic writes). The remaining work is one SSH session on bigblack.

All other success criteria from the ROADMAP are fully satisfied:

- SC1 (seeder writes 16-digit us for post-2025): VERIFIED
- SC3 (Rust processor accepts and outputs us throughout): VERIFIED
- SC4 (bars from reseeded data pass Stathera validation): Script implements validation — executable after migration

The gap is operational (run the migration), not a code defect.

---

_Verified: 2026-03-24T05:36:28Z_
_Verifier: Claude (gsd-verifier)_
