---
phase: 04-microsecond-data-layer
plan: 01
subsystem: storage
tags: [parquet, microsecond, tick-storage, arrow, timestamp]

# Dependency graph
requires:
  - phase: 03-kintsugi-oom-hardening
    provides: repair_direct_parquet.py Arrow pipeline, kintsugi Parquet-only reads
provides:
  - TickStorage stores microsecond timestamps natively (us Int64)
  - Auto-detect ms vs us in all read/filter/partition paths
  - Seeder writes us timestamps (upscales pre-2025 ms data)
  - All Parquet-to-Rust Arrow paths use process_trades_arrow_native (us-aware)
affects: [04-02 reseed, 05 clickhouse-rebuild]

# Tech tracking
tech-stack:
  added: []
  patterns:
    [
      _US_MS_THRESHOLD auto-detection for ms/us dual support,
      process_trades_arrow_native for us-native Arrow path,
    ]

key-files:
  created: []
  modified:
    - python/opendeviationbar/storage/parquet.py
    - python/opendeviationbar/storage/parquet_query.py
    - scripts/tick_cache_seeder.py
    - python/opendeviationbar/orchestration/helpers.py
    - scripts/repair_direct_parquet.py

key-decisions:
  - "Auto-detect ms vs us using _US_MS_THRESHOLD (1e13) in all timestamp conversion and filter paths"
  - "Normalize to microseconds on write (upscale ms*1000), not on read -- single canonical unit in storage"
  - "Preserve open_time_ms/close_time_ms CH conversion in orchestration helpers for Phase 5 scope"

patterns-established:
  - "_US_MS_THRESHOLD auto-detection: all timestamp functions handle both ms and us transparently"
  - "process_trades_arrow_native for us-timestamp Parquet data (replaces ms conversion + process_trades_arrow)"

requirements-completed: [USEC-01, USEC-04]

# Metrics
duration: 6min
completed: 2026-03-24
---

# Phase 4 Plan 1: TickStorage us-native Storage + Seeder + Pipeline Callers Summary

**TickStorage stores native microsecond timestamps with auto-detect ms/us filtering; seeder preserves us precision; all Parquet-to-Rust paths use process_trades_arrow_native**

## Performance

- **Duration:** 6 min
- **Started:** 2026-03-24T05:19:01Z
- **Completed:** 2026-03-24T05:25:01Z
- **Tasks:** 2
- **Files modified:** 5

## Accomplishments

- TickStorage normalizes to microseconds on write (ms\*1000 upscale, us passthrough)
- \_timestamp_to_day and_timestamp_to_year_month auto-detect ms vs us input
- read_ticks and read_ticks_streaming auto-normalize ms filter values to us for backward compat
- Seeder writes 16-digit us timestamps for post-2025 data, upscales pre-2025/futures ms data
- repair_direct_parquet.py and orchestration helpers use process_trades_arrow_native (us-aware)

## Task Commits

Each task was committed atomically:

1. **Task 1: TickStorage microsecond-native storage and auto-detect read filtering** - `64f4f6d` (feat)
2. **Task 2: Seeder stop truncating + pipeline callers use us-aware Arrow path** - `cf38be2` (feat)

## Files Created/Modified

- `python/opendeviationbar/storage/parquet.py` - \_normalize_timestamps targets us, write_ticks uses Datetime(time_unit="us"), read_ticks auto-normalizes ms filters
- `python/opendeviationbar/storage/parquet_query.py` - \_timestamp_to_day/\_timestamp_to_year_month auto-detect ms/us, read_ticks_streaming filter normalization
- `scripts/tick_cache_seeder.py` - Replaced us->ms truncation with ms->us upscale for uniform us storage
- `python/opendeviationbar/orchestration/helpers.py` - \_stream_bars_with_processor uses process_trades_arrow_native for us path
- `scripts/repair_direct_parquet.py` - Uses process_trades_arrow_native, removed us->ms normalization

## Decisions Made

- Auto-detect ms vs us using \_US_MS_THRESHOLD (1e13) -- same constant already existed in parquet.py, moved to parquet_query.py for shared access
- Normalize to microseconds on write rather than read -- single canonical unit avoids per-read conversion overhead
- Preserved open_time_ms/close_time_ms CH conversion in helpers.py -- Phase 5 will change CH to us

## Deviations from Plan

None - plan executed exactly as written.

## Known Stubs

None - all data paths are fully wired.

## Issues Encountered

None.

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- TickStorage is us-native, ready for 04-02 reseed of pre-2025 Parquet files
- All Parquet-to-Rust paths handle us timestamps correctly
- ClickHouse still uses ms timestamps (Phase 5 scope)

---

_Phase: 04-microsecond-data-layer_
_Completed: 2026-03-24_
