---
phase: 04-microsecond-data-layer
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/storage/parquet.py
  - python/opendeviationbar/storage/parquet_query.py
  - scripts/tick_cache_seeder.py
  - python/opendeviationbar/orchestration/helpers.py
  - scripts/repair_direct_parquet.py
autonomous: true
requirements: [USEC-01, USEC-04]

must_haves:
  truths:
    - "Binance seeder writes 16-digit microsecond timestamps to Parquet for post-2025-01-01 data"
    - "TickStorage write_ticks() preserves microsecond timestamps without truncating to milliseconds"
    - "TickStorage read_ticks() returns data with its native timestamp unit and filtering works correctly regardless of caller's timestamp unit"
    - "repair_direct_parquet.py and orchestration helpers process Parquet us timestamps through Rust correctly via timestamp_is_microseconds=true"
    - "Existing consumers (kintsugi, sidecar) continue to work with ms-based API calls against us-stored data"
  artifacts:
    - path: "python/opendeviationbar/storage/parquet.py"
      provides: "TickStorage with us-native storage, auto-detect filtering"
      contains: "_normalize_timestamps_to_us"
    - path: "python/opendeviationbar/storage/parquet_query.py"
      provides: "Timestamp-to-day conversion that handles both ms and us"
      contains: "_detect_timestamp_unit"
    - path: "scripts/tick_cache_seeder.py"
      provides: "Seeder that passes through native us timestamps from Vision"
    - path: "python/opendeviationbar/orchestration/helpers.py"
      provides: "Arrow processing with timestamp_is_microseconds awareness"
    - path: "scripts/repair_direct_parquet.py"
      provides: "Repair script using process_trades_arrow with us timestamps"
  key_links:
    - from: "scripts/tick_cache_seeder.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "write_ticks() without ms truncation"
      pattern: "storage.write_ticks"
    - from: "python/opendeviationbar/storage/parquet.py"
      to: "python/opendeviationbar/storage/parquet_query.py"
      via: "_timestamp_to_day auto-detecting ms vs us"
      pattern: "_timestamp_to_day"
    - from: "scripts/repair_direct_parquet.py"
      to: "src/core_bindings.rs"
      via: "process_trades_arrow with us-aware flag"
      pattern: "process_trades_arrow"
---

<objective>
Update the Parquet tick storage layer to store native microsecond timestamps and update all Parquet-to-Rust pipeline callers to handle microsecond data correctly.

Purpose: Eliminates the lossy ms truncation in the tick data pipeline. Post-2025-01-01 Binance Vision data has native microsecond precision that is currently discarded. This is the foundation for the full us alignment.

Output: TickStorage stores us timestamps, seeder preserves us precision, all Parquet→Rust paths handle us correctly.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/04-microsecond-data-layer/04-CONTEXT.md

@python/opendeviationbar/storage/parquet.py
@python/opendeviationbar/storage/parquet_query.py
@python/opendeviationbar/storage/parquet_io.py
@scripts/tick_cache_seeder.py
@scripts/repair_direct_parquet.py
@python/opendeviationbar/orchestration/helpers.py
@src/helpers.rs
@src/core_bindings.rs
@crates/opendeviationbar-core/src/arrow_export.rs

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/storage/parquet.py:

```python
class TickStorage:
    _normalize_timestamps(ticks, timestamp_col="timestamp") -> pl.DataFrame  # Currently forces ms
    write_ticks(symbol, ticks, *, timestamp_col="timestamp") -> int  # Currently normalizes to ms
    read_ticks(symbol, start_ts=None, end_ts=None, *, timestamp_col="timestamp", max_memory_mb=None) -> pl.DataFrame
    _timestamp_to_day(timestamp_ms: int) -> str  # Delegates to parquet_query
    _timestamp_to_year_month(timestamp_ms: int) -> str

_US_MS_THRESHOLD = 10_000_000_000_000  # 1e13 — already exists, used for detection
```

From python/opendeviationbar/storage/parquet_query.py:

```python
def _timestamp_to_day(timestamp_ms: int) -> str:  # Currently: datetime.fromtimestamp(timestamp_ms / 1000, tz=UTC)
def _timestamp_to_year_month(timestamp_ms: int) -> str
```

From scripts/tick_cache_seeder.py line 224-229:

```python
# Current: Normalize µs → ms for spot data
if market != "um_futures" and not df.is_empty():
    max_ts = df["timestamp"].max()
    if max_ts is not None and max_ts > 1_500_000_000_000_000:
        df = df.with_columns((pl.col("timestamp") // 1000).alias("timestamp"))
```

From src/core_bindings.rs line 455-474:

```rust
fn process_trades_arrow(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    // timestamp_is_microseconds=false: Python API passes millisecond timestamps
    process_from_arrow_columns(processor, record_batch, false)
}

fn process_trades_arrow_native(&mut self, py: Python, batch: PyRecordBatch) -> PyResult<PyRecordBatch> {
    // timestamp_is_microseconds=true: internal path, timestamps already in μs
    process_from_arrow_columns(processor, record_batch, true)
}
```

From python/opendeviationbar/orchestration/helpers.py line 318-343:

```python
# Current: converts us→ms before Arrow processing, then converts Arrow us output→ms for CH
chunk = chunk.with_columns((pl.col("timestamp") // 1000).alias("timestamp"))
bars_arrow = processor.process_trades_arrow(chunk_arrow)
# Then: open_time // 1000 → open_time_ms
```

From scripts/repair_direct_parquet.py line 301-302:

```python
# Current comment: process_trades_arrow() expects ms timestamps (Parquet ticks are ms)
bars_arrow = processor.process_trades_arrow(trades_batch)
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: TickStorage microsecond-native storage and auto-detect read filtering</name>
  <files>
    python/opendeviationbar/storage/parquet.py
    python/opendeviationbar/storage/parquet_query.py
  </files>
  <read_first>
    python/opendeviationbar/storage/parquet.py
    python/opendeviationbar/storage/parquet_query.py
    python/opendeviationbar/storage/CLAUDE.md
    python/opendeviationbar/kintsugi/repair.py (lines near has_ticks/read_ticks calls)
    python/opendeviationbar/kintsugi/discovery.py (lines near has_ticks calls)
  </read_first>
  <action>
    **parquet_query.py changes:**

    1. Rename `_timestamp_to_day(timestamp_ms)` to `_timestamp_to_day(timestamp: int)`. Make it auto-detect ms vs us using the existing `_US_MS_THRESHOLD` constant (move constant to parquet_query.py or import from parquet.py):
       - If `timestamp > _US_MS_THRESHOLD` (1e13): it's microseconds, use `datetime.fromtimestamp(timestamp / 1_000_000, tz=UTC)`
       - Else: it's milliseconds, use `datetime.fromtimestamp(timestamp / 1000, tz=UTC)`
       This ensures callers passing ms (kintsugi) and callers passing us (new seeder) both get correct day strings.

    2. Same treatment for `_timestamp_to_year_month()`.

    **parquet.py changes:**

    3. Replace `_normalize_timestamps()` — instead of normalizing to ms, normalize to MICROSECONDS:
       - Rename to `_normalize_timestamps_to_us(ticks, timestamp_col)` (or keep name but change behavior)
       - Datetime columns → `dt.epoch(time_unit="us")` (not "ms")
       - Int64 values BELOW `_US_MS_THRESHOLD` → multiply by 1000 (ms→us upscale)
       - Int64 values ABOVE `_US_MS_THRESHOLD` → pass through (already us)
       This is the inverse of the old behavior. Update the docstring to say "Normalize timestamps to microseconds."

    4. In `write_ticks()` line 234-238: change the day partitioning cast from `cast(pl.Datetime(time_unit="ms"))` to `cast(pl.Datetime(time_unit="us"))`. The `.dt.strftime("%Y-%m-%d")` call works identically regardless of time unit.

    5. In `read_ticks()` lines 426-429: add auto-detection for the caller's timestamp unit. The caller may pass ms values (kintsugi passes `day_start_ms`, `day_end_ms`). The stored data is now in us. Add a normalization step:
       ```python
       # Auto-detect caller's unit and normalize filter values to us
       if start_ts is not None and start_ts < _US_MS_THRESHOLD:
           start_ts = start_ts * 1000  # ms → us
       if end_ts is not None and end_ts < _US_MS_THRESHOLD:
           end_ts = end_ts * 1000  # ms → us
       ```
       Place this BEFORE the filter expressions (before line 426).

    6. Update `_timestamp_to_day` and `_timestamp_to_year_month` method wrappers in TickStorage class to reflect auto-detection (update docstrings from "millisecond timestamp" to "timestamp in ms or us").

    7. In `read_ticks_streaming()`: apply the same start_ts/end_ts auto-detection normalization as in `read_ticks()`.

    8. The `has_ticks()` function delegates to `_has_ticks` in parquet_query.py. Check if it uses timestamps for anything beyond day-string conversion — if so, ensure auto-detection applies. (It should only use `_timestamp_to_day` for file existence checks, which is already handled by step 1.)

    **Important: Do NOT change the `_US_MS_THRESHOLD` value.** It's `1e13` which correctly separates ms (1e12 range for modern dates) from us (1e15 range). This constant is the foundation of all auto-detection.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 python -c "
from opendeviationbar.storage.parquet_query import _timestamp_to_day
# Test ms input (2025-01-15 midnight UTC in ms)
assert _timestamp_to_day(1736899200000) == '2025-01-15', f'ms failed: {_timestamp_to_day(1736899200000)}'
# Test us input (same moment in us)
assert _timestamp_to_day(1736899200000000) == '2025-01-15', f'us failed: {_timestamp_to_day(1736899200000000)}'
print('_timestamp_to_day auto-detect: PASS')

from opendeviationbar.storage.parquet import TickStorage
import polars as pl

# Test \_normalize_timestamps now targets us

ts = TickStorage()
df_ms = pl.DataFrame({'timestamp': [1736899200000]}) # ms
result = ts.\_normalize_timestamps(df_ms)
assert result['timestamp'][0] > 1e13, f'ms not upscaled: {result[\"timestamp\"][0]}'
print('\_normalize_timestamps upscales ms to us: PASS')

df_us = pl.DataFrame({'timestamp': [1736899200000000]}) # us
result2 = ts.\_normalize_timestamps(df_us)
assert result2['timestamp'][0] == 1736899200000000, f'us changed: {result2[\"timestamp\"][0]}'
print('\_normalize_timestamps preserves us: PASS')
print('ALL CHECKS PASSED')
"
</automated>
</verify>
<acceptance_criteria> - `_timestamp_to_day(1736899200000)` returns "2025-01-15" (ms input) - `_timestamp_to_day(1736899200000000)` returns "2025-01-15" (us input) - `_normalize_timestamps` on ms Int64 data produces values > 1e13 (upscaled to us) - `_normalize_timestamps` on us Int64 data preserves original values - `write_ticks` day partitioning uses `Datetime(time_unit="us")` - `read_ticks` auto-normalizes ms filter values to us before predicate pushdown - grep: `time_unit="us"` appears in write_ticks partitioning code - grep: `_US_MS_THRESHOLD` used in read_ticks filter normalization
</acceptance_criteria>
<done>TickStorage stores microsecond timestamps natively, auto-detects ms vs us in all read/filter paths, backward compatible with ms-based callers</done>
</task>

<task type="auto">
  <name>Task 2: Seeder stop truncating + pipeline callers use us-aware Arrow path</name>
  <files>
    scripts/tick_cache_seeder.py
    python/opendeviationbar/orchestration/helpers.py
    scripts/repair_direct_parquet.py
  </files>
  <read_first>
    scripts/tick_cache_seeder.py (full file — especially _download_vision_day and the µs→ms normalization)
    python/opendeviationbar/orchestration/helpers.py (full file — especially the Arrow processing and timestamp conversion)
    scripts/repair_direct_parquet.py (especially the process_trades_arrow call and surrounding context)
    src/core_bindings.rs (process_trades_arrow and process_trades_arrow_native methods)
  </read_first>
  <action>
    **tick_cache_seeder.py:**

    1. Remove the µs→ms normalization block at lines 224-229 in `_download_vision_day()`:
       ```python
       # DELETE THESE LINES:
       # Normalize µs → ms for spot data (spot switched to µs on 2025-01-01).
       # UM futures timestamps are ALWAYS ms — skip normalization for futures.
       if market != "um_futures" and not df.is_empty():
           max_ts = df["timestamp"].max()
           if max_ts is not None and max_ts > 1_500_000_000_000_000:
               df = df.with_columns((pl.col("timestamp") // 1000).alias("timestamp"))
       ```

    2. Instead, add an UPSCALE for UM futures and pre-2025 spot data (which are in ms) so ALL data going into TickStorage is in microseconds:
       ```python
       # Normalize all timestamps to microseconds for uniform storage.
       # - Post-2025-01-01 spot data: already µs (16-digit), pass through.
       # - Pre-2025 spot data: ms (13-digit), upscale ×1000.
       # - UM futures: always ms, upscale ×1000.
       if not df.is_empty():
           max_ts = df["timestamp"].max()
           if max_ts is not None and max_ts < 1_500_000_000_000_000:
               # ms timestamps → upscale to µs
               df = df.with_columns((pl.col("timestamp") * 1000).alias("timestamp"))
       ```

    3. Update the comment at line 71 from `"timestamp",      # int64 ms epoch` to `"timestamp",      # int64 — ms before 2025, µs from 2025 (normalized to µs before write)`

    **orchestration/helpers.py:**

    4. In the `_stream_bars_from_chunks()` function (or wherever the Arrow processing loop is), update the timestamp handling. Currently lines 318-323 convert us→ms before calling `process_trades_arrow()`. After this change, Parquet data already has us timestamps, so:
       - Remove the `// 1000` conversion that downgrades us→ms (lines 321-323)
       - Change `process_trades_arrow(chunk_arrow)` to `process_trades_arrow_native(chunk_arrow)` which accepts us timestamps
       - The output from `process_trades_arrow_native()` has `open_time`/`close_time` in us (same as `process_trades_arrow` — Rust always outputs us). The downstream conversion to `open_time_ms`/`close_time_ms` (lines 332-343) should remain for now since ClickHouse still uses ms (Phase 5 changes that).

    5. Update the comment at line 301 to reflect that we now use the us-native path.

    **repair_direct_parquet.py:**

    6. Change `processor.process_trades_arrow(trades_batch)` to `processor.process_trades_arrow_native(trades_batch)` since Parquet ticks are now in microseconds.

    7. Update the comment at line 301 from "process_trades_arrow() expects ms timestamps (Parquet ticks are ms)" to "process_trades_arrow_native() expects µs timestamps (Parquet ticks are µs)".

    8. The orphan bar handling after `reset_at_ouroboros()` returns a dict with `open_time_ms`/`close_time_ms` — this is from the dict path in helpers.rs which divides by 1000. This is fine for now (Phase 5 changes the dict path).

    **Important:** The Arrow output from both `process_trades_arrow()` and `process_trades_arrow_native()` produces `open_time`/`close_time` in MICROSECONDS. The downstream code in orchestration/helpers.py that converts `open_time // 1000 → open_time_ms` MUST be preserved — ClickHouse schema still uses ms until Phase 5.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "// 1000" scripts/tick_cache_seeder.py | grep -v "^#" | grep -c "timestamp" && echo "FAIL: seeder still truncates us to ms" || echo "PASS: seeder no longer truncates"; grep -n "process_trades_arrow_native" scripts/repair_direct_parquet.py | head -3 && echo "PASS: repair uses native path" || echo "FAIL: repair not updated"; grep -n "process_trades_arrow_native" python/opendeviationbar/orchestration/helpers.py | head -3 && echo "PASS: orchestration uses native path" || echo "FAIL: orchestration not updated"</automated>
  </verify>
  <acceptance_criteria>
    - grep: `// 1000` does NOT appear near `timestamp` in tick_cache_seeder.py `_download_vision_day()`
    - grep: `* 1000` appears in tick_cache_seeder.py for ms→us upscaling of pre-2025/futures data
    - grep: `process_trades_arrow_native` appears in repair_direct_parquet.py
    - grep: `process_trades_arrow_native` appears in orchestration/helpers.py
    - grep: `open_time_ms` conversion (us→ms for CH) still present in orchestration/helpers.py
    - Seeder writes 16-digit timestamps for post-2025 data (no truncation)
    - Pre-2025 and UM futures data upscaled to us before write
  </acceptance_criteria>
  <done>Seeder preserves native us timestamps, all Parquet→Rust pipeline callers use process_trades_arrow_native for us timestamps, ClickHouse ms conversion preserved for Phase 5</done>
</task>

</tasks>

<verification>
1. TickStorage round-trip test: write us timestamps, read back, verify 16-digit values preserved
2. _timestamp_to_day auto-detection works for both ms and us inputs
3. Seeder _download_vision_day returns us timestamps for post-2025 data
4. repair_direct_parquet uses process_trades_arrow_native
5. orchestration helpers use process_trades_arrow_native but still output open_time_ms for CH
6. All existing Python tests pass: `mise run test-py`
7. Rust tests pass: `mise run test`
</verification>

<success_criteria>

- All Parquet tick data written by seeder contains microsecond timestamps (16-digit for post-2025, ms\*1000 for pre-2025)
- TickStorage auto-detects ms vs us in read/filter paths — backward compatible with ms callers
- All Parquet→Rust Arrow paths use process_trades_arrow_native (us-aware)
- ClickHouse writes still produce open_time_ms/close_time_ms in milliseconds (Phase 5 scope)
- `mise run test-py` passes
- `mise run test` passes
  </success_criteria>

<output>
After completion, create `.planning/phases/04-microsecond-data-layer/04-01-SUMMARY.md`
</output>
