---
phase: 05-microsecond-clickhouse-rebuild
plan: 02
type: execute
wave: 2
depends_on: [05-01]
files_modified:
  - python/opendeviationbar/clickhouse/schema.sql
  - python/opendeviationbar/clickhouse/bulk_operations.py
  - python/opendeviationbar/clickhouse/query_operations.py
  - python/opendeviationbar/clickhouse/cache.py
  - python/opendeviationbar/clickhouse/maintenance.py
  - python/opendeviationbar/clickhouse/stathera_query.py
  - python/opendeviationbar/clickhouse/migrations.py
  - python/opendeviationbar/clickhouse/bar_count_api.py
  - python/opendeviationbar/orchestration/open_deviation_bars.py
  - scripts/repair_direct_parquet.py
autonomous: true
requirements: [USEC-03]

must_haves:
  truths:
    - "ClickHouse schema DDL defines open_time_us and close_time_us columns with microsecond COMMENTs"
    - "All 8 ClickHouse Python files use _us column names in SQL queries"
    - "All SQL date derivations use /1000000 divisor (not /1000)"
    - "repair_direct_parquet.py writes _us columns with no /1000 division"
    - "prj_time_range projection uses close_time_us"
  artifacts:
    - path: "python/opendeviationbar/clickhouse/schema.sql"
      provides: "v2 DDL with _us timestamp columns"
      contains: "close_time_us"
    - path: "scripts/repair_direct_parquet.py"
      provides: "us-native Arrow rebuild pipeline"
      contains: "close_time_us"
    - path: "python/opendeviationbar/clickhouse/bulk_operations.py"
      provides: "us-aware bulk INSERT operations"
      contains: "open_time_us"
  key_links:
    - from: "python/opendeviationbar/clickhouse/schema.sql"
      to: "python/opendeviationbar/clickhouse/bulk_operations.py"
      via: "column names in INSERT statements must match DDL"
      pattern: "close_time_us"
    - from: "scripts/repair_direct_parquet.py"
      to: "python/opendeviationbar/clickhouse/schema.sql"
      via: "Arrow column names must match CH table columns"
      pattern: "open_time_us"
---

<objective>
Migrate the ClickHouse schema DDL and all Python ClickHouse layer code from _ms to _us columns. Update repair_direct_parquet.py to stop dividing by 1000.

Purpose: The ClickHouse table definition and all SQL queries that reference timestamp columns must use \_us naming and /1000000 divisors before the full rebuild can produce us-precision bars.
Output: Updated schema.sql with_us columns, all CH Python files use us SQL patterns, repair_direct_parquet.py outputs us-native columns.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-RESEARCH.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-01-SUMMARY.md

<interfaces>
<!-- Contracts established by Plan 01 that this plan depends on -->

From python/opendeviationbar/constants.py (after Plan 01):

```python
TIMESTAMP_COLUMNS: tuple[str, ...] = ("close_time_us", "open_time_us",)
```

From python/opendeviationbar/validation/day_mode/guards.py (after Plan 01):

```python
DAY_US = 86_400_000_000  # microseconds per UTC day
```

SQL pattern changes required (from 05-RESEARCH.md):

| Old Pattern                               | New Pattern                                  |
| ----------------------------------------- | -------------------------------------------- |
| `toDate(open_time_ms / 1000, 'UTC')`      | `toDate(open_time_us / 1000000, 'UTC')`      |
| `toDate(close_time_ms / 1000, 'UTC')`     | `toDate(close_time_us / 1000000, 'UTC')`     |
| `toDateTime(close_time_ms / 1000, 'UTC')` | `toDateTime(close_time_us / 1000000, 'UTC')` |
| `intDiv(close_time_ms, 1000)`             | `intDiv(close_time_us, 1000000)`             |
| `fromUnixTimestamp64Milli(close_time_ms)` | `fromUnixTimestamp64Micro(close_time_us)`    |
| `pd.to_datetime(..., unit="ms")`          | `pd.to_datetime(..., unit="us")`             |

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Update ClickHouse schema DDL and all CH Python layer files</name>
  <files>
    python/opendeviationbar/clickhouse/schema.sql,
    python/opendeviationbar/clickhouse/bulk_operations.py,
    python/opendeviationbar/clickhouse/query_operations.py,
    python/opendeviationbar/clickhouse/cache.py,
    python/opendeviationbar/clickhouse/maintenance.py,
    python/opendeviationbar/clickhouse/stathera_query.py,
    python/opendeviationbar/clickhouse/migrations.py,
    python/opendeviationbar/clickhouse/bar_count_api.py,
    python/opendeviationbar/orchestration/open_deviation_bars.py
  </files>
  <read_first>
    python/opendeviationbar/clickhouse/schema.sql,
    python/opendeviationbar/clickhouse/bulk_operations.py,
    python/opendeviationbar/clickhouse/query_operations.py,
    python/opendeviationbar/clickhouse/cache.py,
    python/opendeviationbar/clickhouse/maintenance.py,
    python/opendeviationbar/clickhouse/stathera_query.py,
    python/opendeviationbar/clickhouse/migrations.py,
    python/opendeviationbar/clickhouse/bar_count_api.py,
    python/opendeviationbar/orchestration/open_deviation_bars.py
  </read_first>
  <action>
    **schema.sql — Full column rename with updated COMMENTs (USEC-03):**

    Main table `open_deviation_bars`:
    - `close_time_ms Int64 CODEC(Delta, ZSTD)` -> `close_time_us Int64 CODEC(Delta, ZSTD)` with COMMENT 'Bar close time in microseconds since Unix epoch. Use intDiv(close_time_us, 1000000) for epoch seconds.'
    - `open_time_ms Int64 DEFAULT 0 CODEC(Delta, ZSTD)` -> `open_time_us Int64 DEFAULT 0 CODEC(Delta, ZSTD)` with COMMENT 'Bar open time in microseconds since Unix epoch. Use intDiv(open_time_us, 1000000) for epoch seconds.'
    - Update the line comment "Bar close time in milliseconds" -> "Bar close time in microseconds"
    - Update prj_time_range projection: `close_time_ms` -> `close_time_us`
    - Add a migration section at the top for v2.0 explaining the ms->us column rename + new table strategy
    - All existing migration comments can stay as historical reference

    Secondary tables:
    - `data_freshness`: rename `last_close_time_ms` -> `last_close_time_us`
    - `population_checkpoints`: rename `last_trade_timestamp_ms` -> `last_trade_timestamp_us`
    - `kintsugi_log`: rename `gap_start_ms` -> `gap_start_us`, `gap_end_ms` -> `gap_end_us`

    Commented-out materialized view: update `timestamp_ms / 1000` -> `timestamp_us / 1000000`

    **bulk_operations.py:**
    - All string refs `"open_time_ms"` -> `"open_time_us"`, `"close_time_ms"` -> `"close_time_us"`
    - All `// DAY_MS` -> `// DAY_US` (import DAY_US from guards)
    - Rename guard function `_guard_close_time_ms_scale` -> `_guard_close_time_us_scale` if it exists
    - SQL strings: `/ 1000` divisors for date derivation -> `/ 1000000`
    - `pd.to_datetime(..., unit="ms")` -> `unit="us"`

    **query_operations.py:**
    - All `close_time_ms` in SQL strings -> `close_time_us`
    - All `open_time_ms` in SQL strings -> `open_time_us`
    - `toDate(close_time_ms / 1000` -> `toDate(close_time_us / 1000000`
    - `pd.to_datetime(..., unit="ms")` -> `unit="us"`

    **cache.py:**
    - `close_time_ms` in SQL strings -> `close_time_us`
    - `last_close_time_ms` for data_freshness -> `last_close_time_us`
    - Any `/ 1000` SQL divisors -> `/ 1000000`

    **maintenance.py:**
    - `close_time_ms` in DELETE/cleanup SQL -> `close_time_us`
    - Any `/ 1000` SQL divisors -> `/ 1000000`

    **stathera_query.py:**
    - All timestamp column references in SQL builder -> `_us`
    - `/ 1000` divisors -> `/ 1000000`

    **migrations.py:**
    - `intDiv(close_time_ms, 1000)` -> `intDiv(close_time_us, 1000000)`

    **bar_count_api.py:**
    - `close_time_ms` refs -> `close_time_us`

    **orchestration/open_deviation_bars.py:**
    - `"close_time_ms"` dict key refs -> `"close_time_us"`
    - `pd.to_datetime(..., unit="ms")` -> `unit="us"`

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -rn "close_time_ms\|open_time_ms\|last_close_time_ms\|gap_start_ms\|gap_end_ms\|last_trade_timestamp_ms" python/opendeviationbar/clickhouse/ python/opendeviationbar/orchestration/open_deviation_bars.py 2>/dev/null | grep -v "\.pyc" | wc -l</automated>
  </verify>
  <acceptance_criteria>
    - grep -rn "close_time_ms" python/opendeviationbar/clickhouse/schema.sql returns NO matches (only _us)
    - grep -rn "close_time_us" python/opendeviationbar/clickhouse/schema.sql returns matches for the column definitions
    - grep -rn "close_time_ms\|open_time_ms" python/opendeviationbar/clickhouse/bulk_operations.py returns NO matches
    - grep -rn "close_time_ms\|open_time_ms" python/opendeviationbar/clickhouse/query_operations.py returns NO matches
    - grep -rn "/ 1000[^0]" python/opendeviationbar/clickhouse/ returns NO matches (all divisors are /1000000)
    - grep "last_close_time_us" python/opendeviationbar/clickhouse/schema.sql returns the data_freshness column
    - grep "gap_start_us" python/opendeviationbar/clickhouse/schema.sql returns the kintsugi_log column
    - Total count of _ms references across all CH files is 0 (excluding historical migration comments)
  </acceptance_criteria>
  <done>ClickHouse schema DDL uses _us columns with microsecond COMMENTs, all 8 CH Python files use _us naming and /1000000 divisors, orchestration helper uses _us keys</done>
</task>

<task type="auto">
  <name>Task 2: Update repair_direct_parquet.py to output us-native columns</name>
  <files>
    scripts/repair_direct_parquet.py
  </files>
  <read_first>
    scripts/repair_direct_parquet.py
  </read_first>
  <action>
    **Remove the us->ms conversion (lines ~314/317 per research):**
    - Find the lines that divide `open_time // 1000` and alias to `open_time_ms` — change to `pl.col("open_time").alias("open_time_us")` with NO division
    - Same for `close_time // 1000` -> `pl.col("close_time").alias("close_time_us")` with NO division

    **Update SQL queries in the file:**
    - Line ~178 per research: `open_time_ms / 1000` in any SQL string -> `open_time_us / 1000000`
    - Any `close_time_ms` in SQL strings -> `close_time_us`
    - Any `/ 1000` SQL divisors for date derivation -> `/ 1000000`

    **Update data_freshness writes:**
    - If the script updates data_freshness, change `last_close_time_ms` -> `last_close_time_us`

    **Update DAY_MS references:**
    - Any `86_400_000` literal -> `86_400_000_000`
    - Import DAY_US if used

    The Arrow path from Rust already outputs us-native `open_time`/`close_time` columns (no _ms suffix). This task just changes the alias from `_ms` to `_us` and removes the division.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "open_time_ms\|close_time_ms\|// 1000[^0]" scripts/repair_direct_parquet.py | wc -l</automated>
  </verify>
  <acceptance_criteria>
    - grep "open_time_ms\|close_time_ms" scripts/repair_direct_parquet.py returns NO matches
    - grep "open_time_us" scripts/repair_direct_parquet.py returns matches for the column alias
    - grep "// 1000" scripts/repair_direct_parquet.py returns NO matches (no ms division left)
    - grep "/ 1000000" scripts/repair_direct_parquet.py returns matches for SQL date derivation
  </acceptance_criteria>
  <done>repair_direct_parquet.py outputs open_time_us/close_time_us columns with raw microsecond values, all SQL queries use /1000000 divisor</done>
</task>

</tasks>

<verification>
- `grep -rn "close_time_ms\|open_time_ms" python/opendeviationbar/clickhouse/ scripts/repair_direct_parquet.py` returns zero matches (excluding comments about migration history)
- `grep -c "close_time_us" python/opendeviationbar/clickhouse/schema.sql` returns >= 2 (column def + projection)
- Schema.sql contains microsecond COMMENTs on all timestamp columns
</verification>

<success_criteria>

- ClickHouse DDL defines close_time_us and open_time_us with Delta,ZSTD codec and microsecond COMMENTs
- All 4 secondary tables (data_freshness, population_checkpoints, kintsugi_log) use_us column names
- All 8 ClickHouse Python files have zero \_ms column references in active code
- repair_direct_parquet.py outputs us columns with no /1000 division
- prj_time_range projection references close_time_us
  </success_criteria>

<output>
After completion, create `.planning/phases/05-microsecond-clickhouse-rebuild/05-02-SUMMARY.md`
</output>
