---
phase: 05-microsecond-clickhouse-rebuild
plan: 02
subsystem: database
tags: [clickhouse, microsecond, schema, sql, migration]

requires:
  - phase: 05-01
    provides: "TIMESTAMP_COLUMNS tuple with _us names, DAY_US constant"
provides:
  - "ClickHouse DDL with _us timestamp columns and microsecond COMMENTs"
  - "All CH Python layer files using _us column names and /1000000 divisors"
  - "repair_direct_parquet.py outputting us-native columns without /1000 division"
affects: [05-03, 05-04, sidecar, kintsugi, heartbeat]

tech-stack:
  added: []
  patterns:
    - "All ClickHouse timestamp columns use _us suffix with microsecond precision"
    - "SQL date derivation uses intDiv(close_time_us, 1000000) for epoch seconds"
    - "pd.to_datetime unit='us' for converting us timestamps to pandas DatetimeIndex"

key-files:
  created: []
  modified:
    - python/opendeviationbar/clickhouse/schema.sql
    - python/opendeviationbar/clickhouse/bulk_operations.py
    - python/opendeviationbar/clickhouse/query_operations.py
    - python/opendeviationbar/clickhouse/cache.py
    - python/opendeviationbar/clickhouse/maintenance.py
    - python/opendeviationbar/clickhouse/stathera_query.py
    - python/opendeviationbar/clickhouse/migrations.py
    - python/opendeviationbar/clickhouse/bar_count_api.py
    - python/opendeviationbar/clickhouse/checkpoint_operations.py
    - python/opendeviationbar/orchestration/open_deviation_bars.py
    - scripts/repair_direct_parquet.py

key-decisions:
  - "Scale guard updated for us range [1e15, 2.5e15] replacing ms range [1e12, 2.5e12]"
  - "open_time_us DEFAULT changed from close_time_us - intDiv(duration_us, 1000) to close_time_us - duration_us since both are now microseconds"

patterns-established:
  - "ClickHouse _us column naming convention for all timestamp columns"
  - "fromUnixTimestamp64Micro(close_time_us) for ClickHouse timestamp conversion"

requirements-completed: [USEC-03]

duration: 6min
completed: 2026-03-24
---

# Phase 05 Plan 02: ClickHouse Schema DDL and Python Layer ms-to-us Migration Summary

**ClickHouse DDL renamed to \_us timestamp columns with microsecond COMMENTs; all 10 CH Python files use \_us naming, /1000000 divisors, and us-scale validation**

## Performance

- **Duration:** 6 min
- **Started:** 2026-03-24T06:16:18Z
- **Completed:** 2026-03-24T06:22:40Z
- **Tasks:** 2
- **Files modified:** 11

## Accomplishments

- Schema.sql defines close_time_us/open_time_us with Delta,ZSTD codec and microsecond COMMENTs
- All 4 secondary tables (data_freshness, population_checkpoints, kintsugi_log) use_us column names
- All 10 ClickHouse Python files have zero \_ms column references in active code (4 remain in historical migration comments only)
- repair_direct_parquet.py outputs us columns with no /1000 division (raw microsecond passthrough from Arrow)
- prj_time_range projection references close_time_us

## Task Commits

Each task was committed atomically:

1. **Task 1: Update ClickHouse schema DDL and all CH Python layer files** - `5a98ce2` (feat)
2. **Task 2: Update repair_direct_parquet.py to output us-native columns** - `734d475` (feat)

## Files Created/Modified

- `python/opendeviationbar/clickhouse/schema.sql` - DDL with \_us timestamp columns, microsecond COMMENTs
- `python/opendeviationbar/clickhouse/bulk_operations.py` - \_us column refs, us-scale guard, DAY_US cross-midnight gate
- `python/opendeviationbar/clickhouse/query_operations.py` - SQL queries with \_us columns, pd.to_datetime unit="us"
- `python/opendeviationbar/clickhouse/cache.py` - Cache operations with \_us columns, migration helpers updated
- `python/opendeviationbar/clickhouse/maintenance.py` - Comment reference updated to close_time_us
- `python/opendeviationbar/clickhouse/stathera_query.py` - Stathera SQL builder with close_time_us
- `python/opendeviationbar/clickhouse/migrations.py` - intDiv(close_time_us, 1000000) for session derivation
- `python/opendeviationbar/clickhouse/bar_count_api.py` - Bar count queries with close_time_us
- `python/opendeviationbar/clickhouse/checkpoint_operations.py` - last_trade_timestamp_us in SQL and Python
- `python/opendeviationbar/orchestration/open_deviation_bars.py` - Orphan bar handling with \_us keys
- `scripts/repair_direct_parquet.py` - us-native Arrow column aliases, /1000000 SQL divisor

## Decisions Made

- Scale guard for \_us columns validates [1e15, 2.5e15] range (was [1e12, 2.5e12] for ms)
- open_time_us DEFAULT simplified to `close_time_us - duration_us` (no intDiv needed since both are microseconds)

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 3 - Blocking] Updated checkpoint_operations.py for last_trade_timestamp_us**

- **Found during:** Task 1 (CH Python layer update)
- **Issue:** checkpoint_operations.py references `last_trade_timestamp_ms` in SQL INSERT/SELECT queries and CREATE TABLE, which no longer matches the renamed schema.sql column `last_trade_timestamp_us`
- **Fix:** Renamed all `last_trade_timestamp_ms` references to `last_trade_timestamp_us` in checkpoint_operations.py
- **Files modified:** python/opendeviationbar/clickhouse/checkpoint_operations.py
- **Verification:** grep confirms zero \_ms references in active code
- **Committed in:** 5a98ce2 (Task 1 commit)

---

**Total deviations:** 1 auto-fixed (1 blocking)
**Impact on plan:** Essential fix -- without it, checkpoint SQL queries would reference a non-existent column after the schema.sql rename.

## Issues Encountered

None.

## Known Stubs

None -- all \_us column references are fully wired with correct divisors and scale validation.

## Next Phase Readiness

- ClickHouse schema and Python layer fully migrated to \_us naming
- Ready for Plan 03 (sidecar/kintsugi/heartbeat migration) and Plan 04 (full rebuild)
- Callers outside the CH layer (sidecar.py, kintsugi.py, heartbeat) still use \_ms internally; they will be updated in Plan 03

---

_Phase: 05-microsecond-clickhouse-rebuild_
_Completed: 2026-03-24_
