---
phase: 03-clickhouse-schema-migration
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/clickhouse/schema.sql
  - python/opendeviationbar/clickhouse/cache.py
  - scripts/migrate_column_rename_us.py
autonomous: true
requirements: [RENAME-01]

must_haves:
  truths:
    - "schema.sql CREATE TABLE uses close_time_us/open_time_us (already true, preserved)"
    - "schema.sql has a v14.x migration section documenting the _ms to _us rename"
    - "Migration script exists that runs ALTER TABLE RENAME COLUMN on bigblack"
    - "_ensure_schema() DDL strings reference _us column names, not _ms"
  artifacts:
    - path: "python/opendeviationbar/clickhouse/schema.sql"
      provides: "Updated schema with _us naming throughout including migration comments"
      contains: "close_time_us"
    - path: "scripts/migrate_column_rename_us.py"
      provides: "Migration script for bigblack column rename"
      contains: "RENAME COLUMN close_time_ms TO close_time_us"
    - path: "python/opendeviationbar/clickhouse/cache.py"
      provides: "_ensure_schema() with _us DDL references"
      contains: "close_time_us"
  key_links:
    - from: "scripts/migrate_column_rename_us.py"
      to: "bigblack ClickHouse"
      via: "ALTER TABLE RENAME COLUMN (run during deploy)"
      pattern: "RENAME COLUMN"
    - from: "python/opendeviationbar/clickhouse/cache.py"
      to: "python/opendeviationbar/clickhouse/schema.sql"
      via: "_ensure_schema() DDL must match schema.sql column names"
      pattern: "close_time_us"
---

<objective>
Update ClickHouse schema DDL and create migration script for renaming close_time_ms/open_time_ms to close_time_us/open_time_us.

Purpose: RENAME-01 requires the CH table to use \_us column names. Since close_time_ms is NOT in ORDER BY (changed to first_agg_trade_id in Issue #158), ALTER TABLE RENAME COLUMN works directly -- no table recreation needed.

Output: Updated schema.sql, migration script, updated \_ensure_schema() DDL
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@python/opendeviationbar/clickhouse/schema.sql
@python/opendeviationbar/clickhouse/cache.py
</context>

<tasks>

<task type="auto">
  <name>Task 1: Update schema.sql migration comments and create migration script</name>
  <files>python/opendeviationbar/clickhouse/schema.sql, scripts/migrate_column_rename_us.py</files>
  <read_first>
    - python/opendeviationbar/clickhouse/schema.sql (full file -- need all migration comment sections)
    - scripts/migrate_order_by_first_agg_trade_id.py (reference for migration script pattern)
  </read_first>
  <action>
Two sub-tasks:

**A) schema.sql updates:**

1. Add a new migration section header for v14.x (ClickHouse column rename) after the Issue #158 Phase 7 migration section (around line 191). Document:
   - `ALTER TABLE RENAME COLUMN close_time_ms TO close_time_us`
   - `ALTER TABLE RENAME COLUMN open_time_ms TO open_time_us`
   - Note that this works directly because close_time_ms is NOT in ORDER BY
   - Reference the migration script: `scripts/migrate_column_rename_us.py`
2. Update the Issue #164 migration comment (lines 166-177) to use `_us` names:
   - Line 167: `close_time_ms` in projection ORDER BY -> `close_time_us`
   - Lines 172-173: `MODIFY COLUMN close_time_ms` and `open_time_ms` -> `close_time_us` and `open_time_us`
3. Update line 188 comment: "Add skip index for close_time_ms range queries" -> `close_time_us`
4. The CREATE TABLE (line 199+) already uses `_us` -- verify and leave as-is
5. The projection at line 346-347 already uses `close_time_us` -- verify and leave as-is

**B) Create migration script `scripts/migrate_column_rename_us.py`:**
Follow the pattern from `scripts/migrate_order_by_first_agg_trade_id.py` (dry-run by default, `--apply` flag).

The script should:

1. Connect to ClickHouse (use `clickhouse_connect`, respect OPENDEVIATIONBAR_CH_HOSTS)
2. Verify current column names exist (`close_time_ms` and `open_time_ms` in system.columns)
3. If columns already renamed to `_us`, print "Already migrated" and exit 0
4. Show current row count for verification
5. In dry-run mode: print the ALTER statements that would run
6. In `--apply` mode:
   - `ALTER TABLE opendeviationbar_cache.open_deviation_bars RENAME COLUMN close_time_ms TO close_time_us`
   - `ALTER TABLE opendeviationbar_cache.open_deviation_bars RENAME COLUMN open_time_ms TO open_time_us`
   - Also rename in data_freshness table: `ALTER TABLE opendeviationbar_cache.data_freshness RENAME COLUMN last_close_time_ms TO last_close_time_us`
   - Verify row count unchanged after rename
   - Print success with before/after column check

CRITICAL: `ALTER TABLE RENAME COLUMN` is metadata-only in ClickHouse -- instant, no data rewrite. Safe for 42M+ rows.
</action>
<verify>
<automated>python -c "import ast; ast.parse(open('scripts/migrate_column_rename_us.py').read()); print('syntax OK')" && grep -c 'RENAME COLUMN' scripts/migrate_column_rename_us.py</automated>
</verify>
<acceptance_criteria> - schema.sql has v14.x migration section documenting the column rename - Legacy migration comments (Issue #164) updated from_ms to \_us - Migration script exists with dry-run default and --apply flag - Migration script handles idempotency (already-migrated detection)
</acceptance_criteria>
<done>schema.sql migration comments use \_us naming; migration script ready for bigblack deploy</done>
</task>

<task type="auto">
  <name>Task 2: Update _ensure_schema() DDL strings from _ms to _us</name>
  <files>python/opendeviationbar/clickhouse/cache.py</files>
  <read_first>
    - python/opendeviationbar/clickhouse/cache.py (lines 230-320 -- _ensure_schema and _migrate_auxiliary_tables)
  </read_first>
  <action>
Update all DDL strings in `_ensure_schema()` and `_migrate_auxiliary_tables()` that reference `_ms` column names:

1. **Line 254**: Skip index DDL -- change `close_time_ms` to `close_time_us`:
   `"ADD INDEX IF NOT EXISTS idx_close_time close_time_us "`
   Also update the comment on line 250 and log message on line 258.

2. **Lines 260-281**: Legacy schema detection block -- this entire block adds `open_time_ms` column for legacy schemas that only had `timestamp_ms`. Since the migration renames columns to `_us`, update:
   - Line 270 log message: reference `close_time_us` instead of `close_time_ms`
   - Lines 276-278: change to `open_time_us`, `DEFAULT close_time_us - duration_us`, `AFTER close_time_us`
   - Line 281 log message: reference `open_time_us`

3. **Line 299**: Projection DDL -- change `close_time_ms` to `close_time_us`:
   `"(SELECT * ORDER BY (symbol, threshold_decimal_bps, close_time_us))"`

4. **Line 312**: data_freshness CREATE TABLE DDL -- change `last_close_time_ms` to `last_close_time_us`:
   `"    last_close_time_us Int64,"`

Do NOT touch:

- bulk_operations.py, query_operations.py, stathera_query.py, constants.py -- these are Phase 4 (RENAME-02/04)
- Any code outside the clickhouse/cache.py \_ensure_schema() and \_migrate_auxiliary_tables() methods
  </action>
  <verify>
  <automated>grep -n "close_time_ms\|open_time_ms" python/opendeviationbar/clickhouse/cache.py | grep -v "^#" | head -20; echo "---"; python -c "import ast; ast.parse(open('python/opendeviationbar/clickhouse/cache.py').read()); print('syntax OK')"</automated>
  </verify>
  <acceptance_criteria>
  - \_ensure_schema() skip index references close_time_us
  - \_ensure_schema() projection references close_time_us
  - \_ensure_schema() legacy open_time column add references_us names
  - \_migrate_auxiliary_tables() data_freshness table uses last_close_time_us
  - cache.py passes Python syntax check
  - No \_ms references remain in DDL-issuing code paths within_ensure_schema()/\_migrate_auxiliary_tables()
    </acceptance_criteria>
    <done>All schema DDL strings in cache.py use_us column names matching schema.sql</done>
    </task>

</tasks>

<verification>
1. `python -c "import ast; ast.parse(open('python/opendeviationbar/clickhouse/cache.py').read())"` -- syntax valid
2. `python -c "import ast; ast.parse(open('scripts/migrate_column_rename_us.py').read())"` -- syntax valid
3. `grep -c 'close_time_ms\|open_time_ms' python/opendeviationbar/clickhouse/cache.py` -- only in non-DDL code (query strings for Phase 4)
4. `grep 'RENAME COLUMN' scripts/migrate_column_rename_us.py` -- migration statements present
5. schema.sql CREATE TABLE still uses `_us` names (no regression)
</verification>

<success_criteria>

- RENAME-01 migration infrastructure complete: schema.sql documented, script ready, \_ensure_schema() aligned
- Migration script is idempotent and safe for 42M+ row table (metadata-only RENAME COLUMN)
- No functional regressions -- only DDL strings changed, not query code (Phase 4)
  </success_criteria>

<output>
After completion, create `.planning/phases/03-clickhouse-schema-migration/03-01-SUMMARY.md`
</output>
