---
phase: 05-microsecond-clickhouse-rebuild
plan: 04
type: execute
wave: 3
depends_on: [05-02, 05-03]
files_modified:
  - python/opendeviationbar/validation/day_mode/CLAUDE.md
  - python/opendeviationbar/clickhouse/CLAUDE.md
  - scripts/CLAUDE.md
  - CLAUDE.md
autonomous: false
requirements: [USEC-06]

user_setup:
  - service: bigblack-ssh
    why: "Full 35M+ bar rebuild requires SSH access to bigblack for ClickHouse operations"
    env_vars: []
    dashboard_config:
      - task: "Stop sidecar + kintsugi services before table swap"
        location: "bigblack via SSH: sudo systemctl stop opendeviationbar-sidecar opendeviationbar-kintsugi"

must_haves:
  truths:
    - "New open_deviation_bars table has close_time_us and open_time_us columns with microsecond values"
    - "All 35M+ bars rebuilt from Parquet source data with microsecond precision"
    - "Stathera continuity validated across the entire rebuilt dataset (zero gaps, zero overlaps)"
    - "Sidecar, kintsugi, and heartbeat restart and operate correctly with new schema"
    - "CLAUDE.md network documentation updated to reference _us conventions"
  artifacts:
    - path: "python/opendeviationbar/clickhouse/CLAUDE.md"
      provides: "Updated documentation with us conventions"
      contains: "close_time_us"
    - path: "python/opendeviationbar/validation/day_mode/CLAUDE.md"
      provides: "Updated oracle query with us divisors"
      contains: "/ 1000000"
  key_links:
    - from: "scripts/repair_direct_parquet.py"
      to: "ClickHouse open_deviation_bars_v2 table"
      via: "48-worker Arrow rebuild pipeline"
      pattern: "open_time_us"
    - from: "scripts/detect_gaps.py"
      to: "ClickHouse open_deviation_bars table"
      via: "Stathera gap validation SQL"
      pattern: "close_time_us"
---

<objective>
Execute the full 35M+ bar rebuild on bigblack, perform atomic table swap, validate Stathera continuity, and update CLAUDE.md documentation network.

Purpose: This is the culminating plan -- create the new us-schema table, rebuild all bars from Parquet source data, swap tables atomically, and verify the entire dataset has zero gaps and zero overlaps. Then update documentation.
Output: Production ClickHouse running with microsecond-precision bars, services restarted and healthy, documentation updated.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-RESEARCH.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-01-SUMMARY.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-02-SUMMARY.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-03-SUMMARY.md

<interfaces>
<!-- Migration strategy from 05-RESEARCH.md -->

New table + rebuild + atomic swap:

1. Create open_deviation_bars_v2 with_us columns (use updated schema.sql DDL)
2. Run repair_direct_parquet.py --all against v2 table (48 workers)
3. Stop sidecar + kintsugi
4. RENAME TABLE open_deviation_bars TO open_deviation_bars_legacy, open_deviation_bars_v2 TO open_deviation_bars
5. TRUNCATE kintsugi_log (90-day TTL audit log, kintsugi rediscovers from scratch)
6. TRUNCATE population_checkpoints (rebuild bypasses checkpoint logic)
7. DROP + recreate data_freshness (small table, sidecar repopulates)
8. Restart sidecar + kintsugi
9. Validate Stathera continuity
10. DROP TABLE open_deviation_bars_legacy (after validation)
    </interfaces>

</context>

<tasks>

<task type="auto">
  <name>Task 1: Create v2 table, rebuild all bars, and execute atomic swap</name>
  <files></files>
  <read_first>
    python/opendeviationbar/clickhouse/schema.sql,
    scripts/repair_direct_parquet.py
  </read_first>
  <action>
    This task executes the full rebuild on bigblack via SSH. The repair_direct_parquet.py script is already us-native from Plan 02.

    **Step 1: Create the v2 table on bigblack**
    SSH to bigblack and create `open_deviation_bars_v2` using the updated schema.sql DDL (which now has _us columns). Modify the CREATE TABLE name to `open_deviation_bars_v2` temporarily.

    ```sql
    -- On bigblack ClickHouse:
    CREATE TABLE IF NOT EXISTS opendeviationbar_cache.open_deviation_bars_v2
    -- (paste the full DDL from schema.sql but with table name = open_deviation_bars_v2)
    ```

    Also recreate secondary tables with _us columns:
    - DROP + recreate `data_freshness` with `last_close_time_us`
    - ALTER TABLE `population_checkpoints` RENAME COLUMN `last_trade_timestamp_ms` TO `last_trade_timestamp_us`, UPDATE values * 1000
    - DROP + recreate `kintsugi_log` with `gap_start_us`/`gap_end_us` (TRUNCATE first since 90-day TTL)

    **Step 2: Deploy updated code to bigblack**
    The code changes from Plans 01-03 must be deployed to bigblack before the rebuild. Use the standard deploy workflow.

    **Step 3: Run the full rebuild**
    On bigblack, targeting the v2 table:
    ```bash
    # Temporarily set target table to v2 (environment variable or script arg)
    python scripts/repair_direct_parquet.py --all --workers 48 --table open_deviation_bars_v2
    ```

    If repair_direct_parquet.py does not support a `--table` argument, temporarily modify the script to target `open_deviation_bars_v2`, or use an environment variable override. Expected runtime: ~3200 days/min with 48 workers.

    **Step 4: Stop services and atomic swap**
    ```bash
    sudo systemctl stop opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer
    ```

    Execute atomic table swap on ClickHouse:
    ```sql
    RENAME TABLE
      opendeviationbar_cache.open_deviation_bars TO opendeviationbar_cache.open_deviation_bars_legacy,
      opendeviationbar_cache.open_deviation_bars_v2 TO opendeviationbar_cache.open_deviation_bars;
    ```

    **Step 5: Clean up auxiliary tables**
    ```sql
    TRUNCATE TABLE opendeviationbar_cache.kintsugi_log;
    TRUNCATE TABLE opendeviationbar_cache.population_checkpoints;
    ```

    **Step 6: Restart services**
    ```bash
    sudo systemctl start opendeviationbar-sidecar opendeviationbar-kintsugi opendeviationbar-heartbeat.timer
    ```

  </action>
  <verify>
    <automated>MISSING -- Requires bigblack SSH access. Verify manually: ssh bigblack "clickhouse-client --query \"SELECT count() FROM opendeviationbar_cache.open_deviation_bars WHERE close_time_us > 1000000000000000\""</automated>
  </verify>
  <acceptance_criteria>
    - ClickHouse open_deviation_bars table has close_time_us and open_time_us columns (not _ms)
    - Row count in new table is >= 35,000,000 (all bars rebuilt)
    - All close_time_us values are > 1e15 (microsecond scale, not millisecond)
    - Services (sidecar, kintsugi, heartbeat) are running after restart
  </acceptance_criteria>
  <done>All 35M+ bars rebuilt with microsecond timestamps, atomic table swap complete, services restarted</done>
</task>

<task type="checkpoint:human-verify" gate="blocking">
  <name>Task 2: Verify rebuild integrity and service health</name>
  <files></files>
  <action>
    Human verification checkpoint. Claude automated the full rebuild, table swap, and service restart in Task 1.
    Now the user verifies the production system is healthy with microsecond timestamps.

    What was built: Full ClickHouse rebuild with microsecond timestamps across 35M+ bars, atomic table swap, and service restart.

    How to verify:
    1. SSH to bigblack and verify bar count: `clickhouse-client --query "SELECT count() FROM opendeviationbar_cache.open_deviation_bars"`
    2. Verify microsecond scale: `clickhouse-client --query "SELECT min(close_time_us), max(close_time_us) FROM opendeviationbar_cache.open_deviation_bars"` -- both should be 16-digit numbers
    3. Run Stathera validation for BTCUSDT@250: `python scripts/detect_gaps.py --symbol BTCUSDT --threshold 250` -- should show zero gaps
    4. Check sidecar health: `curl http://localhost:8081/health` (or via SSH tunnel)
    5. Check heartbeat log: `journalctl -u opendeviationbar-heartbeat --since "5 min ago" --no-pager` -- should show healthy
    6. Verify kintsugi runs without error: `journalctl -u opendeviationbar-kintsugi --since "5 min ago" --no-pager`

    Resume signal: Type "approved" if all checks pass, or describe issues found.

  </action>
  <verify>
    <automated>echo "checkpoint:human-verify -- awaiting user approval"</automated>
  </verify>
  <acceptance_criteria>
    - User confirms bar count >= 35M
    - User confirms timestamps are 16-digit microsecond values
    - User confirms Stathera validation shows zero gaps
    - User confirms all services healthy
  </acceptance_criteria>
  <done>User has verified the rebuild is correct and all services are healthy</done>
</task>

<task type="auto">
  <name>Task 3: Update CLAUDE.md documentation network for microsecond conventions</name>
  <files>
    python/opendeviationbar/validation/day_mode/CLAUDE.md,
    python/opendeviationbar/clickhouse/CLAUDE.md,
    scripts/CLAUDE.md,
    CLAUDE.md
  </files>
  <read_first>
    python/opendeviationbar/validation/day_mode/CLAUDE.md,
    python/opendeviationbar/clickhouse/CLAUDE.md,
    scripts/CLAUDE.md,
    CLAUDE.md
  </read_first>
  <action>
    Update the CLAUDE.md hub-and-spoke documentation network to reflect the ms->us migration.

    **python/opendeviationbar/validation/day_mode/CLAUDE.md:**
    - Update the oracle query: all `open_time_ms / 1000` -> `open_time_us / 1000000`
    - Update column refs: `close_time_ms` -> `close_time_us`
    - Update invariant definition: `toDate(open_time_us / 1000000, 'UTC') == toDate(close_time_us / 1000000, 'UTC')`
    - Update "Real Production Output" table if column names shown
    - Update interpretation guide

    **python/opendeviationbar/clickhouse/CLAUDE.md:**
    - Update gap detection query: `close_time_ms` -> `close_time_us`, `/ 3600000` -> `/ 3600000000`
    - Update schema references
    - Update example queries

    **scripts/CLAUDE.md:**
    - Update any `close_time_ms` references
    - Update gap detection examples

    **CLAUDE.md (root hub):**
    - Update terminology table: `mutations_sync=1` and other entries that mention `close_time_ms`
    - Update ORDER BY documentation
    - Update the quick reference table if it mentions _ms columns
    - Add a note about v2.0 microsecond migration in the architecture section or a migration note

    **CRITICAL**: Only update documentation that references specific _ms column names or ms divisors. Do not rewrite entire files. Surgical find-and-replace of `close_time_ms` -> `close_time_us`, `open_time_ms` -> `open_time_us`, `/ 1000` -> `/ 1000000` in SQL examples, `gap_start_ms` -> `gap_start_us`, `gap_end_ms` -> `gap_end_us`.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -rn "close_time_ms\|open_time_ms" python/opendeviationbar/validation/day_mode/CLAUDE.md python/opendeviationbar/clickhouse/CLAUDE.md CLAUDE.md 2>/dev/null | wc -l</automated>
  </verify>
  <acceptance_criteria>
    - grep "close_time_ms" python/opendeviationbar/validation/day_mode/CLAUDE.md returns NO matches (only _us)
    - grep "close_time_us" python/opendeviationbar/validation/day_mode/CLAUDE.md returns matches in the oracle query
    - grep "/ 1000000" python/opendeviationbar/validation/day_mode/CLAUDE.md returns matches in SQL examples
    - grep "close_time_ms" python/opendeviationbar/clickhouse/CLAUDE.md returns NO matches in active query examples
    - Root CLAUDE.md references _us columns where appropriate
  </acceptance_criteria>
  <done>CLAUDE.md hub-and-spoke documentation network reflects microsecond conventions throughout, oracle queries use /1000000 divisors, all column refs updated to _us</done>
</task>

</tasks>

<verification>
- ClickHouse on bigblack has open_deviation_bars with _us columns and 35M+ rows
- Stathera validation passes (zero gaps, zero overlaps) for BTCUSDT@250
- Sidecar, kintsugi, and heartbeat services running and healthy
- Documentation updated to _us conventions
</verification>

<success_criteria>

- Full ClickHouse rebuild complete with 35M+ bars at microsecond precision
- Atomic table swap executed successfully
- Stathera continuity validated across entire dataset
- All services (sidecar, kintsugi, heartbeat) operate correctly with new schema
- CLAUDE.md documentation network updated to reflect \_us conventions
  </success_criteria>

<output>
After completion, create `.planning/phases/05-microsecond-clickhouse-rebuild/05-04-SUMMARY.md`
</output>
