---
phase: 05-microsecond-clickhouse-rebuild
plan: 03
type: execute
wave: 2
depends_on: [05-01]
files_modified:
  - python/opendeviationbar/sidecar/__init__.py
  - python/opendeviationbar/sidecar/midnight.py
  - python/opendeviationbar/sidecar/http_server.py
  - python/opendeviationbar/sidecar/checkpoint.py
  - python/opendeviationbar/kintsugi/discovery.py
  - python/opendeviationbar/kintsugi/orchestrator.py
  - python/opendeviationbar/kintsugi/repair.py
  - python/opendeviationbar/kintsugi/verify.py
  - python/opendeviationbar/kintsugi/telemetry.py
  - scripts/detect_gaps.py
  - scripts/heartbeat/checks/stathera.py
  - scripts/heartbeat/checks/freshness.py
  - scripts/heartbeat/checks/clickhouse.py
  - scripts/heartbeat/checks/yesterday.py
  - scripts/heartbeat/checks/vision_oracle.py
  - scripts/cache_status.py
  - scripts/tick_cache_seeder.py
  - scripts/validate_ariadne.py
  - scripts/validate_dedup_hardening.py
  - scripts/verify_checkpoint_integrity.py
  - scripts/oracle_verify_ws_vs_rest.py
  - scripts/populate_full_cache.py
  - scripts/cleanup_oversized_bars.py
autonomous: true
requirements: [USEC-05]

must_haves:
  truths:
    - "Sidecar __init__.py uses DAY_US and _us column names for all bar dict access and SQL"
    - "Kintsugi discovery.py uses _us column names in Stathera gap SQL and _us field names in data classes"
    - "All heartbeat check scripts use /1000000 divisors in ClickHouse SQL"
    - "detect_gaps.py uses _us column names and /1000000 divisors"
    - "No active production script contains _ms timestamp column references"
  artifacts:
    - path: "python/opendeviationbar/sidecar/__init__.py"
      provides: "us-native sidecar orchestration"
      contains: "open_time_us"
    - path: "python/opendeviationbar/kintsugi/discovery.py"
      provides: "us-native gap discovery SQL"
      contains: "close_time_us"
    - path: "scripts/heartbeat/checks/stathera.py"
      provides: "us-native Stathera health check"
      contains: "close_time_us"
    - path: "scripts/detect_gaps.py"
      provides: "us-native gap detection"
      contains: "close_time_us"
  key_links:
    - from: "python/opendeviationbar/sidecar/__init__.py"
      to: "python/opendeviationbar/clickhouse/bulk_operations.py"
      via: "bar dict keys must match CH column names"
      pattern: "close_time_us"
    - from: "python/opendeviationbar/kintsugi/telemetry.py"
      to: "python/opendeviationbar/clickhouse/schema.sql"
      via: "kintsugi_log INSERT column names"
      pattern: "gap_start_us"
---

<objective>
Migrate sidecar, kintsugi, heartbeat, and all production scripts from _ms to _us timestamp conventions.

Purpose: These runtime components read/write bars with timestamp columns and do date arithmetic. They must align with the new \_us schema before services can restart after the ClickHouse rebuild.
Output: All sidecar, kintsugi, heartbeat, and script files use_us column names and microsecond arithmetic.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-RESEARCH.md
@.planning/phases/05-microsecond-clickhouse-rebuild/05-01-SUMMARY.md

<interfaces>
<!-- Contracts from Plan 01 -->

From python/opendeviationbar/validation/day_mode/guards.py (after Plan 01):

```python
DAY_US = 86_400_000_000  # microseconds per UTC day
```

Key constant conversions:

- DAY_MS (86_400_000) -> DAY_US (86_400_000_000)
- HOUR: 3_600_000 -> 3_600_000_000
- SQL: / 1000 -> / 1000000 for all date derivation from timestamps
- pd.to_datetime: unit="ms" -> unit="us"
- Dict keys: "open_time_ms" -> "open_time_us", "close_time_ms" -> "close_time_us"
- kintsugi fields: "gap_start_ms" -> "gap_start_us", "gap_end_ms" -> "gap_end_us"
- data_freshness: "last_close_time_ms" -> "last_close_time_us"
  </interfaces>

</context>

<tasks>

<task type="auto">
  <name>Task 1: Migrate sidecar and kintsugi Python modules to microseconds</name>
  <files>
    python/opendeviationbar/sidecar/__init__.py,
    python/opendeviationbar/sidecar/midnight.py,
    python/opendeviationbar/sidecar/http_server.py,
    python/opendeviationbar/sidecar/checkpoint.py,
    python/opendeviationbar/kintsugi/discovery.py,
    python/opendeviationbar/kintsugi/orchestrator.py,
    python/opendeviationbar/kintsugi/repair.py,
    python/opendeviationbar/kintsugi/verify.py,
    python/opendeviationbar/kintsugi/telemetry.py
  </files>
  <read_first>
    python/opendeviationbar/sidecar/__init__.py,
    python/opendeviationbar/sidecar/midnight.py,
    python/opendeviationbar/sidecar/http_server.py,
    python/opendeviationbar/sidecar/checkpoint.py,
    python/opendeviationbar/kintsugi/discovery.py,
    python/opendeviationbar/kintsugi/orchestrator.py,
    python/opendeviationbar/kintsugi/repair.py,
    python/opendeviationbar/kintsugi/verify.py,
    python/opendeviationbar/kintsugi/telemetry.py
  </read_first>
  <action>
    **Sidecar __init__.py:**
    - All `"open_time_ms"` string refs -> `"open_time_us"`
    - All `"close_time_ms"` refs -> `"close_time_us"`
    - `86_400_000` -> `86_400_000_000` (DAY_MS -> DAY_US arithmetic)
    - `open_ms // 86_400_000` -> `open_us // 86_400_000_000` (and rename local vars)
    - `last_close_time_ms` in data_freshness SQL -> `last_close_time_us`
    - Any `/ 1000` in SQL strings for date derivation -> `/ 1000000`
    - Import DAY_US instead of DAY_MS if imported

    **Sidecar midnight.py:**
    - `toDate(close_time_ms / 1000)` -> `toDate(close_time_us / 1000000)` in all SQL
    - `86_400_000` -> `86_400_000_000`
    - `3_600_000` -> `3_600_000_000`
    - All column name refs: `_ms` -> `_us`

    **Sidecar http_server.py:**
    - `bar.get('close_time_ms', 0)` -> `bar.get('close_time_us', 0)`

    **Sidecar checkpoint.py:**
    - `"last_bar_close_time_ms"` -> `"last_bar_close_time_us"`

    **Kintsugi discovery.py:**
    - SQL: all `close_time_ms`/`open_time_ms` -> `_us`
    - Data class fields: `gap_start_ms`/`gap_end_ms` -> `gap_start_us`/`gap_end_us`
    - Arithmetic: `86_400_000` -> `86_400_000_000`, `3600000` -> `3_600_000_000`
    - SQL divisors: `/ 1000` -> `/ 1000000`

    **Kintsugi orchestrator.py:**
    - `yesterday_end_ms = yesterday_start_ms + 86_400_000 - 1` -> `yesterday_end_us = yesterday_start_us + 86_400_000_000 - 1`
    - All `_ms` variable names and field refs -> `_us`

    **Kintsugi repair.py:**
    - `gap_start_ms`/`gap_end_ms` field refs -> `gap_start_us`/`gap_end_us`

    **Kintsugi verify.py:**
    - `close_time_ms` in SQL window functions -> `close_time_us`
    - SQL divisors: `/ 1000` -> `/ 1000000`

    **Kintsugi telemetry.py:**
    - `gap_start_ms`/`gap_end_ms` in kintsugi_log INSERT SQL -> `gap_start_us`/`gap_end_us`
    - Field refs in data structures -> `_us`

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -rn "close_time_ms\|open_time_ms\|gap_start_ms\|gap_end_ms\|DAY_MS\|last_close_time_ms\|last_bar_close_time_ms" python/opendeviationbar/sidecar/ python/opendeviationbar/kintsugi/ 2>/dev/null | grep -v "\.pyc" | wc -l</automated>
  </verify>
  <acceptance_criteria>
    - grep -rn "close_time_ms\|open_time_ms" python/opendeviationbar/sidecar/ returns NO matches
    - grep -rn "close_time_ms\|open_time_ms" python/opendeviationbar/kintsugi/ returns NO matches
    - grep -rn "gap_start_ms\|gap_end_ms" python/opendeviationbar/kintsugi/ returns NO matches
    - grep -rn "DAY_MS" python/opendeviationbar/sidecar/ python/opendeviationbar/kintsugi/ returns NO matches
    - grep -rn "86_400_000[^_0]" python/opendeviationbar/sidecar/ returns NO matches (no bare ms DAY constant)
    - grep "close_time_us" python/opendeviationbar/sidecar/__init__.py returns matches
    - grep "gap_start_us" python/opendeviationbar/kintsugi/discovery.py returns matches
  </acceptance_criteria>
  <done>Sidecar and kintsugi modules use _us column names, _us field names, microsecond arithmetic, and /1000000 SQL divisors throughout</done>
</task>

<task type="auto">
  <name>Task 2: Migrate heartbeat checks and all production scripts to microseconds</name>
  <files>
    scripts/detect_gaps.py,
    scripts/heartbeat/checks/stathera.py,
    scripts/heartbeat/checks/freshness.py,
    scripts/heartbeat/checks/clickhouse.py,
    scripts/heartbeat/checks/yesterday.py,
    scripts/heartbeat/checks/vision_oracle.py,
    scripts/cache_status.py,
    scripts/tick_cache_seeder.py,
    scripts/validate_ariadne.py,
    scripts/validate_dedup_hardening.py,
    scripts/verify_checkpoint_integrity.py,
    scripts/oracle_verify_ws_vs_rest.py,
    scripts/populate_full_cache.py,
    scripts/cleanup_oversized_bars.py
  </files>
  <read_first>
    scripts/detect_gaps.py,
    scripts/heartbeat/checks/stathera.py,
    scripts/heartbeat/checks/freshness.py,
    scripts/heartbeat/checks/clickhouse.py,
    scripts/heartbeat/checks/yesterday.py,
    scripts/heartbeat/checks/vision_oracle.py,
    scripts/cache_status.py,
    scripts/tick_cache_seeder.py,
    scripts/validate_ariadne.py,
    scripts/validate_dedup_hardening.py,
    scripts/verify_checkpoint_integrity.py,
    scripts/oracle_verify_ws_vs_rest.py,
    scripts/populate_full_cache.py,
    scripts/cleanup_oversized_bars.py
  </read_first>
  <action>
    **scripts/detect_gaps.py:**
    - All `close_time_ms` in SQL -> `close_time_us`
    - `/ 1000` SQL divisors -> `/ 1000000`
    - `prev_close_time_ms` data class field -> `prev_close_time_us`
    - `/ 3600000` (ms -> hours) -> `/ 3_600_000_000` (us -> hours)

    **scripts/heartbeat/checks/stathera.py:**
    - 5+ occurrences of `open_time_ms / 1000` -> `open_time_us / 1000000`
    - `close_time_ms / 1000` -> `close_time_us / 1000000`
    - All column refs: `_ms` -> `_us`

    **scripts/heartbeat/checks/freshness.py:**
    - `intDiv(close_time_ms, 1000)` -> `intDiv(close_time_us, 1000000)`
    - `intDiv(open_time_ms, 1000)` -> `intDiv(open_time_us, 1000000)`
    - All column refs: `_ms` -> `_us`

    **scripts/heartbeat/checks/clickhouse.py:**
    - `intDiv(close_time_ms, 1000)` -> `intDiv(close_time_us, 1000000)`
    - All column refs: `_ms` -> `_us`

    **scripts/heartbeat/checks/yesterday.py:**
    - `toDate(open_time_ms / 1000)` -> `toDate(open_time_us / 1000000)`
    - All column refs: `_ms` -> `_us`

    **scripts/heartbeat/checks/vision_oracle.py:**
    - `86_400_000` -> `86_400_000_000`
    - Any column refs: `_ms` -> `_us`

    **scripts/cache_status.py:**
    - `close_time_ms` refs in SQL -> `close_time_us`

    **scripts/tick_cache_seeder.py:**
    - `86_400_000` -> `86_400_000_000`

    **Validation/utility scripts (validate_ariadne.py, validate_dedup_hardening.py, verify_checkpoint_integrity.py, oracle_verify_ws_vs_rest.py, populate_full_cache.py, cleanup_oversized_bars.py):**
    - For each: grep for `_time_ms`, `/ 1000`, `86_400_000`, `3_600_000`, `DAY_MS`
    - Apply the same pattern: column names _ms -> _us, divisors /1000 -> /1000000, constants *1000
    - `pd.to_datetime(..., unit="ms")` -> `unit="us"` where present

    **NOTE**: LOW-priority research scripts (duration_autocorrelation, tda_*, coarse_to_fine_cascade, microstructure_clickhouse, etc.) are out of scope for this task. They can be updated later if ever run again.

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -rn "close_time_ms\|open_time_ms\|DAY_MS" scripts/detect_gaps.py scripts/heartbeat/checks/ scripts/cache_status.py scripts/tick_cache_seeder.py 2>/dev/null | grep -v "\.pyc" | wc -l</automated>
  </verify>
  <acceptance_criteria>
    - grep -rn "close_time_ms\|open_time_ms" scripts/detect_gaps.py returns NO matches
    - grep -rn "close_time_ms\|open_time_ms" scripts/heartbeat/checks/stathera.py returns NO matches
    - grep -rn "close_time_ms\|open_time_ms" scripts/heartbeat/checks/freshness.py returns NO matches
    - grep -rn "intDiv.*1000[^0]" scripts/heartbeat/checks/ returns NO matches (all use 1000000)
    - grep "close_time_us" scripts/detect_gaps.py returns matches
    - grep "close_time_us" scripts/heartbeat/checks/stathera.py returns matches
    - grep "86_400_000_000" scripts/tick_cache_seeder.py returns matches (if DAY constant used)
  </acceptance_criteria>
  <done>All heartbeat checks, detect_gaps.py, and production scripts use _us column names and microsecond arithmetic with /1000000 divisors</done>
</task>

</tasks>

<verification>
- `grep -rn "close_time_ms\|open_time_ms\|DAY_MS" python/opendeviationbar/sidecar/ python/opendeviationbar/kintsugi/ scripts/detect_gaps.py scripts/heartbeat/checks/` returns zero matches
- `grep -rn "gap_start_ms\|gap_end_ms" python/opendeviationbar/kintsugi/` returns zero matches
</verification>

<success_criteria>

- Sidecar, kintsugi, and heartbeat modules have zero \_ms timestamp column references
- All SQL queries use /1000000 divisors for date derivation
- detect_gaps.py validates using_us columns
- Kintsugi data classes use gap_start_us/gap_end_us field names
- Production scripts (heartbeat, cache_status, tick_cache_seeder) all aligned to us
  </success_criteria>

<output>
After completion, create `.planning/phases/05-microsecond-clickhouse-rebuild/05-03-SUMMARY.md`
</output>
