---
phase: 05-microsecond-clickhouse-rebuild
plan: 03
subsystem: runtime
tags: [microsecond, timestamp, sidecar, kintsugi, heartbeat, stathera, scripts]

requires:
  - phase: 05-microsecond-clickhouse-rebuild-01
    provides: us-native Rust dict output (open_time_us/close_time_us), DAY_US constant

provides:
  - us-native sidecar orchestration (bar dict access, cross-midnight filter, SQL)
  - us-native kintsugi gap discovery, repair, verify, and telemetry
  - us-native heartbeat Stathera checks, freshness, clickhouse stats
  - us-native production scripts (detect_gaps, cache_status, validate_*)

affects: [05-04-rebuild]

tech-stack:
  added: []
  patterns:
    - "All CH SQL: close_time_us / 1000000 for date derivation (not /1000)"
    - "Shard dataclass: gap_start_us/gap_end_us (not _ms)"
    - "Bar dict access: bar.get('close_time_us', 0) throughout runtime"
    - "Day constant: 86_400_000_000 (microseconds) in all day arithmetic"
    - "Boundary conversion: us->ms when calling TickStorage or Binance REST API"

key-files:
  created: []
  modified:
    - python/opendeviationbar/sidecar/__init__.py
    - python/opendeviationbar/sidecar/midnight.py
    - python/opendeviationbar/sidecar/http_server.py
    - python/opendeviationbar/sidecar/checkpoint.py
    - python/opendeviationbar/kintsugi/discovery.py
    - python/opendeviationbar/kintsugi/orchestrator.py
    - python/opendeviationbar/kintsugi/repair.py
    - python/opendeviationbar/kintsugi/verify.py
    - python/opendeviationbar/kintsugi/telemetry.py
    - scripts/detect_gaps.py
    - scripts/heartbeat/checks/stathera.py
    - scripts/heartbeat/checks/freshness.py
    - scripts/heartbeat/checks/clickhouse.py
    - scripts/heartbeat/checks/yesterday.py
    - scripts/cache_status.py
    - scripts/validate_ariadne.py
    - scripts/validate_dedup_hardening.py
    - scripts/verify_checkpoint_integrity.py
    - scripts/oracle_verify_ws_vs_rest.py
    - scripts/populate_full_cache.py
    - scripts/cleanup_oversized_bars.py

key-decisions:
  - "TickStorage and Binance REST API still expect ms -- convert us->ms at boundary"
  - "tick_cache_seeder.py and vision_oracle.py unchanged (Binance REST API timestamps, not CH columns)"
  - "Checkpoint migration: backward-compat reads old _ms keys and converts to _us"

patterns-established:
  - "us->ms boundary: kintsugi repair converts day_start_us//1000 when calling TickStorage.has_ticks() or _process_gap_fallback_starttime()"
  - "Legacy checkpoint migration: _load_checkpoint reads both timestamp_ms and close_time_ms keys, converts to _us"

requirements-completed: [USEC-05]

duration: 15min
completed: 2026-03-24
---

# Phase 05 Plan 03: Sidecar, Kintsugi, Heartbeat, and Scripts Microsecond Migration Summary

**All runtime components (sidecar, kintsugi, heartbeat, 12 production scripts) migrated from \_ms to \_us timestamp conventions with /1000000 SQL divisors and 86_400_000_000 day constants**

## Performance

- **Duration:** 15 min
- **Started:** 2026-03-24T06:16:20Z
- **Completed:** 2026-03-24T06:31:51Z
- **Tasks:** 2
- **Files modified:** 21

## Accomplishments

- Sidecar bar dict access, cross-midnight filter, and CH SQL all use \_us columns
- Kintsugi Shard dataclass fields renamed gap_start_ms/gap_end_ms to gap_start_us/gap_end_us
- Kintsugi discovery SQL, repair timestamp math, verify SQL, and telemetry INSERT all us-native
- All 5 heartbeat check modules migrated (stathera, freshness, clickhouse, yesterday, vision_oracle)
- 7 production scripts migrated (detect_gaps, cache_status, validate_ariadne, validate_dedup, verify_checkpoint, oracle_verify, populate_full_cache, cleanup_oversized_bars)
- Boundary conversion pattern established: us->ms when calling TickStorage or Binance REST API

## Task Commits

1. **Task 1: Migrate sidecar and kintsugi modules** - `a8b1b7c` (feat)
2. **Task 2: Migrate heartbeat checks and production scripts** - `1fc3abd` (feat)

## Files Created/Modified

**Sidecar (4 files):**

- `python/opendeviationbar/sidecar/__init__.py` - cross-midnight filter, data_freshness INSERT, pre-fill DELETE SQL
- `python/opendeviationbar/sidecar/midnight.py` - yesterday repair, sync_flush, heal_day_boundary SQL
- `python/opendeviationbar/sidecar/http_server.py` - SSE event IDs use close_time_us
- `python/opendeviationbar/sidecar/checkpoint.py` - last_bar_close_time_us key, backward-compat migration

**Kintsugi (5 files):**

- `python/opendeviationbar/kintsugi/discovery.py` - Shard dataclass, Stathera audit SQL, genesis discovery SQL
- `python/opendeviationbar/kintsugi/orchestrator.py` - yesterday priority shards, pass telemetry
- `python/opendeviationbar/kintsugi/repair.py` - day-recompute timestamp math, TickStorage boundary conversion
- `python/opendeviationbar/kintsugi/verify.py` - post-repair SQL ORDER BY
- `python/opendeviationbar/kintsugi/telemetry.py` - kintsugi_log INSERT column names

**Heartbeat checks (5 files):**

- `scripts/heartbeat/checks/stathera.py` - Stathera audit SQL, cross-midnight oracle, queue positions
- `scripts/heartbeat/checks/freshness.py` - intDiv(close_time_us, 1000000) for per-symbol freshness
- `scripts/heartbeat/checks/clickhouse.py` - staleness query
- `scripts/heartbeat/checks/yesterday.py` - orphan query
- `scripts/heartbeat/checks/vision_oracle.py` - unchanged (Binance REST API)

**Production scripts (7 files):**

- `scripts/detect_gaps.py` - dataclass fields, SQL, arithmetic
- `scripts/cache_status.py` - SQL min/max queries
- `scripts/validate_ariadne.py` - SQL column refs
- `scripts/validate_dedup_hardening.py` - INSERT column name
- `scripts/verify_checkpoint_integrity.py` - query filters and timestamp math
- `scripts/oracle_verify_ws_vs_rest.py` - CH column refs
- `scripts/populate_full_cache.py` - fromUnixTimestamp64Micro
- `scripts/cleanup_oversized_bars.py` - SQL column ref

## Decisions Made

- **TickStorage/REST boundary**: TickStorage.has_ticks() and Binance REST APIs still expect ms timestamps. Kintsugi repair converts us->ms (day_start_us // 1000) at the boundary rather than changing external APIs.
- **tick_cache_seeder.py unchanged**: The 86_400_000 constant there is for Binance REST API day ranges (ms), not CH column arithmetic. Changing it would break API calls.
- **vision_oracle.py unchanged**: Same reason -- Binance REST API timestamps are always ms.
- **Checkpoint backward compatibility**: \_load_checkpoint now migrates both legacy keys (timestamp_ms and close_time_ms) to close_time_us.

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 1 - Bug] Preserved ms timestamps for Binance REST API boundaries**

- **Found during:** Task 1 (kintsugi repair.py)
- **Issue:** Plan said to change all \_ms to_us, but \_process_gap_fallback_starttime() passes timestamps directly to Binance REST API which expects ms. Converting to us would break API calls.
- **Fix:** Added us->ms conversion (day_start_us // 1000) when calling external APIs that expect ms
- **Files modified:** python/opendeviationbar/kintsugi/repair.py
- **Verification:** Parameter names match function signature (start_ms, end_ms)
- **Committed in:** a8b1b7c (Task 1 commit)

**2. [Rule 1 - Bug] Preserved ms timestamps for TickStorage API boundaries**

- **Found during:** Task 1 (kintsugi repair.py)
- **Issue:** TickStorage.has_ticks() and read_ticks() expect ms timestamps per their docstrings
- **Fix:** Added local ms conversion variables in \_try_parquet_day_replace()
- **Files modified:** python/opendeviationbar/kintsugi/repair.py
- **Committed in:** a8b1b7c (Task 1 commit)

**3. [Rule 1 - Bug] tick_cache_seeder.py and vision_oracle.py left unchanged**

- **Found during:** Task 2
- **Issue:** Plan said to change 86_400_000 in tick_cache_seeder.py, but that constant is used for Binance REST API day ranges (always ms). Similarly, vision_oracle.py computes REST API timestamps.
- **Fix:** Left both files unchanged -- they deal exclusively with Binance REST API timestamps, not CH columns.
- **Committed in:** 1fc3abd (documented in commit message)

---

**Total deviations:** 3 auto-fixed (3 Bug - API boundary preservation)
**Impact on plan:** Essential for correctness. External APIs (Binance REST, TickStorage) still expect ms. Converting those to us would break production. No scope creep.

## Issues Encountered

None beyond the API boundary preservation documented above.

## Next Phase Readiness

- All sidecar, kintsugi, heartbeat, and script files now use \_us conventions
- Ready for plan 05-02 (CH schema migration) and 05-04 (rebuild)
- stathera_query.py (shared SQL module) still uses_ms -- will be migrated in plan 05-02

## Known Stubs

None -- all files have complete \_us migrations with no placeholder values.

## Self-Check: PASSED

- SUMMARY.md: FOUND
- Commit a8b1b7c (Task 1): FOUND
- Commit 1fc3abd (Task 2): FOUND
