# Phase 5: Microsecond ClickHouse Rebuild - Research

**Researched:** 2026-03-23
**Domain:** ClickHouse schema migration, Python timestamp arithmetic, full dataset rebuild
**Confidence:** HIGH

## Summary

Phase 5 migrates the entire ClickHouse schema and Python codebase from millisecond to microsecond timestamps. The Rust processor already outputs microsecond timestamps natively (USEC-04, Phase 4). The Parquet tick cache already stores microsecond timestamps (USEC-01/USEC-02, Phase 4). The gap is: (1) ClickHouse columns still store ms values with `_ms` suffixed names, (2) Python code still does ms arithmetic (`DAY_MS`, `// 1000` conversions), and (3) the 35M+ existing bars need rebuilding with us precision.

**The schema migration cannot use ALTER TABLE RENAME COLUMN** because the column values themselves change (ms -> us). The cleanest approach is: create a new table with `_us` columns, rebuild all data via `repair_direct_parquet.py`, swap tables atomically, then update all Python code to use the new column names and microsecond constants.

**Primary recommendation:** Three-wave approach: (Wave 1) Python constants and arithmetic migration, (Wave 2) ClickHouse schema + Rust helpers.rs + repair_direct_parquet.py alignment to stop dividing by 1000, (Wave 3) full 35M+ bar rebuild with Stathera validation.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices at Claude's discretion.

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

Key constraints from ROADMAP success criteria:

- USEC-03: ClickHouse timestamp columns store microsecond values with updated COMMENTs
- USEC-05: All Python timestamp arithmetic uses microseconds (no ms constants in active code)
- USEC-06: Full ClickHouse rebuild with 48 workers, Stathera validation across entire dataset

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID      | Description                                                                                                          | Research Support                                                                                                                                                                                        |
| ------- | -------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| USEC-03 | ClickHouse schema migration -- all timestamp columns renamed and stored as us. Column COMMENTs updated.              | Schema inventory below identifies 4 primary columns in `open_deviation_bars`, 2 in `data_freshness`, 1 in `population_checkpoints`, 2 in `kintsugi_log`. Migration strategy: new table + atomic swap.   |
| USEC-05 | Python API + sidecar + kintsugi us alignment -- all Python code that does timestamp arithmetic updated from ms to us | Full grep inventory below: 26 Python files in `python/`, 20+ files in `scripts/`, 40+ ClickHouse SQL queries with `/ 1000` divisor, 7 `pd.to_datetime(unit="ms")` calls, `DAY_MS` constant in guards.py |
| USEC-06 | Full ClickHouse rebuild -- reconstruct all 35M+ bars with us timestamps using repair_direct_parquet.py (48 workers)  | repair_direct_parquet.py already uses Arrow zero-copy us-native path; currently divides by 1000 at lines 314/317 to produce ms columns -- removing this division produces us values directly            |

</phase_requirements>

## Project Constraints (from CLAUDE.md)

- **Python 3.13 ONLY** -- never 3.14
- **ClickHouse COMMENTs are SSoT** -- every column MUST have a COMMENT documenting meaning, unit, computation
- **Leverage Rust** -- Python handles I/O only; Rust does heavy lifting
- **Kintsugi-first gap filling** -- all backfill through kintsugi
- **Writer ownership** -- sidecar owns today, kintsugi owns yesterday+older
- **No lightweight DELETE FROM** -- banned; use ALTER TABLE DELETE or DELETE IN PARTITION
- **Local-first CI/CD** -- `mise run check-full`, no GitHub Actions for tests

## Architecture Patterns

### Current State: ms Everywhere

The current data flow is:

```
Parquet (us) -> Rust processor (us internally) -> helpers.rs (/ 1000) -> Python dict (ms)
                                                -> Arrow output (us) -> repair_direct_parquet (/ 1000) -> CH (ms)
```

Two conversion points divide by 1000:

1. `src/helpers.rs:102-103` -- `opendeviationbar_to_dict()` divides `bar.open_time / 1000` and `bar.close_time / 1000`
2. `scripts/repair_direct_parquet.py:314,317` -- Arrow path divides `open_time // 1000` and `close_time // 1000`

### Target State: us End-to-End

```
Parquet (us) -> Rust processor (us internally) -> helpers.rs (NO division) -> Python dict (us)
                                                -> Arrow output (us) -> repair_direct_parquet (NO division) -> CH (us)
```

### ClickHouse Column Rename Decision

**REQUIREMENTS.md says "renamed and stored as us".** The `Out of Scope` section says: "ClickHouse column rename (keep `_ms` suffix for backward compat if needed) -- decided during USEC-03 implementation."

**Recommendation: Rename `_ms` to `_us` in both column names and Python code.** Rationale:

- The `_ms` suffix is documentation -- leaving it while storing us would be actively misleading
- All consumers are internal (sidecar, kintsugi, heartbeat, scripts) -- no external API stability concern
- A full rebuild already creates a new table; renaming is free during rebuild
- The BarOutputSchema contract, constants.py TIMESTAMP_COLUMNS, and all SQL queries change regardless

### Schema Migration Strategy

**Cannot ALTER TABLE in place** because:

1. Column values change (ms -> us, multiply by 1000)
2. Column names change (`_ms` -> `_us`)
3. CODEC specifications should be preserved
4. 35M+ rows need recomputation from source Parquet anyway (USEC-06)

**Strategy: New table + rebuild + atomic swap.**

1. Create `open_deviation_bars_v2` with `_us` columns
2. Run `repair_direct_parquet.py --all` against the v2 table (48 workers, ~3200 days/min)
3. Atomic swap: `RENAME TABLE open_deviation_bars TO open_deviation_bars_legacy, open_deviation_bars_v2 TO open_deviation_bars`
4. Validate Stathera continuity on the new table
5. Drop legacy table after validation

### Secondary Tables Requiring Migration

| Table                    | Columns Affected                                             | Strategy                                                                          |
| ------------------------ | ------------------------------------------------------------ | --------------------------------------------------------------------------------- |
| `open_deviation_bars`    | `open_time_ms`, `close_time_ms`, projection `prj_time_range` | New table + rebuild                                                               |
| `data_freshness`         | `last_close_time_ms`                                         | DROP + recreate (small table, sidecar repopulates)                                |
| `population_checkpoints` | `last_trade_timestamp_ms`                                    | ALTER RENAME + UPDATE (small table, ~100 rows)                                    |
| `kintsugi_log`           | `gap_start_ms`, `gap_end_ms`                                 | ALTER RENAME (historical log, values are contextual, not queried for computation) |

## Comprehensive File Inventory

### Python Files Requiring Changes (26 files in python/)

**Core constants and guards:**

| File                            | Change Type                  | Details                                                                                                                                                                             |
| ------------------------------- | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `validation/day_mode/guards.py` | Constant rename + arithmetic | `DAY_MS = 86_400_000` -> `DAY_US = 86_400_000_000`. All `// DAY_MS` -> `// DAY_US`. Dict key lookups `"open_time_ms"` -> `"open_time_us"`                                           |
| `constants.py`                  | Constant rename              | `TIMESTAMP_COLUMNS = ("close_time_ms", "open_time_ms")` -> `("close_time_us", "open_time_us")`                                                                                      |
| `validation/schemas.py`         | Schema update                | `BarOutputSchema` fields `open_time_ms`/`close_time_ms` -> `open_time_us`/`close_time_us`. Bounds change: `_TS_MS_MIN`/`_TS_MS_MAX` -> `_TS_US_MIN`/`_TS_US_MAX` (multiply by 1000) |

**ClickHouse layer (8 files):**

| File                             | Change Type                             | Details                                                                                                                        |
| -------------------------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `clickhouse/schema.sql`          | Full rewrite of column names + COMMENTs | 4 timestamp columns renamed, all COMMENTs updated to say "microseconds"                                                        |
| `clickhouse/bulk_operations.py`  | Column name refs + day arithmetic       | `"open_time_ms"` -> `"open_time_us"`, `// DAY_MS` -> `// DAY_US`, `_guard_close_time_ms_scale` -> `_guard_close_time_us_scale` |
| `clickhouse/query_operations.py` | SQL queries + pd.to_datetime            | All `close_time_ms` refs in SQL, `unit="ms"` -> `unit="us"`                                                                    |
| `clickhouse/cache.py`            | Column name refs                        | `close_time_ms` in SQL strings, data_freshness table                                                                           |
| `clickhouse/maintenance.py`      | SQL queries                             | `close_time_ms` in DELETE/cleanup queries                                                                                      |
| `clickhouse/stathera_query.py`   | SQL builder                             | Stathera gap detection SQL                                                                                                     |
| `clickhouse/migrations.py`       | SQL queries                             | `intDiv(close_time_ms, 1000)`                                                                                                  |
| `clickhouse/bar_count_api.py`    | SQL queries                             | `close_time_ms` refs                                                                                                           |

**Sidecar (4 files):**

| File                     | Change Type              | Details                                                                                                                               |
| ------------------------ | ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `sidecar/__init__.py`    | Column refs + arithmetic | `"open_time_ms"`, `86_400_000`, `open_ms // 86_400_000`, `last_close_time_ms` in data_freshness SQL                                   |
| `sidecar/midnight.py`    | SQL + arithmetic         | `toDate(close_time_ms / 1000)` -> `toDate(close_time_us / 1000000)`, `86_400_000` -> `86_400_000_000`, `3_600_000` -> `3_600_000_000` |
| `sidecar/http_server.py` | Dict key refs            | `bar.get('close_time_ms', 0)`                                                                                                         |
| `sidecar/checkpoint.py`  | Dict key refs            | `"last_bar_close_time_ms"`                                                                                                            |

**Kintsugi (5 files):**

| File                       | Change Type                           | Details                                                                                           |
| -------------------------- | ------------------------------------- | ------------------------------------------------------------------------------------------------- |
| `kintsugi/discovery.py`    | SQL queries + data class + arithmetic | `gap_start_ms`/`gap_end_ms` field names, `close_time_ms` in SQL, `86_400_000`, `3600000` divisors |
| `kintsugi/orchestrator.py` | Arithmetic                            | `yesterday_end_ms = yesterday_start_ms + 86_400_000 - 1`                                          |
| `kintsugi/repair.py`       | Field refs                            | `gap_start_ms`/`gap_end_ms`                                                                       |
| `kintsugi/verify.py`       | SQL queries                           | `close_time_ms` in window function SQL                                                            |
| `kintsugi/telemetry.py`    | Field refs + SQL                      | `gap_start_ms`/`gap_end_ms` in kintsugi_log INSERT                                                |

**Other Python (6 files):**

| File                                   | Change Type                    | Details                                                                                               |
| -------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------- |
| `backfill.py`                          | Import + arithmetic            | `from ...guards import DAY_MS` -> `DAY_US`                                                            |
| `orchestration/helpers.py`             | Remove us->ms conversion       | Lines 334/338: `(pl.col("open_time") // 1000).alias("open_time_ms")` -- REMOVE division, rename alias |
| `orchestration/open_deviation_bars.py` | Dict key refs + pd.to_datetime | `"close_time_ms"` refs, `unit="ms"`                                                                   |
| `processors/core.py`                   | pd.to_datetime                 | `unit="ms"`                                                                                           |
| `exness.py`                            | pd.to_datetime                 | `unit="ms"`                                                                                           |
| `rectify.py`                           | Column refs                    | `open_time_ms`/`close_time_ms`                                                                        |
| `checkpoint/models.py`                 | Field name                     | `last_trade_timestamp_ms`                                                                             |
| `checkpoint/storage.py`                | Field name                     | `last_trade_timestamp_ms`                                                                             |
| `config/backfill.py`                   | Comment                        | `86400000 = 1 day` in ms                                                                              |

### Rust Files Requiring Changes (1 file)

| File                     | Change                   | Details                                                                                                                                                                                   |
| ------------------------ | ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `src/helpers.rs:102-103` | Remove `/ 1000` division | `opendeviationbar_to_dict()` currently divides by 1000: `bar.open_time / 1000`. Change to output raw us: `bar.open_time`. Also rename dict keys from `"open_time_ms"` to `"open_time_us"` |

**Note:** The Arrow path in `arrow_bindings.rs` already outputs us-native `open_time`/`close_time` columns (no `_ms` suffix). Only `opendeviationbar_to_dict()` (dict path) needs updating.

### Scripts Requiring Changes (20+ files)

**Active production scripts:**

| File                                        | Change Type                     | Details                                                                          |
| ------------------------------------------- | ------------------------------- | -------------------------------------------------------------------------------- |
| `scripts/repair_direct_parquet.py`          | Remove us->ms conversion + SQL  | Lines 314/317: remove `// 1000` division. Line 178: `open_time_ms / 1000` in SQL |
| `scripts/detect_gaps.py`                    | SQL queries + data class fields | `close_time_ms` refs, `/ 1000` divisors, `prev_close_time_ms`                    |
| `scripts/heartbeat/checks/stathera.py`      | SQL queries                     | 5+ occurrences of `open_time_ms / 1000` and `close_time_ms / 1000`               |
| `scripts/heartbeat/checks/freshness.py`     | SQL queries                     | `intDiv(close_time_ms, 1000)`, `intDiv(open_time_ms, 1000)`                      |
| `scripts/heartbeat/checks/clickhouse.py`    | SQL queries                     | `intDiv(close_time_ms, 1000)`                                                    |
| `scripts/heartbeat/checks/yesterday.py`     | SQL queries                     | `toDate(open_time_ms / 1000)`                                                    |
| `scripts/cache_status.py`                   | SQL queries                     | `close_time_ms` refs                                                             |
| `scripts/tick_cache_seeder.py`              | Arithmetic                      | `86_400_000`                                                                     |
| `scripts/heartbeat/checks/vision_oracle.py` | Arithmetic                      | `86_400_000`                                                                     |

**Research/analysis scripts (lower priority):**

| File                                     | Priority | Notes                       |
| ---------------------------------------- | -------- | --------------------------- |
| `scripts/duration_autocorrelation*.py`   | LOW      | Research scripts, can defer |
| `scripts/tda_*.py`                       | LOW      | TDA research scripts        |
| `scripts/coarse_to_fine_cascade.py`      | LOW      | Analysis                    |
| `scripts/microstructure_clickhouse.py`   | LOW      | Analysis                    |
| `scripts/direction_patterns_reaudit.py`  | LOW      | Analysis                    |
| `scripts/hurst_multi_estimator_audit.py` | LOW      | Analysis                    |

**Utility scripts:**

| File                                     | Priority | Notes               |
| ---------------------------------------- | -------- | ------------------- |
| `scripts/validate_ariadne.py`            | MEDIUM   | Validation script   |
| `scripts/validate_dedup_hardening.py`    | MEDIUM   | Validation          |
| `scripts/verify_checkpoint_integrity.py` | MEDIUM   | Validation          |
| `scripts/oracle_verify_ws_vs_rest.py`    | MEDIUM   | Oracle verification |
| `scripts/populate_full_cache.py`         | MEDIUM   | Cache population    |
| `scripts/cleanup_oversized_bars.py`      | MEDIUM   | Maintenance         |
| `scripts/upload_eurusd_to_clickhouse.py` | LOW      | One-off             |
| `scripts/validate_plugin_e2e.py`         | LOW      | Plugin testing      |

### ClickHouse SQL Query Pattern Changes

All ClickHouse SQL queries that derive dates from timestamps need updating:

| Old Pattern                               | New Pattern                                  | Occurrences |
| ----------------------------------------- | -------------------------------------------- | ----------- |
| `toDate(open_time_ms / 1000, 'UTC')`      | `toDate(open_time_us / 1000000, 'UTC')`      | ~15         |
| `toDate(close_time_ms / 1000, 'UTC')`     | `toDate(close_time_us / 1000000, 'UTC')`     | ~10         |
| `toDateTime(close_time_ms / 1000, 'UTC')` | `toDateTime(close_time_us / 1000000, 'UTC')` | ~8          |
| `intDiv(close_time_ms, 1000)`             | `intDiv(close_time_us, 1000000)`             | ~5          |
| `fromUnixTimestamp64Milli(close_time_ms)` | `fromUnixTimestamp64Micro(close_time_us)`    | ~3          |
| `pd.to_datetime(..., unit="ms")`          | `pd.to_datetime(..., unit="us")`             | 7           |

### Test Files Requiring Changes (14 files)

Files referencing `open_time_ms`/`close_time_ms`/`DAY_MS`:

- `tests/test_day_mode_invariant.py`
- `tests/test_data_contracts.py`
- `tests/test_streaming.py`
- `tests/test_rust_bindings.py`
- `tests/test_exchange_sessions.py`
- `tests/test_multi_threshold.py`
- `tests/test_overlap_strategy.py`
- `tests/test_sync_flush_atomic_replace.py`
- `tests/test_ariadne_telemetry.py`
- `tests/test_oracle_regression.py`
- `tests/test_heal_scoping.py`
- `tests/test_rectification.py`
- `tests/test_checkpoint_edge_cases.py`
- `tests/test_oversized_bar_guard.py`

### CLAUDE.md Network Updates

Documentation files referencing ms patterns:

- `python/opendeviationbar/validation/day_mode/CLAUDE.md` -- oracle queries use `/ 1000`
- `python/opendeviationbar/clickhouse/CLAUDE.md` -- gap detection query, examples
- `scripts/CLAUDE.md` -- references to `close_time_ms`
- `CLAUDE.md` (root) -- terminology, API examples mentioning `close_time_ms`

## Don't Hand-Roll

| Problem                 | Don't Build                   | Use Instead                               | Why                                                                    |
| ----------------------- | ----------------------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
| Full dataset rebuild    | Custom migration script       | `repair_direct_parquet.py --all`          | Already proven at 3200+ days/min with 48 workers, Arrow zero-copy path |
| Table swap              | Manual INSERT INTO...SELECT   | `RENAME TABLE` (atomic)                   | ClickHouse RENAME is metadata-only, instant, atomic                    |
| Stathera validation     | New validation query          | `scripts/detect_gaps.py` (updated for us) | Existing tool already validates entire dataset                         |
| Column rename migration | Multiple ALTER TABLE commands | New table DDL + rebuild                   | Cleaner than incremental ALTER TABLE statements                        |

## Common Pitfalls

### Pitfall 1: Division Factor Mismatch

**What goes wrong:** Some code paths get `/ 1000` (ms->s) while others get `/ 1000000` (us->s), producing wrong dates.
**Why it happens:** The change from `close_time_ms / 1000` to `close_time_us / 1000000` is easy to miss in SQL string templates.
**How to avoid:** Grep for ALL instances of `/ 1000` in SQL strings; every one needs review. The new divisor is always `/ 1000000` (us -> seconds).
**Warning signs:** ClickHouse dates in year 2001 (too small) or year 50000+ (too large).

### Pitfall 2: Sidecar Downtime During Migration

**What goes wrong:** Sidecar writes bars with ms timestamps to the new us schema, or vice versa.
**Why it happens:** Code and schema must be deployed atomically.
**How to avoid:** Stop sidecar + kintsugi before the table swap. Deploy code + swap table together. Restart services.
**Warning signs:** Stathera gaps at the migration boundary.

### Pitfall 3: data_freshness Table Mismatch

**What goes wrong:** Sidecar writes us values to `last_close_time_ms` (old column name) or ms values to `last_close_time_us` (new column).
**Why it happens:** data_freshness is a small table forgotten in migration.
**How to avoid:** Recreate data_freshness with `_us` column. Sidecar repopulates on first bar write.

### Pitfall 4: Kintsugi Shard Discovery After Migration

**What goes wrong:** Kintsugi discovers "gaps" because old gap_start_ms/gap_end_ms in kintsugi_log don't match new us values.
**Why it happens:** kintsugi_log stores historical gap timestamps; the 30-day dismiss window compares these.
**How to avoid:** TRUNCATE kintsugi_log before restarting kintsugi (it has a 90-day TTL anyway). Or simply let old entries age out -- kintsugi re-discovers from scratch.

### Pitfall 5: BarOutputSchema Validation Rejects us Timestamps

**What goes wrong:** Pandera schema rejects all bars because us timestamps exceed `_TS_MS_MAX = 2_500_000_000_000`.
**Why it happens:** BarOutputSchema bounds were designed for ms (13-digit values). us values are 16-digit.
**How to avoid:** Update `_TS_MS_MIN`/`_TS_MS_MAX` to `_TS_US_MIN`/`_TS_US_MAX` with appropriate bounds (1e15 to 2.5e15).

### Pitfall 6: Checkpoint last_trade_timestamp_ms

**What goes wrong:** Existing checkpoints have ms timestamps. Resuming with us code skips too many or too few trades.
**Why it happens:** Checkpoint records in CH and local files store `last_trade_timestamp_ms`.
**How to avoid:** For the rebuild via repair_direct_parquet.py, checkpoints are irrelevant (it rebuilds from scratch). For ongoing operations, the field name change in models.py + a simple multiply-by-1000 migration on the CH checkpoint table is sufficient.

## Code Examples

### New DDL for open_deviation_bars (key columns only)

```sql
CREATE TABLE IF NOT EXISTS opendeviationbar_cache.open_deviation_bars (
    symbol LowCardinality(String),
    threshold_decimal_bps UInt32,

    -- Bar close time in microseconds since Unix epoch (was close_time_ms before v2.0)
    close_time_us Int64 CODEC(Delta, ZSTD)
        COMMENT 'Bar close time in microseconds since Unix epoch. Use intDiv(close_time_us, 1000000) for epoch seconds.',
    -- Bar open time in microseconds since Unix epoch (was open_time_ms before v2.0)
    open_time_us Int64 DEFAULT 0 CODEC(Delta, ZSTD)
        COMMENT 'Bar open time in microseconds since Unix epoch. Use intDiv(open_time_us, 1000000) for epoch seconds.',
    -- ... rest of columns unchanged ...
)
```

### New Python constant

```python
# python/opendeviationbar/validation/day_mode/guards.py
DAY_US = 86_400_000_000  # microseconds per UTC day
```

### Updated helpers.rs (dict path)

```rust
// src/helpers.rs - opendeviationbar_to_dict()
dict.set_item(ikey(py, "open_time_us"), bar.open_time)?;   // was: bar.open_time / 1000
dict.set_item(ikey(py, "close_time_us"), bar.close_time)?;  // was: bar.close_time / 1000
```

### Updated repair_direct_parquet.py (Arrow path)

```python
# scripts/repair_direct_parquet.py - _process_day()
# Arrow path: open_time/close_time already in us -- just rename column
if "open_time" in bars_df.columns:
    rename_exprs.append(pl.col("open_time").alias("open_time_us"))  # was: // 1000 -> open_time_ms
    drop_cols.append("open_time")
if "close_time" in bars_df.columns:
    rename_exprs.append(pl.col("close_time").alias("close_time_us"))  # was: // 1000 -> close_time_ms
    drop_cols.append("close_time")
```

### Updated ClickHouse SQL pattern

```python
# Old:
f"toDate(open_time_ms / 1000, 'UTC') = today()"
# New:
f"toDate(open_time_us / 1000000, 'UTC') = today()"
```

### Updated pd.to_datetime pattern

```python
# Old:
df["timestamp"] = pd.to_datetime(df["close_time_ms"], unit="ms", utc=True)
# New:
df["timestamp"] = pd.to_datetime(df["close_time_us"], unit="us", utc=True)
```

## State of the Art

| Old Approach                   | Current Approach                             | When Changed               | Impact                    |
| ------------------------------ | -------------------------------------------- | -------------------------- | ------------------------- |
| ms timestamps in Parquet ticks | us timestamps                                | Phase 4 (USEC-01, USEC-02) | Tick cache is already us  |
| ms timestamps in Rust output   | us timestamps internally, / 1000 at boundary | Phase 4 (USEC-04)          | Rust is us-native         |
| ms in ClickHouse               | **This phase changes to us**                 | Phase 5 (USEC-03)          | End-to-end us alignment   |
| DAY_MS Python constant         | **This phase changes to DAY_US**             | Phase 5 (USEC-05)          | Python arithmetic aligned |

## Validation Architecture

### Test Framework

| Property           | Value                          |
| ------------------ | ------------------------------ |
| Framework          | pytest                         |
| Config file        | `pyproject.toml` [tool.pytest] |
| Quick run command  | `mise run test-py`             |
| Full suite command | `mise run check-full`          |

### Phase Requirements to Test Map

| Req ID  | Behavior                                         | Test Type   | Automated Command                                                         | File Exists?       |
| ------- | ------------------------------------------------ | ----------- | ------------------------------------------------------------------------- | ------------------ |
| USEC-03 | CH schema stores us values with updated COMMENTs | integration | `pytest tests/test_data_contracts.py -x`                                  | Yes (needs update) |
| USEC-05 | Python uses us constants, no ms in active code   | unit        | `pytest tests/test_day_mode_invariant.py tests/test_data_contracts.py -x` | Yes (needs update) |
| USEC-06 | Full rebuild produces valid bars                 | smoke       | `python scripts/detect_gaps.py --symbol BTCUSDT --threshold 250`          | Yes (needs update) |

### Sampling Rate

- **Per task commit:** `mise run test-py`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green + Stathera validation on rebuilt dataset

### Wave 0 Gaps

- [ ] Update `tests/test_data_contracts.py` -- BarOutputSchema bounds for us
- [ ] Update `tests/test_day_mode_invariant.py` -- DAY_US constant, us column names
- [ ] Update all 14 test files referencing `_time_ms` column names

## Open Questions

1. **Column rename in kintsugi_log: rename or keep?**
   - What we know: `kintsugi_log` has `gap_start_ms`/`gap_end_ms` columns storing ms values. These are historical audit entries with 90-day TTL.
   - What's unclear: Whether to rename columns (breaking historical log queries) or keep as-is (misleading column names).
   - Recommendation: Rename to `_us`. TRUNCATE the table -- it's a 90-day TTL audit log. Kintsugi rediscovers all shards from scratch on next pass anyway.

2. **population_checkpoints: migrate or reset?**
   - What we know: Small table (~100 rows) with `last_trade_timestamp_ms` storing ms values.
   - What's unclear: Whether any active population runs depend on these checkpoints.
   - Recommendation: Reset (TRUNCATE). The full rebuild via repair_direct_parquet.py bypasses checkpoint logic entirely. Future populate_cache_resumable calls will create new us-native checkpoints.

3. **Backward compatibility period?**
   - What we know: All consumers are internal (no external API).
   - Recommendation: No backward compat needed. Atomic cutover: stop services, deploy code, swap tables, restart.

## Environment Availability

| Dependency               | Required By                          | Available         | Version | Fallback |
| ------------------------ | ------------------------------------ | ----------------- | ------- | -------- |
| ClickHouse               | Schema migration, rebuild target     | Remote (bigblack) | Native  | --       |
| Parquet tick cache       | Rebuild source data                  | On bigblack       | --      | --       |
| repair_direct_parquet.py | Full rebuild                         | In repo           | Current | --       |
| maturin                  | Rust rebuild after helpers.rs change | Local             | --      | --       |

**Missing dependencies with no fallback:** None.

## Sources

### Primary (HIGH confidence)

- `python/opendeviationbar/clickhouse/schema.sql` -- current DDL, all column definitions
- `src/helpers.rs` -- Rust-to-Python dict conversion with `/ 1000` at lines 102-103
- `scripts/repair_direct_parquet.py` -- Arrow rebuild pipeline with `// 1000` at lines 314/317
- Full grep inventory of `open_time_ms`, `close_time_ms`, `DAY_MS` across codebase

### Secondary (MEDIUM confidence)

- ClickHouse documentation on `RENAME TABLE` atomicity (verified from training data, consistent with known behavior)

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- no new libraries needed, purely internal migration
- Architecture: HIGH -- strategy is straightforward (new table + rebuild + swap), using proven tools
- Pitfalls: HIGH -- all pitfalls identified from direct codebase inspection and prior Phase 4 experience

**Research date:** 2026-03-23
**Valid until:** 2026-04-23 (stable -- no external dependency changes expected)
