# Kintsugi: Gap Reconciliation Package

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Autonomous gap reconciliation engine that discovers Stathera anomalies (trade-ID gaps and overlaps) in ClickHouse, repairs them via day-recompute (Vision-first, REST fallback), and verifies integrity. Named after the Japanese art of repairing pottery with gold. Modularized from a single `kintsugi.py` (Phase 01 v1.5).

---

## Files

| File              | Purpose                                                                                                                                                                                      |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `__init__.py`     | Package docstring, architecture diagram, re-exports of all public symbols from submodules for backward compatibility                                                                         |
| `orchestrator.py` | `kintsugi_pass()` -- the core discover-repair-verify-log cycle; yesterday signal processing; concurrent repair via ThreadPoolExecutor                                                        |
| `discovery.py`    | `discover_shards()` -- Stathera trade-ID audit query against ClickHouse; `Shard` and `KintsugiResult` dataclasses; `_shard_sort_key()`, `_select_repair_candidates()`, genesis gap detection |
| `repair.py`       | `repair_shard()` -- day-recompute via Vision ZIP or REST time-based fetch; threshold validation; today-guard enforcement                                                                     |
| `verify.py`       | `verify_repair()` -- post-repair Stathera continuity check via `lagInFrame()` ClickHouse query to confirm `tid_delta == 0`                                                                   |
| `daemon.py`       | `kintsugi_daemon()` -- long-running adaptive polling loop; `_trailing_edge_fill()` (absorbed from former gap-backfill daemon, #238 Phase 2)                                                  |
| `dead_letter.py`  | `replay_dead_letters()` -- reads dead-letter Parquet files from sidecar flush failures, replays to ClickHouse                                                                                |
| `telemetry.py`    | `_emit_telemetry()` (NDJSON to `kintsugi.ndjson`), `_maybe_log_repair()` (ClickHouse `kintsugi_log` table), `_notify_kintsugi_failures()` (Telegram), `_sd_notify()` (systemd watchdog)      |
| `_helpers.py`     | `_kintsugi_cfg()` (lazy config), `_get_worker_cache()` (thread-local ClickHouse connection), `_is_transient_error()`, NDJSON path constants                                                  |

---

## Discovery-Repair-Verify Flow

```
orchestrator.kintsugi_pass()
    |
    +--> discovery.discover_shards()       # Stathera audit: find tid_delta != 0
    |       returns list[Shard]
    |
    +--> _select_repair_candidates()       # Budget filter, dismiss window, sort
    |       sorted by _shard_sort_key()
    |
    +--> ThreadPoolExecutor(max_workers)
    |       +--> repair.repair_shard()     # Day-recompute per shard
    |       |       Vision ZIP -> _try_vision_day_replace()
    |       |       REST fallback -> _process_gap_fallback_starttime()
    |       |
    |       +--> verify.verify_repair()    # Confirm tid_delta == 0
    |
    +--> telemetry._maybe_log_repair()     # Log to kintsugi_log table
    +--> telemetry._emit_telemetry()       # NDJSON event
    +--> dead_letter.replay_dead_letters() # Replay sidecar dead-letters
```

## Writer Ownership

Kintsugi **owns historical gaps** (older than the sidecar's resume point). The sidecar owns the live stream from its resume TID forward.

## Parquet Trust Gate (v5.0)

Before shard-recompute from Parquet ticks, kintsugi verifies the Parquet file covers the full shard by checking trade-ID completeness.

**Behavior:**

- **Incomplete Parquet**: falls back to Vision/REST with WARNING log.
- **Complete Parquet**: uses Parquet as the repair source (fast path, no download needed).

**Implementation detail:** The gate runs AFTER ticks are loaded to avoid double-reading the Parquet file -- the ticks are already in memory for bar computation.

## Shard Sort Key

`_shard_sort_key()` in `discovery.py`: `(-gap_end_ms, symbol_rank)` -- newest gaps first, BTCUSDT priority within same time bucket. The dismissed window (30 days) skips already-attempted shards logged in `kintsugi_log`.

---

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) -- Parent: Python API layer
- [/CLAUDE.md](/CLAUDE.md) -- Hub: writer ownership (#280), Stathera invariant, kintsugi terminology
- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) -- Daemon config (`mise run kintsugi:daemon`), writer ownership details
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) -- ClickHouse write operations, `kintsugi_log` table
- [/python/opendeviationbar/sidecar/CLAUDE.md](/python/opendeviationbar/sidecar/CLAUDE.md) -- Sidecar dead-letter files consumed by `dead_letter.py`
