---
phase: 02-kintsugi-rest-ch-connection
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/kintsugi.py
  - python/opendeviationbar/config/kintsugi.py
autonomous: true
requirements:
  - STAB-03
  - STAB-04
  - STAB-05
  - STAB-06

must_haves:
  truths:
    - "Kintsugi worker threads reuse one CH connection per thread, not one per function call"
    - "REST-only shards are capped at MAX_CONCURRENT_REST workers (default 2)"
    - "Parquet-cached and Vision-available shards are not blocked by REST workers"
    - "fd count stays at ~4N+50 for N workers during 24-worker repairs"
  artifacts:
    - path: "python/opendeviationbar/kintsugi.py"
      provides: "Thread-local CH cache + REST concurrency limiting"
      contains: "_thread_local"
    - path: "python/opendeviationbar/config/kintsugi.py"
      provides: "max_concurrent_rest config field"
      contains: "max_concurrent_rest"
  key_links:
    - from: "python/opendeviationbar/kintsugi.py"
      to: "threading.local()"
      via: "_get_worker_cache() reused in _try_parquet_day_replace and _try_vision_day_replace"
      pattern: "_get_worker_cache\\(\\)"
    - from: "python/opendeviationbar/kintsugi.py"
      to: "python/opendeviationbar/config/kintsugi.py"
      via: "cfg.max_concurrent_rest governs REST shard pool size"
      pattern: "max_concurrent_rest"
---

<objective>
Eliminate fd exhaustion and hour-long REST repairs in kintsugi by: (1) adding thread-local CH cache reuse so each worker thread gets ONE OpenDeviationBarCache for its lifetime, and (2) splitting concurrency control into three tiers (Parquet-cached / Vision-available / REST-only) so REST-only shards don't saturate the thread pool.

Purpose: kintsugi currently creates a new OpenDeviationBarCache() on every CH write inside worker threads (lines 1160, 1295, 1448, 1558, 2331), causing fd exhaustion at 24 workers. REST-only shards take 1h+ each but share the same 24-worker pool as fast Parquet/Vision shards, diluting throughput.

Output: Modified kintsugi.py with thread-local cache and 3-tier concurrency, plus new config field.
</objective>

<execution_context>
@/Users/terryli/.claude/get-shit-done/workflows/execute-plan.md
@/Users/terryli/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md

@python/opendeviationbar/kintsugi.py
@python/opendeviationbar/config/kintsugi.py
@scripts/repair_direct_parquet.py

<interfaces>
<!-- Thread-local pattern from repair_direct_parquet.py (lines 73-94) — exact template -->
From scripts/repair_direct_parquet.py:
```python
_thread_local = threading.local()

def \_get_worker_cache() -> object:
cache = getattr(\_thread_local, "cache", None)
if cache is None:
from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
\_thread_local.cache = OpenDeviationBarCache()
cache =\_thread_local.cache
return cache

````

From python/opendeviationbar/config/kintsugi.py:
```python
class KintsugiConfig(BaseSettings):
    max_concurrent_repairs: int = 4
    max_concurrent_parquet: int = 64
    # (new field to add: max_concurrent_rest)
````

From python/opendeviationbar/kintsugi.py:

```python
def _shard_is_parquet_cached(shard: Shard) -> bool:
    # Currently checks local Parquet only — no Vision check
def _try_parquet_day_replace(*, shard, day_start_ms, day_end_ms, include_microstructure) -> int:
    # Line 1160: with OpenDeviationBarCache() as cache:  <-- fd leak
def _try_vision_day_replace(*, shard, repair_date, include_microstructure) -> int:
    # Line 1295: with OpenDeviationBarCache() as cache:  <-- fd leak
def verify_repair(shard: Shard) -> bool:
    # Line 1448: with OpenDeviationBarCache() as cache:  <-- fd leak
def _log_repair(result: KintsugiResult) -> None:
    # Line 1558: with OpenDeviationBarCache() as cache:  <-- fd leak
def replay_dead_letters() -> int:
    # Line 2331: with OpenDeviationBarCache() as cache:  <-- fd leak (called from main thread)
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Add thread-local CH cache to kintsugi worker functions</name>
  <files>python/opendeviationbar/kintsugi.py</files>
  <read_first>
    - python/opendeviationbar/kintsugi.py (full file — understand all OpenDeviationBarCache() call sites)
    - scripts/repair_direct_parquet.py (lines 73-94 — the exact threading.local() + _get_worker_cache() pattern to replicate)
  </read_first>
  <action>
Add thread-local CH cache reuse to kintsugi.py, following the exact pattern from repair_direct_parquet.py:

1. At module level (near the existing imports around line 50-60), add:

```python
import threading
_thread_local = threading.local()

def _get_worker_cache():
    """Get or create a thread-local OpenDeviationBarCache.

    Each worker thread gets ONE cache for its lifetime, eliminating
    per-call connection overhead (urllib3.PoolManager with 20 fds each).
    """
    cache = getattr(_thread_local, "cache", None)
    if cache is None:
        from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
        _thread_local.cache = OpenDeviationBarCache()
        cache = _thread_local.cache
    return cache
```

1. Replace `with OpenDeviationBarCache() as cache:` with `cache = _get_worker_cache()` at these WORKER-THREAD call sites (called from ThreadPoolExecutor in \_repair or kintsugi_pass):
   - Line ~1160 in `_try_parquet_day_replace()` — change `with OpenDeviationBarCache() as cache:` to `cache = _get_worker_cache()` and un-indent the `written = cache.store_bars_batch(...)` block
   - Line ~1295 in `_try_vision_day_replace()` — same replacement
   - Line ~1448 in `verify_repair()` — same replacement
   - Line ~1558 in `_log_repair()` — same replacement

2. Do NOT change these MAIN-THREAD call sites (they run once, not in a hot loop):
   - Line ~251 in `discover_shards()` — main thread, single query
   - Line ~437 in `_discover_shards_stathera()` — main thread, single query
   - Line ~561 in `_get_trailing_edge_shards()` — main thread
   - Line ~611 in `_get_dismissed_set()` — main thread
   - Line ~633 in `_get_rate_limited_set()` — main thread
   - Line ~2331 in `replay_dead_letters()` — main thread, infrequent

3. The removal of `with ... as cache:` context manager means the cache stays open for the thread's lifetime. This is intentional — the thread exits when the ThreadPoolExecutor shuts down, and Python's GC closes the connection. The thread-local pattern is proven in repair_direct_parquet.py at 24 workers.
   </action>

  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "_get_worker_cache" python/opendeviationbar/kintsugi.py | head -20 && grep -c "with OpenDeviationBarCache() as cache:" python/opendeviationbar/kintsugi.py</automated>
  </verify>
  <acceptance_criteria>
    - grep "_thread_local = threading.local()" python/opendeviationbar/kintsugi.py returns exactly 1 match
    - grep "def _get_worker_cache" python/opendeviationbar/kintsugi.py returns exactly 1 match
    - grep "_get_worker_cache()" python/opendeviationbar/kintsugi.py returns >= 4 matches (the 4 worker-thread call sites)
    - grep "with OpenDeviationBarCache() as cache:" python/opendeviationbar/kintsugi.py returns exactly 6 matches (the 6 main-thread call sites that remain unchanged)
    - mise run test-py passes (specifically kintsugi-related tests)
  </acceptance_criteria>
  <done>Worker-thread functions (_try_parquet_day_replace, _try_vision_day_replace, verify_repair, _log_repair) use thread-local CH cache. Main-thread functions unchanged. fd count per worker drops from ~80 (4 calls x 20 fds/PoolManager) to ~20 (1 PoolManager reused).</done>
</task>

<task type="auto">
  <name>Task 2: Add 3-tier concurrency control with REST limiting and Vision availability detection</name>
  <files>python/opendeviationbar/kintsugi.py, python/opendeviationbar/config/kintsugi.py</files>
  <read_first>
    - python/opendeviationbar/kintsugi.py (lines 648-668 — _shard_is_parquet_cached; lines 1686-1702 — current 2-tier concurrency logic in kintsugi_pass)
    - python/opendeviationbar/config/kintsugi.py (full file — KintsugiConfig fields)
    - python/opendeviationbar/backfill.py (lines 69-101 — _is_vision_cached_unavailable pattern for checking Vision availability via Redis negative cache)
  </read_first>
  <action>
Implement 3-tier concurrency control: Parquet-cached (64 workers) / Vision-available (24 workers) / REST-only (2 workers).

**Step 1: Add config field** in `python/opendeviationbar/config/kintsugi.py`:

Add after the `max_concurrent_parquet` field (line ~71):

```python
    # Max concurrent REST-only shard repairs (no Parquet cache, no Vision).
    # REST repairs are rate-limited by Binance API (1200 weight/min) and take
    # 30-60 min each. Capping at 2 prevents REST workers from consuming all
    # thread pool slots while Parquet/Vision shards wait.
    max_concurrent_rest: int = 2
```

Add docstring entry after `OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_PARQUET`:

```
    OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REST : int
        Max concurrent REST-only shard repairs (default: 2). REST repairs are
        rate-limited by Binance API and take 30-60 min each.
```

**Step 2: Add Vision availability classifier** in `kintsugi.py`:

Add a new function after `_shard_is_parquet_cached()` (line ~668):

```python
def _shard_has_vision_available(shard: Shard) -> bool:
    """Return True if Vision archives are likely available for this shard's date range.

    Checks the Redis negative cache (populated by backfill.py on Vision 404).
    If no 404 is cached, Vision is presumed available — the actual HTTP check
    happens at repair time. This is a fast heuristic for concurrency planning,
    not a guarantee.
    """
    try:
        from opendeviationbar.backfill import _is_vision_cached_unavailable

        # Check each Ouroboros day in the shard range
        lo_ms = min(shard.gap_start_ms, shard.gap_end_ms)
        hi_ms = max(shard.gap_start_ms, shard.gap_end_ms)
        gap_start_date = datetime.fromtimestamp(lo_ms / 1000, tz=UTC).date()
        gap_end_date = datetime.fromtimestamp(hi_ms / 1000, tz=UTC).date()

        current = gap_start_date
        while current <= gap_end_date:
            if _is_vision_cached_unavailable(shard.symbol, current.strftime("%Y-%m-%d")):
                return False  # At least one day has a cached Vision 404
            current += timedelta(days=1)
        return True  # No cached 404s — Vision presumed available
    except Exception:
        return True  # Optimistic: assume Vision available if check fails
```

**Step 3: Replace 2-tier concurrency with 3-tier** in `kintsugi_pass()`:

Replace the current logic at lines ~1690-1702:

```python
    # Current (2-tier):
    cfg = _kintsugi_cfg()
    if candidates and all(_shard_is_parquet_cached(s) for s in candidates):
        max_concurrent = cfg.max_concurrent_parquet
    else:
        max_concurrent = cfg.max_concurrent_repairs
```

With 3-tier logic that classifies candidates and dispatches in groups:

```python
    cfg = _kintsugi_cfg()

    # Classify candidates into 3 tiers for concurrency control.
    # Parquet-cached: local I/O only, highest parallelism.
    # Vision-available: single HTTP GET per day, medium parallelism.
    # REST-only: hundreds of paginated REST calls, rate-limited, low parallelism.
    parquet_candidates = [s for s in candidates if _shard_is_parquet_cached(s)]
    remaining = [s for s in candidates if s not in parquet_candidates]  # use identity
    vision_candidates = [s for s in remaining if _shard_has_vision_available(s)]
    rest_candidates = [s for s in remaining if s not in vision_candidates]

    if rest_candidates:
        logger.info(
            "kintsugi concurrency: %d Parquet, %d Vision, %d REST-only candidates",
            len(parquet_candidates),
            len(vision_candidates),
            len(rest_candidates),
        )

    # Determine max_concurrent based on the slowest tier present.
    # If ANY REST-only shards exist, limit the REST portion to max_concurrent_rest.
    # Parquet and Vision shards still use their own higher limits.
    if not candidates:
        max_concurrent = cfg.max_concurrent_repairs
    elif parquet_candidates and not remaining:
        # All Parquet-cached — highest parallelism
        max_concurrent = cfg.max_concurrent_parquet
        logger.debug(
            "all %d candidates are Parquet-cached -> using max_concurrent_parquet=%d",
            len(candidates),
            max_concurrent,
        )
    elif not rest_candidates:
        # Parquet + Vision mix, no REST — use standard limit
        max_concurrent = cfg.max_concurrent_repairs
    else:
        # Has REST-only shards — cap REST workers, let fast shards use remaining slots.
        # Strategy: run fast shards (Parquet + Vision) first at full concurrency,
        # then REST shards at capped concurrency.
        # For simplicity in the existing ThreadPoolExecutor pattern, reorder
        # candidates so fast shards run first, and cap total workers to avoid
        # REST shards blocking fast ones.
        max_concurrent = cfg.max_concurrent_repairs
```

Then, right before the ThreadPoolExecutor dispatch (line ~1790), reorder candidates so fast shards execute first:

```python
    # Reorder: Parquet first, Vision second, REST last.
    # ThreadPoolExecutor processes submissions in order when pool is full,
    # so fast shards complete and free slots before REST shards start.
    candidates = parquet_candidates + vision_candidates + rest_candidates
```

Also, if ALL candidates are REST-only, cap max_concurrent to max_concurrent_rest:

```python
    if candidates and all(s in rest_candidates for s in candidates):
        max_concurrent = min(max_concurrent, cfg.max_concurrent_rest)
        logger.info(
            "all %d candidates are REST-only -> capping to max_concurrent_rest=%d",
            len(candidates),
            max_concurrent,
        )
```

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && grep -n "max_concurrent_rest" python/opendeviationbar/config/kintsugi.py python/opendeviationbar/kintsugi.py && grep -n "_shard_has_vision_available" python/opendeviationbar/kintsugi.py | head -10</automated>
  </verify>
  <acceptance_criteria>
    - grep "max_concurrent_rest: int = 2" python/opendeviationbar/config/kintsugi.py returns exactly 1 match
    - grep "OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REST" python/opendeviationbar/config/kintsugi.py returns at least 1 match (in docstring)
    - grep "def _shard_has_vision_available" python/opendeviationbar/kintsugi.py returns exactly 1 match
    - grep "_is_vision_cached_unavailable" python/opendeviationbar/kintsugi.py returns at least 1 match
    - grep "parquet_candidates" python/opendeviationbar/kintsugi.py returns at least 2 matches
    - grep "rest_candidates" python/opendeviationbar/kintsugi.py returns at least 2 matches
    - grep "vision_candidates" python/opendeviationbar/kintsugi.py returns at least 2 matches
    - mise run test-py passes
  </acceptance_criteria>
  <done>Candidates classified into 3 tiers. All-REST batches capped at 2 workers. Fast shards (Parquet/Vision) ordered first so they complete before REST shards consume slots. New OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REST env var (default 2) controls REST cap.</done>
</task>

</tasks>

<verification>
1. `grep -c "_get_worker_cache" python/opendeviationbar/kintsugi.py` >= 5 (1 def + 4 call sites)
2. `grep -c "with OpenDeviationBarCache() as cache:" python/opendeviationbar/kintsugi.py` == 6 (main-thread sites only)
3. `grep "max_concurrent_rest" python/opendeviationbar/config/kintsugi.py` shows the new field
4. `grep "_shard_has_vision_available" python/opendeviationbar/kintsugi.py` shows the new classifier
5. `mise run test-py` passes all tests
</verification>

<success_criteria>

- Thread-local CH cache eliminates per-call connection overhead in worker threads
- 3-tier concurrency prevents REST shards from diluting fast Parquet/Vision throughput
- fd count at 24 workers: ~4*24+50 = ~146 (not 96*20 = 1920)
- New env var OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REST defaults to 2
- All existing tests pass
  </success_criteria>

<output>
After completion, create `.planning/phases/02-kintsugi-rest-ch-connection/02-01-SUMMARY.md`
</output>
