---
phase: 02-kintsugi-rest-ch-connection
plan: 01
subsystem: infra
tags: [kintsugi, threading, clickhouse, connection-pool, concurrency]

requires: []
provides:
  - Thread-local CH cache pattern in kintsugi worker threads
  - 3-tier concurrency control (Parquet/Vision/REST) in kintsugi_pass
  - max_concurrent_rest config field for REST budget capping
affects: [03-rust-hot-path, 04-python-efficiency]

tech-stack:
  added: []
  patterns:
    [
      thread-local CH cache via threading.local(),
      3-tier concurrency classification,
    ]

key-files:
  created: []
  modified:
    - python/opendeviationbar/kintsugi.py
    - python/opendeviationbar/config/kintsugi.py

key-decisions:
  - "Thread-local cache reuses one OpenDeviationBarCache per worker thread lifetime -- no context manager needed since GC closes on thread exit"
  - "Vision availability check uses Redis negative cache heuristic -- optimistic (assumes available if no 404 cached)"
  - "REST-only batches capped at min(max_concurrent_repairs, max_concurrent_rest) to prevent budget dilution"
  - "Candidates reordered Parquet-first, Vision-second, REST-last so fast shards free pool slots before slow ones start"

patterns-established:
  - "_get_worker_cache() pattern: thread-local CH cache reuse for worker threads in ThreadPoolExecutor"
  - "3-tier shard classification: _shard_is_parquet_cached / _shard_has_vision_available / REST fallback"

requirements-completed: [STAB-03, STAB-04, STAB-05, STAB-06]

duration: 4min
completed: 2026-03-20
---

# Phase 02 Plan 01: Kintsugi REST + CH Connection Summary

**Thread-local CH cache eliminates per-call fd churn in kintsugi workers, and 3-tier concurrency control prevents REST-only shards from diluting Parquet/Vision throughput**

## Performance

- **Duration:** 4 min
- **Started:** 2026-03-20T08:26:00Z
- **Completed:** 2026-03-20T08:30:00Z
- **Tasks:** 2
- **Files modified:** 2

## Accomplishments

- Thread-local CH cache at 4 worker-thread call sites reduces fd count from ~80/worker to ~20/worker
- 3-tier concurrency classifies shards by data source (Parquet/Vision/REST) for optimal parallelism
- REST-only batches capped at 2 workers (configurable via OPENDEVIATIONBAR_KINTSUGI_MAX_CONCURRENT_REST)
- Fast shards (Parquet/Vision) reordered to execute first, freeing slots before REST shards start

## Task Commits

Each task was committed atomically:

1. **Task 1: Add thread-local CH cache to kintsugi worker functions** - `cea48f4` (feat)
2. **Task 2: Add 3-tier concurrency control with REST limiting** - `7d3102b` (feat)

## Files Created/Modified

- `python/opendeviationbar/kintsugi.py` - Thread-local cache (\_get_worker_cache), Vision classifier (\_shard_has_vision_available), 3-tier concurrency in kintsugi_pass
- `python/opendeviationbar/config/kintsugi.py` - New max_concurrent_rest field (default: 2)

## Decisions Made

- Thread-local cache pattern copied exactly from repair_direct_parquet.py (proven at 24 workers)
- Main-thread call sites (discover_shards, replay_dead_letters, etc.) left unchanged -- they run once, not in hot loops
- Vision availability uses Redis negative cache as fast heuristic; actual HTTP check happens at repair time
- Removed unused OpenDeviationBarCache imports from worker functions after switching to \_get_worker_cache()

## Deviations from Plan

### Auto-fixed Issues

**1. [Rule 1 - Bug] Removed unused imports in worker functions**

- **Found during:** Task 1
- **Issue:** After replacing `with OpenDeviationBarCache() as cache:` with `_get_worker_cache()`, the `from opendeviationbar.clickhouse import OpenDeviationBarCache` imports at lines 1135, 1253, 1465 became unused
- **Fix:** Removed the 3 unused imports to keep code clean
- **Files modified:** python/opendeviationbar/kintsugi.py
- **Committed in:** cea48f4 (Task 1 commit)

---

**Total deviations:** 1 auto-fixed (1 bug cleanup)
**Impact on plan:** Trivial cleanup of unused imports. No scope creep.

## Issues Encountered

None

## User Setup Required

None - no external service configuration required.

## Next Phase Readiness

- Thread-local CH cache and 3-tier concurrency are ready for production deployment
- Phase 03 (Rust hot path) and Phase 04 (Python efficiency) can proceed independently

---

_Phase: 02-kintsugi-rest-ch-connection_
_Completed: 2026-03-20_
