---
phase: 04-python-efficiency
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/kintsugi.py
  - python/opendeviationbar/backfill.py
  - python/opendeviationbar/clickhouse/bulk_operations.py
autonomous: true
requirements:
  - PYEF-01
  - PYEF-02
  - PYEF-03

must_haves:
  truths:
    - "_kintsugi_cfg() is called once at top of _repair(), bound to local `cfg`, reused at lines ~1001, ~1043, ~1075"
    - "Vision 404 negative cache uses 15-min TTL for dates within 2 days of today, 1-hour TTL otherwise"
    - "store_bars_bulk() non-DatetimeIndex path assigns `df = bars` instead of `df = bars.copy()`"
  artifacts:
    - path: "python/opendeviationbar/kintsugi.py"
      provides: "Config caching in _repair()"
      contains: "cfg = _kintsugi_cfg()"
    - path: "python/opendeviationbar/backfill.py"
      provides: "Dynamic Vision TTL"
      contains: "vision_probe_ttl_recent_s"
    - path: "python/opendeviationbar/clickhouse/bulk_operations.py"
      provides: "Removed redundant copy"
      contains: "df = bars"
  key_links:
    - from: "python/opendeviationbar/kintsugi.py"
      to: "python/opendeviationbar/config/kintsugi.py"
      via: "_kintsugi_cfg() called once, attributes accessed from local"
      pattern: "cfg\\.batch_days|cfg\\.batch_workers"
    - from: "python/opendeviationbar/backfill.py"
      to: "python/opendeviationbar/config/backfill.py"
      via: "vision_probe_ttl_recent_s config field"
      pattern: "vision_probe_ttl_recent_s"
---

<objective>
Eliminate redundant config instantiation in kintsugi _repair(), reduce Vision 404 negative cache TTL for recent dates, and remove unnecessary DataFrame copy in bulk operations.

Purpose: Three surgical Python efficiency fixes identified by the 5-agent performance audit. Each eliminates measurable overhead in production hot paths.
Output: Three modified Python files with zero behavioral changes (pure optimization).
</objective>

<execution_context>
@/Users/terryli/.claude/get-shit-done/workflows/execute-plan.md
@/Users/terryli/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@python/opendeviationbar/kintsugi.py
@python/opendeviationbar/backfill.py
@python/opendeviationbar/clickhouse/bulk_operations.py
@python/opendeviationbar/config/backfill.py

<interfaces>
<!-- Key functions and config the executor needs -->

From python/opendeviationbar/kintsugi.py:

```python
def _kintsugi_cfg() -> KintsugiConfig:
    from opendeviationbar.config.kintsugi import KintsugiConfig
    return KintsugiConfig()
```

Call sites inside \_repair() (lines 1001, 1043, 1075):

```python
batch_days = min(num_days, _kintsugi_cfg().batch_days) if ouroboros_mode == "day" else num_days
if _kintsugi_cfg().batch_workers <= 1 or len(days_to_repair) <= 1:
workers = min(_kintsugi_cfg().batch_workers, len(days_to_repair))
```

From python/opendeviationbar/config/backfill.py:

```python
class BackfillConfig(BaseSettings):
    model_config = SettingsConfigDict(env_prefix="OPENDEVIATIONBAR_BACKFILL_", case_sensitive=False)
    vision_probe_ttl_s: int = 3600  # 1 hour
```

From python/opendeviationbar/backfill.py:

```python
def _mark_vision_unavailable(symbol: str, date_str: str) -> None:
    client.set(f"odb:vision:404:{symbol}:{date_str}", "1", ex=_backfill_cfg().vision_probe_ttl_s)
```

From python/opendeviationbar/clickhouse/bulk_operations.py (line 503):

```python
else:
    df = bars.copy()
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Cache _kintsugi_cfg() in _repair() + add dynamic Vision TTL</name>
  <files>python/opendeviationbar/kintsugi.py, python/opendeviationbar/backfill.py, python/opendeviationbar/config/backfill.py</files>
  <read_first>
    - python/opendeviationbar/kintsugi.py (lines 934-1085 for _repair function)
    - python/opendeviationbar/backfill.py (lines 55-100 for Vision TTL)
    - python/opendeviationbar/config/backfill.py (lines 40-70 for BackfillConfig)
  </read_first>
  <action>
**PYEF-01: Config caching in _repair() (kintsugi.py)**

In `_repair()` (starts at line 934), add `cfg = _kintsugi_cfg()` as the first line after the docstring/initial setup, BEFORE line 994 where the first call occurs.

Replace these 3 call sites with the local `cfg`:

- Line 1001: `_kintsugi_cfg().batch_days` -> `cfg.batch_days`
- Line 1043: `_kintsugi_cfg().batch_workers` -> `cfg.batch_workers`
- Line 1075: `_kintsugi_cfg().batch_workers` -> `cfg.batch_workers`

Do NOT touch `_kintsugi_cfg()` calls outside `_repair()` (e.g., lines 120, 1685, 1740, 2018, 2025, 2027, 2074, 2081, 2082, 2178, 2180, 2276, 2399). Those are in different functions.

**PYEF-02: Dynamic Vision TTL (backfill.py + config/backfill.py)**

In `python/opendeviationbar/config/backfill.py`, add a new field to `BackfillConfig` right after `vision_probe_ttl_s`:

```python
vision_probe_ttl_recent_s: int = 900  # 15 minutes — for dates within 2 days of today
```

In `python/opendeviationbar/backfill.py`, modify `_mark_vision_unavailable()` to accept an optional `date_str` parameter (it already does) and compute TTL dynamically:

1. Import `datetime` and `date` at the top of the function or use existing imports
2. Parse `date_str` (format "YYYY-MM-DD") to a `date` object
3. If `(date.today() - parsed_date).days <= 2`, use `_backfill_cfg().vision_probe_ttl_recent_s` (15 min)
4. Otherwise use `_backfill_cfg().vision_probe_ttl_s` (1 hour, existing behavior)

The date parsing should be wrapped in try/except (fallback to standard 1-hour TTL if parse fails). Use `datetime.strptime(date_str, "%Y-%m-%d").date()` for parsing.

Update the log message at line 355 to reflect the dynamic TTL:

- `"cached 1h"` -> `"cached 15m"` or `"cached 1h"` depending on which TTL was used.
  </action>
  <verify>
  <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "
  from opendeviationbar.config.backfill import BackfillConfig
  cfg = BackfillConfig()
  assert hasattr(cfg, 'vision_probe_ttl_recent_s'), 'Missing vision_probe_ttl_recent_s'
  assert cfg.vision_probe_ttl_recent_s == 900, f'Expected 900, got {cfg.vision_probe_ttl_recent_s}'
  assert cfg.vision_probe_ttl_s == 3600, f'Expected 3600, got {cfg.vision_probe_ttl_s}'
  print('BackfillConfig OK')
  " && python -c "
  import ast, inspect
  from opendeviationbar import kintsugi
  src = inspect.getsource(kintsugi.\_repair)
  assert 'cfg =\_kintsugi_cfg()' in src, 'Missing cfg local binding in_repair()'

# Verify no bare \_kintsugi_cfg() calls remain in \_repair body after the binding

lines = src.split('\n')
found*binding = False
bare_calls = 0
for line in lines:
stripped = line.strip()
if 'cfg = \_kintsugi_cfg()' in stripped:
found_binding = True
continue
if found_binding and '\_kintsugi_cfg()' in stripped:
bare_calls += 1
assert bare_calls == 0, f'Found {bare_calls} bare_kintsugi_cfg() calls after binding'
print('kintsugi.\_repair config caching OK')
"</automated>
</verify>
<acceptance_criteria> - `grep -n "cfg = _kintsugi_cfg()" python/opendeviationbar/kintsugi.py` shows a line inside `_repair()` - `grep -n "cfg\.batch_days" python/opendeviationbar/kintsugi.py` shows at least 1 result - `grep -n "cfg\.batch_workers" python/opendeviationbar/kintsugi.py` shows at least 2 results - `grep -n "\_kintsugi_cfg().batch*" python/opendeviationbar/kintsugi.py`returns NO results (all replaced)
    -`grep -n "vision_probe_ttl_recent_s" python/opendeviationbar/config/backfill.py`shows the new field
    -`grep -n "vision_probe_ttl_recent_s" python/opendeviationbar/backfill.py`shows the dynamic TTL usage
  </acceptance_criteria>
  <done>
    - _kintsugi_cfg() called once at top of _repair(), local`cfg` used for batch_days and batch_workers (0 bare calls remain inside_repair) - Vision 404 TTL is 15 minutes for dates within 2 days of today, 1 hour otherwise - BackfillConfig has vision_probe_ttl_recent_s field defaulting to 900
</done>
</task>

<task type="auto">
  <name>Task 2: Remove redundant DataFrame copy in store_bars_bulk()</name>
  <files>python/opendeviationbar/clickhouse/bulk_operations.py</files>
  <read_first>
    - python/opendeviationbar/clickhouse/bulk_operations.py (lines 490-520 for the copy site and downstream mutations)
  </read_first>
  <action>
**PYEF-03: Remove redundant copy (bulk_operations.py)**

At line 503, change:

```python
df = bars.copy()
```

to:

```python
df = bars
```

This is safe because:

1. Line 508: `df.columns = df.columns.str.lower()` — creates a new Index object, replaces reference
2. Lines 511-514: `df["symbol"] = ...`, `df["threshold_decimal_bps"] = ...` etc. — these are column assignments that mutate the DataFrame, but the caller (`store_bars_bulk()`) is the terminal write path. The caller's DataFrame may be mutated, but `store_bars_bulk()` is always the last consumer of the DataFrame before ClickHouse INSERT.

VERIFY this is safe: Read the callers of `store_bars_bulk()` to confirm none of them use the DataFrame after calling this function. The callers are in `orchestration/` and `checkpoint.py`. If any caller reads the DataFrame AFTER calling `store_bars_bulk()`, keep `.copy()` and add a comment explaining why. The ROADMAP and audit indicate it's safe to remove.

IMPORTANT: The DatetimeIndex path (line 495: `df = bars.reset_index()`) already avoids copy — that's correct and should NOT be changed.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "
import inspect
from opendeviationbar.clickhouse.bulk_operations import BulkStoreMixin
src = inspect.getsource(BulkStoreMixin.store_bars_bulk)

# The non-DatetimeIndex else branch should have 'df = bars' not 'df = bars.copy()'

lines = src.split('\n')
found_else = False
for i, line in enumerate(lines):
if 'df = bars.copy()' in line:
raise AssertionError(f'Found bars.copy() at source line {i}: {line.strip()}')
print('No bars.copy() found — redundant copy removed')

# Verify df = bars still exists

assert any('df = bars' in l and 'copy' not in l for l in lines), 'df = bars assignment missing'
print('df = bars assignment present')
"</automated>
</verify>
<acceptance_criteria> - `grep -n "df = bars.copy()" python/opendeviationbar/clickhouse/bulk_operations.py` returns NO results - `grep -n "df = bars$" python/opendeviationbar/clickhouse/bulk_operations.py` shows the replacement line - The DatetimeIndex path (`df = bars.reset_index()`) is unchanged
</acceptance_criteria>
<done> - `bars.copy()` removed from store_bars_bulk() non-DatetimeIndex path - DatetimeIndex path unchanged (still uses reset_index()) - All existing tests pass
</done>
</task>

</tasks>

<verification>
Run the full Python test suite to confirm no regressions:

```bash
cd /Users/terryli/eon/opendeviationbar-py && mise run test-py
```

Specifically verify kintsugi-related tests:

```bash
pytest tests/test_kintsugi_repair.py tests/test_kintsugi_ordering.py tests/test_chunked_gap_repair.py -v
```

And bulk operations tests:

```bash
pytest tests/test_overlap_strategy.py tests/test_day_mode_invariant.py -v
```

</verification>

<success_criteria>

1. `_kintsugi_cfg()` called exactly once inside `_repair()` body, bound to `cfg`, 3 attribute accesses use `cfg.`
2. `vision_probe_ttl_recent_s = 900` in BackfillConfig, used in `_mark_vision_unavailable()` for dates <= 2 days old
3. `bars.copy()` removed from `store_bars_bulk()` non-DatetimeIndex path
4. All existing Python tests pass with no regressions
   </success_criteria>

<output>
After completion, create `.planning/phases/04-python-efficiency/04-01-SUMMARY.md`
</output>
