---
phase: 01-modularize-long-files-live-engine-rs-checkpoint-py-backfill-py
plan: 03
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/backfill.py
  - python/opendeviationbar/rectify.py
  - python/opendeviationbar/memory_guard.py
  - python/opendeviationbar/vision_cache.py
  - python/opendeviationbar/backfill.pyi
autonomous: true
requirements: []

must_haves:
  truths:
    - "All existing Python tests pass without modification"
    - "All imports from opendeviationbar.backfill work unchanged"
    - "backfill_gap, backfill_all_gaps, BackfillResult importable from opendeviationbar package"
  artifacts:
    - path: "python/opendeviationbar/backfill.py"
      provides: "Core backfill functions (backfill_gap, backfill_all_gaps) -- trimmed from 1423 to ~1000 lines"
      contains: "def backfill_gap"
    - path: "python/opendeviationbar/rectify.py"
      provides: "rectify_recent_bars, _delete_ghost_bar, _fetch_trades_for_bar, _emit_rectification_event"
      contains: "def rectify_recent_bars"
    - path: "python/opendeviationbar/memory_guard.py"
      provides: "_force_allocator_purge, _check_memory_watermark"
      contains: "def _check_memory_watermark"
    - path: "python/opendeviationbar/vision_cache.py"
      provides: "_is_vision_cached_unavailable, _mark_vision_unavailable"
      contains: "def _is_vision_cached_unavailable"
  key_links:
    - from: "python/opendeviationbar/backfill.py"
      to: "rectify.py, memory_guard.py, vision_cache.py"
      via: "from .rectify import, from .memory_guard import, from .vision_cache import"
      pattern: "from \\.rectify import"
    - from: "python/opendeviationbar/__init__.py"
      to: "backfill module"
      via: "from .backfill import"
      pattern: "from \\.backfill import.*BackfillResult"
---

<objective>
Extract 3 cohesive concerns from backfill.py (1,423 lines): bar rectification (~320 lines), memory guard (~80 lines), and Vision negative cache (~50 lines) into sibling modules.

Purpose: Reduce backfill.py to its core gap-fill logic (~1000 lines) by extracting orthogonal concerns into focused files. Unlike live_engine and checkpoint, backfill stays as a file (not converted to package) with extracted siblings.
Output: backfill.py trimmed, 3 new sibling modules, all imports preserved, all tests passing.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/01-modularize-long-files-live-engine-rs-checkpoint-py-backfill-py/01-CONTEXT.md

@python/opendeviationbar/backfill.py
@python/opendeviationbar/backfill.pyi
@python/opendeviationbar/**init**.py (lines 130-141 only)

<interfaces>
<!-- Public imports from backfill that MUST remain working -->
From python/opendeviationbar/__init__.py (lines 132-136):
```python
from .backfill import (
    BackfillResult,
    backfill_all_gaps,
    backfill_gap,
)
```

From python/opendeviationbar/kintsugi/daemon.py (line 35):

```python
from opendeviationbar.backfill import backfill_all_gaps, backfill_gap
```

From python/opendeviationbar/kintsugi/daemon.py (line 97-100):

```python
from opendeviationbar.backfill import (
    _check_rate_limit_before_rest,
    _fetch_aggtrades_by_id,
    _fetch_aggtrades_rest,
)
```

From python/opendeviationbar/kintsugi/daemon.py (line 149):

```python
from opendeviationbar.backfill import _check_memory_watermark, _env_int
```

From python/opendeviationbar/kintsugi/repair.py (line 467):

```python
from opendeviationbar.backfill import (
    _fetch_aggtrades_by_id_with_retry,
    _fetch_trades_ariadne,
    _process_gap_fallback_starttime,
    rectify_recent_bars,
)
```

From python/opendeviationbar/kintsugi/discovery.py (line 530):

```python
from opendeviationbar.backfill import _is_vision_cached_unavailable
```

<!-- backfill.pyi stub exports -->

```python
class BackfillResult: ...
def backfill_gap(...) -> BackfillResult: ...
def backfill_all_gaps(...) -> list[BackfillResult]: ...
```

</interfaces>
</context>

<tasks>

<task type="auto">
  <name>Task 1: Extract rectify.py, memory_guard.py, and vision_cache.py from backfill.py</name>
  <files>
    python/opendeviationbar/backfill.py
    python/opendeviationbar/rectify.py
    python/opendeviationbar/memory_guard.py
    python/opendeviationbar/vision_cache.py
    python/opendeviationbar/backfill.pyi
  </files>
  <read_first>
    python/opendeviationbar/backfill.py
    python/opendeviationbar/backfill.pyi
    python/opendeviationbar/__init__.py (lines 130-141)
    python/opendeviationbar/kintsugi/daemon.py (lines 30-155 -- imports from backfill)
    python/opendeviationbar/kintsugi/repair.py (lines 460-475 -- imports from backfill)
    python/opendeviationbar/kintsugi/discovery.py (lines 525-535 -- imports from backfill)
  </read_first>
  <action>
    Extract 3 cohesive modules from backfill.py as sibling files. backfill.py re-exports extracted symbols for backward compatibility.

    **Step 1: Create vision_cache.py** (~50 lines) containing:
    - `def _is_vision_cached_unavailable(symbol, date_str) -> bool` (lines 69-83)
    - `def _mark_vision_unavailable(symbol, date_str) -> None` (lines 85-133)
    - Imports: logging, `from opendeviationbar.config.resilience import ResilienceConfig` (lazy, inside function body -- keep same pattern as original)
    - Logger: `logger = logging.getLogger(__name__)`
    - Module docstring: "Redis-backed negative cache for Binance Vision 404 responses."

    **Step 2: Create memory_guard.py** (~80 lines) containing:
    - `def _force_allocator_purge() -> None` (lines 136-165)
    - `def _check_memory_watermark(memory_max_bytes: int) -> str` (lines 167-201)
    - Imports: gc, logging, os, ctypes (lazy -- inside function body as original)
    - Logger: `logger = logging.getLogger(__name__)`
    - Module docstring: "Memory watermark and allocator purge utilities for long-running processes."

    **Step 3: Create rectify.py** (~320 lines) containing:
    - `def rectify_recent_bars(...)` (lines 1099-1306)
    - `def _delete_ghost_bar(...)` (lines 1308-1354)
    - `def _fetch_trades_for_bar(...)` (lines 1356-1383)
    - `def _emit_rectification_event(...)` (lines 1385-1423)
    - Imports: logging, os, json, datetime, typing, plus lazy imports inside function bodies (polars, opendeviationbar.clickhouse, opendeviationbar.orchestration.helpers, etc. -- keep same lazy pattern as original)
    - Logger: `logger = logging.getLogger(__name__)`
    - Module docstring: "Bar rectification safety net -- re-fetches and overwrites bars with trade count deficits."

    **Step 4: Update backfill.py** to import and re-export extracted symbols:
    - Remove the extracted function bodies
    - Add at the top (after existing imports):
      ```python
      # Extracted modules -- re-export for backward compatibility
      from .vision_cache import _is_vision_cached_unavailable, _mark_vision_unavailable
      from .memory_guard import _force_allocator_purge, _check_memory_watermark
      from .rectify import rectify_recent_bars, _delete_ghost_bar, _fetch_trades_for_bar, _emit_rectification_event
      ```
    - Update internal callers within backfill.py that call the extracted functions -- they now come via import (already handled by re-export at module level)
    - Keep all other functions in backfill.py unchanged: `_backfill_cfg`, `_resilience_cfg`, `BackfillResult`, `_pair_sort_key`, `_get_cached_symbol_threshold_pairs`, `backfill_gap`, `_check_rate_limit_before_rest`, `_fetch_aggtrades_rest`, `_fetch_aggtrades_by_id`, `_process_gap_fallback_starttime`, `_fetch_and_process_gap`, `_fetch_aggtrades_by_id_with_retry`, `_fetch_trades_ariadne`, `_notify_sustained_failure`, `backfill_all_gaps`

    **Step 5: backfill.pyi** -- No changes needed. The .pyi file only documents BackfillResult, backfill_gap, and backfill_all_gaps, which remain in backfill.py.

    **CRITICAL**:
    - All `from opendeviationbar.backfill import X` must still work for ALL symbols (public and private)
    - kintsugi/daemon.py imports `_check_memory_watermark` from backfill -- must still work via re-export
    - kintsugi/repair.py imports `rectify_recent_bars` from backfill -- must still work via re-export
    - kintsugi/discovery.py imports `_is_vision_cached_unavailable` from backfill -- must still work via re-export
    - No behavioral changes -- pure code movement + re-export

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && python -c "from opendeviationbar.backfill import backfill_gap, backfill_all_gaps, BackfillResult, rectify_recent_bars, _check_memory_watermark, _is_vision_cached_unavailable; print('All imports OK')"</automated>
  </verify>
  <acceptance_criteria>
    - File `python/opendeviationbar/rectify.py` exists and contains `def rectify_recent_bars`
    - File `python/opendeviationbar/rectify.py` contains `def _delete_ghost_bar`
    - File `python/opendeviationbar/rectify.py` contains `def _fetch_trades_for_bar`
    - File `python/opendeviationbar/memory_guard.py` exists and contains `def _check_memory_watermark`
    - File `python/opendeviationbar/memory_guard.py` contains `def _force_allocator_purge`
    - File `python/opendeviationbar/vision_cache.py` exists and contains `def _is_vision_cached_unavailable`
    - File `python/opendeviationbar/vision_cache.py` contains `def _mark_vision_unavailable`
    - File `python/opendeviationbar/backfill.py` contains `from .rectify import rectify_recent_bars`
    - File `python/opendeviationbar/backfill.py` contains `from .memory_guard import`
    - File `python/opendeviationbar/backfill.py` contains `from .vision_cache import`
    - `python -c "from opendeviationbar.backfill import rectify_recent_bars, _check_memory_watermark, _is_vision_cached_unavailable"` exits 0
    - `python -c "from opendeviationbar import BackfillResult, backfill_gap, backfill_all_gaps"` exits 0
  </acceptance_criteria>
  <done>3 modules extracted from backfill.py, all re-exports in place, all imports working</done>
</task>

<task type="auto">
  <name>Task 2: Run Python test suite to confirm zero regressions</name>
  <files>
    python/opendeviationbar/backfill.py
  </files>
  <read_first>
    python/opendeviationbar/backfill.py (first 30 lines -- verify re-exports present)
  </read_first>
  <action>
    Run the Python test suite to confirm no regressions from the backfill.py extraction.

    Run:
    1. `uv run --python 3.13 pytest tests/ -x -q --timeout=60` -- full Python test suite with fail-fast
    2. Specifically verify backfill-related tests pass: `uv run --python 3.13 pytest tests/ -x -q -k "backfill or rectif" --timeout=60`

    If any test fails due to import issues:
    - Check if the test imports a private function that wasn't re-exported in backfill.py
    - Add the missing re-export to backfill.py
    - Re-run the failing test

    Also verify kintsugi imports work:
    ```
    python -c "from opendeviationbar.kintsugi.daemon import KintsugiDaemon; print('kintsugi daemon import OK')" 2>&1 || true
    python -c "from opendeviationbar.kintsugi.repair import _repair_shard; print('kintsugi repair import OK')" 2>&1 || true
    ```
    (These may fail if they need runtime config -- that's OK. The important thing is no ImportError.)

  </action>
  <verify>
    <automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/ -x -q --timeout=60 2>&1 | tail -10</automated>
  </verify>
  <acceptance_criteria>
    - `uv run --python 3.13 pytest tests/ -x -q --timeout=60` exits 0 with all tests passing
  </acceptance_criteria>
  <done>All Python tests pass, zero regressions from backfill extraction</done>
</task>

</tasks>

<verification>
1. `python -c "from opendeviationbar.backfill import backfill_gap, rectify_recent_bars, _check_memory_watermark, _is_vision_cached_unavailable"` -- backward-compatible imports work
2. `python -c "from opendeviationbar.rectify import rectify_recent_bars"` -- direct import from new module works
3. `python -c "from opendeviationbar.memory_guard import _check_memory_watermark"` -- direct import works
4. `python -c "from opendeviationbar.vision_cache import _is_vision_cached_unavailable"` -- direct import works
5. `uv run --python 3.13 pytest tests/ -x -q --timeout=60` -- all tests pass
6. `wc -l python/opendeviationbar/backfill.py` -- reduced from 1423 to ~1000 lines
</verification>

<success_criteria>

- 3 new sibling modules created: rectify.py, memory_guard.py, vision_cache.py
- backfill.py trimmed from 1423 to ~1000 lines
- All symbols still importable from opendeviationbar.backfill (backward compatibility via re-export)
- All Python tests pass unchanged
- kintsugi imports of private backfill functions still work
  </success_criteria>

<output>
After completion, create `.planning/phases/01-modularize-long-files-live-engine-rs-checkpoint-py-backfill-py/01-03-SUMMARY.md`
</output>
