# Checkpoint: Resumable Cache Population

**Parent**: [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

Resumable population engine for building the ClickHouse bar cache from historical tick data. Supports bar-level checkpointing, hybrid local + ClickHouse persistence, and cross-machine resume. Modularized from the `populate_cache_resumable()` section of `backfill.py` (Phase 01 v1.5).

---

## Files

| File           | Purpose                                                                                                                                                                                                                                                  |
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `__init__.py`  | Package entry: `populate_cache_resumable()` main function, re-exports all public symbols from submodules for backward compatibility                                                                                                                      |
| `__init__.pyi` | Type stubs for `populate_cache_resumable()` and all re-exported names                                                                                                                                                                                    |
| `engine.py`    | Processing engine: `_process_day()` (single-day tick fetch + bar construction), `_process_day_multi_threshold()` (fan-out), `_write_bars()` (bounded flush to ClickHouse), `_date_range()`, `_is_ouroboros_boundary()`                                   |
| `models.py`    | Data structures: `PopulationContext` (loop-constant state bundle, ~15 params -> 1 object), `PopulationCheckpoint` (last completed date, bar count, processor state), `_CHECKPOINT_DIR`, `_get_population_config()`                                       |
| `storage.py`   | Checkpoint persistence: `_get_checkpoint_path()` (path resolution with symbol/threshold/ouroboros in filename), `_load_checkpoint_from_clickhouse()`, `_save_checkpoint_to_clickhouse()`, `_checkpoint_provenance()` (hostname, PID, timestamp metadata) |

---

## Checkpoint Flow

```
populate_cache_resumable()              # __init__.py
    |
    +--> storage._get_checkpoint_path()  # Resolve local checkpoint file
    +--> storage._load_checkpoint_*()    # Try local file, then ClickHouse
    |
    +--> for date in engine._date_range(start, end):
    |       |
    |       +--> engine._process_day()   # Fetch ticks (Parquet cache -> Vision -> REST)
    |       |       +--> processor.process_trades_arrow()  # Rust bar construction
    |       |       +--> engine._write_bars()              # Bounded flush (MEM-013)
    |       |
    |       +--> storage._save_checkpoint_*()  # Persist after each day
    |
    +--> Final count verification against ClickHouse
```

## Memory Guards

`_write_bars()` in `engine.py` implements bounded flush (MEM-013): bars are flushed every `OPENDEVIATIONBAR_FLUSH_THRESHOLD` bars (default 10,000) to prevent unbounded memory growth during long population runs. The env var is validated with `max(1, int())` and fallback to 10,000.

`populate_cache_resumable()` in `__init__.py` enforces a `max_days=366` guard by default. Only kintsugi (via `max_days=None`) can bypass this for historical backfill.

---

## Related

- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) -- Parent: Python API layer, long range processing section
- [/CLAUDE.md](/CLAUDE.md) -- Hub: `populate_cache_resumable()` in API table, memory protection
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) -- ClickHouse `population_checkpoints` table used by `storage.py`
- [/python/opendeviationbar/storage/CLAUDE.md](/python/opendeviationbar/storage/CLAUDE.md) -- Tier-1 Parquet ticks cache consumed by `engine._process_day()`
