# Phase 25: Lazy Evaluation + Cache Efficiency - Research

**Researched:** 2026-03-27
**Domain:** Polars lazy evaluation, Parquet column projection, Python in-place sorting
**Confidence:** HIGH

## Summary

Phase 25 addresses two specific inefficiencies in the codebase: (1) `TickStorage.write_ticks()` loads the entire existing Parquet file into memory just to check which `agg_trade_id` values already exist, when it only needs a single column; (2) `_sync_flush_rest_bars_dict()` creates a sorted copy of the bars list for Stathera diagnostic when an in-place sort would avoid the allocation.

Both changes are surgical, low-risk, and mechanically straightforward. The existing code is well-structured -- the inefficiencies are minor and only matter at scale (daily Parquet files with 700K+ ticks, or 200K-bar drain batches).

**Primary recommendation:** Use `pl.scan_parquet().select("agg_trade_id").collect()` for the dedup ID check, then only `pl.read_parquet()` the full file when new rows exist. Replace `sorted()` with `list.sort()` in the dict-based Stathera diagnostic.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- auto-generated infrastructure phase, discuss skipped.

### Claude's Discretion

All implementation choices are at Claude's discretion -- pure infrastructure phase. Use ROADMAP phase goal, success criteria, and codebase conventions to guide decisions.

### Deferred Ideas (OUT OF SCOPE)

None -- discuss phase skipped.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID      | Description                                                                                | Research Support                                                                                                                       |
| ------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------- |
| LAZY-01 | `TickStorage.write_ticks` dedup uses `scan_parquet` for ID-only check (not full file load) | Line 283 of `parquet.py`: replace `pl.read_parquet(parquet_path)` with lazy scan for dedup check, defer full read to merge-needed path |
| LAZY-02 | Stathera diagnostic sort uses in-place sort or avoids sorted() copy                        | Line 617 of `midnight.py`: replace `sorted(batch_bars, ...)` with `batch_bars.sort(...)`. Arrow path already uses Polars sort.         |

</phase_requirements>

## Standard Stack

### Core

| Library | Version          | Purpose                                   | Why Standard                                         |
| ------- | ---------------- | ----------------------------------------- | ---------------------------------------------------- |
| polars  | (project-pinned) | LazyFrame scan_parquet, column projection | Already used throughout codebase for all Parquet I/O |

### Supporting

No additional libraries needed. Both changes use existing Python/Polars primitives.

## Architecture Patterns

### LAZY-01: Two-Phase Dedup in write_ticks

**Current code** (`parquet.py:278-312`):

```python
# Single read: load full file once for both dedup and concat
existing_df = pl.read_parquet(parquet_path)          # <-- loads ALL 7 columns
write_new = write_deduped.join(
    existing_df.select("agg_trade_id"),              # selects after full load
    on="agg_trade_id", how="anti",
)
if not write_new.is_empty():
    # ... merge existing_df with write_new
```

**Problem:** `pl.read_parquet()` materializes the entire file (all 7 columns, ~6.5 MB compressed / ~35 MB in memory for a typical BTCUSDT day). The dedup anti-join only needs the `agg_trade_id` column.

**Optimized pattern:**

```python
# Phase 1: Lazy scan with column projection -- only load agg_trade_id
existing_ids = pl.scan_parquet(parquet_path).select("agg_trade_id").collect()
write_new = write_deduped.join(existing_ids, on="agg_trade_id", how="anti")

if not write_new.is_empty():
    # Phase 2: Only NOW load full file for merge
    existing_df = pl.read_parquet(parquet_path)
    # ... align columns, concat, validate, write
```

**Memory savings:** When all incoming ticks already exist (common case during overlapping seeder runs), the full file is never loaded. When new ticks exist, the full file is loaded once (same as before) -- no regression.

**Key consideration:** The `scan_parquet().select("agg_trade_id").collect()` uses Parquet column projection (predicate pushdown). Polars reads only the `agg_trade_id` column group from the Parquet footer, skipping all other column data. This is a fundamental Parquet feature, not a Polars-specific optimization.

**Edge case:** The comment "Single read: load full file once for both dedup and concat" was the original design rationale. With the two-phase approach, when new rows DO exist, we read the file twice (once for IDs, once for full data). This is acceptable because:

1. The Parquet file is on local SSD (< 1ms seek), and the second read benefits from OS page cache
2. The common case (no new rows) saves the entire full-file read
3. Daily files are ~6.5 MB compressed -- double-read overhead is negligible

### LAZY-02: In-Place Sort for Stathera Diagnostic

**Current code** (`midnight.py:617`):

```python
sorted_bars = sorted(batch_bars, key=lambda b: b.get("first_agg_trade_id", 0))
```

**Problem:** `sorted()` creates a new list. For a 200K-bar drain batch per (symbol, threshold), this duplicates the list's reference array (~1.6 MB of pointers). The dicts themselves are not copied, but the list allocation is unnecessary.

**Optimized pattern:**

```python
batch_bars.sort(key=lambda b: b.get("first_agg_trade_id", 0))
# Use batch_bars directly (now sorted in-place)
```

**Safety analysis:** `batch_bars` is the value from `batches[(sym, thr)]`, a list built during the drain loop. After the Stathera diagnostic, the same `batches` dict is iterated again for CH writes (line 657) and junction telemetry (line 683). Sorting in-place is safe because:

1. The CH write path does NOT depend on original insertion order
2. Sorted order is actually preferable for ClickHouse writes (ascending TID)
3. Junction telemetry uses `max()` and `min()` which are order-independent

**Arrow path note:** The arrow path (`arrow_flush.py:211-257`, `_stathera_diagnostic_polars()`) already uses `pair_df.sort("first_agg_trade_id")` which creates a Polars view, not a Python list copy. No change needed there.

### Anti-Patterns to Avoid

- **Do not remove the full-file read entirely.** When new rows exist, the merge path needs ALL columns from the existing file to align schemas and validate trade-ID continuity.
- **Do not use `scan_parquet` for the merge path.** The merge requires materializing the full existing DataFrame anyway -- lazy evaluation adds overhead for small files.
- **Do not sort the batch_bars list BEFORE the Stathera diagnostic loop.** The sort must happen inside the per-(sym, thr) loop since each batch is independent.

## Don't Hand-Roll

| Problem                   | Don't Build          | Use Instead                               | Why                                                                    |
| ------------------------- | -------------------- | ----------------------------------------- | ---------------------------------------------------------------------- |
| Parquet column projection | Custom column reader | `pl.scan_parquet().select(col).collect()` | Polars handles Parquet row group skipping, predicate pushdown natively |
| In-place list sorting     | Custom sort          | `list.sort(key=...)`                      | Python built-in, stable sort, zero allocation overhead                 |

## Common Pitfalls

### Pitfall 1: Breaking the Merge Path

**What goes wrong:** Removing the full-file `read_parquet` entirely causes the merge path to fail because it needs all columns for schema alignment and `validate_trade_id_continuity()`.
**Why it happens:** Optimizing too aggressively -- the dedup check and the merge are two different operations with different data needs.
**How to avoid:** Two-phase approach: lazy scan for dedup check, full read only when merge is needed.
**Warning signs:** `KeyError` or schema mismatch in the column alignment loop (lines 291-298).

### Pitfall 2: In-Place Sort Mutating Shared State

**What goes wrong:** If `batch_bars` were referenced elsewhere and order mattered, in-place sort would cause subtle bugs.
**Why it happens:** Aliased list references in Python.
**How to avoid:** Verify all downstream consumers of `batches` dict are order-independent. (They are -- confirmed in Architecture Patterns above.)
**Warning signs:** Tests that assert bar ordering in flush output.

### Pitfall 3: scan_parquet on Corrupted Files

**What goes wrong:** `scan_parquet` may fail differently than `read_parquet` on corrupted Parquet files.
**How to avoid:** The `scan_parquet` call is INSIDE the `elif` branch that already passed `_validate_and_recover_parquet()` validation (line 266-268). Corrupted files are caught before reaching the dedup path.

## Code Examples

### LAZY-01: Optimized write_ticks dedup

```python
# Replace lines 282-312 of parquet.py
# Phase 1: Column-projected dedup check (lazy scan, only agg_trade_id)
existing_ids = pl.scan_parquet(parquet_path).select("agg_trade_id").collect()
write_new = write_deduped.join(
    existing_ids,
    on="agg_trade_id",
    how="anti",
)
del existing_ids  # Free ID-only DataFrame immediately

if not write_new.is_empty():
    # Phase 2: Full file read only when merge is needed
    existing_df = pl.read_parquet(parquet_path)
    # Align columns (unchanged from current code)
    for col in existing_df.columns:
        if col not in write_new.columns:
            write_new = write_new.with_columns(
                pl.lit(None)
                .cast(existing_df.schema[col])
                .alias(col)
            )
    write_new = write_new.select(existing_df.columns)
    combined = pl.concat([existing_df, write_new])
    if "agg_trade_id" in combined.columns:
        combined = combined.sort("agg_trade_id")
    # D-07: Validate trade-ID continuity after merge
    if not validate_trade_id_continuity(combined):
        logger.warning(
            "%s %s: trade-ID continuity check failed after merge, rejecting write",
            symbol, day_key,
        )
        del combined, existing_df
        continue
    _atomic_write_parquet(combined, parquet_path)
    del combined
    del existing_df
```

### LAZY-02: In-place sort for Stathera diagnostic

```python
# Replace line 617 of midnight.py
batch_bars.sort(key=lambda b: b.get("first_agg_trade_id", 0))
# Use batch_bars directly instead of sorted_bars
for i in range(1, len(batch_bars)):
    prev_last = batch_bars[i - 1].get("last_agg_trade_id", 0)
    curr_first = batch_bars[i].get("first_agg_trade_id", 0)
    # ... rest unchanged
```

## Validation Architecture

### Test Framework

| Property           | Value                                                  |
| ------------------ | ------------------------------------------------------ |
| Framework          | pytest                                                 |
| Config file        | `pyproject.toml` [tool.pytest.ini_options]             |
| Quick run command  | `uv run --python 3.13 pytest tests/test_storage.py -x` |
| Full suite command | `uv run --python 3.13 pytest tests/ -x`                |

### Phase Requirements -> Test Map

| Req ID  | Behavior                                              | Test Type | Automated Command                                                        | File Exists?                                               |
| ------- | ----------------------------------------------------- | --------- | ------------------------------------------------------------------------ | ---------------------------------------------------------- |
| LAZY-01 | write_ticks dedup uses scan_parquet for ID-only check | unit      | `uv run --python 3.13 pytest tests/test_storage.py -x -k "dedup"`        | Partial (test_storage.py exists, need dedup-specific test) |
| LAZY-02 | Stathera diagnostic uses in-place sort                | unit      | `uv run --python 3.13 pytest tests/test_sync_flush_atomic_replace.py -x` | Partial (test file exists, need sort-specific assertion)   |

### Sampling Rate

- **Per task commit:** `uv run --python 3.13 pytest tests/test_storage.py tests/test_sync_flush_atomic_replace.py -x`
- **Per wave merge:** `uv run --python 3.13 pytest tests/ -x`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

None -- existing test infrastructure covers both files. Tests can be added inline to existing test files.

## Sources

### Primary (HIGH confidence)

- Direct codebase inspection: `parquet.py:278-312` (write_ticks dedup path)
- Direct codebase inspection: `midnight.py:509-710` (\_sync_flush_rest_bars_dict)
- Direct codebase inspection: `arrow_flush.py:211-257` (\_stathera_diagnostic_polars -- already optimized)

### Secondary (MEDIUM confidence)

- Polars documentation: `scan_parquet` supports column projection via `.select()` before `.collect()`

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - no new dependencies, uses existing Polars primitives
- Architecture: HIGH - both changes are mechanical, well-bounded, with clear before/after
- Pitfalls: HIGH - edge cases enumerated and verified against codebase

**Research date:** 2026-03-27
**Valid until:** 2026-04-27 (stable -- no external dependencies)
