# Phase 6: Fix SEED-001 Parquet-First Bar Processing - Research

**Researched:** 2026-03-25
**Domain:** Sidecar startup sequence, Parquet tick processing, Rust processor API
**Confidence:** HIGH

## Summary

This phase replaces the broken SEED-001 "skip Parquet range" logic with "process Parquet ticks into bars via Rust processor." The research confirms all required APIs exist, the atomic write pattern makes concurrent read/write safe, and the `process_trades_arrow_native()` microsecond path is the correct entry point. The change is surgical: modify `_read_parquet_max_tids()` to also return ticks data, process through per-threshold `OpenDeviationBarProcessor` instances, write bars to ClickHouse, and seed `fill_from_rest` with `last_completed_bar_tid` instead of `max(agg_trade_id)`.

Memory is not a concern: 274K BTCUSDT trades at ~60 bytes/trade = ~16 MB in-memory, and Rust processing at 100ms/1M trades means ~27ms wall time. No chunking needed for single-day processing.

**Primary recommendation:** Modify `midnight.py` to add a `_process_parquet_ticks()` function that reads today's Parquet, processes ticks per (symbol, threshold) via `process_trades_arrow_native()`, writes bars to ClickHouse via `store_bars_batch()`, and returns `{symbol: last_completed_bar_tid}` for seeding `fill_from_rest`.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

- Read today's Parquet ticks via `TickStorage.read_ticks()` (returns Polars LazyFrame)
- Convert to Arrow and process through `OpenDeviationBarProcessor.process_trades_arrow_native()` (microsecond timestamps)
- Process each symbol x threshold combination independently
- Write resulting bars to ClickHouse via existing `cache_write()` bulk operations
- Seed engine with `last_completed_bar_tid` (NOT Parquet max_tid) so fill_from_rest picks up forming bar correctly
- TickStorage uses `BINANCE_SPOT_BTCUSDT` directory names, sidecar uses `BTCUSDT` -- must map between naming conventions (prefix `BINANCE_SPOT_`)
- Parquet processing produces completed bars + leaves a forming bar in progress; seed fill_from_rest with last COMPLETED bar's `last_agg_trade_id`; fill_from_rest creates fresh processors starting from that TID, naturally re-processing forming bar's tail trades
- Keep 600s staleness threshold -- if Parquet stale, fall back to full REST fill from crossover TID
- Parquet processing failure is non-fatal: fall back to crossover TID seed

### Claude's Discretion

- Chunking strategy for large Parquet files (500K batch size matches existing `process_trades_polars`)
- Whether to parallelize across symbols or process sequentially
- Exact log format and telemetry fields

### Deferred Ideas (OUT OF SCOPE)

- Feeding Parquet ticks directly into LiveBarEngine's Rust-side `fill_from_rest()` (would require Rust API changes)
- Using checkpoints to carry forming bar state from Parquet processing to fill_from_rest (complex, unnecessary since re-processing a few trades is cheap)

</user_constraints>

<phase_requirements>

## Phase Requirements

| ID          | Description                                                                                          | Research Support                                                                                                                                                           |
| ----------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SEED-FIX-01 | On sidecar restart, process Parquet ticks into bars via Rust processor instead of skipping past them | All APIs verified: `TickStorage.read_ticks()`, `process_trades_arrow_native()`, `store_bars_batch()`. Naming convention mapping documented. Atomic write safety confirmed. |
| SEED-FIX-02 | Seed fill_from_rest with last_completed_bar_tid from Parquet processing, not Parquet max_tid         | `processor.last_completed_bar_tid()` API verified in `core.py:685-694` and `core_bindings.rs:622-626`. Returns last COMPLETED bar's TID, excluding forming bar tail.       |

</phase_requirements>

## Architecture Patterns

### Current Startup Sequence (Broken)

```
_create_engine() in sidecar/__init__.py:
  Step 1: Midnight preflight → crossover TID seeds (e.g., BTCUSDT@3915404668)
  Step 2: Yesterday integrity check
  Step 3: Kill mutations + OPTIMIZE + DELETE today (clean slate)
  Step 4: _read_parquet_max_tids() → OVERWRITES seed with max_tid (BTCUSDT@3915679334)
  Step 5: fill_from_rest starts from max_tid → 274K trades NEVER processed into bars
```

### Fixed Startup Sequence

```
_create_engine() in sidecar/__init__.py:
  Step 1: Midnight preflight → crossover TID seeds
  Step 2: Yesterday integrity check
  Step 3: Kill mutations + OPTIMIZE + DELETE today (clean slate)
  Step 4: _process_parquet_ticks() →
      4a: Read today's Parquet per symbol (TickStorage.read_ticks)
      4b: For each (symbol, threshold): process ticks → bars via process_trades_arrow_native
      4c: Write bars to ClickHouse via store_bars_batch
      4d: Seed engine with last_completed_bar_tid (NOT max agg_trade_id)
  Step 5: fill_from_rest starts from last_completed_bar_tid → fills only remaining gap
```

### Recommended File Changes

```
python/opendeviationbar/sidecar/
├── midnight.py          # MODIFY: Add _process_parquet_ticks(), modify _read_parquet_max_tids()
└── __init__.py          # MODIFY: Replace parquet-first block (lines 366-391) with new logic
```

### Pattern: Parquet Tick Processing

```python
# Source: verified from codebase analysis
from opendeviationbar.storage.parquet import TickStorage
from opendeviationbar.processors.core import OpenDeviationBarProcessor
import polars as pl

storage = TickStorage()

# 1. Read today's Parquet ticks for a symbol
# TickStorage uses bare symbol names for spot ("BTCUSDT")
# NOT "BINANCE_SPOT_BTCUSDT" (that's the checkpoint.py convention)
today_str = datetime.now(UTC).strftime("%Y-%m-%d")
parquet_path = storage.ticks_dir / symbol / f"{today_str}.parquet"

# read_ticks returns materialized pl.DataFrame (lazy scan + collect)
ticks = storage.read_ticks(symbol, start_ts=None, end_ts=None)

# 2. Convert to Arrow and process per threshold
arrow_table = ticks.to_arrow()
processor = OpenDeviationBarProcessor(threshold, symbol=symbol)
bars_arrow = processor.process_trades_arrow_native(arrow_table)

# 3. Get seeding TID
last_completed_tid = processor.last_completed_bar_tid()
# This excludes forming bar tail -- critical for correct fill_from_rest junction

# 4. Write bars to ClickHouse
bars_pl = pl.from_arrow(bars_arrow)
cache.store_bars_batch(symbol=symbol, threshold_decimal_bps=threshold, bars=bars_pl, ...)
```

### Anti-Patterns to Avoid

- **Seeding with max(agg_trade_id) from Parquet**: This is the current bug. max_tid includes trades in the forming bar that haven't completed a bar yet. fill_from_rest must re-process those trades to complete the forming bar.
- **Using process_trades_arrow() instead of process_trades_arrow_native()**: The non-native version expects millisecond timestamps. All Parquet ticks are microseconds post Phase 4/5.
- **Reading with BINANCE*SPOT* prefix**: The seeder uses `_cache_key()` which returns bare symbol for spot (e.g., "BTCUSDT"), NOT "BINANCE_SPOT_BTCUSDT". The sidecar's `_read_parquet_max_tids()` already uses `ticks_dir / symbol / today.parquet` (bare symbol). No prefix mapping needed.

## Critical Findings

### 1. Parquet Concurrent Read/Write Safety (HIGH confidence)

**Finding: SAFE. Atomic write pattern prevents corruption.**

`_atomic_write_parquet()` in `parquet_io.py:142-206` implements:

1. Create temp file in same directory (`.parquet_*.tmp`)
2. Write to temp file via `df.write_parquet()`
3. `os.fsync()` the temp file
4. `temp_path.replace(target_path)` -- POSIX atomic rename on same filesystem

This means the sidecar reading `storage.read_ticks()` will see either:

- The **previous complete** Parquet file (if read happens before rename), or
- The **new complete** Parquet file (if read happens after rename)

It will **never** see a partial/corrupted write. The seeder writes every 5 minutes; the sidecar reads once on startup. Race condition is benign -- worst case is reading a slightly older version (5 min stale), which is well within the 600s staleness threshold.

### 2. Naming Convention: No Prefix Needed (HIGH confidence)

**Finding: Seeder uses bare symbol names for spot market, matching sidecar expectations.**

The seeder's `_cache_key()` function (line 562-570):

```python
def _cache_key(symbol: str, market: str) -> str:
    if market == "um_futures":
        return f"BINANCE_UM_{symbol}"
    return symbol  # bare "BTCUSDT" for spot
```

And `_read_parquet_max_tids()` already uses `ticks_dir / symbol / f"{today_str}.parquet"` (bare symbol). **No BINANCE*SPOT* prefix mapping needed** -- contrary to what CONTEXT.md states. The CONTEXT.md note about mapping conventions is about `checkpoint.py` (which uses `BINANCE_SPOT_`), not the seeder. The sidecar and seeder already agree on bare symbol names.

### 3. Microsecond Timestamps (HIGH confidence)

**Finding: `process_trades_arrow_native()` is the correct path. Both arrow methods pass `timestamp_is_microseconds=true` to Rust.**

From `core_bindings.rs`:

- `process_trades_arrow()` (line 455-477): calls `process_from_arrow_columns(processor, record_batch, true)` -- the `true` means `timestamp_is_microseconds=true`
- `process_trades_arrow_native()` (line 495-517): calls `process_from_arrow_columns(processor, record_batch, true)` -- identical

**Both methods pass `true` for microsecond timestamps.** The difference is API intent: `process_trades_arrow()` is the "public" path, `process_trades_arrow_native()` is the "internal" path. Both actually handle microseconds identically because the comment at line 468 confirms: "ALL Parquet ticks are microseconds after Phase 4/5 reseed."

The Parquet schema columns (from `_CANONICAL_COLS` in `parquet.py:251-253`):

```python
["agg_trade_id", "price", "quantity", "first_trade_id", "last_trade_id", "timestamp", "is_buyer_maker"]
```

The `timestamp` column is Int64 microseconds. The Arrow column mapping in Rust (`process_from_arrow_columns`) expects: `timestamp` (Int64), `price` (Float64), `quantity`/`volume` (Float64).

### 4. Memory & Chunking Analysis (HIGH confidence)

**Finding: No chunking needed for single-day processing. 274K trades = ~16 MB.**

Memory breakdown for BTCUSDT (largest symbol, ~274K trades on restart day):

- Parquet on disk: ~2.5 MB (ZSTD-3 compressed)
- Polars DataFrame: ~274K x 7 cols x 8 bytes = ~15 MB
- Arrow table: same as Polars (zero-copy conversion)
- Rust processing: additional ~50K bars x 200 bytes = ~10 MB
- Total: ~25 MB per symbol

All 16 Binance symbols combined: ~25 MB x 16 = ~400 MB worst case (if all have full-day data). This is well within the sidecar's memory budget (MemoryMax=10G in systemd).

The existing `process_trades_polars()` uses 500K chunk size because it handles multi-month datasets. For single-day restart processing, a single batch is appropriate. However, to match the existing pattern and be defensive, using the same 500K chunk size adds negligible overhead.

### 5. ClickHouse Write Path (HIGH confidence)

**Finding: Use `store_bars_batch()` with `overlap_strategy="off"` (trust mode).**

The sidecar already uses `store_bars_batch()` for sync_flush (line 571 in `midnight.py`). Since we just did a clean-slate DELETE of today's bars in Step 3, there are zero existing bars to conflict with. `overlap_strategy="off"` is correct.

The bars from `process_trades_arrow_native()` need to be converted from Arrow RecordBatch to Polars DataFrame for `store_bars_batch()`:

```python
bars_arrow = processor.process_trades_arrow_native(arrow_table)
bars_pl = pl.from_arrow(bars_arrow)
```

`store_bars_batch()` expects a Polars DataFrame with the standard bar columns (open_time_us, close_time_us, open, high, low, close, volume, etc.).

### 6. process_trades_arrow() vs process_trades_arrow_native() (HIGH confidence)

**Critical discovery:** Both `process_trades_arrow()` and `process_trades_arrow_native()` in `core_bindings.rs` call the same underlying `process_from_arrow_columns(processor, record_batch, true)` with `timestamp_is_microseconds=true`. They produce identical results for microsecond input data.

The CONTEXT.md decision to use `process_trades_arrow_native()` is correct by intent (it's the "internal microsecond path"), but functionally identical. Either would work.

### 7. Forming Bar Junction (HIGH confidence)

**Finding: `last_completed_bar_tid()` is the correct seed, not `last_agg_trade_id()`.**

From `core.py:685-694`:

```python
def last_completed_bar_tid(self) -> int | None:
    """Get last COMPLETED bar's aggregate trade ID.
    Unlike last_agg_trade_id() which includes the forming bar tail,
    this returns the trade ID from the most recently completed or orphaned bar.
    """
```

Seeding fill_from_rest with `last_completed_bar_tid` means:

1. fill_from_rest creates a fresh processor starting from `last_completed_bar_tid`
2. It re-fetches and re-processes the forming bar's tail trades via REST
3. The forming bar completes naturally when new trades arrive
4. A few trades are processed twice (once from Parquet, once from REST) -- correct and cheap

If `last_completed_bar_tid()` returns `None` (no bars produced from Parquet), fall back to the crossover TID seed (unchanged from current behavior).

## Don't Hand-Roll

| Problem               | Don't Build                      | Use Instead                           | Why                                                           |
| --------------------- | -------------------------------- | ------------------------------------- | ------------------------------------------------------------- |
| Arrow conversion      | Custom Parquet-to-Arrow pipeline | `ticks.to_arrow()` (Polars zero-copy) | Polars handles schema mapping correctly                       |
| Microsecond detection | Manual timestamp scale checks    | `process_trades_arrow_native()`       | Rust binding already handles microseconds natively            |
| Bar writing to CH     | Custom INSERT logic              | `store_bars_batch()`                  | Already handles dedup tokens, volume guards, schema alignment |
| Forming bar junction  | Custom TID extraction from bars  | `processor.last_completed_bar_tid()`  | Rust-side API designed for exactly this use case              |

## Common Pitfalls

### Pitfall 1: Seeding with Parquet max_tid Instead of last_completed_bar_tid

**What goes wrong:** This is the CURRENT BUG. fill_from_rest starts past the forming bar boundary, leaving 274K+ trades unprocessed.
**Why it happens:** `max(agg_trade_id)` from Parquet includes trades that are part of a forming (incomplete) bar. Those trades need to be re-processed to produce bars.
**How to avoid:** Use `processor.last_completed_bar_tid()` after processing. If None, fall back to crossover TID.
**Warning signs:** Large gap between Parquet max_tid and first bar's first_agg_trade_id.

### Pitfall 2: Using BINANCE*SPOT* Prefix for Parquet Reads

**What goes wrong:** File not found -- Parquet files for spot are stored under bare symbol names.
**Why it happens:** Confusion between checkpoint.py convention (BINANCE*SPOT*) and seeder convention (bare symbol).
**How to avoid:** Use bare symbol name: `ticks_dir / "BTCUSDT" / "2026-03-25.parquet"`.
**Warning signs:** `_read_parquet_max_tids()` already uses bare symbol names and works.

### Pitfall 3: Forgetting to Process All Thresholds

**What goes wrong:** Only one threshold gets Parquet bars; others have gaps until fill_from_rest.
**Why it happens:** Processing ticks once and writing to one threshold.
**How to avoid:** Loop over `config.thresholds` and create a separate `OpenDeviationBarProcessor` for each.
**Warning signs:** Only one threshold has bars from the Parquet time range.

### Pitfall 4: Not Handling Empty Parquet or No Bars Produced

**What goes wrong:** Crash or incorrect seed on symbols with few trades (e.g., FILUSDT).
**Why it happens:** Parquet exists but has very few trades, processor produces zero complete bars.
**How to avoid:** Check `last_completed_bar_tid()` for None; fall back to crossover TID seed.
**Warning signs:** Crash on `.item()` or similar when result is empty.

## Code Examples

### Main Processing Function

```python
# Source: synthesized from verified codebase patterns
def _process_parquet_ticks(
    symbols: list[str],
    thresholds: list[int],
    ticks_dir: Path | None = None,
) -> dict[str, int]:
    """Process today's Parquet ticks into bars and return seed TIDs.

    Returns dict of {symbol: last_completed_bar_tid} for seeding fill_from_rest.
    """
    from datetime import UTC, datetime
    from opendeviationbar.storage.parquet import TickStorage
    from opendeviationbar.processors.core import OpenDeviationBarProcessor
    from opendeviationbar.clickhouse.cache import OpenDeviationBarCache
    from opendeviationbar import __version__
    import polars as pl

    staleness_threshold_s = 600
    storage = TickStorage()
    if ticks_dir is None:
        ticks_dir = storage.ticks_dir

    today_str = datetime.now(UTC).strftime("%Y-%m-%d")
    result: dict[str, int] = {}

    for symbol in symbols:
        parquet_path = ticks_dir / symbol / f"{today_str}.parquet"
        if not parquet_path.exists():
            continue

        age_s = time.time() - parquet_path.stat().st_mtime
        if age_s > staleness_threshold_s:
            logger.info("parquet-process: %s stale (%.0fs), skipping", symbol, age_s)
            continue

        # Read all ticks for today
        try:
            ticks = pl.scan_parquet(parquet_path).collect()
        except Exception:
            logger.warning("parquet-process: failed to read %s", symbol, exc_info=True)
            continue

        if ticks.is_empty():
            continue

        # Convert to Arrow (zero-copy from Polars)
        arrow_table = ticks.to_arrow()

        # Track best (highest) last_completed_bar_tid across all thresholds
        best_tid: int | None = None
        total_bars = 0

        with OpenDeviationBarCache() as cache:
            for threshold in thresholds:
                processor = OpenDeviationBarProcessor(threshold, symbol=symbol)
                bars_arrow = processor.process_trades_arrow_native(arrow_table)
                bars_pl = pl.from_arrow(bars_arrow)

                if not bars_pl.is_empty():
                    cache.store_bars_batch(
                        symbol=symbol,
                        threshold_decimal_bps=threshold,
                        bars=bars_pl,
                        version=f"{__version__}+parquet",
                        overlap_strategy="off",
                    )
                    total_bars += len(bars_pl)

                tid = processor.last_completed_bar_tid()
                if tid is not None and (best_tid is None or tid > best_tid):
                    best_tid = tid

        if best_tid is not None:
            result[symbol] = best_tid
            logger.info(
                "parquet-process: %s — %d bars across %d thresholds, seed_tid=%d (age=%.0fs)",
                symbol, total_bars, len(thresholds), best_tid, age_s,
            )

    return result
```

### Integration in \_create_engine

```python
# Source: synthesized from current sidecar/__init__.py:366-391
# Replace the current parquet-first block with:
try:
    from .midnight import _process_parquet_ticks

    parquet_seeds = _process_parquet_ticks(config.symbols, config.thresholds)
    if parquet_seeds:
        for symbol, seed_tid in parquet_seeds.items():
            for threshold in config.thresholds:
                engine.seed_trade_id(symbol, threshold, seed_tid)
        logger.info(
            "parquet-process: seeded %d/%d symbols: %s",
            len(parquet_seeds),
            len(config.symbols),
            ", ".join(f"{s}@{tid}" for s, tid in sorted(parquet_seeds.items())),
        )
except Exception:
    logger.warning(
        "parquet-process: failed (non-fatal, falling back to preflight seeds)",
        exc_info=True,
    )
```

## Validation Architecture

### Test Framework

| Property           | Value                                                     |
| ------------------ | --------------------------------------------------------- |
| Framework          | pytest                                                    |
| Config file        | `pyproject.toml` [tool.pytest.ini_options]                |
| Quick run command  | `uv run --python 3.13 pytest tests/test_sidecar.py -x -q` |
| Full suite command | `mise run test-py`                                        |

### Phase Requirements -> Test Map

| Req ID      | Behavior                                     | Test Type | Automated Command                                                                  | File Exists? |
| ----------- | -------------------------------------------- | --------- | ---------------------------------------------------------------------------------- | ------------ |
| SEED-FIX-01 | Parquet ticks processed into bars on restart | unit      | `pytest tests/test_parquet_bar_processing.py -x`                                   | Wave 0       |
| SEED-FIX-02 | Engine seeded with last_completed_bar_tid    | unit      | `pytest tests/test_parquet_bar_processing.py::test_seed_uses_completed_bar_tid -x` | Wave 0       |

### Sampling Rate

- **Per task commit:** `uv run --python 3.13 pytest tests/test_parquet_bar_processing.py -x`
- **Per wave merge:** `mise run test-py`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- [ ] `tests/test_parquet_bar_processing.py` -- covers SEED-FIX-01, SEED-FIX-02 (new file)
- [ ] Test fixtures: mock Parquet files with known trade data for deterministic bar verification

## Sources

### Primary (HIGH confidence)

- `python/opendeviationbar/storage/parquet_io.py` -- `_atomic_write_parquet()` implementation verified (lines 142-206)
- `python/opendeviationbar/storage/parquet.py` -- `TickStorage.read_ticks()` and `write_ticks()` verified
- `src/core_bindings.rs` -- `process_trades_arrow_native()` Rust binding verified (lines 495-517)
- `python/opendeviationbar/processors/core.py` -- `last_completed_bar_tid()` API verified (lines 685-694)
- `python/opendeviationbar/sidecar/__init__.py` -- Current startup sequence verified (lines 253-451)
- `python/opendeviationbar/sidecar/midnight.py` -- `_read_parquet_max_tids()` current implementation verified (lines 690-756)
- `scripts/tick_cache_seeder.py` -- `_cache_key()` naming convention verified (lines 562-570)

### Secondary (MEDIUM confidence)

- CONTEXT.md naming convention note: partially incorrect (says BINANCE*SPOT* prefix needed; actually seeder uses bare symbol names for spot). The note applies to checkpoint.py, not seeder.

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH -- all APIs exist and verified in source code
- Architecture: HIGH -- straightforward pipeline: read Parquet -> process Arrow -> write CH -> seed engine
- Pitfalls: HIGH -- root cause verified from production logs, all edge cases documented

**Research date:** 2026-03-25
**Valid until:** 2026-04-25 (stable; core APIs unlikely to change)
