---
phase: 06-fix-seed-001-process-parquet-ticks
verified: 2026-03-25T10:15:00Z
status: passed
score: 3/3 must-haves verified
re_verification: false
---

# Phase 6: Fix SEED-001 Parquet-First Bar Processing — Verification Report

**Phase Goal:** On sidecar restart, process Parquet ticks into bars via Rust processor instead of skipping past them. Eliminates 274K+ trade gaps caused by SEED-001 advancing the REST seed past deleted bars.
**Verified:** 2026-03-25T10:15:00Z
**Status:** passed
**Re-verification:** No — initial verification

---

## Goal Achievement

### Observable Truths

| #   | Truth                                                                                                                       | Status   | Evidence                                                                                                                                                                                                                          |
| --- | --------------------------------------------------------------------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1   | On sidecar restart with fresh Parquet, ticks are processed into bars via Rust processor and written to ClickHouse           | VERIFIED | `_process_parquet_ticks()` calls `process_trades_arrow_native()` + `store_bars_batch()` in midnight.py:838,844. 7 unit tests pass including `test_fresh_parquet_processes_bars`.                                                  |
| 2   | Engine is seeded with `last_completed_bar_tid` (not Parquet max_tid) so `fill_from_rest` picks up the forming bar correctly | VERIFIED | midnight.py:853 calls `processor.last_completed_bar_tid()`. `test_seed_uses_completed_bar_tid` asserts result is 5000 when max agg_trade_id is 5500.                                                                              |
| 3   | Stale or missing Parquet falls back to crossover TID seeds (no regression from current behavior)                            | VERIFIED | Staleness guard at midnight.py:812-820 skips symbols with mtime > 600s. `test_stale_parquet_skipped` and `test_missing_parquet_returns_empty` both pass. `test_parquet_first_catchup.py` (8 tests) still green — zero regression. |

**Score:** 3/3 truths verified

---

### Required Artifacts

| Artifact                                      | Expected                                                     | Status   | Details                                                                                                                                                                                                |
| --------------------------------------------- | ------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `python/opendeviationbar/sidecar/midnight.py` | `_process_parquet_ticks()` function                          | VERIFIED | Exists at line 759. Substantive (~116 lines). Wired from `__init__.py` via `from .midnight import _process_parquet_ticks`.                                                                             |
| `python/opendeviationbar/sidecar/__init__.py` | Updated parquet-first block calling `_process_parquet_ticks` | VERIFIED | Lines 366-390. Old `_read_parquet_max_tids` import fully removed (grep returns no matches). `parquet_seeds` variable, `parquet-process:` log prefix, `# SEED-FIX-01, SEED-FIX-02` comment all present. |
| `tests/test_parquet_bar_processing.py`        | 7 unit tests for Parquet tick processing and seeding         | VERIFIED | 351-line test file. All 7 tests pass. Contains `test_seed_uses_completed_bar_tid`, `test_fresh_parquet_processes_bars`, `test_all_thresholds_processed`.                                               |

---

### Key Link Verification

| From                  | To                              | Via                                                       | Status | Details                                                                                                                                   |
| --------------------- | ------------------------------- | --------------------------------------------------------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------------- |
| `sidecar/__init__.py` | `sidecar/midnight.py`           | `from .midnight import _process_parquet_ticks`            | WIRED  | `__init__.py` line 370 imports and calls `_process_parquet_ticks(config.symbols, config.thresholds)` at line 372.                         |
| `sidecar/midnight.py` | `storage/parquet.py`            | `TickStorage().read_ticks(symbol)`                        | WIRED  | midnight.py:799 creates `TickStorage()`, midnight.py:824 calls `storage.read_ticks(symbol)`.                                              |
| `sidecar/midnight.py` | `processors/core.py`            | `OpenDeviationBarProcessor.process_trades_arrow_native()` | WIRED  | midnight.py:793-794 lazy-imports `OpenDeviationBarProcessor`. Called at midnight.py:838.                                                  |
| `sidecar/midnight.py` | `clickhouse/bulk_operations.py` | `cache.store_bars_batch()`                                | WIRED  | midnight.py:793 lazy-imports `OpenDeviationBarCache`. `cache.store_bars_batch()` called at midnight.py:844 with `overlap_strategy="off"`. |

---

### Data-Flow Trace (Level 4)

| Artifact                   | Data Variable                    | Source                                                                   | Produces Real Data                                                            | Status  |
| -------------------------- | -------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------------- | ------- |
| `_process_parquet_ticks()` | `ticks` (Polars DataFrame)       | `TickStorage().read_ticks(symbol)` — reads Parquet files from filesystem | Yes — reads real Parquet tick files; empty-DataFrame guard at midnight.py:825 | FLOWING |
| `_process_parquet_ticks()` | `bars_arrow` (Arrow RecordBatch) | `processor.process_trades_arrow_native(arrow_table)` — Rust processor    | Yes — Rust processes trades and returns actual bars                           | FLOWING |
| `_process_parquet_ticks()` | `best_tid` (int)                 | `processor.last_completed_bar_tid()` — returns last completed bar's TID  | Yes — derived from Rust processor state, NOT from Parquet max(agg_trade_id)   | FLOWING |

---

### Behavioral Spot-Checks

| Behavior                                                | Command                                                                  | Result                       | Status |
| ------------------------------------------------------- | ------------------------------------------------------------------------ | ---------------------------- | ------ |
| All 7 new tests pass                                    | `uv run --python 3.13 pytest tests/test_parquet_bar_processing.py -x -q` | 7 passed, 0 failed           | PASS   |
| Existing parquet-first tests pass (regression)          | `uv run --python 3.13 pytest tests/test_parquet_first_catchup.py -x -q`  | 8 passed, 0 failed           | PASS   |
| Old `_read_parquet_max_tids` removed from `__init__.py` | `grep -c "_read_parquet_max_tids" sidecar/__init__.py`                   | 0 matches                    | PASS   |
| `_process_parquet_ticks` present in midnight.py         | `grep -n "def _process_parquet_ticks"`                                   | Found at line 759            | PASS   |
| `last_completed_bar_tid` used (not max agg_trade_id)    | `grep -n "last_completed_bar_tid" midnight.py`                           | Found at lines 772, 787, 853 | PASS   |
| `overlap_strategy="off"` used in store_bars_batch       | `grep -n "overlap_strategy" midnight.py`                                 | Found at line 849            | PASS   |

---

### Requirements Coverage

| Requirement | Source Plan   | Description                                                                | Status    | Evidence                                                                                                                                                                                                                                      |
| ----------- | ------------- | -------------------------------------------------------------------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SEED-FIX-01 | 06-01-PLAN.md | On sidecar restart, process Parquet ticks into bars via Rust processor     | SATISFIED | `_process_parquet_ticks()` in midnight.py reads ticks via `TickStorage.read_ticks()`, processes via `process_trades_arrow_native()`, writes to ClickHouse. `test_fresh_parquet_processes_bars` + `test_all_thresholds_processed` verify this. |
| SEED-FIX-02 | 06-01-PLAN.md | Seed `fill_from_rest` with `last_completed_bar_tid`, not Parquet `max_tid` | SATISFIED | midnight.py:853 uses `processor.last_completed_bar_tid()`. `test_seed_uses_completed_bar_tid` explicitly asserts result=5000 when Parquet max_agg_trade_id=5500.                                                                              |

**Note on REQUIREMENTS.md coverage:** SEED-FIX-01 and SEED-FIX-02 are defined in ROADMAP.md (Phase 6, v5.1 milestone) but are not listed in `.planning/REQUIREMENTS.md` which covers only v5.0 requirements. This is an expected gap — REQUIREMENTS.md has not been updated to include v5.1 requirements. No orphaned requirements exist within the scope of v5.0 REQUIREMENTS.md for this phase.

---

### Anti-Patterns Found

| File | Line | Pattern    | Severity | Impact |
| ---- | ---- | ---------- | -------- | ------ |
| —    | —    | None found | —        | —      |

No TODOs, FIXMEs, stub returns, or empty implementations found in the modified files for this phase. The old `_read_parquet_max_tids()` function is preserved in `midnight.py` (backward compatibility for existing tests) but is no longer called from `__init__.py`.

---

### Human Verification Required

#### 1. Production Gap Elimination Verification

**Test:** On bigblack, restart the sidecar and verify the heartbeat Stathera check shows zero intraday gaps after the Parquet-first fill completes.
**Expected:** Sidecar logs show `parquet-process: seeded N/18 symbols` at startup. Heartbeat (5 min after restart) reports no Stathera gaps for any symbol for today's date.
**Why human:** Requires live bigblack environment with real Parquet tick files, ClickHouse, and the running sidecar service. Cannot verify against production without a deploy.

---

### Gaps Summary

No gaps. All must-haves verified.

The implementation correctly replaces the broken SEED-001 "skip Parquet range" logic:

- Before: `_read_parquet_max_tids()` returned raw `max(agg_trade_id)` and seeded the engine directly, causing `fill_from_rest` to skip the forming-bar tail trades — producing 274K+ trade gaps.
- After: `_process_parquet_ticks()` runs the full Rust processor pipeline (TickStorage → Arrow → `process_trades_arrow_native` → ClickHouse), then seeds with `last_completed_bar_tid` so `fill_from_rest` correctly re-processes the forming bar's tail.

One human verification item remains: confirming the fix eliminates the 274K+ trade gaps in the production environment on bigblack after deploy.

---

_Verified: 2026-03-25T10:15:00Z_
_Verifier: Claude (gsd-verifier)_
