---
phase: 03-kintsugi-parquet-trust
plan: 01
type: execute
wave: 1
depends_on: []
files_modified:
  - python/opendeviationbar/kintsugi/repair.py
  - tests/test_kintsugi_parquet_trust.py
autonomous: true
requirements: [TRUST-01, TRUST-02]

must_haves:
  truths:
    - "Kintsugi skips incomplete Parquet files and falls back to Vision/REST"
    - "Orphan bars from Parquet and Vision repair paths have is_orphan=True"
    - "Complete Parquet files are still used without regression"
  artifacts:
    - path: "python/opendeviationbar/kintsugi/repair.py"
      provides: "Parquet completeness gate + orphan bar fix"
      contains: "get_crossover_tid"
    - path: "tests/test_kintsugi_parquet_trust.py"
      provides: "Tests for completeness gate and orphan marking"
      min_lines: 40
  key_links:
    - from: "python/opendeviationbar/kintsugi/repair.py"
      to: "python/opendeviationbar/crossover.py"
      via: "get_crossover_tid(symbol, next_day)"
      pattern: "get_crossover_tid"
    - from: "python/opendeviationbar/kintsugi/repair.py"
      to: "python/opendeviationbar/storage/parquet.py"
      via: "TickStorage.read_ticks() max agg_trade_id"
      pattern: "agg_trade_id.*max"
---

<objective>
Add Parquet completeness verification gate and orphan bar marking to kintsugi repair paths.

Purpose: Prevent kintsugi from using incomplete Parquet files (missing trailing trades near midnight boundary) as repair sources, which would perpetuate gaps. Also fix orphan bars from Parquet/Vision paths that currently lack `is_orphan=True`.

Output: Modified `repair.py` with completeness gate in `_try_parquet_day_replace()` and orphan marking in both Parquet and Vision paths. Tests proving both behaviors.
</objective>

<execution_context>
@$HOME/.claude/get-shit-done/workflows/execute-plan.md
@$HOME/.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/01-crossover-tid-utility/01-01-SUMMARY.md

@python/opendeviationbar/kintsugi/repair.py
@python/opendeviationbar/crossover.py
@python/opendeviationbar/orchestration/helpers.py
@python/opendeviationbar/storage/parquet.py

<interfaces>
<!-- Key types and contracts the executor needs -->

From python/opendeviationbar/crossover.py:

```python
def get_crossover_tid(symbol: str, date: datetime.date | None = None) -> int | None:
    """First agg_trade_id at or after UTC midnight for a date. None on failure."""
```

From python/opendeviationbar/storage/parquet.py:

```python
class TickStorage:
    def has_ticks(self, symbol, start_ts_ms, end_ts_ms, min_coverage=0.95) -> bool: ...
    def read_ticks(self, symbol, start_ts_ms, end_ts_ms) -> pl.DataFrame: ...
    # Parquet files have column: agg_trade_id (sorted ascending within each daily file)
```

From python/opendeviationbar/orchestration/helpers.py:

```python
def _stream_bars_with_processor(trade_batches, processor, include_incomplete=False, batch_size=10_000):
    """Yields pl.DataFrame batches. When include_incomplete=True, flushes the forming bar via
    processor.get_incomplete_bar() as the final batch. The incomplete bar dict does NOT have
    is_orphan key -- caller must set it."""
```

From python/opendeviationbar/kintsugi/repair.py:

```python
def _try_parquet_day_replace(*, shard, day_start_us, day_end_us, include_microstructure) -> int:
    """Returns bar count written. Returns 0 on failure (caller falls through to Vision)."""

def _try_vision_day_replace(*, shard, repair_date, include_microstructure) -> int:
    """Returns bar count written. Returns 0 on failure (caller falls through to REST)."""
```

</interfaces>
</context>

<tasks>

<task type="auto" tdd="true">
  <name>Task 1: Parquet completeness gate and orphan bar marking</name>
  <files>python/opendeviationbar/kintsugi/repair.py, tests/test_kintsugi_parquet_trust.py</files>
  <behavior>
    - Test 1 (TRUST-01): _try_parquet_day_replace returns 0 when Parquet max(agg_trade_id) < crossover_tid - 1 for the next day
    - Test 2 (TRUST-01): _try_parquet_day_replace proceeds normally when max(agg_trade_id) >= crossover_tid - 1
    - Test 3 (TRUST-01): _try_parquet_day_replace proceeds when get_crossover_tid returns None (crossover lookup failed -- don't block repair)
    - Test 4 (TRUST-02): Parquet path sets is_orphan=True on the last bar when include_incomplete=True produces an incomplete bar
    - Test 5 (TRUST-02): Vision path sets is_orphan=True on the last bar produced by _stream_open_deviation_bars_binance with include_incomplete=True
  </behavior>
  <action>
**TRUST-01 — Parquet completeness gate in `_try_parquet_day_replace()`:**

After the `has_ticks` check (line ~370) and before reading ticks (line ~389), add a completeness verification block:

1. Import `get_crossover_tid` from `opendeviationbar.crossover` and `datetime`
2. Compute the NEXT day's date from `day_start_us`: `next_day = datetime.date.fromtimestamp(day_start_us / 1_000_000) + timedelta(days=1)`
3. Call `crossover_tid = get_crossover_tid(shard.symbol, next_day)` -- this gives the first TID of the next day, meaning all trades up to `crossover_tid - 1` belong to the current day
4. If `crossover_tid is not None`: use Polars lazy scan to get `max(agg_trade_id)` from the Parquet file WITHOUT loading all ticks into memory. Use `storage.read_ticks(shard.symbol, day_start_ms, day_end_ms - 1)` with a `.select(pl.col("agg_trade_id").max())` approach. Actually simpler: after the existing `day_ticks = storage.read_ticks(...)` call (line 389), check `day_ticks["agg_trade_id"].max()` against `crossover_tid - 1`. Move the completeness check to AFTER ticks are loaded (line 391, after the `is_empty` check) to avoid a double-read.
5. If `max_tid < crossover_tid - 1`: log WARNING with the gap size (`crossover_tid - 1 - max_tid` missing trades) and `return 0` (falls through to Vision/REST)
6. If `crossover_tid is None`: log DEBUG "crossover lookup failed, proceeding with Parquet anyway" -- don't block repair when the crossover API is down

**TRUST-02 — Orphan bar marking in Parquet path (`_try_parquet_day_replace`):**

After the `bars_df = pl.concat(bar_frames, ...)` line (~421), before `store_bars_batch`:

1. Check if the last bar in `bars_df` came from `get_incomplete_bar()` (it did if `include_incomplete=True` was used, which it always is in this function)
2. The incomplete bar is always the LAST row. Add: `bars_df = bars_df.with_columns(pl.when(pl.arange(0, pl.count()) == pl.count() - 1).then(pl.lit(True)).otherwise(pl.lit(False)).alias("is_orphan"))` -- but simpler: since ALL bars except the last are complete, just set `is_orphan` column: create a boolean series that is False for all rows except the last which is True.
3. Implementation: `is_orphan_col = [False] * (len(bars_df) - 1) + [True]` then `bars_df = bars_df.with_columns(pl.Series("is_orphan", is_orphan_col))`

**TRUST-02 — Orphan bar marking in Vision path (`_try_vision_day_replace`):**

Same pattern after `bars_df = pl.concat(bar_frames, ...)` (~546) and before `store_bars_batch`:

1. `is_orphan_col = [False] * (len(bars_df) - 1) + [True]` then `bars_df = bars_df.with_columns(pl.Series("is_orphan", is_orphan_col))`

**Tests** (in `tests/test_kintsugi_parquet_trust.py`):

Mock `get_crossover_tid`, `TickStorage`, `_stream_bars_with_processor`, `_get_worker_cache` to test the gate and orphan marking in isolation. Use `unittest.mock.patch` targeting the import paths within `opendeviationbar.kintsugi.repair`.
</action>
<verify>
<automated>cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_kintsugi_parquet_trust.py -xvs 2>&1 | tail -30</automated>
</verify>
<done> - Parquet path returns 0 with WARNING log when max TID < crossover_tid - 1 (TRUST-01) - Parquet path proceeds normally when complete or crossover unavailable (TRUST-01) - Last bar in Parquet repair output has is_orphan=True (TRUST-02) - Last bar in Vision repair output has is_orphan=True (TRUST-02) - All 5 tests pass
</done>
</task>

</tasks>

<verification>
```bash
# All new tests pass
cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/test_kintsugi_parquet_trust.py -xvs

# Existing kintsugi tests still pass (no regression)

cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 pytest tests/ -k "kintsugi" -x --timeout=60

# Lint check

cd /Users/terryli/eon/opendeviationbar-py && uv run --python 3.13 ruff check python/opendeviationbar/kintsugi/repair.py

```
</verification>

<success_criteria>
1. Kintsugi rejects incomplete Parquet files (max TID below crossover boundary) with a WARNING log and falls through to Vision/REST
2. Complete Parquet files still used without any behavior change
3. Orphan bars from both Parquet and Vision repair paths have `is_orphan=True` set on the last bar
4. All tests pass, no regressions in existing kintsugi tests
</success_criteria>

<output>
After completion, create `.planning/phases/03-kintsugi-parquet-trust/03-01-SUMMARY.md`
</output>
```
