# Phase 2: Sidecar Fill Pipeline - Research

**Researched:** 2026-03-24
**Domain:** Sidecar startup fill pipeline (Rust StreamManager + Python orchestration)
**Confidence:** HIGH

## Summary

Phase 2 addresses STREAM-01 and STREAM-02: ensuring `fill_from_rest` returns computed bars (not raw trades) for immediate CH flush, and guaranteeing zero-trade gaps at the REST-to-WS junction boundary.

**The current codebase already implements most of this.** After reading the Rust `live_engine/mod.rs` and `symbol_task.rs`, `fill_from_rest()` already: (1) fetches trades from REST in Rust, (2) processes them through `OpenDeviationBarProcessor` per threshold, (3) pushes completed bars to a ring buffer, (4) preserves processors for WS reuse. The Python side drains bars via `_sync_flush_rest_bars()` and writes to CH. The junction fill logic in `symbol_task.rs` (JUNC-01 through JUNC-04) already handles the REST-to-WS boundary with monotonicity guards and gap detector re-seeding.

**Primary recommendation:** This phase is about verifying, hardening, and potentially fixing edge cases in the existing pipeline -- not building from scratch. The research found that the forming-bar abandonment bug documented in `rest-ws-junction-fix.md` was already fixed by the junction fill code in `symbol_task.rs`. The work remaining is: (1) verify STREAM-01/STREAM-02 hold under production conditions, (2) add assertion-level validation for Stathera continuity at the junction, and (3) ensure no intermediate Python trade DataFrame exists in the path.

<user_constraints>

## User Constraints (from CONTEXT.md)

### Locked Decisions

None -- all implementation choices are at Claude's discretion (pure infrastructure phase).

### Claude's Discretion

All implementation choices. Use ROADMAP phase goal, success criteria, and codebase conventions.

Key technical constraints from CLAUDE.md:

- `fill_from_rest` lives in `sidecar/midnight.py` and is called from `sidecar/__init__.py:365`
- `_sync_flush_rest_bars` in `midnight.py:641` handles the CH write path
- Trades must stay in Rust (`OpenDeviationBarProcessor.process_trades()`) -- no Python DataFrame intermediary
- Stathera continuity: `bar[last_rest].last_agg_trade_id + 1 == bar[first_ws].first_agg_trade_id`
- Writer ownership: sidecar owns today, kintsugi owns yesterday+older
- Clean-slate DELETE today runs BEFORE fill_from_rest (Step 3 of startup sequence)

### Carrying Forward from Phase 1

- Per-partition locks now available in `maintenance.py`
- All DELETEs use `DELETE IN PARTITION`

### Deferred Ideas (OUT OF SCOPE)

None.
</user_constraints>

<phase_requirements>

## Phase Requirements

| ID        | Description                                                                                               | Research Support                                                                                                                                                                                                                                               |
| --------- | --------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| STREAM-01 | `fill_from_rest` returns computed bars directly (not raw trades) for immediate `sync_flush` to ClickHouse | Already implemented in Rust: `fill_from_rest()` in `live_engine/mod.rs:227` processes trades through processors and pushes `CompletedBar` to ring buffer. Python drains via `drain_bars()` and writes to CH in `_sync_flush_rest_bars()`. Verification needed. |
| STREAM-02 | Gap between REST fill completion and WS junction is zero trades (no dropped trades at junction point)     | Junction fill logic in `symbol_task.rs` (JUNC-01 through JUNC-04) bridges REST-last-tid to WS-first-tid. Monotonicity guard prevents duplicate processing. Gap detector re-seeded past junction. Verification needed.                                          |

</phase_requirements>

## Project Constraints (from CLAUDE.md)

- **Python 3.13 ONLY** -- never use 3.14 or any other version
- **Leverage Rust** -- Python handles I/O only, all heavy lifting in Rust crates
- **Kintsugi-First Gap Filling** -- sole gap-filling mechanism for yesterday+older
- **Writer Ownership** -- sidecar owns today (UTC), kintsugi owns yesterday+older
- **Day-Mode Invariant** -- every bar belongs to exactly one UTC day
- **Sidecar Startup Sequence Invariant** -- Steps 1-6 in exact order
- **ClickHouse DELETE strategies** -- lightweight `DELETE FROM` permanently banned
- **Local-First CI/CD** -- no GitHub Actions for testing/linting

## Architecture Patterns

### Current Fill Pipeline (Already Implemented)

The fill pipeline flows as follows:

```
_create_engine() in __init__.py
    |
    1. engine.start_ws()           -- WS starts buffering immediately
    2. (skip checkpoint injection)
    3. KILL MUTATIONS + OPTIMIZE + DELETE IN PARTITION today  -- clean slate
    4. _run_midnight_preflight()   -- get crossover TIDs, seed processors
    5. engine.fill_from_rest()     -- Rust: REST -> processor -> ring buffer
    6. _sync_flush_rest_bars()     -- Python: drain ring buffer -> CH
    7. engine.start()              -- WS processing begins, reuses processors
```

### Rust-Side Data Flow (fill_from_rest)

```
LiveBarEngine::fill_from_rest()
    |
    For each symbol (parallel, semaphore=4):
    |   1. Create fresh processors per threshold
    |   2. Determine from_tid: max(checkpoint_tid, ch_seed_tid)
    |   3. Probe latest_tid for bounded fill
    |   4. Chunked REST pagination (50K trades/chunk)
    |   5. For each trade: maybe_reset_at_midnight() -> process_single_trade()
    |   6. Push CompletedBar to ConcurrentRingBuffer
    |   7. Return (symbol, processors, bar_count, trade_count)
    |
    Collect all processors into pre_created_processors HashMap
    Return total_bars count
```

### Junction Fill (symbol_task.rs)

```
symbol_task() -- when pre_created processors exist:
    |
    JUNC-01: Read first WS trade from pre_buffered_rx
    JUNC-02: If gap between processor.last_agg_trade_id and first_ws.tid:
             -> puck_fill() to bridge the gap (sequential REST)
    JUNC-03: Re-seed gap detector past processor's high-water mark
    JUNC-04: Monotonicity guard -- drop WS trades already processed
    JUNC-05: Drain remaining WS buffer with high_water_tid tracking
```

### Python-Side Sync Flush (\_sync_flush_rest_bars)

```
_sync_flush_rest_bars()
    |
    1. engine.drain_bars()     -- synchronous, gets all REST bars from ring buffer
    2. Group by (symbol, threshold)
    3. Cross-midnight filter (L1)
    4. Valid-pairs filter
    5. store_bars_batch() per pair -> ClickHouse
    6. Populate _junction_state telemetry
```

### Anti-Patterns to Avoid

- **Materializing trades in Python**: Trades MUST stay in Rust between REST fetch and bar computation. The current pipeline correctly does this -- `fill_from_rest()` returns only a bar count, not trade data.
- **Skipping sync_flush before engine.start()**: Bars in the ring buffer from REST fill MUST be written to CH before WS processing begins, otherwise `_heal_day_boundary_gaps()` sees false gaps.
- **Ignoring the junction gap**: Between REST fill completion and first WS trade there is always a time gap (minutes). The junction fill in `symbol_task.rs` handles this.

## Don't Hand-Roll

| Problem                      | Don't Build                  | Use Instead                                          | Why                                                                             |
| ---------------------------- | ---------------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------- |
| REST-to-WS junction bridging | Custom Python gap-fill logic | Existing `puck_fill()` in `symbol_task.rs` (JUNC-02) | Already handles forming bar preservation, monotonicity, gap detector re-seeding |
| Ring buffer drain            | Python polling loop          | `engine.drain_bars()` (synchronous Rust method)      | Avoids `block_on` failures before `engine.start()`                              |
| Stathera verification        | Ad-hoc trade ID checks       | ClickHouse Stathera query via `stathera_query.py`    | Production-proven windowed query with `tid_delta` computation                   |
| Per-partition writes         | Global `store_bars_batch`    | Phase 1's per-partition lock pattern                 | Concurrent partition access, no full-table scan                                 |

## Common Pitfalls

### Pitfall 1: Assuming fill_from_rest is Not Yet Implemented

**What goes wrong:** Planning to build fill_from_rest from scratch when it already exists and works.
**Why it happens:** The requirements describe desired behavior, not delta from current state.
**How to avoid:** The Rust `fill_from_rest()` in `live_engine/mod.rs:227-594` already fetches REST trades, processes them through processors, and pushes bars to ring buffer. The Python `_sync_flush_rest_bars()` in `midnight.py:641-797` already drains and writes to CH. The phase work is VERIFICATION and HARDENING, not greenfield implementation.
**Warning signs:** Creating new files or functions that duplicate existing logic.

### Pitfall 2: Ring Buffer Overflow During REST Fill

**What goes wrong:** 18 symbols x 4 thresholds x 20h of bars can exceed ring buffer capacity.
**Why it happens:** Ring buffer has a fixed capacity (10K default). Heavy fill periods produce many bars.
**How to avoid:** Monitor `dropped_bars` metric during fill. Ring buffer uses overwrite-oldest semantics. The `drain_bars()` call in `_sync_flush_rest_bars()` happens AFTER fill completes, so all bars must fit.
**Warning signs:** `fill_from_rest: ring buffer full, bar dropped` warnings in logs.

### Pitfall 3: Junction Gap During Parallel Fill

**What goes wrong:** While filling symbol N via REST, WS buffers grow for all symbols. By the time `engine.start()` is called, the gap between REST last trade and WS first trade can be large (5+ minutes of trades).
**Why it happens:** REST fill takes ~5 min total across 18 symbols (semaphore=4 parallel).
**How to avoid:** The junction fill (JUNC-02) in `symbol_task.rs` already handles this by calling `puck_fill()` to bridge the gap. Verify this works correctly under production load.
**Warning signs:** Stathera gaps in the first few bars after sidecar startup.

### Pitfall 4: Forming Bar Abandonment at Junction

**What goes wrong:** REST fill builds a forming bar with 100K+ trades. At junction, gap detector fires puck_fill which disrupts the forming bar.
**Why it happens:** Documented root cause in `rest-ws-junction-fix.md`. Was the original motivation for the junction fill logic.
**How to avoid:** The JUNC-01 through JUNC-04 logic in `symbol_task.rs` already addresses this. The gap detector is re-seeded past the processor's high-water mark (JUNC-03), and WS trades below high_water_tid are dropped (JUNC-04).
**Warning signs:** Multi-hour gaps in completed bars after sidecar restart.

### Pitfall 5: committed_floors Not Initialized After fill_from_rest

**What goes wrong:** Bars emitted by fill_from_rest get written to CH by sync_flush, but `committed_floors` in the symbol task doesn't reflect them, causing duplicate emission during WS processing.
**How to avoid:** `committed_floors` is initialized from `processor.last_completed_bar_tid()` (v1.4). Since processors are reused from fill_from_rest, their state already includes REST-produced bars. Verify this initialization happens correctly in `symbol_task.rs`.
**Warning signs:** Overlapping bars in CH after sidecar startup.

## Code Examples

### Key Integration Points

#### 1. Rust fill_from_rest entry (live_engine/mod.rs:227)

```rust
// Source: crates/opendeviationbar-streaming/src/live_engine/mod.rs:227
pub async fn fill_from_rest(&mut self) -> Result<u64, ProcessingError> {
    // Parallel fill: semaphore(4), per-symbol REST pagination
    // Returns total_bars count, stores processors in pre_created_processors
}
```

#### 2. Python sync flush (midnight.py:641)

```python
# Source: python/opendeviationbar/sidecar/midnight.py:641
def _sync_flush_rest_bars(engine, valid_pairs) -> int:
    # drain_bars() -> group by pair -> cross-midnight filter -> store_bars_batch()
    all_bars = engine.drain_bars()  # Synchronous Rust call
    # ... filtering, batching, CH write
    return written
```

#### 3. Junction fill (symbol_task.rs)

```rust
// Source: crates/opendeviationbar-streaming/src/live_engine/symbol_task.rs
// JUNC-02: Bridge REST-last to WS-first via puck_fill
if junction_gap_tid_delta > 0 {
    puck_fill(&symbol, current_last_tid + 1, first_ws_trade.agg_trade_id, ...);
}
// JUNC-03: Re-seed gap detector past processor's high-water mark
gap_detector.seed(current_last_tid.max(first_ws_trade.agg_trade_id));
// JUNC-04: Monotonicity guard
if first_ws_trade.agg_trade_id > current_last_tid { ... }
```

#### 4. Python call site (**init**.py:365)

```python
# Source: python/opendeviationbar/sidecar/__init__.py:365
rest_bars = engine.fill_from_rest()  # Returns bar count (u64), not trades
# ...
if rest_bars > 0:
    flushed = _sync_flush_rest_bars(engine, valid_pairs)
```

## State of the Art

| Old Approach                    | Current Approach                                | When Changed      | Impact                         |
| ------------------------------- | ----------------------------------------------- | ----------------- | ------------------------------ |
| Sequential fill_from_rest       | Parallel fill with Semaphore(4)                 | v13.x (#297)      | 4x speedup for 18 symbols      |
| Checkpoint injection at startup | Skip checkpoint, fresh processors from midnight | v13.45.0 (#295)   | Clean-slate restartability     |
| Full-table ALTER TABLE DELETE   | Per-partition DELETE IN PARTITION               | Phase 1 (v3.0)    | O(1) partition vs O(all parts) |
| No junction fill                | JUNC-01 through JUNC-04 in symbol_task          | v1.0 junction fix | Zero-gap REST-to-WS transition |
| Puck fires at junction          | Gap detector re-seeded past high-water mark     | v1.0 junction fix | No false Puck triggers         |

## Validation Architecture

### Test Framework

| Property           | Value                                                             |
| ------------------ | ----------------------------------------------------------------- |
| Framework          | pytest (Python) + cargo nextest (Rust)                            |
| Config file        | `pyproject.toml` [tool.pytest.ini_options] + `.cargo/config.toml` |
| Quick run command  | `mise run test-py -- -x -k "sidecar or midnight"`                 |
| Full suite command | `mise run check-full`                                             |

### Phase Requirements -> Test Map

| Req ID    | Behavior                                                | Test Type   | Automated Command                                          | File Exists?                             |
| --------- | ------------------------------------------------------- | ----------- | ---------------------------------------------------------- | ---------------------------------------- |
| STREAM-01 | fill_from_rest returns bars, not trades, for sync_flush | integration | `mise run test-py -- -x tests/test_sidecar.py`             | Exists (test_sidecar.py)                 |
| STREAM-01 | No Python DataFrame of trades in fill path              | code audit  | Manual: grep for trade DataFrame creation in sidecar path  | Manual-only -- code audit                |
| STREAM-02 | Stathera continuity at REST-WS junction                 | unit (Rust) | `cargo nextest run -p opendeviationbar-streaming junction` | Exists (test in live_engine/mod.rs:1331) |
| STREAM-02 | Junction fill bridges gap correctly                     | integration | Production verification via heartbeat Stathera check       | Manual-only -- requires live sidecar     |

### Sampling Rate

- **Per task commit:** `mise run test-py -- -x -k "sidecar or midnight"`
- **Per wave merge:** `mise run check-full`
- **Phase gate:** Full suite green before `/gsd:verify-work`

### Wave 0 Gaps

- None -- existing test infrastructure covers phase requirements. The Rust junction test exists at `live_engine/mod.rs:1331` (`test_junction_fill_zero_stathera_gaps`). Python sidecar tests exist at `tests/test_sidecar.py` and `tests/test_midnight_preflight.py`.

## Open Questions

1. **Ring buffer capacity vs REST fill volume**
   - What we know: Ring buffer capacity is 10K (default). 18 symbols x 4 thresholds x ~20h = potentially thousands of bars.
   - What's unclear: Whether capacity is ever exceeded during fill_from_rest for a full 18-symbol fill.
   - Recommendation: Add a diagnostic log or assertion that checks `dropped_bars` metric after fill_from_rest. If >0, increase ring buffer capacity or add intermediate drain.

2. **Stathera verification timing**
   - What we know: `_heal_day_boundary_gaps()` runs after sync_flush to check for Stathera gaps. Junction fill happens later in `engine.start()` -> `symbol_task()`.
   - What's unclear: Whether the heal query can fire before junction fill completes (race condition).
   - Recommendation: Verify that `_heal_day_boundary_gaps()` is called BEFORE `engine.start()` (which it is -- at line 416 in `__init__.py`). But the heal query runs on CH data, not on ring buffer data. Junction fill happens during `engine.start()` so heal sees the pre-junction state. This is acceptable because junction fill bars flow through the normal WS write path.

3. **Success criteria #3: "No intermediate trade DataFrame"**
   - What we know: The current code path goes Python -> `engine.fill_from_rest()` (Rust, returns u64 count) -> `engine.drain_bars()` (Rust, returns completed bars as dicts). No trade DataFrame is created in Python.
   - What's unclear: Whether there is any legacy code path that creates a Python trade DataFrame during fill.
   - Recommendation: Grep audit for any `pd.DataFrame` or `pl.DataFrame` creation in the sidecar fill path. If found, remove it.

## Sources

### Primary (HIGH confidence)

- `crates/opendeviationbar-streaming/src/live_engine/mod.rs:227-594` -- fill_from_rest Rust implementation
- `crates/opendeviationbar-streaming/src/live_engine/symbol_task.rs` -- junction fill (JUNC-01 through JUNC-04)
- `python/opendeviationbar/sidecar/midnight.py:641-797` -- \_sync_flush_rest_bars Python implementation
- `python/opendeviationbar/sidecar/__init__.py:171-416` -- \_create_engine startup sequence
- `crates/opendeviationbar-streaming/src/stream_manager.rs:203-218` -- StreamManager.fill_from_rest delegation

### Secondary (MEDIUM confidence)

- `rest-ws-junction-fix.md` (auto-memory) -- root cause analysis for forming-bar abandonment (verified against current code: fix is implemented)
- `python/opendeviationbar/sidecar/CLAUDE.md` -- module documentation, startup sequence table

## Metadata

**Confidence breakdown:**

- Standard stack: HIGH - no new libraries needed, all existing codebase
- Architecture: HIGH - direct code reading of all key files
- Pitfalls: HIGH - documented root cause analysis + existing fix code verified

**Research date:** 2026-03-24
**Valid until:** 2026-04-24 (stable -- codebase-specific, no external dependency churn)
