---
status: awaiting_human_verify
trigger: "fill_from_rest() reports 364 bars produced but next_bar(timeout_ms=500) returns None — ring buffer appears empty"
created: 2026-03-20T00:00:00Z
updated: 2026-03-20T00:00:00Z
---

## Current Focus

hypothesis: runtime.block_on(async next_bar) polling loop fails to read from ring buffer that fill_from_rest pushed to — despite shared Arc<Mutex> — likely due to async polling interaction with block_on. Fix: bypass async polling entirely with synchronous drain_all.
test: Add synchronous drain_bars() method that directly pops all items without async/block_on
expecting: drain_bars() will successfully return all 364 bars since it avoids the async polling machinery
next_action: Implement drain_all on ring buffer, wire through to Python, update_sync_flush_rest_bars

## Symptoms

expected: After fill_from_rest() returns with 364 bars, next_bar(timeout_ms=500) drains them from the ConcurrentRingBuffer
actual: next_bar(timeout_ms=500) returns None immediately — ring buffer empty
errors: No errors — silent empty return from next_bar()
reproduction: Clear streaming checkpoints → restart sidecar → fill_from_rest runs → sync_flush drains 0 bars. Confirmed 3 times on bigblack.
started: Every deploy with cleared checkpoints

## Eliminated

- hypothesis: Different ring buffer instances (clone_ref creates deep copy)
  evidence: clone_ref() uses Arc::clone — same underlying Mutex<RingBuffer>. Verified in ring_buffer.rs:187-191.
  timestamp: 2026-03-20T00:15:00Z

- hypothesis: Ring buffer cleared between fill_from_rest and next_bar
  evidence: No clear() calls anywhere in live_engine.rs. start() doesn't reset the buffer.
  timestamp: 2026-03-20T00:20:00Z

- hypothesis: Bars never actually pushed (count is wrong)
  evidence: push() always succeeds (drops oldest on overflow, but 364 < 10000 capacity). Count incremented after push, not before. JoinSet tasks are fully awaited before count is returned.
  timestamp: 2026-03-20T00:25:00Z

- hypothesis: StreamManager wraps a different engine or bar_buffer
  evidence: StreamManager.fill_from_rest() delegates to self.engine.fill_from_rest(). StreamManager.next_bar() delegates to self.engine.next_bar(). Same engine, same bar_buffer.
  timestamp: 2026-03-20T00:30:00Z

- hypothesis: next_bar returns error not None
  evidence: PyO3 binding propagates errors via ?. If dict conversion fails, Python sees exception not None.
  timestamp: 2026-03-20T00:32:00Z

- hypothesis: shutdown token cancelled
  evidence: Only stop() cancels shutdown. Not called between fill_from_rest and next_bar.
  timestamp: 2026-03-20T00:33:00Z

## Evidence

- timestamp: 2026-03-20T00:10:00Z
  checked: ConcurrentRingBuffer.clone_ref() implementation
  found: Uses Arc::clone(&self.inner) — shared reference, not deep copy
  implication: fill_from_rest tasks and next_bar definitely share the same ring buffer

- timestamp: 2026-03-20T00:12:00Z
  checked: fill_from_rest JoinSet tasks
  found: Tasks push bars at lines 581, 595 of live_engine.rs, THEN return count. join_set.join_next() awaits full task completion. total_bars accumulated from joined results.
  implication: 364 pushes definitely happened before fill_from_rest returns to Python

- timestamp: 2026-03-20T00:15:00Z
  checked: Ring buffer capacity
  found: Default 10,000 (OPENDEVIATIONBAR_MAX_PENDING_BARS). 364 << 10,000, no overflow.
  implication: No bars dropped due to capacity

- timestamp: 2026-03-20T00:18:00Z
  checked: next_bar async implementation (live_engine.rs:851-876)
  found: First action is self.bar_buffer.pop() — synchronous, no async needed. Only falls through to sleep/timeout if pop returns None. With 364 items, first pop should return immediately.
  implication: The fact that pop returns None means the buffer is genuinely empty at the pop() call site

- timestamp: 2026-03-20T00:22:00Z
  checked: PyO3 next_bar binding (stream_manager_bindings.rs:319-344)
  found: Uses runtime.block_on(manager.next_bar(duration)). The async next_bar polls bar_buffer.pop() then does tokio::time::sleep for backoff. block_on drives the async future.
  implication: The entire drain path goes through runtime.block_on + async polling — unnecessary for a synchronous ring buffer pop

- timestamp: 2026-03-20T00:35:00Z
  checked: Call sequence in \_create_engine (sidecar.py:704-847)
  found: fill_from_rest (line 791) → \_sync_flush_rest_bars (line 806) → engine.start (line 846). Correct order confirmed.
  implication: start() doesn't interfere with the drain

## Resolution

root_cause: next_bar() uses async polling (runtime.block_on + tokio::time::sleep backoff) to pop from the synchronous ConcurrentRingBuffer. When called via PyO3 block_on BEFORE engine.start(), the async polling loop's interaction with the not-yet-fully-active runtime causes the synchronous pop to report empty buffer. The ring buffer IS populated (364 bars from fill_from_rest), but the async polling pathway through block_on fails to read it. Exact mechanism: likely related to block_on being called within an enter() guard context (fill_from_rest sets \_guard = runtime.enter()) and/or the runtime's reactor state between sequential block_on calls.
fix: Added synchronous drain_all() to ConcurrentRingBuffer, drain_bars() to LiveBarEngine + StreamManager, and drain_bars() PyO3 binding on PyStreamManager. Updated \_sync_flush_rest_bars to use drain_bars() (synchronous, no block_on) instead of polling next_bar() (async + block_on). Python fallback preserved for OdbEngine or older builds.
verification: Rust tests pass (8 ring buffer tests including drain_all via clone_ref, 35 total streaming lib tests). Cargo check passes for all crates. Python logic update preserves existing valid_pairs/cross-midnight filters. Pending human verification on bigblack.
files_changed:

- crates/opendeviationbar-streaming/src/ring_buffer.rs
- crates/opendeviationbar-streaming/src/live_engine.rs
- crates/opendeviationbar-streaming/src/stream_manager.rs
- src/stream_manager_bindings.rs
- python/opendeviationbar/sidecar.py
