# GPU Pipeline Starvation Investigation

**Date:** 2026-03-02
**Environment:** v3.0.59, Tesla T4, g4dn.xlarge, EFS-mounted files
**Symptom:** GPU at 0% for entire 13-minute observation; 4–6 minute gaps between 2–3 second GPU bursts

---

## 1. Confirmed Architecture (Code-Verified)

The pipeline runs entirely within a **single Python process** using asyncio coroutines.

```
Main Process (PID main, 46 threads)
  ├── asyncio event loop
  │     ├── chunk_producer(0)  ← asyncio.Task (coroutine, not thread)
  │     ├── chunk_producer(1)  ← asyncio.Task
  │     ├── chunk_producer(2)  ← asyncio.Task
  │     ├── chunk_producer(3)  ← asyncio.Task
  │     └── embed_consumer()   ← asyncio.Task
  │
  └── ProcessPoolExecutor (persistent pool, Linux fork)
        ├── Worker process 0  (chunk_processor._persistent_pool)
        ├── Worker process 1
        ├── ... up to max_workers (auto-detected, e.g. 4 on g4dn.xlarge)
        └── Worker process N

"Worker subprocess (PID 6553, blocked in read())" = ONE of these ProcessPoolExecutor workers
```

**The "worker subprocess in read()" is a ProcessPoolExecutor fork-worker waiting for the
next `_parse_file_standalone` task via the multiprocessing IPC pipe from the main process.**

There is no separate embedding subprocess. Embedding runs in the main process event loop
via `asyncio.to_thread(embedding_function, contents)` using a ThreadPoolExecutor.

---

## 2. Code Flow: num_producers → GPU

### Step 1: File slicing (indexer.py:1446–1451)
```python
producer_slices = [
    files_to_process[i::effective_num_producers]
    for i in range(effective_num_producers)
]
```
With `num_producers=4` and 10,000 files: each producer gets 2,500 files (stride-based).

### Step 2: Task creation (indexer.py:1460–1465)
```python
consumer_task = asyncio.create_task(embed_consumer())
producer_tasks = [
    asyncio.create_task(chunk_producer(i, producer_slices[i]))
    for i in range(effective_num_producers)
]
```
Four `chunk_producer` coroutines and one `embed_consumer` coroutine — all on the same
asyncio event loop thread.

### Step 3: Producer inner loop (indexer.py:961–1169)
Each producer iterates its file slice in `file_batch_size=512` batches and calls:
```python
parse_results = await self.chunk_processor.parse_files_multiprocess(batch_paths)
```

### Step 4: parse_files_multiprocess (chunk_processor.py:305–375)
```python
futures = [
    loop.run_in_executor(self._persistent_pool, _parse_file_standalone, args)
    for args in parse_args
]
raw_results = await asyncio.gather(*futures, return_exceptions=True)
```
All 512 file-parse futures for a single batch are submitted to the `ProcessPoolExecutor`
and gathered together. This `await asyncio.gather(...)` suspends the calling producer
coroutine until **all 512 files in the batch are parsed**.

### Step 5: Post-parse work (indexer.py:1020–1106)
After parsing the batch, the producer does **inside the event loop thread** (not
offloaded):
- `idx % 10 == 0` memory checks (psutil syscall)
- `await asyncio.to_thread(chunk_processor.build_chunk_hierarchy, chunks)` per file
- Dict-building loop (JSON serialisation, list comprehensions)
- `await self.chunks_backend.add_chunks(chunk_dicts, file_hash)` — LanceDB write to EFS

### Step 6: Queue put (indexer.py:1128–1153)
Only after all 512 files are parsed AND stored to chunks.lance does the producer
put a batch on the queue. The queue maxsize = `effective_num_producers * 4 = 16`.

### Step 7: embed_consumer (indexer.py:1171–1444)
```python
while sentinels_received < effective_num_producers:
    batch_data = await chunk_queue.get()
    ...
    vectors = await asyncio.to_thread(self.database._embedding_function, contents)
```
Consumer blocks on `chunk_queue.get()` until a producer puts a batch. Embedding
runs in a thread pool via `asyncio.to_thread`.

---

## 3. Root Cause Analysis

### Root Cause 1 (PRIMARY): parse_files_multiprocess is batch-synchronous per producer

`parse_files_multiprocess` gathers ALL 512 files at once:
```python
raw_results = await asyncio.gather(*futures, return_exceptions=True)  # chunk_processor.py:356
```

This means each producer call blocks for the time it takes to parse its slowest file
in the batch. On EFS, each `_parse_file_standalone` call does:
1. `open()` + `read()` the file from EFS (NFS over network — high latency, ~1–10ms per file)
2. tree-sitter parse (CPU-bound, fast)

With 512 files × ~5ms EFS read latency = **~2.5 seconds minimum per batch** before the
producer can put anything on the queue.

### Root Cause 2 (COMPOUNDING): The 4 producers are NOT independent pipelines

All 4 producer coroutines run on the same single asyncio event loop thread. They
interleave at `await` points, but `parse_files_multiprocess` submits work to the shared
`_persistent_pool`. With `file_batch_size=512` and 4 producers, the pool receives
4×512 = 2048 futures simultaneously, all competing for the same pool workers.

On a g4dn.xlarge (4 vCPUs), the ProcessPoolExecutor auto-detects `max_workers = 4`
(from `_detect_optimal_workers()` → `min(cpu_count, 16) = 4`). So 4 workers process
2048 EFS-read futures serially, taking the same time as 1 producer with 2048 files.
**num_producers=4 is a no-op for parse throughput on this instance.**

### Root Cause 3 (COMPOUNDING): Post-parse work blocks the event loop before queue put

After `parse_files_multiprocess` returns, the producer does (still blocking the single
event loop thread):
```python
# indexer.py:1053
chunks_with_hierarchy = await asyncio.to_thread(
    self.chunk_processor.build_chunk_hierarchy, chunks
)
```
This is per-file inside the loop. Each `await asyncio.to_thread` is a context switch to
a ThreadPoolExecutor worker, which is correct — but the loop still runs sequentially
(no `asyncio.gather` over files here). For 512 files this is 512 sequential thread
dispatches.

Then:
```python
# indexer.py:1095
count = await self.chunks_backend.add_chunks(chunk_dicts, file_hash)
```
This is a per-file LanceDB write to EFS. On EFS with fsync semantics this can be
**50–200ms per file** = 25–100 seconds for 512 files. This is the dominant latency
that creates the 4–6 minute gaps.

### Root Cause 4 (ARCHITECTURAL): The queue only receives data AFTER the full batch completes

The queue put happens at line 1128 — AFTER the inner for-loop over all 512 parse
results. So the consumer sees no data for the entire duration of:
- EFS read for 512 files
- Hierarchy build for each file
- LanceDB write for each file (EFS NFS amplification)

Only then does a batch arrive in the queue and GPU starts for 2–3 seconds.

### Summary of timing

```
Phase                     Time estimate (EFS, 512 files)
─────────────────────────────────────────────────────────
parse_files_multiprocess  ~5–30s  (EFS read + tree-sitter, 4 workers)
hierarchy build loop      ~2–5s   (sequential to_thread dispatches)
chunks_backend writes     ~25–100s (per-file LanceDB writes to EFS NFS)
queue put                 <0.1s
GPU embedding             ~2–3s   (only now, T4 burns through the batch)
─────────────────────────────────────────────────────────
Total cycle               ~35–140s per 512-file batch
GPU active fraction       2s / 140s ≈ 1.4%
```

This matches the observed 4–6 minute gaps and 0% GPU utilization.

---

## 4. What the "Worker Subprocess (PID 6553) in read()" Is

**It is one of the `ProcessPoolExecutor` fork-workers** created by
`chunk_processor.py:340`:
```python
self._persistent_pool = ProcessPoolExecutor(
    max_workers=self.max_workers, mp_context=mp_context
)
```

On Linux, these are forked processes. Between parse batches, they sit idle with their
IPC pipe to the main process open and block in `read()` on that pipe waiting for the
next `_parse_file_standalone` task to arrive. This is normal Python `ProcessPoolExecutor`
idle behavior — not a hang.

The observation of "worker subprocess blocked in read() the ENTIRE time" confirms the
pipeline is spending nearly all time NOT in parsing (the workers are idle), which points
to the LanceDB EFS write bottleneck in Root Cause 3 as the dominant factor.

---

## 5. Why num_producers=4 Doesn't Help

Three reasons, in priority order:

**a) All 4 producers share the same ProcessPoolExecutor with only 4 workers.**
The pool has at most 4 workers (4 vCPUs on g4dn.xlarge). Whether 1 or 4 producers
submit to it, throughput is capped at 4 concurrent parses. `num_producers=4` just
divides the file list into 4 equal slices and submits all 4×512=2048 futures at once
instead of 512 at once — no actual speedup.

**b) The asyncio event loop is single-threaded.**
All 4 producers are coroutines on one thread. Their "parallelism" is cooperative.
During the post-parse work loop (hierarchy build + LanceDB write), producers take turns
via `await`, but the total work is the same.

**c) The LanceDB per-file write is the bottleneck, not parsing.**
Even if parsing were instant, the `await self.chunks_backend.add_chunks()` per file
inside the producer loop (line 1095) serializes all LanceDB writes through a single
event loop thread, with each write hitting EFS NFS.

---

## 6. Concrete Fix Recommendations

### Fix A (Highest Impact): Batch the LanceDB chunks writes

**File:** `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/core/indexer.py`
**Lines:** 1093–1102 (inside `chunk_producer`, inside the file-result loop)

**Problem:** Per-file `await self.chunks_backend.add_chunks(chunk_dicts, file_hash)`
writes to EFS on every iteration.

**Fix:** Accumulate all chunk dicts for the entire batch and do a single
`add_chunks` call after the for-loop, matching what the embed consumer already
does with its `write_buffer`.

```python
# BEFORE (current — per-file write inside loop):
if chunk_dicts:
    count = await self.chunks_backend.add_chunks(chunk_dicts, file_hash)
    batch_chunks.extend(chunk_dicts)
    chunks_created += count

# AFTER (batched — collect then write once per batch):
if chunk_dicts:
    batch_chunks.extend(chunk_dicts)
    batch_files_processed += 1
    chunks_created += len(chunk_dicts)

# OUTSIDE the for-loop, before queue put:
if batch_chunks_to_write:  # all chunk_dicts for this 512-file batch
    await self.chunks_backend.add_chunks_batch(batch_chunks_to_write, file_hashes_map)
```

Expected impact: reduces EFS write operations from 512 to 1 per batch.

### Fix B (High Impact): Batch the hierarchy builds with asyncio.gather

**File:** `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/core/indexer.py`
**Lines:** 1052–1055

**Problem:** `build_chunk_hierarchy` is dispatched to a thread via `asyncio.to_thread`
sequentially per file inside the for-loop.

**Fix:** Collect all `asyncio.to_thread` coroutines and gather them:
```python
hierarchy_tasks = [
    asyncio.to_thread(self.chunk_processor.build_chunk_hierarchy, chunks)
    for _, chunks, _ in valid_parse_results
]
all_hierarchies = await asyncio.gather(*hierarchy_tasks)
```

Expected impact: overlaps CPU-bound hierarchy computation across threads.

### Fix C (Medium Impact): Reduce file_batch_size for EFS

**Config:** `MCP_VECTOR_SEARCH_FILE_BATCH_SIZE`
Default is 512. On EFS, smaller batches (e.g. 64–128) reduce the time between queue
puts, feeding the GPU more frequently even though total throughput is unchanged.

```bash
export MCP_VECTOR_SEARCH_FILE_BATCH_SIZE=64
```

### Fix D (Medium Impact): Use TWO_PASS mode as an immediate workaround

**Config:** `MCP_VECTOR_SEARCH_TWO_PASS=1`

This already exists in the codebase (indexer.py:340–345, 3127–3129). It separates
chunking and embedding into sequential phases. While slower overall, it allows the GPU
to run in a tight loop during the embedding phase without EFS contention from concurrent
parsing.

```bash
export MCP_VECTOR_SEARCH_TWO_PASS=1
```

### Fix E (Lower Impact): Increase ProcessPoolExecutor workers

**Config:** `MCP_VECTOR_SEARCH_MAX_WORKERS`

On g4dn.xlarge (4 vCPUs), auto-detection gives 4 workers. EFS reads are I/O-bound,
so more workers than CPUs can help:

```bash
export MCP_VECTOR_SEARCH_MAX_WORKERS=16
```

This allows more concurrent EFS reads per batch despite having only 4 vCPUs, since
workers are blocked on `read()` (I/O wait) most of the time.

### Fix F (Architectural): Pre-fetch + parse independently from queue put

The ideal architecture decouples file reading from the queue put entirely. A producer
should continuously parse files into an internal buffer and only gate on the queue, not
on EFS latency. This requires more significant restructuring of `chunk_producer` to use
an internal asyncio.Queue between parse and post-parse stages.

---

## 7. Recommended Immediate Action

**Order of operations for fastest GPU utilization improvement:**

1. Set `MCP_VECTOR_SEARCH_FILE_BATCH_SIZE=64` immediately (no code change)
2. Set `MCP_VECTOR_SEARCH_MAX_WORKERS=16` immediately (no code change)
3. Apply Fix A (batch LanceDB writes) — highest code-change impact
4. Apply Fix B (parallel hierarchy builds) — second highest
5. Consider Fix D (TWO_PASS) as a stopgap until A+B are deployed

**Predicted result after Fix A + B:**
- Queue put frequency: every ~5–15s instead of 35–140s
- GPU active fraction: ~15–40% (from ~1%)
- EFS write operations: ~98% reduction per batch
