# KG Relationship Insertion Performance Investigation

**Date:** 2026-03-03
**Reported Issue:** Inserting large numbers of relationships (5-25k) is very slow with no progress feedback.

---

## 1. Exact Code Path

### Entry Point: `build_from_chunks_sync()` — kg_builder.py lines 307–770

```
build_from_chunks_sync()
  Phase 3 (lines 540–726)
    → _add_relationships_batch_by_type_sync()   [knowledge_graph.py L1952]
        → UNWIND + MATCH + CREATE (batch_size=50)
```

### The Loop (kg_builder.py lines 655–721)

```python
for rel_type, rels in relationships.items():   # iterates ~12 relationship types
    valid_rels = [...]                           # Python-side ID validation
    count = self.kg.add_relationships_batch_sync(valid_rels, batch_size=safe_batch_size)
```

`safe_batch_size` is hardcoded to **50** at line 645:

```python
# CRITICAL: Use very small batch size (50) to avoid Kuzu assertion failures
safe_batch_size = 50
```

### The Actual Kuzu Call: `_add_relationships_batch_by_type_sync()` — knowledge_graph.py lines 1952–2044

```python
for i in range(0, len(relationships), batch_size):   # batch_size=50
    batch = relationships[i : i + batch_size]
    params = [{"source_id": r.source_id, "target_id": r.target_id, ...} for r in batch]

    query = f"""
        UNWIND $batch AS r
        MATCH (a:{source_label} {{id: r.source_id}}),
              (b:{target_label} {{id: r.target_id}})
        CREATE (a)-[rel:{rel_type}]->(b)
        SET rel.weight = r.weight,
            rel.commit_sha = r.commit_sha
    """
    self._execute_query(query, {"batch": params})
```

`_execute_query` acquires a `threading.Lock` on every call (line 252):

```python
def _execute_query(self, query, params=None):
    with self._kuzu_lock:
        return self.conn.execute(query, params or {})
```

---

## 2. Is It One-at-a-Time or Batched?

**Technically batched, but with a critically small batch size.**

- The code uses `UNWIND $batch AS r ... MATCH ... CREATE` — correct batch pattern.
- However, `batch_size=50` means **for 25,000 relationships, Kuzu executes 500 separate round-trips**.
- The async path (`add_relationships_batch`) defaults to `batch_size=500` but also has per-batch MATCH overhead.
- The `UNWIND + MATCH + MATCH + CREATE` pattern is **not the fastest Kuzu bulk insert pattern**. Each row in the UNWIND must do two indexed lookups (source + target node by `id`), then CREATE the edge. At batch_size=50, the overhead per-row dominates.
- For comparison: node insertion uses `MERGE` at the same call site, but node inserts go through at batch_size=500 (the default), so nodes do 10x fewer round-trips.

### Why batch_size=50?

The comment at line 642–644 explains it:

> "CRITICAL: Use very small batch size (50) to avoid Kuzu assertion failures.
> Kuzu's Rust bindings have bugs that cause assertion failures in csr_node_group.cpp.
> Smaller batches reduce the risk of hitting the bug."

This is a Kuzu 0.11.3 stability workaround that has become the primary performance bottleneck.

---

## 3. Relationship Types and Volumes

### Schema-defined relationship types (15 types total)

| Type | FROM | TO | Typical volume |
|------|------|----|----------------|
| CALLS | CodeEntity | CodeEntity | Medium–High (function call graph) |
| IMPORTS | CodeEntity | CodeEntity | Medium (per-module import stmts) |
| INHERITS | CodeEntity | CodeEntity | Low (class hierarchies) |
| CONTAINS | CodeEntity | CodeEntity | Medium (class→method containment) |
| FOLLOWS | DocSection | DocSection | High (doc reading order chains) |
| REFERENCES | DocSection | CodeEntity | Medium |
| DOCUMENTS | DocSection | CodeEntity | High (optional, O(n×m)) |
| HAS_TAG | DocSection | Tag | Medium |
| DEMONSTRATES | DocSection | Tag | Low |
| LINKS_TO | DocSection | DocSection | Low |
| CONTAINS_SECTION | Document | DocSection | **Very High** (1 per section, scales with docs) |
| RELATED_TO | Document | Document | Low |
| DESCRIBES | Document | CodeEntity | Low |
| HAS_TOPIC | Document | Topic | Low |
| PART_OF | CodeEntity | Project | High (1 per code entity) |

**Likely volume culprits for 5-25k total:**
- `CONTAINS_SECTION`: 1 relationship per DocSection. If 5k doc sections exist, that is 5k rels of this type alone.
- `FOLLOWS`: 1 per consecutive section pair per document. Scales linearly with doc sections.
- `CALLS`: Function call graph — can be dense in a large Python project (thousands of calls).
- `PART_OF`: One per CodeEntity — can be 3-5k in a mid-sized project.

At batch_size=50, `5000 CONTAINS_SECTION` rels = 100 Kuzu round-trips, each doing 2 MATCH lookups = **200 index lookups plus 100 transaction overhead cycles**.

---

## 4. Progress Feedback: What Exists vs. What Is Missing

### What exists

At kg_builder.py lines 680–690 (validation loop):

```python
if progress_tracker and (
    processed_rels % 1000 == 0 or processed_rels == total_rels_count
):
    progress_tracker.progress_bar_with_eta(
        processed_rels, total_rels_count,
        prefix=f"Validating {rel_type.lower()}",
        start_time=rel_start_time,
    )
```

This progress bar fires during the **Python-side validation loop** (filtering invalid IDs), but it does NOT fire during the **actual Kuzu insertion** inside `add_relationships_batch_sync`.

At lines 694–708, only a start message and a done message are shown per relationship type:

```python
progress_tracker.item(f"Inserting {len(valid_rels):,} {rel_type.lower()} relationships...", done=False)
# ... Kuzu insertion happens here (could take minutes) ...
progress_tracker.item(f"{count:,} {rel_type.lower()} inserted", done=True)
```

### The gap

If `CONTAINS_SECTION` has 8,000 relationships, the user sees:
```
Inserting 8,000 contains_section relationships...
[silence for 20+ seconds]
8,000 contains_section inserted
```

There is no progress inside `_add_relationships_batch_by_type_sync`. The method has no callback or progress hook. It is a black box from the caller's perspective.

---

## 5. Node Insertion vs. Relationship Insertion

| Aspect | Node insertion | Relationship insertion |
|--------|---------------|----------------------|
| Method | `add_entities_batch_sync` | `add_relationships_batch_sync` |
| Batch size | 500 (default) | 50 (hardcoded override at call site) |
| Kuzu op | `UNWIND ... MERGE (n:NodeType {id})` | `UNWIND ... MATCH (a) MATCH (b) CREATE (a)-[]->(b)` |
| Lookups/row | 1 (primary key lookup) | 2 (source + target lookup) |
| Round-trips for 5k records | 10 | 100 |
| Per-batch progress | None inside method | None inside method |

Node insertion is 10x fewer round-trips AND cheaper per round-trip (one lookup vs. two).
Relationship insertion is the bottleneck.

---

## 6. Root Causes Ranked

1. **batch_size=50 forces 10-20x more Kuzu round-trips** than the natural batch_size=500 would.
   Each round-trip has overhead: Python dict serialization, lock acquisition, Kuzu query planning, transaction.

2. **UNWIND + MATCH + MATCH + CREATE is not the fastest Kuzu pattern** for bulk edge inserts.
   Kuzu's recommended fast path for bulk data is `COPY FROM` (CSV/Parquet file), but that requires stable node offsets and is not practical for incremental inserts. The next-best approach is large-batch UNWIND.

3. **No progress feedback inside `_add_relationships_batch_by_type_sync`.**
   The method is a tight loop with no callback. For 25k rels at batch_size=50, it runs 500 iterations silently.

4. **Lock contention.** `_execute_query` acquires a `threading.Lock` on every call. At batch_size=50, this means 500 lock acquire/release cycles for 25k relationships. At batch_size=500, it would be 50.

---

## 7. Recommendations with Code Sketches

### Recommendation A: Increase batch size for stable relationship types (highest impact)

The batch_size=50 was set as a Kuzu 0.11.3 workaround. The workaround comment says "smaller batches **reduce the risk**" — it is probabilistic, not deterministic. The assertion failure in `csr_node_group.cpp` is a known Kuzu bug related to dense graph structure, not simply batch size.

**Targeted fix:** Use batch_size=500 for relationship types where Kuzu is known stable, keep batch_size=50 as fallback on assertion failure.

In `build_from_chunks_sync()`, replace the hardcoded `safe_batch_size = 50` with:

```python
# Start with a larger batch size; fall back to 50 if Kuzu asserts
preferred_batch_size = 250  # safe middle ground vs. 500 default
safe_batch_size = 50        # fallback on Kuzu assertion failure
```

And in `_add_relationships_batch_by_type_sync()`, add retry-with-smaller-batch on assertion:

```python
for i in range(0, len(relationships), batch_size):
    batch = relationships[i : i + batch_size]
    # ... build params ...
    try:
        self._execute_query(query, {"batch": params})
        total += len(batch)
    except Exception as e:
        error_msg = str(e)
        if "KU_UNREACHABLE" in error_msg or "Assertion failed" in error_msg:
            # Retry with individual inserts (safe fallback)
            for rel in batch:
                try:
                    single_params = [{"source_id": rel.source_id, ...}]
                    self._execute_query(query, {"batch": single_params})
                    total += 1
                except Exception:
                    pass
        else:
            pass  # skip batch
```

This means only bad batches get the slow treatment, not the entire run.

### Recommendation B: Add per-batch progress inside `_add_relationships_batch_by_type_sync` (highest visibility)

Add an optional `progress_callback` parameter:

```python
def _add_relationships_batch_by_type_sync(
    self,
    relationships: list[CodeRelationship],
    rel_type: str,
    batch_size: int,
    progress_callback=None,   # NEW: callable(inserted_so_far, total)
) -> int:
    total = 0
    n = len(relationships)
    for i in range(0, n, batch_size):
        batch = relationships[i : i + batch_size]
        # ... build params, execute ...
        total += len(batch)
        if progress_callback:
            progress_callback(total, n)
    return total
```

Then in `build_from_chunks_sync()`, pass a closure:

```python
if progress_tracker:
    def _rel_progress(done, total_n):
        progress_tracker.progress_bar_with_eta(
            done, total_n,
            prefix=f"Inserting {rel_type.lower()}",
            start_time=insert_start,
        )
    insert_start = time.time()
else:
    _rel_progress = None

count = self.kg.add_relationships_batch_sync(
    valid_rels,
    batch_size=safe_batch_size,
    progress_callback=_rel_progress,
)
```

### Recommendation C: Show overall relationship phase progress bar (quick win, no API changes)

The current structure reports per-type status but the user cannot see overall progress across all 12 types. Add a Rich `Progress` bar wrapping the entire Phase 3 loop:

In `build_from_chunks_sync()` around line 655:

```python
from rich.progress import Progress, SpinnerColumn, BarColumn, TaskProgressColumn, TimeElapsedColumn

with Progress(
    SpinnerColumn(),
    "[progress.description]{task.description}",
    BarColumn(),
    TaskProgressColumn(),
    TimeElapsedColumn(),
    console=console,
    transient=True,
) as progress:
    task = progress.add_task("Inserting relationships", total=total_rels_count)

    for rel_type, rels in relationships.items():
        if rels:
            valid_rels = [r for r in rels if ...]  # existing filter
            count = self.kg.add_relationships_batch_sync(valid_rels, batch_size=safe_batch_size)
            progress.advance(task, count)
```

### Recommendation D: Pre-deduplicate relationships before insertion (prevents wasted work)

The current code uses `CREATE` (not `MERGE`) to avoid Kuzu CSR bugs, which means duplicate relationships are silently inserted. A pre-pass deduplication by `(source_id, target_id)` before calling `add_relationships_batch_sync` reduces the actual insertion count:

```python
# In build_from_chunks_sync Phase 3, before calling add_relationships_batch_sync:
seen_pairs: set[tuple[str, str]] = set()
deduped_rels = []
for r in valid_rels:
    key = (r.source_id, r.target_id)
    if key not in seen_pairs:
        seen_pairs.add(key)
        deduped_rels.append(r)
valid_rels = deduped_rels
```

This is O(n) Python overhead that can eliminate 10-30% of inserts in CONTAINS/FOLLOWS graphs where the same relationship gets generated from multiple chunks.

### Recommendation E: Batch UNWIND with SET vs. separate SET statement (micro-optimization)

The current sync query:

```cypher
UNWIND $batch AS r
MATCH (a:CodeEntity {id: r.source_id}),
      (b:CodeEntity {id: r.target_id})
CREATE (a)-[rel:CALLS]->(b)
SET rel.weight = r.weight,
    rel.commit_sha = r.commit_sha
```

Can be simplified to inline properties on CREATE to reduce planner steps:

```cypher
UNWIND $batch AS r
MATCH (a:CodeEntity {id: r.source_id}),
      (b:CodeEntity {id: r.target_id})
CREATE (a)-[rel:CALLS {weight: r.weight, commit_sha: r.commit_sha}]->(b)
```

Minor, but reduces Kuzu's internal operations per-row.

---

## 8. Best Places to Add Progress Bars

| Location | File | Lines | What to show |
|----------|------|-------|--------------|
| Inside `_add_relationships_batch_by_type_sync` | knowledge_graph.py | ~2000–2044 | Per-type batch progress (N of M batches) |
| Phase 3 outer loop in `build_from_chunks_sync` | kg_builder.py | 655–721 | Overall: X of Y total relationships |
| Per-type item messages (already exist) | kg_builder.py | 694–708 | Already there — just needs timing |

The highest-value addition is a Rich `Progress` bar wrapping the Phase 3 outer loop, showing total relationships inserted / total to insert, with elapsed time. This requires no API changes to `knowledge_graph.py`.

---

## 9. Summary Table

| Problem | Severity | Fix |
|---------|----------|-----|
| batch_size=50 (100x more round-trips for 5k rels) | Critical | Raise to 250, fallback on assert |
| No progress inside insertion black box | High | Add progress_callback or wrap outer loop with Rich Progress |
| UNWIND+MATCH+MATCH has 2 lookups/row | Medium | Inline props on CREATE, nothing else practical without COPY FROM |
| No deduplication before insert | Low-Medium | Pre-pass dedup by (source_id, target_id) |
| Lock acquisition per 50-rel batch | Low | Resolved if batch_size increases |

---

*Research saved to: docs/research/kg-relationship-insertion-performance-2026-03-03.md*
