# Kuzu DB Lock Errors During KG Build/Update — Investigation

**Date:** 2026-05-05
**Symptom:** Users get database lock errors when calling `kg_build` or KG update
operations while the MCP server is running.

---

## 1. What Database Is Used and How Connections Work

The knowledge graph uses **Kuzu** (an embedded graph database with Rust bindings).
The database directory is:
```
<project_root>/.mcp-vector-search/knowledge_graph/code_kg/
```

**Critical Kuzu concurrency rule (from the Kuzu Python API docs):**
> "Multiple read-only Database objects can be created with the same database path.
> However, **there cannot be multiple `Database` objects created with the same
> database path**" (in read-write mode).

This is an **exclusive write-lock**: a single Kuzu database directory can only be
opened by one writer at a time. If a second process (or second `kuzu.Database()`
call) tries to open the same path for writing while the first is alive, it raises
a runtime error — which surfaces to the user as a "database lock" error.

---

## 2. How Connections Are Opened and Closed

### `KnowledgeGraph` class (`core/knowledge_graph.py`)

```python
class KnowledgeGraph:
    def __init__(self, db_path: Path):
        self.db = None
        self.conn = None
        self._initialized = False
        self._kuzu_lock = threading.Lock()   # in-process thread lock only

    async def initialize(self):
        db_dir = self.db_path / "code_kg"
        with self._kuzu_lock:
            self.db = kuzu.Database(str(db_dir))   # EXCLUSIVE LOCK acquired here
            self.conn = kuzu.Connection(self.db)
        self._initialized = True

    async def close(self):
        with self._kuzu_lock:
            self.conn = None
            self.db = None                          # EXCLUSIVE LOCK released here
        self._initialized = False
```

**Key observations:**
- `close()` releases the lock by setting `self.db = None`, which drops the Python
  reference and lets the Kuzu `Database` destructor release the OS-level file lock.
- The in-process `threading.Lock` (`_kuzu_lock`) only protects concurrent access
  *within* a single Python process. It does **not** prevent a second process from
  attempting to open the same DB directory.
- There is no read-only mode used anywhere — every caller opens in the default
  read-write (exclusive) mode.

---

## 3. Where the Lock Is Held — Root Causes

### Root Cause A (Primary): MCP `kg_build` handler opens Kuzu directly in-process

The MCP `handle_kg_build` method (`mcp/kg_handlers.py`, lines 76–106) does:

```python
kg = KnowledgeGraph(kg_path)
await kg.initialize()          # opens kuzu.Database — exclusive lock
...
builder = KGBuilder(kg, ...)
await builder.build_from_database(database, ...)   # long-running write operation
await kg.close()               # releases lock only on success or explicit error path
```

**Problem 1 — No subprocess isolation.** The CLI `kg build` command (`cli/commands/kg.py`)
correctly runs the actual Kuzu writes in an **isolated subprocess** (`_kg_subprocess.py`)
after closing all LanceDB handles, specifically because background threads from LanceDB
cause Kuzu segfaults. The MCP `handle_kg_build` does **not** use a subprocess — it calls
`KGBuilder.build_from_database()` directly inside the MCP server process, which already
has LanceDB background threads running.

**Problem 2 — Concurrent MCP calls open the same DB twice.** If two MCP tool calls
arrive simultaneously (or if a user calls `kg_build` while another KG query is still
in flight), two separate `KnowledgeGraph` instances both attempt to call
`kuzu.Database(db_dir)` on the same path. Kuzu will raise a lock error on the second
call because the first instance holds the exclusive write lock.

**Problem 3 — Exception paths may not close the lock.** In `handle_kg_build` the
`await kg.close()` is inside the inner `try` block but NOT in a `finally` clause for
the lock-holding code. If an exception occurs between `await kg.initialize()` and
`await kg.close()`, the `kg` object leaks and the `kuzu.Database` destructor runs only
when the garbage collector collects it (non-deterministic timing). During that window,
any other call that tries to open the same DB will hit a lock error.

```python
# From kg_handlers.py handle_kg_build:
try:
    kg = KnowledgeGraph(kg_path)
    await kg.initialize()          # lock acquired
    ...
    await kg.close()               # lock released — but only on happy path!
finally:
    await database.__aexit__(...)  # LanceDB cleanup; no KG cleanup here
```

### Root Cause B (Secondary): Background KG build from the indexer

If the environment variable `MCP_VECTOR_SEARCH_AUTO_KG=true` is set, the
`SemanticIndexer._build_kg_background()` method fires an `asyncio.create_task` that
opens a `KnowledgeGraph` connection and runs `build_from_database()`. This runs
concurrently with the MCP server's event loop and holds the Kuzu exclusive write lock
for the entire build duration.

If a user then calls `kg_build` (via MCP) or any `kg_*` query tool while the background
build is active, the second `kuzu.Database()` open attempt fails with a lock error.

```python
# indexer.py line 1789:
self._kg_build_task = asyncio.create_task(self._build_kg_background())

# _build_kg_background() opens:
kg = KnowledgeGraph(kg_path)
await kg.initialize()   # exclusive lock — held for entire background build
...
await kg.close()
```

### Root Cause C (Tertiary): KG query handlers are not lock-coordinated

All individual KG query handlers (`handle_kg_stats`, `handle_kg_query`,
`handle_kg_ontology`, `handle_kg_ia`, `handle_trace_execution_flow`,
`handle_kg_history`, `handle_kg_callers_at_commit`) each follow the pattern:

```python
kg = KnowledgeGraph(kg_path)
await kg.initialize()   # opens kuzu.Database exclusively
...
await kg.close()
```

While these are read-only operations semantically, the code opens Kuzu in the
default **read-write** mode every time. If two of these handlers run concurrently
(which is possible under the async MCP server), the second `kg.initialize()` will
fail with a lock error because Kuzu does not allow two simultaneous read-write openers.

---

## 4. What Error Messages Are Produced

Kuzu raises a `RuntimeError` (from Rust) when a second process or second instance
tries to open the same database directory for writing while it is locked. The exact
message is typically:

```
RuntimeError: Cannot open database ... database is locked by another process
```

or on some platforms:

```
RuntimeError: Failed to open file ... (OS error: Resource temporarily unavailable)
```

In the MCP handler, this is caught by the outer `except Exception as e:` block and
returned as:

```json
{"isError": true, "content": [{"text": "Knowledge graph build failed: <lock error>"}]}
```

Or for query tools:

```json
{"isError": true, "content": [{"text": "Knowledge graph query failed: <lock error>"}]}
```

---

## 5. Kuzu Concurrency Model Summary

| Scenario | Allowed? |
|---|---|
| Multiple processes, all read-only (`read_only=True`) | Yes |
| One writer + zero readers | Yes |
| One writer + one or more readers (different `Database` instances, same path) | **No — lock error** |
| Two writers simultaneously | **No — lock error** |
| Two `Connection` objects from the *same* `Database` object | Yes (shared) |

Kuzu is an **embedded database** (like SQLite). It does not have a server process.
Concurrency is managed at the `Database` object level — only ONE `Database` instance
may hold a given path in read-write mode at a time.

---

## 6. Fix Options and Trade-offs

### Option 1: Shared singleton `KnowledgeGraph` instance with read-write coordination (Recommended)

**Approach:** Create one shared `KnowledgeGraph` instance in the MCP server at startup
(similar to how `database` is shared). Protect all access with an `asyncio.Lock` that
distinguishes reads from writes. Query operations take a shared (non-exclusive) logical
lock; build operations take an exclusive lock.

**Implementation sketch:**
```python
# In MCPVectorSearchServer.__init__:
self._kg: KnowledgeGraph | None = None
self._kg_write_lock = asyncio.Lock()

# In MCPVectorSearchServer.initialize():
self._kg = KnowledgeGraph(kg_path)
await self._kg.initialize()   # open once; held for server lifetime

# In all query handlers (kg_stats, kg_query, etc.):
# No additional lock needed — reads are safe on the shared connection

# In kg_build handler:
async with self._kg_write_lock:   # exclusive: block other writes
    await builder.build_from_database(...)
```

**Trade-offs:**
- Eliminates lock errors entirely for in-process concurrent access.
- Requires the Kuzu `Database` to be open for the server lifetime (minor RAM cost).
- Build and query can no longer truly run in parallel (the write lock serializes them).
- If the subprocess model is used for build (see Option 3), the persistent connection
  must be closed before spawning the subprocess.

### Option 2: Open/close per request with an `asyncio.Lock` gate

**Approach:** Keep the current open-per-request pattern, but add a server-level
`asyncio.Lock` that all KG handlers must acquire before opening a `KnowledgeGraph`.
Use `try/finally` to guarantee close.

**Implementation sketch:**
```python
# In MCPVectorSearchServer:
self._kg_lock = asyncio.Lock()

# In every KG handler:
async with self._kg_lock:
    kg = KnowledgeGraph(kg_path)
    try:
        await kg.initialize()
        result = await kg.some_operation()
    finally:
        await kg.close()   # always released, even on exception
```

**Trade-offs:**
- Minimal code change.
- All KG operations are serialized (no parallel queries).
- For long-running builds, all queries block. The lock hold time for `kg_build` is
  several seconds to minutes on large repos.
- Does not fix the cross-process lock problem (CLI `kg build` run while MCP is active).

### Option 3: Use the subprocess model for MCP `kg_build` (matches CLI behavior)

**Approach:** Make `handle_kg_build` in `kg_handlers.py` use the same subprocess
isolation that `cli/commands/kg.py` already uses. The MCP handler would:
1. Dump chunks to a temp JSON file (after closing LanceDB).
2. Spawn `_kg_subprocess.py` as a child process.
3. Wait for the subprocess to finish.

**Trade-offs:**
- Matches the battle-tested CLI path exactly.
- Eliminates the in-process concurrent-connection problem for builds.
- Adds latency from subprocess startup.
- Does not fix lock errors for *query* operations (those still need the asyncio lock
  from Option 2).

### Option 4: Open KG query connections in `read_only=True` mode

**Approach:** For all query-only handlers (`kg_stats`, `kg_query`, `kg_ontology`,
`kg_ia`, `kg_history`, `trace_execution_flow`, `kg_callers_at_commit`), open the
`kuzu.Database` with `read_only=True`. Kuzu allows multiple simultaneous read-only
openers on the same path.

**Implementation change in `KnowledgeGraph`:**
```python
def __init__(self, db_path: Path, read_only: bool = False):
    ...
    self._read_only = read_only

async def initialize(self):
    ...
    self.db = kuzu.Database(str(db_dir), read_only=self._read_only)
```

**Trade-offs:**
- Allows concurrent read queries without any locking overhead.
- Does NOT fix the build vs. query conflict: a read-only opener cannot coexist with
  an active write opener (Kuzu docs say "there cannot be multiple `Database` objects
  created with the same database path" — the restriction is bidirectional for
  read-only vs read-write conflict).
- Actually, re-reading the Kuzu docs more carefully: "Multiple read-only Database
  objects can be created with the same database path. However, there cannot be
  multiple Database objects created with the same database path" — this means multiple
  read-only is OK, but one read-write + anything else is NOT OK.
- So this only helps for concurrent read-only query scenarios, not build + query.

### Option 5: Hybrid (Recommended combination)

Combine Options 1 and 3:

1. **For queries:** Use a shared singleton `KnowledgeGraph` opened in **read-only
   mode** at server startup. All query handlers use the shared connection — no opening
   or closing per request, no lock errors.
2. **For `kg_build`:** Before building, close the shared read-only connection. Run the
   build in an isolated subprocess (as the CLI does). After the subprocess finishes,
   re-open the shared read-only connection.
3. **Guard with an asyncio lock** around the build-close-spawn-reopen sequence to
   prevent concurrent builds.

This matches how databases are typically handled: a persistent read connection for
live queries, and a brief "take offline" for writes.

---

## 7. Summary Diagnosis

| Condition | Lock Held By | Second Opener | Result |
|---|---|---|---|
| MCP server idle, user calls `kg_build` via MCP | Nothing | `handle_kg_build` opens `KnowledgeGraph` | Works |
| MCP server handling a `kg_query` (in-flight), another `kg_build` arrives | `handle_kg_query`'s `KnowledgeGraph` | `handle_kg_build` tries to open same path | **Lock error** |
| `_build_kg_background` running (`AUTO_KG=true`), user calls `kg_build` | Background task's `KnowledgeGraph` | `handle_kg_build` tries to open same path | **Lock error** |
| CLI `mvs kg build` runs while MCP server has an active KG operation | MCP server's `KnowledgeGraph` | CLI subprocess tries to open same path | **Lock error** |
| Two concurrent `kg_query` calls in default (read-write) mode | First query's `KnowledgeGraph` | Second query's `KnowledgeGraph` | **Lock error** |

**The core problem** is that every KG handler instantiates a new `KnowledgeGraph`
and calls `kuzu.Database(path)` without checking whether another instance already
holds the lock — either within the same MCP process (concurrent async requests) or
from an external process (CLI). There is no server-level coordination of Kuzu
database access.

**Immediate minimal fix:** Add `try/finally` around every `kg.initialize()` / `kg.close()`
pair to prevent leaked connections on exceptions, plus add an `asyncio.Lock` in the MCP
server to serialize all KG accesses. This stops concurrent in-process lock collisions.

**Proper fix:** Shared singleton connection with read-only mode for queries, subprocess
isolation for builds (matching the existing CLI architecture).
