# Index Disappearing Investigation

**Date:** 2026-03-12
**Project:** mcp-vector-search
**Researcher:** Research Agent

---

## Summary

Indexed projects do not "lose" their index in the sense of it being deleted. The index is stored
inside the project being indexed (at `<project_root>/.mcp-vector-search/`). The **apparent
disappearance** has two distinct root causes:

1. **Incomplete Phase 2 (embedding):** Chunking (Phase 1) completes and writes `chunks.lance`, but
   the embedding phase (Phase 2) never writes `vectors.lance`. Search fails silently or returns
   nothing because the vector index is empty. This is the exact state of `~/Duetto/cto` right now.

2. **The index directory is gitignored by design, so `git pull` / clone on another machine produces
   an empty `.mcp-vector-search/` directory** that was never committed - users must re-index.

---

## 1. Index Storage Architecture

### Where indexes live

All index data is written **inside the project being indexed**, in a hidden directory:

```
<project_root>/.mcp-vector-search/
    config.json                  # Project config (embedding model, extensions, etc.)
    index_metadata.json          # File mtimes for incremental re-index (may not exist)
    directory_index.json         # Directory structure snapshot
    bm25_index.pkl               # BM25 keyword search index (can be 150MB+)
    migrations.json              # Migration state
    schema_version.json          # Schema version
    vendor.yml / vendor_metadata.json
    progress.json                # Live progress (phase, counts)
    indexing_errors.log          # Error log
    cache/                       # Embedding cache
    lance/
        chunks.lance/            # Phase 1: parsed code chunks (no vectors)
            data/                # 37+ .lance fragment files
            _versions/           # LanceDB manifest files
            _transactions/       # LanceDB transaction files
        vectors.lance/           # Phase 2: embedded vectors (may NOT exist if interrupted)
    knowledge_graph/
        kg.db                    # SQLite knowledge graph (optional, background)
```

### Path resolution chain

```
get_default_index_path(project_root) = project_root / ".mcp-vector-search"
```

Defined in `src/mcp_vector_search/config/defaults.py` lines 527-529.

**Override chain** (highest to lowest priority):
1. Explicit `--index-path` CLI argument
2. `INDEX_PATH` environment variable
3. `config.index_path` from `config.json` (= `<project_root>/.mcp-vector-search`)

### What constitutes a complete, searchable index

A project is "indexed and searchable" only when **both phases are complete**:

| Phase | Backend | Path | Minimum for search |
|-------|---------|------|--------------------|
| Phase 1 (chunking) | `ChunksBackend` | `lance/chunks.lance/` | Required |
| Phase 2 (embedding) | `VectorsBackend` | `lance/vectors.lance/` | **Required for vector search** |
| BM25 | `BM25Backend` | `bm25_index.pkl` | Required for hybrid/keyword search |

`project.py:is_initialized()` checks only that `config.json` and the `.mcp-vector-search/`
directory exist - it does **not** verify that embeddings were written or that vectors are
non-empty.

---

## 2. Why Indexes Appear to Disappear

### Root Cause A: Phase 2 never ran or was interrupted (most common)

The two-phase pipeline in `indexer.py` / `pipeline.py`:

- **Phase 1** parses files and writes chunks to `lance/chunks.lance/` - fast, ~30 seconds for
  a large repo.
- **Phase 2** embeds those chunks and writes to `lance/vectors.lance/` - slow, can take minutes
  to hours for 300K+ chunks.

If the process is killed, crashes, or hits an OOM between phases, `progress.json` shows
`phase: "complete"` but `embedded_chunks: 0`. The BM25 index IS written, but no vector search
works because `vectors.lance/` was never created.

**The `progress.json` "complete" status is misleading** - it means Phase 1 chunking completed,
but does not mean Phase 2 embedding completed.

### Root Cause B: `is_initialized()` only checks directory/config existence

```python
# project.py line 74-79
def is_initialized(self) -> bool:
    config_path = get_default_config_path(self.project_root)
    index_path = get_default_index_path(self.project_root)
    return config_path.exists() and index_path.exists()
```

This returns `True` even if there are zero chunks, zero vectors, or the embedding phase never ran.
The MCP `get_project_status` tool reports `total_chunks: 0` in this case, which is the signal
that re-indexing is needed, but the distinction between "never indexed" and "chunked but not
embedded" is not surfaced.

### Root Cause C: `DEFAULT_IGNORE_PATTERNS` includes `.mcp-vector-search`

In `defaults.py` line 429:
```python
".mcp-vector-search",  # Our own index directory
```

This means the indexer correctly skips its own index directory during file discovery.
**However**, this pattern is also in the `DEFAULT_IGNORE_PATTERNS` list that is checked BEFORE
gitignore rules, using a simple string membership check on path parts (not a gitignore-style
match). This is correct behavior but worth noting.

### Root Cause D: Corruption auto-recovery deletes the table

In `lancedb_backend.py` lines 340-346, if a LanceDB table is detected as corrupted during
`initialize()`, it is **silently deleted**:

```python
shutil.rmtree(table_path)
logger.info(f"Deleted corrupted table: {table_path}")
```

After deletion, the table is recreated empty. If this happens to `chunks.lance`, the entire index
is gone and the project appears unindexed on next startup.

### Root Cause E: `force=True` on `initialize()` deletes the entire index directory

`project.py` line 119-128: when `force=True`, `shutil.rmtree(index_path)` deletes the entire
`.mcp-vector-search/` directory before rebuilding. Any in-flight or external reference to old
paths will find the index gone.

---

## 3. Gitignore Behavior

### Current behavior: CORRECTLY gitignored in `~/Duetto/cto`

The `.gitignore` in `~/Duetto/cto` (line 59) already contains:
```
.mcp-vector-search/
```

This was either added automatically by `ensure_gitignore_entry()` or manually.
`git check-ignore -v .mcp-vector-search` confirms the pattern matches. The directory is
properly gitignored and will not be committed.

### Automatic gitignore update (on init)

`project.py` lines 133-148 call `ensure_gitignore_entry()` during `initialize()`:
- Adds `.mcp-vector-search/` to `.gitignore` if the project is a git repo
- Only adds it once (checks for semantic equivalents before adding)
- Is **non-blocking**: if it fails, initialization continues with a warning
- Does **not** trigger on re-index (only on `init`)

**Gap:** If a user runs `index_project` via MCP without ever running `init` (which is common for
MCP-first workflows), the `ensure_gitignore_entry()` function is NOT called. The `.gitignore`
update only happens via `ProjectManager.initialize()`.

Looking at `project_handlers.py` `handle_index_project()`, it calls the indexer directly
without going through `ProjectManager.initialize()`. This means projects indexed via the MCP
tool may never get the `.gitignore` entry unless `init` was run separately.

---

## 4. Specific Findings: `~/Duetto/cto`

### Current state

```
~/Duetto/cto/.mcp-vector-search/
    config.json          EXISTS  (384-dim MiniLM model, phase-complete)
    bm25_index.pkl       EXISTS  (154 MB - BM25 keyword index is complete)
    directory_index.json EXISTS  (1.2 MB)
    lance/
        chunks.lance/    EXISTS  (37 data files, 165 MB total - Phase 1 COMPLETE)
    vectors.lance/       MISSING (Phase 2 embedding never ran or was interrupted)
    index_metadata.json  MISSING (not written by current architecture)
    progress.json        phase: "complete", embedded_chunks: 0
```

### Diagnosis

**Phase 1 (chunking) completed successfully:**
- 9,347 files chunked
- 308,797 total chunks created
- 37 LanceDB data fragments written
- BM25 index built (154 MB)

**Phase 2 (embedding) never ran or was killed:**
- `embedded_chunks: 0` in `progress.json`
- No `vectors.lance/` directory exists
- Vector search returns empty results

### Gitignore status

`.mcp-vector-search/` is **correctly gitignored** on line 59 of `~/Duetto/cto/.gitignore`.
The entry was already present (manually or via prior `init`). This is not a cause of the problem.

### Why search appears broken

The MCP server's `SemanticSearchEngine` uses `VectorsBackend` for vector search and falls back
to the legacy `LanceVectorDatabase` (the `chunks` table in `lancedb_backend.py`). The `chunks`
table stores chunks but only has vector embeddings if Phase 2 wrote them there. Since Phase 2
did not complete, all search paths return empty results.

BM25/keyword search via `bm25_index.pkl` MAY work because that file exists.

---

## 5. Why the Index Can Disappear on a Team/Multi-Machine Setup

Since `.mcp-vector-search/` is gitignored:
- Developer A indexes on Machine A: index is in `~/project/.mcp-vector-search/` (not committed)
- Developer B clones or pulls: gets no `.mcp-vector-search/` directory
- Developer B runs `get_project_status`: sees `total_chunks: 0`, index appears "gone"
- Every developer must run `index_project` independently on their machine

This is **by design** (index files are large - 150MB+ - and machine-specific due to absolute
paths in metadata). The design is correct but needs clearer user communication.

---

## 6. Recommendations for Making Indexes More Robust

### Priority 1 (Critical): Fix Phase 2 completion detection

The `progress.json` `phase: "complete"` flag is set when Phase 1 (chunking) completes, not
when Phase 2 (embedding) completes. This is the root cause of the misleading "indexed but empty"
state.

**Fix:** Add a separate `embedding_complete` boolean or change `phase` to `"chunking_complete"`
vs `"embedding_complete"`. The `get_project_status` MCP tool should detect and report this
split state clearly.

### Priority 2 (High): Always call `ensure_gitignore_entry` on index, not just init

The `ensure_gitignore_entry()` function is only called in `ProjectManager.initialize()`. When
the MCP `index_project` tool runs without a prior `init`, the `.gitignore` entry is never added.

**Fix:** Call `ensure_gitignore_entry()` at the start of `handle_index_project()` in
`project_handlers.py`, or in `SemanticIndexer.__init__()` when `_mcp_dir` is first established.

### Priority 3 (High): Surface Phase 2 incompleteness in `get_project_status`

Currently `get_project_status` reports `total_chunks` from the database stats, which reflects
the LanceDB `chunks` table (Phase 1). If Phase 2 was never run, `embedded_chunks` in
`progress.json` will be 0 but the tool may still say "9347 files indexed".

**Fix:** Read `progress.json` in `handle_get_project_status` and add:
- `chunks_stored: N` (Phase 1)
- `vectors_embedded: N` (Phase 2)
- `embedding_complete: bool`

### Priority 4 (Medium): Offer automatic Phase 2 resume

The two-phase architecture supports resumable embedding (`VectorsBackend.get_unembedded_chunk_ids`).
If `get_project_status` detects Phase 1 complete but Phase 2 incomplete, it could automatically
offer to resume with `embed_chunks` rather than requiring a full re-index.

### Priority 5 (Low): Write `index_metadata.json` in the new architecture

The `IndexMetadata` class writes `<project_root>/.mcp-vector-search/index_metadata.json` with
file modification times and embedding model info. In the current `~/Duetto/cto` index, this
file is **missing** (`No index_metadata.json`). This means incremental re-indexing falls back
to re-indexing everything. Ensure `IndexMetadata.save()` is called after Phase 2 completes.

### Priority 6 (Low): Document the per-machine index model

Add clear documentation that:
- Each developer must run `index_project` on their own machine
- The index is not committed to git (by design)
- The `bm25_index.pkl` will be 100MB+ for large repos (normal)
- Phase 2 (embedding) is the slow step and can be resumed if interrupted

---

## Files Investigated

| File | Role |
|------|------|
| `src/mcp_vector_search/config/defaults.py` | `get_default_index_path()`, `DEFAULT_IGNORE_PATTERNS` |
| `src/mcp_vector_search/core/project.py` | `ProjectManager.initialize()`, `is_initialized()`, gitignore update |
| `src/mcp_vector_search/core/factory.py` | `resolve_index_path()`, component wiring |
| `src/mcp_vector_search/core/indexer.py` | `SemanticIndexer.__init__()`, `_mcp_dir` resolution |
| `src/mcp_vector_search/core/index_metadata.py` | `IndexMetadata` - file mtime tracking |
| `src/mcp_vector_search/core/index_cleanup.py` | `cleanup_stale_transactions()`, `cleanup_stale_progress()` |
| `src/mcp_vector_search/core/atomic_rebuild.py` | `AtomicRebuildManager` - force-rebuild path |
| `src/mcp_vector_search/core/lancedb_backend.py` | Corruption auto-recovery (silent `rmtree`) |
| `src/mcp_vector_search/core/vectors_backend.py` | `VectorsBackend` - Phase 2 storage |
| `src/mcp_vector_search/utils/gitignore_updater.py` | `ensure_gitignore_entry()` |
| `src/mcp_vector_search/mcp/project_handlers.py` | `handle_get_project_status()`, `handle_index_project()` |
| `src/mcp_vector_search/mcp/server.py` | `MCPVectorSearchServer` - project root resolution |
| `~/Duetto/cto/.gitignore` | Has `.mcp-vector-search/` on line 59 (correctly gitignored) |
| `~/Duetto/cto/.mcp-vector-search/progress.json` | `phase: complete`, `embedded_chunks: 0` (Phase 2 missing) |
