# File Move Re-embedding Bug Investigation

**Date:** 2026-04-26
**Repo:** mcp-vector-search

---

## Summary

The indexer has correct move-detection logic but it can silently fail to fire,
causing moved files to fall through to the standard change-detection loop where
they appear as "new" (hash not found at new path) and get re-embedded.

---

## 1. File Registry / Hash Store

**File:** `src/mcp_vector_search/core/chunks_backend.py:607-647`

`ChunksBackend.get_all_indexed_file_hashes()` scans the LanceDB chunks table
for two columns (`file_path`, `file_hash`) and returns a `dict[str, str]`
(relative path → SHA-256 hash).  This is the single source of truth for change
detection; the legacy `index_metadata.json` (storing mtimes) is only used for
version checks and cleanup, not for incremental change decisions.

---

## 2. Change-Detection Logic

**File:** `src/mcp_vector_search/core/index_metadata.py:164-180`

`IndexMetadata.needs_reindexing()` compares **mtime** only — but this method is
*not* on the hot path for incremental indexing.  The real decision lives in:

**`src/mcp_vector_search/core/indexer.py:620-636`** (pipeline mode) and
**`src/mcp_vector_search/core/indexer.py:1642-1663`** (sequential mode):

```python
stored_hash = indexed_file_hashes.get(rel_path)
if stored_hash is None or stored_hash != file_hash:
    filtered_files.append(f)   # needs re-embedding
```

Change detection uses **content hash (SHA-256)**, not mtime.

---

## 3. Move Detection — What Should Happen

**File:** `src/mcp_vector_search/core/file_move_detector.py:20-123`

`detect_file_moves()` is called before the change-detection loop
(`indexer.py:597-618` and `indexer.py:1095-1116` and `indexer.py:1614-1640`).

It hashes all current on-disk files, then matches DB-orphaned paths to new
paths sharing the same hash.  On a match it calls:

- `chunks_backend.update_file_path(old, new)` — `chunks_backend.py:1009-1042`
- `vectors_backend.update_file_path(old, new)` — `vectors_backend.py:1012-1045`

Both use `lancedb.table.update(where=..., values={file_path: new})` — an
in-place metadata update that **does not re-embed**.

After the update, `indexed_file_hashes` is reloaded so the change-detection
loop sees the new path with its stored hash and skips re-embedding.

---

## 4. Why a Move Can Still Cause Re-embedding (the Bug)

`update_file_path` in `chunks_backend.py:1033` does:

```python
rows = getattr(result, "rows_updated", 0) if result else 0
```

LanceDB's `table.update()` returns `None` in some versions, meaning `rows`
always evaluates to `0` even on success.  The caller logs the count but does
**not gate** the reload on `rows > 0`, so that part is safe.

The actual failure mode is in `detect_file_moves` itself
(`file_move_detector.py:104-121`).  A move is only detected when:

1. The old path is absent from `current_rel_paths` (gone from disk).
2. The new path is absent from `indexed_file_hashes.keys()` (not yet indexed).
3. The counts of orphaned and new paths sharing that hash are equal (1-to-1 or
   N-to-N directory rename).

**Bug trigger:** If the chunks table has **duplicate rows** for the same
`file_path` (e.g., from a failed partial index), `groupby().first()` in
`get_all_indexed_file_hashes()` (`chunks_backend.py:635`) returns one hash
arbitrarily.  If that hash differs from what `compute_file_hash()` produces for
the moved file's content (possible if the table stored a stale hash from a
prior run), condition 2 above fails: the old path is not in `orphaned` (the
stored hash maps to a different set), so the move is invisible to the detector.
The file then fails the `stored_hash != file_hash` check at the new path
(`indexer.py:630`) and is queued for full re-embedding.

**Secondary cause:** The `_index_project_impl` sequential path at
`indexer.py:1676` calls `_phase1_chunk_files(files_to_index, force=force_reindex)`
**without** passing `file_hash_cache`, so the hash computed during move
detection is discarded and `compute_file_hash()` is called again — harmless
but wasteful.

---

## Key File:Line References

| What | Location |
|------|----------|
| Registry (`file_path` → `file_hash` dict) | `chunks_backend.py:607-647` |
| Change-detection decision (pipeline mode) | `indexer.py:625-634` |
| Change-detection decision (sequential mode) | `indexer.py:1645-1658` |
| Move detection entry point | `indexer.py:597-618`, `1095-1116`, `1614-1640` |
| Move detector implementation | `file_move_detector.py:94-121` |
| In-place path update (no re-embed) | `chunks_backend.py:1009-1042`, `vectors_backend.py:1012-1045` |
| Duplicate-row groupby (root cause) | `chunks_backend.py:634-635` |
| Legacy mtime check (not on hot path) | `index_metadata.py:164-180` |

---

## Recommendation

Fix `get_all_indexed_file_hashes()` (`chunks_backend.py:634-635`) to use
`.last()` or validate hash consistency across rows for the same `file_path`,
or deduplicate the chunks table on startup.  This ensures the stored hash
reliably matches `compute_file_hash()` so move detection fires correctly.
