# Issue #82: SemanticIndexer Refactoring Plan

**Date:** 2026-03-08
**File:** `src/mcp_vector_search/core/indexer.py`
**Stats:** ~4,471 lines, 61 methods, cyclomatic complexity 161, grade F
**Goal:** Decompose into cohesive, single-responsibility modules without breaking the public API.

---

## 1. Current Class Inventory

### Module-level functions (outside the class, ~165 lines)

| Function | Lines | Purpose |
|---|---|---|
| `cleanup_stale_locks()` | 51–83 | Remove SQLite journal files from interrupted runs |
| `cleanup_stale_transactions()` | 85–124 | Remove stale LanceDB `.txn` files |
| `cleanup_stale_progress()` | 127–164 | Remove stale `progress.json` from crashed runs |
| `_detect_filesystem_type()` | 167–223 | Detect NFS/NVMe for I/O tuning |
| `HealthStatus` re-export | 228 | noqa import alias from models |

### `SemanticIndexerConfig` dataclass (lines 232–358, ~127 lines)

- `from_env()` classmethod that reads all `MCP_VECTOR_SEARCH_*` env vars once

### `SemanticIndexer` class (lines 361–4472, ~4,111 lines)

All 59 methods are listed below, grouped by responsibility.

---

## 2. Complete Method Inventory by Responsibility Group

### Group A: Initialization & Configuration (~150 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `__init__` | 364 | public | Wires up all sub-objects; 200 lines |

### Group B: Pipeline Orchestration — "index_project" entry points (~600 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `index_project` | 2996 | **PUBLIC API** | Thin wrapper; delegates to `_index_project_impl` |
| `_index_project_impl` | 3065 | private | 354-line orchestrator; calls pipeline or sequential modes, cleanup, trend tracking, BM25, KG |
| `chunk_files` | 2531 | **PUBLIC API** | Phase 1 standalone; ~290 lines |
| `embed_chunks` | 2821 | **PUBLIC API** | Phase 2 standalone; ~142 lines |
| `chunk_and_embed` | 2964 | **PUBLIC API** | Sequential wrapper calling both phases; ~32 lines |
| `re_embed_chunks` | 2274 | **PUBLIC API** | Re-embed without re-parsing; ~70 lines |

### Group C: Pipeline Engine — Producer/Consumer (~770 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_index_with_pipeline` | 763 | private | ~760 lines including inner closures |
| `chunk_producer` (inner) | 961 | closure | Defined inside `_index_with_pipeline` |
| `embed_consumer` (inner) | 1184 | closure | Defined inside `_index_with_pipeline` |

### Group D: Two-Phase Core Processing (~650 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_phase1_chunk_files` | 1629 | private | ~360 lines; parse + chunk + store to `chunks.lance` |
| `_phase2_embed_chunks` | 2024 | private | ~250 lines; embed pending chunks, write to `vectors.lance` |
| `_flush_pending_chunks` (inner) | 1808 | closure | Inside `_phase1_chunk_files` |

### Group E: Single-File Indexing (~260 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `index_file` | 3709 | **PUBLIC API** | ~142 lines; parse + chunk + embed single file |
| `_index_file_safe` | 3691 | private | Error-wrapper around `index_file`; 17 lines |
| `reindex_file` | 3852 | **PUBLIC API** | One-liner delegate to `index_file(force=True)` |
| `remove_file` | 3863 | **PUBLIC API** | Delete chunks from DB; ~19 lines |
| `_parse_and_prepare_file` | 3420 | private | Parse + hierarchy + metrics, no DB write; ~55 lines |
| `_process_file_batch` | 3476 | private | Batch-parse multiple files; ~215 lines |

### Group F: Change Detection & File Move Handling (~105 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_detect_file_moves` | 1525 | private | Hash-based rename/move detection; ~103 lines |

### Group G: Atomic Rebuild (~185 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_atomic_rebuild_databases` | 2345 | private | Prepare `.new` DB dirs for force rebuild; ~75 lines |
| `_finalize_atomic_rebuild` | 2431 | private | Atomically rename `.new` → final; ~100 lines |

### Group H: BM25 Index (~100 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_build_bm25_index` | 4242 | private | Read chunks table, build + save BM25 index; ~99 lines |

### Group I: KG Background Build (~37 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_build_kg_background` | 568 | private | Background asyncio task; ~37 lines |
| `get_kg_status` | 606 | **PUBLIC API** | Returns `_kg_build_status` string; 7 lines |

### Group J: Status & Stats Queries (~250 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `get_two_phase_status` | 4000 | **PUBLIC API** | Dict of phase1/phase2 counts; ~64 lines |
| `get_project_status` | 4064 | **PUBLIC API** | Returns `ProjectStatus` dataclass; ~70 lines |
| `get_indexing_stats` | 4136 | **PUBLIC API** | Returns stats dict; ~45 lines |
| `get_indexed_count` | 4182 | **PUBLIC API** | Count unique indexed files; ~17 lines |
| `get_files_to_index` | 4200 | **PUBLIC API** | Async file discovery + change filter; ~41 lines |

### Group K: Metadata & Health (~120 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `get_index_metadata` | 4347 | **PUBLIC API** | Delegate to `self.metadata.get_index_metadata()` |
| `get_index_version` | 3908 | **PUBLIC API** | Delegate to `self.metadata.get_index_version()` |
| `needs_reindex_for_version` | 3916 | **PUBLIC API** | Delegate to `self.metadata.needs_reindex_for_version()` |
| `health_check` | 4360 | **PUBLIC API** | Structural health check (< 100ms); ~86 lines |
| `_save_embedding_metadata` | 4446 | private | Persist model name + dims after indexing; ~26 lines |
| `_auto_rebuild_metadata` | 3953 | private | Rebuild `index_metadata.json` from chunks DB; ~38 lines |

### Group L: Ignore Patterns (thin wrappers, ~23 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `add_ignore_pattern` | 3884 | **PUBLIC API** | Delegate to `self.file_discovery` |
| `remove_ignore_pattern` | 3892 | **PUBLIC API** | Delegate to `self.file_discovery` |
| `get_ignore_patterns` | 3900 | **PUBLIC API** | Delegate to `self.file_discovery` |

### Group M: Embedding Utilities (~33 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `get_embedding_model_name` | 614 | **PUBLIC API** | Introspect DB for model name; ~23 lines |
| `apply_auto_optimizations` | 638 | public (semi-internal) | Invoke CodebaseProfiler; ~63 lines |

### Group N: Auto-Migration (~60 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_run_auto_migrations` | 702 | private | Register + run pending DB migrations; ~60 lines |

### Group O: Chunk Conversion Utilities (~35 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_build_hierarchy_path` | 1989 | private | Build dotted path from class/function names; 8 lines |
| `_get_project_name` | 1998 | private | Monorepo subproject lookup; ~25 lines |

### Group P: Backward-Compat Delegates (~70 lines)

| Method | Line | Visibility | Notes |
|---|---|---|---|
| `_find_indexable_files` | 3925 | private (used by auto_indexer) | Delegate to `file_discovery` |
| `_should_index_file` | 3929 | private (used by auto_indexer) | Delegate to `file_discovery` |
| `_should_ignore_path` | 3935 | private | Delegate to `file_discovery` |
| `_needs_reindexing` | 3941 | private (used by auto_indexer) | Delegate to `metadata` |
| `_load_index_metadata` | 3945 | private (used by auto_indexer) | Delegate to `metadata` |
| `_save_index_metadata` | 3949 | private | Delegate to `metadata` |
| `_parse_file` | 3992 | private | Delegate to `chunk_processor` |
| `_build_chunk_hierarchy` | 3996 | private | Delegate to `chunk_processor` |

---

## 3. Established Public API Surface

The following methods are called from outside `SemanticIndexer` and **must remain stable** (same signature and import location `core.indexer.SemanticIndexer`):

### Called by CLI commands (index.py, embed.py, reindex.py, status.py, search.py, watch.py, index_background.py)
- `index_project(force_reindex, show_progress, skip_relationships, phase, metrics_json, pipeline)`
- `chunk_files(fresh)`
- `embed_chunks(fresh, batch_size)`
- `chunk_and_embed(fresh, batch_size)`
- `re_embed_chunks(batch_size, checkpoint_interval)`
- `reindex_file(file_path)`
- `index_file(file_path, force_reindex, skip_delete)`
- `remove_file(file_path)`
- `get_indexed_count()`
- `get_indexing_stats(db_stats)`
- `get_files_to_index(force_reindex, progress_callback, file_limit)`
- `get_two_phase_status()`
- `get_index_version()`
- `needs_reindex_for_version()`
- `get_ignore_patterns()`

### Called by MCP server (server.py, project_handlers.py)
- `index_project(...)`
- `embed_chunks(...)`
- `get_kg_status()`
- `needs_reindex_for_version()`
- `get_index_version()`

### Called by watcher / auto_indexer
- `index_file(file_path)`
- `_find_indexable_files()` — auto_indexer uses this private method directly
- `_needs_reindexing(file_path, metadata)` — auto_indexer uses this private method directly
- `_process_file_batch(file_paths, force_reindex, skip_delete, metadata_dict)` — auto_indexer
- `_load_index_metadata()` — auto_indexer

### Accessed as attributes (not just called)
- `indexer.cancellation_flag` (set externally by CLI)
- `indexer.project_root`
- `indexer.chunks_backend._db` (checked directly in index_background.py)
- `indexer.chunks_backend.initialize()`
- `indexer.vectors_backend._db`
- `indexer.vectors_backend.initialize()`
- `indexer.directory_index.rebuild_from_files()`
- `indexer.directory_index.save()`
- `indexer.database.get_all_chunks()`
- `indexer.relationship_store.compute_and_store()`

### Also exported at module level
- `SemanticIndexerConfig` — imported directly by `factory.py`, `index.py`

---

## 4. Existing Sub-Objects (Already Extracted)

`SemanticIndexer.__init__` already wires up these well-scoped helper objects:

| Attribute | Class | Module |
|---|---|---|
| `self.file_discovery` | `FileDiscovery` | `core.file_discovery` |
| `self.metadata` | `IndexMetadata` | `core.index_metadata` |
| `self.metrics_collector` | `IndexerMetricsCollector` | `core.metrics_collector` |
| `self.chunk_processor` | `ChunkProcessor` | `core.chunk_processor` |
| `self.monorepo_detector` | `MonorepoDetector` | `utils.monorepo` |
| `self.parser_registry` | (from `get_parser_registry`) | `parsers.registry` |
| `self.directory_index` | `DirectoryIndex` | `core.directory_index` |
| `self.relationship_store` | `RelationshipStore` | `core.relationships` |
| `self.trend_tracker` | `TrendTracker` | `analysis.trends` |
| `self.chunks_backend` | `ChunksBackend` | `core.chunks_backend` |
| `self.vectors_backend` | `VectorsBackend` | `core.vectors_backend` |
| `self.memory_monitor` | `MemoryMonitor` | `core.memory_monitor` |
| `self.database` | `VectorDatabase` | `core.database` |

The class is already a coordinator of these objects. The problem is that the glue code between them (pipelines, phase logic, change detection, atomic rebuild) was never extracted.

---

## 5. Proposed New File/Class Structure

### New files to create

#### `core/pipeline.py` — Pipeline Engine (Group C + parts of D)
**Responsibility:** The producer-consumer pipeline that overlaps Phase 1 and Phase 2.

**Proposed class:** `IndexPipeline`

**Methods to move (from `_index_with_pipeline`):**
- `run(force_reindex) -> tuple[int, int, int]` (extracted from `_index_with_pipeline`)
- Inner `chunk_producer()` and `embed_consumer()` become methods or stay as nested closures within `run()`

**Dependencies needed from `SemanticIndexer`:** `file_discovery`, `metadata`, `chunks_backend`, `vectors_backend`, `database`, `chunk_processor`, `memory_monitor`, `progress_tracker`, `cancellation_flag`, `batch_size`, `embed_batch_size`, `project_root`, `use_multiprocessing`, `_indexer_config`, `_detect_file_moves()`, `_build_hierarchy_path()`

**Estimated lines:** ~800 (the current `_index_with_pipeline` is ~760 lines, this adds ~40 for class boilerplate)

**Risk:** HIGH — This is the most complex method in the file. It contains two tightly coupled inner closures (`chunk_producer`, `embed_consumer`) that share nonlocal state (`files_indexed`, `chunks_created`, `chunks_embedded`, `pipeline_cancel`, `consumer_task`, `chunk_queue`). The closures reference many `self.*` attributes and methods. Extraction requires either passing all dependencies as constructor args or making this a nested class that holds a reference to the parent indexer.

---

#### `core/index_phases.py` — Two-Phase Sequential Processing (Group D + F + G + H + N)
**Responsibility:** Sequential Phase 1/Phase 2 logic, change detection, atomic rebuild, BM25, and migration.

**Proposed class:** `IndexPhaseRunner`

**Methods to move:**
- `_phase1_chunk_files(files, force, file_hash_cache)`
- `_phase2_embed_chunks(batch_size, checkpoint_interval)`
- `_detect_file_moves(current_files, indexed_file_hashes)`
- `_atomic_rebuild_databases(force)`
- `_finalize_atomic_rebuild()`
- `_build_bm25_index()`
- `_run_auto_migrations()`

**Module-level functions that move with it:**
- `cleanup_stale_locks()`
- `cleanup_stale_transactions()`
- `cleanup_stale_progress()`
- `_detect_filesystem_type()`

**Dependencies from `SemanticIndexer`:** `chunks_backend`, `vectors_backend`, `database`, `chunk_processor`, `memory_monitor`, `progress_tracker`, `metadata`, `project_root`, `_mcp_dir`, `_indexer_config`, `_atomic_rebuild_active`, `_build_hierarchy_path()`

**Estimated lines:** ~820

**Risk:** MEDIUM — Methods have clear boundaries but `_phase1_chunk_files` contains an inner closure `_flush_pending_chunks` and references `self._atomic_rebuild_active` state flag. The flag would need to be managed externally (returned or stored on the runner object).

---

#### `core/chunk_dict.py` — Chunk Conversion Utilities (Groups O)
**Responsibility:** Pure conversion of `CodeChunk` objects to storage dicts.

**Functions to extract (currently methods):**
- `_build_hierarchy_path(chunk) -> str`
- `_get_project_name(rel_path, monorepo_detector) -> str`
- `chunk_to_storage_dict(chunk, rel_path, file_hash, monorepo_detector) -> dict` — new helper that consolidates the repeated inline `chunk_dict = {...}` pattern (appears in `_index_with_pipeline`/`chunk_producer` AND `index_file` AND `_phase1_chunk_files`)

**Dependencies:** Only `CodeChunk`, `MonorepoDetector`. No `self` state needed.

**Estimated lines:** ~80

**Risk:** LOW — Pure functions with no side effects. However, the dict-building pattern is duplicated in 3 places; extraction also enables deduplication.

---

### Changes to existing `SemanticIndexer` (what stays)

After extraction, `SemanticIndexer` retains:

1. `__init__` — wiring only
2. All **public API methods** (orchestration entry points): `index_project`, `_index_project_impl` (reduced to ~100 lines of orchestration), `chunk_files`, `embed_chunks`, `chunk_and_embed`, `re_embed_chunks`
3. Single-file operations: `index_file`, `_index_file_safe`, `reindex_file`, `remove_file`, `_parse_and_prepare_file`, `_process_file_batch`
4. Status/stats group: `get_project_status`, `get_two_phase_status`, `get_indexing_stats`, `get_indexed_count`, `get_files_to_index`
5. Metadata/health: `get_index_metadata`, `get_index_version`, `needs_reindex_for_version`, `health_check`, `_save_embedding_metadata`, `_auto_rebuild_metadata`
6. Config/embedding: `get_embedding_model_name`, `apply_auto_optimizations`
7. Ignore patterns: `add_ignore_pattern`, `remove_ignore_pattern`, `get_ignore_patterns`
8. KG: `_build_kg_background`, `get_kg_status`
9. Backward-compat delegates: all `_find_indexable_files`, `_needs_reindexing` etc.

**Estimated residual size:** ~2,000 lines (down from 4,111 lines in the class body)

---

## 6. Extraction Difficulty and Cross-Dependency Matrix

### Methods with the highest cross-dependency count

| Method | Accesses N self.* attrs | Complexity driver |
|---|---|---|
| `_index_with_pipeline` | 15+ | Inner closures sharing nonlocal + cancellation flag + queue management |
| `_index_project_impl` | 12+ | Orchestrates 6 other groups sequentially; 354 lines |
| `_phase1_chunk_files` | 10+ | Inner closure `_flush_pending_chunks`; references `_atomic_rebuild_active` |
| `chunk_files` | 8 | Initialises backends, calls atomic rebuild, delegates to `_phase1` |
| `index_file` | 10 | Touches chunks_backend, vectors_backend, database, metadata, chunk_processor, memory_monitor |

### The `_atomic_rebuild_active` state problem

`_atomic_rebuild_active` is a boolean set on `self` by `_atomic_rebuild_databases()` and read by `_phase1_chunk_files`, `index_file`, `_process_file_batch`, and `_parse_and_prepare_file`. Any extraction must either:
- Keep it on `SemanticIndexer` and pass it as a parameter to extracted methods, or
- Bundle it into an `IndexPhaseRunner` that holds its own copy and syncs back on completion.

The second approach is cleaner but requires `chunk_files()` / `index_project()` to receive the result flag back from the runner.

### The chunk-dict duplication

The `chunk_dict = { "chunk_id": ..., "file_path": ..., ... }` construction appears identically in 3 locations:
- `_index_with_pipeline` / `chunk_producer` (lines ~1058–1090)
- `index_file` (lines ~3773–3800)
- `_phase1_chunk_files` (via the flush path)

Extracting `chunk_to_storage_dict()` to `chunk_dict.py` removes ~80 lines of duplication and is a prerequisite for the pipeline extraction.

---

## 7. Recommended Extraction Order (Safest First)

### Step 1: Extract `chunk_to_storage_dict()` utility — `core/chunk_dict.py`
**Risk:** LOW
**Why first:** Pure functions. Eliminates duplication. Prerequisite for pipeline extraction.
**What to do:** Move `_build_hierarchy_path` and `_get_project_name` as standalone functions. Add `chunk_to_storage_dict()`. Replace the three inline dict literals. Keep `_build_hierarchy_path` as a thin delegate on `SemanticIndexer` for any tests that call it directly.
**Line delta:** `indexer.py` loses ~90 lines; new file gets ~80 lines.

### Step 2: Extract module-level cleanup functions — `core/index_cleanup.py`
**Risk:** VERY LOW
**Why safe:** These are already standalone functions (not methods). They have no dependency on `SemanticIndexer` state. The only call sites are inside `_index_project_impl`.
**What to do:** Move `cleanup_stale_locks`, `cleanup_stale_transactions`, `cleanup_stale_progress`, `_detect_filesystem_type` to a new file. Update the two import sites (`_index_project_impl` and `embed_consumer` in `_index_with_pipeline`).
**Line delta:** `indexer.py` loses ~165 lines; new file gets ~165 lines.

### Step 3: Extract `_detect_file_moves` — inline or to `core/change_detector.py`
**Risk:** LOW
**Why safe:** Self-contained method. Only inputs are `current_files` and `indexed_file_hashes`. Only output is a tuple. Only dependency is `compute_file_hash` (already from `chunks_backend`), `self.project_root`, and `self.progress_tracker`.
**What to do:** Either a standalone function taking `project_root` and `progress_tracker` as arguments, or a small `ChangeDetector` class. Called from `_index_with_pipeline` and `_index_project_impl`.
**Line delta:** `indexer.py` loses ~103 lines.

### Step 4: Extract `_build_bm25_index` and `_run_auto_migrations` — `core/index_post_processing.py`
**Risk:** LOW–MEDIUM
**Why:** These are well-bounded. `_build_bm25_index` only reads `self.chunks_backend._table`, `self.progress_tracker`, and `self._mcp_dir`. `_run_auto_migrations` only needs `self.project_root`. Both are called once at the end of indexing.
**What to do:** Extracted functions accepting the relevant backends/paths as arguments. Or thin wrapper methods that forward to a `PostIndexProcessor`.
**Line delta:** `indexer.py` loses ~160 lines.

### Step 5: Extract atomic rebuild pair — `core/atomic_rebuild.py`
**Risk:** MEDIUM
**Why:** `_atomic_rebuild_databases` mutates `self.chunks_backend`, `self.vectors_backend`, and `self.database` (replaces them with `.new` instances). This mutation coupling is the main risk. Mitigation: return the new backend instances from the function rather than assigning to `self`.
**What to do:** `AtomicRebuildManager` class that takes the current backends and returns new ones. `SemanticIndexer.chunk_files()` and `_index_project_impl` receive the new backends back and re-assign.
**Line delta:** `indexer.py` loses ~185 lines; new file ~200 lines.

### Step 6: Extract Phase 2 embedding — `core/embedding_runner.py`
**Risk:** MEDIUM
**Why:** `_phase2_embed_chunks` accesses `self.chunks_backend`, `self.vectors_backend`, `self.database`, `self.memory_monitor`, `self.progress_tracker`, `self._indexer_config`. It returns `(chunks_embedded, batches_processed)` with no side effects on `self` state beyond the backends.
**What to do:** `EmbeddingRunner` class constructed with the required dependencies. `_phase2_embed_chunks` becomes `EmbeddingRunner.run()`.
**Line delta:** `indexer.py` loses ~250 lines; new file ~280 lines.

### Step 7: Extract `_phase1_chunk_files` — `core/chunking_runner.py`
**Risk:** MEDIUM
**Why:** Similar to Phase 2 but also references `_atomic_rebuild_active`. The inner closure `_flush_pending_chunks` shares batch-level state. Manageable with a `ChunkingRunner` class that owns `_atomic_rebuild_active`.
**What to do:** `ChunkingRunner` class. Inner `_flush_pending_chunks` becomes a private method.
**Line delta:** `indexer.py` loses ~360 lines; new file ~400 lines.

### Step 8: Extract `_index_with_pipeline` — `core/pipeline.py`
**Risk:** HIGH
**Why:** The inner closures share `nonlocal` state (`files_indexed`, `chunks_created`, `pipeline_cancel`, `consumer_task`, etc.). The consumer closure embeds the `embed_consumer` logic that duplicates Phase 2 memory management. Correctly extracting this requires understanding all the nonlocal variable lifetimes.
**Recommended approach:** Lift the inner closures to private methods of an `IndexPipeline` class. Replace `nonlocal` with instance attributes protected by the asyncio task ordering (only one consumer, N producers).
**Prerequisite:** Steps 1, 3, 6, and 7 should be done first so chunk-dict and embedding logic are already in standalone units.
**Line delta:** `indexer.py` loses ~770 lines; new file ~840 lines.

---

## 8. Estimated Line Counts for Final State

| File | Estimated Lines | Notes |
|---|---|---|
| `core/indexer.py` (residual) | ~1,700 | Down from 4,471 |
| `core/pipeline.py` (new) | ~840 | Pipeline engine |
| `core/index_phases.py` (new) | ~820 | Phase 1/2 sequential runners |
| `core/chunk_dict.py` (new) | ~80 | Conversion utilities |
| `core/index_cleanup.py` (new) | ~165 | Stale file cleanup functions |
| `core/atomic_rebuild.py` (new) | ~200 | Atomic rebuild manager |

Total: ~3,805 lines distributed across 6 files vs 4,471 in one.

---

## 9. Test Coverage Inventory

### Unit test files touching SemanticIndexer

| File | Lines | Focus |
|---|---|---|
| `tests/unit/core/test_indexer.py` | 567 | Core indexing: project, file, batch, metadata, path resolution |
| `tests/unit/core/test_pipeline_parallelism.py` | 335 | Pipeline mode vs sequential, producer-consumer |
| `tests/unit/core/test_indexer_multiprocessing.py` | 178 | Multiprocessing flag, workers, error handling |
| `tests/unit/core/test_indexer_collectors.py` | 497 | Metric collectors during indexing |
| `tests/unit/core/test_index_metadata_and_health.py` | 331 | `get_index_metadata`, `health_check` |
| `tests/unit/cli/test_index_cancellation.py` | ~? | Cancellation via `cancellation_flag` |
| `tests/unit/core/test_exception_hierarchy.py` | ~? | Exception types raised by indexer |

**Total dedicated test lines:** ~1,908+ lines across 5 measured files.

### Test coverage implications for refactoring

- `test_pipeline_parallelism.py` mocks `_index_with_pipeline` by name — Step 8 extraction must update mock targets or move tests to `test_pipeline.py`.
- `test_indexer_multiprocessing.py` constructs `SemanticIndexer` with `use_multiprocessing=False` — constructor signature must not change.
- `test_indexer_collectors.py` calls `index_file()` directly — this stays on `SemanticIndexer`.
- `test_index_metadata_and_health.py` tests `health_check()` and `get_index_metadata()` — these stay on `SemanticIndexer`.
- `auto_indexer.py` calls `_find_indexable_files`, `_needs_reindexing`, `_process_file_batch`, `_load_index_metadata` directly — these backward-compat delegates must stay on `SemanticIndexer` after extraction (or `auto_indexer.py` must be updated in the same PR).

---

## 10. Risks and Gotchas

### Gotcha 1: `_atomic_rebuild_active` boolean mutation
`_atomic_rebuild_databases()` switches `self.chunks_backend` and `self.vectors_backend` to new instances pointing at `.new` directories. Any extraction must either return the new instances or keep atomic rebuild on `SemanticIndexer`. The simplest safe approach: return `(bool, chunks_backend, vectors_backend)` from the extracted function and re-assign on `SemanticIndexer`.

### Gotcha 2: `cancellation_flag` accessed externally
`index.py` does `indexer.cancellation_flag = cancellation_flag` before calling `index_project`. The pipeline reads it inside `chunk_producer`. If pipeline is extracted to its own class, the flag must be passed at `IndexPipeline.__init__` or as a parameter to `run()`.

### Gotcha 3: Duplicate chunk-dict construction
The `chunk_dict = { "chunk_id": ..., ... }` literal appears 3 times with slight differences (the `index_file` path includes `"calls"`, `"imports"`, `"inherits_from"` in the chunks dict but not in the vectors dict). Before extracting, audit all 3 call sites to confirm which fields each consumer actually needs.

### Gotcha 4: Inner closures capturing many variables
`chunk_producer` and `embed_consumer` use `nonlocal files_indexed, chunks_created, chunks_embedded`. They also capture `pipeline_cancel`, `consumer_task`, `chunk_queue`, `file_batch_size`, `effective_num_producers` from `_index_with_pipeline`'s local scope. Converting these to class methods requires promoting all these to instance attributes on `IndexPipeline`.

### Gotcha 5: `SemanticIndexerConfig` export
`SemanticIndexerConfig` is imported alongside `SemanticIndexer` in multiple callers:
```python
from ...core.indexer import SemanticIndexer, SemanticIndexerConfig
```
If `SemanticIndexerConfig` is moved to its own module, a re-export must be added to `indexer.py` to keep all import sites working without changes.

### Gotcha 6: Progress tracker stderr writes
Several methods write to `sys.stderr` directly (`_phase2_embed_chunks`, `_build_bm25_index`, `embed_chunks`). These are bypassing the `progress_tracker` abstraction. Extraction surfaces this inconsistency — consider routing all output through `progress_tracker` as part of the cleanup.

### Gotcha 7: `auto_indexer.py` uses private methods
`auto_indexer.py` calls `_find_indexable_files`, `_needs_reindexing`, `_process_file_batch`, and `_load_index_metadata` directly on the indexer. These 4 methods are already thin delegates but they must not be removed even after extraction. `auto_indexer.py` should ideally be updated to call the public equivalents or the delegation.

---

## 11. Summary Recommendation

**Highest-ROI first step:** Extract `chunk_to_storage_dict()` and the cleanup functions (Steps 1 + 2) — zero risk, removes ~255 lines, creates reusable utilities.

**Biggest complexity reduction:** Extracting `_index_with_pipeline` (Step 8) removes ~770 lines and the bulk of the cyclomatic complexity. Do it last after the smaller extractions have simplified its dependencies.

**Recommended PR sequence for #82:**
1. PR A: `chunk_dict.py` + `index_cleanup.py` (Steps 1–2, ~255 lines extracted)
2. PR B: `_detect_file_moves` + `_build_bm25_index` + `_run_auto_migrations` (Steps 3–4, ~263 lines)
3. PR C: `atomic_rebuild.py` + Phase 2 `EmbeddingRunner` (Steps 5–6, ~435 lines)
4. PR D: Phase 1 `ChunkingRunner` (Step 7, ~360 lines)
5. PR E: `IndexPipeline` extraction (Step 8, ~770 lines — highest risk, done last)

Each PR should be accompanied by tests for the extracted class and should not change any public API or constructor signature of `SemanticIndexer`.
