## Task 7: 代码库全量索引（build_engine + CodeSearchStore）

**Files:**
- Create: `gitai/agent_core/retrieval/factory.py`
- Create: `gitai/agent_core/retrieval/code_store.py`
- Modify: `gitai/agent_core/rag/chunker.py`（只加一个公开别名）
- Modify: `gitai/agent_core/retrieval/__init__.py`
- Test: `tests/test_code_store.py`

**Interfaces:**
- Consumes: `retrieval.engine.HybridSearchEngine`（Task 4，含 `vector=` 缓存入口）、`retrieval.rerank.build_llm_reranker`（Task 5）、`Config` 新字段（Task 6）、`rag.chunker.chunk_code` + `FUNC_DEF_PATTERNS`、`rag.indexer._SUPPORTED_EXTENSIONS` / `_SKIP_DIRS`
- Produces:
  - `factory.build_engine(cfg: Any = None, *, embedder=None, vectors=None, sparse=None) -> HybridSearchEngine`
  - `class CodeSearchStore`：`__init__(self, workspace: str = ".", *, cfg: Any = None, embedder=None)`、`sync(self) -> dict`、`status(self) -> dict`、`search(self, query: str, top_k: int = 5, *, max_per_file: int | None = None) -> list[dict]`、`clear(self) -> int`、`ensure_synced(self) -> dict`、属性 `ready: bool`
  - `setup_code_workspace(workspace: str) -> None`、`get_code_store(workspace: str | None = None) -> CodeSearchStore`（与 `doc_rag`/`history_rag` 同构的按 workspace 单例）
  - `sync()`/`status()` 共同键：`files`、`chunks`、`indexed_files`、`eligible_files`、`coverage`（0~1 float）、`degraded`（`list[str]` 相对路径）、`skipped_unreadable`（`int`）；`sync()` 额外 `added_files`/`updated_files`/`removed_files`/`unchanged_files`/`elapsed`；`status()` 额外 `store_path`/`vectors_path`/`last_sync`/`built`
  - `search()` 每项键：`id`、`path`（相对工作区的 posix 路径）、`chunk_index`、`score`、`snippet`（块首个非空行，截 120 字）、`degraded`（bool）

- [ ] **Step 1: 写失败测试**

新建 `tests/test_code_store.py`：

```python
"""retrieval.code_store — 全量覆盖是硬约束，规模由检索期过滤而非索引截断吸收。"""

import json
from types import SimpleNamespace

import numpy as np
import pytest

from gitai.agent_core.retrieval.code_store import CodeSearchStore, get_code_store
from gitai.agent_core.retrieval.factory import build_engine


def _cfg(**over):
    base = dict(enable_hybrid_retrieval=True, retrieval_recall_k=200, retrieval_max_per_file=2,
                rerank_with_llm=False, rerank_candidate_k=20,
                code_index_degrade_file_bytes=512_000, code_index_degraded_head_bytes=8_000,
                exclude_patterns=[])
    base.update(over)
    return SimpleNamespace(**base)


def _write(root, rel, text):
    p = root / rel
    p.parent.mkdir(parents=True, exist_ok=True)
    p.write_text(text, encoding="utf-8")
    return p


class TestBuildEngine:
    def test_defaults_when_cfg_missing(self, monkeypatch):
        monkeypatch.setattr("gitai.agent_core.retrieval.factory.load_config", lambda *a, **k: _cfg())
        eng = build_engine()
        assert eng.recall_k == 200 and eng.max_per_file == 2

    def test_hybrid_disabled_gives_dense_only_engine(self):
        eng = build_engine(_cfg(enable_hybrid_retrieval=False))
        assert eng.enable_sparse is False

    def test_rerank_built_only_when_configured(self, monkeypatch):
        monkeypatch.setattr("gitai.agent_core.retrieval.factory.build_llm_reranker",
                            lambda cfg: "RERANKER")
        assert build_engine(_cfg()).reranker is None
        assert build_engine(_cfg(rerank_with_llm=True)).reranker == "RERANKER"


class TestFullCoverage:
    """用户裁定的硬约束：不得静默丢弃文件。"""

    def test_sync_indexes_all_files_without_truncation(self, tmp_path):
        for i in range(60):
            _write(tmp_path, f"pkg/mod{i}.py", f"def uniq_symbol_{i}():\n    return {i}\n")
        store = CodeSearchStore(str(tmp_path), cfg=_cfg())
        info = store.sync()
        assert info["indexed_files"] == 60 == info["eligible_files"]
        assert info["coverage"] == 1.0
        assert store.search("uniq_symbol_37", top_k=1)[0]["path"] == "pkg/mod37.py"

    def test_oversized_file_is_degraded_not_dropped(self, tmp_path):
        body = "\n".join(f" filler_line_{i} = {i}" for i in range(200))
        _write(tmp_path, "huge.py", "def needle_symbol():\n    return 1\n" + body)
        store = CodeSearchStore(str(tmp_path), cfg=_cfg(code_index_degrade_file_bytes=200,
                                                        code_index_degraded_head_bytes=60))
        info = store.sync()
        assert info["degraded"] == ["huge.py"]
        assert info["indexed_files"] == 1 and info["coverage"] == 1.0
        hits = store.search("needle_symbol", top_k=3)
        assert hits and hits[0]["path"] == "huge.py" and hits[0]["degraded"] is True

    def test_undecodable_file_reported_as_unreadable_not_silently_skipped(self, tmp_path):
        _write(tmp_path, "good.py", "def alpha():\n    return 1\n")
        (tmp_path / "blob.py").write_bytes(b"\x00\xff" * 200)
        info = CodeSearchStore(str(tmp_path), cfg=_cfg()).sync()
        assert info["skipped_unreadable"] == 1
        assert info["indexed_files"] == 1


class TestIncremental:
    def test_second_sync_reports_all_unchanged(self, tmp_path):
        _write(tmp_path, "a.py", "def alpha():\n    return 1\n")
        _write(tmp_path, "b.py", "def beta():\n    return 2\n")
        store = CodeSearchStore(str(tmp_path), cfg=_cfg())
        store.sync()
        info = store.sync()
        assert info["added_files"] == 0 and info["updated_files"] == 0
        assert info["unchanged_files"] == 2

    def test_changed_file_is_rechunked_and_old_chunks_dropped(self, tmp_path):
        _write(tmp_path, "a.py", "def gone_symbol():\n    return 1\n")
        store = CodeSearchStore(str(tmp_path), cfg=_cfg())
        store.sync()
        _write(tmp_path, "a.py", "def fresh_symbol():\n    return 2\n")
        assert store.sync()["updated_files"] == 1
        assert store.search("gone_symbol", top_k=3) == []
        assert store.search("fresh_symbol", top_k=1)[0]["path"] == "a.py"

    def test_deleted_file_is_removed_from_index(self, tmp_path):
        _write(tmp_path, "a.py", "def alpha():\n    return 1\n")
        store = CodeSearchStore(str(tmp_path), cfg=_cfg())
        store.sync()
        (tmp_path / "a.py").unlink()
        assert store.sync()["removed_files"] == 1
        assert store.search("alpha", top_k=3) == []

    def test_ensure_synced_only_builds_once(self, tmp_path):
        _write(tmp_path, "a.py", "def alpha():\n    return 1\n")
        store = CodeSearchStore(str(tmp_path), cfg=_cfg())
        assert store.ready is False
        store.ensure_synced()
        assert store.ready is True
        assert store.ensure_synced()["unchanged_files"] == 1


class TestExclusions:
    def test_noise_and_self_index_dirs_are_skipped(self, tmp_path):
        _write(tmp_path, "src/keep.py", "def keep_symbol():\n    return 1\n")
        _write(tmp_path, ".git/config.py", "def git_noise():\n    return 1\n")
        _write(tmp_path, "__pycache__/cache.py", "def cache_noise():\n    return 1\n")
        _write(tmp_path, ".gitai/doc_helper.py", "def vector_noise():\n    return 1\n")
        _write(tmp_path, "agent_workspace/tmp.py", "def tmp_noise():\n    return 1\n")
        _write(tmp_path, "notes/exclude_me.py", "def excluded_symbol():\n    return 1\n")
        store = CodeSearchStore(str(tmp_path), cfg=_cfg(exclude_patterns=["notes/*"]))
        info = store.sync()
        assert info["indexed_files"] == 1 == info["eligible_files"]
        assert store.search("excluded_symbol", top_k=3) == []
        assert store.search("git_noise", top_k=3) == []


class TestSymlinkSafety:
    def test_symlinked_directory_is_not_followed(self, tmp_path):
        import os

        real = tmp_path / "real"
        _write(real, "a.py", "def alpha():\n    return 1\n")
        try:
            os.symlink(real, tmp_path / "link", target_is_directory=True)
        except (OSError, NotImplementedError):
            pytest.skip("当前平台/权限不允许创建符号链接")
        info = CodeSearchStore(str(tmp_path), cfg=_cfg()).sync()
        assert info["indexed_files"] == 1


class TestPersistence:
    def test_manifest_roundtrip_keeps_coverage_statement(self, tmp_path):
        _write(tmp_path, "a.py", "def alpha():\n    return 1\n")
        CodeSearchStore(str(tmp_path), cfg=_cfg()).sync()
        manifest = json.loads((tmp_path / ".gitai" / "code_index.json").read_text(encoding="utf-8"))
        assert manifest["version"] == 2
        assert manifest["manifest"]["a.py"]["chunks"][0]["content_hash"]

        fresh = CodeSearchStore(str(tmp_path), cfg=_cfg())
        assert fresh.status()["built"] is False
        assert fresh.sync()["unchanged_files"] == 1
        assert fresh.status()["built"] is True

    def test_tfidf_backend_writes_no_vector_sidecar(self, tmp_path):
        _write(tmp_path, "a.py", "def alpha():\n    return 1\n")
        CodeSearchStore(str(tmp_path), cfg=_cfg()).sync()
        assert not (tmp_path / ".gitai" / "code_index_vectors.json").exists()


class _CountingEmbedder:
    backend = "litellm"
    dim = 8

    def __init__(self):
        self.calls = 0

    def embed(self, text):
        self.calls += 1
        vec = np.zeros(self.dim, dtype=np.float32)
        vec[abs(hash(text)) % self.dim] = 1.0
        return vec

    def embed_batch(self, texts):
        return [self.embed(t) for t in texts]


class TestVectorReuseForApiEmbedding:
    def test_litellm_backend_writes_sidecar_and_skips_reembedding(self, tmp_path):
        _write(tmp_path, "a.py", "def alpha():\n    return 1\n")
        first = _CountingEmbedder()
        CodeSearchStore(str(tmp_path), cfg=_cfg(), embedder=first).sync()
        assert first.calls > 0
        assert (tmp_path / ".gitai" / "code_index_vectors.json").exists()

        second = _CountingEmbedder()
        store2 = CodeSearchStore(str(tmp_path), cfg=_cfg(), embedder=second)
        store2.sync()
        assert second.calls == 0  # 未变更块直接复用磁盘向量
        assert store2.search("alpha", top_k=1)


class TestSingleton:
    def test_get_code_store_is_per_workspace_and_reused(self, tmp_path):
        a = get_code_store(str(tmp_path))
        assert a is get_code_store(str(tmp_path))
        assert a is not get_code_store(str(tmp_path / "other"))
```

- [ ] **Step 2: 运行确认失败**

Run: `python -m pytest tests/test_code_store.py -q`
Expected: `ModuleNotFoundError: No module named 'gitai.agent_core.retrieval.code_store'`

- [ ] **Step 3: 实现 `factory.py` 与 `chunker.py` 别名**

- `chunker.py`：在 `_FUNC_DEF_PATTERNS` 定义后加一行公开别名 `FUNC_DEF_PATTERNS = _FUNC_DEF_PATTERNS`（无行为变化，避免跨模块访问私有名）。
- `factory.py`：模块顶层 `from gitai.config import load_config`（供测测试 monkeypatch）、`from gitai.agent_core.retrieval.rerank import build_llm_reranker`。`build_engine(cfg=None, *, embedder=None, vectors=None, sparse=None)`：`cfg` 为 None 时 `try: cfg = load_config() except Exception: cfg = None`；全部字段用 `getattr(cfg, name, 默认)` 取（`enable_hybrid_retrieval=True`/`retrieval_recall_k=200`/`retrieval_max_per_file=2`/`rerank_with_llm=False`）；`reranker = build_llm_reranker(cfg) if rerank_with_llm else None`；返回 `HybridSearchEngine(embedder=embedder, vectors=vectors, sparse=sparse, reranker=reranker, enable_sparse=enable_hybrid, recall_k=recall_k, max_per_file=max_per_file)`。
- 确认 `HybridSearchEngine` 已把 `enable_sparse`/`recall_k`/`max_per_file`/`reranker`/`sparse`/`vectors` 存在同名公开属性上（Task 4 未存则补存，不改行为）。

- [ ] **Step 4: 实现 `code_store.py`**

结构与两个既有 store 同构（`threading.Lock`、模块级 `_instances: dict[str, CodeSearchStore]` + `_instance_lock` + `_default_workspace` + `setup_code_workspace`；测试会直接清 `_instances`，名字不得自创别名）。

**默认配置的取法**：`code_store.py` 顶层 `from gitai.config import load_config`；`CodeSearchStore.__init__(workspace, *, cfg=None, embedder=None)` 在 `cfg is None` 时 `try: cfg = load_config() except Exception: cfg = None`，所有字段一律 `getattr(cfg, name, 默认值)` 取。必须走这个模块级名字（而非局部 import），因为 Task 10 的工具层测试要 monkeypatch 它来压低降级阈值。

**eligible 判定**（顺序短路，全是噪声/自索引排除，无规模截断）：`rglob("*")` → 是文件 → `suffix.lower() in _SUPPORTED_EXTENSIONS`（从 `rag.indexer` 导入）→ 路径任一段不在 `_SKIP_DIRS`（同上）且不在 `{".git", ".gitai", "agent_workspace"}` → `cfg.exclude_patterns` 的每个模式用 `fnmatch` 匹配相对 posix 路径均不命中 → 不是 symlink（`p.is_symlink()` 与 `p.resolve()` 仍在工作区内双重判定）。

**不可解码判定**：`raw = p.read_bytes()`；`raw` 含 `b"\x00"`，或 `raw.decode("utf-8", errors="replace")` 中 `\ufffd` 占比 > 5% → 计入 `skipped_unreadable`，不入索引（IO 声明，非规模截断）；否则用 `errors="replace"` 的解码结果作文本。

**降级切块**：`len(raw) > cfg.code_index_degrade_file_bytes` 时不丢弃，而是：前 `cfg.code_index_degraded_head_bytes` 字节作为一块 + 每行匹配 `FUNC_DEF_PATTERNS` 的行拼块（每 40 行一块，保证块数有界）；metadata `{"file_path": rel, "chunk_index": i, "degraded": True, "text": ...}`；块 id 用 `f"{rel}#d{i}"` 以免与正常块 `f"{rel}#{i}"` 冲突；降级文件相对路径逐个进 `degraded` 列表。

**sync 算法**：1) 收集 eligible；2) 读 manifest（`<ws>/.gitai/code_index.json`）与 sidecar（`code_index_vectors.json`，仅 `embedder.backend == "litellm"` 时读写）；3) 逐文件：`(mtime, size)` 与 manifest 相同且本进程已建过 → `unchanged_files`；相同但未建过 → 从磁盘文本重切分，`engine.add(id, text, meta, vector=sidecar.get(content_hash))`（sidecar 命中则不走 embedder）；不同/新增 → 先 `engine.remove` 该 path 旧块再写入 → `updated_files`/`added_files`；4) manifest 有而磁盘无 → 删其全部块 + 条目 → `removed_files`；5) 写回 manifest（`{path: {mtime, size, chunks: [{id, chunk_index, content_hash}]}}`，`content_hash = sha1(text).hexdigest()[:16]`）与 sidecar（`{content_hash: [floats]}`）；6) `coverage = indexed_files / eligible_files`（eligible=0 计 1.0）；`elapsed` 保留 2 位小数。

**内存优先**：`self._built_here: bool` 标记本进程是否已建 dense；`search()` 未建则先 `ensure_synced()`。`clear()` 返回清理块数并删除两个产物文件。`status()` 不扫盘（快），但用 manifest 条目数给出 `indexed_files`/`chunks`/`coverage`/`degraded`/`built`。

`__init__.py` 追加导出 `build_engine`、`CodeSearchStore`、`get_code_store`、`setup_code_workspace`。

- [ ] **Step 5: 运行确认通过**

Run: `python -m pytest tests/test_code_store.py -q`
Expected: 全绿（≥ 15 passed；符号链接用例在无权限平台上可 `skipped`）

- [ ] **Step 6: 提交**

```bash
git add gitai/agent_core/retrieval/ gitai/agent_core/rag/chunker.py tests/test_code_store.py
git commit -m "feat(retrieval): full-coverage code index with degraded chunking for oversized files"
```

---

## Task 8: 两个自包含适配器接入内核（doc_rag / history_rag）
