Metadata-Version: 2.4
Name: codecompass-ai
Version: 0.1.0
Summary: AI-powered repository onboarding assistant
License: MIT
Requires-Python: >=3.10
Requires-Dist: gitpython>=3.1.0
Requires-Dist: lancedb>=0.6.0
Requires-Dist: langchain-core>=0.3.0
Requires-Dist: langchain-ollama>=1.0.1
Requires-Dist: langgraph>=0.2.0
Requires-Dist: ollama>=0.5.4
Requires-Dist: pyarrow>=15.0.0
Requires-Dist: pydantic-settings>=2.0.0
Requires-Dist: pydantic>=2.0.0
Requires-Dist: python-dotenv>=1.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: tree-sitter-javascript>=0.21.0
Requires-Dist: tree-sitter-python>=0.21.0
Requires-Dist: tree-sitter>=0.21.0
Requires-Dist: typer>=0.9.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
Requires-Dist: pytest>=7.0.0; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# CodeCompass
AI-Powered Repository Onboarding Assistant

## Setup

## RAG

### Retrieval Strategy Evaluation

Evaluated retrieval strategies on 19 test queries across 5 categories (feature search, concept search, API search, debug search, natural language).

#### Results

| Strategy | Recall@5 | Precision@5 | MRR |
|----------|----------|-------------|-----|
| **HyDE** | **0.733** | 0.295 | 0.671 |
| Query Expansion | 0.718 | 0.295 | 0.737 |
| Baseline (vector search) | 0.644 | 0.263 | 0.754 |
| Query Expansion + Context | 0.428 | 0.189 | 0.399 |

**HyDE** (Hypothetical Document Embedding) performed best, improving recall by **13.9%** over baseline. This approach generates hypothetical code matching the query, then searches for similar real code.

#### Context-Aware Expansion: Negative Result

The context-aware strategy (providing repo imports to the LLM) performed surprisingly poorly. I hypothesized the prompt was suboptimal, so I tested 4 variations:

| Variant | Description | Recall@5 |
|---------|-------------|----------|
| v4: Rule-based | No LLM, keyword matching | 0.691 |
| v2: Pick from list | LLM selects relevant imports | 0.665 |
| v3: Minimal prompt | Simple prompt | 0.644 |
| v1: Fewer imports | 15 imports instead of 100 | 0.600 |
| Original | 100 imports, verbose prompt | 0.428 |

**Finding**: Even the best context variant (rule-based, no LLM) underperformed simple query expansion. Adding repository context introduces noise rather than helping retrieval.

**Takeaway**: Simpler strategies (HyDE, query expansion) outperform complex context-aware approaches for code search.

## Finetuning

I considered finetuning with xlam-60k from salesforce but after running an tool-calling evaluation script on base model with curated dataset, it already does a good job. So we only finetune for better code explanation.

Considred datasets like CodeSearchNet, code_x_glue_ct_code_to_text but found 
Magicoder-OSS-Instruct-75K to have the best detailed yet concise explanations suitable for our case.
Since we expect user queries to be more abstract questions where code explanation is not the only response to the query, in fact it's only for a subset of most queries, I decided to use a two-pass architecture with hot-swap adapaters, using only the fine-tuned model for explanation portion so we don't see performance degradation for other tasks.

│  User Query                                                  │
│      ↓                                                       │
│  [Base Model] → Tool selection (95% accurate, no training)  │
│      ↓                                                       │
│  [Tools] → Code chunks                                       │
│      ↓                                                       │
│  [Fine-tuned Model] → Explanations only ← TRAIN THIS ONLY   │
│      ↓                                                       │
│  [Base Model] → Final synthesized answer  

And the user shouldn't be downloading two full qwen models so I decided to fine tune with LoRA.