Metadata-Version: 2.3
Name: codesleuth
Version: 0.1.0
Summary: CodeSleuth is a local-first code search and retrieval tool optimized for LLM integration
Keywords: code search,semantic search,code retrieval,local-first,llm integration,mlx,faiss,embeddings,tree-sitter,developer tools,code understanding,code intelligence,code assistant,apple silicon,arm optimization,vector search,semantic indexing
Author: Jack Nam
Author-email: naminalstack+codesleuth@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Dist: faiss-cpu (>=1.10.0,<2.0.0)
Requires-Dist: loguru (>=0.7.3,<0.8.0)
Requires-Dist: numpy (>=2.2.4,<3.0.0)
Requires-Dist: pydantic (>=2.11.0,<3.0.0)
Requires-Dist: torch (>=2.6.0,<3.0.0)
Requires-Dist: tqdm (>=4.67.1,<5.0.0)
Requires-Dist: transformers (>=4.50.2,<5.0.0)
Requires-Dist: tree-sitter (>=0.24.0,<0.25.0)
Requires-Dist: tree-sitter-language-pack (>=0.6.1,<0.7.0)
Description-Content-Type: text/markdown

# CodeSleuth

CodeSleuth is a local-first code search and retrieval tool optimized for LLM integration. It creates semantic embeddings for code snippets and provides fast, efficient search capabilities without requiring external services or cloud dependencies.

## Key Features

### 🚀 Local-First Architecture

-   **Zero External Dependencies**: All processing happens locally on your machine
-   **No Cloud Services**: Your code never leaves your system
-   **Fast Indexing**: Optimized for large codebases
-   **Efficient Storage**: Compact index format for quick loading

### 🧠 LLM-Optimized

-   **Semantic Search**: Find code based on natural language descriptions
-   **Context-Aware Results**: Perfect for LLM code generation and analysis
-   **Natural Language Queries**: Search using everyday language
-   **Code Understanding**: Built for LLM code comprehension tasks

### ⚡ Performance Optimizations

-   **MLX Integration**: Native support for Apple Silicon (M1/M2/M3)
-   **Smart Index Selection**:
    -   FAISS with HNSW for x86/AMD architectures
    -   Optimized FAISS for ARM processors
-   **Efficient Embedding**: Fast code chunk processing
-   **Memory-Efficient**: Smart chunking and caching

### 🔍 Search Capabilities

-   **Semantic Search**: Find code by meaning, not just text
-   **Lexical Search**: Precise text-based search with regex support
-   **Function Definition Search**: Quickly locate function and method definitions
-   **Reference Search**: Find all usages of a symbol
-   **File Search**: Search for files by name or pattern

### 🌐 Language Support

Supports a wide range of programming languages including:

-   Python
-   JavaScript/TypeScript
-   Java
-   C/C++
-   PHP
-   Go
-   Rust
-   And more...

## Installation

```bash
pip install codesleuth
```

## Quick Start

```python
from codesleuth import CodeSleuth
from codesleuth.config import CodeSleuthConfig, EmbeddingModel

# Initialize CodeSleuth with your repository path
config = CodeSleuthConfig(repo_path="/path/to/your/repo")
codesleuth = CodeSleuth(config)

# Index your repository (uses MLX on Apple Silicon, FAISS otherwise)
codesleuth.index_repository()

# Check if semantic search is available
if codesleuth.is_semantic_search_available():
    # Search for code semantically (perfect for LLM integration)
    results = codesleuth.semantic_search.search(
        "authentication service implementation",
        top_k=5,
        similarity_threshold=0.7
    )
else:
    # Fall back to lexical search if semantic search isn't available
    results = codesleuth.lexical_search.search(
        "authentication service",
        max_results=5
    )

# Use with your LLM
for result in results:
    print(f"Found relevant code in {result['file_path']}:")
    print(result['code'])
```

## Configuration

CodeSleuth automatically optimizes for your hardware:

```python
from codesleuth.config import CodeSleuthConfig, ParserConfig, IndexConfig, EmbeddingModel

config = CodeSleuthConfig(
    repo_path="/path/to/repo",
    parser=ParserConfig(
        chunk_size=100,
        chunk_overlap=20,
        ignore_patterns=["node_modules/*", "dist/*"]
    ),
    index=IndexConfig(
        model_name=EmbeddingModel.BGE_M3,  # Uses MLX on Apple Silicon
        dimension=1024,
        use_mlx=True,  # Automatically use MLX on Apple Silicon
        use_gpu=False,  # Set to True to use GPU if available
        batch_size=32,  # Adjust for your memory constraints
        hnsw_m=16,  # Number of connections per node in HNSW index
        hnsw_ef_construction=100,  # Search depth during construction
        hnsw_ef_search=64  # Search depth during search
    ),
    search=SearchConfig(
        max_results=10,
        min_similarity=0.5,
        max_grep_results=50
    )
)
```

## Advanced Usage

### LLM Integration

```python
# Example with an LLM
from codesleuth import CodeSleuth
from your_llm import LLM

codesleuth = CodeSleuth(config)
llm = LLM()

# Check semantic search availability
if codesleuth.is_semantic_search_available():
    # Search for relevant code
    results = codesleuth.semantic_search.search(
        "implement user authentication with JWT",
        top_k=3
    )
else:
    # Fall back to lexical search
    results = codesleuth.lexical_search.search(
        "user authentication JWT",
        max_results=3
    )

# Use the results with your LLM
context = "\n".join(result["code"] for result in results)
response = llm.generate(f"Based on this code:\n{context}\n\nImplement a similar authentication system.")
```

### Performance Tuning

```python
# Optimize for your specific use case
config = CodeSleuthConfig(
    repo_path="/path/to/repo",
    parser=ParserConfig(
        chunk_size=150,  # Larger chunks for better semantic understanding
        chunk_overlap=30,  # More overlap for better context
    ),
    index=IndexConfig(
        model_name=EmbeddingModel.BGE_M3,
        dimension=1024,
        use_mlx=True,  # Use MLX on Apple Silicon
        batch_size=64,  # Larger batch size for faster processing
        hnsw_m=32,  # More connections for better recall
        hnsw_ef_construction=200,  # Higher quality index
        hnsw_ef_search=50  # Balance between speed and accuracy
    ),
    search=SearchConfig(
        max_results=20,  # More results for better coverage
        min_similarity=0.6,  # Stricter similarity threshold
        max_grep_results=100  # More grep results for better context
    )
)
```

## Architecture

CodeSleuth is built with performance in mind:

1. **Code Parsing**: Uses Tree-sitter for fast, accurate code parsing
2. **Embedding Generation**:
    - MLX on Apple Silicon for native performance
    - Optimized FAISS on other architectures
3. **Index Storage**: Efficient binary format for quick loading
4. **Search**: HNSW-based similarity search for fast results

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

## License

This project is licensed under the MIT License - see the LICENSE file for details.

