Metadata-Version: 2.4
Name: ai-repo-mapper
Version: 1.4.0
Summary: A lightweight AST-driven repository mapping tool with security, reliability, and export formats
Author: AI Repo Mapper Contributors
License-Expression: MIT
Project-URL: Homepage, https://github.com/siso-kh/project_summeriser
Project-URL: Repository, https://github.com/siso-kh/project_summeriser
Project-URL: Issues, https://github.com/siso-kh/project_summeriser/issues
Keywords: code-analysis,ast,tree-sitter,repository,mapping
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Documentation
Classifier: Topic :: Software Development :: Quality Assurance
Requires-Python: >=3.8
Description-Content-Type: text/markdown
Requires-Dist: pathspec>=0.12.0
Requires-Dist: tree-sitter>=0.21.0
Requires-Dist: tree-sitter-python>=0.21.0
Requires-Dist: tree-sitter-javascript>=0.21.0
Requires-Dist: tree-sitter-typescript>=0.21.0
Requires-Dist: tree-sitter-go>=0.21.0
Requires-Dist: tree-sitter-html>=0.21.0
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: tiktoken>=0.5.0; extra == "dev"

# AI Repo Mapper

[![PyPI version](https://badge.fury.io/py/ai-repo-mapper.svg)](https://pypi.org/project/ai-repo-mapper/)
[![Downloads](https://static.pepy.tech/badge/ai-repo-mapper)](https://pypi.org/project/ai-repo-mapper/)
[![Python](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![GitHub](https://img.shields.io/badge/GitHub-siso--kh%2Fproject__summeriser-black)](https://github.com/siso-kh/project_summeriser)

A lightweight, AST-driven repository mapping tool designed to scan codebases, parse source code files, and extract structured signatures (classes, functions, and key declarations) without relying on fragile regular expressions.

## Table of Contents
- [Overview](#overview)
- [Architecture & Project Structure](#architecture--project-structure)
- [Features](#features)
- [Prerequisites](#prerequisites)
- [Installation & Deployment](#installation--deployment)
- [Quick Start: Testing the App & UX](#quick-start-testing-the-app--ux)
- [Usage](#usage)
- [CLI Arguments](#cli-arguments)
- [Export Formats](#export-formats)
- [Using as a System-Wide Tool](#using-as-a-system-wide-tool)
- [Publishing to PyPI](#publishing-to-pypi-make-available-to-all-users)
- [Running Tests](#running-tests)
- [Performance](#performance)
- [Future Improvements](#future-improvements)

---

## Overview

**AI Repo Mapper** addresses the challenge of understanding large or unfamiliar codebases by extracting a high-level representation of code structure. Utilizing [Tree-sitter](https://tree-sitter.github.io/tree-sitter/) for robust Abstract Syntax Tree (AST) parsing, it generates accurate function and class signature maps while strictly respecting `.gitignore` rules and path exclusion configurations.

---

## Architecture & Project Structure

The project is structured into clear modular components:

```
ai-repo-mapper/
├── main.py              # CLI entry point parsing user arguments
├── map_repo.py          # Repository mapper orchestrator with export formats
├── scanner.py           # Gitignore loader and path matching logic via pathspec
├── parser_engine.py     # Tree-sitter AST extraction engine (Python, JS, TS, Go, HTML)
├── token_counter.py     # Token estimation for LLM context planning
├── config.json          # Configuration for ignored dirs and supported extensions
└── tests/
    ├── run_test.py          # Original test runner with mock snippets
    ├── test_fixes.py        # Unit tests for bug fixes (39 tests)
    ├── test_integration.py  # Integration tests for CLI workflow (53 tests)
    ├── test_security.py     # Security tests (20 tests)
    ├── test_reliability.py  # Reliability tests (17 tests)
    ├── test_ux.py           # UX tests (21 tests)
    └── test_benchmarks.py   # Performance benchmarks with regression thresholds (24 tests)
```

---

## Features

- **Multi-Language AST Parsing**: Precise extraction of code signatures using Tree-sitter parsers for Python, JavaScript, TypeScript, Go, and HTML.
- **Smart Filtering**: Automatically respects `.gitignore` rules and custom exclusion directives using `pathspec`.
- **Multiple Export Formats**: Output maps as Markdown (default), JSON, or YAML.
- **Smart Import Resolution**: `--dump-smart` follows imports and extracts only the referenced symbols.
- **Token Counting**: Estimates token count for LLM context planning.
- **Robust Error Handling**: Graceful handling of invalid config, syntax errors, binary files, and Unicode content.
- **Security Features**: Path traversal protection, symlink detection, file size limits, binary detection, and automatic secret redaction.
- **Performance Benchmarks**: Built-in benchmark suite with regression thresholds to catch slowdowns.
- **Reliability**: Structured logging, config validation, deterministic output, and graceful error handling.
- **User Experience**: Progress bars, color output, consistent exit codes, and timing info.
- **Comprehensive Testing**: 174 tests covering unit, integration, security, reliability, UX, and performance scenarios.

---

## Prerequisites

- **Python**: Version 3.8 or higher.
- **C Compiler**: Required by `tree-sitter` for building/loading language grammars on some platforms.

---

## Installation

### Option 1: Install from GitHub (Recommended)
```bash
pip install git+https://github.com/siso-kh/project_summeriser.git
```

### Option 2: Install via PyPI (when published)
```bash
pip install ai-repo-mapper
```

### Option 3: Install from source (for development)
```bash
git clone https://github.com/siso-kh/project_summeriser.git
cd project_summeriser
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -e .
```

### Option 4: Manual installation
```bash
git clone https://github.com/siso-kh/project_summeriser.git
cd project_summeriser
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
pip install -r requirements.txt
```

---

## Quick Start: Testing the App & UX

Follow these steps to install, run, and verify the tool works correctly on your machine.

### Step 1: Clone & Install

```bash
# Clone the repository
git clone https://github.com/siso-kh/project_summeriser.git
cd project_summeriser

# Create a virtual environment (recommended)
python -m venv venv

# Activate it
# On macOS/Linux:
source venv/bin/activate
# On Windows:
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt
```

### Step 2: Verify Installation

```bash
# Check that the CLI works
python main.py --help

# You should see:
# usage: repo-mapper [-h] [--output OUTPUT] [--format {markdown,json,yaml}]
#                    ...

# Check the version
python main.py --version
# Output: repo-mapper 1.1.0
```

### Step 3: Scan a Directory (Basic Test)

```bash
# Scan the current directory (the project itself)
python main.py

# Expected output:
# Scanning target directory: ./
# Found XX valid files. Starting AST mapping...
# [Success] Processed mapping saved to: codebase_map.md
# [Info] Estimated token count: XXX Tokens

# Open the output file to verify
# On macOS:  open codebase_map.md
# On Linux:  xdg-open codebase_map.md
# On Windows: start codebase_map.md
```

### Step 4: Test Export Formats

```bash
# Test JSON export
python main.py --format json --output test_map.json
cat test_map.json | head -20
# Should show valid JSON with "metadata" and "files" keys

# Test YAML export
python main.py --format yaml --output test_map.yaml
cat test_map.yaml | head -20
# Should show valid YAML with metadata and files sections

# Clean up test files
rm -f test_map.json test_map.yaml codebase_map.md
```

### Step 5: Test Security Features

```bash
# Test path traversal protection (should be blocked)
mkdir -p /tmp/test_escalation
echo 'SECRET = "password123"' > /tmp/test_escalation/secret.py
python main.py /tmp/test_escalation --output /tmp/test_out.md
# Should work, but secret should be redacted in output
grep -o 'REDACTED' /tmp/test_out.md && echo 'Secret was redacted!'

# Test symlink detection
ln -sf /etc/passwd test_link.txt 2>/dev/null || true
python main.py --output test_out.md
grep -c 'test_link.txt' test_out.md  # Should be 0 (symlink skipped)

# Clean up
rm -rf /tmp/test_escalation /tmp/test_out.md test_link.txt test_out.md
```

### Step 6: Test Verbose Logging

```bash
# Run with verbose output
python main.py --verbose
# Should show DEBUG-level log messages with timestamps
```

### Step 7: Test Error Handling

```bash
# Test with a file path instead of directory (should fail gracefully)
touch test_file.txt
python main.py test_file.txt 2>&1
# Expected: error message about file instead of directory, exit code 1

echo "Exit code: $?"  # Should print: Exit code: 1

# Test with invalid arguments
python main.py --invalid-flag 2>&1
# Expected: usage message, exit code 2

echo "Exit code: $?"  # Should print: Exit code: 2

# Clean up
rm -f test_file.txt
```

### Step 8: Test on a Real Project

```bash
# Scan a well-known open-source project (e.g., requests, flask)
pip install requests  # If not installed
python main.py $(python -c "import requests; print(requests.__path__[0])")

# Or scan any Python project you have
dir="path/to/your/other/project"
python main.py "$dir" --output my_project_map.md --format json
cat my_project_map.md | head -30

rm -f my_project_map.md
```

### Step 9: Run the Automated Test Suite

```bash
# Navigate to tests directory
cd tests

# Run each test suite one by one
python run_test.py          # Original AST tests (2 files)
python test_fixes.py        # Unit tests (39 tests)
python test_integration.py  # Integration tests (53 tests)
python test_security.py     # Security tests (20 tests)
python test_reliability.py  # Reliability tests (17 tests)
python test_ux.py           # UX tests (21 tests)
python test_benchmarks.py   # Performance benchmarks (24 tests)

# All should print: "All tests passed! ✅"
```

### Step 10: Verify Exit Codes

```bash
# Success (exit code 0)
python main.py .
echo "Exit code: $?"

# Bad arguments (exit code 2)
python main.py --bogus 2>/dev/null
echo "Exit code: $?"

# Expected output:
# Exit code: 0
# Exit code: 2
```

---

### Quick Test Checklist

| # | Test | What to Look For | Pass? |
|---|------|-------------------|-------|
| 1 | `python main.py --help` | Shows usage info | ☐ |
| 2 | `python main.py` | Creates `codebase_map.md` | ☐ |
| 3 | `python main.py --format json` | Creates valid JSON file | ☐ |
| 4 | `python main.py --format yaml` | Creates valid YAML file | ☐ |
| 5 | `python main.py --verbose` | Shows debug logs | ☐ |
| 6 | `python main.py --version` | Shows `repo-mapper 1.1.0` | ☐ |
| 7 | `python main.py <file>` | Error message, exit code 1 | ☐ |
| 8 | `python main.py --bogus` | Error message, exit code 2 | ☐ |
| 9 | Secret in code → output | API keys are redacted | ☐ |
| 10 | `cd tests && python test_ux.py` | 21/21 tests pass | ☐ |

---

## Usage

After installing (`pip install ai-repo-mapper`), you can use either command:

```bash
# Both work identically:
repo_mapper          # underscore
repo-mapper          # hyphen

# Or run as a module:
python -m ai_repo_mapper
```

### Examples

```bash
# Scan current directory, output to codebase_map.md
repo_mapper

# Scan specific directory
repo_mapper /path/to/project

# Custom output file
repo_mapper --output my_map.md

# Export as JSON
repo_mapper --format json --output map.json

# Export as YAML
repo_mapper --format yaml --output map.yaml

# Dump all source files
repo_mapper --dump all

# Dump only programming files
repo_mapper --dump logic

# Dump specific files
repo_mapper --dump-files main.py scanner.py

# Smart dump - follows imports
repo_mapper --dump-smart main.py

# Show version
repo_mapper --version
```

**Note:** You can also run directly with `python main.py` if installed from source.

---

## CLI Arguments

| Argument | Description | Default |
|----------|-------------|---------|
| `target_dir` | Directory to scan | `./` |
| `--output` | Output file path | `codebase_map.md` |
| `--format` | Output format: `markdown`, `json`, `yaml` | `markdown` |
| `--dump` | Dump mode: `none`, `logic`, `all` | `none` |
| `--dump-files` | Specific files to dump completely | `[]` |
| `--dump-smart` | Target file + auto-dump its imports | `[]` |
| `--follow-symlinks` | Follow symbolic links (security risk) | `false` |
| `--max-file-size` | Maximum file size in MB | `10` |
| `--no-redact` | Disable automatic secret redaction | `false` |
| `--verbose` | Enable verbose debug logging | `false` |

---

## Export Formats

### Markdown (default)
```bash
python main.py --output map.md
```
Produces a human-readable Markdown file with code signatures.

### JSON
```bash
python main.py --format json --output map.json
```
Produces structured JSON with metadata and file entries:
```json
{
  "metadata": {
    "total_files": 13,
    "generated_by": "AI Repo Mapper"
  },
  "files": [
    {
      "path": "./main.py",
      "extension": ".py",
      "signatures": ["def main():"],
      "source": null
    }
  ]
}
```

### YAML
```bash
python main.py --format yaml --output map.yaml
```
Produces YAML without requiring PyYAML dependency.

---

## Using as a System-Wide Tool

After installing via `pip install ai-repo-mapper`, both commands are available globally:

```bash
repo_mapper    # Use underscore
repo-mapper    # Or use hyphen
```

### Quick Access from Any Directory

```bash
# From any project directory, just run:
repo_mapper

# This will scan the current directory and create codebase_map.md
```

### Windows Users (PowerShell/CMD)

```powershell
# After pip install, just type:
repo_mapper

# Or:
repo-mapper
```

### macOS/Linux Users

```bash
# After pip install, just type:
repo_mapper

# Or add to your shell profile for autocomplete:
echo 'alias rm="repo_mapper"' >> ~/.bashrc
```

---

## Security

AI Repo Mapper includes built-in security features:

| Feature | Description |
|---------|-------------|
| **Path Traversal Protection** | Prevents accessing files outside the target directory |
| **Symlink Detection** | Skips symbolic links by default (use `--follow-symlinks` to enable) |
| **File Size Limits** | Skips files larger than `--max-file-size` (default: 10MB) |
| **Binary Detection** | Automatically skips binary files |
| **Secret Redaction** | Detects and redacts API keys, passwords, and tokens in output |

```bash
# Enable symlink following (use with caution)
repo-mapper --follow-symlinks /path/to/project

# Set custom file size limit
repo-mapper --max-file-size 5 /path/to/project

# Disable secret redaction
repo-mapper --no-redact /path/to/project
```

---

## Reliability

AI Repo Mapper ensures reliability through:

| Feature | Description |
|---------|-------------|
| **Structured Logging** | Uses Python `logging` module with timestamps and log levels |
| **Config Validation** | Validates config.json schema on load, falls back to defaults |
| **Deterministic Output** | Same input always produces same output (for caching) |
| **Graceful Error Handling** | Never crashes, always provides useful error messages |
| **Verbose Mode** | Use `--verbose` for detailed debug output |

```bash
# Enable verbose logging
repo-mapper --verbose /path/to/project

# Log levels: DEBUG, INFO, WARNING, ERROR
# Default level: INFO
# Verbose level: DEBUG
```

---

## Running Tests

```bash
# Run all tests
cd tests
python run_test.py          # Original AST extraction tests
python test_fixes.py        # Unit tests for bug fixes (39 tests)
python test_integration.py  # Integration tests for CLI workflow (53 tests)
python test_security.py     # Security tests (20 tests)
python test_reliability.py  # Reliability tests (17 tests)
python test_ux.py           # UX tests (21 tests)
python test_benchmarks.py   # Performance benchmarks with regression thresholds (24 tests)

# Run from project root
python -c "import sys; sys.path.insert(0, '.'); from tests.run_test import setup_and_execute_tests; setup_and_execute_tests()"

# Run benchmarks with regression checks (exits with code 1 if regression detected)
cd tests && python test_benchmarks.py
```

---

## User Experience

AI Repo Mapper provides a polished CLI experience:

| Feature | Description |
|---------|-------------|
| **Progress Bars** | Visual progress indicator for large codebases |
| **Color Output** | Color-coded terminal output for better readability |
| **Exit Codes** | Consistent exit codes (0=success, 1=error, 2=bad args) |
| **Timing Info** | Displays scan duration and file count in stderr |

```bash
# Color output is enabled by default in terminal
repo-mapper /path/to/project

# Disable colors (for piping or non-TTY)
NO_COLOR=1 repo-mapper /path/to/project
```

---

## Performance

Benchmarks are included in `tests/test_benchmarks.py` with regression thresholds:

| Component | Throughput |
|-----------|------------|
| Scanner | ~300 files/sec (with security checks) |
| Parser | ~200 files/sec |
| Full Pipeline (500 files) | ~8 seconds |
| Token Counter | ~1.4M tokens/sec |

To run benchmarks:
```bash
cd tests && python test_benchmarks.py
```

---

## Architecture Decisions

| Decision | Rationale |
|----------|-----------|
| Tree-sitter over regex | Robust AST parsing, handles edge cases |
| pathspec for gitignore | Native gitignore syntax support |
| Structured logging | Observability without cluttering stdout |
| Security by default | Symlinks blocked, secrets redacted |
| Deterministic output | Same input = same output (for caching) |

---

## Future Improvements

- **Additional Language Support**: Expand Tree-sitter integrations to support Rust, C++, and Java.
- **Parallel Processing**: Implement multi-process scanning for large monorepos.
- **Incremental Scanning**: Only scan changed files for faster re-scans.
- **Caching**: Cache AST parsing results for repeated runs.
- **CI/CD Integration**: GitHub Actions for automated testing on every commit.
- **AI Integration**: Support feeding extracted repository maps into LLM context windows for code analysis.

---

## Publishing to PyPI (Make Available to All Users)

To make this tool installable by anyone via `pip install ai-repo-mapper`, follow these steps:

### Step 1: Install Build Tools

```bash
pip install build twine
```

### Step 2: Create a PyPI Account

1. Go to [https://pypi.org](https://pypi.org)
2. Create an account (use a dedicated email)
3. Enable 2FA for security

### Step 3: Build the Package

```bash
# From the project root (where pyproject.toml is)
python -m build
```

This creates two files in `dist/`:
```
dist/
├── ai_repo_mapper-1.1.0.tar.gz    # Source distribution
└── ai_repo_mapper-1.1.0-py3-none-any.whl  # Wheel (pre-built)
```

### Step 4: Upload to TestPyPI (Recommended First)

```bash
# Test on TestPyPI first
twine upload --repository testpypi dist/*

# Test installation from TestPyPI
pip install --index-url https://test.pypi.org/simple/ ai-repo-mapper

# Verify it works
repo-mapper --version
```

### Step 5: Upload to Real PyPI

```bash
# Upload to real PyPI
twine upload dist/*

# You'll be prompted for your PyPI username and password
```

### Step 6: Verify Public Installation

```bash
# Now anyone can install it!
pip install ai-repo-mapper

# Verify
repo-mapper --version
# Output: repo-mapper 1.1.0

# Or run as module
python -m ai_repo_mapper --help
```

### Step 7: Add Badges to README

Add these badges to the top of your README:

```markdown
[![PyPI version](https://badge.fury.io/py/ai-repo-mapper.svg)](https://pypi.org/project/ai-repo-mapper/)
[![Downloads](https://static.pepy.tech/badge/ai-repo-mapper)](https://pypi.org/project/ai-repo-mapper/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
```

---

### Publishing Checklist

| # | Task | Status |
|---|------|--------|
| 1 | `pyproject.toml` has correct metadata | ☐ |
| 2 | Version matches in `__init__.py` and `pyproject.toml` | ☐ |
| 3 | `python -m build` succeeds | ☐ |
| 4 | Test on TestPyPI | ☐ |
| 5 | Upload to real PyPI | ☐ |
| 6 | `pip install ai-repo-mapper` works | ☐ |
| 7 | `repo-mapper --help` shows usage | ☐ |
| 8 | Add badges to README | ☐ |

---

### Updating the Package

When you release new versions:

```bash
# 1. Update version in pyproject.toml and __init__.py
# 2. Build new version
python -m build

# 3. Upload (PyPI doesn't allow reusing version numbers)
twine upload dist/*

# 4. Clean up old builds
rm -rf dist/ build/ *.egg-info/
```

---

## Contributing

1. Fork the repository
2. Create a feature branch (`git checkout -b feature/amazing-feature`)
3. Run the test suite (`cd tests && python run_test.py`)
4. Commit your changes (`git commit -m 'Add amazing feature'`)
5. Push to the branch (`git push origin feature/amazing-feature`)
6. Open a Pull Request

---

*Note: This project was implemented with the help of AI (Gemini) and Codebuff.*
