
## Executive Summary

ast-tools is in **strong shape** post-external-review. All 5 fixes (A-E) verified correct. Primary risk is **path traversal** in file-path-accepting tools. Test suite has 6 failures due to missing dependency (`tree_sitter_python`), not code bugs. Overall coverage at 50% — acceptable for a tool-heavy project but room for improvement.

**Overall Health: 7.5/10** (up from 6.8 pre-external-review)

---

## ✅ Strengths

| Area | Status | Notes |
|------|--------|-------|
| External Review Fixes | ✅ All 5 correct | Fixes A-E verified in code |
| Subprocess Safety | ✅ Clean | Zero `shell=True`, all list args |
| Secrets | ✅ Clean | Zero hardcoded credentials |
| Import Hygiene | ✅ Clean | All packages have `__init__.py` |
| Dead Code | ✅ Minimal | Only 2 TODOs in production code |
| Lint | ⚠️ Minor | 40 non-W293 issues (ARG001, I001, etc.) |
| Test Pass Rate | 98.4% | 367/373 passed (6 failures = missing dep) |

---

## 🔴 CRITICAL Findings

None found. No immediate security vulnerabilities or data loss risks.

---

## 🟠 HIGH Findings

### H1: Path Traversal in File-Path-Accepting Tools

**Location:** Multiple tools  
**Risk:** An LLM providing malicious `file_path` parameters could read/write files outside the declared `project_path`.

**Affected Tools:**
| Tool | File | Line | Issue |
|------|------|------|-------|
| `ast_read` | `tools/ast_read.py` | 16 | `Path(args["file"]).resolve()` — no containment check |
| `ast_edit` | `tools/ast_edit.py` | 130 | Same pattern |
| `ast_generate_stub` | `tools/ast_generate_stub.py` | 12 | Same pattern |
| `ast_refactor_extract_interface` | `tools/ast_refactor_extract_interface.py` | 12 | Same pattern |
| `code_validate` | `tools/code_validate.py` | 48 | Same pattern |

**Current Code (ast_read.py:16):**
```python
file_path = Path(args["file"]).resolve()
# No check: file_path.is_relative_to(project_path)
```

**Recommended Fix:**
```python
file_path = Path(args["file"]).resolve()
project_path = Path(args.get("project_path", ".")).resolve()
if not file_path.is_relative_to(project_path):
    return {"error": "Path outside project boundary", "error_code": "PATH_TRAVERSAL"}
```

**Effort:** Low (5 tools × 3 lines each)  
**Risk Mitigation:** HIGH — prevents LLM agent from escaping sandbox

---

## 🟡 MEDIUM Findings

### M1: Test Dependency Gap (6 failures)

**Location:** `tests/test_project_tools.py`, `tests/test_phase3_polish.py`  
**Issue:** `tree_sitter_python` not installed in test environment

**Failing Tests:**
1. `test_ast_read_syntax_error` — test bug (expects `"error"`, tool returns `"parse_error"`)
2. `test_ts_parse_python` — missing `tree_sitter_python`
3. `test_ts_grep_function_definition` — missing `tree_sitter_python`
4. `test_ts_grep_class_definition` — missing `tree_sitter_python`
5. `test_ts_read_extracts_functions` — missing `tree_sitter_python`
6. `test_ts_read_extracts_classes` — missing `tree_sitter_python`

**Fix:** Add `tree_sitter_python` to `pyproject.toml` dev-dependencies AND fix test assertion.

### M2: Inconsistent Error Key in `ast_read` Test

**Location:** `tests/test_phase3_polish.py:243`  
**Issue:** Test asserts `"error" in result` but tool returns `"parse_error"` for syntax errors. The tool's behavior is intentional (distinguishes file I/O errors from parse errors). Test is outdated.

**Fix:** Update test to check `"parse_error" in result or "error" in result`.

### M3: Watcher Test Coverage (19%)

**Location:** `src/ast_tools/tools/watcher.py`  
**Issue:** Filesystem event-driven code is hard to test in CI. Core logic (symbol extraction, impact analysis) has better coverage but watcher daemon itself is undertested.

**Impact:** Low — watcher is auxiliary, not critical path.

### M4: Oversized Files

**Files >500 lines:**
| File | Lines | Recommendation |
|------|-------|----------------|
| `code_validate.py` | 704 | Split by language (Python/TS/Go/Rust validators) |
| `extractor.py` | 698 | Extract language-specific extractors |
| `queries.py` | 596 | Split by concern (symbols/dependencies/search) |
| `semantic_search.py` | 475 | Borderline — monitor |
| `lsp_tools.py` | 430 | Acceptable |

**Effort:** Medium (2-3 hours for `code_validate.py` split)  
**Impact:** Maintainability, not functionality

---

## 🟢 LOW Findings

### L1: Lint Issues (40 non-whitespace)

**Breakdown:**
- `ARG001` (unused function args): 28 occurrences — mostly in tool handler signatures (MCP interface requires specific signature)
- `ARG002` (unused method args): 1 occurrence
- `I001` (import sorting): 1 file (`ast_query.py`)
- `F541` (f-string without placeholders): 1 occurrence
- `E741` (ambiguous variable name `l`): 1 occurrence
- `SIM102` (nested if): 1 occurrence
- `RUF001` (unicode): 1 occurrence
- `W292` (no newline at EOF): 2 files

**Note:** Most `ARG001` are false positives — MCP tool handlers must accept `args: dict` even if unused.

### L2: Module-Level Mutable State

**Location:** `tools/watcher.py:6`
```python
_active_daemon: WatcherDaemon | None = None
```
**Risk:** State leaks across sessions in long-running MCP server. Low risk for local use.

### L3: `secret_sanitizer.py` — 0% Coverage

**Location:** `src/ast_tools/utils/secret_sanitizer.py`  
**Issue:** 77 lines, zero tests. Critical security utility should have tests.

---

## 📊 Metrics Dashboard

| Metric | Value | Target | Status |
|--------|-------|--------|--------|
| **Test Pass Rate** | 98.4% (367/373) | 100% | 🟡 (6 failures = missing dep) |
| **Overall Coverage** | 50% | 70%+ | 🟡 |
| **Lint Errors** | 40 (non-W293) | 0 | 🟡 |
| **Dead Code** | None detected | — | ✅ |
| **Secrets** | 0 | 0 | ✅ |
| **Subprocess Safety** | 100% list args | 100% | ✅ |
| **Oversized Files** | 3 (>500 lines) | 0 | 🟡 |
| **TODOs** | 2 | 0 | ✅ |

---

## 🛠️ Prioritized Action Plan

### Immediate (This Week)
1. **H1: Path traversal fix** — Add `is_relative_to()` checks to 5 tools
   - Effort: 30 min | Risk: HIGH mitigation

2. **M1: Install test dependency** — Add `tree_sitter_python` to dev deps
   - Effort: 5 min | Impact: 5 tests fixed

3. **M2: Fix test assertion** — Update `test_ast_read_syntax_error`
   - Effort: 2 min | Impact: 1 test fixed

### Short Term (This Month)
4. **M4: Split `code_validate.py`** — Extract per-language validators
   - Effort: 2-3 hours | Impact: Maintainability

5. **L3: Add `secret_sanitizer.py` tests**
   - Effort: 1 hour | Impact: Security confidence

6. **L1: Clean up lint issues** — Fix I001, F541, E741, SIM102, RUF001
   - Effort: 15 min | Impact: Code quality

### Ongoing
7. **M3: Watcher coverage** — Add integration tests with `pyfakefs` if instability observed
8. **M4: Split `extractor.py` and `queries.py`** — When touching those files next

---

## ✅ Verification Evidence

- Forward audit: All 5 external review fixes verified in source code
- Reverse audit: No dead code, no missing `__init__.py`, no secrets
- Adversarial audit: No `shell=True`, no `eval`/`exec`, no injection vectors found
- Bug review: No silent error swallowing, no connection leaks, no race conditions
- Lint: `ruff check` run, 40 non-whitespace issues documented
- Tests: Full suite run, 367 passed / 6 failed (all failures diagnosed)

---

## Conclusion

ast-tools is **production-ready** with the path traversal fix applied. The external review fixes were correctly implemented. The test suite is reliable once `tree_sitter_python` is added to dev dependencies. The main improvement opportunities are splitting oversized files and adding tests for `secret_sanitizer.py`.

**Recommendation:** Apply H1 (path traversal) + M1 (test dep) + M2 (test fix) as a single commit. Defer M4 (file splits) to next maintenance window.


################################################################################
ROADMAP
################################################################################


================================================================================
FILE: docs/roadmap/ROADMAP.md
================================================================================

# AST-Tools Ecosystem — Comprehensive Roadmap

> **Version:** 1.0.0  
> **Author:** Lucien (Lead Digital Architect, RapidWebs Enterprise)  
> **Date:** 2026-07-31  
> **Status:** Draft — Pending ADRs, Phase Audits, and Final Sign-off  
> **License:** MIT  

---

## Table of Contents

1. [Executive Summary](#executive-summary)
2. [Vision Statement](#vision-statement)
3. [Core Architecture Principles](#core-architecture-principles)
4. [Project Inventory — Current State](#project-inventory--current-state)
5. [Full Feature Taxonomy](#full-feature-taxonomy)
6. [Monetization & Distribution Model](#monetization--distribution-model)
7. [Implementation Phases Overview](#implementation-phases-overview)
8. [ADR Index](#adr-index)
9. [Planning Documents](#planning-documents)

---

## Executive Summary

AST-Tools is a structural code analysis and editing platform built on the Model Context Protocol (MCP). It provides 43+ tools for semantic code search, AST-based refactoring, dependency analysis, impact assessment, and index management. Currently at v0.1.0 (Beta), the project has a working MCP server, 17+ CLI tools, 3 Hermes plugins, a SQLite+vector database backend, and a CI/CD pipeline with GitHub Actions publishing to PyPI.

**The gap:** The project exists as a powerful MCP server but lacks the infrastructure, ecosystem, and productization layers needed for production-grade adoption, monetization, and multi-platform agent integration. This roadmap defines the complete path from "functional MCP server" to "self-sustaining code intelligence platform."

**Target state:** A fully self-contained code intelligence ecosystem with:
- Multi-platform agent integration (Hermes, Gemini CLI, Claude Code, Qwen Code)
- Web dashboard for configuration, insights, and administration
- Tiered SaaS/self-hosted monetization
- Data lifecycle management (init → doctor → curator → vacuum → backup/restore → uninstall)
- Knowledge graph support
- SDK for programmatic consumption
- Multi-machine distributed deployment support

---

## Vision Statement

AST-Tools will become the **universal code intelligence layer** — an open-source, agent-agnostic platform that any AI agent (Hermes, Claude Code, Gemini CLI, Qwen Code, etc.) can plug into for deep structural understanding of any codebase.

**Core identity:** We are an infrastructure layer, not an application. Our job is to parse, index, analyze, and serve code intelligence to any consumer that speaks our protocol.

**Why this matters:** Every coding agent needs to understand code structure. Currently, every agent builds its own partial solution (Claude Code has grep+regex, Gemini CLI has its own index, Codex has a different one). AST-Tools provides a standardized, open-source, and comprehensive solution that any agent can adopt.

**The 10-year ambition:** A codebase knowledge graph that spans millions of open-source repositories — a "Google for code structure" that agents query instead of rebuilding indexes from scratch.

---

## Core Architecture Principles

1. **Agent-Agnostic Protocol:** All functionality exposed via MCP. No Hermes-specific dependencies in core. The Hermes plugins, SKILL.md files, and CLI extensions are adapters that wrap the MCP protocol.

2. **Data Lifecycle First:** Every piece of data must have an init → maintain → curator → vacuum → backup → cleanup → uninstall lifecycle. No orphaned data, no stale indexes, no unbounded database growth.

3. **Layered Monetization:** Free tier (CLI + MCP + raw stats) → Paid tier (formatted reports, PDF, docs) → Pro tier (dashboard, multi-machine, advanced analytics). Never degrade the free tier; always add value on top.

4. **Self-Healing Infrastructure:** Doctor command diagnoses and fixes common issues. Curator daemon prunes stale data. Vacuum reclaims space. Backup/restore provides safety nets. The system should survive machine failures gracefully.

5. **Privacy by Default:** PII redaction during curation, optional encryption for backups, no telemetry without opt-in. On-premise first, SaaS optional.

6. **Distributed by Design:** The index format and query protocol must support multi-machine deployments from day one. A single SQLite database scales to ~1M symbols; beyond that, the architecture should support sharding and federation.

---

## Project Inventory — Current State

### What Exists (v0.1.0)

| Component | Status | Detail |
|-----------|--------|--------|
| **MCP Server** | ✅ Complete (43 tools) | 17 core + 26 utility/analysis/index tools |
| **CLI** | ✅ Complete | `ast-tools-server`, `ast-tools-project`, `ast-tools` entry points |
| **Hermes Plugins** | ✅ 3 shipped | ast-tools-context, ast-tools-tokens, ast-tools-codebase-index |
| **SQLite Database** | ✅ Schema v5 | Symbols, embeddings, edges, dependency metrics, audit_log |
| **FTS5 Search** | ✅ <10ms | Full-text search over indexed symbols |
| **Vector Search** | ✅ <50ms | sqlite-vec with sentence-transformers (bge-small-en-v1.5) |
| **6-Factor RRF Fusion** | ✅ <100ms hybrid | Semantic (40%) + recency (15%) + usage (15%) + kind (10%) + proximity (10%) + callgraph (10%) |
| **CI/CD Pipeline** | ✅ GitHub Actions | Lint → test (matrix 3.10-3.13) → build → publish PyPI |
| **Pre-commit hooks** | ✅ Configured | `.pre-commit-config.yaml` with ruff |
| **Tests** | ✅ 461+ tests | Pytest suite with coverage across 33+ files |
| **Build** | ✅ Wheel + sdist | Hatchling build, published to PyPI |
| **Docs** | ⚠️ Partial | Quickstart, CLI reference, usage rules, troubleshooting exist; user-facing docs incomplete |
| **Knowledge Graph** | ⚠️ Foundation | `knn_builder.py` exists, KNN graph built from embeddings, but no formal KG query layer |
| **Curator** | ⚠️ Partial | `curator/daemon.py` exists but not production-ready |
| **File Watcher** | ⚠️ Partial | `watcher/daemon.py` exists, watchdog-based, reindex dispatch stubbed |

### What's Missing (Full Gap Analysis)

#### Infrastructure & Data Lifecycle
- [ ] `~/.ast-tools/` config directory with `config/tokens.yaml`
- [ ] First-time setup wizard (db init, model download, index creation)
- [ ] Doctor command (healthcheck: db integrity, model loaded, index consistent)
- [ ] Maintenance commands (vacuum, curation, pruning, dedup, cleanup)
- [ ] Shutdown/uninstall logic (cleanup config, db, caches)
- [ ] Data backup/restore (local + remote, incremental, optional encryption)
- [ ] PII redaction during curator runs
- [ ] Logging with rotation

#### Agent Ecosystem
- [ ] Bundled SKILL.md files (cross-platform: Hermes, Claude Code, Gemini CLI, Qwen Code, etc.)
- [ ] Official Hermes plugins (3 shipped, need maintenance + 1 more: project-context)
- [ ] Gemini CLI extension
- [ ] Claude Code extension
- [ ] Qwen Code extension
- [ ] VS Code extension (MCP-based)

#### SDK & API
- [ ] Python SDK (programmatic consumption of code intelligence)
- [ ] TypeScript SDK (for web dashboard and Node.js consumers)
- [ ] REST API (optional HTTP gateway alongside MCP)

#### Knowledge Graph
- [ ] Formal knowledge graph query layer (beyond KNN)
- [ ] Cross-repository symbol resolution
- [ ] Graph traversal API (neighbors, paths, clusters)
- [ ] Concept extraction (what does this codebase "do"?)

#### Deployment & Operations
- [ ] Docker image
- [ ] docker-compose.yml (server + dashboard)
- [ ] Systemd service file
- [ ] Multi-machine / distributed server support
- [ ] Healthcheck endpoint

#### Monetization & Productization
- [ ] Tiered feature gating
- [ ] Report generation (markdown → docx → PDF → HTML)
- [ ] Web dashboard (Tailwind + React + shadcn/ui)
- [ ] SaaS deployment option
- [ ] License key / subscription management

---

## Full Feature Taxonomy

### Core Infrastructure (Foundation Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| CORE-01 | Config directory (`~/.ast-tools/`) | P0 | 0 | Standardized config file structure |
| CORE-02 | tokens.yaml | P0 | 0 | Token budget, context length, threshold config |
| CORE-03 | Logging framework | P0 | 0 | Structured logging with rotation |
| CORE-04 | Error handling standardization | P0 | 0 | Consistent error codes across all tools |

### Data Lifecycle (Operations Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| DATA-01 | Setup wizard | P0 | 1 | Interactive first-time setup (db init, model download, index creation) |
| DATA-02 | Doctor command | P0 | 1 | Healthcheck: db integrity, model loaded, index consistent, dependencies |
| DATA-03 | Vacuum | P1 | 1 | SQLite vacuum + reindex, space reclamation |
| DATA-04 | Curation daemon | P1 | 1 | Stale data pruning, orphaned symbol cleanup, dedup |
| DATA-05 | Cleanup command | P1 | 1 | Remove temporary files, stale indexes, expired caches |
| DATA-06 | Deduplication engine | P2 | 3 | Content-hash based dedup with confidence scoring |
| DATA-07 | Uninstall logic | P0 | 3 | Clean removal of all artifacts (config, db, caches, logs) |

### SDK & API (Consumption Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| SDK-01 | Python SDK | P1 | 2 | Programmatic consumption of code intelligence |
| SDK-02 | REST API gateway | P2 | 3 | HTTP gateway alongside MCP |
| SDK-03 | TypeScript SDK | P2 | 4 | For web dashboard and Node.js consumers |

### Knowledge Graph (Intelligence Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| KG-01 | Knowledge graph query layer | P1 | 2 | Formal KG queries beyond KNN (neighbors, paths, clusters) |
| KG-02 | Cross-repo symbol resolution | P2 | 4 | Resolve symbols across repository boundaries |
| KG-03 | Graph traversal API | P2 | 4 | Breadth-first, depth-first, shortest path traversals |
| KG-04 | Concept extraction | P2 | 5 | High-level understanding of codebase purpose |

### Agent Ecosystem (Integration Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| AGENT-01 | SKILL.md cross-platform bundle | P0 | 0 | Platform-agnostic skill files for Hermes, Claude Code, etc. |
| AGENT-02 | Hermes plugin maintenance | P1 | 1 | Update plugins: config-driven tokens.yaml, project-context |
| AGENT-03 | Gemini CLI extension | P2 | 4 | Gemini CLI adapter for ast-tools tools |
| AGENT-04 | Claude Code extension | P2 | 4 | Claude Code adapter |
| AGENT-05 | Qwen Code extension | P3 | 5 | Qwen Code adapter |
| AGENT-06 | VS Code extension | P3 | 5 | MCP-based VS Code extension |

### Deployment & Operations (Infrastructure Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| OPS-01 | Docker image | P1 | 2 | Production Docker image |
| OPS-02 | docker-compose.yml | P1 | 2 | Server + optional dashboard compose |
| OPS-03 | Systemd service | P1 | 2 | Service file for persistent operation |
| OPS-04 | Multi-machine support | P2 | 4 | Distributed server deployment |
| OPS-05 | Healthcheck endpoint | P1 | 2 | MCP healthcheck for orchestration |

### Backup & Disaster Recovery (Safety Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| BACKUP-01 | Local backup | P1 | 3 | Archive + compress indexes and config |
| BACKUP-02 | Remote backup | P2 | 3 | S3/SFTP/rsync remote backup |
| BACKUP-03 | Incremental backup | P2 | 3 | Only changed data since last backup |
| BACKUP-04 | Encryption | P2 | 3 | Optional GPG/AES encryption for backups |
| BACKUP-05 | Restore command | P1 | 3 | Full + selective restore from backup |

### Reporting & Insights (Value Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| REPORT-01 | Text/CSV/JSON stats | P1 | 2 | Raw codebase insights (free tier) |
| REPORT-02 | Markdown reports | P2 | 3 | Human-readable markdown insight reports (paid) |
| REPORT-03 | DOCX reports | P3 | 4 | Word document format (paid) |
| REPORT-04 | PDF reports | P3 | 4 | PDF generation (paid) |
| REPORT-05 | HTML dashboard | P2 | 3 | Tailwind + React + shadcn/ui dashboard (pro) |
| REPORT-06 | Custom report builder | P3 | 5 | User-configurable report content and layout |

### Privacy & Security (Trust Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| SEC-01 | PII redaction in curator | P1 | 1 | Redact emails, keys, passwords from db during curator runs |
| SEC-02 | Backup encryption | P2 | 3 | AES-256-GCM encryption for backup artifacts |
| SEC-03 | Audit logging | P1 | 0 | Structured audit log for all destructive operations |
| SEC-04 | Configuration validation | P1 | 0 | Schema validation for all config files |

### Dashboard & Administration (Management Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| DASH-01 | Web dashboard (local) | P2 | 3 | Local web UI for configuration, insights, administration |
| DASH-02 | SaaS deployment | P3 | 5 | Cloud-hosted dashboard with multi-tenant support |
| DASH-03 | Usage analytics | P2 | 4 | Index health, query frequency, storage trends |

### Monetization Infrastructure (Revenue Layer)

| ID | Feature | Priority | Phase | Description |
|----|---------|----------|-------|-------------|
| MON-01 | Tier gating | P3 | 4 | Feature access control by tier |
| MON-02 | License key system | P3 | 4 | Offline license validation |
| MON-03 | Subscription management | P3 | 5 | Stripe integration for SaaS |
| MON-04 | Usage metering | P3 | 5 | API call tracking for metered billing |

---

## Monetization & Distribution Model

### Tier Structure

| Tier | Price (est.) | Access | Features |
|------|-------------|--------|----------|
| **Free** | $0 | MCP server, CLI, all core tools | Raw stats (text/CSV/JSON), single machine, community support |
| **Team** | $29/mo | Everything free + | Markdown reports, docx/pdf export, backup/restore, email support |
| **Pro** | $49/mo | Everything team + | Web dashboard, advanced analytics, multi-machine, dedicated support |
| **Enterprise** | Custom | Everything pro + | SaaS deployment, SSO, SLA, on-premise license, custom integrations |

### Distribution Channels

| Channel | Tier | Purpose |
|---------|------|---------|
| PyPI | Free | pip install ast-tools |
| Docker Hub | Free/Team | docker pull rapidwebs/ast-tools |
| GitHub Releases | Free | Source + pre-built wheels |
| npm | Free | @rapidwebs/ast-tools-sdk (TypeScript) |
| SaaS Portal | Pro/Enterprise | Managed dashboard + API |

---

## Implementation Phases Overview

| Phase | Name | Focus | Effort (est.) | Dependencies |
|-------|------|-------|---------------|-------------|
| **0** | Foundation & Configuration | Config directory, tokens.yaml, logging, audit log, SKILL.md bundle | 1 week | None |
| **1** | Data Lifecycle & Operations | Setup wizard, doctor, vacuum, curator, PII redaction | 2 weeks | Phase 0 |
| **2** | SDK, Knowledge Graph, Docker | Python SDK, KG query layer, Docker image, systemd | 3 weeks | Phase 1 |
| **3** | Backup, Reporting, Dashboard | Backup/restore, insights, HTML dashboard, uninstall | 3 weeks | Phase 2 |
| **4** | Agent Ecosystem & Multi-Machine | Gemini/Claude Code extensions, distributed support, analytics | 3 weeks | Phase 3 |
| **5** | Monetization & Advanced Features | Tier gating, SaaS, pro dashboard, concept extraction | 4 weeks | Phase 4 |

**Total estimated: 16 weeks (4 months)**

See `docs/roadmap/phases/` for detailed phase implementation plans.

---

## ADR Index

Key Architecture Decision Records are at `docs/roadmap/adrs/`:

| ADR | Title | Status |
|-----|-------|--------|
| ADR-001 | Config Directory Structure | Draft |
| ADR-002 | Data Lifecycle Architecture | Draft |
| ADR-003 | Monetization vs Open Source Boundary | Draft |
| ADR-004 | Knowledge Graph Storage Format | Draft |
| ADR-005 | Multi-Platform Agent Strategy | Draft |
| ADR-006 | Backup & Encryption Architecture | Draft |

---

## Planning Documents

Located at `docs/roadmap/planning/`:

| Document | Purpose |
|----------|---------|
| SHORT_TERM.md | Next 4 weeks — Phases 0-1 |
| LONG_TERM.md | 4-16 week outlook — Phases 2-5 |
| RISK_REGISTER.md | Identified risks and mitigation strategies |
| DEPENDENCY_MAP.md | Cross-phase dependency graph |
| MILESTONE_TIMELINE.md | Gantt-style timeline view |

---

*This roadmap is a living document. Phase details evolve as audits complete and user feedback is incorporated.*

================================================================================
FILE: docs/roadmap/planning/LONG_TERM.md
================================================================================

# Long-Term Plan — 4-16 Weeks (Phases 2-5)

> **Focus:** SDK, Knowledge Graph, Docker, Backup, Reporting, Dashboard, Agent Ecosystem, Multi-Machine, Monetization  
> **Timeline:** Phases 2-5, ~12 weeks total  
> **Prerequisites:** Phases 0-1 complete  

---

## Phase 2: SDK, Knowledge Graph, Docker (3 weeks)

### Prerequisites
- [x] Config directory (`~/.ast-tools/`) established
- [x] Data lifecycle commands operational (init, doctor, vacuum, curator)
- [x] Logging and audit infrastructure

### Deliverables
- [ ] Python SDK package (`ast-tools-sdk`)
- [ ] Knowledge graph query layer
- [ ] Docker image (`rapidwebs/ast-tools`)
- [ ] docker-compose.yml (server + optional dashboard)
- [ ] Systemd service file (`.service`)

### Key Decisions Needed
- SDK API surface: what's included vs MCP-only
- Knowledge graph: formal query API design
- Docker image: alpine vs slim, model bundling strategy

### Risk
- SDK API may change as MCP tools evolve — version pinning required
- Docker image with embedding model is ~1.5GB (model + dependencies) — CI build time may exceed 30min limits

---

## Phase 3: Backup, Reporting, Dashboard (3 weeks)

### Prerequisites
- [x] Python SDK published
- [x] Docker image available
- [x] Systemd service operational

### Deliverables
- [ ] Local backup/restore
- [ ] Remote backup (S3 backend)
- [ ] Incremental backup
- [ ] Encryption (AES-256-GCM)
- [ ] Text/CSV/JSON insights reporting (free tier)
- [ ] Markdown reporting (paid tier)
- [ ] Deduplication engine
- [ ] Uninstall command
- [ ] HTML dashboard (Tailwind + React + shadcn/ui) — local deployment

### Key Decisions Needed
- Backup retention policy (number of backups, age-based pruning)
- Dashboard tech stack confirmed (React vs vanilla, Tailwind CDN vs bundled)
- Report generation library (markdown→PDF: pandoc? weasyprint? wkhtmltopdf?)

### Risk
- HTML dashboard requires Node.js — adds build step complexity
- PDF generation is notoriously fragile — extensive testing needed

---

## Phase 4: Agent Ecosystem & Multi-Machine (3 weeks)

### Prerequisites
- [x] Backup/restore operational
- [x] Dashboard deployed
- [x] Uninstall command available

### Deliverables
- [ ] Gemini CLI extension
- [ ] Claude Code integration (CLAUDE.md + tuck)
- [ ] Multi-machine / distributed server support
- [ ] DOCX/PDF report generation (paid tier)
- [ ] Cross-repository symbol resolution
- [ ] Graph traversal API (BFS, DFS, shortest path)
- [ ] Usage analytics

### Key Decisions Needed
- Multi-machine architecture: SQLite shared via network FS vs separate DBs with merge
- Cross-repo resolution: how to match symbols across repos (by name? hash? canonical path?)
- Analytics: privacy-first (local-only) vs optional telemetry

### Risk
- Multi-machine is the highest-complexity feature in the roadmap — may need earlier prototyping
- Cross-repo resolution without an external symbol server is limited

### Dependency
- ADR-004 (Knowledge Graph Format) execution must be complete

---

## Phase 5: Monetization & Advanced Features (4 weeks)

### Prerequisites
- [x] Agent ecosystem operational
- [x] Multi-machine support available
- [x] Dashboard deployed

### Deliverables
- [ ] Tier gating system (free/team/pro/enterprise)
- [ ] License key system (offline-capable)
- [ ] SaaS deployment (multi-tenant dashboard)
- [ ] Subscription management (Stripe integration)
- [ ] Usage metering
- [ ] Concept extraction (high-level codebase understanding)
- [ ] Custom report builder
- [ ] VS Code extension
- [ ] Qwen Code extension
- [ ] Pro dashboard (advanced analytics, multi-machine admin)

### Key Decisions Needed
- Licensing model finalized: MIT core + paid extensions (ADR-003)
- SaaS pricing ($29/$49/custom)
- Stripe vs Paddle vs self-managed for payment processing

### Risk
- Concept extraction quality is hard to predict — may need AI model fine-tuning
- SaaS adds significant operational overhead (billing, auth, support)
- Licensing enforcement without telemetry is challenging

---

## Post-Launch (16+ Weeks)

### Potential Directions
- **ast-tools Hub:** A community registry of indexed open-source projects
- **AST-Tools Cloud:** Fully managed SaaS for enterprises
- **CI/CD Integration:** Native GitHub Actions, GitLab CI, Jenkins plugins
- **IDE Extension Marketplace:** VS Code, JetBrains, Vim/Neovim
- **Language Expansion:** Full support for 50+ languages via tree-sitter
- **Code Review Integration:** Auto-generated PR summaries based on impact analysis

---

## Gantt Overview

```
Phase 0 |▓▓▓▓▓▓▓▓▓▓▓▓▓▓|  
Phase 1 |       ▓▓▓▓▓▓▓▓▓▓▓▓▓▓|  
Phase 2 |              ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓|  
Phase 3 |                           ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓|  
Phase 4 |                                       ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓|  
Phase 5 |                                                    ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓|
        └────Month 1────┴────Month 2────┴────Month 3────┴────Month 4────┘
```

================================================================================
FILE: docs/roadmap/planning/SHORT_TERM.md
================================================================================

# Short-Term Plan — Next 4 Weeks (Phases 0-1)

> **Focus:** Foundation, Configuration, Data Lifecycle, Operations  
> **Timeline:** 3 weeks Phase 0 + 2 weeks Phase 1 (can overlap 1 week)  
> **Total:** ~4 weeks

---

## Week 1-2: Phase 0 — Foundation & Configuration

### Deliverables
- [ ] `~/.ast-tools/` config directory with XDG support
- [ ] `~/.ast-tools/config/tokens.yaml` with JSON Schema validation
- [ ] Logging framework with rotation
- [ ] Audit log for destructive operations
- [ ] SKILL.md cross-platform bundle (Hermes, Claude Code, generic)
- [ ] ast-tools-tokens plugin updated to read tokens.yaml
- [ ] Config file validation command

### Tasks
1. **Config directory module** — `src/ast_tools/config/` with loader, validation, schema
2. **tokens.yaml schema** — Define: per-tool budgets, model context lengths, compression/warning thresholds
3. **Logging setup** — Structured JSON logging, log rotation (size+time based), `logs/` directory
4. **Audit trail** — All destructive ops logged to `audit.log` with timestamp, action, user, params
5. **Plugin update** — `ast-tools-tokens` reads from `~/.ast-tools/config/tokens.yaml` instead of hardcoded values, falls back to defaults
6. **SKILL.md bundle** — Platform-agnostic skill files documenting all 43 tools, install + usage instructions
7. **Config validation** — `ast-tools config validate` command

### Dependencies
- ADR-001 (Config Directory Structure) — finalized
- `src/ast_tools/utils/` — needs config loader module

### Risk
- Config file migration from `~/.cache/ast-tools/` may have edge cases
- Plugin backwards compatibility with hardcoded values during transition

---

## Week 3-4: Phase 1 — Data Lifecycle & Operations

### Deliverables
- [ ] Setup wizard (interactive + `--non-interactive` mode)
- [ ] Doctor command with health score
- [ ] Vacuum command
- [ ] Curation daemon with scheduled execution
- [ ] PII redaction in curator
- [ ] Cleanup command
- [ ] Hermes plugin maintenance (project-context plugin)

### Tasks
1. **Setup wizard** — `ast-tools init`: detect env, create config, download model, create initial index
2. **Doctor command** — `ast-tools doctor`: check db integrity, model presence, index consistency, config validity, dependency availability, output health score
3. **Vacuum command** — `ast-tools vacuum`: SQLite VACUUM + REINDEX, temp file cleanup, old log rotation
4. **Curator daemon** — `ast-tools curator`: prune stale symbols, dedup, cleanup orphans. Configurable schedule (cron expression). `ast-tools curator run` for one-shot execution
5. **PII redaction** — Add to curator: scan symbol names/comments for emails, API keys, tokens, file paths. Configurable action (redact/flag/remove)
6. **Cleanup command** — `ast-tools cleanup`: `cache/tmp/`, stale model variants, expired caches
7. **Plugin maintenance** — Update ast-tools-context to reference tokens.yaml thresholds. Create ast-tools-project-context plugin (injects project metadata on session start)

### Dependencies
- Phase 0 (config directory, logging, audit)
- ADR-002 (Data Lifecycle Architecture) — finalized

### Risk
- Model download can fail (large file, network interruption) — need resume support
- Curator daemon must not conflict with concurrent index operations (locking)

---

## Immediate Next Steps (Day 1-3)

1. Write Phase 0 draft implementation doc
2. Run forward + reverse + adversarial audits on Phase 0 draft
3. Finalize Phase 0 document
4. Begin Phase 0 implementation (config directory module first)
5. Begin Phase 1 draft and audit in parallel with Phase 0 implementation

## Key Decisions Needed

- [ ] Approve ADR-001 (Config Directory Structure)
- [ ] Approve ADR-002 (Data Lifecycle Architecture)
- [ ] Approve tokens.yaml schema design (per-tool budgets)
- [ ] Model download strategy: bundled vs download-on-init

================================================================================
FILE: docs/roadmap/planning/RISK_REGISTER.md
================================================================================

# Risk Register — AST-Tools Ecosystem Implementation

> **Date:** 2026-07-31  
> **Author:** Lucien  
> **Status:** Living document — updated as risks are identified or retired  

---

## Risk Scoring

| Score | Likelihood × Impact | Response |
|-------|---------------------|----------|
| 🔴 Critical | High × High | Must mitigate before proceeding |
| 🟠 High | Medium × High or High × Medium | Active mitigation required |
| 🟡 Medium | Low × High or Medium × Medium | Monitor, plan response |
| 🔵 Low | Low × Medium or Low × Low | Accept or defer |

---

## Phase 0 Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R0-01 | Config migration from `~/.cache/ast-tools/` breaks existing deployments | Medium | High | 🟠 High | Fallback: if `~/.ast-tools/` doesn't exist but `~/.cache/ast-tools/` does, migrate on first run. Document manual migration steps. |
| R0-02 | Plugin backwards compatibility breaks during tokens.yaml transition | Medium | High | 🟠 High | Ship both codepaths during transition: try tokens.yaml first, fall back to hardcoded defaults. Remove hardcoded path in Phase 2. |
| R0-03 | SKILL.md files become stale after tools are updated | High | Low | 🟡 Medium | Add SKILL.md generation to CI pipeline — auto-regenerate when tool schemas change. |

---

## Phase 1 Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R1-01 | Model download fails during setup wizard | Medium | High | 🟠 High | Add resume support (download to tmp, verify checksum). Offer `--skip-model` flag for offline environments. |
| R1-02 | Curator daemon conflicts with concurrent index operations | Medium | Medium | 🟡 Medium | Use SQLite WAL mode with retry logic. Curator acquires a write lock; other operations wait with timeout. |
| R1-03 | Doctor command has false negatives (reports health but system broken) | Low | High | 🟡 Medium | Exhaustive check list: db integrity, model responds, index returns results, config valid, dependencies available. |
| R1-04 | PII redaction produces false positives (flags legitimate code) | Medium | Medium | 🟡 Medium | Default to "flag for review" rather than "auto-redact". Configurable allowlist. |

---

## Phase 2 Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R2-01 | SDK API changes as MCP tools evolve — breaks consumers | High | Medium | 🟡 Medium | Version SDK in lockstep with MCP tools. Semantic versioning: breaking changes = major version bump. |
| R2-02 | Docker image exceeds CI build time limits (model = ~1.5GB) | High | High | 🟠 High | Multi-stage build: base image without model, model downloaded at runtime. Or use GitHub Actions with larger runners. |
| R2-03 | Knowledge graph queries are too slow on large codebases (>100K symbols) | Medium | High | 🟠 High | Benchmark with 100K symbol dataset. Optimize with indexes. Consider paginated query results. Document performance envelope. |

---

## Phase 3 Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R3-01 | Backup takes too long for large databases (>1GB) | Medium | Medium | 🟡 Medium | Default to database-only backup (skip models). Offer incremental mode. Show progress bar during backup. |
| R3-02 | PDF report generation is fragile across platforms | High | Medium | 🟡 Medium | Pin PDF generation tool version. Test across Linux/macOS/Windows. Use simple HTML→PDF (weasyprint) rather than complex layouts. |
| R3-03 | Encryption key management confuses users | Medium | Low | 🔵 Low | Clear CLI prompts with password strength meter. Document key recovery: "No password = no data. There is no backdoor." |

---

## Phase 4 Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R4-01 | Multi-machine support introduces consistency issues | High | High | 🔴 Critical | Start with shared-database model (NFS-backed SQLite). Document limitations. Only add distributed model if demand justifies. |
| R4-02 | Cross-repo symbol matching is unreliable | High | Medium | 🟠 High | Start with exact-match (symbol name + file path). Add fuzzy matching as optional enhancement. Document accuracy guarantees. |

---

## Phase 5 Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R5-01 | Licensing enforcement without telemetry is challenging | Medium | High | 🟠 High | Use offline license file validation (signed JWT). Periodic manual verification prompts. Accept that determined users can bypass — focus on honest users. |
| R5-02 | SaaS adds significant operational overhead | High | Medium | 🟠 High | Start with self-hosted only. SaaS only if demand justifies dedicated ops. Use existing infrastructure (Railway, Render, Fly.io) for MVP. |
| R5-03 | Concept extraction quality disappoints users | Medium | High | 🟠 High | Set expectations: "Experimental — accuracy may vary." Make it a configurable feature, not the headline deliverable. |

---

## Cross-Phase Risks

| ID | Risk | Likelihood | Impact | Score | Mitigation |
|----|------|-----------|--------|-------|------------|
| R0-01 | Feature creep — roadmap scope exceeds capacity | High | High | 🔴 Critical | Strict phase scoping. Each phase has a hard scope boundary. Features not in current phase go to "Icebox" backlog. Monthly reprioritization. |
| R0-02 | Single developer bottleneck (Lucien only) | High | High | 🔴 Critical | Prioritize self-documenting code and comprehensive CI. Write contribution guide (CONTRIBUTING.md) early. Build for handoff-readiness. |
| R0-03 | Test suite becomes brittle at >500 tests | Medium | Medium | 🟡 Medium | Invest in test infrastructure: conftest fixtures, test factories, flaky test detection, test tagging. |
| R0-04 | 4GB workstation RAM limits development velocity | High | Medium | 🟠 High | Offload heavy builds/tests to server (rapidwebs). Use targeted test runs instead of full suite. Consider GitHub Codespaces for model-dependent work. |

---

## Retired Risks

| ID | Risk | Retired | Reason |
|----|------|---------|--------|
| — | — | — | — |

################################################################################
DOCUMENT SIZE SUMMARY
################################################################################

7012 ast-tools-doc-review-context.txt
