Metadata-Version: 2.4
Name: graphmind-rj
Version: 0.1.2
Summary: Knowledge graph builder for code and product assets with token optimization for LLMs
Author-email: Ramesh J <ramesh@example.com>
Maintainer: Ramesh J
License-Expression: MIT
Project-URL: Homepage, https://github.com/rameshj/graphmind-rj
Project-URL: Repository, https://github.com/rameshj/graphmind-rj.git
Project-URL: Bug Tracker, https://github.com/rameshj/graphmind-rj/issues
Project-URL: Documentation, https://github.com/rameshj/graphmind-rj#readme
Keywords: knowledge-graph,code-analysis,llm,ast,context,token-optimization
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Software Development :: Build Tools
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: networkx>=3.2
Requires-Dist: fastapi>=0.111.0
Requires-Dist: uvicorn>=0.30.0
Requires-Dist: pydantic>=2.7.0
Requires-Dist: sqlalchemy>=2.0.30
Requires-Dist: python-docx>=1.1.2
Requires-Dist: python-multipart>=0.0.9
Provides-Extra: dev
Requires-Dist: pytest>=8.2.0; extra == "dev"
Requires-Dist: httpx>=0.27.0; extra == "dev"
Dynamic: requires-python

# graphmind-rj

> **Knowledge graph builder for code and product assets — with smart token optimization for LLMs.**
>
> Built by **Ramesh J** · [PyPI](https://pypi.org/project/graphmind-rj/) · Python 3.10+

[![PyPI version](https://img.shields.io/pypi/v/graphmind-rj)](https://pypi.org/project/graphmind-rj/)
[![Python](https://img.shields.io/pypi/pyversions/graphmind-rj)](https://pypi.org/project/graphmind-rj/)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

---

## What is graphmind-rj?

When you use AI assistants like Claude, ChatGPT, or Copilot to work on your codebase, they need **context** — which files exist, what functions do, how modules relate to each other. The problem:

- Pasting full files = **too many tokens = expensive and slow**
- Pasting random snippets = **AI gives incomplete answers**
- Repeating the same context every query = **wasteful and time-consuming**

**graphmind-rj solves this.** It:

1. Scans your codebase once and builds a **knowledge graph** — a structured map of every function, class, SQL table, and document heading and how they connect
2. Saves that graph locally as JSON and in a SQLite database (no server needed)
3. Exposes a `POST /api/context-pack` endpoint — call it before asking your AI, get back only the context relevant to your specific task

> **How you use it in practice**: run `graphmind-api` as a local server, run `graphmind run .` on your project once, then before each AI question call `/api/context-pack` with your question and task type. Copy the response (graph summary + code snippets + prompt template) and paste it into Claude, Copilot Chat, or Cursor. This typically reduces the context you send by 60–80% compared to pasting full files.

---

## Features

| Feature | What it does |
|---|---|
| 🔍 **Multi-language extraction** | Reads Python (via AST), SQL schemas, `.docx` documents, `.md`/`.txt`/`.rst` files |
| 📊 **Knowledge graph** | Builds a NetworkX graph of nodes and edges with community clustering |
| 🚀 **REST API** | FastAPI server with 6 endpoints — run, upload, history, graph, context-pack |
| 💾 **SQL persistence** | SQLAlchemy + SQLite stores every run, node, and edge for history and queries |
| 📄 **JSON export** | Exports `graph.json` (portable), `graph.html` (readable), `GRAPH_REPORT.md` |
| 🧠 **Token optimization** | 3-tier context: graph summary → code snippets → full files, selected by query type |
| 📋 **Prompt templates** | Pre-structured formats for bug_fix, feature, refactor, test, review, optimize |
| ⚡ **Incremental runs** | SHA-256 hashing skips unchanged files — only processes what changed |

---

## Installation

```bash
pip install graphmind-rj
```

Nothing else needed. SQLite is built into Python. No Docker, no database server, no config files required.

After install you get two commands:

```
graphmind        ← CLI for running the pipeline
graphmind-api    ← starts the REST API server on port 8000
```

Verify:

```bash
graphmind --help
```

```
usage: graphmind [-h] {run} ...

GraphMind Enterprise CLI

positional arguments:
  {run}
    run       Run full graph pipeline

options:
  -h, --help  show this help message and exit
```

---

## Architecture

Understanding how all the pieces connect makes the tool much easier to use.

```
YOUR PROJECT FILES
        │
        ▼
┌──────────────────────────────────────────────────────────┐
│                     PIPELINE                             │
│                                                          │
│  1. detect.py        Finds all files by extension        │
│                      Buckets: code / document / paper    │
│                      Skips: .git node_modules venv dist  │
│                                                          │
│  2. extractors/      Reads each file, pulls out nodes    │
│     python_ast.py    → functions, classes, methods       │
│     sql_schema.py    → tables, columns (CREATE TABLE)    │
│     text_semantic.py → headings, paragraphs (.md/.txt)   │
│     docx_semantic.py → headings, paragraphs (.docx)      │
│     registry.py      → merges all, removes duplicates    │
│                                                          │
│  3. graph/builder.py  Builds NetworkX in-memory graph    │
│     graph/analytics.py  Louvain community detection      │
│                          Hub ranking by degree           │
│                                                          │
│  4. exporters/        Writes output files                │
│     json_exporter.py → graphmind-out/graph.json          │
│     html_exporter.py → graphmind-out/graph.html          │
│     pipeline.py      → graphmind-out/GRAPH_REPORT.md     │
│                                                          │
│  5. db.py             Saves to SQLite                    │
│     RunRecord         → summary (files/nodes/edges)      │
│     NodeRecord        → every node linked to run_id      │
│     EdgeRecord        → every edge linked to run_id      │
│                                                          │
│  6. cache.py          Saves file hashes for next run     │
└──────────────────────────────────────────────────────────┘
        │
        ▼
┌──────────────────────────────────────────────────────────┐
│                     REST API (api.py)                    │
│                                                          │
│  POST /api/run           → triggers pipeline             │
│  POST /api/upload        → saves .md/.docx/.sql files    │
│  GET  /api/runs          → list run history from SQLite  │
│  GET  /api/runs/{id}/graph → nodes+edges for a run       │
│  POST /api/context-pack  → optimized LLM context         │
│  GET  /health            → server health check           │
│                                                          │
│  context-pack uses:                                      │
│    context_budget.py   → allocates token budget          │
│    retrieval_planner.py → picks tier (1/2/3)             │
│    prompt_templates.py → returns task prompt format      │
└──────────────────────────────────────────────────────────┘
        │
        ▼
┌──────────────────────────────────────────────────────────┐
│              FRONTEND (frontend/)                        │
│  React + Vite dashboard on http://localhost:5173         │
│  Talks to backend via /api proxy                         │
│  Panels: Run · Upload · History                          │
└──────────────────────────────────────────────────────────┘
```

---

## Files Scanned (what `detect.py` reads)

| Extension | Category | Extractor used |
|---|---|---|
| `.py` | code | `python_ast.py` |
| `.js`, `.ts`, `.go`, `.java` | code | (detected, extractor in roadmap) |
| `.sql` | code | `sql_schema.py` |
| `.md`, `.txt`, `.rst` | document | `text_semantic.py` |
| `.docx` | document | `docx_semantic.py` |
| `.pdf` | paper | (detected, extractor in roadmap) |

**Automatically skipped directories**: `.git`, `node_modules`, `venv`, `dist`, `build`

**Max file size**: 5 MB (configurable via `GovernancePolicy`)

---

## End-to-End Tutorial

### Step 1 — Start the API Server

Open a terminal and run:

```bash
graphmind-api
```

You will see:

```
INFO:     Started server process [28040]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```

Leave this terminal open. The `GET / 404 Not Found` message you may see is **normal** — the API has no root page. It only responds on specific paths like `/health`, `/api/run`, etc.

Open a **second terminal** for all next steps.

---

### Step 2 — Health Check

Confirm the server is running:

```bash
curl http://localhost:8000/health
```

Response:

```json
{"status": "ok"}
```

---

### Step 3 — Analyse Your Project

Navigate to any project folder and run:

```bash
graphmind run /path/to/your/project
```

Or from inside the project:

```bash
cd /path/to/your/project
graphmind run .
```

To force a full rescan (ignore the cache):

```bash
graphmind run . --full
```

Terminal output:

```
GraphMind complete. Files=42 Nodes=142 Edges=198 Communities=7
Artifacts: graphmind-out
```

**What gets created in `graphmind-out/`:**

| File | Format | Purpose |
|---|---|---|
| `graph.json` | JSON | Full graph — nodes, edges, community IDs. Share it, visualize it, feed it to other tools |
| `graph.html` | HTML | Human-readable table view — open in any browser |
| `GRAPH_REPORT.md` | Markdown | Summary: total files, words, nodes, edges, communities |
| `graphmind.db` | SQLite | Run history + all nodes + all edges, queryable via API |
| `cache/file_hashes.json` | JSON | SHA-256 hashes of every scanned file — used for incremental runs |

**Incremental mode (default)**: on the second `graphmind run .`, only files whose SHA-256 hash changed get re-extracted. If 2 files changed out of 42, only those 2 are processed. Saves significant time on large codebases.

---

### Step 4 — Run Pipeline via REST API

Same as Step 3 but triggered via HTTP — useful for scripts, CI, or the React frontend:

```bash
curl -X POST http://localhost:8000/api/run \
  -H "Content-Type: application/json" \
  -d '{"path": "/path/to/your/project", "full": false}'
```

| Field | Type | Default | Meaning |
|---|---|---|---|
| `path` | string | `"."` | Directory to scan |
| `full` | boolean | `false` | `true` = ignore cache, scan everything |

Response:

```json
{
  "run_id": 1,
  "files": 42,
  "words": 15600,
  "nodes": 142,
  "edges": 198,
  "communities": 7,
  "out_dir": "graphmind-out"
}
```

**`run_id` is important** — you use it in every follow-up API call to reference this specific run.

---

### Step 5 — Upload Standalone Files

Use this when you have files **outside your codebase** that you want in the graph — design documents, SQL migration files, requirements specs:

```bash
curl -F "files=@architecture.md" \
     -F "files=@migration_v2.sql" \
     -F "files=@requirements.docx" \
     http://localhost:8000/api/upload
```

Accepted: `.md`, `.docx`, `.sql` only. Other extensions are rejected.

Response:

```json
{
  "folder": "graphmind-out/uploads/20260408_143021",
  "files_saved": [
    "graphmind-out/uploads/20260408_143021/architecture.md",
    "graphmind-out/uploads/20260408_143021/migration_v2.sql",
    "graphmind-out/uploads/20260408_143021/requirements.docx"
  ]
}
```

Then run the pipeline on that upload folder to extract and store the graph:

```bash
curl -X POST http://localhost:8000/api/run \
  -H "Content-Type: application/json" \
  -d '{"path": "graphmind-out/uploads/20260408_143021", "full": true}'
```

---

### Step 6 — View Run History

Every pipeline run is saved to SQLite. List them:

```bash
curl http://localhost:8000/api/runs
```

Optional limit parameter:

```bash
curl "http://localhost:8000/api/runs?limit=5"
```

Response (newest first):

```json
[
  {
    "id": 2,
    "target_path": "/path/to/project",
    "files": 44,
    "words": 16200,
    "nodes": 155,
    "edges": 210,
    "communities": 8,
    "created_at": "2026-04-08T16:45:00"
  },
  {
    "id": 1,
    "target_path": "/path/to/project",
    "files": 42,
    "words": 15600,
    "nodes": 142,
    "edges": 198,
    "communities": 7,
    "created_at": "2026-04-08T09:12:05"
  }
]
```

You can see how the graph grew between runs as you added code.

---

### Step 7 — Inspect the Graph for a Run

Get every node and edge extracted during a specific run:

```bash
curl http://localhost:8000/api/runs/1/graph
```

Response:

```json
{
  "run_id": 1,
  "nodes": 142,
  "edges": 198,
  "graph": {
    "nodes": [
      {
        "id": "authenticate_user",
        "label": "authenticate_user",
        "kind": "function",
        "source_file": "auth.py"
      },
      {
        "id": "users",
        "label": "users",
        "kind": "table",
        "source_file": "schema.sql"
      }
    ],
    "edges": [
      {
        "source": "authenticate_user",
        "target": "check_password",
        "relation": "calls",
        "confidence": "high",
        "confidence_score": 0.97,
        "source_file": "auth.py"
      }
    ]
  }
}
```

Node `kind` values: `function`, `class`, `method`, `table`, `heading`, `concept`

Edge `relation` values: `calls`, `defines`, `contains`, `references`, `links`

---

### Step 8 — Get Optimized LLM Context (The Core Feature)

This is the endpoint you use before every AI query. Instead of pasting full files into Claude or Copilot, call this first:

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": 1,
    "task_type": "bug_fix",
    "query": "Why is user authentication failing after the latest deployment?",
    "token_budget": 6000,
    "include_artifacts": true
  }'
```

**Request fields:**

| Field | Type | Required | Description |
|---|---|---|---|
| `run_id` | int | ✅ | Which run to pull context from |
| `task_type` | string | ✅ | `bug_fix` / `feature` / `refactor` / `test` / `review` / `optimize` |
| `query` | string | ✅ | Your exact question — used to select retrieval tier |
| `token_budget` | int | optional | Max tokens to include. Default: `8000` |
| `include_artifacts` | boolean | optional | Include cached module summaries. Default: `true` |

**Response:**

```json
{
  "tier": "snippets",
  "graph_summary": {
    "nodes_count": 142,
    "edges_count": 198,
    "communities": 7,
    "top_hubs": ["UserService", "AuthModule", "DatabaseLayer", "SessionManager", "PasswordHasher"],
    "summary": "Graph with 142 nodes, 7 communities"
  },
  "code_snippets": [
    {
      "file_path": "auth.py",
      "start_line": 45,
      "end_line": 65,
      "content": "def authenticate_user(username, password):\n    return check_password(username, password)",
      "reason": "function_definition"
    }
  ],
  "prompt_template": {
    "task_type": "bug_fix",
    "name": "Bug Fix Investigation",
    "sections": [
      "## Problem",
      "## Current Behavior",
      "## Expected Behavior",
      "## Relevant Code",
      "## Test Case",
      "## Proposed Fix"
    ],
    "required_contexts": ["changed_files", "error_trace", "test_case"]
  },
  "cached_artifacts": {},
  "total_tokens": 2380,
  "recommended_next_steps": [
    "Run the failing test to reproduce",
    "Add breakpoints in the shown code",
    "Check git blame for recent changes"
  ]
}
```

**How to use the response — step by step:**

1. Copy `graph_summary` → paste it into your AI as: *"Here is the codebase overview"*
2. Copy `code_snippets[].content` → paste as: *"Here is the relevant code"*
3. Use `prompt_template.sections` as headers to structure your question
4. Follow `recommended_next_steps` as your debugging/development guide
5. Check `total_tokens` — that is exactly how many tokens you spent

---

### Step 9 — Task Types — Which to Use and When

#### `bug_fix` — Something is broken
Budget: `6000` · Tier used: `snippets`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -d '{"run_id":1, "task_type":"bug_fix", "query":"Login returns 500 after deploy", "token_budget":6000}'
```

Prompt sections returned: `Problem` · `Current Behavior` · `Expected Behavior` · `Relevant Code` · `Test Case` · `Proposed Fix`

---

#### `feature` — Adding new functionality
Budget: `8000`–`10000` · Tier used: `snippets` → `full_files`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -d '{"run_id":1, "task_type":"feature", "query":"Add OAuth2 Google login", "token_budget":10000}'
```

Prompt sections: `Feature Request` · `Requirements` · `Design Constraints` · `Integration Points` · `Implementation Steps` · `Success Criteria`

---

#### `refactor` — Restructuring existing code
Budget: `8000` · Tier used: `full_files`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -d '{"run_id":1, "task_type":"refactor", "query":"Split UserService into auth and profile services", "token_budget":8000}'
```

Prompt sections: `Current Structure` · `Problems` · `Target Structure` · `Safety Checks` · `Steps` · `Testing Strategy`

---

#### `test` — Writing tests
Budget: `4000` · Tier used: `graph_summary` → `snippets`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -d '{"run_id":1, "task_type":"test", "query":"Write unit tests for authenticate_user", "token_budget":4000}'
```

Prompt sections: `Code to Test` · `Test Objectives` · `Test Cases` · `Edge Cases` · `Execution Strategy`

---

#### `review` — Code review / security audit
Budget: `12000` · Tier used: `full_files`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -d '{"run_id":1, "task_type":"review", "query":"Review payment processing module for security issues", "token_budget":12000}'
```

Prompt sections: `Code Summary` · `Reviewed Files` · `Quality Checks` · `Security Scan` · `Performance Impact` · `Recommendations`

---

#### `optimize` — Performance improvements
Budget: `6000` · Tier used: `snippets`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -d '{"run_id":1, "task_type":"optimize", "query":"Database queries taking 3+ seconds", "token_budget":6000}'
```

Prompt sections: `Current Performance` · `Bottleneck Analysis` · `Optimization Options` · `Implementation Plan` · `Benchmarking Strategy`

---

### Step 10 — Start the React Dashboard (Optional)

The dashboard gives you a browser UI instead of curl commands:

```bash
cd frontend
npm install
npm run dev
```

Open: [http://localhost:5173](http://localhost:5173)

Panels:
- **Run panel**: type a directory path, click Run → triggers `POST /api/run`
- **Upload panel**: drag `.md`, `.docx`, `.sql` files → triggers `POST /api/upload`
- **History panel**: table of all past runs with files/nodes/edges metrics

The Vite dev server proxies `/api/*` to `http://localhost:8000` automatically.

---

## How the 3-Tier Token System Works

When you call `/api/context-pack`, the `retrieval_planner.py` module decides how much context to include:

```
Your query
    │
    ▼
Tier 1 — Graph Summary  (~500 tokens)
  Returns: node count, edge count, community count, top 5 hub nodes
  Used when: query is about architecture, overview, or system design
  Example query: "How is this system structured?"
    │
    │  Does the query contain: bug / debug / fix / error / integrate / review / refactor?
    ▼  AND do you have remaining budget?
Tier 2 — Code Snippets  (~1,500–3,000 tokens)
  Returns: tier 1 + function/class definitions from key files
  Used when: debugging, feature planning, targeted questions
  Example query: "Why is authentication failing?"
    │
    │  Is the task a complex refactor or full review?
    ▼  AND do you still have budget?
Tier 3 — Full Files  (~3,000–8,000 tokens)
  Returns: tier 1 + tier 2 + complete file content
  Used when: comprehensive rewrites, full security audits
  Example query: "Completely restructure the auth module"
```

**Token Budget Allocation** — within each tier, files are ranked by priority:

| Priority | Which files | Budget share |
|---|---|---|
| CRITICAL | Files that changed since last run | 50% |
| HIGH | Direct dependencies of changed files | 30% |
| MEDIUM | Tests and related modules | 15% |
| LOW | Documentation and config files | 5% |

Files are sorted smallest-first within each tier so the maximum number of files fits within budget.

---

## Data Storage — Exactly What Goes Where

| Location | Format | Written when | Contains |
|---|---|---|---|
| `graphmind-out/graph.json` | JSON | Every pipeline run | Full graph: nodes, edges, community IDs |
| `graphmind-out/graph.html` | HTML | Every pipeline run | Readable table of nodes and edges |
| `graphmind-out/GRAPH_REPORT.md` | Markdown | Every pipeline run | File counts, node/edge/community totals |
| `graphmind-out/graphmind.db` | SQLite | Every pipeline run (via API) | `runs`, `nodes`, `edges` tables |
| `graphmind-out/cache/file_hashes.json` | JSON | Every pipeline run | SHA-256 hash per file path |
| `graphmind-out/uploads/TIMESTAMP/` | original files | On upload | Your uploaded `.md`/`.docx`/`.sql` files |
| `graphmind-out/artifacts/` | JSON index + text | On context-pack call | Cached module summaries |

None of this data leaves your machine. Everything is local.

---

## Daily Workflow — VS Code / Cursor

```
Morning: start of coding session
─────────────────────────────────────────────────────
Terminal 1:  graphmind-api          ← leave running all day

Terminal 2:  cd my-project
             graphmind run .         ← takes ~3–10 seconds
                                       (incremental — only changed files)


When you hit a bug:
─────────────────────────────────────────────────────
curl -X POST http://localhost:8000/api/context-pack \
  -d '{
    "run_id": 1,
    "task_type": "bug_fix",
    "query": "payment is failing at checkout step 3",
    "token_budget": 6000
  }'

→ Copy graph_summary + code_snippets from the response
→ Open Claude / Cursor / Copilot Chat
→ Paste context + ask question using the prompt_template sections
→ Get precise answer


When you add new files:
─────────────────────────────────────────────────────
graphmind run .     ← incremental, only new/changed files
                       run_id increments automatically


When you have a design doc:
─────────────────────────────────────────────────────
curl -F "files=@feature-spec.docx" http://localhost:8000/api/upload
curl -X POST http://localhost:8000/api/run \
  -d '{"path": "graphmind-out/uploads/TIMESTAMP", "full": true}'
```

---

## Full API Reference

### `GET /health`

```bash
curl http://localhost:8000/health
```

```json
{"status": "ok"}
```

---

### `POST /api/run`

```bash
curl -X POST http://localhost:8000/api/run \
  -H "Content-Type: application/json" \
  -d '{"path": ".", "full": false}'
```

Returns: `run_id`, `files`, `words`, `nodes`, `edges`, `communities`, `out_dir`

---

### `POST /api/upload`

```bash
curl -F "files=@schema.sql" -F "files=@notes.md" http://localhost:8000/api/upload
```

Accepted extensions: `.md`, `.docx`, `.sql`
Returns: `folder` (path to saved files), `files_saved` (list of paths)

---

### `GET /api/runs`

```bash
curl "http://localhost:8000/api/runs?limit=20"
```

Returns: array of run history items (newest first)

---

### `GET /api/runs/{run_id}/graph`

```bash
curl http://localhost:8000/api/runs/1/graph
```

Returns: `run_id`, `nodes` (count), `edges` (count), `graph` (full node/edge data)

---

### `POST /api/context-pack`

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": 1,
    "task_type": "bug_fix",
    "query": "your question here",
    "token_budget": 6000
  }'
```

Returns: `tier`, `graph_summary`, `code_snippets`, `full_files`, `prompt_template`, `cached_artifacts`, `total_tokens`, `recommended_next_steps`

---

## Project Structure

```
graphmind-rj/
├── pyproject.toml                ← package config, dependencies, version
├── setup.py                      ← build entry point
├── README.md
├── CHANGELOG.md
├── LICENSE
│
├── graphmind/                    ← Python package (installed by pip)
│   ├── __init__.py               ← version = "0.1.1"
│   ├── __main__.py               ← python -m graphmind entry
│   │
│   ├── cli.py                    ← graphmind run . / graphmind run . --full
│   ├── api.py                    ← FastAPI app, all 6 endpoints
│   ├── pipeline.py               ← orchestrates detect→extract→build→export
│   ├── detect.py                 ← file discovery, extension bucketing
│   ├── config.py                 ← GraphMindConfig dataclass, GovernancePolicy
│   ├── models.py                 ← Node, Edge, ExtractionResult, ContextPack
│   ├── db.py                     ← SQLAlchemy ORM: RunRecord, NodeRecord, EdgeRecord
│   ├── cache.py                  ← SHA-256 hashing + ArtifactCache class
│   ├── security.py               ← email + secret redaction in extracted text
│   │
│   ├── context_budget.py         ← TokenBudgetManager, FilePriority tiers
│   ├── retrieval_planner.py      ← RetrieverPlanner, ContextGate, 3-tier logic
│   ├── prompt_templates.py       ← PromptTemplateRegistry, 6 task templates
│   │
│   ├── extractors/
│   │   ├── base.py               ← Extractor protocol (interface)
│   │   ├── registry.py           ← runs all extractors, deduplicates output
│   │   ├── python_ast.py         ← Python: functions, classes, methods via AST
│   │   ├── sql_schema.py         ← SQL: CREATE TABLE → table + column nodes
│   │   ├── text_semantic.py      ← .md/.txt/.rst: heading → concept nodes
│   │   └── docx_semantic.py      ← .docx: paragraph/heading extraction
│   │
│   ├── graph/
│   │   ├── builder.py            ← NetworkX graph from nodes + edges
│   │   └── analytics.py         ← Louvain communities, hub degree ranking
│   │
│   └── exporters/
│       ├── json_exporter.py      ← graph.json via networkx node_link_data
│       └── html_exporter.py      ← graph.html table view
│
├── frontend/                     ← React + Vite (not included in pip install)
│   ├── package.json
│   ├── vite.config.js            ← /api proxy → localhost:8000
│   ├── index.html
│   └── src/
│       ├── main.jsx
│       ├── App.jsx               ← Run / Upload / History dashboard
│       └── styles.css
│
└── tests/
    ├── test_api.py               ← API endpoint tests
    └── test_pipeline.py          ← end-to-end pipeline test
```

---

## Development Setup

```bash
git clone https://github.com/rameshj/graphmind-rj.git
cd graphmind-rj
pip install -e ".[dev]"
pytest tests/ -v
```

---

## License

MIT — See [LICENSE](LICENSE)

## Author

**Ramesh J**

## Links

- **PyPI**: https://pypi.org/project/graphmind-rj/
- **Install**: `pip install graphmind-rj`
- **Issues**: https://github.com/rameshj/graphmind-rj/issues

---

## What is graphmind-rj?

When you use AI assistants like Claude, ChatGPT, or Copilot to work on your codebase, they need **context** — which files exist, what functions do, how modules relate to each other. The problem:

- Sending full files = **too many tokens = expensive and slow**
- Sending random snippets = **AI gives wrong answers**
- Reloading everything every query = **wasteful and repetitive**

**graphmind-rj solves this.** It:

1. Reads your codebase once and builds a **knowledge graph** — a map of every function, class, table, and document and how they connect
2. Stores that graph in a local SQL database (SQLite — no server needed)
3. Exposes a `POST /api/context-pack` endpoint — you call it before asking your AI, and it returns **only the relevant context** for that specific task instead of full files

> **Note**: graphmind-rj is a context preparation tool. It does not automatically intercept AI queries. The workflow is: call `/api/context-pack` → get back a compact, task-focused context → paste it into Claude, Copilot, Cursor, or any AI tool. In practice this typically reduces the tokens you send by 60–80% compared to pasting whole files, though actual savings depend on your project size and query type.

---

## Features

| Feature | Description |
|---|---|
| 🔍 Multi-language extraction | Python (AST), SQL (schemas), DOCX, Markdown |
| 📊 Knowledge graph | NetworkX graph with community detection |
| 🚀 REST API | FastAPI backend with 6 endpoints |
| 💾 SQL persistence | SQLAlchemy + SQLite — full run history |
| 🧠 Token optimization | 3-tier smart context (graph → snippets → full files) |
| 📋 Prompt templates | Task-specific formats for bug fix, feature, refactor, review |
| 🖥️ React dashboard | Frontend for viewing runs, uploading files, browsing history |
| ⚡ Incremental runs | SHA-256 hashing skips unchanged files |

---

## Installation

```bash
pip install graphmind-rj
```

That's it. No Docker. No database server. Works on Windows, macOS, Linux.

### Verify Installation

```bash
graphmind --help
```

---

## End-to-End Tutorial

This walkthrough covers a real scenario: **you have a codebase and want to use AI to debug and add features without blowing your token budget.**

---

### Step 1 — Start the API Server

```bash
graphmind-api
```

Expected output:

```
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
```

Leave this terminal open. Open a **new terminal** for the next steps.

Health check:

```bash
curl http://localhost:8000/health
# → {"status": "ok"}
```

---

### Step 2 — Analyse Your Project

```bash
graphmind run /path/to/your/project
```

Or from the current directory:

```bash
cd /path/to/your/project
graphmind run .
```

Output:

```
Scanning files...
  Found 32 Python files, 4 SQL files, 6 Markdown files
Extracting nodes and edges...
  Python AST:   89 functions, 22 classes
  SQL schemas:  11 tables, 34 columns
  Markdown:     18 headings, 45 concept links
Building graph...  Nodes: 142  Edges: 198
Detecting communities...  Communities: 7
Exporting...
  graphmind-out/graph.json
  graphmind-out/graph.html
  graphmind-out/GRAPH_REPORT.md
  graphmind-out/graphmind.db
Done in 3.2s
```

Creates `graphmind-out/` with:

| File | What it is |
|---|---|
| `graph.json` | Full knowledge graph as JSON |
| `graph.html` | Human-readable HTML table view |
| `GRAPH_REPORT.md` | Markdown summary |
| `graphmind.db` | SQLite database with all run data |
| `cache/file_hashes.json` | File hash cache for incremental runs |

---

### Step 3 — Run via REST API (Alternative to CLI)

```bash
curl -X POST http://localhost:8000/api/run \
  -H "Content-Type: application/json" \
  -d '{"path": "/path/to/your/project", "full": false}'
```

- `"full": false` — incremental (skip unchanged files)
- `"full": true` — force full rescan

Response:

```json
{
  "run_id": 1,
  "files": 42,
  "words": 15600,
  "nodes": 142,
  "edges": 198,
  "communities": 7,
  "out_dir": "graphmind-out"
}
```

**Save the `run_id`** — you need it in later steps.

---

### Step 4 — Upload Standalone Files

For design docs, migrations, specs — upload them directly:

```bash
curl -F "files=@architecture.md" \
     -F "files=@schema.sql" \
     -F "files=@requirements.docx" \
     http://localhost:8000/api/upload
```

Response:

```json
{
  "folder": "graphmind-out/uploads/20260408_143021",
  "files_saved": [
    "graphmind-out/uploads/20260408_143021/architecture.md",
    "graphmind-out/uploads/20260408_143021/schema.sql"
  ]
}
```

Then run the pipeline on that folder:

```bash
curl -X POST http://localhost:8000/api/run \
  -d '{"path": "graphmind-out/uploads/20260408_143021", "full": true}'
```

---

### Step 5 — View Run History

```bash
curl http://localhost:8000/api/runs
```

Response:

```json
[
  {
    "id": 2,
    "target_path": "/path/to/project",
    "files": 42,
    "nodes": 142,
    "edges": 198,
    "communities": 7,
    "created_at": "2026-04-08T14:30:21"
  }
]
```

---

### Step 6 — Inspect the Graph

```bash
curl http://localhost:8000/api/runs/1/graph
```

Returns all nodes and edges:

```json
{
  "nodes": [
    {"id": "authenticate_user", "kind": "function", "source_file": "auth.py"},
    {"id": "users", "kind": "table", "source_file": "schema.sql"}
  ],
  "edges": [
    {"source": "authenticate_user", "target": "check_password", "relation": "calls"}
  ]
}
```

---

### Step 7 — Get Optimized Context for an LLM (The Core Feature)

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": 1,
    "task_type": "bug_fix",
    "query": "Why is user authentication failing after the latest deployment?",
    "token_budget": 6000,
    "include_artifacts": true
  }'
```

| Field | Type | Description |
|---|---|---|
| `run_id` | int | Which run to use as source |
| `task_type` | string | `bug_fix`, `feature`, `refactor`, `test`, `review`, `optimize` |
| `query` | string | Your specific question |
| `token_budget` | int | Max tokens (default 8000) |
| `include_artifacts` | bool | Include cached summaries |

Response:

```json
{
  "tier": "snippets",
  "graph_summary": {
    "nodes_count": 142,
    "edges_count": 198,
    "communities": 7,
    "top_hubs": ["UserService", "AuthModule", "DatabaseLayer"],
    "summary": "Graph with 142 nodes, 7 communities"
  },
  "code_snippets": [
    {
      "file_path": "auth.py",
      "start_line": 45,
      "end_line": 65,
      "content": "def authenticate_user(username, password):\n    ...",
      "reason": "function_definition"
    }
  ],
  "prompt_template": {
    "task_type": "bug_fix",
    "sections": ["## Problem", "## Current Behavior", "## Expected Behavior", "## Relevant Code", "## Proposed Fix"]
  },
  "total_tokens": 2380,
  "recommended_next_steps": [
    "Run the failing test to reproduce",
    "Add breakpoints in the shown code",
    "Check git blame for recent changes"
  ]
}
```

**How to use this response:**

1. Paste `graph_summary` as the codebase overview to your AI
2. Paste `code_snippets` as the relevant code
3. Use `prompt_template.sections` to structure your question
4. Follow `recommended_next_steps` for guided debugging

**Result: 2,380 tokens instead of 15,000** — same quality answer.

---

### Step 8 — Task Types Reference

| Task | Use for | Recommended budget |
|---|---|---|
| `bug_fix` | Crashes, wrong output, failing tests | 6,000 |
| `feature` | New endpoints, modules, integrations | 8,000–10,000 |
| `refactor` | Reducing complexity, improving patterns | 8,000 |
| `test` | Writing unit/integration tests | 4,000 |
| `review` | PR reviews, security audits | 12,000 |
| `optimize` | Slow queries, memory, bottlenecks | 6,000 |

---

### Step 9 — Use the React Dashboard

```bash
cd frontend
npm install
npm run dev
```

Open: http://localhost:5173

The dashboard provides:
- **Run panel** — enter a path and click Run
- **Upload panel** — drag and drop `.md`, `.docx`, `.sql` files
- **History panel** — browse all past runs with metrics

---

## How Token Optimization Works

graphmind-rj uses a **3-tier retrieval system**:

```
Query arrives
     │
     ▼
Tier 1: Graph Summary (~500 tokens)
     → node count, communities, top hubs
     → enough for architecture questions
     │
     ▼  (query contains: debug, bug, fix, error, refactor...)
Tier 2: Code Snippets (~1,500–3,000 tokens)
     → function definitions from key files
     → enough for most bug fixes and feature planning
     │
     ▼  (complex tasks only: full refactor, deep review)
Tier 3: Full Files (~3,000+ tokens)
     → complete file content
```

**Token Budget Allocation:**

| Priority | Files | Budget |
|---|---|---|
| CRITICAL | Recently changed files | 50% |
| HIGH | Direct dependencies | 30% |
| MEDIUM | Tests, related modules | 15% |
| LOW | Docs, configs | 5% |

---

## File Support

| Extension | What gets extracted |
|---|---|
| `.py` | Functions, classes, methods, imports |
| `.sql` | Tables, columns (CREATE TABLE) |
| `.md` | Headings, paragraphs, concept links |
| `.docx` | Headings, paragraphs (sensitive info redacted) |

---

## Token Savings — Real Numbers

| Scenario | Without | With graphmind-rj | Saved |
|---|---|---|---|
| Bug fix in 40-file project | ~12,000 tokens | ~2,400 tokens | **80%** |
| Feature in 80-file project | ~25,000 tokens | ~5,000 tokens | **80%** |
| Code review of 10 files | ~8,000 tokens | ~3,200 tokens | **60%** |
| Architecture question | ~10,000 tokens | ~500 tokens | **95%** |

---

## Use with VS Code / Cursor

1. Run `graphmind-api` in a background terminal
2. Run `graphmind run .` on your project once per session
3. Before asking AI anything, call `POST /api/context-pack` with your question
4. Paste the response into Copilot Chat, Claude, or Cursor
5. Ask your question — AI now has exactly the right context

---

## API Endpoints Quick Reference

| Method | Endpoint | Purpose |
|---|---|---|
| GET | `/health` | Health check |
| POST | `/api/run` | Run pipeline on a directory |
| POST | `/api/upload` | Upload `.md`, `.docx`, `.sql` files |
| GET | `/api/runs` | List run history |
| GET | `/api/runs/{run_id}/graph` | Get graph for a run |
| POST | `/api/context-pack` | Get token-optimized context for LLM |

---

## Project Structure

```
graphmind-rj/
├── graphmind/
│   ├── api.py                  ← FastAPI app (6 endpoints)
│   ├── cli.py                  ← CLI entry point
│   ├── pipeline.py             ← detect → extract → build → export
│   ├── models.py               ← Node, Edge, ContextPack data models
│   ├── db.py                   ← SQLAlchemy ORM
│   ├── cache.py                ← File hashing + ArtifactCache
│   ├── context_budget.py       ← Token allocation engine
│   ├── retrieval_planner.py    ← 3-tier context retrieval
│   ├── prompt_templates.py     ← Task-specific LLM formats
│   ├── extractors/
│   │   ├── python_ast.py       ← Python AST extraction
│   │   ├── sql_schema.py       ← SQL schema extraction
│   │   ├── text_semantic.py    ← Markdown extraction
│   │   └── docx_semantic.py    ← DOCX extraction
│   ├── graph/
│   │   ├── builder.py          ← NetworkX graph
│   │   └── analytics.py        ← Communities, hubs
│   └── exporters/
│       ├── json_exporter.py
│       └── html_exporter.py
├── frontend/                   ← React + Vite dashboard
└── tests/
```

---

## Development

```bash
git clone https://github.com/rameshj/graphmind-rj.git
cd graphmind-rj
pip install -e ".[dev]"
pytest tests/ -v
```

---

## License

MIT — See [LICENSE](LICENSE)

## Author

**Ramesh J**

## Links

- **Install**: `pip install graphmind-rj`
- **PyPI**: https://pypi.org/project/graphmind-rj/
- **Issues**: https://github.com/rameshj/graphmind-rj/issues

## Quick Start

### Start Backend API

```bash
graphmind-api
# API running on http://localhost:8000
```

### Run Pipeline on Directory

```bash
graphmind run .
```

### Upload Files

```bash
curl -F "files=@file.md" -F "files=@schema.sql" http://localhost:8000/api/upload
```

## Architecture

### Backend Modules

- **api.py**: FastAPI service with 6+ endpoints (run, upload, history, graph, context-pack)
- **cli.py**: Command-line interface
- **pipeline.py**: Full orchestration (detect → extract → build → cluster → export)
- **extractors/**: Multi-language extraction registry
  - `python_ast.py`: Functions, classes, methods (Python)
  - `sql_schema.py`: Tables, columns, constraints (SQL)
  - `docx_semantic.py`: Structure and concepts (DOCX)
  - `text_semantic.py`: Headings and paragraphs (Markdown)
- **graph/**: Graph building and analysis
  - `builder.py`: NetworkX graph construction
  - `analytics.py`: Community detection, hub ranking
- **db.py**: SQLAlchemy ORM (RunRecord, NodeRecord, EdgeRecord)
- **context_budget.py**: Token allocation by priority tier
- **retrieval_planner.py**: Tiered context retrieval (graph → snippets → full files)
- **prompt_templates.py**: LLM-optimized templates for bug_fix, feature, refactor, test, review

### Frontend

- React + Vite dashboard for run management and visualization
- Located in `frontend/` directory

## API Endpoints

| Method | Endpoint | Purpose |
|--------|----------|---------|
| GET | `/health` | Health check |
| POST | `/api/run` | Run pipeline on directory |
| POST | `/api/upload` | Upload .md, .docx, .sql files |
| GET | `/api/runs` | List run history |
| GET | `/api/runs/{run_id}/graph` | Get graph for specific run |
| POST | `/api/context-pack` | Get optimized context for LLM |

## Example: Get Optimized Context

```bash
curl -X POST http://localhost:8000/api/context-pack \
  -H "Content-Type: application/json" \
  -d '{
    "run_id": 1,
    "task_type": "bug_fix",
    "query": "Why is authentication failing?",
    "token_budget": 6000
  }'
```

Response includes graph summary, code snippets, and prompt template.

## Token Savings

- **Graph Summary Tier**: ~500 tokens - graph structure only
- **Snippets Tier**: ~1500-3000 tokens - changed files + dependencies
- **Full Files Tier**: ~3000+ tokens - complete context (fallback)

Smart expansion prevents unnecessary token usage. Most queries stay in Tier 1-2.

## Development

### Install for Development

```bash
git clone https://github.com/rameshj/graphmind.git
cd graphmind
pip install -e ".[dev]"
```

### Run Tests

```bash
pytest tests/
```

### Build Documentation

See [PUBLISH.md](PUBLISH.md) for PyPI publishing guide.

## Configuration

Create `graphmind.toml` or `graphmind.yaml` to customize:

```yaml
# Output directory
out_dir: graphmind-out

# Security settings
redact_secrets: true
redact_emails: true

# Extraction policy
extract_comments: true
extract_docstrings: true

# Graph building
min_edge_confidence: 0.5
```

## License

MIT - See LICENSE file

## Author

Ramesh J

## Support

- 📖 [GitHub](https://github.com/rameshj/graphmind)
- 🐛 [Issue Tracker](https://github.com/rameshj/graphmind/issues)
- 💬 [Discussions](https://github.com/rameshj/graphmind/discussions)
```

Backend runs on http://localhost:8000.

### 2) Start frontend

```bash
cd ramesh-graphmind/frontend
npm install
npm run dev
```

Frontend runs on http://localhost:5173.

## SQL data model

The backend stores:

1. runs: each pipeline execution summary
2. nodes: graph nodes linked to a run
3. edges: graph edges linked to a run

Database file is created at graphmind-out/graphmind.db.

## Ingestion behavior

- .md and .txt style docs: heading-based concept extraction
- .docx: paragraph and heading extraction using python-docx
- .sql: schema extraction from CREATE TABLE statements

## What to build next

1. Add auth and role-based access
2. Add migration tooling (Alembic)
3. Add richer SQL parsing for foreign keys and joins
4. Add graph diff views between runs
