Metadata-Version: 2.4
Name: sciogen
Version: 0.1.1
Summary: Codebase intelligence layer — a queryable knowledge graph over your code.
Project-URL: Homepage, https://ayanbag.github.io/sciogen/
Project-URL: Repository, https://github.com/ayanbag/sciogen
Project-URL: Documentation, https://ayanbag.github.io/sciogen/docs.html
Project-URL: Issues, https://github.com/ayanbag/sciogen/issues
Author: sciogen contributors
License: MIT
License-File: LICENSE
Keywords: code-search,knowledge-graph,mcp,static-analysis,tree-sitter
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Requires-Python: >=3.10
Requires-Dist: chromadb>=0.5
Requires-Dist: kuzu>=0.7
Requires-Dist: mcp>=1.0
Requires-Dist: rich>=13.7
Requires-Dist: tree-sitter-language-pack<1.0,>=0.6
Requires-Dist: tree-sitter<0.26,>=0.22
Requires-Dist: typer>=0.12
Requires-Dist: watchdog>=4.0
Provides-Extra: dev
Requires-Dist: pytest-timeout; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: embeddings
Requires-Dist: einops; extra == 'embeddings'
Requires-Dist: sentence-transformers>=3.0; extra == 'embeddings'
Description-Content-Type: text/markdown

<img src="site/public/logo.png" alt="Image" width="100%" />

<div align="center">
  <p>
    <a href="https://pypi.org/project/sciogen/"><img alt="PyPI" src="https://img.shields.io/pypi/v/sciogen.svg"></a>
    <a href="https://pypi.org/project/sciogen/"><img alt="Python versions" src="https://img.shields.io/pypi/pyversions/sciogen.svg"></a>
    <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-blue.svg"></a>
  </p>
</div>

**Codebase intelligence layer** — a queryable knowledge graph over your code.

sciogen parses a codebase once, resolves every reference to the exact definition it
points to, and materializes the result into an embedded triple store. After that,
questions like *"who calls `AuthService.login`?"*, *"what breaks if I change
`UserModel.find_by_email`?"*, or *"where is the password hashing logic?"* are
sub-second index lookups that return **typed objects with file/line locations** —
not grep results, not raw source dumps.

sciogen is a **static analysis and indexing tool**. It never calls an LLM, never
counts tokens, and has no concept of a context window. It is to codebases what
Elasticsearch is to documents: precomputed, queryable structure.

```
pip install sciogen
sciogen index .
```

```
  ███████╗ ██████╗██╗ ██████╗  ██████╗ ███████╗███╗   ██╗
  ██╔════╝██╔════╝██║██╔═══██╗██╔════╝ ██╔════╝████╗  ██║
  ███████╗██║     ██║██║   ██║██║  ███╗█████╗  ██╔██╗ ██║
  ╚════██║██║     ██║██║   ██║██║   ██║██╔══╝  ██║╚██╗██║
  ███████║╚██████╗██║╚██████╔╝╚██████╔╝███████╗██║ ╚████║
  ╚══════╝ ╚═════╝╚═╝ ╚═════╝  ╚═════╝ ╚══════╝╚═╝  ╚═══╝
  Codebase Intelligence Layer  v0.1.0

✓ Scanning project structure...
✓ Reading 143 files...
✓ Parsing Python, TypeScript...
✓ Building knowledge graph...
✓ Embedding 1,204 symbols...
✓ Linking dependencies...
✓ Done! 1,204 nodes · 3,456 edges · indexed in 4.2s
  Ready. Your codebase is now queryable.
```

## Why

A coding agent understands a codebase by **reading files** — and files are the
wrong granularity. To learn one function's callers it reads thousands of lines
of source into its context window, burning tokens on noise. sciogen precomputes
the structure once, so the same answer is a handful of typed records with exact
`file:line` locations.

You run `sciogen index .` once. The knowledge graph is written to a `.sciogen/`
directory **inside the codebase**. From then on, an agent working in that repo
reads the graph instead of the raw files — the callers of a symbol, the blast
radius of a change, where a concept lives — for a fraction of the tokens a
file-by-file crawl would cost. sciogen itself never calls an LLM and is unaware
of tokens; the savings are a consequence of returning structure instead of
source.

```
  index once  ─────────────►  .sciogen/  (graph stored in the repo)
  (sciogen index .)                │
                                   ▼
  agent asks: "who calls login?"  →  typed records + file:line
  agent asks: "what breaks if …?"  →  impact subgraph, not 40 files
```

## The two ways it's used

**You — from the terminal.** `sciogen index .` builds the graph; `sciogen
explore` opens it as an interactive browser GUI to inspect the codebase
visually.

**Your coding agent — over MCP.** `sciogen mcp .` exposes the graph to any
agent (Claude Code, etc.) through the Model Context Protocol, so it can query
callers, dependencies, impact, and semantic search directly instead of reading
files. See [docs/mcp.md](docs/mcp.md).

### How the agent connects

The key inversion: **sciogen never calls an LLM — the LLM calls sciogen.**
sciogen is a tool the agent uses, not a model. There are three parties:

```
  the LLM   ⇄   agent host (Claude Code, Cursor…)   ⇄   sciogen   ⇄   .sciogen/ graph
  └ the model ┘  └─ owns the LLM connection ────────┘  └── just a tool server ──┘
```

Two ways the agent reaches sciogen — pick whichever it supports:

- **MCP (structured).** Register the server once and the host calls typed tools
  that return JSON. Claude Code: `claude mcp add sciogen -- sciogen mcp .`
  (or a `.mcp.json` in the repo root).
- **CLI (zero-config).** Any agent with shell access just runs `sciogen callers
  …` / `sciogen impact …` and reads the table from stdout — no setup.

To make the agent use sciogen **autonomously** — querying the graph before
every modification, refreshing the index after every edit, without being told —
run the one-time setup:

```bash
sciogen setup-agent . --hooks
```

It registers the MCP server (`.mcp.json`), adds usage rules to `CLAUDE.md` so
the agent prefers graph queries over file crawling, and installs a Claude Code
hook that silently re-indexes after each file edit. Details in
[docs/mcp.md](docs/mcp.md).

Either way sciogen returns typed records with `file:line` locations, so the
agent opens only the files it actually needs. Full walkthrough, including a
one-turn example, is in [docs/mcp.md](docs/mcp.md).

## What the graph gives an agent

- **Exact call graphs** — a call to `login()` resolves to
  `src/auth/service.py:AuthService.login`, not the string `"login"`. References
  static analysis cannot prove (dynamic dispatch, injected dependencies) are kept
  with a **low confidence score** rather than silently dropped, so the caller
  chooses how much to trust each edge.
- **Impact analysis** — everything that may break if a symbol changes:
  transitive callers with hop counts, subclasses, implementors, tests, mutators —
  and the same for a whole PR at once from a unified diff.
- **Semantic + hybrid search** — local embeddings at four granularities (file,
  class, function, chunk); hybrid mode expands vector hits through the graph.
- **Incremental by design** — SHA256 differ with a stat fast path; a single-file
  change re-indexes in under a second, and dependent files are re-linked without
  being re-parsed.
- **Interactive visualization** — `sciogen explore` renders the graph in your
  browser from one self-contained HTML file. No server.

## Getting started

```bash
pip install sciogen
# optional, for real code-optimized semantic search (larger, one model download):
pip install "sciogen[embeddings]"

cd your-project
sciogen index .          # build the graph (stored in ./.sciogen; incremental after)
sciogen explore          # inspect it visually in the browser
sciogen mcp .            # serve it to your coding agent over MCP
```

Add `.sciogen/` to your `.gitignore` — it's machine-local and regenerable.

> **`sciogen` not recognized after install?** Your Python scripts folder isn't
> on `PATH` (common with Microsoft Store Python on Windows). Run
> `python -m sciogen index .` instead, install via `pipx install sciogen`, or
> add the folder to `PATH` — see
> [the quickstart troubleshooting section](docs/quickstart.md#sciogen-not-recognized-after-install)
> for the exact commands.

You can also query directly from the terminal for a quick look:

```bash
sciogen search "password hashing"      # semantic search
sciogen callers AuthService.login      # who calls this?
sciogen impact UserModel.find_by_email # what breaks if this changes?
sciogen deps src/auth/service.py       # imports, transitive deps
```

See [docs/cli.md](docs/cli.md) for every command.

## Architecture in one paragraph

`sciogen index` runs a five-stage pipeline per file: **tree-sitter** parses
(100+ languages, one API), a **normalizer** converts the language-specific AST
into a universal IR (only this layer knows languages), the **symbol resolver**
traces every reference to its exact definition with a confidence score, the
**graph builder** materializes typed nodes/edges into **KuzuDB** (Cypher), and a
**SHA256 differ** backed by **SQLite** makes re-runs incremental. Embeddings of
normalized symbol summaries (never raw code) go to **ChromaDB** at four
granularities. Queries are deterministic: structural queries hit KuzuDB, semantic
queries hit ChromaDB, hybrid uses vector hits as seeds for graph expansion. The
full design — including the performance architecture (parallel parsing, batched
transactional writes, embed deduplication, incremental re-resolution) — is in
[docs/architecture.md](docs/architecture.md).

## Storage

Everything is embedded — no servers, no ports, one `.sciogen/` directory:

| Store | Role |
|-------|------|
| KuzuDB | Graph topology — nodes, typed edges, confidence scores |
| ChromaDB | Vector embeddings at file/class/function/chunk granularity |
| SQLite | File hashes, symbol table, reference records, schema version |

Add `.sciogen/` to your `.gitignore`.

## Documentation

| Doc | Contents |
|-----|----------|
| [docs/quickstart.md](docs/quickstart.md) | Install, first index, first queries |
| [docs/cli.md](docs/cli.md) | Every command and flag |
| [docs/mcp.md](docs/mcp.md) | MCP tools and agent setup — how an agent consumes the graph |
| [docs/schema.md](docs/schema.md) | Node types, edge types, confidence model |
| [docs/architecture.md](docs/architecture.md) | The two phases, every design decision, performance architecture |
| [docs/api.md](docs/api.md) | Embedded Python API — only if you're building tooling *on top of* sciogen |

## Measuring the savings

sciogen itself never counts tokens (hard design rule) — so the measurement
lives in a standalone companion utility. It parses Claude Code's local session
transcripts and compares token consumption between sessions that used sciogen
and sessions that didn't, normalized per user prompt:

```bash
python tools/claude_token_monitor.py --project your-project --days 14
```

It reports fresh input, cache reads, and output tokens per session, flags which
sessions used sciogen (MCP tools or CLI calls), and prints the per-prompt
delta. Observational, not a controlled experiment — compare similar work for a
fair read. `--json` for machine-readable output, `--watch` for a live view.

## Supported languages

Full symbol extraction: **Python**, **JavaScript**, **TypeScript/TSX**.
File-level indexing (parse check + file search): Go, Rust, Java, Ruby, PHP, C,
C++, C#, Kotlin, Swift, Scala, Lua. Adding full support for a language means
writing one normalizer — see [docs/architecture.md](docs/architecture.md#ast-normalizer).

## Development

```bash
git clone <repo> && cd sciogen
python -m venv .venv && .venv/Scripts/activate   # or bin/activate
pip install -e ".[dev]"
pytest
```

MIT license.
