Everything sciogen does, and why.

Introduction #

sciogen is a static analysis and indexing tool. It parses a codebase once, resolves every reference to the exact definition it points to, and materializes the result into an embedded triple store. After that, questions like "who calls AuthService.login?" or "what breaks if I change find_by_email?" are sub-second index lookups that return typed records with exact file:line locations.

Why files are the wrong granularity

A coding agent understands code by reading files — and to learn one function's callers it pulls thousands of lines into its context window, burning tokens on noise. sciogen precomputes the call graph, the dependency graph, the type hierarchy, and semantic embeddings, so the same answer costs a handful of typed records.

You run sciogen index . once. The graph is written to a .sciogen/ directory inside the repo. From then on, an agent working in that repo queries the graph — over MCP or the CLI — and opens only the files the graph points it to.

Two phases, no ML routing

Indexing turns source into a typed, confidence-scored graph plus multi-granularity embeddings. Querying answers questions deterministically — which store answers is fixed by the method called, never by a model: structural queries hit KuzuDB, semantic queries hit ChromaDB, hybrid uses vector hits as seeds for graph expansion.

Getting started #

Install

$ pip install sciogen

That's the full indexer, query engine, CLI, MCP server, and explore GUI. Semantic search works out of the box on a deterministic hashing embedder (keyword-level quality, zero downloads). For code-optimized semantic search, add the embeddings extra — one larger local model download, nothing leaves your machine:

$ pip install "sciogen[embeddings]"

$ python -m sciogen index .

# 1. Print the folder pip installed the command into:
> python -c "import sysconfig, os; print(sysconfig.get_path('scripts', os.name + '_user'))"

# 2. Add it to your user PATH (paste the folder from step 1 in place of <folder>):
> [Environment]::SetEnvironmentVariable('Path', "$([Environment]::GetEnvironmentVariable('Path','User'));<folder>", 'User')

# 3. Open a NEW terminal — existing windows keep the old PATH.

$ echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc   # or ~/.zshrc

$ pipx install sciogen

Index a project

$ sciogen index .

✓ Scanning project structure...
✓ Reading 143 files...
✓ Parsing Python, TypeScript...
✓ Building knowledge graph...
✓ Embedding 1,204 symbols...
✓ Linking dependencies...
✓ Done! 1,204 nodes · 3,456 edges · indexed in 4.2s
  Ready. Your codebase is now queryable.

The index lives in .sciogen/ at the project root — add it to your .gitignore, it's machine-local and regenerable. Re-run any time: it's incremental. Unchanged files are skipped via a stat + SHA256 differ, so a no-op re-index is effectively instant and a single-file change takes under a second.

Explore it visually

$ sciogen explore

Opens the knowledge graph in your browser from a single self-contained HTML file — no server, no port. The view starts collapsed to file nodes; click a file to reveal its classes and functions, click a function to reveal its call edges. Node color encodes type, node size encodes in-degree (hotspots), edge dash style encodes confidence. It's a snapshot — re-run sciogen index then sciogen explore after changes.

Query from the terminal

$ sciogen search "password hashing"       # semantic search
$ sciogen callers AuthService.login       # who calls this?
$ sciogen impact UserModel.find_by_email  # what breaks if this changes?
$ sciogen deps src/auth/service.py        # imports + transitive deps

Using with coding agents #

Three parties, not two

The piece people miss is the host in the middle. The host (Claude Code, Cursor, Cline, Claude Desktop…) is what actually talks to the LLM — it holds the API key and routes the model's tool-call requests. sciogen is only a tool server: it answers graph queries and returns JSON. It has no idea which model is on the other side, which makes it LLM-agnostic by construction.

What happens in one turn

You ask your agent: "is it safe to change hash_password?"

1. Host sends the model your message + sciogen's tool list
2. Model decides to call analyze_impact("hash_password")
3. Host routes the request to the sciogen mcp subprocess
4. sciogen queries .sciogen/ → JSON: impacted symbols, hops, confidence
5. Host feeds that JSON back to the model as the tool result
6. Model answers — having read ~200 tokens of structure, not 3 files

Steps 3–4 are the only ones sciogen takes part in. Everything about the model is the host's job.

Path 1 — MCP (structured)

Register the server once; the host launches it and calls typed tools that return JSON. For Claude Code, one command per project:

$ claude mcp add sciogen -- sciogen mcp .

Or a .mcp.json in the repo root — shareable with your team; Claude Code, Cursor, and Windsurf all read this shape:

{
  "mcpServers": {
    "sciogen": {
      "command": "sciogen",
      "args": ["mcp", "."]
    }
  }
}

Path 2 — CLI (zero-config)

If the agent can run shell commands (Claude Code can, out of the box), it needs no setup at all — it runs a command and reads the table from stdout:

$ sciogen callers AuthService.login --depth 2

hops  conf  via    symbol                    location
   1  0.90  CALLS  AuthService.check_pass…   src/auth/service.py:22
   2  0.40  CALLS  handle_login              src/app.py:4

MCP vs CLI: MCP is structured — typed JSON, formal tool schemas, best when the host supports it. The CLI is universal — any agent with a shell. They expose the same query engine; pick whichever your agent can reach.

Autonomous use — one command

An agent reaches for a tool on its own when three things are true: the tool says when to use itself, the project's agent memory says to prefer it, and the data never goes stale. setup-agent wires all three:

$ sciogen setup-agent . --hooks

  ✓ .mcp.json — sciogen MCP server registered
  ✓ CLAUDE.md — autonomous-usage rules added
  ✓ .claude/settings.json — re-index hook installed

• .mcp.json registers the server — Claude Code, Cursor, and Windsurf read it automatically; commit it to share with your team.
• CLAUDE.md gets a marker-guarded rules block the agent reads every session: query the graph before modifying a symbol, use search instead of grep, open only the file:line locations results point to. Existing content is preserved.
• --hooks installs a Claude Code hook that runs sciogen index . --quiet after every file edit — the graph refreshes itself, sub-second, so the agent never queries stale structure.

The tool descriptions themselves are trigger-oriented too — analyze_impact announces "use BEFORE any rename or refactor", get_context_for_task says "START HERE for any broad task". The result: you give the agent a task, it finds the right files through the graph, checks impact before touching anything, edits, and the hook re-indexes behind it. No human in the loop for any of it.

MCP tools #

Eight tools, each a thin wrapper over the query engine — the MCP layer contains no logic of its own. Every tool returns JSON where node references carry exact file / line_start / line_end, so the agent reads source only when it actually needs to.

Architecture #

Indexing runs five stages per file. Only the normalizer knows languages; everything downstream — resolver, graph builder, query engine, MCP — is language-agnostic. And only changed files are ever re-processed.

The full system, interactive

Everything on one map — the indexing pipeline, the triple store, and the query side. Click any component to see what it does, or play a flow to watch how data moves.

1 — Discovery + two-tier differ

One walk over the project tree, pruning ignored directories. Change detection is two-tier because it's the hot path of re-indexing: first a stat fast path — if a file's (size, mtime) signature matches what SQLite recorded, the file is never even opened. Then SHA256 truth — only stat-changed files are read and hashed; if the hash still matches (a touch, a branch switch that restored content), the file is not re-parsed. mtime is never trusted to declare a file changed, only to prove it unchanged — which is what makes this correct in CI/CD.

2 — AST normalizer

Converts language-specific tree-sitter ASTs into a universal IR: the same ClassNode, FunctionNode, call sites, imports, and mutations regardless of source language. This is what makes multi-language support composable — adding a language is one normalizer class; unsupported languages fall back to a generic normalizer that still yields a searchable file node.

3 — Symbol resolver

Tree-sitter gives syntax, not semantics — without this step a CALLS edge is a string, not a pointer. The resolver builds a scope table from each file's imports, then traces every reference to the exact definition: a call to login() resolves to src/auth/service.py:AuthService.login, not the string "login". References that can't be statically proven (dynamic dispatch, injected dependencies) still produce edges — with a low confidence score rather than being silently dropped.

4 — Graph builder

Materializes resolved nodes and typed edges into KuzuDB. Confidence is never filtered at write time: the graph stores everything, and callers set their own threshold at query time.

5 — Embeddings

Alongside the graph, ChromaDB holds embeddings at four granularities — file, class, function, chunk — because no single granularity answers all queries well: broad "how is auth structured?" needs file vectors; precise "the block that rate-limits login" needs chunk vectors. What gets embedded is always a normalized summary (name + file + params + docstring + call list), never raw source — raw code embeds noise.

Technology choices #

Every component, what it does, and why it was chosen over the alternatives. The through-line: everything is embedded — no servers, no ports, no long-running processes anywhere.

Performance design #

The contract: first index fast, re-index near-instant, queries sub-second. Measured on sciogen's own source: first index 25.4s, no-op re-index 0.2s.

• Parallel parsing. Parsing + normalization is CPU-bound and per-file independent, so large batches fan out across cores. Small batches stay serial — pool startup costs more than parsing a handful of files.
• Two-tier differ. Stat signature short-circuit before any file is opened; SHA256 as the source of truth.
• Batched transactional writes. Prepared statements inside one explicit transaction per stage — not one auto-commit round trip per row.
• Embedding deduplication. Embedding is the slowest stage by an order of magnitude. Each node's summary is content-hashed; only changed summaries are re-embedded. A whitespace refactor doesn't re-embed the file.
• Incremental re-resolution. When file A changes, files importing A are re-resolved from reference records persisted in SQLite — without re-parsing them. Renaming a function in A correctly rewires call edges out of an untouched B.
• Query cache keyed by index version. Every index run bumps a counter; one bump invalidates every cached result, and repeated agent queries on an unchanged index are dictionary lookups.
• Lazy imports. The heavy stores load only when first touched, so CLI startup stays instant.

Graph schema & confidence #

Node kinds

Edge kinds

The confidence model

Confidence exists because static analysis is incomplete, not wrong. Dropping unprovable edges would fabricate a misleadingly clean graph — so sciogen keeps them and scores them, and you choose the cutoff at query time.

On multi-hop traversals sciogen reports path confidence — the minimum edge confidence along the best path, because a chain is only as trustworthy as its weakest link: min(0.9, 0.4, 1.0) → 0.4.

CLI reference #

Only index shows the banner and progress flow; query commands print their result and nothing else. Files that fail to parse are skipped with a single ⚠ line and logged — indexing never stops for one bad file.

Configuration #

Everything has working defaults; environment variables override:

Contributing #

sciogen is open source and contributions are welcome — bug reports, language-normalizer improvements, docs, and features all move it forward. The complete guide lives in CONTRIBUTING.md; the essentials are below.

Set up a dev environment

sciogen targets Python 3.10+ and builds with hatchling. Install it editable (-e) so source edits — including the version and CLI banner — take effect immediately, with no reinstall:

$ git clone https://github.com/ayanbag/sciogen
$ cd sciogen
$ pip install -e ".[dev,embeddings]"

Then confirm your environment is live — both must print the same version; if they disagree you're running a stale install:

$ python -c "import sciogen; print(sciogen.__version__)"
$ python -m sciogen --version

Run the tests

Every change should keep the suite green. New behavior needs a test; a bug fix should add a regression test that fails before and passes after.

$ pytest                # full suite
$ pytest -p no:randomly # deterministic order

Versioning

The version lives in a single place — __version__ in sciogen/__init__.py. pyproject.toml reads it via hatchling and the CLI banner imports the same constant, so bump that one line and everything stays in lockstep. Never add a second version string.