Metadata-Version: 2.4
Name: repodigest-py
Version: 0.1.0
Summary: Deterministic codebase context synthesizer and token budgeter
License: MIT
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click>=8.0.0
Requires-Dist: rich>=13.0.0
Requires-Dist: tiktoken>=0.5.0
Dynamic: license-file

# RepoDigest

**A local-first codebase context synthesizer and token budgeter.**

RepoDigest statically analyzes a Python codebase — no network calls, no external services, no
code ever leaves your machine — and turns it into a token-budgeted, LLM-ready context around a
target symbol, file, or plain-English question. It replaces "paste the whole repo into the
prompt" with a deterministic pipeline: parse the AST, build the call graph, rank or target a
symbol, and pack outward from it until the budget runs out.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.10%2B-blue.svg)](pyproject.toml)
[![Tests](https://img.shields.io/badge/tests-29%20passing-brightgreen.svg)](tests/)
[![Dependencies](https://img.shields.io/badge/dependencies-tiktoken%20%7C%20click%20%7C%20rich-lightgrey.svg)](pyproject.toml)

> The test badge reflects the local `pytest` suite as of the latest commit — there is no CI
> workflow configured yet, so treat it as a snapshot, not a live status check.

---

## Table of Contents

- [The Core Problem & Motivation](#the-core-problem--motivation)
- [System Architecture & Algorithms](#system-architecture--algorithms)
  - [1. AST Parsing Pipeline](#1-ast-parsing-pipeline)
  - [2. Dependency Graph Construction](#2-dependency-graph-construction)
  - [3. Context Packing Algorithm](#3-context-packing-algorithm)
  - [4. Local BM25 Ranking Engine](#4-local-bm25-ranking-engine)
- [Installation](#installation)
- [Complete CLI Reference & Workflows](#complete-cli-reference--workflows)
- [Programmatic Python API](#programmatic-python-api)
- [Test Verification & Benchmarks](#test-verification--benchmarks)
- [Development & Solo Authorship](#development--solo-authorship)

---

## The Core Problem & Motivation

Feeding an LLM "the whole codebase" is the default move when a model needs context it doesn't
already have — and it's a bad one, for three compounding reasons:

**1. Context window bloat and cost.** Every additional token in the prompt is billed, whether or
not the model needs it. A 50-file service dumped wholesale into a prompt might be 200K tokens of
which perhaps 2K are actually relevant to the question being asked. That's not a rounding error —
it's two orders of magnitude of wasted spend, repeated on every single call.

**2. "Lost in the middle."** Even when a model's context window is technically large enough,
published attention studies (e.g. Liu et al., *"Lost in the Middle: How Language Models Use Long
Contexts"*) consistently show that model recall of information degrades — often severely — for
content placed in the middle of a long context, versus the beginning or end. Dumping an entire
repository doesn't just cost tokens; it actively *degrades* the model's ability to find and use
the one function that actually matters, by burying it in surrounding noise.

**3. Irrelevant surface area invites wrong answers.** More unrelated code in context means more
opportunities for the model to anchor on a similarly-named function in an unrelated module, or to
"helpfully" incorporate a pattern from code that was never meant to be relevant to the task.

### Why not just concatenate files?

Naive bundling scripts (cat all `.py` files into one blob, optionally with file-path headers) are
simple, but they have no notion of *relevance* or *relationship*. They can't answer "what depends
on this function" or "what does this function depend on" — every file is either fully in or fully
out. They also can't do interface-only rendering: a caller either gets the whole 200-line
implementation or nothing, when often it only needs the four-line signature to reason correctly
about how to call it.

### Why not just use vector-only RAG?

Embedding-based retrieval (chunk the repo, embed the chunks, retrieve by cosine similarity) solves
part of the relevance problem, but introduces its own failure modes:

- **Chunk boundaries don't respect code structure.** A fixed-size or paragraph-based chunker will
  routinely split a function in half, or merge the tail of one function with the head of the next
  — handing the model a syntactically incomplete fragment.
- **No structural guarantees.** Semantic similarity between a query and a docstring says nothing
  about whether that symbol is actually *reachable* from — or a dependency of — the code the user
  is working on. A vector store has no concept of "caller" or "callee."
- **Non-determinism and drift.** Embedding models change between versions, similarity thresholds
  need tuning, and the same query can silently retrieve different results after a model upgrade.
- **Infrastructure weight.** A vector database is one more moving part, one more thing to index,
  invalidate, and keep in sync with the source tree, for a problem that — for a single codebase on
  a single machine — a deterministic call graph solves for free.

RepoDigest takes a **deterministic AST + DAG** approach instead: parse the exact source with
Python's own compiler front-end, build an exact (name-resolved) call graph from it, and traverse
that graph outward from a precise starting point. The same target, the same repository state,
always produces the same packed context — no embeddings, no similarity thresholds, no drift.

---

## System Architecture & Algorithms

```mermaid
flowchart LR
    subgraph Input
        A[Python source tree]
    end

    A --> P["parsers/py_parser.py\nAST symbol extraction"]
    P --> G["graph/call_graph.py\nCross-file call graph"]
    P --> B["packer/budget.py\nSymbol registry"]
    G --> B
    Q[["Query or --entry"]] --> R["search/ranker.py\nBM25 ranking"]
    R -->|seed symbol| B
    B --> S["packer/serializer.py\nXML / Markdown"]
    S --> O[Token-budgeted context]
```

### 1. AST Parsing Pipeline

`repodigest.parsers.py_parser` uses Python's built-in `ast` module — the same parser CPython uses
to compile your code — so extraction is exact, not regex-approximated. `parse_source`/`parse_file`
walk a module's top-level body and, for every `ClassDef`, its nested body, extracting:

```
source text
     │
     ▼
 ast.parse()                     # exact syntax tree, same as the interpreter uses
     │
     ▼
 walk tree.body (top level only) ─────────────┐
     │                                        │
     ├─ Import / ImportFrom  → ImportInfo      │  (module, name, alias, lineno)
     ├─ FunctionDef/AsyncFunctionDef → FunctionInfo
     │     ├─ args (positional/varargs/kwonly/kwargs, with annotations & defaults)
     │     ├─ returns (unparsed annotation)
     │     ├─ docstring (ast.get_docstring)
     │     └─ source: full body OR signature-only stub
     └─ ClassDef → ClassInfo
           ├─ bases / decorators
           ├─ docstring
           ├─ methods: tuple[FunctionInfo, ...]  (same extraction, is_method=True)
           └─ source: full class body OR every method stubbed
```

Every function, method, and class carries **two renderable forms**, selected by `mode`:

- **`mode="full"`** — the verbatim original text, decorators included. (This is *not* simply
  `ast.get_source_segment(source, node)`: that function anchors on `node.lineno`, which for a
  decorated function/class points at the `def`/`class` line rather than the decorator above it —
  silently dropping decorators. RepoDigest's `_full_source` helper slices from the first
  decorator's line instead, when one is present.)
- **`mode="signature"`** — the body is rebuilt as an AST stub (`_render_signature` /
  `_render_class_signature`) keeping only the docstring and an `...` placeholder, then
  re-serialized with `ast.unparse`. For a class, *every method* is stubbed this way, so the whole
  class's public shape is visible in a fraction of the tokens its implementation would cost.

This is the mechanism behind the CLI's `--signatures-only` flag: it doesn't strip lines with a
regex — it round-trips through the AST, so decorators, type annotations, and multi-line default
arguments are always rendered correctly.

### 2. Dependency Graph Construction

`repodigest.graph.call_graph.CallGraph.from_directory(root)` builds a **fully-qualified,
cross-file** symbol graph in two passes:

**Pass 1 — Symbol table.** Every file under `root` is parsed, and every function, method, and
class is assigned a fully-qualified name derived from its file path and nesting:

```
src/repodigest/packer/budget.py  →  module qualname "packer.budget"
    class ContextPacker          →  "packer.budget.ContextPacker"
        def pack(...)            →  "packer.budget.ContextPacker.pack"
    def build_symbol_registry    →  "packer.budget.build_symbol_registry"
```

These qualnames are collected into a `symbol_table: dict[str, set[str]]` mapping *simple* name
(`"pack"`) → the set of *fully-qualified* names that share it (there can be more than one
`pack` across a codebase).

**Pass 2 — Edge resolution.** Every function/method body is walked for `ast.Call` nodes. For a
call like `helper()` or `self.other()`, the callee's simple name (`helper`, `other`) is looked up
in the symbol table, and a directed edge `caller_qualname -> callee_qualname` is added for
**every** qualified symbol sharing that name.

This is a deliberate **name-based resolution** trade-off, not full import-alias tracing: it never
needs to resolve `import numpy as np` chains or re-exports to find a match, which keeps it fast
and dependency-free, at the cost of occasionally over-linking two unrelated functions that happen
to share a name (e.g. two different `run()` methods on unrelated classes). For the purpose this
graph serves — finding *plausibly relevant* neighboring context under a token budget, not proving
program correctness — that trade-off is the right one.

Edges are stored **bidirectionally** for O(1) lookups in both directions:

```python
self._callees: dict[str, set[str]]   # caller -> {callees}
self._callers: dict[str, set[str]]   # callee -> {callers}
```

```python
graph.get_callees("packer.budget.ContextPacker.pack")   # downstream: what `pack` calls
graph.get_callers("packer.budget.ContextPacker.pack")   # upstream: what calls `pack`
```

### 3. Context Packing Algorithm

`ContextPacker.pack(target)` is the formal core of RepoDigest. Given a target (an explicit
symbol, a file path, or a BM25-ranked seed), it runs as follows:

```
1.  roots ← resolve(target)
        - if target is a known symbol:      roots = [target]
        - if target is a file path:         roots = every top-level function/class
                                             defined directly in that file
                                             (nested methods excluded — they're
                                             already inlined in their class's
                                             full source)

2.  order ← roots ++ bfs_alternate(roots)

    bfs_alternate(roots):
        seen ← roots
        frontier_down, frontier_up ← roots, roots
        loop while either frontier is non-empty:
            # one hop of DOWNSTREAM (callees) for every node in frontier_down
            next_down ← unseen callees of frontier_down, sorted, marked seen
            # one hop of UPSTREAM (callers) for every node in frontier_up
            next_up   ← unseen callers of frontier_up,   sorted, marked seen
            append next_down, then next_up, to order
            frontier_down, frontier_up ← next_down, next_up

3.  used ← 0 ;  packed ← [] ;  skipped ← []
    for symbol in order:
        text   ← full_text   if symbol ∈ roots or signatures_only == False
                  else        signature_text
        tokens ← count_tokens(text)
        if used + tokens > budget:
            skipped.append(symbol)      # SKIP — never truncate — keep scanning
            continue
        packed.append((symbol, text, tokens))
        used += tokens

4.  return PackedContext(packed, skipped, total_tokens=used, budget)
```

Three properties fall directly out of this design:

- **Distance-ordered relevance.** Because expansion alternates one hop of downstream, one hop of
  upstream, per round — rather than exhausting one direction first — a symbol two hops downstream
  and a symbol two hops upstream are considered at roughly the same priority, instead of one
  direction systematically starving the other.
- **The budget is never exceeded.** The check is `used + tokens > budget`, evaluated before a
  symbol is ever appended — there is no code path that produces `total_tokens > budget`.
- **The "skip-on-overflow" heuristic.** When a candidate doesn't fit, RepoDigest does not truncate
  it (a truncated function body is often worse than useless — syntactically broken, semantically
  misleading) and does not stop the whole traversal (a heuristic like "stop at first overflow"
  would leave remaining budget on the table if a smaller, more-distant symbol would still fit). It
  skips that one candidate and keeps scanning, so the final packed set is the best-effort greedy
  fit of *closest-first* symbols into the remaining space.

### 4. Local BM25 Ranking Engine

When you don't know the exact symbol name, `repodigest.search.ranker.SymbolRanker` scores every
symbol against a free-text query with a from-scratch BM25 implementation — no external search
library, no vector store.

**Sub-token tokenization.** Symbol and identifier names are rarely single dictionary words —
`compute_loss` and `computeLossValue` both need to match a query containing "loss." The tokenizer
handles both naming conventions:

```python
_CAMEL_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")   # insert a boundary before a capital
_NON_ALNUM = re.compile(r"[^A-Za-z0-9]+")                 # split on everything else, incl. "_"

tokenize("compute_loss")      # -> ["compute", "loss"]
tokenize("computeLossValue")  # -> ["compute", "loss", "value"]
```

**BM25 scoring.** For a query term with document frequency `df` across `N` documents (symbols),
against a document of length `dl` and average document length `avgdl`:

```
idf(term)        = ln( (N - df + 0.5) / (df + 0.5) + 1 )

score(q, doc)     = Σ_{term ∈ q}  idf(term) · tf(term, doc) · (k1 + 1)
                                  ─────────────────────────────────────
                                  tf(term, doc) + k1 · (1 - b + b · dl/avgdl)
```

with the standard tuning constants **k1 = 1.5** and **b = 0.75** — `k1` controls how quickly
additional term occurrences saturate in their contribution to the score, and `b` controls how
strongly a document's length is normalized against the corpus average (`b=0` disables length
normalization entirely; `b=1` fully normalizes).

**Corpus construction.** Each symbol's document is its **qualified name + docstring + rendered
signature** — i.e. its `signature_text` (which already embeds the docstring, per §1) prefixed
with its qualname — deliberately *not* its full implementation body. This keeps a query like
"where is the loss calculated" matching on the vocabulary a human would use to *describe* a
symbol's purpose, rather than being drowned out by common keywords (`self`, `return`, `if`) that
appear inside unrelated function bodies far more often than in any docstring.

**Automatic root seeding.** When `--entry` is omitted and `--query` is given, the CLI builds this
corpus over every symbol in the registry, ranks it, and feeds the single top-scoring symbol
(`SymbolRanker.top(query)`) straight into `ContextPacker.pack(...)` as the root — the query never
touches the packer directly, it only ever selects where the deterministic graph traversal begins.

---

## Installation

Requires Python 3.10+. This project uses a local `venv/` checked in at the repo root.

```bash
# from the repo root
./venv/bin/pip install -e .
```

This installs RepoDigest in editable mode and registers the `repodigest` console script (wired
via `[project.scripts]` in `pyproject.toml`) inside `venv/bin/`.

---

## Complete CLI Reference & Workflows

After installing, use the `repodigest` binary directly (`./venv/bin/repodigest`, or just
`repodigest` once `venv/bin` is on your `PATH`).

### `repodigest pack <path>`

Pack a token-budgeted context around a target symbol, file, or natural-language query.

| Flag | Type | Default | Description |
|---|---|---|---|
| `PATH` | argument | — | Directory to scan (the repo/package root) |
| `--budget` | `int` | `4000` | Token budget for the packed context |
| `--entry` | `str` | `None` | Target symbol (`pkg.mod.Class.method`) or file path, relative to `PATH` |
| `--query` | `str` | `None` | Natural-language query; the top BM25 match seeds the pack |
| `--format` | `xml \| markdown` | `markdown` | Output serialization |
| `--signatures-only` | flag | off | Render non-root dependencies as signatures only (root is always full) |
| `--output` | `path` | stdout | Write to a file instead of printing |

Exactly one of `--entry` or `--query` must be given; passing neither raises a usage error, and an
unmatched `--query` (BM25 score of zero against every symbol) raises a clear CLI error rather than
silently packing nothing.

**Example — target a known symbol, Markdown to stdout (tight budget, to show skip-on-overflow):**

```console
$ repodigest pack src/repodigest --entry packer.budget.ContextPacker.pack --budget 500
```

Real captured output (some fenced code blocks omitted below with `# ...` for brevity — the actual
output includes the full body of every packed symbol):

```markdown
## src/repodigest/packer/budget.py

def pack(self, target: str) -> PackedContext:
    ...
# ... (4 more fully-packed symbols from budget.py and tokenizer.py) ...

---
**Token usage:** 499 / 500 (5 symbols packed, 6 skipped)
```

The loop only ever adds a symbol if it fits, so `total_tokens` never exceeds `budget` — here it
comes right up against the ceiling without going over, with the 6 next-closest candidates skipped
rather than truncated.

**Example — find the relevant code by asking a question instead of naming a symbol:**

```console
$ repodigest pack src/repodigest --query "Where is the token budget enforced?" --budget 4000
```

**Example — keep the target's full body, but dependencies as signatures only, to fit more breadth
into the same budget:**

```console
$ repodigest pack src/repodigest --entry cli.pack_command --budget 3000 --signatures-only
```

**Example — XML output, written to a file for downstream tooling:**

```console
$ repodigest pack src/repodigest --entry graph.call_graph.CallGraph --format xml --output context.xml
$ head -6 context.xml
<context>
  <file path="src/repodigest/graph/call_graph.py">
    <symbol name="graph.call_graph.CallGraph" kind="class" tokens="631">
<![CDATA[
@dataclass
class CallGraph:
```

### `repodigest graph <path>`

Print a Rich-rendered summary table of every module's classes/functions/methods, plus the total
number of call-graph edges — a quick sanity check before packing, or a way to spot which modules
are the most interconnected (and therefore most likely to pull in a lot of context via BFS).

```console
$ repodigest graph src/repodigest
      RepoDigest graph summary: src/repodigest
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓
┃ Module            ┃ Classes ┃ Functions ┃ Methods ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩
│ cli               │       0 │         3 │       0 │
│ graph.call_graph  │       1 │         2 │       6 │
│ packer.budget     │       4 │         1 │       4 │
│ packer.serializer │       0 │         4 │       0 │
│ packer.tokenizer  │       0 │         3 │       0 │
│ parsers.py_parser │       4 │        10 │       0 │
│ search.ranker     │       2 │         1 │       4 │
└───────────────────┴─────────┴───────────┴─────────┘
49 symbols across 7 modules, 56 call edges.
```

---

## Programmatic Python API

Every layer is a plain, importable Python object — the CLI is a thin wrapper over the same calls
you'd make from a script or a custom agent loop.

```python
from pathlib import Path

from repodigest.graph.call_graph import CallGraph
from repodigest.packer.budget import ContextPacker, build_symbol_registry
from repodigest.packer.serializer import to_markdown, to_xml
from repodigest.search.ranker import SymbolRanker

root = Path("src/repodigest")

# Build once, reuse across multiple pack() calls — both are pure functions of the
# source tree, so there's no need to re-parse for every query in a session.
call_graph = CallGraph.from_directory(root)
registry = build_symbol_registry(root)

# Inspect the graph directly
print(call_graph.get_callers("packer.budget.ContextPacker.pack"))   # upstream
print(call_graph.get_callees("packer.budget.ContextPacker.pack"))   # downstream

# Pack a known symbol
packer = ContextPacker(root, budget=4000, signatures_only=True, call_graph=call_graph, registry=registry)
result = packer.pack("packer.budget.ContextPacker.pack")
print(f"{result.total_tokens}/{result.budget} tokens, {len(result.skipped)} symbols skipped")
print(to_markdown(result))

# Seed a pack target from a natural-language question — e.g. inside a custom
# agent loop that needs to find "the right code" before it can act on a task.
corpus = {qual: f"{qual} {record.signature_text}" for qual, record in registry.items()}
ranker = SymbolRanker(corpus)

target = ranker.top("Where is the token budget enforced?")
if target is not None:
    result = packer.pack(target)
    xml_context = to_xml(result)   # feed this directly into an LLM prompt
```

`ContextPacker` accepts an already-built `call_graph`/`registry`, so a long-running process (an
agent server, a REPL) can amortize the one-time AST-parsing cost across many `pack()` calls
against the same snapshot of the repository.

---

## Test Verification & Benchmarks

```bash
./venv/bin/pytest tests/ -v
```

```
============================== 29 passed in 0.21s ==============================
```

| Suite | Tests | Covers |
|---|---|---|
| `tests/test_parser.py` | 6 | Import/function/class/docstring extraction; full vs. signature-only rendering; decorator-line preservation in full mode |
| `tests/test_call_graph.py` | 5 | Cross-file edge construction; upstream/downstream lookups; unknown-symbol handling |
| `tests/test_packer.py` | 6 | BFS traversal order; **strict** budget enforcement (skip-not-truncate); file-entrypoint root resolution; unknown-target errors |
| `tests/test_search.py` | 5 | BM25 tokenization; relevance ranking; `top_k`; empty-corpus edge case |
| `tests/test_cli.py` | 7 | End-to-end `pack`/`graph` behavior across both output formats, `--output`, and error paths |

All fixtures are synthetic, in-repo Python snippets (built via `tmp_path`) rather than mocks —
every test exercises the real `ast` parser, the real call-graph builder, and the real BM25 math.

**On "benchmarks":** RepoDigest does not yet ship a dedicated performance benchmark suite (e.g.
timing packing against a large real-world monorepo). The number above — the full 29-test suite
running in ~0.2s on the author's machine — is the only currently-measured timing figure; treat it
as a smoke-test data point, not a throughput guarantee. Formal benchmarking against larger
codebases is open, tracked work.

---

## Development & Solo Authorship

RepoDigest is authored and maintained by **Nagendra Prasad Konakanchi**
([@nagendra-kon](https://github.com/nagendra-kon)).

This is currently a solo project without a formal contribution process, but issues and pull
requests are welcome:

- **Bug reports / feature requests:** open a GitHub issue with a minimal reproduction (a small
  Python snippet is usually enough, given how the test suite is structured).
- **Pull requests:** please keep changes scoped and include tests — every module in this
  repository (`parsers`, `graph`, `packer`, `search`, `cli`) has a corresponding `tests/test_*.py`
  file, and new behavior should extend the matching one rather than introduce a new pattern.
  Before opening a PR, run the full suite:

  ```bash
  ./venv/bin/pytest tests/ -v
  ```

- **Coding conventions:** the codebase favors small, single-purpose dataclasses and functions
  over inheritance hierarchies, and explicit `from __future__ import annotations` + `X | None`
  style type hints. Runtime dependencies are kept deliberately minimal: `tiktoken` (tokenization),
  `click` and `rich` (the CLI) are actively used by every module described above. `requests` is
  also declared in `pyproject.toml` from the project's initial scaffolding but is not yet imported
  anywhere — it's reserved for a future phase (e.g. calling a remote model API) rather than a
  currently-active dependency; if that phase doesn't materialize, it should be dropped.

Licensed under the [MIT License](LICENSE).
