Metadata-Version: 2.4
Name: refs-mcp
Version: 0.2.0
Summary: FastMCP server for managing the refs/ reference-repo tree
License: MIT
Requires-Python: >=3.11
Requires-Dist: click>=8.1
Requires-Dist: fastmcp>=3.3.1
Requires-Dist: githubkit>=0.13
Requires-Dist: opentelemetry-api>=1.20
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20
Requires-Dist: opentelemetry-sdk>=1.20
Requires-Dist: pydantic>=2.6
Requires-Dist: tree-sitter-language-pack>=1.8
Requires-Dist: tzdata>=2024.1
Description-Content-Type: text/markdown

# refs

You have a `~/dev/refs/` folder too, don't you?

The one where every time you want to read someone's codebase — study
`drizzle-orm`'s query builder, skim `next.js` internals, grep through
`playwright` — you run

```bash
git clone https://github.com/foo/bar.git ~/dev/refs/bar
```

and then months later you've got a hundred loose checkouts at the top
level, half of them stale, two of them accidentally the same repo under
different names, and no idea which ones are your work vs. someone else's
code. You keep meaning to organize it. You never do. Nobody wants to
write a housekeeping tool for their own scratch directory.

This is that tool.

## What it does

Run `./reorg.sh --apply` and your pile becomes this:

```
refs/
├── facebook/
│   └── react/
├── vuejs/
│   └── vue/
├── microsoft/
│   ├── playwright/
│   └── ...
└── ... (one directory per GitHub owner)
```

No matter where a clone was before — flat at top, nested in a
mis-named folder, duplicated under two names — it ends up at
`<owner>/<repo>/` based on its actual `git remote`. Duplicate clones of
the same upstream get parked aside for review instead of clobbering.
Random non-repo content (your notes, binaries, scratch dirs) gets evicted
out of the tree entirely. Empty pseudo-owner folders get pruned.

Every move is journaled. `./reorg.sh undo --apply` reverses the most
recent run. Nothing magical — plain `mv`, `rmdir`, `git`.

## The daily loop

```bash
./reorg.sh --apply     # you cloned some more stuff; put it where it belongs
./status.sh            # quick look at what's dirty and what has updates
./update.sh            # git pull every clean repo with remote changes
```

## About the git repo you're looking at

This directory is itself a git repo, but it doesn't track the hundreds of
cloned reference repos — those are checkouts you can always re-clone. It
only tracks the tooling (`reorg.sh`, `status.sh`, `update.sh`) plus the
two generated indexes (`index.md`, `index.html`). A whitelist-mode
[`.gitignore`](.gitignore) enforces this. More on that below.

## Tools

### `reorg.sh`

Reorganizes loose clones into the `<owner>/<repo>/` layout, keeps an index
(`index.md`, `index.html`) up to date, evicts non-repo content out of the
collection, and journals every move so you can reverse it.

```bash
./reorg.sh                    # dry-run preview
./reorg.sh --apply            # perform moves, write journal
./reorg.sh undo --apply       # reverse the most recent apply
./reorg.sh --help             # full option list
```

Destinations (overridable):

| purpose | default path |
| --- | --- |
| duplicate-URL losers | `$(dirname $ROOT)/!CONFLICTS-RESOLVE/` |
| non-repo top-level content | `$(dirname $ROOT)/!WORK-PRODUCT-FROM-REFS/` |
| apply journals | `${XDG_STATE_HOME:-~/.local/state}/reorg/<basename-of-ROOT>/` |

### `status.sh`

Reports dirty/clean + has-updates for every repo in the tree. Walks depth 1
and 2, so both loose-at-top and `<owner>/<repo>/` layouts are covered.

### `update.sh`

`git pull` every clean repo that has remote changes. Skips dirty repos with
a note.

### `test-reorg.sh`

Integration test harness for `reorg.sh`. Runs `shellcheck`, then 42 tests
over synthetic git fixtures (fake `.git/config` files — no network, no
real repos needed).

```bash
./test-reorg.sh                 # full suite
./test-reorg.sh --no-shellcheck # skip the lint step
```

## MCP server (FastMCP)

`refs_mcp/` ships a [FastMCP][fastmcp] server that exposes the same
operations as MCP tools, with structured outputs typed by Pydantic. Useful
when an agent wants to inspect or operate on the tree through the
[Model Context Protocol][mcp] instead of shelling out.

### Install + run

```bash
uv sync --frozen                      # install deps from uv.lock
uv run --frozen refs stdio        # serve over MCP stdio
uv run --frozen refs http         # serve over streamable-HTTP (daemon)
uv run --frozen refs --help       # full CLI surface
```

The CLI is a [click][click] group with two transport subcommands —
`stdio` for Claude Desktop / IDE clients that spawn the binary, and
`http` for daemons under systemd / Docker / k8s. Anchored to the
canonical pattern in
[`../refs/modelcontextprotocol/servers/src/git/`][mcp-git-ref] and
[`../refs/github/github-mcp-server/`][gh-mcp-ref].

Or build a frozen binary and invoke it directly:

```bash
uv run --frozen python -m refs_mcp._build     # produces dist/refs.exe
./dist/refs stdio                              # no Python needed at run time
```

The build wrapper installs a `logging.Handler` on PyInstaller's logger
that `sys.exit`s on every record at WARNING or above — warnings genuinely
fail the build. There is no `--werror` flag in PyInstaller or uv
(verified via JiT reads of both upstreams).

The canonical dev commands use `uv run --frozen` so dependency
resolution is locked to `uv.lock` and uv never rewrites the lockfile as
a side effect:

```bash
uv run --frozen pytest tests/                  # 177 tests
uv run --frozen ruff format refs_mcp tests     # format
uv run --frozen ruff check  refs_mcp tests     # lint
uv run --frozen pyright refs_mcp               # static type check
uv run --frozen refs selftest                  # live MCP smoke (in-process)
uv run --frozen refs selftest --binary dist/refs.exe   # vs the frozen binary
```

CI is a three-tier pipeline in `.github/workflows/ci.yml`, fronted by
a tiny `changes` gate. Capability is preserved across tiers — macOS,
Python 3.11/3.13, PyInstaller build, MCP stdio smoke, artifact upload,
and release upload are all defined in the workflow. What varies per
event is which tier fires AND whether the cheap default tier fires at
all on PR/branch-push (a docs-only change skips it entirely — Windows
never starts).

| Event | `changes` | `test` | `test-full` | `build` | `release` |
|---|:---:|:---:|:---:|:---:|:---:|
| pull_request to `main` (code change) | ✓ | ✓ | — | — | — |
| pull_request to `main` (docs only) | ✓ | — | — | — | — |
| push to `main` (code change) | ✓ | ✓ | — | — | — |
| push to `main` (docs only) | ✓ | — | — | — | — |
| `workflow_dispatch` with `full-ci: true` | — | ✓ | ✓ | ✓ | — |
| push tag `v*` | — | ✓ | ✓ | ✓ | ✓ |

**`changes`** is a cheap ubuntu probe (`dorny/paths-filter@v4`) that
flags whether this push touched any runtime-relevant file:
`refs_mcp/**`, `tests/**`, `pyproject.toml`,
`uv.lock`, `refs.spec`, `.github/workflows/ci.yml`, or any `*.sh`.
If only docs/comments/README changed, every downstream tier skips and
the PR's required check passes without spinning up Windows. The gate
itself is bypassed on `workflow_dispatch` + tag push (the operator
explicitly asked for the full run).

**`test`** is the default cheap signal — `ubuntu-latest` and
`windows-latest` on Python 3.12 only. Linux runs ruff + pyright +
shellcheck + bash `test-reorg.sh` + pytest; Windows runs ruff +
pytest. No macOS, no extra Python versions.

**`test-full`** is extra source-pytest coverage: `{ubuntu, windows}`
× `{3.11, 3.13}` + `macos-latest/3.12`. Source pytest only, no extra
gates (pyright + shellcheck are version-independent and already
covered by the canonical `test` cell).

**`build`** produces per-OS PyInstaller binaries pinned to py3.13
(the shipped interpreter), runs a CLI smoke + the full 6-tool live
MCP stdio smoke against three shallow-cloned target repos
(`MicrosoftDocs/mcp` + `BurntSushi/ripgrep` +
`modelcontextprotocol/python-sdk`), and uploads each binary as an
artifact (retention 7 days for non-tag runs).

**`release`** downloads the three per-OS binaries and attaches them
to a GitHub release via [`softprops/action-gh-release@v3`][gh-release].

To run the full sweep manually (e.g. before cutting a release), use
the Actions tab and dispatch the `ci` workflow with `full-ci: true`.

Set `REFS_ROOT` to point at a tree other than the current directory.
`REFS_CONFLICTS_DIR`, `REFS_WORK_PRODUCT_DIR`, and `REFS_JOURNAL_DIR`
override the matching reorg.sh defaults.

Add to a Claude Code project's `.mcp.json`:

```json
{
  "mcpServers": {
    "refs": {
      "command": "uv",
      "args": ["run", "refs", "stdio"],
      "cwd": "<absolute path to your refs/ tree>"
    }
  }
}
```

Or, when the frozen binary is on PATH:

```json
{
  "mcpServers": {
    "refs": {
      "command": "refs",
      "args": ["stdio"],
      "cwd": "<absolute path to your refs/ tree>"
    }
  }
}
```

### Tools

Evidence-graded search + symbol extraction (read-only by construction):

| Tool                       | Read-only | Implementation |
| -------------------------- | --------- | -------------- |
| `refs_find_repo`           | yes       | Bounded-scope repo lookup by `owner/repo`, exact name, or fuzzy substring. |
| `refs_search_evidence`     | yes       | `rg --json` content search with verdict labels (MATCH / VALIDATED_EMPTY / INVALID_EMPTY / TRUNCATED / FAILED / SKIPPED_UNSAFE). |
| `refs_prove_absence`       | yes       | Single-pass literal probe + positive control; strict VALIDATED_EMPTY or FAILED. |
| `refs_inspect_terms`       | yes       | Batch verdict map over a list of terms in one repo. |
| `refs_list_symbols`        | yes       | Language-agnostic symbol extraction via [`tree-sitter-language-pack`][tslp] (306 languages) with `DEFINED_AND_TESTED` / `DEFINED_ONLY` / `DEFINED_PRIVATE` verdicts. |
| `refs_find_symbol`         | yes       | Resolve one symbol's signature + tests + head SHA across any supported language. |
| `refs_host_tools`          | yes       | Probe-based inventory of host CLIs (rg, git, gh, …) + Python-module gates (tree-sitter) with feature-gate map. |
| `refs_discover_remote`     | yes       | `gh search repos` for cross-org discovery (needs `gh` on PATH). |
| `refs_discover_remote_org` | yes       | `gh repo list <owner>` for enumerating an org's repos. |
| `refs_preseed_run`         | no        | Auto-clone the curated upstream-reference set (FastMCP, MCP SDK + spec, Pydantic, ripgrep, OTel semconv, pgr, MicrosoftDocs/mcp). |

### Symbol extraction (language-agnostic)

`refs_list_symbols` and `refs_find_symbol` route per-file to an
extractor stack modeled on the [Sourcegraph + GitHub code-nav
architecture][scip]:

| Tier          | Extractor                                          | Status |
| ------------- | -------------------------------------------------- | ------ |
| **primary**   | tree-sitter via [`tree-sitter-language-pack`][tslp] (306 languages) | shipping |
| **enrichment** | SCIP / LSP semantic indexes — override structural records with semantic-precise resolution (overloaded / imported / external symbols). | protocol locked; future work |
| **fallback**  | Universal Ctags subprocess — for languages tree-sitter doesn't ship a grammar for | protocol locked; future work |

The `Symbol` IR carries `name`, `kind` (function / async_function /
method / class / struct / enum / trait / interface / type_alias / impl
/ constant / variable), `language`, `signature`, `line`, `end_line`,
`visibility`, `parent`, `decorators`, `extractor` (which extractor
produced this record — useful for verdict explainability), plus the
`DEFINED_AND_TESTED` / `DEFINED_ONLY` / `DEFINED_PRIVATE` / `NOT_FOUND`
verdict.

Across the smoke targets, tree-sitter extracts:

| Repo                                | Language     | Symbols |
| ----------------------------------- | ------------ | ------- |
| `modelcontextprotocol/python-sdk`   | Python       | 631     |
| `MicrosoftDocs/mcp`                 | TypeScript   | 100     |
| `BurntSushi/ripgrep`                | Rust         | 3472    |

Tree management (every tool runs in native Python or git/gh subprocess — no shell wrappers):

| Tool                       | Read-only | Implementation |
| -------------------------- | --------- | -------------- |
| `refs_help`                | yes       | Returns the typed `HelpDocument` — read this first per session. |
| `refs_status`              | yes       | Filesystem walk + `configparser` on `.git/config`. Offline. |
| `refs_discover`            | yes       | Same walk; optionally writes `$REFS_JOURNAL_DIR/refs.inventory.json`. |
| `refs_reorg_preview`       | yes       | Pure planner — builds `ReorgPlan` from typed inventory, no IO. |
| `refs_reorg_apply`         | no        | `pathlib` + `shutil.move` + `csv.writer` for the TSV journal. |
| `refs_reorg_undo`          | no        | `csv.reader` reads the journal; moves reversed with `shutil`. |
| `refs_clone`               | no        | `gh repo clone` (github.com, auth-aware) or `git clone`; dry-run default; refuses non-GitHub and overwrites. |
| `refs_update_check`        | yes       | `git status --porcelain=v2 --branch` + `git ls-remote` for `has_updates`. |
| `refs_update_apply`        | no        | `git stash push` → `git pull --ff-only` → `git stash pop`; per-repo concurrency. |
| `refs_degit_export`        | no        | `git archive` into an export dir; source clone untouched. |
| `refs_sparse_materialize`  | no        | `git sparse-checkout set`; traversal patterns refused. |
| `refs_index_generate`      | no        | Pure-Python renderer of `index.md` + `index.html` from typed `Inventory`. |
| `refs_journal_latest`      | yes       | `csv.reader` over the latest TSV journal. |
| `refs_events_tail`         | yes       | Read the JSONL operation events log. |

### Feature gates (host_tools probe)

Each tool that depends on a host CLI is gated by the bootstrap
`host_tools` probe — `shutil.which` is not enough; the probe **runs**
each tool's `--version` and only marks it OK on success. Features that
need an absent tool return a typed `FAILED` verdict with the install
hint, not a silent zero. See `refs_host_tools` for the live snapshot.

Per-feature gates land in `refs_mcp.host_tools._CURATED_FEATURES`:

| Feature                  | Required tool(s)            | Used by |
| ------------------------ | --------------------------- | ------- |
| `content_search`         | `rg`                        | `refs_search_evidence`, `refs_prove_absence`, `refs_inspect_terms` |
| `file_enum`              | `rg`                        | Positive-control probes, corpus building |
| `structural_symbols`     | Python `tree_sitter_language_pack` | `refs_list_symbols`, `refs_find_symbol` (gated on Python import, not a CLI) |
| `auto_clone`             | `gh` (preferred) or `git`   | `refs_mcp.auto_clone.ensure_repo` |
| `git_status`             | `git`                       | `refs_update_check` sweep |
| `git_pull` / `git_stash` | `git`                       | `refs_update_apply` sweep |
| `git_archive_local`      | `git`                       | `refs_degit_export` |
| `git_sparse_checkout`    | `git`                       | `refs_sparse_materialize` |
| `remote_repo_search`     | `gh`                        | `refs_discover_remote` |
| `remote_org_list`        | `gh`                        | `refs_discover_remote_org` |
| `remote_archive_tarball` | `gh`                        | Future: archive without local clone (`gh api repos/{o}/{r}/tarball/{ref}`) |
| `remote_metadata`        | `gh`                        | Default-branch / head SHA / disk size without clone |
| `pre_clone_size_inspection` | `gh`                     | `refs_discover_remote` populates `DiscoverableRepo.size_kb` |

### `~/.refs/` layout (XDG-friendly)

Bootstrap writes per-run artifacts to a user-level layout (overridable
via env vars; defaults follow the XDG Base Directory spec on Linux).
Actual on-disk layout as implemented in `refs_mcp.user_config.UserPaths`:

```
~/.refs/
├── .refs.toml                  # user preferences (auto_clone_allowed_hosts, etc.)
├── repos/                      # default refs_root (cloned upstreams)
│   └── <owner>/<repo>/
├── state/
│   ├── events.jsonl            # structured operation events
│   ├── server-runs.jsonl       # one entry per server boot (run_id, transport, env)
│   ├── logs/                   # date-rotated structured file logs (trace_id-correlated)
│   └── traces/                 # OTel spans as JSONL per run (file exporter)
├── data/
│   └── replay/                 # captured request/response artifacts per run
└── cache/
    └── symbol-index/           # symbol-index cache keyed by (repo_path, head_sha)
```

Override individual subtrees with `XDG_STATE_HOME` / `XDG_DATA_HOME` /
`XDG_CACHE_HOME` (each gets `/refs` appended), or the more-specific
`REFS_STATE_DIR` / `REFS_DATA_DIR` / `REFS_CACHE_DIR`. Setting
`REFS_HOME` overrides the **whole tree** — XDG vars are ignored when
REFS_HOME is set so the layout stays cohesive (no surprise splits). The
config file (`.refs.toml`) lives directly under the home root; relocate
it via `REFS_HOME`.

The shell scripts (`reorg.sh`, `status.sh`, `update.sh`) remain in the
repo for shell-CLI users. The MCP server's reorg/status/update paths do
not call them — they share the on-disk contract (whitelist `.gitignore`,
TSV journal format, `<owner>/<repo>` layout) so a journal written by
either path is reversible by the other.

### Resources

| URI                            | MIME                       | Content |
| ------------------------------ | -------------------------- | ------- |
| `refs://inventory`             | `application/json`         | Structured `Inventory` of the tree (typed `ResourceResult`). |
| `refs://inventory/schema`      | `application/schema+json`  | JSON Schema of the `Inventory` model. |
| `refs://repo/{owner}/{repo}`   | `application/json`         | One typed `RepoRecord` by owner+repo. |
| `refs://config`                | `application/json`         | Sanitized server config (no secrets). |
| `refs://journal/latest`        | `application/json`         | Latest reorg journal, parsed. |
| `refs://events/latest`         | `application/x-ndjson`     | Tail of the structured operation events log (NDJSON). |
| `refs://index.md`              | `text/markdown`            | `index.md` rendered from typed `Inventory`. |
| `refs://index.html`            | `text/html`                | `index.html` rendered from typed `Inventory`. |
| `refs://help`                  | `application/json`         | Typed `HelpDocument` — same content as the `refs_help` tool. |
| `refs://help/markdown`         | `text/markdown`            | Help rendered as markdown for humans. |

### Prompts

| Name                            | Purpose |
| ------------------------------- | ------- |
| `refs-agent-onboarding`         | Read-this-first onboarding for any agent connecting to the server. |
| `audit-refs-tree`               | Audit-before-mutate workflow. |
| `clone-reference-repo`          | Safe clone workflow (refs_clone preview → apply → verify). |
| `refresh-reference-tree`        | status → check → update sequence. |
| `prepare-agent-reference-pack`  | Surface local upstream docs from the inventory for a given topic. |
| `export-plain-source-snapshot`  | degit-style snapshot via `refs_degit_export`. |

### Help surface (lessons baked into the server)

`refs_help` and the two `refs://help[/markdown]` resources expose a typed
guide for anyone — operator or agent — about to use this server. The
content is generic and transferable: which tools are offline vs
network-bound, why generated indexes are never parsed as data, what the
journal contract is for undo, why the OpenTelemetry tracer is a no-op
without an exporter, why owner/repo names with `.`, `..`, control chars,
or Windows reserved names are rejected, and which upstream docs are
already cloned under the refs root. Tip severities (`info` / `warning`
/ `critical`) match how aggressively a client should surface them.

### Generated indexes are not canonical

`index.md` and `index.html` are presentation artifacts produced by
`reorg.sh`. The MCP server's source of truth is the structured inventory
built from a filesystem walk + `.git/config` reads. No tool in this
package parses `index.md` or `index.html` as data.

### Mid-2026 MCP / Claude features in scope

What the server uses today:

- **Structured tool outputs and output schemas** — every tool returns a
  Pydantic model, so FastMCP emits both `content` (legacy text JSON) and
  `structuredContent` with a generated `outputSchema` (MCP 2025-11-25 §
  Tool Result).
- **Tool annotations** — `read_only_hint`, `destructive_hint`,
  `idempotent_hint`, `open_world_hint` are set on every tool so the
  client can render safe-vs-destructive UI.
- **Resources with MIME** — JSON resources are addressable and
  discoverable via `resources/list`.
- **Prompts** — `audit-refs-tree` teaches an agent the right
  read-before-mutate workflow.
- **Progress + log notifications** — `refs_reorg_apply` and
  `refs_update_apply` report bracketed progress to the MCP client via
  `ctx.report_progress` and emit structured status via `ctx.log`. The
  asyncio streaming runner in `refs_mcp.runner` is the canonical
  subprocess entrypoint and remains available for any future tool that
  needs line-by-line subprocess forwarding.
- **OpenTelemetry** — FastMCP emits per-call SERVER spans automatically;
  this package adds a CLIENT-kind span around every subprocess call
  (git, gh, ripgrep) carrying `process.command`, `process.command_args`,
  `process.working_directory`, `process.exit.code`, duration, and
  ERROR status on non-zero exits. Bootstrap (`refs_mcp.bootstrap`) wires
  the `TracerProvider` once per process with a file `SpanExporter`
  writing to `~/.refs/traces/<run_id>.jsonl` so every run has captured
  spans on disk by default. OTLP export is also wired:
  - `OTEL_EXPORTER_OTLP_ENDPOINT=…` adds the `OTLPSpanExporter`
    (proto-http) via `BatchSpanProcessor`. The exporter reads the
    standard `OTEL_EXPORTER_OTLP_*` configuration variables. The
    dependency ships as a hard runtime dep — no `[otlp]` extra dance.
  - `OTEL_SERVICE_NAME=…` (default `refs`) attaches a meaningful
    `service.name` Resource attribute. W3C trace context, semconv
    `service.*` / `host.*` / `process.*` / `os.*` attributes are
    populated by `refs_mcp.run_metadata`.

What is deliberately not used:

- **Sampling** — server-requests-LLM. This server doesn't generate text;
  no natural use case.
- **Elicitation** — destructive ops are gated by `destructive_hint` and
  by the default-dry-run contract; a separate "ask the human" round-trip
  adds friction without safety.
- **Roots** — server is launched in a specific working directory (or via
  `REFS_ROOT`); root negotiation isn't useful when the path is part of
  the launch contract.
- **Background tasks (`execution.taskSupport`, SEP-1686)** — would
  require a runner like Docket; advertised once an MVP integrates one.
- **Resource subscriptions / `subscriptions/listen`** — useful for
  watching inventory changes but out of scope for the initial drop.

### Tests

```bash
uv run pytest tests/
```

The pytest suite covers path safety, GitHub URL parsing, subprocess
runner (including timeout and missing-executable paths), discovery on a
synthetic tree, native reorg / git_ops / operations behavior over the
git/gh subprocess paths (status-sweep, update-sweep, clone, archive,
sparse-checkout), verdict-graded search + symbol extraction, host_tools
probe + feature-gate derivation, auto-clone allowlist + path safety,
MCP server smoke (tools/resources/prompts list, structured inventory
return, annotation presence), OpenTelemetry span emission, and the
benchmark harness.

The pre-existing `./test-reorg.sh` Bash suite remains authoritative for
filesystem moves, conflicts, eviction, and journal/undo semantics.

[fastmcp]: https://github.com/jlowin/fastmcp
[mcp]: https://modelcontextprotocol.io/specification
[click]: https://github.com/pallets/click
[mcp-git-ref]: https://github.com/modelcontextprotocol/servers/tree/main/src/git
[gh-mcp-ref]: https://github.com/github/github-mcp-server
[tslp]: https://github.com/kreuzberg-dev/tree-sitter-language-pack
[scip]: https://github.com/sourcegraph/scip
[gh-release]: https://github.com/softprops/action-gh-release

## `.gitignore` is a whitelist

`refs/.gitignore` uses whitelist-mode:

```
/*
!/reorg.sh
!/test-reorg.sh
...
```

Everything at the top level is ignored *except* the explicitly un-ignored
entries. GitHub owner directories (over a hundred of them, in practice)
are correctly excluded — git never sees them, so no embedded-repo
warnings, no accidental staging.

To keep a new top-level file, add a `!/filename` line. `reorg.sh`'s
top-level allowlist also reads this file: anything not on the static or
dynamic (GitHub owner) allowlist will be evicted to
`!WORK-PRODUCT-FROM-REFS/` on the next `--apply`.
