Metadata-Version: 2.4
Name: mdsift
Version: 0.3.0
Summary: Sift a markdown workspace: BM25F field-weighted search, structural grep, and multi-layer graph exploration (PPR) for AI agents. Live-updating MCP server, no embeddings required.
Project-URL: Homepage, https://github.com/YOUR_USER/mdsift
Project-URL: Issues, https://github.com/YOUR_USER/mdsift/issues
Author: mdsift contributors
License-Expression: MIT
License-File: LICENSE
Keywords: agent,bm25f,knowledge-graph,markdown,mcp,pagerank,retrieval,search,tree-sitter
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Requires-Dist: numpy>=1.24
Requires-Dist: scipy>=1.10
Requires-Dist: tree-sitter-markdown>=0.3
Requires-Dist: tree-sitter>=0.21
Requires-Dist: watchdog>=3.0
Provides-Extra: dev
Requires-Dist: build; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Requires-Dist: twine; extra == 'dev'
Description-Content-Type: text/markdown

# mdsift

**Sift a markdown workspace.** Field-weighted keyword search, structural grep,
and multi-layer graph exploration for AI agents — live-updating, embedding-free,
exposed as an MCP server.

Point it at a folder of markdown. Your agent gets seven tools that stay fresh
as files change, with sub-10 ms queries and zero model downloads.

## Why not just BM25 + a wikilink graph?

Plenty of tools do flat BM25 over markdown. mdsift differs in four ways:

1. **BM25F, not BM25.** Tree-sitter parses every file into fields — title,
   headings, code identifiers, link text, body — and ranking weights them
   (a hit in a title or a function name beats a hit in prose). Code
   identifiers are subtokenized, so both `refreshAccessToken` and `token` match.
2. **A multi-layer graph with agent-controlled weights.** Three edge families —
   explicit links (markdown + wiki-links), folder hierarchy, and statistical
   TF-IDF similarity — queried together via Personalized PageRank. The agent
   dials per-edge-type multipliers per query: "related by links" vs "related
   by vocabulary" vs "related by location" are different questions.
3. **Structural grep.** Regex scoped to *where* in the document: only inside
   code blocks (optionally by language), only in headings, only in link text.
   Powered by ripgrep underneath, so it is fast and always index-fresh.
4. **No embeddings.** Pure lexical + graph statistics. Instant cold start,
   fully offline, no GPU, no model cache.

## Install

```bash
pip install mdsift          # once published; from source: pip install -e .
# ripgrep must be on PATH:  apt install ripgrep | brew install ripgrep
```

## Use

```bash
# MCP server (stdio) for Claude Desktop / Claude Code / any MCP client
mdsift-server /path/to/workspace

# CLI
mdsift /path/to/workspace search  '{"query": "auth retry", "top_n": 5}'
mdsift /path/to/workspace grep    '{"pattern": "retry.*backoff", "scope": "code", "lang": "python"}'
mdsift /path/to/workspace explore '{"seeds": {"docs/auth.md": 1.0}, "preset": "semantic"}'
mdsift /path/to/workspace clusters '{"min_size": 5}'
```

MCP client config:

```json
{"mcpServers": {"mdsift": {"command": "mdsift-server", "args": ["/path/to/workspace"]}}}
```

## Tools

| Tool | Use when |
|---|---|
| `search` | Fuzzy/conceptual queries; BM25F over weighted fields. Hits include the best-matching **section snippet**, so agents often skip the full-file read. `expand=true` merges graph-related files. |
| `read_section` | Fetch one section by heading (fuzzy-matched) — the token-efficient follow-up to search. |
| `grep` | Exact strings / regex. `scope=code\|headings\|body\|links`, `lang` filter, path glob/regex. Never stale. |
| `deps` | Direct neighborhood of one file: links in/out, folder relations, similarity edges. |
| `explore` | PPR from seed files. Presets `structural` / `semantic` / `spatial` / `balanced`, or explicit weights (`0` prunes a layer). |
| `clusters` | Topic map: graph communities + TF-IDF labels (preset `topics`). |
| `relate` | Weighted shortest connection between two files. |
| `status` | Corpus, vocabulary, and edge counts. |

Composition patterns agents use well: `grep` for candidates → `search` to rank
them; `search` for seeds → `explore` to widen context; `clusters` on first
contact with an unfamiliar workspace.

See [docs/token-efficiency.md](docs/token-efficiency.md) for an honest
analysis of when this saves agent tokens — the short version: ranked results
and PPR exploration replace multi-turn read loops, and section snippets attack
the file-consumption side, where most tokens actually live.

All tools accept `format` (`toon` default — 45–60% fewer tokens on tabular
results — or `json`), `max_tokens` (server-side response budget), and search
accepts `session` (repeat-snippet dedup), `granularity` (`file`/`section`),
`tags` (frontmatter filter), and `recency` (opt-in freshness boost).
Frontmatter parsing covers Agent Skill (SKILL.md) files: multi-line block-scalar
descriptions and nested `metadata:` are parsed, and skill `name`/`description`
are indexed so skills are searchable by capability. Format coverage is
tested against realistic CLAUDE.md (including `@path` imports, which become
graph edges), AGENTS.md (root + nested per-directory), MADR and Nygard-style
ADRs, and SKILL.md fixtures — see tests/test_agent_formats.py — plus the
full GitHub Copilot customization surface (copilot-instructions.md,
*.instructions.md/applyTo, *.prompt.md, *.chatmode.md, *.agent.md,
.github/skills), Cursor .mdc rules (indexed by default), and Claude Rules —
see tests/test_copilot_formats.py. Hidden directories (.github, .cursor,
.claude) are scanned, grepped, and watched.

## Performance

Measured on a synthetic 2,000-file workspace (`python bench.py` reproduces):

| Path | Latency |
|---|---|
| Cold build | ~2.5 s |
| `search` | p50 ≈ 3 ms |
| `explore` (PPR over 4 layers) | p50 ≈ 5 ms |
| `grep` scoped | p50 ≈ 45 ms (ripgrep-bound) |
| `clusters` cold / cached | ~300 ms / ~3 ms |
| **Live file update** (parse → index → graph) | **p50 ≈ 0.5 ms** |

How: columnar postings (`int32` ids + `uint16` tfs) with numpy-vectorized
scoring and a delta/tombstone tier for updates; per-edge-type scipy CSR
matrices summed at query time so agent-supplied weights cost microseconds;
similarity edges maintained via rare-term candidate shortlisting (never
all-pairs); folder nodes with capped sibling edges keep the graph linear.
One index serves both ranking and similarity vectors.

## Architecture

```
watchdog (200 ms debounce)
  └─ tree-sitter (block + inline grammars)
       ├─ BM25F index  ── search, similarity vectors
       ├─ span index   ── structural grep (via ripgrep --json)
       └─ graph        ── link / sibling / parent / similar layers
                          └─ PPR, clusters, shortest path
```

Update cost is O(changed file + shortlisted neighbors), never O(corpus).

## Rust migration path

The data layout (interned ids, columnar postings, CSR layers) and the MCP JSON
contract port 1:1 to Rust. First step if ever needed: move `index.search` and
`graph.ppr` into a PyO3 extension; nothing else changes.

## Development

```bash
pip install -e ".[dev]"
pytest tests/ -q      # 11 tests
python bench.py       # correctness + latency suite on 2,000 generated files
```

## License

MIT. Dependencies: numpy/scipy (BSD-3), tree-sitter + grammar (MIT),
watchdog (Apache-2.0); ripgrep (MIT/Unlicense) is invoked as a subprocess,
not bundled.
