Metadata-Version: 2.4
Name: silica-core
Version: 0.8.1
Summary: Retrieval tools for coding agents: files, search, read, code_pack, write_note over a folder, no model in the loop
Author: Alessandro Carosia
License-Expression: MIT
Project-URL: Homepage, https://github.com/kiycoh/silica-core
Project-URL: Repository, https://github.com/kiycoh/silica-core
Project-URL: Issues, https://github.com/kiycoh/silica-core/issues
Classifier: Environment :: Console
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Text Processing
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: httpx
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Requires-Dist: python-dotenv>=1.0
Requires-Dist: networkx>=3
Requires-Dist: markdown-it-py
Requires-Dist: orjson
Requires-Dist: snowballstemmer
Requires-Dist: stop-words>=2025.11.4
Requires-Dist: tree-sitter>=0.25
Requires-Dist: tree-sitter-language-pack>=1.13
Requires-Dist: pypdfium2>=5
Requires-Dist: mammoth>=1.9
Requires-Dist: striprtf>=0.0.29
Requires-Dist: xlrd>=2.0
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Requires-Dist: import-linter>=2.0; extra == "dev"
Requires-Dist: ruff>=0.16; extra == "dev"
Provides-Extra: connect
Requires-Dist: websockets>=16.0; extra == "connect"
Provides-Extra: mcp
Requires-Dist: mcp<2,>=1.0; extra == "mcp"
Provides-Extra: dense
Requires-Dist: model2vec>=0.5; extra == "dense"
Requires-Dist: numpy>=1.26; extra == "dense"
Provides-Extra: all
Requires-Dist: silica-core[connect,dense,mcp]; extra == "all"
Dynamic: license-file

<div align="center">

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/banner.svg" />
  <img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/banner-light.svg" alt="Silica Core" width="590" />
</picture>

**Retrieval tools for coding agents. No model in the loop.**

Point it at a folder and the agent you already use gets five tools that return located evidence, never answers.

<p align="center">
  <a href="https://pypi.org/project/silica-core/"><img src="https://img.shields.io/pypi/v/silica-core?style=flat&labelColor=000000&color=000000" alt="PyPI" /></a>
  <a href="https://github.com/kiycoh/silica-core/blob/main/LICENSE"><img src="https://img.shields.io/github/license/kiycoh/silica-core?style=flat&labelColor=000000&color=000000" alt="License" /></a>
</p>

</div>

<!-- The MCP registry proves ownership of the PyPI package by finding this
     string in the description PyPI renders, which is this file. -->
<!-- mcp-name: io.github.kiycoh/silica-core -->

Silica works over the markdown, code, PDFs and office files under one root and
serves them to [Claude Code, Cursor, Zed, Codex and every other harness it
knows](#harnesses), as MCP tools or as shell commands that print the same JSON.
A hit is a path, a section, a line, a window of text and the numbers to judge
it by. Silica never answers, summarises, plans or remembers for you: the
harness owns the loop.

## Install

```bash
uv tool install 'silica-core[mcp]'
```

`pipx install 'silica-core[mcp]'` works the same. The package is `silica-core`,
the command is `silica`, the tools are `silica_*`: the distribution carries the
name, the code keeps the namespace.

## Quickstart

In any folder of markdown, code, PDFs or office files:

```bash
silica init                               # adopt the folder and build the first index
silica search "leveled compaction" -k 5  # return the best located passages
silica setup claude                      # register the MCP server, write the guidance block into ~/.claude/CLAUDE.md
```

<p align="center">
  <img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/search-hit.png" alt="silica search &quot;leveled compaction&quot; over nine LSM papers: the hit carries the file, p. 5 and the passage; beside it the PDF is open on that page with the same passage highlighted" width="900" />
</p>

Nine arXiv papers in a folder, indexed in 2.1 s. The hit names the file, the
page and the passage, and the page beside it is the check, not a promise.

## Benchmarks

Silica's closest comparisons are zvec-grep and ck: local retrieval tools over
files. Each arm received the same BEIR SciFact and NFCorpus documents and
queries. Scores are nDCG@10 (higher is better), shown as SciFact · NFCorpus.

| Mode | Silica | zvec-grep 0.2.2 | ck 0.7.11 |
|---|---:|---:|---:|
| Lexical | **0.662 · 0.311** | 0.649 · 0.297 | 0.630 · 0.289 |
| Hybrid, same `potion-retrieval-32M` embedder | **0.675 · 0.328** | 0.672 · 0.330 | — |

Silica's lexical index needs no model; the competitors serve lexical search
from an index that also includes embeddings. BEIR's published BM25 baselines
are 0.665 · 0.325. On the hybrid row Silica and zvec-grep tie at the 95%
interval: the same vectors rank the same, with no daemon, no vector store and
no bundled model. See [benchmarks](public/benchmarks.md) for corpora, methods,
intervals and commands.

On code, grep then `silica_code_pack` as the only reader was measured against
grep and file reads: twelve repository questions with Haiku 4.5, the four
hardest again with Sonnet 5, and no task separated the two arms on the answer,
as none did with Opus 5 the day before. Where the answer is one symbol's body
in a file not worth reading whole the pack halves the cost, two constants in
two files at $0.067 against $0.034 and a twelve-line label function at $0.054
against $0.033, and on the local task Sonnet pays a third less with it
($0.200 against $0.134) because it finds the symbol inside the pack. Numbers
per model and per task in
[benchmarks](public/benchmarks.md#what-the-harness-measurements-say).

Searching code, not only reading it: with the code index on
(`SILICA_INDEX_CODE`, one unit per function, method, class or constant),
Silica ranks the answering file ahead of zvec-grep on zvec-grep's own twenty
SWE-QA questions with the same embedder, file MRR 0.68 against 0.54, the
interval clear of zero, and ties on twelve local ones. Asked for the way the
plugin now asks (an imperative description, a prompt hook, a CLAUDE.md block),
Sonnet ran the search in 17 of 20 runs and closed the question in 4.7 turns
instead of 6.5, at the same cost, the judge up nine points inside its
interval. Numbers in
[benchmarks](public/benchmarks.md#code-search-against-zvec-grep).

## Tools

| Tool | Shell | Returns |
|---|---|---|
| `silica_files` | `silica files` | the inventory and what the index did with each file: `indexed`, `changed`, `excluded`, `failed`, `unconverted` |
| `silica_search` | `silica search` | ranked passages: path, section, line, raw BM25, matched terms, `coverage`, and the query terms absent from the corpus |
| `silica_read` | `silica read` | a slice by lines or by heading (a page, in a PDF), the outline, and a `version` to carry forward |
| `silica_code_pack` | `silica code-pack` | an AST context pack for one source file inside a character budget |
| `silica_write_note` | `silica write-note` | one atomic write, linted for structure and unresolved wikilinks |

Search indexes markdown and the PDFs that carry a text layer, and, with
`SILICA_INDEX_CODE` set, source files one unit per function, method, class or
constant: a hit's `section` is the symbol, `span` its lines, and
`silica_read(path, section=…)` serves the body. Without the switch a `.py` is
`excluded` in `silica_files` and served by `silica_read` by line. For a symbol
whose name is known, grep wins; for a question that names none, the search
comes first, and the plugin's prompt hook asks the model to say so before it
greps (measured 2026-09-10: with the ask in place the search ran in 17 of 20
runs). The contract, the reply shapes and the acceptance checks are in
[TOOLS.md](TOOLS.md); nothing there needs an API key, a model or a network.

## How search says no

A ranked list always has a top, even when the corpus does not answer. Three
fields show how much the result is worth:

- `coverage`: the share of the query's idf mass the hit's matched terms carry.
  Near 1, every rare term matched; near 0, only common words did.
- `terms_absent`: query terms that occur nowhere in the corpus.
- `matched_terms`: which words this hit actually contains.

<p align="center">
  <img src="https://raw.githubusercontent.com/kiycoh/silica-core/main/assets/search-says-no.png" alt="silica search &quot;raft consensus log replication&quot; over the same nine papers: terms_absent lists raft and consensus, coverage falls to 0.19, and the top hit is a passage about data replication rather than Raft" width="900" />
</p>

`raft consensus log replication` over the same nine papers: `raft` and
`consensus` occur in none of them, `coverage` falls to 0.19, and the top hit
is about data replication, not Raft. Silica exposes the signals and lets the
harness decide whether to stop, read or rephrase.

Ranking is BM25 over documents, then over the heading sections of the top
documents, returning each hit's densest window. `--also QUERY` ranks another
query group separately and fuses the results, useful for searching a concept
and its exact identifiers together. Searches refresh small changes inline;
larger ones return `stale` with `pending` until `silica index` catches up, and
interrupted builds resume. Use `<root>/.silicaignore` to exclude content.

### The dense leg

Optional section embeddings catch paraphrases that share no rare words with
the answer. After they are built, every search fuses them with BM25. Use any
OpenAI-compatible `/v1/embeddings` endpoint:

```bash
ollama pull nomic-embed-text
export SILICA_EMBEDDING_BASE_URL=http://localhost:11434/v1 SILICA_EMBEDDING_MODEL=nomic-embed-text
export SILICA_EMBEDDING_DOC_PREFIX='search_document: ' SILICA_EMBEDDING_QUERY_PREFIX='search_query: '  # what nomic's card asks for
silica index --embed
```

Or use a local static model, with no server:

```bash
uv tool install 'silica-core[mcp,dense]'
export SILICA_EMBEDDING_MODEL=model2vec/minishlab/potion-retrieval-32M
silica index --embed
```

Text leaves the machine only for an endpoint that is not on it, and only after
`silica index --embed --allow-remote` grants that host once. Until then search
stays lexical. The reply always says whether the dense leg ran.

## What it reads

- **PDF with a text layer:** searched directly, one section per page; no conversion or sidecar.
- **Scanned PDF, image, PPTX, XLSX:** `silica import` uses [MinerU](https://github.com/opendatalab/MinerU) to create a guarded `.md` sidecar. Silica reports it as stale if the source changes.
- **DOCX, EPUB, FB2, RTF, XLS, ODF:** converted with no extra.
- **Audio and video:** `ffmpeg` plus `SILICA_STT_BASE_URL`.
- **CSV and other data files:** readable by line but excluded from search.

`silica doctor` says which lanes this machine has. Extras: `[connect]` adds the
Obsidian bridge, `[dense]` the local embedder, `[all]` everything.

## Harnesses

`silica setup <client>` writes the registration into the client's own config.
It backs up existing config and refuses malformed files; for `claude` it also
puts a guidance block between markers into `~/.claude/CLAUDE.md`, when to
search before grep, replaced on a second run and nothing else touched. `silica setup --list`
shows paths for `claude`, `codex`, `cursor`, `windsurf`, `zed`, `cline`, `roo`,
`continue`, `goose`, `opencode`, `openhands`, `gemini`, `dsh`, `hermes`,
`openclaw`, `agent-zero`, `claude-desktop`, `lmstudio`, `anythingllm` and
`librechat`; `shell`, `python` and `generic` print recipes for anything else. By default the server serves the
folder the client opens in; use `--vault DIR` or `SILICA_VAULT` for a fixed root.

`npx skills add kiycoh/silica-core` installs the skill that tells an agent
when to reach for the tools. It installs nothing else: on its own it leaves an
agent holding instructions for tools that are not there. Setup details, shell
recipes, Docker and the optional REPL are in [docs/harnesses.md](docs/harnesses.md).

## Other surfaces

- **Extended tools:** `silica mcp --extended` adds the wikilink tools, `silica_links`, `silica_backlinks`, `silica_orphans` and `silica_unresolved`, over the same root.
- **Obsidian:** `silica connect` hosts the bridge the Obsidian plugin dials into, so writes land through the vault API while the app is open. The five core tools do not need it.
- **REPL:** `silica repl` runs a small reference agent over the same tools. It is the only surface that needs a model (`SILICA_MODEL`).

The core keeps no memory lane, prompt injection, summaries or undo journal.
Undo is git.

## License

MIT. See [LICENSE](LICENSE).
