Metadata-Version: 2.4
Name: interactive-semantic-hunt
Version: 0.6.0
Summary: Interactive semantic search for code, inspired by fzf
Author-email: David Kristiansen <david@kristiansen.tech>
License-Expression: MIT
Project-URL: Homepage, https://github.com/davetothek/interactive-semantic-hunt
Project-URL: Changelog, https://github.com/davetothek/interactive-semantic-hunt/blob/main/CHANGELOG.md
Project-URL: Repository, https://github.com/davetothek/interactive-semantic-hunt
Project-URL: Issues, https://github.com/davetothek/interactive-semantic-hunt/issues
Keywords: search,semantic,embeddings,code,fzf,tui,mcp
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2.5.2
Requires-Dist: pyyaml>=6.0
Requires-Dist: textual>=0.70.0
Requires-Dist: tree-sitter>=0.26.0
Requires-Dist: tree-sitter-c>=0.24.2
Requires-Dist: tree-sitter-cpp>=0.23.4
Provides-Extra: llama
Requires-Dist: llama-cpp-python>=0.2.75; extra == "llama"
Requires-Dist: huggingface-hub>=0.23.0; extra == "llama"
Provides-Extra: st
Requires-Dist: sentence-transformers; extra == "st"
Dynamic: license-file

# ish — Interactive Semantic Hunt

Semantic search for code, inspired by `fzf`. Point `ish` at a directory. It parses
each source file into named chunks, embeds them with a local model, and ranks them
against your query. Everything runs on your machine.

Languages: Python, C and C++, Markdown, AsciiDoc, YAML, and JSON. Documentation
and configuration are indexed beside the code they describe, so one query
searches all of it.

## Install

```sh
pip install interactive-semantic-hunt
```

Nothing in that install compiles: the default backend reaches Ollama over
HTTP with the standard library. A backend that runs the model in this
process is an extra — `[llama]` for llama.cpp, `[st]` for
sentence-transformers.

To work on ish itself, clone it and run `uv sync`.

The default embedding backend is Ollama, which keeps the model resident so no
run pays a model load. Start it once and pull the embedding model:

```sh
ollama serve
ollama pull nomic-embed-text
```

Set `OLLAMA_HOST` to reach a daemon elsewhere.

Two other backends need no daemon:

- `--embedder llama.cpp` downloads a GGUF model and loads it per run. Slower per
  query, faster for a first index of a large tree. Needs the `[llama]` extra.
- `--embedder st` uses sentence-transformers. Needs the `[st]` extra.

## Use

List every chunk under a path:

```sh
ish src/
```

```
src/ish/domain/chunk.py:7-28  class  Chunk
```

Search for a query:

```sh
ish "parse a python file" src/
```

```
[0.71] src/ish/adapters/parser/python.py:14-31  method  PythonParser.parse
```

Run the interactive picker and open the selection in your editor. Type to
search, `up`/`down` or `ctrl+p`/`ctrl+n` to move, `enter` to choose, `ctrl+t`
to change the theme, `escape` to quit. The picker lists what the index held
last time before it looks for changes, and answers from that while the
refresh runs, with the progress in the header. Narrow without leaving the
query line:

```text
state machine transitions              every language
lang:cpp state machine transitions     the implementation
lang:yaml under:/10.System/ state      the tests that cover it
type:doc how do I configure this       the prose, not the code
type:test,doc retry backoff            the tests and what they document
```

Press Tab to finish a filter word, in the picker and in Neovim alike. `ty`
becomes `type:`, `lang:cp` becomes `lang:cpp`, and `under:/s` becomes
`under:/src/`; a word with several answers grows as far as they agree and
names the rest. `ish-complete` does the work from the command line, and
`complete_filter` does it over MCP, so any picker can call whichever it has.

The shell finishes the same words on the command line. Source
`contrib/shell/ish.bash` from `~/.bashrc`, or copy `contrib/shell/_ish` to a
directory on zsh's `$fpath`, and `ish lang:cp<Tab>` becomes `ish lang:cpp`.
Both call `ish-complete`, so the shell learns nothing about the languages.

`lang:`, `under:`, and `type:` work in the query line of every interface —
the command line, the picker, Neovim, and MCP. The words are taken out
before the query is embedded, so the model sees the question rather than
how it was narrowed.

A language may be named however it comes to mind. `c`, `c++`, `cxx`, `h`, and
`hpp` all mean `cpp`, because one parser reads them all; `adoc` means
`asciidoc`, `md` means `markdown`, `py` means `python`, and `yml` means `yaml`.

`type:` sorts every chunk into exactly one kind. A path decides before a
language does, so a YAML fixture counts as a test rather than as config:

| kind | what it holds |
|---|---|
| `code` | Python, C, C++, and anything a plugin parser adds |
| `doc` | Markdown and AsciiDoc |
| `test` | anything under `tests/`, `spec/`, `fixtures/`, plus `test_*` and `conftest.py` |
| `config` | YAML, JSON, and TOML outside a test path |

A repository that names its trees its own way can say so, in
`.ish/config.toml`:

```toml
type_patterns = [
  "test:/[0-9.]*(Tests|Verification)/",
  "doc:/[0-9.]*Specification/",
]
```

The first match wins; anything unmatched keeps the reading above.

A config beside a subtree adds to the one above it, so a tree settles only
what it names and inherits the rest. Searching inside an already-indexed tree
reads that tree's index and narrows the answers to the path, rather than
starting a second index of the same files.

```sh
nvim $(ish -i src/)
```

### Options

| Flag | Purpose |
|---|---|
| `-i`, `--interactive` | Run the TUI picker |
| `-c`, `--config PATH` | Read the options from this file, not from the search upward |
| `--embedder {llama.cpp,ollama,st}` | Select the embedding backend (default: ollama) |
| `-v`, `-vv` | Increase log detail |
| `--color {auto,always,never}` | Control log color |
| `--limit N` | Maximum search results |
| `--ignore DIR ...` | Directory names to skip (default `.git .venv venv __pycache__`) |
| `--include REGEX ...` | Index only paths matching these patterns, anchored at the tree root |
| `--exclude REGEX ...` | Never index paths matching these patterns |
| `--git`, `--no-git` | Skip files git ignores (default: on) |
| `--unignore REGEX ...` | Index these paths although git ignores them, anchored at the tree root |
| `--lang LANG ...` | Show results only from these languages |
| `--under REGEX` | Show results only from matching paths |
| `--type TYPE ...` | Show results only of these kinds: `code`, `doc`, `test`, `config` |
| `--type-patterns TYPE:REGEX ...` | Say what a path holds, overriding the built-in reading |
| `--max-chunks N` | Index a file that yields more chunks than this as one chunk (default 1000) |
| `--context-tokens N` | How many tokens the model reads of each chunk (default 2048) |
| `--model NAME` | Override the backend model |
| `--refresh` | Bring every stored index at or below the path up to date first |
| `--reindex` | Discard the stored index and build it again |
| `--forget` | Remove the stored index of the path and stop. The vectors stay |
| `--sweep` | Delete the vectors no index of the path's repository references, and stop |
| `--no-cache` | Index in memory only, leaving nothing on disk |
| `--tui-theme NAME` | Theme for the picker (default: the Textual default) |

Logs go to stderr, so you can pipe stdout safely. A file that cannot be read
or parsed is counted in one line; `-v` names them.

### Theme the picker

`ctrl+t` steps to the next theme while the picker runs and names the one it
lands on. The choice lasts for that run. Say where the next run starts with
`tui_theme`, in the config file or as `--tui-theme NAME`:

```toml
tui_theme = "solarized-light"
```

Textual holds a theme for a light terminal as well as for a dark one, and
the preview follows: it reads the source in `gruvbox-light` under a light
theme and in `gruvbox-dark` under a dark one. A name no theme answers to is
reported in the picker, which then keeps the default.

`NO_COLOR` still wins over all of it. Set it and the picker draws in
monochrome, the preview included, whatever theme is chosen.

## Use from Neovim

`contrib/nvim/ish.lua` is an fzf-lua picker. Copy it to `lua/utils/ish.lua` and
bind it:

```lua
map('n', '<leader>fi', function() require('utils.ish').search() end,
    { desc = 'Semantic search (ish)' })
```

It reads `--format grep`, so the built-in previewer opens each result at its
line, and prints the rank in the leftmost column. `search_lang({'cpp'})`,
`search_type({'doc'})`, and `search_here()` narrow it, as does a `lang:`,
`type:`, or `under:` word typed into the query.

While the index refreshes, `require('utils.ish').statusline()` renders a bar
for a statusline — `ish ███░░░░░ 38%` — and an empty string when idle. It reads `vim.g.ish_index_status`, which the picker keeps up to date, and
shows `ish ✓` briefly when a refresh finishes. Nothing is reported through
`vim.notify`: with `cmdheight = 0` there is no command line to put a message
in, so nvim draws one over the last screen row — the statusline itself.

The picker never blocks the editor: results are written as they arrive, so
typing stays smooth however long a search takes.

Saving a buffer tells the server to refresh, once a search has started it,
so an edit is searchable moments after the save. The server would otherwise
notice on its next poll, up to `refresh_seconds` later.

`contrib/nvim/ish_server.lua` keeps one `ish-mcp` process per session. It
starts on the first search and is reused after that, which cuts a keystroke
from about 500 ms to about 150 ms. Copy it beside the picker.

## Use from VS Code

`contrib/vscode/` is an extension that drives the same `ish-mcp` server
from a QuickPick. Press `Ctrl+Alt+I` and type. The highlighted chunk shows
in the editor beside the picker, Enter opens it with its lines selected, and
Tab finishes a filter word. While the index refreshes, the status bar shows
`ish ███░░░░░ 38%`. It needs `ish-mcp` on PATH, like the Neovim client, and
its README says how to build, install, and release it. It ships on its own
schedule, apart from the PyPI package.

## Use from Python

```python
from ish.interfaces.python.api import Ish

with Ish("src/") as ish:
    for chunk, score in ish.search("type:doc how to configure", limit=5):
        print(score, chunk.path, chunk.symbol)
    print(ish.status())
```

`Ish` holds the index open, so a second query costs a search rather than a
process start. It offers `search()`, `chunks()`, `index()`, `refresh_all()`,
and `status()`, and reads `lang:`, `type:`, and `under:` out of the query
exactly as the other interfaces do.

## Use from an agent

`ish-mcp` serves the same search over the Model Context Protocol, so an agent
can query the index directly. Add it to a project with `.mcp.json`:

```json
{
  "mcpServers": {
    "ish": { "command": "uv", "args": ["run", "ish-mcp"] }
  }
}
```

It offers `search_code`, `list_chunks`, `index_status`, `refresh_index`, and
`complete_filter`. The server stays resident, so a query costs about 58 ms
rather than a process start, and a filter word completes from the registries
the server already holds rather than through a fresh `ish-complete`.

The server re-checks a tree every `refresh_seconds`, 30 by default. An
editor knows the moment a file is saved, so the Neovim and VS Code clients
call `refresh_index` then, and an edit is searchable moments later. A client
of your own should do the same.

A call may narrow one search with `lang`, `under`, `type`, and `limit`, or
write the same filters into the query text. It cannot change
what is indexed — those settings come from the config file only, so no single
call can shrink an index that another call depends on.

## Keep the index current

`ish-serve` keeps the indexes of a tree current, for every client that
reads them. It refreshes the tree as soon as it starts, then every
`refresh_seconds`, 30 by default, or when a client asks, and it answers no
query:

```sh
ish-serve /path/to/tree
```

The server writes a status file under `$XDG_RUNTIME_DIR/ish/`, named after
the tree the way its index file is, every few seconds while it runs. Every
client reads the index itself. It walks up from the path it asks about, and
when it finds the status of a live server whose refresh reaches that path,
it leaves the refreshing to the server and reads what is stored:

- `ish "query" path` walks nothing and embeds nothing before it answers.
  Measured on a firmware tree of 6,266 chunks, a cold query took 691 ms on
  its own and 274 ms beside a server, the difference being the walk of the
  tree. While the server is mid-refresh, the run says so on stderr, with
  the step the server is at, how fast it embeds, and about how long is
  left, because the answer may not hold the latest edit yet.
- `ish-mcp` starts no refresh thread for a tree the server keeps, and
  `refresh_index` asks the server for a refresh now. `index_status` reports
  what the server is doing.
- Nothing in the editor clients changes.

A search goes first. Ollama serves one embedding request at a time, so a
query that arrives while the server embeds waits for the text in flight, and
for nothing else: the server sends one text a request, the query leaves a
marker under the runtime directory, and the server holds its next request
back for 2 seconds after the last query. Measured beside a server embedding
chunks of about 250 tokens while a picker types 0.3 s apart, the first
keystroke costs 115 ms and every keystroke after it 23 ms, which is what a
query costs beside an idle daemon. Beside chunks of about 1000 tokens the
first keystroke costs 45 ms, since the daemon reads a long text in pieces
and the query slips in between them. One text a request costs the server about
a fifth of its throughput on small chunks and nothing on large ones, and
changes no vector.

Without a server everything works as before: `ish` brings the index up to
date in front of the answer, behind a line that shows what it is doing, and
`ish-mcp` refreshes on a thread of its own. A server that was killed leaves
a status file nobody writes, and after 30 seconds the clients read past it
and refresh themselves again. A second `ish-serve` for the same tree exits
at once, so a shell hook may start one without checking first, and one that
finds the status of a dead server takes its place.

One server keeps a whole repository current: every index at or below its
tree, and every index of another checkout of the same repository, wherever
that worktree is on disk. A second `ish-serve` for any checkout of the
repository exits at once, and a server whose own tree is deleted stops.
`--once` refreshes once and exits, for a cron job or a hook.

## Index

The index persists in SQLite under `$XDG_DATA_HOME/ish/`, one file per scanned
tree. **A search of a parent reads the indexes below it and refreshes none** —
choosing one of them to write to would be wrong — so it warns and offers
`--refresh`, which visits each tree in turn. A repeated query reuses it, so only changed files are parsed and only new
text is embedded. A renamed file re-embeds nothing.

Beside each index file, a `-matrix` file holds the vectors of the model in
use. A search maps that file into memory instead of reading every vector out
of SQLite, which is what a fresh process used to pay first. A file older than
the index is read past and written again, so it can be deleted at any time.

Each index records the tree it was built from, so searching a directory also
searches every index below it. Index the parts of a large project separately and
search the whole from its root:

```sh
ish "warm" project/docs        # index one part
ish "warm" project/firmware    # and another
ish "how is exposure set" project    # searches both
```

When a release changes how files divide into chunks, the next refresh
reads every file again under the new division and embeds only text it has
never seen. Parsing a 10,000-file tree costs seconds. Embedding it costs
hours, and a vector is keyed by its text, so none of that is paid twice.

Searching a parent never rewrites an index below it. Pass `--no-federate` to use
only the index of the exact path.

### Worktrees and other checkouts

The vectors of a repository live in one shared store beside the indexes,
named after the repository the way git names it, so every checkout of the
repository reaches the same file. A vector is keyed by the text it stands
for, so a second checkout, such as a git worktree, finds the vectors of the
first and embeds only what differs. Measured on a firmware tree of 6260
chunks, a worktree shared 100 % of its chunk hashes with the main checkout
and embedded nothing. Parsing it took 0.9 s. Chunk rows and file stamps stay
in the index of each tree, because a result names a file to open.

A search reads only the indexes of the checkout it starts in, so a worktree
kept under `.claude/worktrees/` does not join a search of the main tree, and
a search from a directory above several checkouts shows each chunk once. An
index whose tree is gone is left out of every search and said so once.
`ish --forget PATH` removes it. The vectors stay in the shared store, because
a checkout at an older commit still needs the vectors of text the main
checkout has since changed: on one firmware tree, sweeping the vectors no
index referenced took a worktree from 100 % reuse to 87 %. Sweep them only on
purpose, with `ish --sweep PATH`, which deletes the vectors no index of the
repository references.

A Markdown or AsciiDoc file with no heading is one chunk, named after the
file, when it holds a line of prose. A file of attribute definitions,
includes, and table rows holds nothing a query asks for and stays out.
`index_status` and `Ish.status()` count the files that were read and
yielded nothing, so a tree of such files is not mistaken for an index.

The index records where each chunk is — its path, line range, kind, and name —
together with the embedding vector. It does not store the source, so it is not a
second readable copy of your code. Previews are read from the file, which also
means they always show the current content.

## Configure

Every command-line option is also a key in the config file, under the same
name. The one exception is `--config` itself: a file cannot name where to
find itself. Put project settings in `.ish/config.toml` at the root of your
repository:

```toml
embedder = "ollama"
model = "mxbai-embed-large"
limit = 10
ignore = [".git", ".venv", "build", "node_modules"]

# Regular expressions, searched against the path.
exclude = ["/vendor/", "_pb2\\.py$", "(_test|_spec)\\.py$"]
```

`include` and `exclude` take regular expressions rather than globs, so
alternation works. `exclude` is searched against the path from the top of the
checkout the tree belongs to, written with a leading slash, so `/vendor/`
matches at any depth and `^/vendor/` only at the top. Outside a repository it
is searched against the whole path. `include` is anchored at the root of the
tree, so `30\.Firmware/` indexes that directory and not
`99.Artifacts/30.Firmware/`. Write `(?:.*/)?30\.Firmware/` to take the name at
any depth. `exclude` wins over `include`.

Both `ignore` and `exclude` judge only what lies inside the checkout, never
the directories above it. A git worktree kept under `.claude/worktrees/` is
indexable with the same configuration that keeps `.claude` out of the main
index.

`--git` is on by default, so anything a `.gitignore` covers stays out of the
index. Pass `--no-git` to index it all anyway, or name the one ignored tree
that belongs in the index with `unignore`, anchored at the root like
`include`, and let git keep filtering the rest:

```toml
unignore = ["11\\.SystemSpec/"]   # a checkout of another system, hidden from git
```

### Keep generated code out

Generated code is the one thing worth excluding by hand. It is large, it is
repetitive, and nobody searches it by meaning. On one firmware tree, generated
headers were **99% of the oversized C and C++ text**: 10.3 MB of 10.4 MB, and
the largest single definition held 2,949,177 characters. A 26 MB generated JSON
register map produced 32,768 chunks and took 76 seconds to parse.

Embedding costs about one chunk per second, so that is hours of work for text
no query wants.

Name the patterns in `exclude`:

```toml
exclude = [
    "/generated/",           # a directory that holds nothing written by hand
    "_pb2\\.py$",             # protobuf
    "\\.g\\.(c|h|cpp)$",       # a generator's own suffix
    "/build/",               # anything a build wrote
    "register_map.*\\.json$", # a generated register map
]
```

Two things to know:

- `exclude` is index scope, so it decides what is *in* the index. Narrowing it
  later does not remove what a wider run already stored; run `--reindex` for
  that.
- A pattern is a regular expression searched against the whole path, so
  `/generated/` matches at any depth and needs no wildcards.
- A pattern that matches a directory, written with its trailing slash, keeps
  the walk out of it. `/build/` never enters `build`, so a tree of 272,364
  generated files costs nothing at all. A pattern that names files, such as
  `_pb2\.py$`, still reads every directory to find them.

Check a pattern before you pay to index it. An empty query lists what the
filter allows, and `--no-cache` keeps the trial out of the stored index:

```sh
ish "" . --no-cache | wc -l                          # everything today
ish "" . --no-cache --exclude '/generated/' | wc -l  # what the pattern leaves
```

If most of a tree is generated, it is usually less work to exclude the
directory than to name each suffix.

A file that slips past every pattern is caught by its size in chunks. A file
that yields more than `max_chunks` of them, 1000 by default, is indexed as
one chunk under its own name, and the run says so once. A file that large is
generated: the largest hand-written file measured split into 374 pieces, the
register map into 32,768. Raise the limit for a tree that is different.

`--lang` and `--under` narrow what a search *returns*. They never change what is
indexed, so a narrowed query cannot shrink the index:

```sh
ish "how is ranking done" --lang python
ish "installation steps" --lang markdown asciidoc
ish "parse a header" --under '/include/'
```

User-level defaults go in `~/.config/ish/config.toml`, which honors
`XDG_CONFIG_HOME`. Later sources win:

```
defaults
  < ~/.config/ish/config.toml
  < .ish/config.toml            (every one from the target path upward,
                                 or the one file --config names)
  < ISH_* environment
  < command line
```

A flat `ish.toml` is still read, beside `.ish/config.toml` and in the user
directory alike, so a file written before this keeps working. Prefer
`.ish/config.toml`: it keeps a tree's settings beside anything else the tool
leaves there.

**Every** config file from the target path upward applies, outermost first, and
each settles only the keys it names. So a file beside a subtree adds to the one
above it rather than replacing it — a subtree can set `git = false` for itself
and still inherit the `type_patterns` the repository above it set.

Set any option from the environment with the `ISH_` prefix, for example
`ISH_LIMIT=20` or `ISH_IGNORE=build,dist`.

### Read more of each chunk

A chunk is capped at 8,000 characters because Ollama serves an embedding
model a window of 2048 tokens, and a model drops what lies past its window
with no signal. `nomic-embed-text` accepts 8192. Ask for it, and the cap
follows to 32,000 characters, so a chunk carries about four times the
meaning:

```toml
context_tokens = 8192
```

A model given more of a text produces another vector for it, so the change
embeds every chunk again under the new window, and the vectors made under
the old one stay for a return to it. Measure before paying that:
`/benchmark` in this repository reports top-1 and MRR for a query set, and
a wider window is worth its re-index only when the table moves. The
`llama.cpp` backend takes the window the same way. The `st` backend reads
it from the model card, so there only the cap follows.

### Name one config file

Point ish at a file with `--config PATH`, its short form `-c`, or the
`ISH_CONFIG` variable:

```sh
ish "how is ranking done" --config ~/work/firmware.toml
ISH_CONFIG=~/work/firmware.toml ish -i 30.Firmware/
```

The named file stands in place of the files above the tree, which ish then
does not look for. Your user file still applies below it, so a machine-wide
preference survives a file chosen for one run. The flag wins over the
variable. A named file that is not there is an error: a mistyped path would
otherwise run on the defaults and say nothing about it.

### Share a file with other tools

A config file may keep the options under a `[tool.ish]` table, where
`[tool.black]` and `[tool.ruff]` also live, so one file can hold sections for
several tools:

```toml
[project]
name = "firmware"

[tool.black]
line-length = 88

[tool.ish]
limit = 10
ignore = [".git", "build"]
```

ish reads that table alone then, and passes over every other section. A file
with no `[tool.ish]` table is a flat list of options, as above, so a file
written before this keeps working.

ish does not look for a `pyproject.toml` on its own. Name one with
`--config` or `ISH_CONFIG` to have its `[tool.ish]` table read.

## Extend

Two things are made to be added: a language and an embedding backend.
Each has one folder, one table, and a recipe at the top of the folder's
`__init__.py`.

**A language** is a class with a `language`, a set of `suffixes`, and a
`parse(path, source)` that returns chunks. Put it in a new module under
`src/ish/adapters/parser/` and add one line to `PARSERS` in that package's
`__init__.py`:

```python
PARSERS = {
    ...
    "toml": Language(TomlParser, aliases=frozenset({"tml"}), category="config"),
}
```

That line is the whole registration. It says how to build the parser, what
else a reader may call the language, and whether it holds code, prose, or
configuration. File discovery, the `--languages` and `--lang` choices, the
chunk size cap and the registry tests all follow from it. Nothing else needs
editing.

A parser that belongs to one project rather than to ish goes in
`~/.config/ish/parsers/` as a module with a `parser()` function. It joins the
same table at run time, may declare `aliases` and a `category` of its own, and
may replace a built-in language.

**A backend** subclasses `PrefixingEmbedder`, implements `_embed(texts)` and
a `from_option(model)` classmethod, and takes one line in `EMBEDDERS` in
`src/ish/adapters/embedder/__init__.py`. The `--embedder` choices derive from
the keys. A backend that needs a package the default install does not carry
is an extra in `pyproject.toml`, and a model trained with task prefixes gets
a row in `prefixes.py`.

## Develop

```sh
uv run poe check   # lint, typecheck, test
```

The architecture is ports and adapters. `.claude/CLAUDE.md` describes the
layers, the composition root, and the decisions behind them.
