Metadata-Version: 2.4
Name: interactive-semantic-hunt
Version: 0.5.0
Summary: Interactive semantic search for code, inspired by fzf
Author-email: David Kristiansen <david@kristiansen.tech>
License-Expression: MIT
Project-URL: Homepage, https://github.com/davetothek/interactive-semantic-hunt
Project-URL: Changelog, https://github.com/davetothek/interactive-semantic-hunt/blob/main/CHANGELOG.md
Project-URL: Repository, https://github.com/davetothek/interactive-semantic-hunt
Project-URL: Issues, https://github.com/davetothek/interactive-semantic-hunt/issues
Keywords: search,semantic,embeddings,code,fzf,tui,mcp
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Software Development
Classifier: Topic :: Text Processing :: Indexing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: numpy>=2.5.2
Requires-Dist: pyyaml>=6.0
Requires-Dist: textual>=0.70.0
Requires-Dist: tree-sitter>=0.26.0
Requires-Dist: tree-sitter-c>=0.24.2
Requires-Dist: tree-sitter-cpp>=0.23.4
Provides-Extra: llama
Requires-Dist: llama-cpp-python>=0.2.75; extra == "llama"
Requires-Dist: huggingface-hub>=0.23.0; extra == "llama"
Provides-Extra: st
Requires-Dist: sentence-transformers; extra == "st"
Dynamic: license-file

# ish — Interactive Semantic Hunt

Semantic search for code, inspired by `fzf`. Point `ish` at a directory. It parses
each source file into named chunks, embeds them with a local model, and ranks them
against your query. Everything runs on your machine.

Languages: Python, C and C++, Markdown, AsciiDoc, YAML, and JSON. Documentation
and configuration are indexed beside the code they describe, so one query
searches all of it.

## Install

```sh
pip install interactive-semantic-hunt
```

Nothing in that install compiles: the default backend reaches Ollama over
HTTP with the standard library. A backend that runs the model in this
process is an extra — `[llama]` for llama.cpp, `[st]` for
sentence-transformers.

To work on ish itself, clone it and run `uv sync`.

The default embedding backend is Ollama, which keeps the model resident so no
run pays a model load. Start it once and pull the embedding model:

```sh
ollama serve
ollama pull nomic-embed-text
```

Set `OLLAMA_HOST` to reach a daemon elsewhere.

Two other backends need no daemon:

- `--embedder llama.cpp` downloads a GGUF model and loads it per run. Slower per
  query, faster for a first index of a large tree. Needs the `[llama]` extra.
- `--embedder st` uses sentence-transformers. Needs the `[st]` extra.

## Use

List every chunk under a path:

```sh
ish src/
```

```
src/ish/domain/chunk.py:7-28  class  Chunk
```

Search for a query:

```sh
ish "parse a python file" src/
```

```
[0.71] src/ish/adapters/parser/python.py:14-31  method  PythonParser.parse
```

Run the interactive picker and open the selection in your editor. Type to
search, `up`/`down` or `ctrl+p`/`ctrl+n` to move, `enter` to choose, `ctrl+t`
to change the theme, `escape` to quit. The picker lists what the index held
last time before it looks for changes, and answers from that while the
refresh runs, with the progress in the header. Narrow without leaving the
query line:

```text
state machine transitions              every language
lang:cpp state machine transitions     the implementation
lang:yaml under:/10.System/ state      the tests that cover it
type:doc how do I configure this       the prose, not the code
type:test,doc retry backoff            the tests and what they document
```

Press Tab to finish a filter word, in the picker and in Neovim alike. `ty`
becomes `type:`, `lang:cp` becomes `lang:cpp`, and `under:/s` becomes
`under:/src/`; a word with several answers grows as far as they agree and
names the rest. `ish-complete` does the work from the command line, and
`complete_filter` does it over MCP, so any picker can call whichever it has.

The shell finishes the same words on the command line. Source
`contrib/shell/ish.bash` from `~/.bashrc`, or copy `contrib/shell/_ish` to a
directory on zsh's `$fpath`, and `ish lang:cp<Tab>` becomes `ish lang:cpp`.
Both call `ish-complete`, so the shell learns nothing about the languages.

`lang:`, `under:`, and `type:` work in the query line of every interface —
the command line, the picker, Neovim, and MCP. The words are taken out
before the query is embedded, so the model sees the question rather than
how it was narrowed.

A language may be named however it comes to mind. `c`, `c++`, `cxx`, `h`, and
`hpp` all mean `cpp`, because one parser reads them all; `adoc` means
`asciidoc`, `md` means `markdown`, `py` means `python`, and `yml` means `yaml`.

`type:` sorts every chunk into exactly one kind. A path decides before a
language does, so a YAML fixture counts as a test rather than as config:

| kind | what it holds |
|---|---|
| `code` | Python, C, C++, and anything a plugin parser adds |
| `doc` | Markdown and AsciiDoc |
| `test` | anything under `tests/`, `spec/`, `fixtures/`, plus `test_*` and `conftest.py` |
| `config` | YAML, JSON, and TOML outside a test path |

A repository that names its trees its own way can say so, in
`.ish/config.toml`:

```toml
type_patterns = [
  "test:/[0-9.]*(Tests|Verification)/",
  "doc:/[0-9.]*Specification/",
]
```

The first match wins; anything unmatched keeps the reading above.

A config beside a subtree adds to the one above it, so a tree settles only
what it names and inherits the rest. Searching inside an already-indexed tree
reads that tree's index and narrows the answers to the path, rather than
starting a second index of the same files.

```sh
nvim $(ish -i src/)
```

### Options

| Flag | Purpose |
|---|---|
| `-i`, `--interactive` | Run the TUI picker |
| `-c`, `--config PATH` | Read the options from this file, not from the search upward |
| `--embedder {llama.cpp,ollama,st}` | Select the embedding backend (default: ollama) |
| `-v`, `-vv` | Increase log detail |
| `--color {auto,always,never}` | Control log color |
| `--limit N` | Maximum search results |
| `--ignore DIR ...` | Directory names to skip (default `.git .venv venv __pycache__`) |
| `--include REGEX ...` | Index only paths matching these patterns, anchored at the tree root |
| `--exclude REGEX ...` | Never index paths matching these patterns |
| `--git`, `--no-git` | Skip files git ignores (default: on) |
| `--unignore REGEX ...` | Index these paths although git ignores them, anchored at the tree root |
| `--lang LANG ...` | Show results only from these languages |
| `--under REGEX` | Show results only from matching paths |
| `--type TYPE ...` | Show results only of these kinds: `code`, `doc`, `test`, `config` |
| `--type-patterns TYPE:REGEX ...` | Say what a path holds, overriding the built-in reading |
| `--max-chunks N` | Index a file that yields more chunks than this as one chunk (default 1000) |
| `--context-tokens N` | How many tokens the model reads of each chunk (default 2048) |
| `--model NAME` | Override the backend model |
| `--refresh` | Bring every stored index at or below the path up to date first |
| `--reindex` | Discard the stored index and build it again |
| `--no-cache` | Index in memory only, leaving nothing on disk |
| `--tui-theme NAME` | Theme for the picker (default: the Textual default) |

Logs go to stderr, so you can pipe stdout safely. A file that cannot be read
or parsed is counted in one line; `-v` names them.

### Theme the picker

`ctrl+t` steps to the next theme while the picker runs and names the one it
lands on. The choice lasts for that run. Say where the next run starts with
`tui_theme`, in the config file or as `--tui-theme NAME`:

```toml
tui_theme = "solarized-light"
```

Textual holds a theme for a light terminal as well as for a dark one, and
the preview follows: it reads the source in `gruvbox-light` under a light
theme and in `gruvbox-dark` under a dark one. A name no theme answers to is
reported in the picker, which then keeps the default.

`NO_COLOR` still wins over all of it. Set it and the picker draws in
monochrome, the preview included, whatever theme is chosen.

## Use from Neovim

`contrib/nvim/ish.lua` is an fzf-lua picker. Copy it to `lua/utils/ish.lua` and
bind it:

```lua
map('n', '<leader>fi', function() require('utils.ish').search() end,
    { desc = 'Semantic search (ish)' })
```

It reads `--format grep`, so the built-in previewer opens each result at its
line, and prints the rank in the leftmost column. `search_lang({'cpp'})`,
`search_type({'doc'})`, and `search_here()` narrow it, as does a `lang:`,
`type:`, or `under:` word typed into the query.

While the index refreshes, `require('utils.ish').statusline()` renders a bar
for a statusline — `ish ███░░░░░ 38%` — and an empty string when idle. It reads `vim.g.ish_index_status`, which the picker keeps up to date, and
shows `ish ✓` briefly when a refresh finishes. Nothing is reported through
`vim.notify`: with `cmdheight = 0` there is no command line to put a message
in, so nvim draws one over the last screen row — the statusline itself.

The picker never blocks the editor: results are written as they arrive, so
typing stays smooth however long a search takes.

Saving a buffer tells the server to refresh, once a search has started it,
so an edit is searchable moments after the save. The server would otherwise
notice on its next poll, up to `refresh_seconds` later.

`contrib/nvim/ish_server.lua` keeps one `ish-mcp` process per session. It
starts on the first search and is reused after that, which cuts a keystroke
from about 500 ms to about 150 ms. Copy it beside the picker.

## Use from VS Code

`contrib/vscode/` is an extension that drives the same `ish-mcp` server
from a QuickPick. Press `Ctrl+Alt+I` and type. The highlighted chunk shows
in the editor beside the picker, Enter opens it with its lines selected, and
Tab finishes a filter word. While the index refreshes, the status bar shows
`ish ███░░░░░ 38%`. It needs `ish-mcp` on PATH, like the Neovim client, and
its README says how to build, install, and release it. It ships on its own
schedule, apart from the PyPI package.

## Use from Python

```python
from ish.interfaces.python.api import Ish

with Ish("src/") as ish:
    for chunk, score in ish.search("type:doc how to configure", limit=5):
        print(score, chunk.path, chunk.symbol)
    print(ish.status())
```

`Ish` holds the index open, so a second query costs a search rather than a
process start. It offers `search()`, `chunks()`, `index()`, `refresh_all()`,
and `status()`, and reads `lang:`, `type:`, and `under:` out of the query
exactly as the other interfaces do.

## Use from an agent

`ish-mcp` serves the same search over the Model Context Protocol, so an agent
can query the index directly. Add it to a project with `.mcp.json`:

```json
{
  "mcpServers": {
    "ish": { "command": "uv", "args": ["run", "ish-mcp"] }
  }
}
```

It offers `search_code`, `list_chunks`, `index_status`, `refresh_index`, and
`complete_filter`. The server stays resident, so a query costs about 58 ms
rather than a process start, and a filter word completes from the registries
the server already holds rather than through a fresh `ish-complete`.

The server re-checks a tree every `refresh_seconds`, 30 by default. An
editor knows the moment a file is saved, so the Neovim and VS Code clients
call `refresh_index` then, and an edit is searchable moments later. A client
of your own should do the same.

A call may narrow one search with `lang`, `under`, `type`, and `limit`, or
write the same filters into the query text. It cannot change
what is indexed — those settings come from the config file only, so no single
call can shrink an index that another call depends on.

## Index

The index persists in SQLite under `$XDG_DATA_HOME/ish/`, one file per scanned
tree. **A search of a parent reads the indexes below it and refreshes none** —
choosing one of them to write to would be wrong — so it warns and offers
`--refresh`, which visits each tree in turn. A repeated query reuses it, so only changed files are parsed and only new
text is embedded. A renamed file re-embeds nothing.

Each index records the tree it was built from, so searching a directory also
searches every index below it. Index the parts of a large project separately and
search the whole from its root:

```sh
ish "warm" project/docs        # index one part
ish "warm" project/firmware    # and another
ish "how is exposure set" project    # searches both
```

When a release changes how files divide into chunks, the next refresh
reads every file again under the new division and embeds only text it has
never seen. Parsing a 10,000-file tree costs seconds. Embedding it costs
hours, and a vector is keyed by its text, so none of that is paid twice.

Searching a parent never rewrites an index below it. Pass `--no-federate` to use
only the index of the exact path.

A Markdown or AsciiDoc file with no heading is one chunk, named after the
file, when it holds a line of prose. A file of attribute definitions,
includes, and table rows holds nothing a query asks for and stays out.
`index_status` and `Ish.status()` count the files that were read and
yielded nothing, so a tree of such files is not mistaken for an index.

The index records where each chunk is — its path, line range, kind, and name —
together with the embedding vector. It does not store the source, so it is not a
second readable copy of your code. Previews are read from the file, which also
means they always show the current content.

## Configure

Every command-line option is also a key in the config file, under the same
name. The one exception is `--config` itself: a file cannot name where to
find itself. Put project settings in `.ish/config.toml` at the root of your
repository:

```toml
embedder = "ollama"
model = "mxbai-embed-large"
limit = 10
ignore = [".git", ".venv", "build", "node_modules"]

# Regular expressions, searched against the path.
exclude = ["/vendor/", "_pb2\\.py$", "(_test|_spec)\\.py$"]
```

`include` and `exclude` take regular expressions rather than globs, so
alternation works. `exclude` is searched against the whole path, so `/vendor/`
matches at any depth. `include` is anchored at the root of the tree, so
`30\.Firmware/` indexes that directory and not `99.Artifacts/30.Firmware/`.
Write `(?:.*/)?30\.Firmware/` to take the name at any depth. `exclude` wins
over `include`.

`--git` is on by default, so anything a `.gitignore` covers stays out of the
index. Pass `--no-git` to index it all anyway, or name the one ignored tree
that belongs in the index with `unignore`, anchored at the root like
`include`, and let git keep filtering the rest:

```toml
unignore = ["11\\.SystemSpec/"]   # a checkout of another system, hidden from git
```

### Keep generated code out

Generated code is the one thing worth excluding by hand. It is large, it is
repetitive, and nobody searches it by meaning. On one firmware tree, generated
headers were **99% of the oversized C and C++ text**: 10.3 MB of 10.4 MB, and
the largest single definition held 2,949,177 characters. A 26 MB generated JSON
register map produced 32,768 chunks and took 76 seconds to parse.

Embedding costs about one chunk per second, so that is hours of work for text
no query wants.

Name the patterns in `exclude`:

```toml
exclude = [
    "/generated/",           # a directory that holds nothing written by hand
    "_pb2\\.py$",             # protobuf
    "\\.g\\.(c|h|cpp)$",       # a generator's own suffix
    "/build/",               # anything a build wrote
    "register_map.*\\.json$", # a generated register map
]
```

Two things to know:

- `exclude` is index scope, so it decides what is *in* the index. Narrowing it
  later does not remove what a wider run already stored; run `--reindex` for
  that.
- A pattern is a regular expression searched against the whole path, so
  `/generated/` matches at any depth and needs no wildcards.
- A pattern that matches a directory, written with its trailing slash, keeps
  the walk out of it. `/build/` never enters `build`, so a tree of 272,364
  generated files costs nothing at all. A pattern that names files, such as
  `_pb2\.py$`, still reads every directory to find them.

Check a pattern before you pay to index it. An empty query lists what the
filter allows, and `--no-cache` keeps the trial out of the stored index:

```sh
ish "" . --no-cache | wc -l                          # everything today
ish "" . --no-cache --exclude '/generated/' | wc -l  # what the pattern leaves
```

If most of a tree is generated, it is usually less work to exclude the
directory than to name each suffix.

A file that slips past every pattern is caught by its size in chunks. A file
that yields more than `max_chunks` of them, 1000 by default, is indexed as
one chunk under its own name, and the run says so once. A file that large is
generated: the largest hand-written file measured split into 374 pieces, the
register map into 32,768. Raise the limit for a tree that is different.

`--lang` and `--under` narrow what a search *returns*. They never change what is
indexed, so a narrowed query cannot shrink the index:

```sh
ish "how is ranking done" --lang python
ish "installation steps" --lang markdown asciidoc
ish "parse a header" --under '/include/'
```

User-level defaults go in `~/.config/ish/config.toml`, which honors
`XDG_CONFIG_HOME`. Later sources win:

```
defaults
  < ~/.config/ish/config.toml
  < .ish/config.toml            (every one from the target path upward,
                                 or the one file --config names)
  < ISH_* environment
  < command line
```

A flat `ish.toml` is still read, beside `.ish/config.toml` and in the user
directory alike, so a file written before this keeps working. Prefer
`.ish/config.toml`: it keeps a tree's settings beside anything else the tool
leaves there.

**Every** config file from the target path upward applies, outermost first, and
each settles only the keys it names. So a file beside a subtree adds to the one
above it rather than replacing it — a subtree can set `git = false` for itself
and still inherit the `type_patterns` the repository above it set.

Set any option from the environment with the `ISH_` prefix, for example
`ISH_LIMIT=20` or `ISH_IGNORE=build,dist`.

### Read more of each chunk

A chunk is capped at 8,000 characters because Ollama serves an embedding
model a window of 2048 tokens, and a model drops what lies past its window
with no signal. `nomic-embed-text` accepts 8192. Ask for it, and the cap
follows to 32,000 characters, so a chunk carries about four times the
meaning:

```toml
context_tokens = 8192
```

A model given more of a text produces another vector for it, so the change
embeds every chunk again under the new window, and the vectors made under
the old one stay for a return to it. Measure before paying that:
`/benchmark` in this repository reports top-1 and MRR for a query set, and
a wider window is worth its re-index only when the table moves. The
`llama.cpp` backend takes the window the same way. The `st` backend reads
it from the model card, so there only the cap follows.

### Name one config file

Point ish at a file with `--config PATH`, its short form `-c`, or the
`ISH_CONFIG` variable:

```sh
ish "how is ranking done" --config ~/work/firmware.toml
ISH_CONFIG=~/work/firmware.toml ish -i 30.Firmware/
```

The named file stands in place of the files above the tree, which ish then
does not look for. Your user file still applies below it, so a machine-wide
preference survives a file chosen for one run. The flag wins over the
variable. A named file that is not there is an error: a mistyped path would
otherwise run on the defaults and say nothing about it.

### Share a file with other tools

A config file may keep the options under a `[tool.ish]` table, where
`[tool.black]` and `[tool.ruff]` also live, so one file can hold sections for
several tools:

```toml
[project]
name = "firmware"

[tool.black]
line-length = 88

[tool.ish]
limit = 10
ignore = [".git", "build"]
```

ish reads that table alone then, and passes over every other section. A file
with no `[tool.ish]` table is a flat list of options, as above, so a file
written before this keeps working.

ish does not look for a `pyproject.toml` on its own. Name one with
`--config` or `ISH_CONFIG` to have its `[tool.ish]` table read.

## Extend

Two things are made to be added: a language and an embedding backend.
Each has one folder, one table, and a recipe at the top of the folder's
`__init__.py`.

**A language** is a class with a `language`, a set of `suffixes`, and a
`parse(path, source)` that returns chunks. Put it in a new module under
`src/ish/adapters/parser/` and add one line to `PARSERS` in that package's
`__init__.py`:

```python
PARSERS = {
    ...
    "toml": Language(TomlParser, aliases=frozenset({"tml"}), category="config"),
}
```

That line is the whole registration. It says how to build the parser, what
else a reader may call the language, and whether it holds code, prose, or
configuration. File discovery, the `--languages` and `--lang` choices, the
chunk size cap and the registry tests all follow from it. Nothing else needs
editing.

A parser that belongs to one project rather than to ish goes in
`~/.config/ish/parsers/` as a module with a `parser()` function. It joins the
same table at run time, may declare `aliases` and a `category` of its own, and
may replace a built-in language.

**A backend** subclasses `PrefixingEmbedder`, implements `_embed(texts)` and
a `from_option(model)` classmethod, and takes one line in `EMBEDDERS` in
`src/ish/adapters/embedder/__init__.py`. The `--embedder` choices derive from
the keys. A backend that needs a package the default install does not carry
is an extra in `pyproject.toml`, and a model trained with task prefixes gets
a row in `prefixes.py`.

## Develop

```sh
uv run poe check   # lint, typecheck, test
```

The architecture is ports and adapters. `.claude/CLAUDE.md` describes the
layers, the composition root, and the decisions behind them.
