Metadata-Version: 2.5
Name: hikidasu
Version: 0.1.0
Summary: 日本語の markdown ノートを、常駐した埋め込みモデル（ruri-v3）で意味検索する CLI / Japanese-first semantic search over markdown notes with a resident embedding server
Project-URL: Homepage, https://github.com/Rererr/hikidasu
Project-URL: Repository, https://github.com/Rererr/hikidasu
Author: Rererr
License-Expression: MIT
License-File: LICENSE
Keywords: claude-code,embedding,japanese,knowledge-base,markdown,ruri,search
Classifier: Natural Language :: Japanese
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Text Processing :: Indexing
Requires-Python: >=3.10
Requires-Dist: numpy>=1.26
Requires-Dist: sentence-transformers>=3.0
Requires-Dist: torch>=2.2
Description-Content-Type: text/markdown

# hikidasu

[日本語版 README](README.md)

Semantic search over a directory of markdown notes, in Japanese, with a resident embedding server.
Built to answer one question with measurements instead of opinions: *when does grep miss, and is adding embeddings worth it for a personal knowledge base read by a coding agent?*

- `hikidasu "質問文"` returns the top notes (path, cosine, frontmatter description) in ~50 ms once the server is warm
- `hikidasu-serve` keeps [cl-nagoya/ruri-v3-310m](https://huggingface.co/cl-nagoya/ruri-v3-310m) loaded behind a Unix socket (mode 0600), re-embeds only notes whose sha256 changed, and exits after an idle hour
- `bench/` holds the measurement scripts used in the write-up: grep baseline, qmd (BM25 / vector / hybrid), direct embedding Recall@5, and an agent-in-the-loop benchmark driven by `claude -p`

## Install

With `uv` it is a one-liner:

```sh
uvx --from git+https://github.com/Rererr/hikidasu hikidasu "リトライの上限回数はどう決めるべき？"
```

For regular use, `uv tool install git+https://github.com/Rererr/hikidasu` puts `hikidasu` and `hikidasu-serve` on your PATH (`pip install git+https://github.com/Rererr/hikidasu` works too).
Python 3.10 or later, macOS or Linux.

Point it at your notes with environment variables:

```sh
export HIKIDASU_KB=~/notes                 # notes directory (default ~/notes); every *.md below it is indexed recursively
export HIKIDASU_DIRS=patterns,decisions    # optional: restrict to these top-level subdirectories
hikidasu "リトライの上限回数はどう決めるべき？"
```

The first run downloads the model (about 1.2 GB in safetensors) and installs the dependencies (about 900 MB including torch), so it takes a few minutes.
Later calls take tens of milliseconds.
`hikidasu --status` shows the resident server, `--stop` stops it, `--version` prints the version.
Set `HIKIDASU_OFFLINE=1` once the model is cached to forbid network access.

The index and the server socket live under `~/.cache/hikidasu/<hash of KB>/`, one directory per KB, so several note collections can be resident at the same time.

## What the tool does not do

- It never says "no match". Cosine scores of unanswerable questions (0.82–0.86 on our corpus) overlap with those of correct paraphrase hits, so the caller has to read the returned descriptions and decide. In the agent benchmark that judgment was made correctly by the agent in every unanswerable case.
- It returns 8 results by default because, on our corpus, questions whose answer spans three notes needed rank 6–8 once the question was rephrased.
- Queries ending in a short question clause after a full stop (「…出続ける。なぜ？」) are also embedded without that clause; the per-document max of the two is used. The rule only fires after 「。」.

## Measurements (104 Japanese notes, 2026-10-04)

Two question sets of 20 (lex 5 / synonym 6 / JA↔EN 3 / multi-note 3 / no-answer 3). Set B was written by a separate agent that had not seen set A or the tool. Recall@5 excludes the no-answer questions.

| Searcher (input) | Set A | Set B (held-out) |
|---|---|---|
| grep, per-term OR, ranked by hit lines (keywords) | 0.666 | 0.471 |
| qmd `search`, BM25 (keywords) | 0.402 | 0.382 |
| qmd `search`, BM25 (full sentence) | 0.000 | 0.000 |
| ruri-v3-310m, plain (sentence) | 0.941 | 1.000 |
| hikidasu as shipped (sentence) | 1.000 | 1.000 |

Agent-in-the-loop (Claude Code headless, 20 held-out questions, sequential): with sonnet all three instructions (grep / hikidasu / hybrid) reached the correct note in 100% of cases; input tokens were 44.8k / 22.6k / 26.4k and cost $0.077 / $0.026 / $0.028 per question, wall clock 9–10 s each. The agent expands synonyms on its own and reads the human-written index, which is why grep instructions do not lose; the embedding route mainly halves cost.

qmd's BM25 lost to grep on Japanese because it indexes CJK runs as per-character exact phrases and ANDs every term; `third_party/qmd-patch/` is an experimental word-segmentation patch that helps keyword queries a little and sentence queries not at all.

## License

MIT for the code in this repository (see LICENSE, Copyright (c) 2026 Rererr). Third-party models and the qmd patch are covered in NOTICE.md; the patch keeps upstream qmd's MIT notice.
