Metadata-Version: 2.4
Name: lorebox
Version: 0.2.0
Summary: Private, offline document search and Q&A for your own notes
Author: lorebox contributors
License-Expression: MIT
Project-URL: Homepage, https://atomgit.com/NJPJI_JK/lorebox
Project-URL: Repository, https://atomgit.com/NJPJI_JK/lorebox
Project-URL: Issues, https://atomgit.com/NJPJI_JK/lorebox/issues
Project-URL: Changelog, https://atomgit.com/NJPJI_JK/lorebox/blob/main/CHANGELOG.md
Keywords: rag,retrieval-augmented-generation,offline,local-first,ollama,chromadb,embeddings,semantic-search,notes,cli
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: End Users/Desktop
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Text Processing :: Indexing
Classifier: Topic :: Utilities
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: chromadb>=0.4.0
Requires-Dist: click>=8.0.0
Requires-Dist: requests>=2.28.0
Requires-Dist: pypdf>=3.0.0
Provides-Extra: local-embed
Requires-Dist: onnxruntime>=1.17; extra == "local-embed"
Dynamic: license-file

# lorebox

A private, offline document Q&A tool. Drop your Markdown / text notes in, ask
questions in natural language, and get answers grounded in your own files —
nothing leaves your machine.

一个本地私密文档问答工具。把你的 Markdown / 纯文本笔记丢进去，用自然语言提问，
答案只基于你自己的文件生成——数据不出本机。

It uses [Ollama](https://ollama.com) for embeddings and generation, and
[chromadb](https://www.trychroma.com) for local vector search. No cloud, no API
key, no telemetry. If you would rather not run a model server, `--embed-backend
onnx` computes the embeddings with chromadb's bundled model instead — indexing
and `search` then need no Ollama at all, only `ask` does.

它用 [Ollama](https://ollama.com) 做嵌入和生成，用
[chromadb](https://www.trychroma.com) 做本地向量检索。没有云服务、没有 API Key、
没有遥测上报。不想跑模型服务的话，加 `--embed-backend onnx` 改用 chromadb 自带的
模型算嵌入——这样索引和 `search` 完全不需要 Ollama，只有 `ask` 需要。

**Repository / 仓库地址**：<https://atomgit.com/NJPJI_JK/lorebox>
（同一项目同时发布在 <https://gitcode.com/NJPJI_JK/lorebox>）

## Why this exists / 为什么做这个

I had a growing pile of notes and course material I wanted to query without
uploading any of it to a third-party service. Most RAG demos assume a paid API
key and a cloud vector DB. This is the offline version of that idea.

我积攒了一堆笔记和课程资料，想查询它们又不愿上传到任何第三方服务。市面上大多数
RAG demo 都假设你有付费 API Key 和云端向量库。这个项目就是那个想法的离线版本。

## Requirements / 环境要求

- Python 3.9+
- [Ollama](https://ollama.com) running locally (`ollama serve`) — required for the
  default embedding backend and for `ask`; indexing and `search` do not need it
  when you pass `--embed-backend onnx`
  / 本地运行 Ollama（默认嵌入后端与 `ask` 需要；用 `--embed-backend onnx` 时
  索引与检索不需要）
- Pull the models you want to use, e.g. / 拉取你想用的模型，例如：

```
ollama pull nomic-embed-text
ollama pull qwen2.5
```

Not sure whether your setup is ready? `/ 不确定环境是否就绪：`

```
lorebox doctor
```

## Install / 安装

```
pip install -e .
```

## Usage / 使用

Index your notes — Markdown, plain text and PDF are supported /
索引你的笔记（支持 Markdown、纯文本和 PDF）：

```
lorebox add ./notes
lorebox add notes/a.md --overlap 200 --max-chars 800
lorebox add ./notes --dry-run          # list what would be indexed / 先看会处理哪些文件
```

`--ext .org` adds a suffix the default set doesn't cover; the default set is
`.md .markdown .txt .rst .pdf`. / `--ext .org` 可以追加默认集合之外的后缀；默认集合是
`.md .markdown .txt .rst .pdf`。

Ask a question / 提问：

```
lorebox ask "what did I write about the final exam?"
lorebox ask "我笔记里写了期末考试考什么？"
```

The answer is printed along with the source chunks it used / 答案后面会列出用到的来源片段：

```
lorebox ask "..." --no-sources         # hide the source list / 不显示来源列表
lorebox ask "..." --path notes/course  # search one folder only / 只在这个目录里检索
lorebox ask "..." --json               # {question, answer, sources} / 结构化输出
lorebox ask "..." --stream             # print as it generates / 边生成边打印
```

`--json` and `--stream` cannot be combined: structured output needs the complete
answer. / `--json` 与 `--stream` 不能同时使用，结构化输出需要拿到完整答案。

Look at the index / 查看索引：

```
lorebox list                   # one line per file / 每个文件一行
lorebox list --chunks          # one line per chunk / 每个片段一行
lorebox stats                  # file / chunk / character counts / 统计信息
lorebox stats --files --top 5  # biggest files first / 按字符数从大到小排，只看前 5 个
lorebox doctor                 # check Ollama, the models and the store / 自检本机环境
```

Move an index between machines / 在机器之间迁移索引：

```
lorebox export --out dump.json --with-embeddings
lorebox import dump.json       # no re-embedding needed / 不需要重新嵌入
```

`import` only accepts a dump made with `--with-embeddings`; a plain dump carries
no vectors, so it is refused rather than half-imported. /
`import` 只接受 `--with-embeddings` 导出的 dump；普通 dump 不含向量，会被直接拒绝，
不会只导入一半。

Search without a model / 不用模型直接检索：

```
lorebox search "what did I write about the final exam?"
lorebox search "..." --path notes/course
```

`search` only retrieves: it prints the closest chunks and their distances, and
never calls a language model. With `--embed-backend onnx` it needs no Ollama at
all. / `search` 只做检索：打印最相近的片段与距离，不调用语言模型。配合
`--embed-backend onnx` 时完全不需要 Ollama。

Embedding backends / 嵌入后端：

```
lorebox add ./notes --embed-backend onnx
lorebox search "..." --embed-backend onnx
```

`onnx` runs all-MiniLM-L6-v2 locally through chromadb, so no model server is
needed. Two things worth knowing: the model is downloaded once on first use
(after that it really is offline), and an index can only be queried with the
backend that built it — the two produce vectors of different width, so lorebox
records which backend an index used and refuses to mix them. /
`onnx` 通过 chromadb 在本机运行 all-MiniLM-L6-v2，不需要模型服务。两点需要注意：
模型会在首次使用时下载一次（之后才是真正离线）；一个索引只能用建它时的那个后端
查询——两者向量宽度不同，lorebox 会记录索引用的是哪个后端并拒绝混用。

Maintain the index / 维护索引：

```
lorebox remove notes/a.md      # drop one file / 删掉某个文件
```

`--store` picks the index directory (default `.lorebox/`); `--model`,
`--embed-model`, `--top-k`, `--overlap`, `--max-chars` and `--out` apply to the
commands where they make sense. Set `OLLAMA_HOST` if Ollama is not on
`http://localhost:11434`. / `--store` 指定索引目录（默认 `.lorebox/`）；
`--model`、`--embed-model`、`--top-k`、`--overlap`、`--max-chars`、`--out` 按各命令
的用途适用。Ollama 不在默认地址时用 `OLLAMA_HOST` 指定。

## Development / 开发与测试

The unit tests need `chromadb` (installed together with the package) and do not
need a running Ollama instance.
单元测试需要 `chromadb`（装包时会一起装上），但不需要本地跑着 Ollama。

```
python -m unittest discover -s tests
```

They cover the chunker, the store and the command line. Please keep them green
before opening a pull request.
用例覆盖分块、索引存储和命令行三部分。提 PR 前请保持全绿。

How the pieces fit together is described in
[docs/architecture.md](docs/architecture.md); what a good change looks like is in
[CONTRIBUTING.md](CONTRIBUTING.md).
各模块如何拼在一起见 [docs/architecture.md](docs/architecture.md)；什么样的改动好评审见
[CONTRIBUTING.md](CONTRIBUTING.md)。

## Layout / 目录结构

- `ollama.py` — thin client for the local Ollama HTTP API / 本地 Ollama HTTP 客户端
- `chunker.py` — paragraph-based chunking / 按段落分块
- `store.py` — chromadb persistent index / chromadb 持久化索引
- `retrieve.py` — embed a query and fetch top-k chunks / 查询嵌入 + top-k 检索
- `ask.py` — prompt assembly and generation / prompt 组装与生成
- `cli.py` — the command line: `add` / `ask` / `list` / `remove` / `export` /
  `import` / `stats` / `doctor`
  / 命令行入口：`add` / `ask` / `list` / `remove` / `export` / `import` / `stats` / `doctor`
- `tests/` — unit tests; no running Ollama needed / 单元测试（不需要本地跑着 Ollama）
- `docs/architecture.md` — module map, index layout, design trade-offs / 模块结构、索引布局与设计取舍

## Current limitations / 当前限制

- Retrieval is plain vector similarity — no reranking, no score threshold. /
  检索就是简单的向量相似度，没有重排序，也没有相似度阈值。
- Incremental indexing works at file granularity: a file is skipped only when its
  contents are unchanged, so editing one paragraph re-embeds the whole file. /
  增量索引以文件为单位：只有内容完全没变才会跳过，改一个段落就会让整个文件重新嵌入。
- Only Markdown, plain text and PDF are read; docx and epub are not. /
  目前只读 Markdown、纯文本和 PDF，docx 与 epub 尚不支持。
- The `onnx` embedding backend downloads its model on first use, so the very first
  run needs network access. / `onnx` 嵌入后端首次使用时会下载模型，因此第一次
  运行需要联网。

## Maintenance plan / 维护计划

Three passes are done — the known bugs, the usability gaps (streaming,
structured output, scoped search, import/export, batch embeddings, self-check),
and the local embedding backend. What changed, per release, is in
[CHANGELOG.md](CHANGELOG.md).

三轮已完成：已知问题、体验缺口（流式输出、结构化输出、范围检索、导入导出、批量
嵌入、环境自检），以及本地嵌入后端。每次改动见 [CHANGELOG.md](CHANGELOG.md)。

What is left, roughly in order / 后续方向，大致按优先级：

1. **Only re-embed what changed.** One edited paragraph currently costs a full
   file re-embed.
   **只重算变更部分。** 现在改一个段落会让整个文件重新嵌入。
2. **Rerank the retrieved chunks** and add a similarity threshold.
   **检索结果重排序**，并加上相似度阈值。
3. **More formats**: docx and epub.
   **支持更多格式**：docx、epub。
4. **Ship it on PyPI** so people can `pip install lorebox` instead of cloning.
   **发布到 PyPI**，让用户 `pip install` 而不用 clone 源码。
5. **A server-free path for `ask` too.** Generating an answer still needs Ollama;
   the embedding side already has a local option.
   **给 `ask` 也提供无服务端的路径。** 生成答案目前仍需 Ollama，而嵌入侧已经
   有了本地选项。

Contributions are welcome at any point — see [CONTRIBUTING.md](CONTRIBUTING.md).

任何阶段都欢迎参与贡献，见 [CONTRIBUTING.md](CONTRIBUTING.md)。

## Contributing / 参与贡献

See [CONTRIBUTING.md](CONTRIBUTING.md). / 详见 [CONTRIBUTING.md](CONTRIBUTING.md)。

## License / 许可证

[MIT](LICENSE)
