Metadata-Version: 2.5
Name: litesearch
Version: 0.1.42
Summary: search through files with fts5, vectors and get reranked results. Fast
Project-URL: Repository, https://github.com/Karthik777/litesearch
Project-URL: Documentation, https://Karthik777.github.io/litesearch
Author-email: 71293 <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: document search,fts+vectors,nbdev,semantic search,text search,vector search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Requires-Dist: chonkie>=1.6.0
Requires-Dist: codesigs>=0.0.2
Requires-Dist: fastlite>=0.2.4
Requires-Dist: model2vec>=0.7.0
Requires-Dist: pdf-oxide>=0.3.17
Requires-Dist: pdflite>=0.0.1
Requires-Dist: pillow>=12.1.1
Requires-Dist: tokenizers>=0.22.2
Requires-Dist: usearch>=2.23.0
Description-Content-Type: text/markdown

# litesearch


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

> **NB** Reading this on GitHub? The formatted [documentation](https://Karthik777.github.io/litesearch/) is nicer.

litesearch stores and searches documents in one SQLite file. FTS5 keyword search and SIMD vector
search, fused by Reciprocal Rank Fusion. No server.

Two ways in. Pick by one question: do you want the defaults decided for you?

| route | use it when | what it costs |
|----|----|----|
| [\[`Index`\](https://Karthik777.github.io/litesearch/api.html#index)](https://Karthik777.github.io/litesearch/api.html) | you want to search a folder of documents or code | nothing. Encoder, dtype, chunk size, retrieval and tree all come from `evals/` |
| [\[`database()`\](https://Karthik777.github.io/litesearch/core.html#database)](https://Karthik777.github.io/litesearch/core.html) | you need your own columns, encoder, SQL or float32 vectors | six decisions, one of which fails silently |

Start at [`Index`](https://Karthik777.github.io/litesearch/api.html#index). Drop to [`database()`](https://Karthik777.github.io/litesearch/core.html#database) when it stops fitting: it is the same object underneath,
reachable as `Index.db`.

## Install

``` python
# usearch SQLite extensions are configured automatically on first import
# (macOS needs one extra step — see litesearch.postfix)
!uv add litesearch
```

No extras. `rerank=True` wants flashrank, [`FastEncode`](https://Karthik777.github.io/litesearch/utils.html#fastencode) wants onnxruntime, and each is imported
when used and says what to install. `pip install litesearch` gets the rest.

## Route 1: [`Index`](https://Karthik777.github.io/litesearch/api.html#index)

Six methods. `add` ingests, `search` returns chunks, `sections` returns sections, `read` opens
one, `toc` lists the corpus, `context` assembles an answer.

``` python
ix = Index()                                        # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]
```

Every hit carries a `heading` breadcrumb and a `node_id`, because [`Index`](https://Karthik777.github.io/litesearch/api.html#index) builds a document tree
at ingest. That is what turns “which 512 characters” into “which section”.

``` python
sec = ix.sections('how does multi-head attention work', limit=2)   # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]
```

``` python
ix.read(sec[0]['node_id'])['text'][:300]            # one whole section, reassembled
```

``` python
ix.toc(summaries=False)                             # the corpus — no embeddings computed at all
```

One knob is left to you. `rerank=True` runs a flashrank cross-encoder over the top 30 candidates:
+0.026 to +0.077 weighted MRR, positive in all twelve measured cells, at roughly 10x query latency
and a 4 MB download on first use.

``` python
ix.search('how does multi-head attention work', rerank=True)
```

For code, `add_code` uses the AST instead of headings, and its tree is module › class › function:

``` python
ix.add_code('litesearch')      # a directory, or an installed package name
```

## Route 2: [`database()`](https://Karthik777.github.io/litesearch/core.html#database)

[`database()`](https://Karthik777.github.io/litesearch/core.html#database) returns a [fastlite](https://fastlite.answer.ai/) `Database` patched with usearch’s
SIMD distance functions. Pass a path to persist, omit it for memory.

``` python
db = database()
vecs = dict(v1=np.ones((100,), dtype=np.float32).tobytes(),
            v2=np.zeros((100,), dtype=np.float32).tobytes())
{m: db.q(f'select distance_{m}_f32(:v1,:v2) as d', vecs)[0]['d']
 for m in ['sqeuclidean', 'divergence', 'inner', 'cosine']}
```

Four metrics, `cosine`, `sqeuclidean`, `inner` and `divergence`, each in `f32`, `f16`, `f64` and
`i8`, running inside SQL.

Route 1 by hand is eight lines, and one of them is a trap:

``` python
enc   = static_embedder()             # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)

# float16, because that is what a store holds. Handing it float32 fails quietly: every distance
# comes back 0 and the ranking degrades to keyword-only with no error.
emb   = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16)

texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
         'stochastic gradient descent and learning rate schedules',
         'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
                 upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()

q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)
```

[`Index`](https://Karthik777.github.io/litesearch/api.html#index) exists because those eight lines have to be right every time.

## Threads: one connection or one each

`database(shared=True)` wraps the connection so threads can share it. Each statement runs alone on it, and a transaction keeps it until the commit. The default connection takes one thread at a time, so give each thread its own [`database()`](https://Karthik777.github.io/litesearch/core.html#database) on the same WAL file. `python -m evals.shared_conn` measures both on a 20,000-row store with an HNSW index, on a 14-core arm64 Mac.

|  | shared | one each | one each, memstatus off |
|----|----|----|----|
| FTS queries/s, 8 threads | 608 | 441 | 6,820 |
| brute-force vector scans/s, 8 threads | 70 | 176 | 168 |
| HNSW queries/s, 8 threads | 4,705 | 2,748 | 2,587 |
| FTS p50 / p99, 4 readers beside a writer | 106 / 193 ms | 13.4 / 33.0 ms | 1.6 / 3.4 ms |
| resident memory, HNSW index loaded | 17 MB | 17 MB per thread | 17 MB per thread |

With SQLite’s defaults, every allocation takes one process-wide statistics mutex, and threads on separate connections queue on it. Running `apsw.shutdown(); apsw.config(apsw.SQLITE_CONFIG_MEMSTATUS, 0); apsw.initialize()` before the first connection opens removes it. `apsw.memory_used()` then returns 0. litesearch does not set it.

Share one connection when threads already share one database object and writes are short. A writer that commits back to back on a shared connection makes readers wait a median 106 ms. Give each thread its own connection, with the statistics off, for parallel reads or a steady writer. `evals/RESULTS.md` has the method and every thread count.

## What the evaluation says

`evals/` runs 120 known-item queries per genre over three corpora (EU legislation, arXiv papers, a
19th-century astrology treatise) in five query flavours, scoring section-level MRR weighted so
three quarters of the mass sits where the query is not a copy of the answer.
`python -m evals.decide` reproduces every number.

Above the line is on by default. Below it is off and stays off.

| change | Δ weighted MRR | verdict |
|----|----|----|
| [`pre()`](https://Karthik777.github.io/litesearch/data.html#pre) on the FTS leg | +0.016 to +0.093 | on since 0.1.6 |
| 512-char chunks over page-sized | +0.06 to +0.12 | [`Index`](https://Karthik777.github.io/litesearch/api.html#index) default |
| cross-encoder rerank | +0.026 to +0.077 | `rerank=True`, the one lever worth deciding |
| HNSW ANN vector leg | −0.005 | on by default; buy the speed |
|  |  |  |
| document tree, for ranking | −0.052 to +0.011 | a wash. Built for `toc`, `read`, `sections` |
| heading prefix on the chunk | ±0.02, sign flips by genre | a wash |
| deeper fanout alone | −0.014 to −0.068 | pays only with a reranker |
| late chunking | −0.033 to −0.053 | deleted; the code is in `evals/latechunk.py` |
| entity graph leg | −0.070 to −0.160 | removed from `context`; [vruksha](https://github.com/vedicreader/vruksha) only, see `evals/RESULTS.md` |

Three findings worth more than a table row.

**The encoder is not the lever.** Across `potion-32M`, `bge-small`, `jina-v2-sm` and `egemma-300m`
the spread is 0.018 to 0.046, and the static model wins one genre outright at ~1,700x cheaper
indexing. The default is `potion-multilingual-128M`, so non-Latin scripts are covered without
choosing an encoder.

**The tree does not improve ranking and is still worth building.** Section ranking is a wash.
Section assembly is not: on the Sanskrit corpus `context()` roughly doubles verse-level recall
over plain chunk search, 0.190 to 0.340, the largest single effect in `evals/`.

**FTS alone looks unbeatable here, and that is the benchmark’s fault.** Keyword-only retrieval
with [`pre()`](https://Karthik777.github.io/litesearch/data.html#pre) beats hybrid in all 24 paired cells, because every query in the main set is a lexical
transformation of its target. `evals/multihop.py` builds the corrective: a bridge set where the
answer shares no token with the question. There FTS cannot score at all and the vector leg reaches
the target at rank 1 between 53% and 84% of the time.

## Beyond the two routes

| module | what you get |
|----|----|
| [`litesearch.tree`](https://Karthik777.github.io/litesearch/tree.html) | the tree directly: `add_dir`, `doc_search`, `context`, custom chunkers |
| [`litesearch.data`](https://Karthik777.github.io/litesearch/data.html) | [`file_parse`](https://Karthik777.github.io/litesearch/data.html#file_parse) for any file, [`pyparse`](https://Karthik777.github.io/litesearch/data.html#pyparse) for code, FTS query preprocessing |
| [`litesearch.utils`](https://Karthik777.github.io/litesearch/utils.html) | encoders: static, ONNX [`FastEncode`](https://Karthik777.github.io/litesearch/utils.html#fastencode), image and multimodal |
| [`litesearch.topics`](https://Karthik777.github.io/litesearch/topics.html) | clusters and topic labels off the ANN index |
| [`litesearch.sanskrit`](https://Karthik777.github.io/litesearch/sanskrit.html) | the cross-script FTS5 tokenizer |
| [`litesearch.quality`](https://Karthik777.github.io/litesearch/quality.html) | which documents in a store are retrieval noise |

Three things live in their own packages: [pdflite](https://github.com/vedicreader/pdflite) reads
PDFs, [ganapati](https://github.com/vedicreader/ganapati) does Sanskrit metre, verse chunking and
lemmas, and [vruksha](https://github.com/vedicreader/vruksha) builds an entity graph (external; it measures
negative for retrieval and is no longer wired into `context`, see `evals/RESULTS.md`).

**Cross-script search is on for every store**, not only Sanskrit ones. The `sanskrit` FTS5
tokenizer emits an ASCII fold of each token beside it, so `श्रीमाता`, `śrīmātā` and `srimata` all
reach the same row; a plain-English fold beside it lets `Bheeshma` and `Bhishma` reach `भीष्म` and
`Bhīṣma` too. Purely additive: English keeps its tokens, the second fold grows the index 14.7%, and it is the largest
measured retrieval win here: 1.000 Devanagari to verse recall for every encoder tested. One cost:
a store built with this chain cannot be opened by a connection that has not registered the
tokenizer, plain `sqlite3` included.

## Next Steps

- **[examples/01_simple_rag.ipynb](https://Karthik777.github.io/litesearch/examples/simple_rag.html)**, ingest a folder of PDFs, chunk with chonkie, rerank with FlashRank
- **[examples/02_tool_use.ipynb](https://Karthik777.github.io/litesearch/examples/tool_use.html)**, wire litesearch into an LLM tool-use loop
- **[api docs](https://Karthik777.github.io/litesearch/api.html)**, [`Index`](https://Karthik777.github.io/litesearch/api.html#index), and what each default is worth
- **[core docs](https://Karthik777.github.io/litesearch/core.html)**, [`database`](https://Karthik777.github.io/litesearch/core.html#database), `get_store`, `search`, [`rrf_all`](https://Karthik777.github.io/litesearch/core.html#rrf_all), [`rank_fts`](https://Karthik777.github.io/litesearch/core.html#rank_fts), [`correct`](https://Karthik777.github.io/litesearch/core.html#correct), [`fts_or`](https://Karthik777.github.io/litesearch/core.html#fts_or), `vec_search`
- **[tree docs](https://Karthik777.github.io/litesearch/tree.html)**, `add_dir`, `toc`, `read`, `sections`, `context`
- **[vishalakshi](https://github.com/vedicreader/vishalakshi)**, a litesearch-backed vault, and the first caller nominated to port onto [`Index`](https://Karthik777.github.io/litesearch/api.html#index); see the [api page](https://Karthik777.github.io/litesearch/api.html) for what that port should test

## Acknowledgements

A big thank you to [@yfedoseev](https://github.com/yfedoseev) for [pdf-oxide](https://github.com/yfedoseev/pdf-oxide), which powers the PDF extraction functionality in `litesearch.data`.
