Metadata-Version: 2.5
Name: litesearch
Version: 0.1.34
Summary: search through files with fts5, vectors and get reranked results. Fast
Project-URL: Repository, https://github.com/Karthik777/litesearch
Project-URL: Documentation, https://Karthik777.github.io/litesearch
Author-email: 71293 <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: document search,fts+vectors,nbdev,semantic search,text search,vector search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Requires-Dist: chonkie>=1.6.0
Requires-Dist: codesigs>=0.0.2
Requires-Dist: fastlite>=0.2.4
Requires-Dist: model2vec>=0.7.0
Requires-Dist: pdf-oxide>=0.3.17
Requires-Dist: pdflite>=0.0.1
Requires-Dist: pillow>=12.1.1
Requires-Dist: tokenizers>=0.22.2
Requires-Dist: usearch>=2.23.0
Description-Content-Type: text/markdown

# litesearch


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

> **NB** Reading this on GitHub? The formatted [documentation](https://Karthik777.github.io/litesearch/) is nicer.

litesearch stores and searches documents in one SQLite file. FTS5 keyword search and SIMD vector
search, fused by Reciprocal Rank Fusion. No server.

Two ways in. Pick by one question: do you want the defaults decided for you?

| route | use it when | what it costs |
|----|----|----|
| [\[`Index`\](https://Karthik777.github.io/litesearch/api.html#index)](07_api.ipynb) | you want to search a folder of documents or code | nothing. Encoder, dtype, chunk size, retrieval and tree all come from `evals/` |
| [\[`database()`\](https://Karthik777.github.io/litesearch/core.html#database)](01_core.ipynb) | you need your own columns, encoder, SQL or float32 vectors | six decisions, one of which fails silently |

Start at [`Index`](https://Karthik777.github.io/litesearch/api.html#index). Drop to [`database()`](https://Karthik777.github.io/litesearch/core.html#database) when it stops fitting: it is the same object underneath,
reachable as `Index.db`.

## Install

``` python
# usearch SQLite extensions are configured automatically on first import
# (macOS needs one extra step — see litesearch.postfix)
!uv add litesearch
```

No extras. `rerank=True` wants flashrank, [`FastEncode`](https://Karthik777.github.io/litesearch/utils.html#fastencode) wants onnxruntime, and each is imported
when used and says what to install. `pip install litesearch` gets the rest.

## Route 1: [`Index`](https://Karthik777.github.io/litesearch/api.html#index)

Six methods. `add` ingests, `search` returns chunks, `sections` returns sections, `read` opens
one, `toc` lists the corpus, `context` assembles an answer.

``` python
ix = Index()                                        # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]
```

Every hit carries a `heading` breadcrumb and a `node_id`, because [`Index`](https://Karthik777.github.io/litesearch/api.html#index) builds a document tree
at ingest. That is what turns “which 512 characters” into “which section”.

``` python
sec = ix.sections('how does multi-head attention work', limit=2)   # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]
```

``` python
ix.read(sec[0]['node_id'])['text'][:300]            # one whole section, reassembled
```

``` python
ix.toc(summaries=False)                             # the corpus — no embeddings computed at all
```

One knob is left to you. `rerank=True` runs a flashrank cross-encoder over the top 30 candidates:
+0.026 to +0.077 weighted MRR, positive in all twelve measured cells, at roughly 10x query latency
and a 4 MB download on first use.

``` python
ix.search('how does multi-head attention work', rerank=True)
```

For code, `add_code` uses the AST instead of headings, and its tree is module › class › function:

``` python
ix.add_code('litesearch')      # a directory, or an installed package name
```

## Route 2: [`database()`](https://Karthik777.github.io/litesearch/core.html#database)

[`database()`](https://Karthik777.github.io/litesearch/core.html#database) returns a [fastlite](https://fastlite.answer.ai/) `Database` patched with usearch’s
SIMD distance functions. Pass a path to persist, omit it for memory.

``` python
db = database()
vecs = dict(v1=np.ones((100,), dtype=np.float32).tobytes(),
            v2=np.zeros((100,), dtype=np.float32).tobytes())
{m: db.q(f'select distance_{m}_f32(:v1,:v2) as d', vecs)[0]['d']
 for m in ['sqeuclidean', 'divergence', 'inner', 'cosine']}
```

Four metrics, `cosine`, `sqeuclidean`, `inner` and `divergence`, each in `f32`, `f16`, `f64` and
`i8`, running inside SQL.

Route 1 by hand is eight lines, and one of them is a trap:

``` python
enc   = static_embedder()             # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)

# float16, because that is what a store holds. Handing it float32 fails quietly: every distance
# comes back 0 and the ranking degrades to keyword-only with no error.
emb   = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16)

texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
         'stochastic gradient descent and learning rate schedules',
         'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
                 upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()

q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)
```

[`Index`](https://Karthik777.github.io/litesearch/api.html#index) exists because those eight lines have to be right every time.

## What the evaluation says

`evals/` runs 120 known-item queries per genre over three corpora (EU legislation, arXiv papers, a
19th-century astrology treatise) in five query flavours, scoring section-level MRR weighted so
three quarters of the mass sits where the query is not a copy of the answer.
`python -m evals.decide` reproduces every number.

Above the line is on by default. Below it is off and stays off.

| change | Δ weighted MRR | verdict |
|----|----|----|
| [`pre()`](https://Karthik777.github.io/litesearch/data.html#pre) on the FTS leg | +0.016 to +0.093 | on since 0.1.6 |
| 512-char chunks over page-sized | +0.06 to +0.12 | [`Index`](https://Karthik777.github.io/litesearch/api.html#index) default |
| cross-encoder rerank | +0.026 to +0.077 | `rerank=True`, the one lever worth deciding |
| HNSW ANN vector leg | −0.005 | on by default; buy the speed |
|  |  |  |
| document tree, for ranking | −0.052 to +0.011 | a wash. Built for `toc`, `read`, `sections` |
| heading prefix on the chunk | ±0.02, sign flips by genre | a wash |
| deeper fanout alone | −0.014 to −0.068 | pays only with a reranker |
| late chunking | −0.033 to −0.053 | deleted; the code is in `evals/latechunk.py` |
| entity graph leg | −0.070 to −0.160 | [vruksha](https://github.com/vedicreader/vruksha), opt-in |

Three findings worth more than a table row.

**The encoder is not the lever.** Across `potion-32M`, `bge-small`, `jina-v2-sm` and `egemma-300m`
the spread is 0.018 to 0.046, and the static model wins one genre outright at ~1,700x cheaper
indexing. The default is `potion-multilingual-128M`, so non-Latin scripts are covered without
choosing an encoder.

**The tree does not improve ranking and is still worth building.** Section ranking is a wash.
Section assembly is not: on the Sanskrit corpus `context()` roughly doubles verse-level recall
over plain chunk search, 0.190 to 0.340, the largest single effect in `evals/`.

**FTS alone looks unbeatable here, and that is the benchmark’s fault.** Keyword-only retrieval
with [`pre()`](https://Karthik777.github.io/litesearch/data.html#pre) beats hybrid in all 24 paired cells, because every query in the main set is a lexical
transformation of its target. `evals/multihop.py` builds the corrective: a bridge set where the
answer shares no token with the question. There FTS cannot score at all and the vector leg reaches
the target at rank 1 between 53% and 84% of the time.

## Beyond the two routes

| module | what you get |
|----|----|
| [`litesearch.tree`](06_tree.ipynb) | the tree directly: `add_dir`, `doc_search`, `context`, custom chunkers |
| [`litesearch.data`](02_data.ipynb) | [`file_parse`](https://Karthik777.github.io/litesearch/data.html#file_parse) for any file, [`pyparse`](https://Karthik777.github.io/litesearch/data.html#pyparse) for code, FTS query preprocessing |
| [`litesearch.utils`](03_utils.ipynb) | encoders: static, ONNX [`FastEncode`](https://Karthik777.github.io/litesearch/utils.html#fastencode), image and multimodal |
| [`litesearch.topics`](05_topics.ipynb) | clusters and topic labels off the ANN index |
| [`litesearch.sanskrit`](09_sanskrit.ipynb) | the cross-script FTS5 tokenizer |
| [`litesearch.quality`](11_quality.ipynb) | which documents in a store are retrieval noise |

Three things live in their own packages: [pdflite](https://github.com/vedicreader/pdflite) reads
PDFs, [ganapati](https://github.com/vedicreader/ganapati) does Sanskrit metre, verse chunking and
lemmas, and [vruksha](https://github.com/vedicreader/vruksha) builds the entity graph.

**Cross-script search is on for every store**, not only Sanskrit ones. The `sanskrit` FTS5
tokenizer emits an ASCII fold of each token beside it, so `श्रीमाता`, `śrīmātā` and `srimata` all
reach the same row. Purely additive, ordinary English tokenises identically, and it is the largest
measured retrieval win here: 1.000 Devanagari to verse recall for every encoder tested. One cost:
a store built with this chain cannot be opened by a connection that has not registered the
tokenizer, plain `sqlite3` included.

## Next Steps

- **[examples/01_simple_rag.ipynb](examples/01_simple_rag.ipynb)**, ingest a folder of PDFs, chunk with chonkie, rerank with FlashRank
- **[examples/02_tool_use.ipynb](examples/02_tool_use.ipynb)**, wire litesearch into an LLM tool-use loop
- **[api docs](https://Karthik777.github.io/litesearch/api.html)**, [`Index`](https://Karthik777.github.io/litesearch/api.html#index), and what each default is worth
- **[core docs](https://Karthik777.github.io/litesearch/core.html)**, [`database`](https://Karthik777.github.io/litesearch/core.html#database), `get_store`, `search`, [`rrf_all`](https://Karthik777.github.io/litesearch/core.html#rrf_all), `vec_search`
- **[tree docs](https://Karthik777.github.io/litesearch/tree.html)**, `add_dir`, `toc`, `read`, `sections`, `context`
- **[vishalakshi](https://github.com/vedicreader/vishalakshi)**, a litesearch-backed vault, and the first caller nominated to port onto [`Index`](https://Karthik777.github.io/litesearch/api.html#index); see the [api page](07_api.ipynb) for what that port should test

## Acknowledgements

A big thank you to [@yfedoseev](https://github.com/yfedoseev) for [pdf-oxide](https://github.com/yfedoseev/pdf-oxide), which powers the PDF extraction functionality in `litesearch.data`.
