Metadata-Version: 2.5
Name: litesearch
Version: 0.1.49
Summary: search through files with fts5, vectors and get reranked results. Fast
Project-URL: Repository, https://github.com/Karthik777/litesearch
Project-URL: Documentation, https://Karthik777.github.io/litesearch
Author-email: 71293 <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: document search,fts+vectors,nbdev,semantic search,text search,vector search
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Requires-Dist: chonkie>=1.6.0
Requires-Dist: codesigs>=0.0.2
Requires-Dist: fastlite>=0.2.4
Requires-Dist: model2vec>=0.7.0
Requires-Dist: pdf-oxide>=0.3.17
Requires-Dist: pdflite>=0.0.1
Requires-Dist: pillow>=12.1.1
Requires-Dist: tokenizers>=0.22.2
Requires-Dist: usearch>=2.23.0
Provides-Extra: dev
Requires-Dist: datasets>=5.0.1; extra == 'dev'
Requires-Dist: ddgs; extra == 'dev'
Requires-Dist: fastprogress>=1.1.5; extra == 'dev'
Requires-Dist: flashrank>=0.2.10; extra == 'dev'
Requires-Dist: jupyterlab; extra == 'dev'
Requires-Dist: jupyterlab-quarto; extra == 'dev'
Requires-Dist: model2vec[distill]>=0.7.0; extra == 'dev'
Requires-Dist: nbdev; extra == 'dev'
Requires-Dist: notebook; extra == 'dev'
Requires-Dist: onnx>=1.22.0; extra == 'dev'
Requires-Dist: onnxruntime>=1.24.3; extra == 'dev'
Requires-Dist: pyperclip; extra == 'dev'
Requires-Dist: rishi>=0.0.2; extra == 'dev'
Requires-Dist: selectolax; extra == 'dev'
Description-Content-Type: text/markdown

# litesearch


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

> Reading this on GitHub? The formatted [documentation](https://Karthik777.github.io/litesearch/) is easier to read.

litesearch adds search to an application without a search server. Your documents, their search indexes and their vectors live in one SQLite file, and every query runs two kinds of search over them and merges the results.

The first kind is keyword search. SQLite’s FTS5 extension keeps a full-text index, which lists for every word the rows that contain it. FTS5 ranks matching rows with bm25, a score that is higher when a row contains the query’s words more often and when those words are rare in the collection. Keyword search is exact. That is what you want for names, error codes and identifiers, and it finds nothing when a document says the same thing in different words.

The second kind is vector search. An encoder, also called an embedding model, turns a piece of text into a list of a few hundred numbers called an embedding. Texts with similar meanings get embeddings that are close together. litesearch stores an embedding beside each piece of text, embeds the query the same way, and returns the rows whose embeddings are closest. Vector search finds “ending a membership” when the user typed “cancel my subscription”, and it is weak at exact names.

Each kind of search returns a ranked list. litesearch merges the two lists with Reciprocal Rank Fusion (RRF): a row earns 1/(60 + its position) from each list it appears in, and the amounts add up. A row near the top of both lists beats a row at the top of only one. RRF uses positions only, so it never has to compare a bm25 score with a vector distance, and there is nothing to tune.

There are two ways to use the library.

| route | use it when | what you decide |
|----|----|----|
| [\[`Index`\](https://Karthik777.github.io/litesearch/api.html#index)](https://Karthik777.github.io/litesearch/api.html) | you want to search a folder of documents, a set of strings, or a code base | only whether to rerank. The encoder, vector type, chunk size, search strategy and document tree are already chosen, each from a measurement in `evals/` |
| [\[`database()`\](https://Karthik777.github.io/litesearch/core.html#database)](https://Karthik777.github.io/litesearch/core.html) | you need your own columns, your own encoder, your own SQL, or float32 vectors | everything, including one setting (the vector dtype) that produces a wrong ranking with only a warning when you get it wrong |

Start with [`Index`](https://Karthik777.github.io/litesearch/api.html#index). It is built on [`database()`](https://Karthik777.github.io/litesearch/core.html#database), and `Index.db` is the database underneath, so you can run a lower-level query against the same data whenever [`Index`](https://Karthik777.github.io/litesearch/api.html#index) does not do what you need.

## Install

``` python
# usearch's SQLite extension is fetched, probed and configured on first use (see litesearch.postfix)
!uv add litesearch
```

litesearch has no optional extras. Two features need a package that is not installed with it: `rerank=True` needs `flashrank`, and the ONNX encoder [`FastEncode`](https://Karthik777.github.io/litesearch/utils.html#fastencode) needs `onnxruntime`. Each is imported only when you use the feature, and the error message names the package to install. `pip install litesearch` installs everything else.

## Route 1: [`Index`](https://Karthik777.github.io/litesearch/api.html#index)

An [`Index`](https://Karthik777.github.io/litesearch/api.html#index) is a searchable collection. `add` puts files, folders or text into it. Each document is cut into chunks, short passages of about 512 bytes, and each chunk is stored with its embedding. `search` returns the chunks that best match a query.

The other methods answer questions about whole documents and sections. `sections` ranks sections instead of chunks, `read` returns the full text of one section, `toc` lists everything in the index as a table of contents, and `context` collects the best sections for a query in a form you can pass to a language model.

``` python
ix = Index()                                        # pass a path to keep it on disk
ix.add('pdfs/attention_is_all_you_need.pdf')
hits = ix.search('how does multi-head attention work', limit=3)
[(h['heading'], h['page']) for h in hits]
```

Every hit has a `heading` and a `node_id`. While it ingests a document, [`Index`](https://Karthik777.github.io/litesearch/api.html#index) builds a document tree from the document’s headings: the document is the root, its chapters and sections are the nodes below, and each chunk records the node it belongs to. `heading` is the path of headings above the chunk, such as `attention is all you need › Abstract`, and `node_id` names the section as the document’s id, `#`, and the section’s number. A hit then tells you which section matched, which is what a reader or a model needs, and not only which 512 bytes.

``` python
sec = ix.sections('how does multi-head attention work', limit=2)   # ranked *sections*, not chunks
[(s['node_id'], (s['snippets'] or [''])[0][:60]) for s in sec]
```

``` python
ix.read(sec[0]['node_id'])['text'][:300]            # one whole section, reassembled
```

``` python
ix.toc(summaries=False)                             # the corpus — no embeddings computed at all
```

One setting is left to you. `rerank=True` passes the top 30 candidates to a cross-encoder from flashrank, a small model that reads the query and each candidate together and scores how well they match. That is slower than comparing embeddings and more accurate. In the evaluation it improved weighted MRR by 0.026 to 0.077 and was positive in all twelve measured cells. It makes a query roughly 10 times slower and downloads a 4 MB model the first time.

``` python
ix.search('how does multi-head attention work', rerank=True)
```

For source code, use `add_code`. It cuts Python files at top-level functions and classes using Python’s own parser. Code chunks get no document tree, so they have no `heading` or `node_id`, and `toc`, `sections` and `read` do not cover them:

``` python
ix.add_code('litesearch')      # a directory, or the name of an installed package
```

## Route 2: [`database()`](https://Karthik777.github.io/litesearch/core.html#database)

[`database()`](https://Karthik777.github.io/litesearch/core.html#database) opens a SQLite database and returns a [fastlite](https://fastlite.answer.ai/) `Database`. fastlite is a thin layer over SQLite that gives each table a Python object. The first vector search loads usearch’s SQLite extension into the connection, which adds vector distance functions you can call from SQL. `db.usearch_sql()` loads it before that, for SQL of your own. Pass a file path to keep the database on disk, or nothing for one held in memory.

``` python
db = database()
vecs = dict(v1=np.ones((100,), dtype=np.float32).tobytes(),
            v2=np.zeros((100,), dtype=np.float32).tobytes())
({m: db.q(f'select distance_{m}_f32(:v1,:v2) as d', vecs)[0]['d']
  for m in ['sqeuclidean', 'divergence', 'inner', 'cosine']}
 if db.usearch_sql() else 'usearch is off here, as on macOS 14')
```

There are four distance functions: `cosine`, `sqeuclidean` (squared Euclidean distance), `inner` (one minus the dot product) and `divergence` (Jensen-Shannon divergence). Each comes in four versions for the number type the vectors are stored in: `f32`, `f16`, `f64` and `i8`. Smaller results mean closer vectors.

The cell below does by hand what [`Index`](https://Karthik777.github.io/litesearch/api.html#index) does for you. It loads an encoder, creates a store (a table with a text column, an embedding column and a full-text index), embeds five sentences, and searches them.

The comment in the middle points at the one mistake that is easy to make. An embedding is stored as raw bytes, so a search has to be told the number type the bytes hold. litesearch assumes float16, which uses half the space of float32. The static encoder returns float32, so this code converts to float16 before storing. If you store float32 and search as float16, every distance comes back empty, litesearch prints a warning, and the vector half of the search returns rows in table order.

``` python
enc   = static_embedder()             # model2vec: no GPU, no ONNX runtime
store = db.get_store(hash=True, ann=True)

# float16, because that is what a store holds. Handing it float32 fails quietly: every distance
# comes back 0 and the ranking degrades to keyword-only with no error.
emb   = lambda xs: np.asarray(enc.encode(list(xs)), dtype=np.float16)

texts = ['attention mechanisms in neural networks', 'transformer architecture for sequences',
         'stochastic gradient descent and learning rate schedules',
         'positional encoding and token embeddings', 'dropout reduces overfitting']
store.insert_all([dict(content=t, embedding=e.tobytes()) for t, e in zip(texts, emb(texts))],
                 upsert=True, hash_id='id', hash_id_columns=['content'])
store.rebuild_index()

q = 'self-attention mechanism'
db.search(q, emb([q])[0].tobytes(), columns=['content'], limit=2)
```

[`Index`](https://Karthik777.github.io/litesearch/api.html#index) exists so that this setup is written once, correctly.

## Threads: share one connection, or give each thread its own

A SQLite connection used from several threads needs care. `database(shared=True)` wraps the connection so that threads can share it safely: each statement runs on its own, and a thread that opens a transaction keeps the connection until it commits. The default connection allows one thread at a time, so the alternative is to give each thread its own [`database()`](https://Karthik777.github.io/litesearch/core.html#database) on the same file. SQLite’s WAL mode, which litesearch turns on, lets those connections read while another writes.

`python -m evals.shared_conn` measures both on a store of 20,000 rows with an HNSW index, on a 14-core arm64 Mac:

|  | shared | one each | one each, memstatus off |
|----|----|----|----|
| FTS queries/s, 8 threads | 608 | 441 | 6,820 |
| brute-force vector scans/s, 8 threads | 70 | 176 | 168 |
| HNSW queries/s, 8 threads | 4,705 | 2,748 | 2,587 |
| FTS p50 / p99, 4 readers beside a writer | 106 / 193 ms | 13.4 / 33.0 ms | 1.6 / 3.4 ms |
| resident memory, HNSW index loaded | 17 MB | 17 MB per thread | 17 MB per thread |

The last column needs explaining. By default SQLite counts every memory allocation under one lock shared by the whole process, so threads on separate connections still wait for each other. Running `apsw.shutdown(); apsw.config(apsw.SQLITE_CONFIG_MEMSTATUS, 0); apsw.initialize()` before the first connection opens turns that counting off. After that, `apsw.memory_used()` returns 0. litesearch does not do this for you, because it changes the whole process.

Share one connection when your threads already share one database object and writes are short. On a shared connection, a writer that commits continuously makes readers wait a median of 106 ms. For parallel reads, or a steady writer, give each thread its own connection and turn the counting off. `evals/RESULTS.md` has the method and the results for every thread count.

## What the evaluation says

The defaults come from `evals/`. It runs 120 known-item queries per corpus over three corpora: EU legislation, arXiv papers and a 19th-century astrology treatise. A known-item query is one written to find a specific passage. The queries come in five styles, from a copy of the passage’s words to a paraphrase that shares none of them. The score is MRR (mean reciprocal rank: 1 if the right section is first, 1/2 if second, and so on, averaged over queries), counted per section and weighted so that three quarters of the weight falls on queries that do not copy the answer’s words. `python -m evals.decide` reproduces every number.

The changes above the blank row are on by default. The ones below it are off and stay off.

| change | change in weighted MRR | decision |
|----|----|----|
| [`pre()`](https://Karthik777.github.io/litesearch/data.html#pre) on the keyword search: drop stopwords, match word prefixes, join the words with OR | +0.016 to +0.093 | on since 0.1.6 |
| 512-byte chunks instead of whole pages | +0.06 to +0.12 | the [`Index`](https://Karthik777.github.io/litesearch/api.html#index) default |
| cross-encoder rerank | +0.026 to +0.077 | `rerank=True`, the one setting worth deciding |
| HNSW index for the vector search | -0.005 | on by default, for its speed |
|  |  |  |
| document tree, used for ranking | -0.052 to +0.011 | no difference. The tree is built for `toc`, `read` and `sections` |
| heading text added to each chunk | about ±0.02, sign depends on the corpus | no difference |
| more candidates without a reranker | -0.014 to -0.068 | helps only with a reranker |
| late chunking | -0.033 to -0.053 | removed; the code is in `evals/latechunk.py` |
| entity graph | -0.070 to -0.160 | removed from `context`; lives in [vruksha](https://github.com/vedicreader/vruksha), see `evals/RESULTS.md` |

Three results matter more than any one row.

The choice of encoder matters little. Across `potion-32M`, `bge-small`, `jina-v2-sm` and `egemma-300m` the spread is 0.018 to 0.046. The static model, which looks up a vector per token instead of running a neural network, wins one corpus outright and indexes about 1,700 times more cheaply. The default is `potion-multilingual-128M`, a static model that also covers scripts other than Latin.

The document tree does not improve ranking, and it is still worth building. Ranking sections instead of chunks makes no difference. Assembling them does: on a Sanskrit corpus, `context()` nearly doubles verse-level recall over plain chunk search, from 0.190 to 0.340, the largest single effect in `evals/`.

Keyword search alone beats the hybrid search on this benchmark, and the benchmark is the reason. With [`pre()`](https://Karthik777.github.io/litesearch/data.html#pre), keyword-only search beats hybrid in all 24 paired cells, because every query in the main set is a rewording of words from its target. `evals/multihop.py` builds a second set where the answer shares no word with the question. There keyword search cannot score at all, and the vector search finds the target at rank 1 between 53% and 84% of the time.

## The other modules

| module | what you get |
|----|----|
| [`litesearch.tree`](https://Karthik777.github.io/litesearch/tree.html) | the document tree directly: `add_dir`, `doc_search`, `context`, custom chunkers |
| [`litesearch.data`](https://Karthik777.github.io/litesearch/data.html) | [`file_parse`](https://Karthik777.github.io/litesearch/data.html#file_parse) for any file, [`pyparse`](https://Karthik777.github.io/litesearch/data.html#pyparse) for Python code, and the helpers that turn a typed query into an FTS5 query |
| [`litesearch.utils`](https://Karthik777.github.io/litesearch/utils.html) | encoders: static, ONNX [`FastEncode`](https://Karthik777.github.io/litesearch/utils.html#fastencode), image and multimodal |
| [`litesearch.topics`](https://Karthik777.github.io/litesearch/topics.html) | groups of similar chunks and labels for them, read off the HNSW index |
| [`litesearch.sanskrit`](https://Karthik777.github.io/litesearch/sanskrit.html) | the FTS5 tokenizer that matches words across scripts |
| [`litesearch.quality`](https://Karthik777.github.io/litesearch/quality.html) | which documents in a store only add noise to search results |

Three related jobs live in separate packages. [pdflite](https://github.com/vedicreader/pdflite) reads PDFs. [ganapati](https://github.com/vedicreader/ganapati) handles Sanskrit metre, verse chunking and lemmas. [vruksha](https://github.com/vedicreader/vruksha) builds an entity graph; it measured negative for retrieval and `context` no longer calls it (see `evals/RESULTS.md`).

Every store matches words across scripts, not only Sanskrit stores. A tokenizer is the part of FTS5 that splits text into the words it indexes. litesearch’s default tokenizer adds, beside each word, a copy of it reduced to plain ASCII letters, so `श्रीमाता` (Devanagari), `śrīmātā` (romanised with diacritics) and `srimata` all reach the same row. It also adds a second, looser copy that follows English spelling habits, so `Bheeshma` and `Bhishma` reach `भीष्म` and `Bhīṣma`. English words keep their own entries, so nothing that matched before stops matching. The looser copy makes the index 14.7% larger. This is the largest retrieval gain measured here: a Devanagari query finds the right verse every time (recall 1.000) with every encoder tested. The cost is that a database built with this tokenizer cannot be searched by a connection that has not registered it, which includes the plain `sqlite3` command-line tool. Open it with [`database()`](https://Karthik777.github.io/litesearch/core.html#database).

## Next steps

- [examples/01_simple_rag.ipynb](https://Karthik777.github.io/litesearch/examples/simple_rag.html) ingests a folder of PDFs, chunks them with chonkie and reranks with FlashRank.
- [examples/02_tool_use.ipynb](https://Karthik777.github.io/litesearch/examples/tool_use.html) gives litesearch to a language model as a tool.
- The [api page](https://Karthik777.github.io/litesearch/api.html) covers [`Index`](https://Karthik777.github.io/litesearch/api.html#index) and what each default is worth.
- The [core page](https://Karthik777.github.io/litesearch/core.html) covers [`database`](https://Karthik777.github.io/litesearch/core.html#database), `get_store`, `search`, [`rrf_all`](https://Karthik777.github.io/litesearch/core.html#rrf_all), [`rank_fts`](https://Karthik777.github.io/litesearch/core.html#rank_fts), [`correct`](https://Karthik777.github.io/litesearch/core.html#correct), [`fts_or`](https://Karthik777.github.io/litesearch/core.html#fts_or) and `vec_search`.
- The [tree page](https://Karthik777.github.io/litesearch/tree.html) covers `add_dir`, `toc`, `read`, `sections` and `context`.
- [vishalakshi](https://github.com/vedicreader/vishalakshi) is a document vault built on [`Index`](https://Karthik777.github.io/litesearch/api.html#index).

## Acknowledgements

Thank you to [@yfedoseev](https://github.com/yfedoseev) for [pdf-oxide](https://github.com/yfedoseev/pdf-oxide), which litesearch uses to extract text from PDFs.
