Metadata-Version: 2.2
Name: nind
Version: 4.3.1
Summary: Python frontend for the Nind Indexing Module, with its native C++ index stack built in
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# NIND — "nouvelle indexation"

[![PyPI version](https://img.shields.io/pypi/v/nind.svg)](https://pypi.org/project/nind/)
[![License: LGPL v3](https://img.shields.io/badge/license-LGPL--3.0-blue.svg)](LICENCE.md)
[![Build](https://github.com/aymara/nind/actions/workflows/build.yml/badge.svg)](https://github.com/aymara/nind/actions/workflows/build.yml)
[![Docs](https://github.com/aymara/nind/actions/workflows/docs.yml/badge.svg)](https://github.com/aymara/nind/actions/workflows/docs.yml)

**nind** is a flat-file inverted-index module: a simple, dependency-light
indexing and search engine built around plain binary files instead of a
database or a heavyweight library like Lucene. It ships as:

- a production **C++** library (`src/cpp/`), and
- a **Python package**, `nind` (`src/py/nind/`) — an ergonomic frontend to
  the same binary formats (backed by a compiled `nind._native` extension,
  pybind11 bindings over the C++ stack), plus a modern BM25-style search
  API layered on top.

📖 **Full documentation:** https://aymara.github.io/nind/

## Why nind?

In 2014, **Atejcon** designed and built nind to meet the indexing needs of
an application such as **ANT'inno**'s `ANT'box`, aiming for something
simpler and faster than the alternatives available at the time.

**CEA LIST/DIASI/LVIC** later adapted nind for its `Amose` search engine,
replacing the deprecated `S2` and `Lucene` indexers. nind was extended to
match `Amose`'s richer functional needs — in particular its more elaborate
term-type model and the various corpus-level measures its relevance
calculations require.

nind indexes and searches using nothing but flat files: no database, no
server process. Because those files are also usable directly, the indexed
corpus can be inspected and analyzed offline, with tools as simple as the
command line — see [Command-line tools](#command-line-tools) below.

To keep the binary file formats unambiguous and toolable, nind specifies
them in an **EBNF** (Extended Backus-Naur Form) grammar; class comments
throughout the source quote the relevant grammar rules directly above each
class. That the format is written down formally is also what makes an
ergonomic, non-C++ way of reading and writing the same files possible in
the first place — which is exactly what the Python package above is.

nind writes **format v2**: 64-bit document ids, no 16 MiB ceiling on a
term's postings, and long postings lists cut into blocks of 128 whose
statistics let a top-k search skip what can't reach the top. Its grammar is
in `docs/proposals/format-v2-blocks.md`; the source comments still describe
v1. Indexes written in format v1 by older versions are still read; merges
rewrite them in format v2. Amose, and
other programs that write the files themselves, need a nind that writes v1
until they move to v2.

## Installation

```bash
pip install nind
```

This installs the `nind` Python package together with its compiled
`nind._native` extension (prebuilt wheels are published for Linux, macOS
and Windows). Building from source (e.g. an sdist install, or an
unsupported platform) requires a C++14 compiler; `pip`/`uv` pull in the
rest of the build toolchain (`scikit-build-core`, `pybind11`, `cmake`,
`ninja`) automatically — no manual setup needed.

Requires Python ≥ 3.8.

## Quickstart

Index a handful of files and run a BM25-ranked search over them:

```python
from nind.nind_engine import NindIndexer, NindEngine

indexer = NindIndexer(index_dir="indices", prefix="corpus")
indexer.index_files([
    "src/py/nind/NindFile.py",
    "src/py/nind/NindPadFile.py",
    "src/py/nind/nind_engine.py",
])

engine = NindEngine("indices")
for doc_id, score in engine.search("ejcrit nombre", top_k=5):
    print(f"{doc_id}: {score:.3f}")
```

`index_dir` must already exist; each input file becomes one document,
identified externally by its position in the list (`0`, `1`, `2`, ...).
A JSON Lines corpus (pyserini's format) is indexed with
`indexer.index_jsonl("corpus.jsonl")`; its records' ids are stored, and
`engine.search(query, doc_names=True)` returns them. Tokenizer options (each identifier
split optional, lowercasing) and `local_index=False` (a much smaller index,
enough for `NindEngine`) are `NindIndexer` arguments.
To add, replace or remove documents in an existing index (or build one
file by file), use an incremental update instead of rebuilding:

```python
with indexer.update() as update:
    for path in more_files:
        update.add_file(path)       # returns the new document's id
    update.remove(1)
```

An update costs what it changes, whatever the size of the index (88k documents added to 8.8M in
2.6 s): its documents go to a new segment, the replaced and removed ones are masked, segments are
merged now and then. For a large corpus (more than about 10M documents), give the build a memory
budget: `index_jsonl(path, memory_gb=16)` picks the fewest shards that fit in it (one, a direct
build, if the corpus fits), builds them in worker processes and keeps them as segments (no merge,
and faster: 20.5 s vs 60.1 s for MS MARCO, 306 s vs 958 s for its 138M passages). `shards=N` sets
their number instead. Results are always those of one index of the live documents;
`indexer.force_merge()` makes it one index. See `docs/proposals/segments.md`.

Searching is thread-safe: a server can call `engine.search` from many threads at once (the
GIL is released). To make one query faster on idle cores, search segments in parallel with
`NindEngine("indices", search_threads=4)` (or `search(..., threads=4)`): the same results, about
2× faster with 4 threads. The default (1) suits a server answering many queries at once. Many search
processes on one index can share its documents' lengths instead of copying them,
`NindEngine("indices", map_lengths=True)` (0.5 GB less each at 138M documents, searches 3-6% slower).
See the [benchmarks](https://aymara.github.io/nind/benchmarks.html) against Lucene, tantivy and
bm25s.

`NindEngine` also exposes the building blocks it uses internally
(`get_term_id`, `get_df`, `get_tf`, `get_doc_len`) for inspecting relevance
or a specific document directly.

See the [Quickstart guide](https://aymara.github.io/nind/quickstart.html)
for more, including how to open index files produced by the C++
implementation directly.

## Command-line tools

Every low-level reader module doubles as a runnable diagnostic script
(`analyse`/`dump`/`debug` its own file format), and three higher-level
scripts operate on a whole indexed corpus:

```bash
cd src/py/nind
uv run python3 Nind_search.py <fichier lexiconindex> <terme cherché>
uv run python3 Nind_dumpDocument.py <fichier nind> <n° doc>
uv run python3 Nind_trouveMotsSansOccurrence.py <fichier>
```

See the [command-line tools guide](https://aymara.github.io/nind/cli.html)
for the full list and usage of each.

## Milvus full text search

`NindMilvusClient` ([`examples/milvus/`](examples/milvus/), one file to copy
into your project) is a `pymilvus.MilvusClient` in which nind replaces Milvus's
built-in BM25. Code written for Milvus full text search runs unchanged:
collections, inserts, upserts, deletes, searches with filters and fields, and
hybrid searches with vector fields. On BEIR corpora and MS MARCO (8.8M
passages), it gives the same nDCG@10 and the same top 10 for 99–100% of
queries as Milvus 3.0's own BM25, and its searches returning ids are 2 to 10
times faster. It needs nind ≥ 4.2 and pymilvus 3.0. See the
[Milvus guide](https://aymara.github.io/nind/milvus.html).

## How it works

nind's binary files are a small fixed header, one or more fixed-size
*indirection blocks* mapping an integer identifier to an `(offset,
length)` pair, the variable-length data those indirections point to, and a
trailer holding format-specific data plus an identification stamp. Every
concrete format (the lexicon, the inverted term index, the per-document
local index, the reverse lexicon) builds on that same envelope, adding
only the meaning of its own records. An index built from Python also
stores its tokenizer options, its documents' lengths and, optionally, their
names (`.ninddocnames`: the corpus's own document ids) — see the
[architecture guide](https://aymara.github.io/nind/architecture.html) for
the full module layering, on both the C++ and Python sides.

## Development

- **C++**: `NIND_DIST=/path/to/dist ./gbuild.sh` (wraps CMake + ctest;
  see `./gbuild.sh -h` for build-mode/sanitizer/generator options).
- **Python**: `uv sync` (builds `nind._native` and installs the package
  editable, plus dev dependencies), then `uv run pytest tst/py -v`.

See `CLAUDE.md` for the full architecture reference, build details, and
release process.

## License

nind is distributed under the **GNU Lesser General Public License v3**
(LGPL-3.0) — see [LICENCE.md](LICENCE.md).
