Metadata-Version: 2.2
Name: nind
Version: 3.1.2
Summary: Python frontend for the Nind Indexing Module, with its native C++ index stack built in
Requires-Python: >=3.8
Description-Content-Type: text/markdown

# NIND — "nouvelle indexation"

[![PyPI version](https://img.shields.io/pypi/v/nind.svg)](https://pypi.org/project/nind/)
[![License: LGPL v3](https://img.shields.io/badge/license-LGPL--3.0-blue.svg)](LICENCE.md)
[![Build](https://github.com/aymara/nind/actions/workflows/build.yml/badge.svg)](https://github.com/aymara/nind/actions/workflows/build.yml)
[![Docs](https://github.com/aymara/nind/actions/workflows/docs.yml/badge.svg)](https://github.com/aymara/nind/actions/workflows/docs.yml)

**nind** is a flat-file inverted-index module: a simple, dependency-light
indexing and search engine built around plain binary files instead of a
database or a heavyweight library like Lucene. It ships as:

- a production **C++** library (`src/cpp/`), and
- a **Python package**, `nind` (`src/py/nind/`) — an ergonomic frontend to
  the same binary formats (backed by a compiled `nind._native` extension,
  pybind11 bindings over the C++ stack), plus a modern BM25-style search
  API layered on top.

📖 **Full documentation:** https://aymara.github.io/nind/

## Why nind?

In 2014, **Atejcon** designed and built nind to meet the indexing needs of
an application such as **ANT'inno**'s `ANT'box`, aiming for something
simpler and faster than the alternatives available at the time.

**CEA LIST/DIASI/LVIC** later adapted nind for its `Amose` search engine,
replacing the deprecated `S2` and `Lucene` indexers. nind was extended to
match `Amose`'s richer functional needs — in particular its more elaborate
term-type model and the various corpus-level measures its relevance
calculations require.

nind indexes and searches using nothing but flat files: no database, no
server process. Because those files are also usable directly, the indexed
corpus can be inspected and analyzed offline, with tools as simple as the
command line — see [Command-line tools](#command-line-tools) below.

To keep the binary file formats unambiguous and toolable, nind specifies
them in an **EBNF** (Extended Backus-Naur Form) grammar; class comments
throughout the source quote the relevant grammar rules directly above each
class. That the format is written down formally is also what makes an
ergonomic, non-C++ way of reading and writing the same files possible in
the first place — which is exactly what the Python package above is.

nind writes **format v2**: 64-bit document ids, no 16 MiB ceiling on a
term's postings, and long postings lists cut into blocks of 128 whose
statistics let a top-k search skip what can't reach the top. Its grammar is
in `docs/proposals/format-v2-blocks.md`; the source comments still describe
v1. Indexes written in format v1 by older versions are still read. Updating
one with `NindIndexer.update()` first rebuilds it in format v2. Amose, and
other programs that write the files themselves, need a nind that writes v1
until they move to v2.

## Installation

```bash
pip install nind
```

This installs the `nind` Python package together with its compiled
`nind._native` extension (prebuilt wheels are published for Linux, macOS
and Windows). Building from source (e.g. an sdist install, or an
unsupported platform) requires a C++14 compiler; `pip`/`uv` pull in the
rest of the build toolchain (`scikit-build-core`, `pybind11`, `cmake`,
`ninja`) automatically — no manual setup needed.

Requires Python ≥ 3.8.

## Quickstart

Index a handful of files and run a BM25-ranked search over them:

```python
from nind.nind_engine import NindIndexer, NindEngine

indexer = NindIndexer(index_dir="indices", prefix="corpus")
indexer.index_files([
    "src/py/nind/NindFile.py",
    "src/py/nind/NindPadFile.py",
    "src/py/nind/nind_engine.py",
])

engine = NindEngine("indices")
for doc_id, score in engine.search("ejcrit nombre", top_k=5):
    print(f"{doc_id}: {score:.3f}")
```

`index_dir` must already exist; each input file becomes one document,
identified externally by its position in the list (`0`, `1`, `2`, ...).
A JSON Lines corpus (pyserini's format) is indexed with
`indexer.index_jsonl("corpus.jsonl")`; its records' ids are stored, and
`engine.search(query, doc_names=True)` returns them. Tokenizer options (each identifier
split optional, lowercasing) and `local_index=False` (a much smaller index,
enough for `NindEngine`) are `NindIndexer` arguments.
To add, replace or remove documents in an existing index (or build one
file by file), use an incremental update instead of rebuilding:

```python
with indexer.update() as update:
    for path in more_files:
        update.add_file(path)       # returns the new document's id
    update.remove(1)
```

`NindEngine` also exposes the building blocks it uses internally
(`get_term_id`, `get_df`, `get_tf`, `get_doc_len`) for inspecting relevance
or a specific document directly.

See the [Quickstart guide](https://aymara.github.io/nind/quickstart.html)
for more, including how to open index files produced by the C++
implementation directly.

## Command-line tools

Every low-level reader module doubles as a runnable diagnostic script
(`analyse`/`dump`/`debug` its own file format), and three higher-level
scripts operate on a whole indexed corpus:

```bash
cd src/py/nind
uv run python3 Nind_search.py <fichier lexiconindex> <terme cherché>
uv run python3 Nind_dumpDocument.py <fichier nind> <n° doc>
uv run python3 Nind_trouveMotsSansOccurrence.py <fichier>
```

See the [command-line tools guide](https://aymara.github.io/nind/cli.html)
for the full list and usage of each.

## How it works

nind's binary files are a small fixed header, one or more fixed-size
*indirection blocks* mapping an integer identifier to an `(offset,
length)` pair, the variable-length data those indirections point to, and a
trailer holding format-specific data plus an identification stamp. Every
concrete format (the lexicon, the inverted term index, the per-document
local index, the reverse lexicon) builds on that same envelope, adding
only the meaning of its own records. An index built from Python also
stores its tokenizer options, its documents' lengths and, optionally, their
names (`.ninddocnames`: the corpus's own document ids) — see the
[architecture guide](https://aymara.github.io/nind/architecture.html) for
the full module layering, on both the C++ and Python sides.

## Development

- **C++**: `NIND_DIST=/path/to/dist ./gbuild.sh` (wraps CMake + ctest;
  see `./gbuild.sh -h` for build-mode/sanitizer/generator options).
- **Python**: `uv sync` (builds `nind._native` and installs the package
  editable, plus dev dependencies), then `uv run pytest tst/py -v`.

See `CLAUDE.md` for the full architecture reference, build details, and
release process.

## License

nind is distributed under the **GNU Lesser General Public License v3**
(LGPL-3.0) — see [LICENCE.md](LICENCE.md).
