# whoosh-ng : Full Technical Documentation



## DOCUMENT: Analysis

# Analysis API

Classes and functions for turning text into indexable "tokens" (usually words).
Analysis is the first step in the indexing pipeline: an analyzer tokenizes text
and applies zero or more filters to the resulting token stream.

## Overview

Three general categories of objects make up the analysis pipeline:

- **Tokenizers** split text into individual tokens (words, n-grams, identifiers).
  Every tokenizer is callable: `tokenizer(text) -> iterator of Token objects`.
- **Filters** transform one token stream into another. Common operations include
  lowercasing, stop-word removal, stemming, and synonym expansion. Every filter
  is callable: `filter(token_generator) -> token_generator`.
- **Analyzers** compose a tokenizer and zero or more filters into a single unit.
  Every analyzer is callable and can be used directly as a field's `analyzer`
  argument.

Tokenizers and filters are combined using the `|` operator:

```python
my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

The first item must be a tokenizer; subsequent items must be filters.

## Composition

### Composable

```python
class whoosh.analysis.Composable
```

Base class for tokenizers and filters, providing `|` composition.

**Attributes:**
- `is_morph (bool)`: Whether this object performs morphological transformation
  (e.g. stemming). Defaults to `False`.

**Methods:**

#### `__or__(self, other)`

Combines this object with `other` using `CompositeAnalyzer`.

```python
analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

### CompositeAnalyzer

```python
class whoosh.analysis.CompositeAnalyzer
```

Composed analyzer created by chaining a tokenizer and filters with `|`.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, LowercaseFilter, StopFilter

analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
tokens = list(analyzer("Hello world, this is a test"))
```

## Token

```python
class whoosh.analysis.Token
```

Represents a single token (usually a word) extracted from source text.
Tokenizers yield the **same** `Token` object repeatedly (for performance), so
consumers must not hold references between iterations.

**Slots:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `text` | `str` | The text of this token |
| `pos` | `int` | Token position (if `positions=True`) |
| `startchar` | `int` | Start character offset (if `chars=True`) |
| `endchar` | `int` | End character offset (if `chars=True`) |
| `original` | `str` | Original text before filters (if `keeporiginal=True`) |
| `positions` | `bool` | Whether position info was requested |
| `chars` | `bool` | Whether character offsets were requested |
| `stopped` | `bool` | Set by `StopFilter` |
| `boost` | `float` | Token boost factor (default `1.0`) |
| `removestops` | `bool` | Whether stop words should be removed |
| `mode` | `str` | `'index'` or `'query'` |
| `boosts` | `dict` | Per-position boost values (if requested) |
| `tokenize` | `bool` | Whether tokenization should proceed |
| `matched` | `bool` | Used during highlighting |
| `fieldname` | `str` | Field name for this token |

**Methods:**

#### `copy()`

Returns a new `Token` with the same attribute values. Use this if you need to
retain a token between iterations.

```python
def remove_duplicates(stream):
    last = None
    for t in stream:
        if last != t.text:
            yield t
        last = t.text
```

## Utility Functions

### entoken

```python
whoosh.analysis.entoken(
    textstream,
    positions=False,
    chars=False,
    start_pos=0,
    start_char=0,
    **kwargs
) -> Iterator[Token]
```

Converts a sequence of strings into a stream of `Token` objects.

### unstopped

```python
whoosh.analysis.unstopped(tokenstream) -> Iterator[Token]
```

Removes tokens where `token.stopped` is `True`.

## Analyzers

### Analyzer (Base)

```python
class whoosh.analysis.Analyzer
```

Abstract base class for all analyzers. Subclasses implement `__call__`.

### CompositeAnalyzer

Created automatically when you use `|` to compose tokenizers and filters.

### Predefined Analyzers

#### IDAnalyzer

```python
whoosh.analysis.IDAnalyzer(lowercase=False) -> Analyzer
```

Yields the entire input as a single token. Deprecated; use `IDTokenizer` directly.

- `lowercase (bool)`: If True, add a `LowercaseFilter`.

#### KeywordAnalyzer

```python
whoosh.analysis.KeywordAnalyzer(
    lowercase=False,
    commas=False
) -> Analyzer
```

Splits on whitespace or commas. Suitable for field values that are lists of
keywords.

- `lowercase (bool)`: Lowercase each token.
- `commas (bool)`: Split on commas instead of whitespace.

**Example:**
```python
from whoosh.analysis import KeywordAnalyzer

an = KeywordAnalyzer(lowercase=True, commas=True)
list(an("Hello, WORLD, test"))
# => ["hello", "world", "test"]
```

#### RegexAnalyzer

```python
whoosh.analysis.RegexAnalyzer(
    expression=r"\w+(\.?\w+)*",
    gaps=False
) -> Analyzer
```

Deprecated; use `RegexTokenizer` directly.

#### SimpleAnalyzer

```python
whoosh.analysis.SimpleAnalyzer(
    expression=default_pattern,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer` with `LowercaseFilter`.

- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = SimpleAnalyzer()
list(an("Hello there, this is a TEST"))
# => ["hello", "there", "this", "is", "a", "test"]
```

#### StandardAnalyzer

```python
whoosh.analysis.StandardAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and optional `StopFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length (default `None`, no limit).
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = StandardAnalyzer()
list(an("Testing is testing and testing"))
# => ["testing", "testing", "testing"]
```

#### StemmingAnalyzer

```python
whoosh.analysis.StemmingAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False,
    stemfn=stem,
    ignore=None,
    cachesize=50000
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, optional `StopFilter`, and
`StemFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length.
- `gaps`: If True, split on the expression instead of matching it.
- `stemfn`: Stemming function (default: Porter stemmer for English).
- `ignore`: Words to not stem (set).
- `cachesize`: Stem cache size (default `50000`). Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
an = StemmingAnalyzer()
list(an("Testing is testing and testing"))
# => ["test", "test", "test"]
```

#### FancyAnalyzer

```python
whoosh.analysis.FancyAnalyzer(
    expression=r"\s+",
    stoplist=STOP_WORDS,
    minsize=2,
    gaps=True,
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
) -> Analyzer
```

Composes `RegexTokenizer`, `IntraWordFilter`, `LowercaseFilter`, and `StopFilter`.
Splits on whitespace and breaks compound words into subwords.

**Example:**
```python
an = FancyAnalyzer()
list(an("Should I call getInt or get_real?"))
# => ["should", "call", "get", "int", "get", "real"]
```

#### LanguageAnalyzer

```python
whoosh.analysis.LanguageAnalyzer(
    lang,
    expression=default_pattern,
    gaps=False,
    cachesize=50000
) -> Analyzer
```

Configures a language-specific analyzer with `LowercaseFilter`, `StopFilter`,
and `StemFilter`.

- `lang`: Language code (e.g., `"en"`, `"es"`, `"fr"`).
- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.
- `cachesize`: Stem cache size.

Available languages: `ar`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `hu`, `it`,
`no`, `pt`, `ro`, `ru`, `es`, `sv`, `tr`.

See `whoosh.lang` for `has_stemmer()` and `has_stopwords()` helper functions.

## Tokenizers

All tokenizers inherit from `Tokenizer`.

### Tokenizer

```python
class whoosh.analysis.Tokenizer
```

Base class for tokenizers. Each tokenizer is callable and yields `Token`
objects.

### RegexTokenizer

```python
class whoosh.analysis.RegexTokenizer(
    expression=default_pattern,
    gaps=False
)
```

Uses a regular expression to extract tokens from text. Each match of the
expression equals one token; group 0 (the entire match) is used as the text.

- `expression`: Compiled regex or pattern string.
- `gaps`: If True, split on the expression rather than matching it.

**Example:**
```python
from whoosh.analysis import RegexTokenizer

rext = RegexTokenizer()
list(rext("hi there 3.141 big-time under_score"))
# => ["hi", "there", "3.141", "big", "time", "under_score"]
```

### IDTokenizer

```python
class whoosh.analysis.IDTokenizer
```

Yields the entire input string as a single token. Used for indexed but
untokenized fields (e.g., document paths).

### CharsetTokenizer

```python
class whoosh.analysis.CharsetTokenizer(charmap)
```

Tokenizes and translates text according to a character mapping dictionary.
Characters that map to `None` are treated as token break characters.

- `charmap`: Mapping from integer character codes to unicode characters
  (as used by `unicode.translate()`).

### PathTokenizer

```python
class whoosh.analysis.PathTokenizer(expression="[^/]+")
```

Tokenizes path strings into hierarchical prefixes. Given `"/a/b/c"`, yields
`["/a", "/a/b", "/a/b/c"]`.

### NgramTokenizer

```python
class whoosh.analysis.NgramTokenizer(minsize, maxsize=None)
```

Splits input text into N-grams instead of words. Unlike `RegexTokenizer`, this
tokenizer does not use a regex, so grams may include whitespace and punctuation.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).

**Example:**
```python
from whoosh.analysis import NgramTokenizer

ngt = NgramTokenizer(4)
list(ngt("hi there"))
# => ["hi t", "i th", " the", "ther", "here"]
```

### CachedRegexTokenizer

```python
class whoosh.analysis.CachedRegexTokenizer(
    expression=default_pattern,
    gaps=False,
    maxsize=8192
)
```

A `RegexTokenizer` wrapper that caches tokenization results for repeated
strings, trading memory for speed.

- `expression`: Regex pattern.
- `gaps`: If True, split on the expression.
- `maxsize`: Maximum cache size (LRU eviction when exceeded).

### SpaceSeparatedTokenizer

```python
whoosh.analysis.SpaceSeparatedTokenizer() -> RegexTokenizer
```

Returns a `RegexTokenizer` that splits on whitespace.

### CommaSeparatedTokenizer

```python
whoosh.analysis.CommaSeparatedTokenizer() -> CompositeAnalyzer
```

Returns a composed analyzer that splits on commas and strips whitespace.

## Filters

All filters inherit from `Filter`.

### Filter

```python
class whoosh.analysis.Filter
```

Base class for filters. Subclasses implement `__call__(self, tokens)` which
takes a token generator and returns a token generator.

- `is_morph (bool)`: Set to `True` for morphological filters (e.g., stemming).
  This allows the filter to be bypassed during query analysis if desired.

### STOP_WORDS

```python
whoosh.analysis.STOP_WORDS
```

A frozenset of common English stop words: `"a"`, `"an"`, `"and"`, `"the"`, etc.
Used as the default stoplist for `StopFilter` and `StandardAnalyzer`.

### url_pattern

```python
whoosh.analysis.url_pattern
```

A compiled regex useful for URL filtering.

### LowercaseFilter

```python
class whoosh.analysis.LowercaseFilter
```

Lowercases token text using `unicode.lower()`.

**Example:**
```python
rext = RegexTokenizer() | LowercaseFilter()
list(rext("This is a TEST"))
# => ["this", "is", "a", "test"]
```

### StopFilter

```python
class whoosh.analysis.StopFilter(
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    renumber=True,
    lang=None
)
```

Marks and optionally removes stop words from the token stream.

- `stoplist`: Set of words to filter out (defaults to `STOP_WORDS`).
- `minsize`: Minimum token length; shorter tokens are removed (default `2`).
- `maxsize`: Maximum token length; longer tokens are removed (default `None`).
- `renumber`: Renumber positions to account for removed tokens (default `True`).
- `lang`: If set, loads stop words for the given language code.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StopFilter

stopper = RegexTokenizer() | StopFilter()
list(stopper("this is a test"))
# => ["test"]
```

### StripFilter

```python
class whoosh.analysis.StripFilter
```

Calls `unicode.strip()` on each token's text.

### CharsetFilter

```python
class whoosh.analysis.CharsetFilter(charmap)
```

Translates token text using `unicode.translate()` with the given character map.
Useful for case folding and accent folding.

- `charmap`: Dictionary mapping character ordinals to unicode characters.

**Example:**
```python
from whoosh.support.charset import accent_map

rext = RegexTokenizer() | CharsetFilter(accent_map)
list(rext("café"))
# => ["cafe"]
```

### DelimitedAttributeFilter

```python
class whoosh.analysis.DelimitedAttributeFilter(
    delimiter="^",
    attribute="boost",
    default=1.0,
    type=float
)
```

Looks for delimiter characters in token text and extracts data after the
delimiter into a named token attribute.

- `delimiter`: Separator character (default `"^"`).
- `attribute`: Attribute name on the token (default `"boost"`).
- `default`: Default value if no delimiter is found (default `1.0`).
- `type`: Type to cast the extracted value (default `float`).

**Example:**
```python
from whoosh.analysis import RegexTokenizer, DelimitedAttributeFilter

daf = DelimitedAttributeFilter()
an = RegexTokenizer(r"\S+") | daf
for t in an(u"image 3.14^2 render"):
    print(t.text, t.boost)
# image 1.0
# 3.14 2.0
# render 1.0
```

### SubstitutionFilter

```python
class whoosh.analysis.SubstitutionFilter(pattern, replacement)
```

Performs regex substitution on token text using `re.sub()`.

- `pattern`: Pattern string or compiled regex.
- `replacement`: Replacement text.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, SubstitutionFilter

# Remove hyphens
ana = RegexTokenizer(r"\S+") | SubstitutionFilter("-", "")
```

### MultiFilter

```python
class whoosh.analysis.MultiFilter(**kwargs)
```

Selects between two or more sub-filters based on the `mode` attribute of the
token stream. Useful for using different filters during indexing vs. querying.

- Keyword arguments map mode names to filter instances.

**Example:**
```python
from whoosh.analysis import MultiFilter, IntraWordFilter

iwf_index = IntraWordFilter(mergewords=True, mergenums=True)
iwf_query = IntraWordFilter(mergewords=False, mergenums=False)
mf = MultiFilter(index=iwf_index, query=iwf_query)
```

### TeeFilter

```python
class whoosh.analysis.TeeFilter(*filters)
```

Interleaves the results of two or more filter chains. Requires at least two
filters. Note: this filter is slow because it creates token copies.

**Example:**
```python
# Lowercase in one branch, reverse in another
f1 = LowercaseFilter()
f2 = ReverseTextFilter()
ana = RegexTokenizer(r"\S+") | TeeFilter(f1, f2)
```

### ReverseTextFilter

```python
class whoosh.analysis.ReverseTextFilter
```

Reverses the text of each token.

**Example:**
```python
an = RegexTokenizer() | ReverseTextFilter()
list(an("hello there"))
# => ["olleh", "ereht"]
```

### PassFilter

```python
class whoosh.analysis.PassFilter
```

Identity filter; passes tokens through unchanged.

### LoggingFilter

```python
class whoosh.analysis.LoggingFilter(logger=None)
```

Prints debug log entries for every token that passes through.

- `logger`: Logger instance (defaults to `whoosh.analysis` logger).

## Intraword Filters

### IntraWordFilter

```python
class whoosh.analysis.IntraWordFilter(
    delims="-_'\"()!@#$%^&*[]{}<>\\|;:,./?`~+=",
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
)
```

Splits words into subwords and performs optional merging. Based on
WordDelimiterFilter in Solr.

- `delims`: String of delimiter characters.
- `splitwords`: Split at case transitions (e.g., `PowerShot` → `Power`, `Shot`).
- `splitnums`: Split at letter-number transitions (e.g., `SD500` → `SD`, `500`).
- `mergewords`: Merge consecutive alphabetic subwords.
- `mergenums`: Merge consecutive numeric subwords.

### CompoundWordFilter

```python
class whoosh.analysis.CompoundWordFilter(wordset, keep_compound=True)
```

Breaks compound tokens into their constituent parts if they match words in the
given wordset. Useful for agglutinative languages and trademarks.

- `wordset`: A set (or any `__contains__` object) of known words.
- `keep_compound`: If True, keep the original compound token in the stream.

### BiWordFilter

```python
class whoosh.analysis.BiWordFilter(sep="-")
```

Merges adjacent tokens into bigram tokens. Useful for pseudo-phrase searching.

- `sep`: Separator string for bigrams.

### ShingleFilter

```python
class whoosh.analysis.ShingleFilter(size=2, sep="-")
```

Merges N adjacent tokens into multi-word tokens (shingles).

- `size`: Number of tokens to combine.
- `sep`: Separator string.

**Note:** For `size=2`, `BiWordFilter` is faster.

## Morphological Filters

### StemFilter

```python
class whoosh.analysis.StemFilter(
    stemfn=stem,
    lang=None,
    ignore=None,
    cachesize=50000
)
```

Stems tokens using the Porter stemming algorithm (or a language-specific
stemmer if `lang` is specified).

- `stemfn`: Stemming function (default: Porter stemmer).
- `lang`: Language code to override `stemfn` with a Snowball stemmer.
- `ignore`: Set of words to not stem (defaults to stemming all words).
- `cachesize`: Cache size for stemmed words. Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StemFilter

stemmer = RegexTokenizer() | StemFilter()
list(stemmer("fundamentally willows"))
# => ["fundament", "willow"]
```

### PyStemmerFilter

```python
class whoosh.analysis.PyStemmerFilter(
    lang="english",
    ignore=None,
    cachesize=10000
)
```

Subclass of `StemFilter` that uses the third-party `py-stemmer` library.
Requires the py-stemmer package to be installed.

**Methods:**
- `algorithms()`: Returns available stemming algorithms from py-stemmer.

### DoubleMetaphoneFilter

```python
class whoosh.analysis.DoubleMetaphoneFilter(
    primary_boost=1.0,
    secondary_boost=0.5,
    combine=False
)
```

Encodes tokens using Lawrence Philips's Double Metaphone algorithm. Useful
for phonetic matching of names and places.

- `primary_boost`: Boost factor for the primary code token.
- `secondary_boost`: Boost factor for the secondary code token.
- `combine`: If True, keep the original token alongside the encoded tokens.

## N-gram Filters and Analyzers

### NgramFilter

```python
class whoosh.analysis.NgramFilter(minsize, maxsize=None, at=None)
```

Splits token text into N-grams of varying sizes.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).
- `at`: `'start'` for prefix grams, `'end'` for suffix grams, or `None`
  for all position grams.

### NgramAnalyzer

```python
whoosh.analysis.NgramAnalyzer(minsize, maxsize=None) -> Analyzer
```

Composes `NgramTokenizer` with `LowercaseFilter`.

### NgramWordAnalyzer

```python
whoosh.analysis.NgramWordAnalyzer(
    minsize,
    maxsize=None,
    tokenizer=None,
    at=None
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and `NgramFilter`. Use this
when you want sub-word n-grams (without whitespace) rather than raw
character n-grams.


## DOCUMENT: Automata

# Automata API

Module for constructing and manipulating finite state automata (FSAs),
including NFAs, DFAs, finite state transducers (FSTs), Levenshtein
automata, and regular expression automata. Used internally for spelling
correction, fuzzy term queries, and term dictionary operations.

The automata module is a refactored package with submodules. All classes
and functions are importable directly from `whoosh.automata`.

## Module Functions

### `parse_glob`

```python
whoosh.automata.parse_glob(pattern, _glob_multi="*", _glob_single="?", _glob_range1="[", _glob_range2="]") -> NFA
```

Parses a glob-style pattern string and returns an NFA that matches strings
matching the pattern.

**Parameters:**
- `pattern`: Glob pattern string (`*` matches any sequence, `?` matches any
  single character).
- `_glob_multi`, `_glob_single`: Override the wildcard characters.
- `_glob_range1`, `_glob_range2`: Override the range syntax brackets.

### `glob_automaton`

```python
whoosh.automata.glob_automaton(pattern) -> NFA
```

Convenience function that parses a glob pattern and returns an NFA.

## FSA (Finite State Automaton) Classes

### `FSA`

```python
class whoosh.automata.FSA(initial)
```

Base class for finite state automata.

**Constructor:**
- `initial`: The initial state.

**Attributes:**
- `initial`: Initial state.
- `transitions`: Dict mapping source states to dicts mapping labels to
  target states.
- `final_states`: Set of accepting (final) states.

**Methods:**
- `__eq__(other)`: Compares initial state, final states, and transitions.
- `all_states()`: Returns a set of all states reachable in the automaton.
- `all_labels()`: Returns a set of all transition labels.
- `get_labels(src)`: Yields all labels leaving state `src`.
- `generate_all(state=None, sofar="")`: Yields all strings accepted by the
  automaton.
- `move(state, label)`: Returns the state reached by following `label` from
  `state`, or `None`.
- `moves(state, labels)`: Yields `(label, next_state)` pairs.
- `next(state)`: Yields target states reachable from `state` via any label.
- `is_final(state)`: Returns `True` if `state` is a final state.
- `start()`: Returns the initial state.
- `has_path_to(target)`: Returns `True` if there is a path to `target`.

### `Marker`

```python
class whoosh.automata.Marker(name)
```

Marker object used as a special transition label in NFAs (e.g., `ANY`,
`EPSILON`).

### `EPSILON`

```python
whoosh.automata.EPSILON = Marker("EPSILON")
```

Special marker representing an epsilon transition (no input consumed).

### `ANY`

```python
whoosh.automata.ANY = Marker("ANY")
```

Special marker representing a transition that matches any input character.

### `NFA`

```python
class whoosh.automata.NFA(initial)
```

Nondeterministic Finite Automaton. Extends `FSA` with epsilon transitions
and NFA-specific construction methods.

**Methods:**
- `add_transition(src, label, dst)`: Adds a transition from `src` to `dst`
  consuming `label`.
- `add_final_state(state, final=True)`: Marks `state` as a final/accepting
  state.
- `epsilon_closure(state)`: Returns the set of states reachable from `state`
  via epsilon transitions.
- `to_dfa()`: Converts this NFA to an equivalent DFA and returns it.

### `DFA`

```python
class whoosh.automata.DFA(initial)
```

Deterministic Finite Automaton. Extends `FSA` with DFA-specific operations.

**Methods:**
- `next_valid_string(string)`: Finds the lexicographically smallest string
  accepted by the DFA that is greater than or equal to `string`.
- `to_dfa()`: Returns self (already a DFA).

### `renumber_dfa`

```python
whoosh.automata.renumber_dfa(dfa, base=0) -> DFA
```

Renumerates the states of a DFA to integers starting at `base`.

### `u_to_utf8`

```python
whoosh.automata.u_to_utf8(dfa, base=0) -> DFA
```

Converts a Unicode DFA to a UTF-8 DFA.

### `find_all_matches`

```python
whoosh.automata.find_all_matches(dfa, lookup_func, first=unull)
```

Yields all strings accepted by the DFA, using `lookup_func` to determine
which strings exist in the dictionary.

**Parameters:**
- `dfa`: A deterministic finite automaton.
- `lookup_func`: Function called with each candidate string; returns the
  string if found in the dictionary.
- `first`: First string to start matching from (default `chr(0)`).

### `reverse_nfa`

```python
whoosh.automata.reverse_nfa(n) -> NFA
```

Returns the reverse of an NFA (reversed transitions, swapped initial
and final states).

### `product`

```python
whoosh.automata.product(dfa1, op, dfa2) -> DFA
```

Computes the product of two DFAs using a binary operation.

**Parameters:**
- `dfa1`, `dfa2`: Input DFAs.
- `op`: A function `(set1, set2) -> set` computing the output final states
  from the two input final state sets.

### `intersection`

```python
whoosh.automata.intersection(dfa1, dfa2) -> DFA
```

Returns the intersection of two DFAs.

### `union`

```python
whoosh.automata.union(dfa1, dfa2) -> DFA
```

Returns the union of two DFAs.

### `epsilon_nfa`

```python
whoosh.automata.epsilon_nfa() -> NFA
```

Returns an NFA that accepts only the empty string.

### `dot_nfa`

```python
whoosh.automata.dot_nfa() -> NFA
```

Returns an NFA that accepts any single character.

### `basic_nfa`

```python
whoosh.automata.basic_nfa(label) -> NFA
```

Returns an NFA that accepts exactly the string `label`.

### `charset_nfa`

```python
whoosh.automata.charset_nfa(labels) -> NFA
```

Returns an NFA that accepts any single character in `labels`.

### `string_nfa`

```python
whoosh.automata.string_nfa(string) -> NFA
```

Returns an NFA that accepts exactly `string`.

### `choice_nfa`

```python
whoosh.automata.choice_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts strings accepted by either `n1` or `n2`.

### `concat_nfa`

```python
whoosh.automata.concat_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts the concatenation of `n1` and `n2`.

### `star_nfa`

```python
whoosh.automata.star_nfa(n) -> NFA
```

Returns an NFA that accepts zero or more repetitions of `n`.

### `plus_nfa`

```python
whoosh.automata.plus_nfa(n) -> NFA
```

Returns an NFA that accepts one or more repetitions of `n`.

### `optional_nfa`

```python
whoosh.automata.optional_nfa(n) -> NFA
```

Returns an NFA that accepts zero or one occurrence of `n`.

### `strings_dfa`

```python
whoosh.automata.strings_dfa(strings) -> DFA
```

Constructs a minimal DFA that accepts exactly the given strings.

### `add_suffix`

```python
whoosh.automata.add_suffix(dfa, nodes, last, downto, seen)
```

Internal function for adding suffixes to a trie during DFA construction.

## Levenshtein Automata

### `levenshtein_automaton`

```python
whoosh.automata.levenshtein_automaton(term, k, prefix=0) -> NFA
```

Constructs an NFA that matches all strings within edit distance `k` of
`term`. This is the core function for fuzzy term queries and spelling
suggestions.

**Parameters:**
- `term`: The reference string to compute edit distance from.
- `k`: Maximum edit distance (number of insertions, deletions, or
  substitutions).
- `prefix`: If positive, require matched strings to share this length of
  prefix with `term` (speeds up matching significantly).

**Returns:** An NFA that can be converted to a DFA via `.to_dfa()`.

```python
from whoosh.automata import levenshtein_automaton

nfa = levenshtein_automaton("hello", k=1, prefix=0)
dfa = nfa.to_dfa()
```

## RegEx

### `parse`

```python
whoosh.automata.parse(pattern) -> NFA
```

Parses a regular expression pattern string and returns an NFA.

**Parameters:**
- `pattern`: A regex pattern string (Python `re`-style syntax).

### `RegexBuilder`

```python
class whoosh.automata.RegexBuilder(pattern)
```

Helper class for building NFAs from regex patterns.

## FST (Finite State Transducer) Classes

### `Values`

```python
class whoosh.automata.Values
```

Abstract base class for value types stored in FST arcs.

### `IntValues`

```python
class whoosh.automata.IntValues
```

Stores integer values in FST arcs.

### `SequenceValues`

```python
class whoosh.automata.SequenceValues
```

Base class for value types that store sequences of values.

### `BytesValues`

```python
class whoosh.automata.BytesValues
```

Stores byte string values in FST arcs.

### `ArrayValues`

```python
class whoosh.automata.ArrayValues
```

Stores arrays of values in FST arcs.

### `IntListValues`

```python
class whoosh.automata.IntListValues
```

Stores lists of integers in FST arcs.

### `Node`

```python
class whoosh.automata.Node
```

Base class for nodes in an FST.

### `ComboNode`

```python
class whoosh.automata.ComboNode
```

Base class for nodes that combine multiple sub-nodes (intersection, union).

### `UnionNode`

```python
class whoosh.automata.UnionNode
```

A node that represents the union of multiple sub-nodes.

### `IntersectionNode`

```python
class whoosh.automata.IntersectionNode
```

A node that represents the intersection of multiple sub-nodes.

### `BaseCursor`

```python
class whoosh.automata.BaseCursor
```

Base class for cursors that iterate over FST contents.

### `Cursor`

```python
class whoosh.automata.Cursor
```

Concrete cursor for iterating over an FST, supporting `next()`, `find()`,
`text()`, and other navigation methods.

### `UncompiledNode`

```python
class whoosh.automata.UncompiledNode
```

Represents an FST node that has not yet been compiled into a binary
representation. Used during FST construction.

### `Arc`

```python
class whoosh.automata.Arc
```

Represents a single arc in an FST, with a label, target node, and associated
value.

### `GraphWriter`

```python
class whoosh.automata.GraphWriter
```

Writes an FST to a binary file on disk or to an in-memory buffer.

### `BaseGraphReader`

```python
class whoosh.automata.BaseGraphReader
```

Base class for reading FSTs from disk.

### `GraphReader`

```python
class whoosh.automata.GraphReader
```

Concrete reader for FSTs stored on disk. Supports `find()`, `next()`, and
`text()` for navigating the graph.

### `to_labels`

```python
whoosh.automata.to_labels(key)
```

Converts a key (string, int, etc.) into a list of FST arc labels.

### `within`

```python
whoosh.automata.within(graph, text, k=1, prefix=0, address=None)
```

Uses a pre-built FST and a Levenshtein automaton to find all keys in the
graph within edit distance `k` of `text`.

**Parameters:**
- `graph`: A `GraphReader` instance.
- `text`: The search term.
- `k`: Maximum edit distance.
- `prefix`: Required shared prefix length.
- `address`: Optional starting address in the graph.

### `dump_graph`

```python
whoosh.automata.dump_graph(graph, address=None, tab=0, out=None)
```

Debug utility that prints the structure of an FST to stdout or a file.

### `FileVersionError`

```python
class whoosh.automata.FileVersionError
```

Raised when reading an FST file with an incompatible version.

### `InactiveCursor`

```python
class whoosh.automata.InactiveCursor
```

Raised when operating on a cursor that is not at a valid position.


## DOCUMENT: Backends

# Backends API

Storage backend abstractions.

## Backend (ABC)

```python
class whoosh.backends.abc.Backend
```

Abstract base class for storage backends.

### Methods

#### `create()`

Create a new segment.

#### `open()`

Open an existing segment.

#### `close()`

Close the backend.

#### `commit()`

Commit changes.

#### `startup()`

Called on backend startup.

#### `shutdown()`

Called on backend shutdown.

---

## FileBackend

```python
class whoosh.backends.file.FileBackend
```

Default backend storing index as files.

```python
from whoosh.backends.file import FileBackend
from whoosh.store.filestore import FileStorage

storage = FileStorage("indexdir")
backend = FileBackend(storage=storage)
```

---

## SQLiteBackend

```python
class whoosh.backends.sqlite.SQLiteBackend
```

Stores index in SQLite database.

```python
from whoosh.backends.sqlite import SQLiteBackend
from whoosh.store.sqlite import SQLiteStorage

storage = SQLiteStorage("index.db")
backend = SQLiteBackend(storage=storage)
```

---

## MemoryBackend

```python
class whoosh.backends.memory.MemoryBackend
```

In-memory backend for testing.

```python
from whoosh.backends.memory import MemoryBackend

backend = MemoryBackend()
```

---

## BackendRegistry

```python
class whoosh.registry.BackendRegistry
```

Register backends:

```python
BackendRegistry.register("my_backend", MyBackendClass, "my_package")
backend = BackendRegistry.get("my_backend")
```


## DOCUMENT: Classify

# Classify API

Classes and functions for classifying and extracting information from
documents. This module provides query expansion models, similarity
functions (shingling, simhash), and clustering algorithms.

## Expansion Models

### `ExpansionModel`

```python
class whoosh.classify.ExpansionModel(doc_count, field_length)
```

Abstract base class for query expansion models. Subclass to implement custom
expansion scoring.

**Constructor:**
- `doc_count`: Total number of documents in the collection.
- `field_length`: Total length of the field across all documents.

**Computed Attributes:**
- `N`: Document count.
- `collection_total`: Total field length.
- `mean_length`: Average field length (`collection_total / N`).

**Methods:**
- `normalizer(maxweight, top_total)`: Returns a normalization factor.
- `score(weight_in_top, weight_in_collection, top_total)`: Returns the
  expansion score for a term.

### `Bo1Model`

```python
class whoosh.classify.Bo1Model(doc_count, field_length)
```

Bayesian One-Poisson expansion model. One of the standard query expansion
models.

### `Bo2Model`

```python
class whoosh.classify.Bo2Model(doc_count, field_length)
```

Bayesian Two-Poisson expansion model. Another standard query expansion model.

### `KLModel`

```python
class whoosh.classify.KLModel(doc_count, field_length)
```

Kullback-Leibler divergence-based expansion model.

## Expander

### `Expander`

```python
class whoosh.classify.Expander(
    ixreader,
    fieldname,
    model=Bo1Model
)
```

Uses an `ExpansionModel` to expand the set of query terms based on the top N
result documents.

**Constructor:**
- `ixreader`: An `IndexReader` object.
- `fieldname`: The name of the field to expand terms from.
- `model`: An `ExpansionModel` class or instance. Defaults to `Bo1Model`.

**Methods:**

#### `add(vector)`

Adds forward-index information about one of the "top N" documents.

- `vector`: A series of `(text, weight)` tuples, such as is returned by
  `Reader.vector_as("weight", docnum, fieldname)`.

#### `add_document(docnum)`

Adds a document's term vector to the expander. If the field has a term vector,
uses it; otherwise falls back to stored field text.

#### `add_text(string)`

Adds a text string by indexing it with the field's analyzer.

#### `expanded_terms(number, normalize=True)`

Returns the N most important terms in the vectors added so far, ranked by
the expansion model's score.

- `number`: Number of terms to return.
- `normalize`: Whether to normalize weights.
- Returns: List of `(term, weight)` tuples, sorted by weight descending.

```python
from whoosh.classify import Expander, Bo1Model

expander = Expander(ix.reader(), "content")
for docnum in results.ids()[:10]:
    expander.add_document(docnum)

for word, weight in expander.expanded_terms(5):
    print(word, weight)
```

## Similarity Functions

### `shingles`

```python
whoosh.classify.shingles(input, size=2) -> iterable
```

Generates `(shingle, frequency)` pairs from a string by sliding a window of
the given size over the input.

**Parameters:**
- `input`: The input string.
- `size`: The shingle size (default `2`).

```python
from whoosh.classify import shingles

for shingle, freq in shingles("hello world", size=2):
    print(shingle, freq)
```

### `simhash`

```python
whoosh.classify.simhash(features, hashbits=32) -> int
```

Computes a simhash (perceptual hash) from a sequence of weighted features.
Simhashes that are similar produce similar hash values, allowing fast
near-duplicate detection via Hamming distance.

**Parameters:**
- `features`: Iterable of `(feature, weight)` tuples.
- `hashbits`: Number of bits in the hash (default `32`).
- Returns: An integer hash value.

```python
from whoosh.classify import shingles, simhash

h1 = simhash(shingles(text1))
h2 = simhash(shingles(text2))
from whoosh.classify import hamming_distance
dist = hamming_distance(h1, h2)
```

### `hamming_distance`

```python
whoosh.classify.hamming_distance(first_hash, other_hash, hashbits=32) -> int
```

Computes the Hamming distance between two hash values. A small distance
indicates high similarity.

**Parameters:**
- `first_hash`: First hash integer.
- `other_hash`: Second hash integer.
- `hashbits`: Number of bits in the hashes (default `32`).

## Clustering

### `kmeans`

```python
whoosh.classify.kmeans(
    data,
    k,
    t=0.0001,
    distfun=None,
    maxiter=50,
    centers=None
) -> (labels, centroids)
```

One-dimensional K-means clustering. Assigns each data point to the nearest
of `k` centroids and returns cluster labels and final centroids.

**Parameters:**
- `data`: List of data points (numeric values).
- `k`: Number of clusters.
- `t`: Tolerance; stops if centroid changes are below this value.
- `distfun`: Optional distance function (unused if `None`).
- `maxiter`: Maximum iterations (default `50`).
- `centers`: Optional list of initial centroids. If `None`, selects `k`
  random points from `data`.

**Returns:** A tuple `(labels, centroids)` where `labels` is a list of
cluster assignments per data point and `centroids` is the list of final
centroid positions.

### `two_pass_variance`

```python
whoosh.classify.two_pass_variance(data) -> float
```

Computes the sample variance of a data list using the two-pass algorithm
(first pass computes the mean, second pass accumulates squared deviations).

### `weighted_incremental_variance`

```python
whoosh.classify.weighted_incremental_variance(data_weight_pairs) -> float
```

Computes the weighted variance incrementally from a sequence of
`(value, weight)` pairs.

### `swin`

```python
whoosh.classify.swin(data, size) -> list
```

Sliding window clustering that groups data points where the range (max - min)
within a window of `size` is below a threshold. Uses variance for ranking.

**Parameters:**
- `data`: Sorted list of data points.
- `size`: Maximum window range (max - min) for clustering.

**Returns:** A list of `(left, right, count, variance)` tuples representing
clusters, sorted by count descending then by variance ascending.


## DOCUMENT: Codecs

# Codecs API

Classes and interfaces for how Whoosh writes and reads the inverted index,
postings, and per-document values. The codecs module is a refactored package
exposing the same public API as the former monolithic module.

## Module Functions

### `default_codec`

```python
whoosh.codec.default_codec(*args, **kwargs) -> Codec
```

Returns the default codec used by the index. Currently returns a
`W3Codec` instance.

```python
from whoosh.codec import default_codec
codec = default_codec()
```

## Exceptions

### `OutOfOrderError`

```python
whoosh.codec.OutOfOrderError
```

Raised when documents are added to a field out of order. Fields must
receive documents in ascending docnum order.

## Base Classes

### `Codec`

```python
class whoosh.codec.Codec
```

Abstract base class for index codecs. Subclasses implement methods for
writing and reading the index format.

**Class Attributes:**
- `length_stats (bool)`: If `True`, the codec stores per-document field
  length statistics. Default `True`.

**Methods:**

#### `per_document_writer(storage, segment)`

Abstract. Returns a `PerDocumentWriter` for writing per-document values
(columns, term vectors) to the given segment.

#### `field_writer(storage, segment)`

Abstract. Returns a `FieldWriter` for writing postings to the given segment.

#### `postings_writer(dbfile, byteids=False)`

Abstract. Returns a `PostingsWriter` for writing posting lists to `dbfile`.

#### `postings_reader(dbfile, terminfo, format_, term=None, scorer=None)`

Abstract. Returns a `Matcher` for reading postings from `dbfile`.

#### `automata(storage, segment)`

Returns an `Automata` instance for spelling correction using automata-based
edit distance. Default returns a base `Automata()` object.

#### `terms_reader(storage, segment)`

Abstract. Returns a `TermsReader` for reading the term dictionary and
postings of the given segment.

#### `per_document_reader(storage, segment)`

Abstract. Returns a `PerDocumentReader` for reading per-document values
from the given segment.

#### `new_segment(storage, indexname)`

Abstract. Creates and returns a new `Segment` object for the given storage
and index name.

### `WrappingCodec`

```python
class whoosh.codec.WrappingCodec(child)
```

A `Codec` that delegates all operations to a child codec. Useful for
creating codec wrappers that modify or intercept specific operations.

**Constructor:**
- `child`: The underlying `Codec` instance to wrap.

All methods delegate to the child codec:
`per_document_writer()`, `field_writer()`, `postings_writer()`,
`postings_reader()`, `automata()`, `terms_reader()`, `per_document_reader()`,
`new_segment()`.

## Writer Classes

### `PerDocumentWriter`

```python
class whoosh.codec.PerDocumentWriter
```

Abstract base class for writing per-document values (columns, term vectors).

**Methods:**

#### `start_doc(docnum)`

Abstract. Called when starting to write a new document.

#### `add_field(fieldname, fieldobj, value, length)`

Abstract. Adds a field value to the current document.

#### `add_column_value(fieldname, columnobj, value)`

Abstract. Adds a column value. Raises `NotImplementedError` if the codec
doesn't support columns.

#### `add_vector_items(fieldname, fieldobj, items)`

Abstract. Adds term vector items.

#### `add_vector_matcher(fieldname, fieldobj, vmatcher)`

Convenience method that reads items from a `Matcher` and calls
`add_vector_items()`.

#### `finish_doc()`

Called when finishing a document. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `FieldWriter`

```python
class whoosh.codec.FieldWriter
```

Abstract base class for writing postings (inverted index) data.

**Methods:**

#### `add_postings(schema, lengths, items)`

Translates a generator of `(fieldname, btext, docnum, weight, vbytes)`
tuples into calls to `start_field()`, `start_term()`, `add()`,
`finish_term()`, and `finish_field()`.

**Parameters:**
- `schema`: The `Schema` object.
- `lengths`: Optional `FieldLengthTable` for document field lengths.
- `items`: Iterable of posting tuples.

#### `start_field(fieldname, fieldobj)`

Abstract. Called when starting a new field.

#### `start_term(text)`

Abstract. Called when starting a new term within a field.

#### `add(docnum, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `add_spell_word(fieldname, text)`

Called to add a word to the spelling index. Default does nothing.

#### `finish_term()`

Abstract. Called when finishing a term.

#### `finish_field()`

Called when finishing a field. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `PostingsWriter`

```python
class whoosh.codec.PostingsWriter
```

Abstract base class for writing posting lists (the inverted index).

**Methods:**

#### `start_postings(format_, terminfo)`

Abstract. Starts writing postings for a new term.

#### `add_posting(id_, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `finish_postings(allow_compact=True)`

Called when finished writing postings. Default does nothing.

#### `written()`

Abstract. Returns `True` if this writer has already written to disk.

## Reader Classes

### `FieldCursor`

```python
class whoosh.codec.FieldCursor
```

Abstract base class for iterating over terms in a field.

**Methods:**
- `first()`: Move to the first term.
- `find(string)`: Find a term matching or closest to `string`.
- `next()`: Move to the next term.
- `term()`: Returns the current term's text.

### `EmptyCursor`

```python
class whoosh.codec.EmptyCursor
```

A `FieldCursor` representing an empty field. All methods return `None` or
`False`.

### `TermsReader`

```python
class whoosh.codec.TermsReader
```

Abstract base class for reading the term dictionary and postings of a
segment.

**Methods:**
- `__contains__(term)`: Returns `True` if the term exists.
- `cursor(fieldname, fieldobj)`: Returns a `FieldCursor`.
- `terms()`: Yields `(fieldname, text)` tuples for all terms.
- `terms_from(fieldname, prefix)`: Yields terms from `fieldname` starting
  with `prefix`.
- `items()`: Yields `((fieldname, text), TermInfo)` tuples.
- `items_from(fieldname, prefix)`: Like `items()` but filtered by prefix.
- `term_info(fieldname, text)`: Returns a `TermInfo` for the term.
- `frequency(fieldname, text)`: Returns the total frequency.
- `doc_frequency(fieldname, text)`: Returns the document frequency.
- `matcher(fieldname, text, format_, scorer=None)`: Returns a `Matcher`.
- `indexed_field_names()`: Yields names of indexed fields.
- `close()`: Close the reader.

### `PerDocumentReader`

```python
class whoosh.codec.PerDocumentReader
```

Abstract base class for reading per-document values (columns, term vectors,
stored fields).

**Methods:**
- `close()`: Close the reader.
- `doc_count()`: Returns number of non-deleted documents.
- `doc_count_all()`: Returns total document count (including deleted).
- `has_deletions()`: Returns `True` if any documents are deleted.
- `is_deleted(docnum)`: Returns `True` if docnum is deleted.
- `deleted_docs()`: Yields docnums of deleted documents.
- `all_doc_ids()`: Yields docnums of all non-deleted documents.
- `supports_columns()`: Returns `True` if column storage is supported.
- `has_column(fieldname)`: Returns `True` if field has a column.
- `list_columns()`: Yields names of available columns.
- `column_reader(fieldname, column)`: Returns a column reader.
- `doc_field_length(docnum, fieldname)`: Returns field length for docnum.
- `field_length(fieldname)`: Returns total field length.
- `min_field_length(fieldname)`: Returns minimum field length.
- `max_field_length(fieldname)`: Returns maximum field length.
- `has_vector(docnum, fieldname)`: Returns `True` if docnum has a vector.
- `vector(docnum, fieldname, format_)`: Returns a `Matcher` for the vector.
- `stored_fields(docnum)`: Returns dict of stored field values.
- `all_stored_field()`: Yields stored fields for all documents.

### `MultiPerDocumentReader`

```python
class whoosh.codec.MultiPerDocumentReader(readers, offset=0)
```

Combines multiple `PerDocumentReader` instances into one for multi-segment
indices.

**Constructor:**
- `readers`: List of `PerDocumentReader` instances.
- `offset`: Base document offset (usually `0`).

## Automata

### `Automata`

```python
class whoosh.codec.Automata
```

Provides static methods for automata-based term matching, used by the
spelling corrector.

**Static Methods:**

#### `levenshtein_dfa(uterm, maxdist, prefix=0)`

Returns a deterministic finite automaton (DFA) that matches all edit-distance
variants of `uterm` within `maxdist` edits, optionally requiring a minimum
shared prefix of length `prefix`.

#### `find_matches(dfa, cur)`

Given a DFA and a `FieldCursor`, yields all matching terms.

**Methods:**

#### `terms_within(fieldcur, uterm, maxdist, prefix=0)`

Returns an iterator of matching terms within the given edit distance of
`uterm`.

## Segment

### `Segment`

```python
class whoosh.codec.Segment
```

Represents a segment of the index. Instances are pickled into the TOC file
to describe on-disk files.

**Class Attributes:**
- `COMPOUND_EXT = ".seg"`: Extension for compound segment files.

**Instance Attributes:**
- `indexname`: Base name of the segment.
- `segid`: Random unique ID string.
- `compound (bool)`: Whether this segment uses compound file format.

**Methods:**
- `make_filename(ext)`: Returns `f"{segment_id()}{ext}"`.
- `list_files(storage)`: Lists all files belonging to this segment.
- `create_file(storage, ext, **kwargs)`: Creates a new file for this segment.
- `open_file(storage, ext, **kwargs)`: Opens a file for this segment.
- `create_compound_file(storage)`: Combines all segment files into a
  compound `.seg` file.
- `open_compound_file(storage)`: Opens the compound segment file.
- `doc_count_all()`: Abstract. Returns total document count.
- `doc_count()`: Returns non-deleted document count.
- `set_doc_count(doccount)`: Sets the document count.
- `has_deletions()`: Returns `True` if any documents are deleted.
- `deleted_count()`: Abstract. Returns number of deleted documents.
- `deleted_docs()`: Abstract. Yields docnums of deleted documents.
- `delete_document(docnum, delete=True)`: Abstract. Deletes/undeletes a
  document.
- `is_deleted(docnum)`: Abstract. Returns `True` if docnum is deleted.
- `should_assemble()`: Returns `True` by default. Override to control
  compound file behavior.
- `validate(storage)`: Checks on-disk integrity of this segment.
- `segment_id()`: Returns the unique segment identifier string.
- `is_compound()`: Returns `True` if this segment uses compound file format.

### `WrappingSegment`

```python
class whoosh.codec.WrappingSegment(child)
```

A `Segment` that delegates all operations to a child segment.

**Constructor:**
- `child`: The underlying `Segment` instance to wrap.

## W3 Codec (Default)

The `W3` codec ("Whoosh 3") is the default index format, storing postings in
compressed blocks for efficient reading and skipping.

### `W3Codec`

```python
class whoosh.codec.whoosh3.W3Codec(blocklimit=128, compression=3, inlinelimit=1)
```

The default codec. Uses compressed blocks and term inlining for efficient
storage and fast lookups.

**Constructor:**
- `blocklimit`: Number of postings per block (default `128`).
- `compression`: zlib compression level (default `3`, `0` = no compression).
- `inlinelimit`: Maximum number of postings to inline directly in the term
  info (default `1`).

**File Extensions:**
- `.trm`: Term dictionary
- `.pst`: Postings
- `.vps`: Vector postings
- `.col`: Per-document value columns

### `W3PerDocWriter`

Writer for per-document values using the W3 format. Handles columns,
stored fields, term vectors, and field lengths.

### `W3FieldWriter`

Writer for the inverted term index using the W3 format. Uses a
`OrderedHashWriter` for the term dictionary and posts to a postings file.

### `W3LeafMatcher`

```python
class whoosh.codec.whoosh3.W3LeafMatcher(postfile, startoffset, length, format_, term=None, byteids=None, scorer=None)
```

Reads on-disk postings from the postings file and presents the
`Matcher` interface. Supports block-level skipping and lazy block loading.

**Optimization methods:**
- `block_min_id()`: Returns the first doc ID in the current block.
- `block_max_id()`: Returns the last doc ID in the current block.
- `block_min_length()`: Returns the minimum field length in the current block.
- `block_max_length()`: Returns the maximum field length in the current block.
- `block_max_weight()`: Returns the maximum weight in the current block.
- `skip_to_quality(minquality)`: Skips blocks exceeding a quality threshold.

### `W3TermsReader`

Reader for the term dictionary using the W3 format. Uses an
`OrderedHashReader` for fast lookups.

### `W3TermInfo`

```python
class whoosh.codec.whoosh3.W3TermInfo
```

Stores term statistics and posting location information. Supports inlining
small posting sets directly in the term dictionary for fast lookups.

**Flags:**
- `_FLAG_OFFSET` (0): Postings stored at an offset in the postings file.
- `_FLAG_INLINE_PICKLE` (1): Postings inlined as a pickled tuple.
- `_FLAG_INLINE_COMPACT` (2): Single posting compactly inlined.
- `_FLAG_INLINE_COMPACT_SHORT` (3): Multiple postings compactly inlined.

**Methods:**
- `add_block(block)`: Merges block statistics into this term info.
- `set_extent(offset, length)`: Sets offset and length of postings in file.
- `extent()`: Returns `(offset, length)`.
- `set_inlined(ids, weights, values)`: Sets inlined posting data.
- `set_compact_inline(id_, weight, value)`: Sets single inlined posting.
- `set_compact_short_inline(ids, weights, values)`: Sets multiple compact
  inlined postings.
- `is_inlined()`: Returns `True` if postings are inlined.
- `inlined_postings()`: Returns `(ids, weights, values)` tuples for inlined
  postings.
- `to_bytes()` / `from_bytes()`: Serialize/deserialize.

### `W3Segment`

```python
class whoosh.codec.whoosh3.W3Segment(codec, indexname, doccount=0, segid=None, deleted=None)
```

Segment class for the W3 codec. Stores a reference to the codec, document
count, and deleted document set.

## Plain Text Codec (Debugging)

### `PlainTextCodec`

```python
class whoosh.codec.plaintext.PlainTextCodec
```

A codec that stores the index as human-readable plain text. Intended for
debugging and manual inspection, not for production use.

**Class Attributes:**
- `length_stats = False`

**File extensions:**
- `.dcs`: Document (stored fields, columns, vectors)
- `.trm`: Term dictionary (plain text)

### `PlainPerDocWriter`

Plain text writer for per-document values.

### `PlainPerDocReader`

Plain text reader for per-document values.

### `PlainFieldWriter`

Plain text writer for the inverted index.

### `PlainTermsReader`

Plain text reader for the term dictionary.

### `PlainSegment`

```python
class whoosh.codec.plaintext.PlainSegment(indexname)
```

Segment class for the plain text codec. Does not support compound files
(`should_assume()` returns `False`).

## Memory Codec

### `MemoryCodec`

```python
class whoosh.codec.memory.MemoryCodec
```

An in-memory-only codec for testing. Stores all data in Python objects
rather than on disk.

**Class Attributes:**
- `storage`: A `RamStorage` instance.
- `segment`: A `MemSegment` instance.

**Methods:**
- `writer(schema)`: Returns a `MemWriter`.
- `reader(schema)`: Returns a `SegmentReader`.

### `MemWriter`

```python
class whoosh.codec.memory.MemWriter
```

A `SegmentWriter` subclass that commits immediately without merging.

### `MemPerDocWriter`

In-memory writer for per-document values.

### `MemPerDocReader`

In-memory reader for per-document values.

### `MemFieldWriter`

In-memory writer for the inverted index.

### `MemTermsReader`

In-memory reader for the term dictionary.

### `MemSegment`

```python
class whoosh.codec.memory.MemSegment(codec, indexname)
```

In-memory segment storing all data in Python dictionaries (inverted index,
stored fields, lengths, vectors, term infos). Uses a `Lock` for thread-safe
access.


## DOCUMENT: Collectors

# Collectors API

Classes and functions for gathering search results. Collectors are used
internally by `Searcher.search()` to collect matching documents and build
`Results` objects. The collectors module is a refactored package exposing the
same public API as the former monolithic module.

## Overview

A `Collector` iterates over matching documents in an index, collects
information about them, and produces a `Results` object. The base `Collector`
class defines the interface; specialized subclasses implement different
collection strategies (top-N, unlimited, sorting, filtering, faceting, etc.).

## Core Classes

### `Collector`

```python
class whoosh.collectors.Collector
```

Abstract base class for all collectors. Subclasses must implement `collect()`
and `results()`.

**Methods:**

#### `prepare(top_searcher, q, context)`

Called before a search begins. Sets up `self.top_searcher`, `self.q`,
`self.context`, `self.starttime`, and `self.docset`.

#### `run()`

Iterates over sub-searchers, calling `set_subsearcher()` and
`collect_matches()` for each, then calls `finish()`.

#### `set_subsearcher(subsearcher, offset)`

Called when moving to a new sub-searcher. Sets `self.subsearcher`,
`self.offset`, and `self.matcher`.

#### `collect(sub_docnum)`

Called for every matched document. Must add the document to results and
return a sort key. Subclasses must implement this.

- `sub_docnum`: Segment-relative document number. Add `self.offset` to get
  the top-level document number.

#### `sort_key(sub_docnum)`

Returns a sort key for the current match without the side effect of adding
the document to results. Subclasses must implement this.

#### `collect_matches()`

Calls `matches()` and then `collect()` for each matched document.

#### `matches()`

Yields segment-relative document numbers for matches in the current
sub-searcher.

#### `count()`

Returns the total number of matching documents.

#### `all_ids()`

Returns a sequence of docnums matched in this collector.

#### `computes_count()`

Returns `True` if the collector naturally computes the exact count of
matching documents.

#### `finish()`

Called after the search completes. Sets `self.runtime`.

#### `remove(global_docnum)`

Removes a document from the collector using its global docnum.

#### `results()`

Returns a `Results` object. Subclasses must implement this.

### `ilen`

```python
whoosh.collectors.ilen(iterator) -> int
```

Counts the number of items in an iterator without loading it all into memory.

## Scored Collectors

### `ScoredCollector`

```python
class whoosh.collectors.ScoredCollector(replace=10)
```

Base class for collectors that sort by document score.

**Constructor:**
- `replace`: Number of matches between attempts to replace the matcher with
  a more efficient version.

### `TopCollector`

```python
class whoosh.collectors.TopCollector(
    limit=10,
    usequality=True,
    **kwargs
)
```

A collector that returns only the top N scored results.

**Constructor:**
- `limit`: Maximum number of results to return.
- `usequality`: Whether to use block-quality optimizations for faster
  search. Can be set to `False` for debugging.

**Notes:**
- When `usequality=True`, `computes_count()` returns `False` and
  `all_ids()` requires re-searching.
- Uses a min-heap to efficiently track the top N documents.

### `UnlimitedCollector`

```python
class whoosh.collectors.UnlimitedCollector(reverse=False)
```

A collector that returns **all** scored results. Sorts by score (descending
by default).

**Constructor:**
- `reverse`: If `True`, sort results in ascending order (lowest scores first).

### `UnsortedCollector`

```python
class whoosh.collectors.UnsortedCollector
```

A collector that returns results in document order (no sorting). Used when
the search weighting is `None`.

## Wrapping Collectors

### `WrappingCollector`

```python
class whoosh.collectors.WrappingCollector(child)
```

Base class for collectors that wrap other collectors. Delegates most
operations to the child collector while adding additional behavior.

**Constructor:**
- `child`: The collector to wrap.

**Methods** (all delegated to child):
`top_searcher`, `context`, `prepare`, `set_subsearcher`, `all_ids`,
`count`, `collect_matches`, `sort_key`, `collect`, `remove`, `matches`,
`finish`, `results()`

### `SortingCollector`

```python
class whoosh.collectors.SortingCollector(
    sortedby,
    limit=10,
    reverse=False
)
```

A collector that returns results sorted by a `FacetType` object.

**Constructor:**
- `sortedby`: A `FacetType` or field name to sort by.
- `limit`: Maximum number of results (0 for no limit).
- `reverse`: If `True`, reverse the overall sort order.

### `FilterCollector`

```python
class whoosh.collectors.FilterCollector(
    child,
    allow=None,
    restrict=None
)
```

A collector that allows and/or restricts certain document numbers in
results.

A document is discarded if:
- `allow` is set and the docnum is not in the allowed set, or
- `restrict` is set and the docnum is in the restricted set.

**Constructor:**
- `child`: The collector to wrap.
- `allow`: A query, `Results` object, or set-like of allowed docnums.
  `None` means everything is allowed.
- `restrict`: A query, `Results` object, or set-like of disallowed docnums.
  `None` means nothing is disallowed.

**Attributes:**
- `filtered_count`: Number of documents filtered out.

### `FacetCollector`

```python
class whoosh.collectors.FacetCollector(child, groupedby, maptype=None)
```

A collector that creates groups of documents based on facet objects. Used
when `groupedby` is specified in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `groupedby`: A field name, `FacetType`, dict, or `Facets` object.
- `maptype`: Default `FacetMap` class for facets that don't specify one.

**Attributes:**
- `facetmaps`: Dictionary of facet name to `FacetMap` objects.

### `CollapseCollector`

```python
class whoosh.collectors.CollapseCollector(
    child,
    keyfacet,
    limit=1,
    order=None
)
```

A collector that eliminates all but the top N results sharing the same facet
key. Useful for "dedup" or grouped result views.

**Constructor:**
- `child`: The collector to wrap.
- `keyfacet`: A `FacetType` to collapse on. All but the top N documents
  sharing a key are eliminated.
- `limit`: Maximum documents to keep per key (default `1`).
- `order`: Optional `FacetType` to determine which documents are "top" within
  each group. Defaults to the results order (e.g., highest score).

**Attributes:**
- `collapsed_counts`: Dictionary mapping keys to the number of documents
  eliminated.

### `TimeLimitCollector`

```python
class whoosh.collectors.TimeLimitCollector(
    child,
    timelimit,
    greedy=False,
    use_alarm=True
)
```

A collector that raises a `TimeLimit` exception if the search exceeds a
time limit. Partial results are still available via `results()`.

**Constructor:**
- `child`: The collector to wrap.
- `timelimit`: Maximum search time in seconds.
- `greedy`: If `True`, finish adding the current hit before raising.
- `use_alarm`: If `True` (default), use `signal.SIGALRM` on Unix for
  immediate interruption. On Windows, time is only checked between
  documents.

```python
from whoosh.searching import TimeLimit

uc = collectors.UnlimitedCollector()
tlc = TimeLimitCollector(uc, timelimit=5.8)
try:
    searcher.search_with_collector(myquery, tlc)
except TimeLimit:
    print("Search timed out!")
# Still get partial results:
print(tlc.results())
```

### `TermsCollector`

```python
class whoosh.collectors.TermsCollector(child, settype=set)
```

A collector that records which terms appeared in which matched documents.
Used when `terms=True` in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `settype`: Set type to use for docnum collections (default `set`).

**Attributes:**
- `termdocs`: Dict mapping `(fieldname, text)` tuples to arrays of docnums.
- `docterms`: Dict mapping docnums to lists of `(fieldname, text)` tuples.

## Exceptions

### `TimeLimit`

```python
from whoosh.searching import TimeLimit
```

Raised by `TimeLimitCollector` when the search exceeds the time limit.
Partial results are still available from the collector.


## DOCUMENT: Columns

# Columns API

Classes for storing per-document values (column-oriented storage) used for
fast sorting, faceting, and filtering. Columns are the mechanism by which
Whoosh stores field values alongside the inverted index, in a column-oriented
layout for efficient range access.

The default column type for most fields is `VarBytesColumn`, although numeric
and date fields use `NumericColumn`. Expert users may use other column types
that may be faster or more storage-efficient based on the field contents.

A `Column` object stores configuration information and provides two important
methods: `writer()` to return a `ColumnWriter` and `reader()` to return a
`ColumnReader`.

## Module Functions

### `bytes_column`

```python
whoosh.columns.bytes_column
```

A default `VarBytesColumn` instance used as the column type for string fields.

### `numeric_column`

```python
whoosh.columns.numeric_column
```

A default `NumericColumn` instance used as the column type for numeric fields.

## Base Classes

### `Column`

```python
class whoosh.columns.Column
```

Base class for all column types.

**Class Attributes:**
- `reversible (bool)`: Whether values can be reversed for descending sort.
  Default `False`.

**Methods:**
- `writer(dbfile)`: Returns a `ColumnWriter` for this column type.
- `reader(dbfile, basepos, length, doccount)`: Returns a `ColumnReader` for
  this column type.
- `default_value(reverse=False)`: Returns the default value for documents
  without a column value at index time.
- `stores_lists()`: Returns `True` if the column stores a list of values per
  document instead of a single value.

### `ColumnWriter`

```python
class whoosh.columns.ColumnWriter(dbfile)
```

Base class for writing column values to disk.

**Constructor:**
- `dbfile`: The `StructFile` to write to.

**Methods:**
- `fill(docnum)`: Fills any gap in docnums up to `docnum` with default values.
- `add(docnum, value)`: Adds a value for the given docnum.
- `finish(docnum)`: Called when done writing. Default does nothing.

### `ColumnReader`

```python
class whoosh.columns.ColumnReader(dbfile, basepos, length, doccount)
```

Base class for reading column values from disk.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `basepos`: The offset within the file at which the column starts.
- `length`: The length in bytes the column occupies in the file.
- `doccount`: The number of rows (documents) in the column.

**Methods:**
- `__getitem__(docnum)`: Returns the value for the given docnum.
- `sort_key(docnum)`: Returns the value for sorting (defaults to
  `__getitem__`).
- `__iter__()`: Yields values for all documents.
- `load()`: Returns a list of all values.
- `set_reverse()`: Prepares the reader for reverse iteration.

## Concrete Column Types

### `VarBytesColumn`

```python
class whoosh.columns.VarBytesColumn(
    allow_offsets=True,
    write_offsets_cutoff=2**15
)
```

Stores variable-length byte strings. The default value for documents without
a value is `b''` (empty bytes).

**Constructor:**
- `allow_offsets`: Whether to write offsets for faster lookup when there are
  many rows. Default `True`.
- `write_offsets_cutoff`: Write offsets when there are more than this many
  rows (default `2**15`).

### `FixedBytesColumn`

```python
class whoosh.columns.FixedBytesColumn(blocksize, default=emptybytes)
```

Stores fixed-length byte strings, saving space by not storing the length of
each value.

**Constructor:**
- `blocksize`: Fixed size of each value in bytes.
- `default`: Default value for documents without a value.

### `RefBytesColumn`

```python
class whoosh.columns.RefBytesColumn(
    cachesize=1000,
    stable=True,
    default=emptybytes
)
```

Stores references to unique values rather than the values themselves, saving
space when the field has few unique values. Uses a `DocIdSet` to track which
documents contain each value.

**Constructor:**
- `cachesize`: Size of the LRU cache for value lookups (default `1000`).
- `stable`: Whether to use a stable sort of references (default `True`).
- `default`: Default value for missing documents.

### `NumericColumn`

```python
class whoosh.columns.NumericColumn(
    typecode,
    default=None,
    nullable=False
)
```

Stores numbers (int, float, datetime) encoded as binary values. Extends
`FixedBytesColumn`.

**Constructor:**
- `typecode`: A `struct` typecode string (e.g., `"I"` for unsigned int,
  `"q"` for long, `"d"` for float).
- `default`: Default numeric value (None for the type's zero value).
- `nullable`: Whether `None` values are allowed.

### `BitColumn`

```python
class whoosh.columns.BitColumn
```

Stores boolean values as a bitmap. Each value is either `True` (1) or
`False` (0). Uses a `BitSet` internally.

### `CompressedBytesColumn`

```python
class whoosh.columns.CompressedBytesColumn(default=emptybytes)
```

Wraps a `VarBytesColumn` with zlib compression for the value bytes.

### `CompressedBlockColumn`

```python
class whoosh.columns.CompressedBlockColumn
```

Stores values with block-level zlib compression. More efficient for large
columns.

### `StructColumn`

```python
class whoosh.columns.StructColumn(struct, name)
```

Wraps a `FixedBytesColumn` to store structured binary data (e.g., tuples
encoded with `struct`).

**Constructor:**
- `struct`: A `struct.Struct` object defining the format.
- `name`: Field name for error messages.

### `EmptyColumnReader`

```python
class whoosh.columns.EmptyColumnReader(default, doccount)
```

A `ColumnReader` that returns a constant default value for every document.
Used when a field has no column.

### `MultiColumnReader`

```python
class whoosh.columns.MultiColumnReader(readers)
```

Combines multiple `ColumnReader` instances into one for multi-segment indices.

**Constructor:**
- `readers`: List of `ColumnReader` instances (one per segment).

### `TranslatingColumnReader`

```python
class whoosh.columns.TranslatingColumnReader(child, translator)
```

Wraps a `ColumnReader` to apply a translation function to the values.

**Constructor:**
- `child`: The underlying `ColumnReader`.
- `translator`: Function that maps sort keys to human-readable values.

### `WrappedColumn`

```python
class whoosh.columns.WrappedColumn(child)
```

Base class for column wrappers that adapt another column type.

### `WrappedColumnWriter`

```python
class whoosh.columns.WrappedColumnWriter(child)
```

Base class for column writer wrappers.

### `WrappedColumnReader`

```python
class whoosh.columns.WrappedColumnReader(child)
```

Base class for column reader wrappers.

### `ClampedNumericColumn`

```python
class whoosh.columns.ClampedNumericColumn(child, clampfn)
```

Wraps a `NumericColumn` to clamp values to a valid range before sorting.

**Constructor:**
- `child`: The wrapped `NumericColumn`.
- `clampfn`: Function that clamps a value to the valid range.

### `PickleColumn`

```python
class whoosh.columns.PickleColumn(child, ...)
```

Wraps another column to store pickled Python objects.

### `ListColumn`

```python
class whoosh.columns.ListColumn(child)
```

Base class for columns that store multiple values per document.

### `ListColumnReader`

```python
class whoosh.columns.ListColumnReader(child)
```

Reader for list-valued columns.

### `VarBytesListColumn`

```python
class whoosh.columns.VarBytesListColumn
```

A `ListColumn` variant of `VarBytesColumn` that stores lists of byte strings.

### `FixedBytesListColumn`

```python
class whoosh.columns.FixedBytesListColumn(blocksize)
```

A `ListColumn` variant of `FixedBytesColumn` that stores lists of fixed-size
byte strings.


## DOCUMENT: Core

# Core API

The core module provides the main `Index` class and related functions for managing indexes.

## Functions

### create_in

```python
whoosh.index.create_in(
    dirname: str,
    schema: Schema,
    indexname: str = "MAIN",
    create: bool = True,
    **kwargs
) -> FileIndex
```

Create a new index in the given directory.

**Args:**
- `dirname (str)`: Path to the directory where the index will be stored.
- `schema (Schema)`: The `Schema` object defining the index fields.
- `indexname (str)`: Name of the index. Allows multiple indexes in the same directory.
- `create (bool)`: If True, create the index even if it already exists (clears existing).

**Returns:**
- `FileIndex`: A new index object.

**Example:**
```python
from whoosh.index import create_in
from whoosh.fields import Schema, TEXT

schema = Schema(title=TEXT(stored=True), content=TEXT)
index = create_in("indexdir", schema)
```

---

### open_dir

```python
whoosh.index.open_dir(
    dirname: str,
    indexname: str = "MAIN",
    readonly: bool = False,
    **kwargs
) -> FileIndex
```

Open an existing index.

**Args:**
- `dirname (str)`: Path to the index directory.
- `indexname (str)`: Name of the index to open.
- `readonly (bool)`: If True, open in read-only mode.

**Returns:**
- `FileIndex`: An index object.

**Example:**
```python
from whoosh.index import open_dir
index = open_dir("indexdir")
```

---

### exists_in

```python
whoosh.index.exists_in(
    dirname: str,
    indexname: str = "MAIN",
    **kwargs
) -> bool
```

Check if a valid index exists in the given directory.

**Returns:**
- `bool`: True if the index exists.

---

### create_index

```python
whoosh.index.create_index(
    schema: Schema,
    storage: Storage,
    indexname: str = "MAIN",
    create: bool = True,
    **kwargs
) -> Index
```

Low-level index creation. Use `create_in` instead unless you need custom storage.

---

### open_index

```python
whoosh.index.open_index(
    storage: Storage,
    indexname: str = "MAIN",
    readonly: bool = False,
    **kwargs
) -> Index
```

Low-level index opening. Use `open_dir` instead unless you need custom storage.

## Classes

### Index (Base Class)

```python
class whoosh.index.Index
```

Abstract base class for index objects. Provides common methods for reading and writing.

**Methods:**

#### `writer()`

```python
writer = ix.writer(
    timeout: float = 0.0,
    delay: float = 0.1,
    limitmb: int = 128,
    **kwargs
) -> IndexWriter
```

Return a writer for this index.

**Args:**
- `timeout (float)`: Max seconds to wait for write lock.
- `delay (float)`: Seconds between lock retries.
- `limitmb (int)`: Maximum size of posting pool runs.

**Returns:**
- `IndexWriter`: A writer object.

**Example:**
```python
writer = ix.writer()
writer.add_document(title="Hello", content="World")
writer.commit()
```

---

#### `searcher()`

```python
searcher = ix.searcher(
    weighting: WeightingModel = None,
    **kwargs
) -> Searcher
```

Return a searcher for the current index state.

**Returns:**
- `Searcher`: A searcher object.

**Example:**
```python
with ix.searcher() as searcher:
    results = searcher.search("query")
```

---

#### `reader()`

```python
reader = ix.reader() -> IndexReader
```

Return a reader for the current index state.

---

#### `commit()`

```python
ix.commit(mergetype=None, optimize=None, merge=None)
```

Convenience method: create a writer, call commit, and close.

---

#### `optimize()`

```python
ix.optimize()
```

Merge all segments into a single segment.

---

#### `add_field()`

```python
ix.add_field(fieldname: str, fieldtype, **kwargs)
```

Add a field to the index schema.

---

#### `remove_field()`

```python
ix.remove_field(fieldname: str, **kwargs)
```

Remove a field from the index schema.

---

#### `doc_count()`

```python
count = ix.doc_count() -> int
```

Return the number of documents in the index.

---

#### `doc_count_all()`

```python
count = ix.doc_count_all() -> int
```

Return the total number of documents (including deleted).

---

#### `lock()`

```python
lock = ix.lock(name: str) -> Lock
```

Acquire a named lock on the index.

## FileIndex

The concrete implementation returned by `create_in` and `open_dir`.

All `Index` methods are available. Additional methods:

### `_read_toc()`

Read the table of contents.

### `_write_toc()`

Write the table of contents.

## Exceptions

### LockError

Raised when the index is locked by another writer.

```python
from whoosh.index import LockError

try:
    writer = ix.writer(timeout=5.0)
except LockError:
    print("Index is locked, try again later")
```

### IndexMissingError

Raised when trying to open a non-existent index.

## Constants

### IndexVersion

Current index format version.

---

# Index API

## Index

```python
class whoosh.index.Index
```

Base index class providing reading and writing access.

### Methods

- `writer(**kwargs)` -> `IndexWriter`
- `searcher(**kwargs)` -> `Searcher`
- `reader()` -> `IndexReader`
- `commit(mergetype=None, optimize=None, merge=None)`
- `optimize()`
- `add_field(fieldname, fieldtype, **kwargs)`
- `remove_field(fieldname, **kwargs)`
- `doc_count() -> int`
- `doc_count_all() -> int`
- `lock(name) -> Lock`

## IndexingError

```python
class whoosh.writing.IndexingError(Exception)
```

Raised when an indexing operation fails.

## Exceptions

```python
class whoosh.index.LockError(Exception)
class whoosh.index.IndexMissingError(Exception)
```


## DOCUMENT: Events

# Event Bus & Hooks API

Loose coupling through events and lightweight hooks.

## Event Bus

```python
class whoosh.event_bus.EventBus
```

Publish/subscribe event system supporting both synchronous and asynchronous
listeners. The module-level singleton `event_bus` is available for use:

```python
from whoosh.event_bus import event_bus
```

### Methods

#### `subscribe()`

```python
@event_bus.subscribe(DocumentIndexed)
async def handler(event: DocumentIndexed):
    print(f"Indexed document: {event.document_id}")

# Or without decorator
event_bus.subscribe(DocumentIndexed)(handler)
```

Register a handler function (synchronous or async) for a specific event type.
The decorator takes the event class as an argument. Returns the handler
unchanged so it can be used normally.

---

#### `publish()`

```python
from whoosh.event_bus import event_bus, DocumentIndexed

event_bus.publish(DocumentIndexed(document_id="doc123"))
```

Publish an event to all subscribers. If listeners are async coroutines and no
event loop is running, they are executed via `asyncio.run()`. If an event
loop is running, tasks are scheduled on it. Exceptions in listeners are
swallowed.

---

#### `clear()`

```python
event_bus.clear()
```

Remove all subscribers.

---

## Events

Events are immutable dataclasses.

### `Event`

```python
@dataclass(frozen=True)
class Event:
    pass
```

Base class for all events.

---

### `DocumentIndexed`

```python
@dataclass(frozen=True)
class DocumentIndexed(Event):
    document_id: str
```

Published when a document is indexed. Contains the document ID.

---

### `SearchExecuted`

```python
@dataclass(frozen=True)
class SearchExecuted(Event):
    query: str
```

Published after a search is executed. Contains the query string.

---

## Hooks

Hook system for cross-cutting concerns. Hooks are registered globally using a
module-level registry.

### `hookimpl`

```python
from whoosh.hooks import hookimpl

@hookimpl
def before_search(context):
    context.query = optimize_query(context.query)
    return context
```

Decorator that marks a function as a hook implementation. Returns a `HookImpl`
wrapper.

### `register_hook()`

```python
from whoosh.hooks import register_hook, hookimpl

@hookimpl
def before_search(context):
    ...

register_hook("before_search", before_search)
```

Register a `HookImpl` under a named hook. Multiple hooks can be registered
per name; they are called in registration order.

### `call_hook()`

```python
from whoosh.hooks import call_hook

results = await call_hook("before_search", context)
```

Async function that calls all hooks registered under the given name. Returns
a list of results from each hook's execution. Exceptions in individual hooks
are logged but do not stop execution.

---
## Example: Event Bus

```python
from whoosh.event_bus import event_bus, DocumentIndexed, SearchExecuted

@event_bus.subscribe(DocumentIndexed)
async def on_document_indexed(event: DocumentIndexed):
    print(f"Document indexed: {event.document_id}")

@event_bus.subscribe(SearchExecuted)
async def on_search_executed(event: SearchExecuted):
    print(f"Search executed: {event.query}")

# Publish events
event_bus.publish(DocumentIndexed(document_id="doc123"))
event_bus.publish(SearchExecuted(query="hello world"))
```

## Example: Hooks

```python
from whoosh.hooks import hookimpl, register_hook, call_hook
import asyncio

@hookimpl
def before_search(context):
    print(f"Searching for: {context['query']}")
    return context

register_hook("before_search", before_search)

# Call hooks
context = {"query": "hello"}
results = asyncio.run(call_hook("before_search", context))
```


## DOCUMENT: Fields

# Fields API

Define the structure of your index with field types.

## Schema

```python
class whoosh.fields.Schema
```

The `Schema` class defines the fields available in an index.

### Constructor

```python
schema = Schema(
    title=TEXT(stored=True),
    content=TEXT,
    path=ID(stored=True, unique=True),
    tags=KEYWORD(lowercase=True),
    rating=NUMERIC(float, stored=True),
    published=DATETIME(stored=True),
    active=BOOLEAN
)
```

### Methods

#### `add()`

```python
schema.add(
    fieldname: str,
    fieldtype,
    glob: bool = False,
    **kwargs
)
```

Add a field to the schema. If `glob=True`, the fieldname is treated as a glob pattern.

#### `remove()`

```python
schema.remove(fieldname: str, **kwargs)
```

Remove a field from the schema.

#### `items()`

```python
for name, field in schema.items():
    print(name, field)
```

Return a list of (fieldname, field object) pairs.

#### `names()`

```python
names = schema.names()
```

Return a list of field names.

## FieldType Base Class

```python
class whoosh.fields.FieldType
```

Base class for all field types.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `format` | `Format` | Defines how the field is indexed |
| `vector` | `Format` or None | Optional per-document vector format |
| `scorable` | `bool` | Whether field length is stored (for BM25F) |
| `stored` | `bool` | Whether field value is stored in index |
| `unique` | `bool` | Whether field uniquely identifies documents |

### Methods

#### `index()`

Convert a value into indexed items.

#### `indexable()`

Check if the value can be indexed.

#### `spelling_fieldname()`

Return the field name used for spelling data.

#### `spellable_words()`

Generate spellable words from a value.

## Built-in Field Types

### TEXT

```python
whoosh.fields.TEXT(
    stored: bool = False,
    unique: bool = False,
    phrase: bool = True,
    analyzer: Analyzer = None,
    field_boost: float = 1.0,
    **kwargs
)
```

Full-text field with tokenization and optional phrase search.

**Example:**
```python
title = TEXT(stored=True)
body = TEXT(analyzer=StemmingAnalyzer(), phrase=False)
```

---

### ID

```python
whoosh.fields.ID(
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Untokenized identifier field. Stores the entire value as a single term.

**Example:**
```python
path = ID(stored=True, unique=True)
slug = ID(stored=True)
```

---

### KEYWORD

```python
whoosh.fields.KEYWORD(
    stored: bool = False,
    lowercase: bool = False,
    commas: bool = False,
    scorable: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Space or comma-separated keywords. Phrase search is not supported.

**Example:**
```python
tags = KEYWORD(lowercase=True, commas=True, stored=True)
```

---

### STORED

```python
whoosh.fields.STORED(
    stored: bool = True,
    unique: bool = False,
    **kwargs
)
```

Stored-only field. Not indexed or searchable.

**Example:**
```python
icon = STORED()
description = STORED()
```

---

### NUMERIC

```python
whoosh.fields.NUMERIC(
    numtype: type = int,
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Numeric field for integers or floats.

**Example:**
```python
rating = NUMERIC(float, stored=True)
count = NUMERIC(int)
price = NUMERIC(float, stored=True, sortable=True)
```

---

### DATETIME

```python
whoosh.fields.DATETIME(
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Date/time field. Stores `datetime` objects.

**Example:**
```python
published = DATETIME(stored=True)
updated = DATETIME()
```

---

### BOOLEAN

```python
whoosh.fields.BOOLEAN(
    stored: bool = False,
    unique: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Boolean field. Searchable with `yes`, `no`, `true`, `false`, `1`, `0`, `t`, `f`.

**Example:**
```python
published = BOOLEAN(stored=True)
```

---

### NGRAM

```python
whoosh.fields.NGRAM(
    minsize: int = 2,
    maxsize: int = 5,
    stored: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Character n-gram field.

---

### NGRAMWORDS

```python
whoosh.fields.NGRAMWORDS(
    minsize: int = 2,
    maxsize: int = 5,
    stored: bool = False,
    field_boost: float = 1.0,
    **kwargs
)
```

Word-level n-gram field.

---

### VectorField

```python
whoosh.fields.VectorField(
    dimensions: int,
    metric: str = "cosine",
    provider: str = "numpy",
    stored: bool = False,
    **kwargs
)
```

Field for storing and searching vector embeddings.

**Args:**
- `dimensions (int)`: Embedding dimension (e.g., 384 for all-MiniLM-L6-v2).
- `metric (str)`: Similarity metric: `"cosine"`, `"euclidean"`, `"dot"`.
- `provider (str)`: Vector provider name from registry.

**Example:**
```python
embedding = VectorField(dimensions=384, metric="cosine", stored=True)
```

## SchemaBuilder

Fluent API for building schemas:

```python
from whoosh.fields import SchemaBuilder

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("path", ID(stored=True, unique=True))
    .field("content", TEXT)
    .field("tags", KEYWORD(lowercase=True))
    .field("published", DATETIME(stored=True))
    .build()
)
```

## Constants

- `whoosh.fields.STORED`: Stored-only field type
- `whoosh.fields.TEXT`: Full-text field
- `whoosh.fields.ID`: Identifier field
- `whoosh.fields.KEYWORD`: Keyword field
- `whoosh.fields.NUMERIC`: Numeric field
- `whoosh.fields.DATETIME`: Date/time field
- `whoosh.fields.BOOLEAN`: Boolean field


## DOCUMENT: Filedb Storage

# File DB / Storage API

Classes for storing and retrieving index data on disk or in memory. The
`Storage` class is the main entry point for persisting an index.

## Storage Classes

### `Storage`

```python
class whoosh.filedb.filestore.Storage(path=None)
```

Abstract base class for storage backends. A `Storage` manages a filesystem-
or memory-based location where index files can be created, read, and
manipulated.

**Constructor:**
- `path`: Optional path string. Subclasses may use this to set the storage
  location.

**Methods:**

#### `create_file(name, **kwargs)`

Creates and returns a file object for writing.

#### `open_file(name, **kwargs)`

Opens and returns a file object for reading.

#### `list()`

Returns a list of all filenames in this storage.

#### `exists(name)`

Returns `True` if a file/named item exists in the storage.

#### `file_exists(name)`

Alias for `exists()`.

#### `file_length(name)`

Returns the length of file `name` in bytes.

#### `rename(src, dst)`

Renames a file from `src` to `dst`.

#### `delete_file(name)`

Deletes file `name` from storage.

#### `destroy()`

Deletes all files and the storage itself.

#### `temp_storage()`

Creates and returns a temporary isolated `Storage` for scratch space.

#### `supports_mmap`

Returns `True` if this storage supports memory-mapped file access.

**Properties:**
- `schema`: The `Schema` for this storage (if it holds an index).
- `lock`: The lock object used for this storage.

### `FileStorage`

```python
class whoosh.filedb.filestore.FileStorage(
    path,
    cachesize_limit=40,
    supports_mmap=None,
    **kwargs
)
```

A `Storage` subclass that uses the operating system's filesystem.

**Constructor:**
- `path`: A `Path` (or string path) to the directory where files are stored.
- `cachesize_limit`: Maximum number of open file handles to cache.
- `supports_mmap`: If `None`, auto-detected; otherwise force enable/disable.

**Methods:** All `Storage` methods plus:
- `create_index(schema, indexname="index", ...)`: Creates and returns a new
  `Index` object.
- `open_index(indexname="index", ...)`: Opens an existing `Index`.
- `lock(name)`: Returns a lock object for the given lock name.

### `RamStorage`

```python
class whoosh.filedb.filestore.RamStorage(cachesize_limit=10)
```

A `Storage` subclass that keeps all files in memory as bytes. Useful for
testing and small indexes.

**Constructor:**
- `cachesize_limit`: Maximum number of files to cache as decoded objects.

**Methods:** All `Storage` methods plus:
- `create_index(schema, ...)`: Creates an in-memory `Index`.
- `save_to_file(filename, ...)`: Saves the entire storage to a file.
- `load_from_file(filename, ...)`: Loads storage contents from a file.

### `OverlayStorage`

```python
class whoosh.filedb.filestore.OverlayStorage(base, overlay)
```

A `Storage` wrapper that presents two storage layers: a base and an overlay.
Files in the overlay take precedence over the base.

**Constructor:**
- `base`: The base `Storage` (e.g., read-only original).
- `overlay`: The overlay `Storage` (e.g., writable copy).

## Storage Exceptions

### `StorageError`

```python
class whoosh.filedb.filestore.StorageError
```

Base exception for storage-related errors.

### `ReadOnlyError`

```python
class whoosh.filedb.filestore.ReadOnlyError(StorageError)
```

Raised when attempting to write to a read-only storage.

## File Tables

### `HashWriter`

```python
class whoosh.filedb.filetables.HashWriter(dbfile, keycoder=None, keydecoder=None, data_encoder=None, data_decoder=None, **kwargs)
```

Writes key-value pairs to a file, with optional indexing by key.

**Constructor:**
- `dbfile`: The `StructFile` to write to.
- `keycoder`: Function to encode keys for storage.
- `keydecoder`: Function to decode keys from storage.
- `data_encoder`: Function to encode values.
- `data_decoder`: Function to decode values.

### `HashReader`

```python
class whoosh.filedb.filetables.HashReader(dbfile, length, keycoder=None, keydecoder=None, data_decoder=None, **kwargs)
```

Reads key-value pairs from a file written by `HashWriter`.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `length`: Length of the data section.
- `keycoder`/`keydecoder`/`data_decoder`: Same as `HashWriter`.

**Methods:**
- `__getitem__(key)`: Returns the value for `key`.
- `keys()`: Yields all keys.
- `values()`: Yields all values.
- `items()`: Yields `(key, value)` pairs.
- `keys_from(prefixbytes)`: Yields keys starting at `prefixbytes`.
- `items_from(prefixbytes)`: Yields `(key, value)` pairs starting at prefix.
- `closest_key_pos(key)`: Returns the position of the closest matching key.
- `range_for_key(key)`: Returns `(startpos, endpos)` for a key range.

### `OrderedHashWriter`

```python
class whoosh.filedb.filetables.OrderedHashWriter(HashWriter)
```

A `HashWriter` that maintains keys in sorted order.

### `OrderedHashReader`

```python
class whoosh.filedb.filetables.OrderedHashReader(HashReader)`

A `HashReader` for reading data written by `OrderedHashWriter`. Preserves
key ordering for efficient prefix iteration.

### `FieldedOrderedHashWriter`

```python
class whoosh.filedb.filetables.FieldedOrderedHashWriter(HashWriter)
```

An `OrderedHashWriter` that stores an extra "fieldmap" in the extras dict,
mapping field names to numeric IDs.

### `FieldedOrderedHashReader`

```python
class whoosh.filedb.filetables.FieldedOrderedHashReader(HashReader)
```

Reader for data written by `FieldedOrderedHashWriter`.

## Struct File

### `StructFile`

```python
class whoosh.filedb.structfile.StructFile(name, source, cachesize_limit=40)
```

Wraps a file object and adds methods for reading/writing packed binary
values, arrays, varints, and pickle objects.

**Methods include:**
- `read_int()`, `write_int(n)`: Read/write a 4-byte signed integer.
- `read_long()`, `write_long(n)`: Read/write a 8-byte signed integer.
- `read_uint()`, `write_uint(n)`: Read/write unsigned int.
- `read_ulong()`, `write_ulong(n)`: Read/write unsigned long.
- `read_float()`, `write_float(n)`: Read/write a float.
- `read_ushort()`, `write_ushort(n)`: Read/write unsigned short.
- `read_byte()`, `write_byte(b)`: Read/write a single byte.
- `write_array(arr)`: Write an array of values.
- `get_array(offset, typecode, length)`: Read an array from offset.
- `write_pickle(obj)`: Pickle and write an object.
- `read_pickle()`: Read and unpickle an object.
- `get(offset, length)`: Read `length` bytes from `offset`.
- `get_int()`, `get_uint()`, `get_long()`, `get_float()`, `get_byte()`:
  Read a single value from the given offset.

### `BufferFile`

```python
class whoosh.filedb.structfile.BufferFile
```

A `StructFile` that wraps an in-memory byte buffer.

### `ChecksumFile`

```python
class whoosh.filedb.structfile.ChecksumFile(dbfile)
```

A `StructFile` wrapper that computes a checksum as data is written, for
integrity verification.

## Compound Storage

### `CompoundStorage`

```python
class whoosh.filedb.compound.CompoundStorage(dbfile, use_mmap=True)
```

Treats a single file as a container for multiple sub-files. Used for compound
segment files.

**Methods:**
- `create_file(name)`: Create a sub-file within the compound file.
- `open_file(name)`: Open a sub-file for reading.
- `list()`: List all sub-file names.
- `close()`: Close the compound storage.

### `SubFile`

```python
class whoosh.filedb.compound.SubFile
```

A file-like object representing a sub-file within a `CompoundStorage`.

### `CompoundWriter`

```python
class whoosh.filedb.compound.CompoundWriter(storage)
```

Writes a compound file by assembling multiple files from a storage.

**Methods:**
- `create_file(name)`: Reserve a filename in the compound file.
- `save_as_files(dest_storage, fn_generator)`: Assemble the compound file
  from source files into the destination storage.

## Storage Utility Functions

### `copy_storage`

```python
whoosh.filedb.filestore.copy_storage(sourcestore, deststore)
```

Copies all files from one storage to another.

### `copy_to_ram`

```python
whoosh.filedb.filestore.copy_to_ram(storage)
```

Reads all files from a storage into a `RamStorage` and returns it.


## DOCUMENT: Formats

# Formats API

Classes that control how posting information (frequencies, positions,
character offsets, and weights) is encoded and stored for each field in the
index. The `Format` object is a factory and encoder/decoder for the
value strings stored alongside each posting.

## Module Functions

### `tokens`

```python
whoosh.formats.tokens(value, analyzer, kwargs)
```

Takes a text `value` and an `analyzer`, runs the analyzer on the value, and
returns the resulting token generator (wrapped with `unstopped()` to ignore
`STOP` tokens). Used internally by `Format.word_values()`.

## Format Classes

All format classes accept a `field_boost` parameter (default `1.0`) that
scales the score of all queries matching terms in that field.

### `Format`

```python
class whoosh.formats.Format(field_boost=1.0, **options)
```

Abstract base class for all posting formats. Format objects are
field-level objects: one is created per `Field` and shared across all
postings for that field.

**Attributes:**
- `posting_size (int)`: Fixed byte size of encoded postings, or `None`/`-1`
  if variable-size.
- `textual (bool)`: Whether this format expects string tokens (vs. bytes).
  Default `True`.

**Methods:**

#### `word_values(value, analyzer, **kwargs)`

Abstract. Takes a text value, runs it through the analyzer, and yields
`(tokentext, frequency, weight, valuestring)` tuples.

#### `encode(value)`

Abstract. Encodes raw posting data into the value string bytes.

#### `decode_frequency(valuestring)`

Abstract. Decodes the frequency (term count in document) from the value
string.

#### `decode_weight(valuestring)`

Abstract. Decodes the weight (total boost contribution) from the value string.

#### `combine(valuestrings)`

Abstract. Combines multiple value strings (from overlapping segments) into
a single value string.

#### `supports(name)`

Returns `True` if this format supports interpreting its postings as `name`
(e.g., `"frequency"`, `"positions"`, `"characters"`, `"position_boosts"`,
`"character_boosts"`). Equivalent to `hasattr(self, "decode_" + name)`.

#### `decoder(name)`

Returns the `decode_<name>` method for the given attribute name.

#### `decode_as(astype, valuestring)`

Calls the appropriate `decode_<astype>` method on `valuestring` and returns
the result.

#### `fixed_value_size()`

Returns `self.posting_size` if positive, otherwise `None`.

#### `__eq__(other)`

Returns `True` if `other` is the same class with equal `__dict__`.

### `Existence`

```python
class whoosh.formats.Existence(field_boost=1.0, **options)
```

Indexes only whether a term occurred in a document—not its frequency or
positions. Useful for non-scorable fields like paths.

- `posting_size = 0`
- Supports: `frequency` (always 1), `weight` (always `field_boost`)
- `encode()` returns empty bytes

### `Frequency`

```python
class whoosh.formats.Frequency(field_boost=1.0, boost_as_freq=False, **options)
```

Stores term frequency information (term count per document) for each posting.

- `posting_size = _INT_SIZE` (4 bytes)
- Supports: `frequency`, `weight`
- `encode()` encodes the count as a packed unsigned int
- `boost_as_freq`: If `True`, boosts are interpreted as frequency boosts

```python
from whoosh.formats import Frequency
fmt = Frequency(field_boost=1.0)
```

### `Positions`

```python
class whoosh.formats.Positions(field_boost=1.0, **options)
```

Stores position information (term offsets within the document) in each
posting, enabling phrase queries and "near" queries.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode(poslist)` encodes positions using variable-length delta encoding
- Positions are stored as delta-encoded variable-length integers

```python
from whoosh.formats import Positions
fmt = Positions()
```

### `Characters`

```python
class whoosh.formats.Characters(field_boost=1.0, **options)
```

Extends `Positions` to also store character start and end offsets for each
term occurrence, enabling character-precise highlighting.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`
- `encode()` encodes (position, startchar, endchar) triples with delta
  encoding

### `PositionBoosts`

```python
class whoosh.formats.PositionBoosts(field_boost=1.0, **options)
```

Extends `Positions` to store per-position boost values in addition to
positions.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode()` encodes `(position, boost)` pairs

### `CharacterBoosts`

```python
class whoosh.formats.CharacterBoosts(field_boost=1.0, **options)
```

Extends `Characters` to store per-position boost values along with
character offsets.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`, `character_boosts`
- `encode()` encodes `(position, startchar, endchar, boost)` tuples


## DOCUMENT: Highlight

# Highlight API

Classes and functions for highlighting matches in search result fragments.
The highlight module is a refactored package exposing the same public API as
the former monolithic module.

## Overview

The highlighting system has four components:

- **Fragmenters** split text into fragments.
- **Fragment Scorers** score fragments to determine which to display.
- **Formatters** render fragments as output (HTML, plain text, etc.).
- **Highlighter** ties these together and is used by `Searcher.highlights()`.

## Module-level Functions

### `highlight`

```python
whoosh.highlight.highlight(
    text: str,
    terms: list[str],
    analyzer,
    fragmenter,
    formatter,
    top: int = 3,
    scorer=None,
    minscore: int = 1,
    order=SCORE,
    mode: str = "query"
) -> str
```

Highlights the matched terms in `text` and returns a formatted string.

- `text`: The text to highlight.
- `terms`: A list of matched terms (strings).
- `analyzer`: The analyzer for the field.
- `fragmenter`: A `Fragmenter` instance or class.
- `formatter`: A `Formatter` instance or class.
- `top`: Maximum number of fragments to return.
- `scorer`: Optional fragment scorer (defaults to `BasicFragmentScorer`).
- `minscore`: Minimum score for a fragment to be included.
- `order`: Sort order for fragments (`FIRST`, `SCORE`, `LONGER`, `SHORTER`).
- `mode`: Analysis mode, typically `"query"` or `"index"`.

### `mkfrag`

```python
whoosh.highlight.mkfrag(
    text: str,
    tokens,
    startchar=None,
    endchar=None,
    charsbefore: int = 0,
    charsafter: int = 0
) -> Fragment
```

Returns a `Fragment` object based on `Token` objects in `tokens`.

### `get_text`

```python
whoosh.highlight.get_text(
    original: str,
    token,
    replace: bool
) -> str
```

Returns the text to use for a match when formatting. If `replace` is `False`,
returns the original text between `token.startchar` and `token.endchar`. If
`True`, returns `token.text`.

### `set_matched_filter`

```python
whoosh.highlight.set_matched_filter(
    tokens,
    termset: frozenset
) -> Iterator[Token]
```

Marks tokens as matched if their `text` attribute is in `termset`. Used for
phrase-agnostic highlighting.

### `set_matched_filter_phrases`

```python
whoosh.highlight.set_matched_filter_phrases(
    tokens,
    text: str,
    terms,
    phrases
) -> Iterator[Token]
```

Marks tokens as matched using phrase-aware logic. Highlights only tokens that
are part of matched phrases.

### `top_fragments`

```python
whoosh.highlight.top_fragments(
    fragments,
    count: int,
    scorer,
    order,
    minscore: int = 1
) -> list[Fragment]
```

Returns the best `count` fragments sorted by `order`, filtered by `minscore`.

## Constants

### `DEFAULT_CHARLIMIT`

```python
whoosh.highlight.DEFAULT_CHARLIMIT = 2**15
```

Default character limit for fragments.

### Sort Order Constants

```python
whoosh.highlight.FIRST   # Sort passages from earlier in the document first
whoosh.highlight.SCORE   # Sort higher scored passages first
whoosh.highlight.LONGER  # Sort longer passages first
whoosh.highlight.SHORTER # Sort shorter passages first
```

## Formatters

### `Formatter`

```python
class whoosh.highlight.Formatter
```

Base class for formatters. Subclasses implement `format_token()` to define
how matched tokens are rendered.

**Methods:**

- `format_token(text, token, replace=False)`: Returns formatted text for a
  matched token.
- `format_fragment(fragment, replace=False)`: Returns formatted text for a
  `Fragment`.
- `format(fragments, replace=False)`: Returns formatted text for a list of
  fragments, joined by `between`.

**Attributes:**
- `between`: String inserted between formatted fragments (default `"..."`).

### `NullFormatter`

```python
class whoosh.highlight.NullFormatter(Formatter)
```

A formatter that does not modify the string. Returns fragments unformatted.

### `UppercaseFormatter`

```python
class whoosh.highlight.UppercaseFormatter(between="...")
```

Formats matched terms in uppercase.

### `HtmlFormatter`

```python
class whoosh.highlight.HtmlFormatter(
    tagname="strong",
    between="...",
    classname="match",
    termclass="term",
    maxclasses=5,
    attrquote='"'
)
```

Wraps matched terms in HTML tags with CSS class names. Two classes are
applied to each match: `classname` (same for all matches) and `termclass`
(different for each term, e.g. `term0`, `term1`).

- `tagname`: The HTML tag to wrap matches (default `"strong"`).
- `between`: Text inserted between fragments.
- `classname`: CSS class applied to all matched term tags.
- `termclass`: CSS class prefix for per-term classes.
- `maxclasses`: Maximum number of distinct per-term class numbers.
- `attrquote`: Quote character for attribute values.

**Methods:**
- `clean()`: Clears the internal term-to-classname mapping dictionary.

### `GenshiFormatter`

```python
class whoosh.highlight.GenshiFormatter(qname="strong", between="...")
```

Formats matched terms as Genshi event streams (requires the Genshi library).

## Fragmenters

### `Fragmenter`

```python
class whoosh.highlight.Fragmenter
```

Base class for fragmenters. Subclasses implement `fragment_tokens()` and/or
`fragment_matches()`.

**Methods:**
- `must_retokenize()`: Returns `True` if this fragmenter needs to re-tokenize
  the text (calls `fragment_tokens` with all tokens). Returns `False` if it can
  work from matched token positions alone (calls `fragment_matches`).

### `WholeFragmenter`

```python
class whoosh.highlight.WholeFragmenter(charlimit=DEFAULT_CHARLIMIT)
```

Does not fragment text. Returns the entire text as one fragment. Useful for
highlighting short fields.

```python
results.fragmenter = WholeFragmenter()
```

### `SentenceFragmenter`

```python
class whoosh.highlight.SentenceFragmenter(
    maxchars: int = 200,
    sentencechars=".!?",
    charlimit=DEFAULT_CHARLIMIT
)
```

Breaks text at sentence-ending punctuation (`.`, `!`, `?`).

- `maxchars`: Maximum characters per fragment.
- `sentencechars`: Characters that indicate sentence boundaries.
- `charlimit`: Maximum character position to process.

**Note:** Should be used with an analyzer that does not remove stop words.

### `ContextFragmenter`

```python
class whoosh.highlight.ContextFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    charlimit=DEFAULT_CHARLIMIT
)
```

The default fragmenter. Finds matched terms and includes `surround` characters
of context before and after each match.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters to include around matches.
- `charlimit`: Maximum character position to process.

### `PinpointFragmenter`

```python
class whoosh.highlight.PinpointFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    autotrim: bool = False,
    charlimit=DEFAULT_CHARLIMIT
)
```

A non-retokenizing fragmenter that builds fragments from character positions of
matched terms. Faster than `ContextFragmenter` because it doesn't need to
re-tokenize text.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters around matches.
- `autotrim`: If `True`, trims fragments to the nearest spaces.
- `charlimit`: Maximum character position to process.

### `NullFragmeter`

Alias for `WholeFragmenter`.

### `Fragment`

```python
class whoosh.highlight.Fragment(
    text: str,
    matches,
    startchar: int = 0,
    endchar: int = -1
)
```

Represents a fragment (excerpt) from a hit document. Stores the start and end
character offsets and the list of matched term objects.

**Attributes:**
- `text`: The original source text.
- `matches`: List of objects with `startchar` and `endchar` attributes.
- `startchar`: Start index of the fragment.
- `endchar`: End index of the fragment.
- `matched_terms`: Set of text values of matched terms.

**Methods:**
- `overlaps(fragment)`: Returns `True` if this fragment overlaps the given one.
- `overlapped_length(fragment)`: Returns the combined length of overlapping
  fragments.

### `FragmentScorer`

```python
class whoosh.highlight.FragmentScorer
```

Base class for fragment scoring objects. Subclasses implement `__call__()`
to score a `Fragment`.

### `BasicFragmentScorer`

```python
class whoosh.highlight.BasicFragmentScorer
```

Scores fragments by summing the boosts of matched terms, then multiplying by
the number of distinct matched terms (favors diversity).

## Highlighter

### `Highlighter`

```python
class whoosh.highlight.Highlighter(
    fragmenter=None,
    scorer=None,
    formatter=None,
    always_retokenize: bool = False,
    order=SCORE
)
```

Main highlighter object used by `Searcher.highlights()`.

- `fragmenter`: Fragmenter instance (defaults to `ContextFragmenter`).
- `scorer`: Fragment scorer (defaults to `BasicFragmentScorer`).
- `formatter`: Formatter instance (defaults to `HtmlFormatter(tagname="b")`).
- `always_retokenize`: If `True`, always re-tokenize text instead of using
  character offsets from postings.
- `order`: Sort order for fragments.

**Methods:**
- `highlight_hit(hitobj, fieldname, top=3, minscore=1, strict_phrase=False)`:
  Returns the highlighted string for a single hit in a given field.
- `can_load_chars(results, fieldname)`: Returns `True` if the field supports
  "pinpoint" highlighting using stored character offsets.


## DOCUMENT: Idsets

# Idsets API

Specialized set implementations for storing sorted lists of positive
integers (document IDs). These are more memory-efficient than the built-in
`set` for certain use cases, though they are slower for most operations since
they are pure Python.

## Overview

The `DocIdSet` class is the abstract base class. Concrete implementations
include `BitSet`, `OnDiskBitSet`, `SortedIntSet`, `RoaringIdSet`, and
`MultiIdSet`. The `AutoIdSet` function selects the best implementation
based on the contents.

## Module Functions

### `autoset`

```python
whoosh.idsets.autoset
```

A factory that creates an appropriate `DocIdSet` subclass based on the
contents of a given iterable. If all integers in the set are below 10,000,
returns a `BitSet`; otherwise returns a `SortedIntSet`.

## `DocIdSet`

```python
class whoosh.idsets.DocIdSet
```

Abstract base class for set implementations specialized toward storing sorted
lists of positive integers.

**Inheritance:** Inherits from `set`-like interface.

**Methods:**
- `__eq__(other)`: Compares two `DocIdSet` instances by iterating.
- `__len__()`: Returns the number of elements. Override in subclasses.
- `__iter__()`: Yields elements in sorted order. Override in subclasses.
- `__contains__(i)`: Returns `True` if `i` is in the set.
- `__or__(other)`: Returns `self.union(other)`.
- `__and__(other)`: Returns `self.intersection(other)`.
- `__sub__(other)`: Returns `self.difference(other)`.
- `copy()`: Returns a copy of this set.
- `add(n)`: Adds `n` to the set.
- `discard(n)`: Removes `n` from the set (no error if absent).
- `update(other)`: Adds all elements from `other`.
- `intersection_update(other)`: Removes elements not in `other`.
- `difference_update(other)`: Removes all elements in `other`.
- `invert_update(size)`: In-place inversion over the range `[0, size)`.
- `intersection(other)`: Returns a new set with elements in both.
- `union(other)`: Returns a new set with elements from both.
- `difference(other)`: Returns a new set with elements in self but not other.
- `invert(size)`: Returns a new set that is the inversion over `[0, size)`.
- `isdisjoint(other)`: Returns `True` if no elements are shared.
- `before(i)`: Returns the previous integer in the set before `i`, or `None`.
- `after(i)`: Returns the next integer in the set after `i`, or `None`.
- `first()`: Returns the first (lowest) integer.
- `last()`: Returns the last (highest) integer.

## `BaseBitSet`

```python
class whoosh.idsets.BaseBitSet(DocIdSet)
```

Base class for bitmap-backed `DocIdSet` implementations. Uses a bytes-based
bitmap where each bit represents membership of an integer.

**Abstract Methods to Override:**
- `byte_count()`: Returns the number of bytes in the bitmap.
- `_get_byte(i)`: Returns the byte at index `i`.
- `_iter_bytes()`: Yields all bytes in the bitmap.

**Inherited Methods:** All `DocIdSet` methods with efficient bitmap
implementations of `__len__`, `__iter__`, `__contains__`, `first`, and
`last`.

## `OnDiskBitSet`

```python
class whoosh.idsets.OnDiskBitSet(file, doc_count)
```

A `BaseBitSet` that reads the bitmap from a file on disk, using `mmap` for
memory efficiency.

**Constructor:**
- `file`: A file-like object (opened in binary mode) containing the bitmap.
- `doc_count`: Total number of documents (bits) represented.

```python
from whoosh.idsets import OnDiskBitSet

with open("deletions.dat", "rb") as f:
    bs = OnDiskBitSet(f, doc_count=10000)
    if 42 in bs:
        print("Document 42 is deleted")
```

## `BitSet`

```python
class whoosh.idsets.BitSet
```

A `BaseBitSet` that stores the bitmap in memory as a `bytearray`. Fast for
membership tests and set operations on small ranges of integers.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import BitSet

bs = BitSet([0, 5, 10, 15])
print(5 in bs)  # True
print(bs.first())  # 0
print(len(bs))   # 4
```

**Methods:**
- `from_blob(data)`: Create a `BitSet` from raw bytes.
- `tostring()`: Returns the bitmap as a `bytes` string.
- `set_reverse()`: Prepares the set for reverse iteration.

## `SortedIntSet`

```python
class whoosh.idsets.SortedIntSet
```

A `DocIdSet` that stores integers as a sorted list of Python `int` objects.
More memory-efficient than `BitSet` for sparse sets but slower for membership
tests.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import SortedIntSet

sis = SortedIntSet([100, 500, 999])
print(500 in sis)  # True
print(sis.after(200))  # 500
```

## `ReverseIdSet`

```python
class whoosh.idsets.ReverseIdSet(child)
```

Wraps another `DocIdSet` to reverse the interpretation of integers. Instead
of representing membership directly, the set represents the *complement* of
the inner set. Useful for representing deleted documents.

**Constructor:**
- `child`: The `DocIdSet` to reverse.

**Example:** If `child` represents documents `{3, 7, 9}`, then
`ReverseIdSet(child)` represents all documents *except* `{3, 7, 9}`.

## `RoaringIdSet`

```python
class whoosh.idsets.RoaringIdSet
```

A `DocIdSet` that partitions integers into 16-bit buckets and uses `BitSet`
within each bucket. More memory-efficient than a single flat `BitSet` for
large, sparse sets of integers.

**Constructor:**
- Optional initial iterable of integers.

**Methods:**
- `from_bytes(data)`: Deserialize from bytes.
- `to_bytes()`: Serialize to bytes.
- `to_bytes_list()`: Returns a list of `(bucket, bytes)` pairs.

## `MultiIdSet`

```python
class whoosh.idsets.MultiIdSet(readers, offsets=None)
```

Combines multiple `DocIdSet` instances into one, handling document ID offsets
automatically. Used for combining deletions across multiple segments.

**Constructor:**
- `readers`: List of `DocIdSet` instances (one per segment).
- `offsets`: Optional list of base docnum offsets for each reader. If
  omitted, offsets are computed automatically.

**Methods:**
- `__contains__(i)`: Checks the appropriate sub-set based on offsets.
- `__iter__()`: Iterates over all integers in all sub-sets.
- `__len__()`: Returns the total count across all sub-sets.


## DOCUMENT: Lang

# Language Support API

Language detection helpers, stemmer selection, stop-word lists, and
language-specific modules (Snowball stemmers, ISRI stemmer, Soundex,
Double Metaphone, etc.).

## Module Overview

The `whoosh.lang` package provides functions for detecting and selecting
language-specific resources (stemmers, stop words) and submodules containing
stemmers for various languages.

## Supported Languages

```python
whoosh.lang.languages = ("ar", "da", "nl", "en", "fi", "fr", "de", "hu",
                         "it", "no", "pt", "ro", "ru", "es", "sv", "tr")
```

Two-letter ISO 639-1 language codes for which stemmers or stop-word lists
are available.

## Language Aliases

```python
whoosh.lang.aliases = { ... }
```

A dictionary mapping alternate language identifiers to their canonical
two-letter codes. Includes ISO 639-3 three-letter codes, English names,
and native-language names.

## Exceptions

### `NoStemmer`

```python
class whoosh.lang.NoStemmer
```

Raised by `stemmer_for_language()` when no stemmer is available for the
given language.

### `NoStopWords`

```python
class whoosh.lang.NoStopWords
```

Raised by `stopwords_for_language()` when no stop-word list is available for
the given language.

## Language Functions

### `two_letter_code`

```python
whoosh.lang.two_letter_code(name) -> str or None
```

Converts a language identifier to its canonical two-letter code. Accepts
two-letter codes, ISO 639-3 codes, English names, and native-language names.

```python
from whoosh.lang import two_letter_code

code = two_letter_code("french")   # 'fr'
code = two_letter_code("deutsch")  # 'de'
code = two_letter_code("español")  # 'es'
```

### `has_stemmer`

```python
whoosh.lang.has_stemmer(lang) -> bool
```

Returns `True` if a stemmer is available for the given language.

### `has_stopwords`

```python
whoosh.lang.has_stopwords(lang) -> bool
```

Returns `True` if a stop-word list is available for the given language.

### `stemmer_for_language`

```python
whoosh.lang.stemmer_for_language(lang) -> callable
```

Returns a stemmer function for the given language. Raises `NoStemmer` if
no stemmer is available.

**Supported languages and stemmers:**
- `"en"` / `"en_porter"`: Original Porter stemmer (`whoosh.lang.porter`)
- `"ar"`: ISRI Arabic stemmer (`whoosh.lang.isri`)
- `"da"`: Danish Snowball stemmer
- `"nl"`: Dutch Snowball stemmer
- `"en"`: English Snowball stemmer
- `"fi"`: Finnish Snowball stemmer
- `"fr"`: French Snowball stemmer
- `"de"`: German Snowball stemmer
- `"hu"`: Hungarian Snowball stemmer
- `"it"`: Italian Snowball stemmer
- `"no"`: Norwegian Snowball stemmer
- `"pt"`: Portuguese Snowball stemmer
- `"ro"`: (no stemmer currently)
- `"ru"`: Russian Snowball stemmer
- `"es"`: Spanish Snowball stemmer
- `"sv"`: Swedish Snowball stemmer
- `"tr"`: (no stemmer currently)

```python
from whoosh.lang import stemmer_for_language

stem = stemmer_for_language("en")
print(stem("running"))  # 'run'
```

### `stopwords_for_language`

```python
whoosh.lang.stopwords_for_language(lang) -> list
```

Returns the stop-word list for the given language. Raises `NoStopWords` if
no stop-word list is available.

```python
from whoosh.lang import stopwords_for_language

stops = stopwords_for_language("en")
```

## Snowball Stemmers

The `whoosh.lang.snowball` subpackage contains stemmers implementing the
Snowball stemming algorithms for various languages.

### Available Stemmers

| Module | Class | Language |
|--------|-------|----------|
| `snowball.english` | `EnglishStemmer` | English |
| `snowball.dutch` | `DutchStemmer` | Dutch |
| `snowball.finnish` | `FinnishStemmer` | Finnish |
| `snowball.french` | `FrenchStemmer` | French |
| `snowball.german` | `GermanStemmer` | German |
| `snowball.hungarian` | `HungarianStemmer` | Hungarian |
| `snowball.italian` | `ItalianStemmer` | Italian |
| `snowball.norwegian` | `NorwegianStemmer` | Norwegian |
| `snowball.portugese` | `PortugueseStemmer` | Portuguese |
| `snowball.russian` | `RussianStemmer` | Russian |
| `snowball.romanian` | `RomanianStemmer` | Romanian |
| `snowball.spanish` | `SpanishStemmer` | Spanish |
| `snowball.swedish` | `SwedishStemmer` | Swedish |
| `snowball.danish` | `DanishStemmer` | Danish |

### Base Classes

```python
class whoosh.lang.snowball.bases._ScandinavianStemmer
class whoosh.lang.snowball.bases._StandardStemmer
```

Internal base classes for Snowball stemmers. User code should use the
language-specific stemmer classes directly.

### `classes`

```python
whoosh.lang.snowball.classes = {"da": DanishStemmer, "nl": DutchStemmer, ...}
```

Dictionary mapping two-letter language codes to Snowball stemmer classes.

## Porter Stemmer

### `whoosh.lang.porter`

The original Porter stemming algorithm, faster but less accurate than
Snowball English stemmer.

#### `stem`

```python
whoosh.lang.porter.stem(w) -> str
```

Stems a single English word using the Porter algorithm.

## ISRI Stemmer

### `whoosh.lang.isri.ISRIStemmer`

```python
class whoosh.lang.isri.ISRIStemmer
```

Arabic stemmer based on the Information Science Research Institute (ISRI)
algorithm. Does not use a root dictionary.

#### `stem`

```python
def ISRIStemmer.stem(word) -> str
```

Stems an Arabic word.

## Double Metaphone

### `whoosh.lang.dmetaphone.double_metaphone`

```python
whoosh.lang.dmetaphone.double_metaphone(text) -> tuple
```

Returns a tuple of `(primary, secondary)` metaphone codes for the given
text, using the Double Metaphone algorithm.

## Soundex

### `whoosh.lang.phonetic`

Soundex implementations for phonetic matching.

#### `soundex_en`

```python
whoosh.lang.phonetic.soundex_en(word) -> str
```

English Soundex encoding.

#### `soundex_esp`

```python
whoosh.lang.phonetic.soundex_esp(word) -> str
```

Spanish Soundex encoding.

#### `soundex_ar`

```python
whoosh.lang.phonetic.soundex_ar(word) -> str
```

Arabic Soundex encoding.

## WordNet Thesaurus

### `whoosh.lang.wordnet.Thesaurus`

```python
class whoosh.lang.wordnet.Thesaurus
```

Provides synonym expansion based on WordNet-style data.

**Methods:**
- `synonyms(word)`: Returns the set of synonyms for `word`.
- `__contains__(word)`: Returns `True` if `word` is in the thesaurus.

### Functions

```python
whoosh.lang.wordnet.parse_file(f) -> dict
whoosh.lang.wordnet.make_index(storage, indexname, word2nums, num2words)
whoosh.lang.wordnet.synonyms(word2nums, num2words, word) -> set
```

## Lovins Stemmer

### `whoosh.lang.lovins`

A suffix-stripping stemmer by Lovins. Functions include:
- `stem(word)`: Main stemming function.
- `remove_ending(word)`: Removes suffixes.
- `fix_ending(word)`: Fixes the word ending after stemming.

## Paice-Husk Stemmer

### `whoosh.lang.paicehusk.PaiceHuskStemmer`

```python
class whoosh.lang.paicehusk.PaiceHuskStemmer(rules)
```

A rule-based stemmer using Paice-Husk rules.

#### `stem`

```python
def PaiceHuskStemmer.stem(word) -> str
```

Stems a word using the Paice-Husk algorithm.

**Usage note:** The module also exposes a pre-configured stemmer:
```python
whoosh.lang.paicehusk.stem = PaiceHuskStemmer(defaultrules).stem
```


## DOCUMENT: Matching

# Matching API

Classes and functions for iterating over and combining result sets during
searching. The matching module is a refactored package exposing the same
public API as the former monolithic module.

## Overview

When you search an index, Whoosh creates `Matcher` objects representing the
postings (document IDs and scores) produced by query objects. Matchers can
be combined (e.g., union, intersection) to build compound queries. The
matching module provides the core `Matcher` class hierarchy, utility
functions, and concrete implementations for various query types.

## Core Matcher Classes

### `Matcher`

```python
class whoosh.matching.Matcher
```

Abstract base class for all matchers. Concrete subclasses implement
`__init__()` and the `_set()` and `_maybe_values()` methods.

**Methods:**

#### `init = property(is_active)`

Property that returns whether the matcher is "active" (at top of segment
postings, not exhausted).

#### `init(view, docnum, score)`

Called when the matcher is initialized.

#### `set(matcher)`

Replaces this matcher with another one.

#### `copy()`

Returns a copy of this matcher.

#### `all_ids()`

Returns a list of docnums matched by this matcher.

#### `matches(matcher)`

Returns `True` if any of the current matches in `self` also match in
`matcher`.

#### `skip_to(docid)`

Advances the matcher to the first match at or after `docid`.

#### `skip_to_intersect(matcher)`

Moves this matcher to the earliest matching docnum that is also matched in
`matcher`.

#### `next()`

Advances the matcher to the next match.

#### `next_in_segment()`

Advances to the next match in the current segment.

#### `next_segment(matcher)`

Advances to the next segment in the context of `matcher`.

#### `is_active(in_segment=False)`

Returns `True` if this matcher has more matches to process.

#### `all_matching_segments()`

Generates `(segment_num, matcher)` pairs for all matching segments.

#### `doc()`

Returns the current document number of this matcher. May advance to next
document if not already on one.

#### `docnum()`

Returns the current docnum (segment-relative) of the matcher.

#### `score()`

Returns the current match's score.

#### `value()`

Returns the current match's value (e.g., the decoded stored value of the
term).

#### `supports()`

Returns `True` if `value()` is supported.

#### `value_matches()`

Returns the value at the current match.

#### `all_values()`

Returns a list of all values in this matcher.

#### `supports_lee()`

Returns `True` if the matcher uses lazy evaluation.

#### `lee`

Returns the current "lazy evaluation extension" value (for term vectors).

#### `spans()`

If the postings include positions, returns a list of `Position` objects for
the current match.

#### `spans()`

Returns the spans (positions) of the match in the current document.

#### `next_type()`

Returns the type of the next match.

#### `copy()`

Returns a shallow copy of this matcher.

### `Child`

```python
class whoosh.matching.Child
```

Mixin class for matchers that wrap other matchers.

### `FilterMixin`

```python
class whoosh.matching.FilterMixin
```

Mixin for matchers used as filters (boolean scoring, no relevance).

### `Custom`

```python
class whoosh.matching.Custom
```

Mixin for matchers that return a custom score from `score()` rather than 1.

### `Constant`

```python
class whoosh.matching.Constant
```

Mixin for matchers whose score is always the same value.

### `Coord`

```python
class whoosh.matching.Coord
```

Mixin for matchers that compute coordination factor (for phrase and other
queries that benefit from it).

## Concrete Matcher Classes

### `ListUnion`

```python
class whoosh.matching.ListUnion(matcher, items, maptype=None)
```

Base class for matchers that combine multiple matchers with a list of keys.

#### `filter`

```python
class whoosh.matching.filter
```

Decorator for creating filter matchers (boolean matchers with no relevance).

### `Union`

```python
class whoosh.matching.Union(matcher, items)
```

Base class for the `OR` operator.

### `Intersection`

```python
class whoosh.matching.Intersection(matcher, items)
```

The `AND` operator. A document matches only if it appears in all the child
matchers.

#### `IntersectionFilter`

```python
class whoosh.matching.IntersectionFilter(matcher, items)
```

A filter (no scoring) version of intersection.

### `And`

```python
class whoosh.matching.And(matcher, items)
```

Alias for `Intersection`.

### `Or`

```python
class whoosh.matching.Or(matcher, items)
```

Alias for `Union`.

### `Not`

```python
class whoosh.matching.Not(matcher, a, b)
```

The `NOT` operator. Matches all documents in `a` that are not in `b`.

### `Require`

```python
class whoosh.matching.Require(matcher, a, b)
```

Matches documents in `a` only if they also appear in `b`, but does not add
`b`'s score.

### `AndNot`

```python
class whoosh.matching.AndNot(matcher, a, b)
```

Matches documents in `a` that are not in `b`.

#### `AndMaybe`

```python
class whoosh.matching.AndMaybe(matcher, a, b)
```

Matches documents in `a`, adding `b`'s score if present.

### `BinaryUnion`

```python
class whoosh.matching.BinaryUnion(items)
```

Efficient intersection of exactly two matchers.

#### `BinaryUnion2`

```python
class whoosh.matching.BinaryUnion2
```

Optimized binary union for two items.

### `TreeMatcher`

```python
class whoosh.matching.TreeMatcher
```

A matcher that wraps a `Tree` object for combining results.

### `NestedParent`

```python
class whoosh.matching.NestedParent(parent, child, bools=False)
```

Matches parent documents that have at least one child document matched by
the child matcher. Used for nested document queries.

### `NestedChildren`

```python
class who which.matching.NestedChildren(parentmatch, child)
```

Matches child documents for a given parent document.

### `LengthMatcher`

```python
class whoosh.matching.LengthMatcher(child, q, polarity=False)
```

Matches documents based on field length (used by `Every` query).

### `Filter`

```python
class whoosh.matching.Filter(matcher)
```

Converts any matcher into a filter (no scoring).

### `AlwaysFilter`

```python
class whoosh.matching.AlwaysFilter
```

A filter that matches all documents.

### `NeverFilter`

```python
class whoosh.matching.NeverFilter
```

A filter that matches no documents.

### `PseudoMatcher`

```python
class whoosh.matching.PseudoMatcher
```

Base class for pseudo-matchers used in span queries.

## Matching Utilities

### `current_spans`

```python
whoosh.matching.current_spans(matcher) -> list
```

Returns a list of `Span` objects for the current match in `matcher`, or an
empty list if the matcher doesn't support positions.

### `disjunction_score`

```python
whoosh.matching.disjunction_score(matcher) -> float
```

Returns the sum of `matcher.score()` and the scores of all child matchers of
type `Union`.

### `intersection_score`

```python
whoosh.matching.intersection_score(matcher) -> float
```

Returns the sum of `matcher.score()` and all child matchers of type
`Intersection`.

### `child_count`

```python
whoosh.matching.child_count(matcher) -> int
```

Returns the number of child matchers in `matcher`.

### `has_quality`

```python
whoosh.matching.has_quality(matcher) -> bool
```

Returns `True` if `matcher` has a `query` attribute (i.e., is a
`QueryMatcher`-derived object, or a combination of such matchers).

### `has_untranslated`

```python
whoosh.matching.has_untranslated(matcher) -> bool
```

Returns `True` if the matcher has an `untranslated` attribute (set by
certain wrapper matchers like `TimeLimited`).

### `wrap`

```python
whoosh.matching.wrap(matcher)
```

Returns `matcher` if it has a `.copy()` method, otherwise wraps it in an
`AutoMatcher`.

### `wrap2`

```python
whoosh.matching.wrap2(a, b, m)
```

Returns either a `BinaryUnion2` or an `AutoMatcher` depending on whether `a`
and `b` are list-compatible.

### `unified`

```python
whoosh.matching.unified(matcher)
```

Returns `matcher` if it has an `untranslated` attribute, otherwise returns
`None`.

### `deletion`

```python
whoosh.matching.deletion(matcher)
```

If `matcher` has a `parent` attribute, returns the parent, otherwise returns
`None`.

### `AutoMatcher`

```python
class whoosh.matching.AutoMatcher(m, **kwargs)
```

A general-purpose matcher that wraps arbitrary objects and adds default
behavior for scoring, docnums, and other features. Created by `wrap()`.

### `MatchingTimeLimit`

```python
class whoosh.matching.MatchingTimeLimit
```

A lightweight exception raised when a query matcher exceeds a time limit.

### `TimeLimited`

```python
class whoosh.matching.TimeLimited(child, maxsteps=100, timeout=None, currenttime=None)
```

Wrapper that wraps a `Matcher` to enforce a time limit. Raises
`MatchingTimeLimit` if the time limit is exceeded.

**Parameters:**
- `child`: The matcher to wrap.
- `maxsteps`: Check time every N documents (default `100`).
- `timeout`: Maximum time in seconds (default `None`, no limit).
- `currenttime`: Optional function to use for getting the current time.

### `TermMatcher`

```python
class whoosh.matching.TermMatcher(postings, text, qname, scorer=None, boost=1.0)
```

Matches documents containing a specific term.

**Constructor:**
- `postings`: A `Postings` object from the index reader.
- `text`: The term text.
- `qname`: The query name for this term.
- `scorer`: Optional `Scorer` object.
- `boost`: Boost factor for this term's score.

### `MultiScorer`

```python
class whoosh.matching.MultiScorer(numgroups, start_i=0)
```

A `Scorer` that combines the scores from multiple scorers into one, weighted
across groups of segments.

### `RangeMatcher`

```python
class whoosh.matching.RangeMatcher(start_matcher, end_matcher, query)
```

Matches documents within a range of term values.

### `RegexMatcher`

```python
class whoosh.matching.RegexMatcher(regex, qname, boost=1.0)
```

Matches documents whose terms match a compiled regex.

### `SpanMatcher`

```python
class whoosh.matching.SpanMatcher(matcher, order=0, end=0)
```

Matches spans (positions) within documents.

### `SpanOverlap`

```python
class whoosh.matching.SpanOverlap(l, r)
```

Matches overlapping spans from two matchers.

### `SpanNear`

```python
class whoosh.matching.SpanNear(l, r, slop=1, ordered=True)
```

Matches spans that are near each other within a document.

### `SpanCondition`

```python
class whoosh.matching.SpanCondition(l, r)
```

Matches a condition on spans.

### `SpanBefore`

```python
class whoosh.matching.SpanBefore(l, r, end=0)
```

Matches spans before a given position.

### `SpanAfter`

```python
class whoosh.matching.SpanAfter(l, r, end=0)
```

Matches spans after a given position.

### `SpanOutside`

```python
class whoosh.matching.SpanOutside(l, r, end=0)
```

Matches spans outside a given range.

### `SpanFirst`

```python
class whoosh.matching.SpanFirst(l, start=0, end=1)
```

Matches spans at the beginning of a document.

### `SpanNot`

```python
class whoosh.matching.SpanNot(l, r)
```

Matches spans in `l` that are not in `r`.

### `SpanOr`

```python
class whoosh.matching.SpanOr(items)
```

Logical OR for span matchers.

### `SpanAnd`

```python
class whoosh.matching.SpanAnd(l, r)
```

Logical AND for span matchers.


## DOCUMENT: Middleware

# Middleware API

Reference for the middleware pipeline.

## MiddlewareContext

```python
class whoosh.middleware.context.MiddlewareContext
```

Context object passed through all middleware hooks.

### Attributes

| Attribute | Type | Description |
|-----------|------|-------------|
| `operation` | `str` | Operation type: `index`, `search`, `delete`, `commit` |
| `index` | `Any` | The index object |
| `backend` | `Any` | The backend storage object |
| `writer` | `Any` | The index writer |
| `searcher` | `Any` | The searcher |
| `document` | `dict \| None` | Document being indexed/deleted |
| `query` | `str` | Search query string |
| `collector` | `Any` | Collector instance |
| `results` | `Any` | Search results |
| `labels` | `dict` | Middleware identification labels |
| `metadata` | `dict` | Arbitrary middleware communication data |

### Methods

#### `copy()`

```python
ctx_copy = context.copy()
```

Create a shallow copy.

---

## Middleware

```python
class whoosh.middleware.base.Middleware
```

Base class for all middleware.

### Methods

#### `startup()`

```python
def startup(self, context: MiddlewareContext) -> None:
    """Called once on initialization."""
```

#### `shutdown()`

```python
def shutdown(self, context: MiddlewareContext) -> None:
    """Called once on teardown."""
```

#### `before_index()`

```python
def before_index(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called before indexing a document."""
```

#### `after_index()`

```python
def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called after indexing a document."""
```

#### `before_delete()`

```python
def before_delete(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called before deleting a document."""
```

#### `after_delete()`

```python
def after_delete(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called after deleting a document."""
```

#### `before_search()`

```python
def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called before executing a search."""
```

#### `after_search()`

```python
def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
    """Called after search results are returned."""
```

#### `on_error()`

```python
def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
    """Called on exception. Re-raise by default."""
```

#### `on_commit()`

```python
def on_commit(self, context: MiddlewareContext) -> None:
    """Called after commit."""
```

---

## MiddlewareChain

```python
class whoosh.middleware.chain.MiddlewareChain
```

Orchestrates ordered middleware execution.

### Methods

#### `add()`

```python
chain.add(middleware: Middleware)
```

Add a middleware.

---

#### `extend()`

```python
chain.extend(middlewares: list[Middleware])
```

Add multiple middlewares.

---

#### `run_before()`

```python
context = chain.run_before(
    hook_name: str,
    context: MiddlewareContext,
    fail_open: bool = False
)
```

Run before hooks in order.

---

#### `run_after()`

```python
context = chain.run_after(
    hook_name: str,
    context: MiddlewareContext,
    fail_open: bool = False
)
```

Run after hooks in reverse.

---

#### `run_on_error()`

```python
chain.run_on_error(context, exc, fail_open=False)
```

Call on_error hooks.

---

## Built-in Middleware Classes

### MetricsMiddleware

```python
class whoosh.middleware.base.MetricsMiddleware
```

Tracks indexing/search counts.

#### Methods

##### `after_index()`

Increment documents indexed count.

##### `after_search()`

Increment searches executed count.

##### `get_metrics()`

```python
metrics = metrics_mw.get_metrics() -> dict
```

Return collected metrics.

---

### CacheMiddleware

```python
class whoosh.middleware.base.CacheMiddleware
```

In-memory search cache.

#### Methods

##### `before_search()`

Check cache for query.

##### `after_search()`

Store results in cache.

##### `get_cached()`

```python
cached = cache_mw.get_cached(query: str) -> Any
```

##### `set_cached()`

```python
cache_mw.set_cached(query: str, results: Any)
```

---

### CompressionMiddleware

```python
class whoosh.middleware.base.CompressionMiddleware
```

Mark documents for compression at backend level.

---

### EncryptionMiddleware

```python
class whoosh.middleware.base.EncryptionMiddleware
```

Mark documents for encryption at backend level.

---

## Exceptions

### StopOperation

```python
class whoosh.middleware.exceptions.StopOperation(Exception)
```

Raise to abort an operation.

---

## Integration Helpers

### `apply_middleware_to_writer()`

```python
def apply_middleware_to_writer(
    writer: IndexWriter,
    middleware: list[Middleware] = None
) -> MiddlewareWriter
```

---

### `apply_middleware_to_searcher()`

```python
def apply_middleware_to_searcher(
    searcher: Searcher,
    middleware: list[Middleware] = None
) -> MiddlewareSearcher
```


## DOCUMENT: Modern

# Modern API

The `whoosh_modern` package provides the modern, fully-typed surface of
Whoosh-NG. It includes data sources, schema discovery, validation,
middleware, profiling, autocomplete, vector search integration, and an
optimized batch writer.

## Data Sources

```python
from whoosh_modern.data_sources import (
    DataSource,
    SQLSource,
    RESTSource,
    FastCSVSource,
    JSONSource,
    GraphQLSource,
    PydanticSource,
    PandasSource,
    PolarsSource,
    ParquetSource,
    PeeweeSource,
    TortoiseSource,
    SQLAlchemySource,
    ObservableDataSource,
    DataSourceConfig,
)
```

### DataSource Protocol

The `DataSource` protocol defines the interface for all data source
implementations:

```python
class DataSource(Protocol):
    @property
    def name(self) -> str
    def discover_schema(self) -> Schema
    def iter_documents(self) -> Iterator[Document]
    def stream_batches(self, batch_size=1000) -> Iterator[list[dict]]
    def health_check(self) -> bool
```

Additional capability protocols are available:
- `IncrementalDataSource` — supports `iter_changes(since)` for incremental sync
- `AsyncDataSource` — supports `aiter_documents()` for async iteration
- `RefreshableDataSource` — supports `refresh()`
- `CountableDataSource` — supports `document_count()`
- `MetadataDataSource` — supports `metadata()`
- `ObservableDataSource` — supports `add_observer()`/`remove_observer()`

### SQLSource

```python
from whoosh_modern.data_sources import SQLSource

source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles WHERE status='published'",
    incremental_field="updated_at",
    id_field="id",
    pool_size=5,
)
schema = source.discover_schema()
docs = list(source.iter_documents())
batches = list(source.stream_batches(batch_size=1000))
count = source.document_count()
meta = source.metadata()
```

### RESTSource

```python
from whoosh_modern.data_sources import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    method="GET",
    pagination="page",  # or "offset", "cursor"
    page_size=50,
    headers={"Authorization": "Bearer token"},
    auth={"type": "bearer", "token": "..."},
    document_path="results",  # Extract from nested response
)
schema = source.discover_schema()
docs = list(source.iter_documents())
```

### DataSourceConfig

Declarative configuration with a factory:

```python
from whoosh_modern.data_sources import DataSourceConfig

config = DataSourceConfig(
    type="sql",
    connection=conn,
    query="SELECT * FROM articles",
    sequential_field="updated_at",
)
source = config.create()  # Returns a configured SQLSource
```

Supported types: `sql`, `sqlalchemy`, `rest`, `csv`, `json`, `graphql`,
`pydantic`, `pandas`, `polars`, `parquet`, `peewee`, `tortoise`.

## Schema Discovery

```python
from whoosh_modern.schema_discovery import SchemaDiscovery

# From column metadata (list of (name, sql_type) tuples)
columns = [("id", "INTEGER"), ("title", "TEXT"), ("published", "TIMESTAMP")]
schema = SchemaDiscovery.from_result_set(columns)

# From sample documents (auto-detects types)
schema = SchemaDiscovery.from_sample(docs)

# Optimized variant (drops non-searchable TEXT, infers IDs/booleans)
schema = SchemaDiscovery.from_sample_optimized(docs, searchable_text=["title", "content"])

# Detect ID field from schema
id_field = SchemaDiscovery.detect_id_field(dict(schema))
```

### SQL Type Mapping

`SchemaDiscovery` includes a built-in SQL type map: `VARCHAR`→`TEXT`,
`INTEGER`→`NUMERIC`, `BOOLEAN`→`BOOLEAN`, `TIMESTAMP`→`DATETIME`, `JSON`→
`KEYWORD`, `UUID`→`ID`, etc.

## FacetManager

```python
from whoosh_modern.facets import FacetManager, TermsFacet, RangeFacet, DateRangeFacet

manager = FacetManager(schema)
# Or with manual config:
manager = FacetManager(schema, config={"price": {"type": "range", "buckets": [...]}})

facets = manager.get_facets()          # Auto-discovered + manual facets
config = manager.get_facet_config("category")
stats = manager.get_facet_stats()
manager.set_manual_override("price", {"type": "range", "buckets": ["0-100", "100-500"]})
```

Auto-discovery rules:
- `TEXT`, `KEYWORD`, `BOOLEAN`, `ID` → `TermsFacet`
- `NUMERIC` → `RangeFacet`
- `DATETIME` → `DateRangeFacet`

## Validation Framework

```python
from whoosh_modern.validation import ValidationFramework, ValidationResult

validator = ValidationFramework()
results = validator.validate(source)

for result in results:
    print(f"Level {result.level}: passed={result.passed}")
    for warning in result.warnings:
        print(f"  Warning: {warning}")
    for error in result.errors:
        print(f"  Error: {error}")
```

Four validation levels:

| Level | Method | Purpose |
|-------|--------|---------|
| 1 | `validate_structural()` | DataSource availability, schema detection |
| 2 | `validate_search()` | Indexable fields, term vectors, searchable analyzers |
| 3 | `validate_performance()` | Performance warnings (e.g., TEXT fields on large datasets) |
| 4 | `validate_runtime()` | Sample iteration, type conformance |

## Middleware Pipeline

```python
from whoosh_modern.middleware import (
    Middleware,
    MiddlewarePipeline,
    RetryMiddleware,
    LoggingMiddleware,
    CacheMiddleware,
)

pipeline = MiddlewarePipeline(
    RetryMiddleware(attempts=3, backoff="exponential", jitter=True),
    LoggingMiddleware(level=logging.INFO),
    CacheMiddleware(maxsize=128),
)

@pipeline.wrap
def my_operation():
    return searcher.search(query)

result = pipeline.execute(my_operation)
```

### Middleware Types

- **`Middleware`** (abstract): Subclass and implement `wrap(operation)`.
- **`RetryMiddleware`**: Retries failed operations with exponential or linear
  backoff, with optional jitter.
- **`LoggingMiddleware`**: Logs execution time and errors.
- **`CacheMiddleware`**: Caches results keyed by operation name and arguments.
  Exposes `stats` property and `clear()` method.

## SearchView

```python
from whoosh_modern.views import SearchView

view = SearchView(
    name="articles",
    source=source,
    fields={"title": fields.TEXT(stored=True)},  # Field type overrides
    facets={"category": {"type": "terms", "limit": 50}},
    incremental_field="updated_at",
    strict=False,  # Raise on validation failures
    middleware=[LoggingMiddleware()],
    schema_version="1.0",
)

ix = view.build("indexdir")       # Create/populate index
count = view.reindex()            # Full reindex
count = view.refresh()            # Incremental refresh
results = view.validate()         # Run validation
view.evolve_schema({"new_field": fields.TEXT})  # Add fields without reindexing
```

## Optimized Writer

```python
from whoosh_modern.writer import ModernIndex, ModernIndexWriter

# Create or open an optimized index
index = ModernIndex.create("indexdir", schema=my_schema)
# Or open existing:
# index = ModernIndex.open("indexdir")

# Optimized writer for batch processing millions of docs
with index.writer(batch_size=5000, limitmb=512, multisegment=True) as writer:
    for batch in source.stream_batches(batch_size=5000):
        writer.add_batch(batch)
        # Or: writer.add_batches(source.stream_batches(batch_size=5000))

print(f"Documents indexed: {writer.doc_count}")

# Access searcher
with index.searcher() as searcher:
    results = searcher.search(query)
```

### Key Optimizations

- **Multisegment mode**: No merging during indexing (set with
  `multisegment=True`)
- **Reduced Python overhead**: Batch-oriented add API
- **Configurable memory limits**: `limitmb` parameter controls buffering

## Analysis Extensions

```python
from whoosh_modern.analysis import StemmingAnalyzer, get_stemmer, register_stemmer
```

### StemmingAnalyzer

Enhanced analyzer with pluggable stemmer backends:

```python
# Auto-detect best available stemmer
analyzer = StemmingAnalyzer(stemmer="auto")

# Explicit internal stemmer
analyzer = StemmingAnalyzer(stemmer="internal")

# PyStemmer (requires pip install whoosh-ng[fast-stemming])
analyzer = StemmingAnalyzer(stemmer="pystemmer")

# Custom stemmer provider
analyzer = StemmingAnalyzer(stemmer=my_custom_stemmer)

# Full parameters
analyzer = StemmingAnalyzer(
    expression=r"\S+",
    stoplist=None,
    minsize=2,
    maxsize=None,
    gaps=False,
    stemmer="auto",
    ignore=None,
    cachesize=50000,
)
```

### Stemmer Providers

```python
from whoosh_modern.analysis import get_stemmer, list_available_backends

# Get a stemmer provider
stemmer = get_stemmer("auto", "english")
stemmed = stemmer.stem("running")  # "run"

# List available backends
backends = list_available_backends()
# {"internal": "available", "pystemmer": "not installed"}

# Register a custom stemmer
@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower()

# Priority: PyStemmer > Internal (whoosh.lang)
```

### StemmerProvider Protocol

```python
class StemmerProvider(Protocol):
    def stem(self, word: str) -> str
    @property
    def name(self) -> str
    @property
    def language(self) -> str
```

Available providers:
- `InternalStemmerProvider` — wraps Whoosh's built-in `whoosh.lang.porter.stem`
- `PyStemmerProvider` — wraps the PyStemmer library (fastest)
- `IdentityStemmerProvider` — no-op stemmer for testing

## Autocomplete

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create an autocomplete provider
provider = create_autocomplete("inverted")

# Add phrases
provider.add(["hello world", "hello there", "goodbye world"])

# Search
hits = provider.search("hello", limit=10)
for hit in hits:
    print(hit.text, hit.score)
```

### Classes

- **`AutocompleteHit`**: Simple data class with `text` and `score` attributes.
- **`AutocompleteProvider` (Plugin)**: Abstract base for autocomplete
  implementations. Implements `add()` and `search()`.
- **`InvertedIndexAutocomplete`**: Default provider using prefix matching with
  a scoring function favoring exact prefix matches.

## Exceptions

```python
from whoosh_modern.exceptions import (
    DataSourceError,
    DataSourceNotFoundError,
    DocumentIterationError,
    SchemaDiscoveryError,
    ValidationError,
)
```

All exceptions inherit from `DataSourceError`, which carries optional
`source` and `field` context attributes.

## Storage Providers

```python
from whoosh_modern.storage import (
    FileStorage,
    AsyncFileStorage,
    S3Storage,
    HybridStorage,
    AsyncHybridStorage,
)
```

### FileStorage

Local filesystem storage. Keys are relative paths under ``root``.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

### AsyncFileStorage

Async variant of ``FileStorage``. All operations run on a worker thread
via ``asyncio.to_thread``.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

### S3Storage

S3-compatible blob storage. ``boto3`` is required only when this provider
is used; it is imported lazily so the rest of Whoosh-NG does not depend on
it. A ``client`` may be injected for testing.

```python
from whoosh_modern.storage import S3Storage

storage = S3Storage(bucket="my-index-bucket", prefix="segments")
storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

### HybridStorage

Compose a local cache with a remote backend for cloud-native indexes.
The remote is the source of truth; the local cache is a write-through
performance layer.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")  # served from cache after first read
storage.invalidate("segment_1.dat")   # force refresh from remote
storage.prefetch(["segment_2.dat"])   # warm cache proactively
```

Read path:

1. local cache hit → return immediately
2. cache miss → read from remote, write-through into cache, return

Write path:

- ``remote.write(key, data)`` (source of truth)
- on success → ``local_cache.write(key, data)``
- on failure → raise before polluting cache

### AsyncHybridStorage

Async variant of ``HybridStorage``. Remote operations are executed on a
worker thread via ``asyncio.to_thread`` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = AsyncHybridStorage(local_cache="./cache", remote=remote)

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```


## DOCUMENT: Overview

# API Overview

This section provides a comprehensive reference of the Whoosh-NG public API.

## Modules

| Module | Description |
|--------|-------------|
| `whoosh.index` | High-level index creation, opening, and management |
| `whoosh.fields` | Schema and field type definitions |
| `whoosh.writing` | Writer classes and merge policies |
| `whoosh.searching` | Searcher, Results, and collectors |
| `whoosh.query` | Query classes and parsers |
| `whoosh.qparser` | Query parser implementation |
| `whoosh.analysis` | Tokenizers, filters, and analyzers |
| `whoosh.highlight` | Search result highlighting |
| `whoosh.spelling` | Spelling correction |
| `whoosh.sorting` | Facets and sorting |
| `whoosh.event_bus` | Event system |
| `whoosh.hooks` | Hook system |
| `whoosh.middleware` | Middleware pipeline |
| `whoosh.plugins` | Plugin system and registry |
| `whoosh.backends` | Storage backends |
| `whoosh.vector` | Vector search providers |
| `whoosh_modern.autocomplete` | Autocomplete providers |
| `whoosh_fastapi` | FastAPI integration |

## Quick Reference

### Index Lifecycle

```python
from whoosh.index import create_in, open_dir, exists_in

# Create
ix = create_in("indexdir", schema)

# Open
ix = open_dir("indexdir")

# Check
if exists_in("indexdir"):
    ix = open_dir("indexdir")
```

### Writing

```python
with ix.writer() as writer:
    writer.add_document(field1=value1, field2=value2)
    writer.commit()
```

### Reading

```python
from whoosh.qparser import QueryParser

with ix.searcher() as searcher:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("query")
    results = searcher.search(q)
```

### Schema

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    count=NUMERIC(int, stored=True)
)
```


## DOCUMENT: Plugins

# Plugins & Registry

Extend Whoosh-NG through the plugin system and registries.

## Plugin System

### BasePlugin

```python
class whoosh.plugins.base.BasePlugin
```

All plugins inherit from this class.

#### Attributes

- `name (str)`: Plugin name.
- `version (str)`: Plugin version.
- `dependencies (list[str])`: Required plugins.

#### Methods

##### `setup()`

```python
def setup(self, registry) -> None:
    """Called when the plugin is enabled."""
```

##### `teardown()`

```python
def teardown(self, registry) -> None:
    """Called when the plugin is disabled."""
```

##### `middleware()`

```python
def middleware(self) -> list[Middleware]:
    """Return middleware instances."""
```

---

### PluginManager

```python
class whoosh.plugins.manager.PluginManager
```

Manages plugin lifecycle.

#### Methods

##### `load_plugins()`

```python
PluginManager.load_plugins()
```

Auto-discover plugins from entry points.

---

##### `register()`

```python
PluginManager.register(name: str, plugin: BasePlugin)
```

Register a plugin manually.

---

##### `enable()`

```python
PluginManager.enable(name: str)
```

Enable a registered plugin.

---

##### `disable()`

```python
PluginManager.disable(name: str)
```

Disable a plugin.

---

##### `get()`

```python
plugin = PluginManager.get(name: str)
```

Get a plugin instance.

---

##### `list_plugins()`

```python
plugins = PluginManager.list_plugins()
```

List all registered plugins.

---

##### `get_middleware_chain()`

```python
chain = PluginManager.get_middleware_chain()
```

Get the combined middleware chain from all plugins.

---

## Registry System

Registries provide centralized object management.

### Registry Base

```python
class whoosh.registry.base.Registry
```

Generic registry.

#### Methods

##### `register()`

```python
Registry.register(
    key: str,
    value: Any,
    owner: str = None
)
```

Register a value.

---

##### `un


## DOCUMENT: Query

# Query API

Build and execute queries programmatically.

## QueryParser

```python
class whoosh.qparser.QueryParser(
    fieldname: str,
    schema: Schema,
    group=AndGroup,
    **kwargs
)
```

Convert a query string into a Query object.

### Methods

#### `parse()`

```python
query = qp.parse(querystring)
```

Parse a query string.

---

#### `tokenize()`

```python
tokens = qp.tokenize(querystring)
```

Tokenize a query string without parsing.

---

### MultifieldParser

```python
class whoosh.qparser.MultifieldParser(
    fieldnames: list,
    schema: Schema,
    fieldboosts: dict = None,
    group=OrGroup,
    **kwargs
)
```

Search multiple fields with different boosts.

**Example:**
```python
from whoosh.qparser import MultifieldParser

qp = MultifieldParser(
    ["title", "content"],
    schema,
    fieldboosts={"title": 2.0}
)
```

## Query Classes

All queries inherit from `Query`:

```python
class whoosh.query.Query
```

### Methods

#### `matcher()`

```python
matcher = query.matcher(searcher, context=None)
```

Return a matcher for executing the query.

#### `__and__()`, `__or__()`, `__invert__()`

Combine queries with `&`, `|`, `-`.

### Leaf Queries

#### Term

```python
Term(fieldname: str, text: str, boost: float = 1.0)
```

Match a specific term.

---

#### Phrase

```python
Phrase(fieldname: str, words: list, boost: float = 1.0, slop: int = 1)
```

Match a phrase.

---

#### Prefix

```python
Prefix(fieldname: str, text: str, boost: float = 1.0)
```

Match terms starting with `text`.

---

#### Wildcard

```python
Wildcard(fieldname: str, text: str, boost: float = 1.0)
```

Match terms with `?` and `*` wildcards.

---

#### FuzzyTerm

```python
FuzzyTerm(
    fieldname: str,
    text: str,
    maxdist: int = 2,
    prefix: int = 0,
    boost: float = 1.0
)
```

Fuzzy match with edit distance.

---

#### Range

```python
NumericRange(
    fieldname: str,
    start: Any,
    end: Any,
    startexact: bool = False,
    endexact: bool = False,
    boost: float = 1.0
)
```

Numeric range query.

```python
DateRange(
    fieldname: str,
    start: datetime,
    end: datetime,
    startexact: bool = False,
    endexact: bool = False,
    boost: float = 1.0
)
```

Date range query.

---

#### Every

```python
Every(fieldname: str, boost: float = 1.0)
```

Match every document with any term in this field.

### Boolean Queries

#### And

```python
And(children: list, boost: float = 1.0)
```

All children must match.

---

#### Or

```python
Or(children: list, boost: float = 1.0)
```

Any child must match.

---

#### Not

```python
Not(query, exclude)
```

Match docs matching query but not exclude.

---

#### DisjunctionMax

```python
DisjunctionMax(
    children: list,
    tiebreak: float = 0.0,
    boost: float = 1.0
)
```

OR-like with scoring tiebreaker.

### Special Queries

#### Require

```python
Require(match, requires)
```

Match must have `match`, and at least one of `requires`.

---

#### AndMaybe

```python
AndMaybe(must, should)
```

Must match `must`, optionally boosting with `should`.

---

#### Boost

```python
Boost(q, factor)
```

Multiply score by factor.

---

#### ConstantScore

```python
ConstantScore(q, score=1.0)
```

Assign constant score.

## Query Operators

```python
q1 & q2      # And
q1 | q2      # Or
~q1          # Not
q1 ^ q2      # DisjunctionMax
```

## Plugins

```python
from whoosh.qparser import QueryParserPlugin

class RangePlugin(QueryParserPlugin):
    def __init__(self):
        pass

    def evaluate(self, env, signode):
        # Return a query node
        return query.Range(signode.fieldname, ...)
```

## Exceptions

```python
class whoosh.qparser.QueryParserError(Exception)
```

Raised on parse errors.


## DOCUMENT: Reading

# Reading API

Classes and functions for reading from an index. The reading module is a
refactored package exposing the same public API as the former monolithic
module.

## Overview

The reading module provides classes for accessing documents, terms, and
postings in an index. The main entry points are `IndexReader` objects obtained
from a searcher. These readers allow you to enumerate terms, access stored
fields, iterate postings, and get term frequencies.

## Core Classes

### `IndexReader`

```python
class whoosh.reading.IndexReader
```

Abstract base class for reading index data. Concrete subclasses include
`SegmentReader` and `MultiReader` (which wraps multiple segment readers).

### `MultiReader`

```python
class whoosh.reading.MultiReader(readers, base=None)
```

Combines multiple `IndexReader` instances into one. All docnums are treated
as relative to the combined index.

**Constructor:**
- `readers`: A list of `IndexReader` instances.
- `base`: Optional list of cumulative document count offsets for each reader.

**Methods:**

#### `doc_frequency(fieldname, text)`

Returns the total number of documents that have the given term in the given
field across all sub-readers.

#### `documents()`

Yields dictionaries of stored fields for each document in the index.

#### `stored_fields(docnum)`

Returns a dictionary of stored field values for the given document number
(index-wide docnum).

```python
r = my_index.reader()
print(r.stored_fields(20))
```

#### `all_stored_fields()`

Yields a `(docnum, stored_fields)` tuple for each document in the index.

#### `terms(fieldname)`

Yields `(fieldname, text)` tuples for every term in the given field.

#### `terms_from(segmentreader,fieldnameprefix)`

Low-level method for multi-reader.

#### `has_termvector(docnum, fieldname)`

Returns `True` if the document has a term vector for the given field.

#### `term_vector(docnum, fieldname)`

Returns a `TermVector` for the given document and field.

#### `is_deleted(docnum)`

Returns `True` if the given document (index-wide docnum) is deleted.

#### `all_doc_ids()`

Returns a sorted array of non-deleted document IDs.

#### `min_spam(fieldname)`

Returns the minimum spam value for the given field.

#### `set_spam(fieldname)`

Returns the set spam value for the given field.

#### `has_exact_length(docnum)`

Returns `True` if the exact length is known for `docnum`.

#### `doc_field_length(docnum, fieldname=None, default=1)`

Returns the length of the given field in the given document.

```python
r = my_index.reader()
length = r.doc_field_length(20, "content")
```

#### `max_field_length(fieldname)`

Returns the maximum length of the given field across all documents.

#### `iter_fieldname`

Low-level method for multi-reader.

#### `lexicon(fieldname)`

Returns an array of all unique terms in the given field, sorted.

#### `expanded_lexicon(fieldname)`

Low-level method that yields terms without the overhead of building an array.

#### `term_info(fieldname, text)`

Returns a `TermInfo` object for the given term, or `None` if the term does
not appear in the index.

#### `terminfos(fieldname)`

Yields `(text, TermInfo)` pairs for the given field.

#### `postings(fieldname, text, stype=None)`

Returns a `Matcher` for the postings list of the given term.

```python
r = my_index.reader()
m = r.postings("content", "whoosh")
for docnum, score in m:
    print("doc %d has term" % docnum)
```

#### `_all_postings(fieldname)`

Low-level. Yields `(text, matcher)` pairs for all terms in a field.

#### `_posting_fragments()`

Low-level.

#### `has_vector(docnum, fieldname)`

Returns `True` if the given field has a term vector in the given document.

#### `vectors(docnum)`

Yields `(fieldname, TermVector)` pairs for all term vectors in the document.

#### `all_items(fieldname)`

Yields `(term, weight, docfreq)` tuples for every term in the given field.

#### `frequency(fieldname, text)`

Returns the total frequency of the term across all documents.

#### `idf(term)`

Returns an iterator of `(docnum, idf)` pairs for the given term.

#### `spelling`

Returns a `SpellingAnalyzer` for the given field.

#### `doc_term(slicenum, fieldname, word)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

#### `doc_diff(slicenum, fieldname, text, num)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

### `SegmentReader`

```python
class whoosh.reading.SegmentReader(segment, schema, storage, base=True)
```

Reader for a single segment of the index.

**Constructor:**
- `segment`: The `Segment` object.
- `schema`: The `Schema` object.
- `storage`: The `Storage` instance.
- `base`: Base document number offset (usually `True`, meaning compute it).

### `MultiID3Reader`

```python
class whoosh.reading.MultiID3Reader(readers, base)
```

Combines multiple readers that have ID3 codec.

### `TermInfo`

```python
class whoosh.reading.TermInfo(
    df=0,
    weight=0,
    minlength=0,
    maxlen=0,
    maxnum=0,
    numdocs=0,
    scorable=True
)
```

Information about a term in the index.

**Attributes:**
- `df`: Document frequency (number of documents containing the term).
- `weight`: Total term frequency across all documents.
- `minlength`: Minimum document length where the term appears.
- `maxlength`: Maximum document length where the term appears. This is `0`
  if lengths are not stored.
- `maxnum`: Maximum number of occurrences per document.
- `numdocs`: Number of documents where the term has a non-zero contribution
  to the score.
- `scorable`: Whether this term is scorable.

## Term Vector

### `TermVector`

```python
class whoosh.reading.TermVector(docnum, fieldname, format_, terms, store_term_vector)
```

Represents the term vector for a single document/field pair.

**Methods:**

#### `tokens(text=None)`

Yields `(t, w, v, p)` tuples for terms in this field.

- `t`: The term string.
- `w`: The term weight (frequency in this document).
- `v`: The list of positions where the term occurs. (`None` if positions
  are not stored.)
- `p`: The list of characters where the term occurs. (`None` if character
  vectors are not stored.)

#### `items(text=None)`

Like `tokens()` but includes term strings in the result.

```python
tv = my_index.reader().term_vector(0, "content")
for token, frequency, positions, chars in tv.tokens():
    print(token, frequency, positions)
```

**Parameters:**
- `text`: Optional `Bytes` object. If given, only yield terms starting with
  this text (used for multi-byte tokenization).

## Reader Utilities

### `get_storage`

```python
whoosh.reading.get_storage(searcher) -> Storage
```

Returns the storage object associated with the searcher.

### `get_index_schema`

```python
whoosh.reading.get_index_schema(searcher) -> Schema
```

Returns the schema object associated with the searcher.

### `load_termdocs`

```python
whoosh.reading.load_termdocs(reader, fieldname, text) -> list
```

Returns a list of document numbers that have the given term.

### `read_pattern`

```python
whoosh.reading.read_pattern(reader, fieldname, expression) -> list
```

Returns sorted term list from `reader.lexicon(fieldname)` filtered to those
matching `expression`.

### `read_terminfo`

```python
whoosh.reading.read_terminfo(reader, fieldname, text) -> TermInfo or None
```

Returns a `TermInfo` for the given term, or `None` if not found.


## DOCUMENT: Searching

# Searching API

Execute queries and retrieve results.

## Searcher

```python
class whoosh.searching.Searcher
```

The Searcher is the primary interface for reading from the index.

### Methods

#### `search()`

```python
results = searcher.search(query, limit=10, **kwargs)
```

Execute a query and return Results.

**Args:**
- `query`: The query to run.
- `limit (int)`: Maximum number of results. Use `None` for all results.

**Returns:**
- `Results`: A Results object.

---

#### `search_page()`

```python
results = searcher.search_page(query, pagenum, pagelen=10)
```

Get a page of results.

---

#### `search_with_collector()`

```python
searcher.search_with_collector(query, collector)
```

Advanced search with custom collector.

---

#### `find()`

```python
results = searcher.find("field", "text")
```

Convenience method to search a single field.

---

#### `documents()`

```python
docs = list(searcher.documents(fieldname=value))
```

Get stored documents matching a term.

---

#### `document()`

```python
doc = searcher.document(fieldname=value)
```

Get a single stored document.

---

#### `lexicon()`

```python
terms = list(searcher.lexicon("fieldname"))
```

List all terms in a field.

---

#### `all_stored_fields()`

```python
for fields in searcher.all_stored_fields():
    print(fields)
```

Iterate over all stored fields.

---

#### `all_features()`

```python
with searcher.all_features() as features:
    facets = features.facet(facet)
```

Get facet counts across all documents.

## Results

```python
class whoosh.searching.Results
```

List-like container for matched documents.

### Methods

#### `__len__()`

```python
total = len(results)
```

Total matching documents (may recount).

#### `scored_length()`

```python
scored = results.scored_length()
```

Number of scored/sorted documents in this results object.

#### `__getitem__()`

```python
hit = results[0]
hits = results[0:10]
```

Get a hit by index or slice.

#### `has_matched_terms()`

```python
if results.has_matched_terms():
    print(results.matched_terms())
```

Check if matched terms were collected.

#### `iter_matched_terms()`

Iterate over (docnum, term) pairs.

#### `upgrade()`

Move docs from another Results to top.

#### `extend()`

Append docs from another Results.

#### `upgrade_and_extend()`

Upgrade docs and append rest.

#### `filtered_count`

Number of documents filtered out.

#### `collapsed_counts`

Dict of collapse keys to filtered counts.

## Hit

```python
class whoosh.searching.Hit
```

A single matched document.

### Attributes

- `hit["fieldname"]`: Stored field value
- `hit.score`: Relevance score
- `hit.docnum`: Internal document number

### Methods

#### `highlights()`

```python
snippets = hit.highlights("content", top=3)
```

Get highlighted snippets.

#### `matched_terms()`

```python
terms = hit.matched_terms()
```

Get terms that matched (if `terms=True`).

## Highlight

```python
from whoosh.highlight import highlight, Fragment

snippets = hit.highlights(
    "content",
    top=3,
    fragmenter=None,
    formatter=None
)
```

## Collectors

```python
from whoosh.collectors import Collector, FacetCollector, TimeLimitCollector
```

## Sorting and Facets

```python
from whoosh import sorting

facet = sorting.FieldFacet("category")
results = searcher.search(query, sortedby="date")
```


## DOCUMENT: Sorting

# Sorting API

Classes and functions for faceting and sorting search results. The sorting
module is a refactored package exposing the same public API as the former
monolithic module.

## Overview

Sorting and faceting use `FacetType` objects to compute sort keys for documents.
A `FacetType` creates a `Categorizer` that computes a key for each document.
The key is used for sorting and grouping. `FacetMap` objects hold the
results of grouping documents by a facet.

## Facet Types

### `FacetType`

```python
class whoosh.sorting.FacetType
```

Base class for "facets" — aspects that can be sorted and/or faceted.

**Attributes:**
- `maptype`: Default `FacetMap` class to use for this facet.

**Methods:**

#### `categorizer(global_searcher)`

Returns a `Categorizer` corresponding to this facet.

- `global_searcher`: A parent searcher for global document ID references.

#### `map(default=None)`

Returns a `FacetMap` instance for holding facet results.

#### `default_name()`

Returns the default name for this facet (default `"facet"`).

### `Categorizer`

```python
class whoosh.sorting.Categorizer
```

Base class for objects that compute a key value for a document for sorting and
faceting. Created by `FacetType` objects via `categorizer()`.

**Attributes:**
- `allow_overlap (bool)`: If `True`, use `keys_for()` to allow overlapping
  groups. Default `False`.
- `needs_current (bool)`: If `True`, the categorizer needs the matcher to be
  in a valid state when `key_for()` is called. Default `False`.

**Methods:**

#### `set_searcher(segment_searcher, docoffset)`

Called when the collector moves to a new segment. Sets up segment-specific
data.

- `segment_searcher`: The atomic sub-searcher for the current segment.
- `docoffset`: Offset of the segment's docnums relative to the full index.

#### `key_for(matcher, segment_docnum)`

Returns a sort key for the current match.

- `matcher`: A `Matcher` object. If `needs_current` is `False`, do not use
  this object as it may be inconsistent.
- `segment_docnum`: Segment-relative document number.

#### `keys_for(matcher, segment_docnum)`

Yields multiple keys for the current match. Called instead of `key_for()`
when `allow_overlap` is `True`.

#### `key_to_name(key)`

Translates the sort key into a human-readable representation for facet
group names (e.g., converts an integer date sort key to a `datetime`).

### `FieldFacet`

```python
class whoosh.sorting.FieldFacet(
    fieldname,
    reverse=False,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets by the contents of a field.

**Constructor:**
- `fieldname`: Name of the field to sort/facet on.
- `reverse`: If `True`, reverse the sort order.
- `allow_overlap`: If `True`, allow documents to appear in multiple groups
  when they have multiple terms in the field.
- `maptype`: `FacetMap` class for holding results.

```python
paths = FieldFacet("path", reverse=True)
tags = FieldFacet("tag")
results = searcher.search(myquery, sortedby=paths, groupedby=tags)
```

### `ColumnCategorizer`

Categorizer that reads values from a column for sorting. Used when a field
has a column type.

### `ReversedColumnCategorizer`

Categorizer that reverses column values for fields that are not naturally
reversible.

### `OverlappingCategorizer`

```python
class whoosh.sorting.OverlappingCategorizer
```

Categorizer used when `allow_overlap=True`. A single document can belong to
multiple facet groups.

### `PostingCategorizer`

```python
class whoosh.sorting.PostingCategorizer
```

Categorizer for fields without column values. Builds an array caching the
order of all documents. Used as a fallback; prefer setting
`sortable=True` on fields.

### `QueryFacet`

```python
class whoosh.sorting.QueryFacet(
    querydict: dict,
    other=None,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets based on the results of a series of queries.

**Constructor:**
- `querydict`: Dictionary mapping keys to `Query` objects.
- `other`: Key to use for documents matching no queries.

### `RangeFacet`

```python
class whoosh.sorting.RangeFacet(
    fieldname,
    start,
    end,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on numeric ranges. Ranges are inclusive at the start and
exclusive at the end.

```python
prices = RangeFacet("price", 0, 1000, 100)
results = searcher.search(myquery, groupedby=prices)
```

- `fieldname`: The numeric field to facet on.
- `start`: Start of the entire range.
- `end`: End of the entire range.
- `gap`: Size of each bucket (can be a sequence for progressive gaps).
- `hardend`: If `True`, clamp the last bucket to `end`.

### `DateRangeFacet`

```python
class whoosh.sorting.DateRangeFacet(
    fieldname,
    startdate,
    enddate,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on date ranges. Extends `RangeFacet` but uses
`datetime` objects for start/end and `timedelta`/`relativedelta` for gaps.
Generates `DateRange` queries instead of `TermRange` queries.

```python
from datetime import datetime
from whoosh.support.relativedelta import relativedelta

startdate = datetime(1920, 1, 1)
enddate = datetime.now()
gap = relativedelta(years=5)
bdays = DateRangeFacet("birthday", startdate, enddate, gap)
```

### `ScoreFacet`

```python
class whoosh.sorting.ScoreFacet
```

Uses a document's relevance score as a sorting criterion.

```python
tag_score = MultiFacet(["tag", ScoreFacet()])
results = searcher.search(myquery, sortedby=tag_score)
```

### `FunctionFacet`

```python
class whoosh.sorting.FunctionFacet(fn)
```

Lets you pass an arbitrary function that computes the sort key. The function
is called with `(searcher, docid)` where `docid` is an absolute index
document number.

```python
fn = lambda s, docid: s.doc_field_length(docid, "content")
lengths = FunctionFacet(fn)
```

### `TranslateFacet`

```python
class whoosh.sorting.TranslateFacet(fn, *facets)
```

Applies a custom function to the key generated by one or more wrapped facets.
Useful for custom collation, such as Unicode Collation Algorithm (UCA) sorting.

```python
from pyuca import Collator

c = Collator("allkeys.txt")
facet = FieldFacet("name")
facet = TranslateFacet(c.sort_key, facet)
results = searcher.search(myquery, sortedby=facet)
```

**Constructor:**
- `fn`: Function applied to the computed key values.
- `*facets`: One or more `FacetType` objects whose keys are passed to `fn`.

### `StoredFieldFacet`

```python
class whoosh.sorting.StoredFieldFacet(
    fieldname,
    allow_overlap=False,
    split_fn=None,
    maptype=None
)
```

Sorts/groups using the value in an unindexed, stored field (e.g., `STORED`).
Usually slower than using an indexed field.

**Constructor:**
- `fieldname`: Name of the stored field.
- `allow_overlap`: If `True`, when grouping, allow documents to appear in
  multiple groups when they have multiple values (split by `split_fn` or
  `string.split()`).
- `split_fn`: Custom function to split a stored field value into multiple
  facet values (only used when `allow_overlap=True`).

### `MultiFacet`

```python
class whoosh.sorting.MultiFacet(items=None, maptype=None)
```

Sorts/facets by the combination of multiple sub-facets.

```python
facet = MultiFacet([FieldFacet("tag"), FieldFacet("path")])
results = searcher.search(myquery, sortedby=facet)
```

Strings in the items list are treated as field names:

```python
facet = MultiFacet(["tag", "path"])
```

**Methods:**
- `from_sortedby(sortedby)`: Class method that creates a `MultiFacet` from
  a field name, facet, or list thereof.
- `add_field(fieldname, reverse=False)`: Add a `FieldFacet`.
- `add_query(querydict, other=None, allow_overlap=False)`: Add a `QueryFacet`.
- `add_score()`: Add a `ScoreFacet`.
- `add_facet(facet)`: Add an arbitrary `FacetType`.

### `Facets`

```python
class whoosh.sorting.Facets(x=None)
```

Maps facet names to `FacetType` objects for creating multiple independent
groupings of documents.

```python
facets = Facets()
facets.add_field("tag")
facets.add_facet("price", RangeFacet("price", 0, 1000, 100))
results = searcher.search(myquery, groupedby=facets)

tag_groups = results.groups("tag")
price_groups = results.groups("price")
```

**Class Methods:**
- `from_groupedby(groupedby)`: Creates a `Facets` object from a field name,
  `FacetType`, dict, list, or another `Facets` object.

**Methods:**
- `names()`: Returns an iterator of facet names.
- `items()`: Returns a list of `(name, facet)` tuples.
- `add_field(fieldname, **kwargs)`: Adds a `FieldFacet`.
- `add_query(name, querydict, **kwargs)`: Adds a `QueryFacet`.
- `add_facet(name, facet)`: Adds a `FacetType` under the given name.
- `add_facets(facets, replace=True)`: Adds the contents of a `Facets` or
  `dict` to this object.

## Facet Maps

### `FacetMap`

```python
class whoosh.sorting.FacetMap
```

Base class for objects holding the results of grouping search results by a
facet. Use `as_dict()` to access results.

```python
myfacet = FieldFacet("size", maptype=OrderedList)
myfacet = FieldFacet("size", maptype=Count)
```

**Methods:**
- `add(groupname, docid, sortkey)`: Adds a document to the facet results.
- `as_dict()`: Returns a dictionary mapping group names to values.

### `OrderedList`

```python
class whoosh.sorting.OrderedList
```

Stores a list of document numbers for each group, in sorted order.

### `UnorderedList`

```python
class whoosh.sorting.UnorderedList
```

Stores a list of document numbers for each group in arbitrary order. Slightly
faster and more memory-efficient than `OrderedList` when ordering doesn't
matter.

### `Count`

```python
class whoosh.sorting.Count
```

Stores the count of documents in each group.

### `Best`

```python
class whoosh.sorting.Best
```

Stores the "best" (highest sort key) document in each group.

## Sorting Utilities

### `add_sortable`

```python
whoosh.sorting.add_sortable(
    writer,
    fieldname,
    facet,
    column=None
)
```

Adds a per-document value column to an existing field, making it sortable.
Useful for retrofitting fields that were created without `sortable=True`.

**Example:**
```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    facet = sorting.FieldFacet("price")
    sorting.add_sortable(w, "price", facet)
```

**Parameters:**
- `writer`: An `IndexWriter` object.
- `fieldname`: Name of the field to add sortable values to.
- `facet`: A `FacetType` object to generate per-document values.
- `column`: Optional `ColumnType` to store the values. If omitted, uses the
  field's default column type.


## DOCUMENT: Spelling

# Spelling API

Functions and classes for correcting typos in user queries using edit-distance
(Damerau-Levenshtein) matching against the terms in the index.

## Corrector Objects

### `Corrector`

```python
class whoosh.spelling.Corrector
```

Base class for spelling correction objects. Concrete subclasses implement the
`_suggestions()` method.

**Methods:**

#### `suggest(text, limit=5, maxdist=2, prefix=0)`

Returns a list of suggested corrections for `text`, ranked by edit distance
then by frequency.

- `text`: The text to check. Will **not** be added to suggestions even if it
  appears in the index.
- `limit`: Maximum number of suggestions to return.
- `maxdist`: Maximum edit distance to look at (values > 2 are inefficient).
- `prefix`: Require suggestions to share this length of prefix with `text`.
  Increasing to even `1` dramatically speeds up suggestions.

#### `_suggestions(text, maxdist, prefix)`

Low-level method yielding `(score, suggestion)` tuples. Subclasses must
implement this.

### `ReaderCorrector`

```python
class whoosh.spelling.ReaderCorrector(reader, fieldname, fieldobj)
```

Suggests corrections based on terms in a specific field of an `IndexReader`.

**Ranks suggestions by edit distance, then by highest to lowest frequency.**

**Constructor:**
- `reader`: An `IndexReader` object.
- `fieldname`: The name of the field to get suggestions from.
- `fieldobj`: The `FieldType` for the field.

### `ListCorrector`

```python
class whoosh.spelling.ListCorrector(wordlist)
```

Suggests corrections based on a sorted list of strings.

**Constructor:**
- `wordlist`: A sorted list of words to match against.

### `MultiCorrector`

```python
class whoosh.spelling.MultiCorrector(correctors, op)
```

Merges suggestions from a list of sub-correctors.

**Constructor:**
- `correctors`: List of `Corrector` objects.
- `op`: A function (e.g., `max` or `operator.add`) to combine scores from
  multiple correctors for the same suggestion.

## Query Correction

### `Correction`

```python
class whoosh.spelling.Correction(q, qstring, corr_q, tokens)
```

Represents the corrected version of a user query string.

**Attributes:**
- `query`: The corrected `Query` object.
- `string`: The corrected user query string.
- `original_query`: The original `Query` object.
- `original_string`: The original user query string.
- `tokens`: List of token objects representing corrected words.

**Methods:**

#### `format_string(formatter)`

Highlights corrected words in the original query string using the given
`Formatter`.

```python
from whoosh import highlight

correction = searcher.correct_query(q, qstring)
hf = highlight.HtmlFormatter(classname="change")
html = correction.format_string(hf)
```

- `formatter`: A `Formatter` instance (or class, which will be instantiated).
- Returns: Formatted string, typically with corrections emphasized.

### `QueryCorrector`

```python
class whoosh.spelling.QueryCorrector(fieldname)
```

Base class for objects that correct words in a user query.

**Constructor:**
- `fieldname`: The default field name for corrections.

**Methods:**

#### `correct_query(q, qstring)`

Returns a `Correction` object representing the corrected form of the given
query.

- `q`: The original `Query` tree to be corrected.
- `qstring`: The original user query string (may be `None`).
- Returns: A `Correction` object.

#### `field()`

Returns the field name this corrector operates on.

### `SimpleQueryCorrector`

```python
class whoosh.spelling.SimpleQueryCorrector(
    correctors: dict,
    terms: list,
    aliases=None,
    prefix: int = 0,
    maxdist: int = 2
)
```

A simple query corrector based on a mapping of field names to `Corrector`
objects, and a list of `(fieldname, text)` tuples to correct.

**Constructor:**
- `correctors`: Dictionary mapping field names to `Corrector` objects.
- `terms`: Sequence of `(fieldname, text)` tuples representing terms to be
  corrected.
- `aliases`: Dictionary mapping field names in the query to field names for
  spelling suggestions.
- `prefix`: Suggested replacement words must share this number of initial
  characters. Default `0`.
- `maxdist`: Maximum edit distance for suggestions. Values > 2 may be slow.


## DOCUMENT: Writing

# Writing API

Write, update, and delete documents using the `IndexWriter` interface.

## IndexWriter

```python
class whoosh.writing.IndexWriter
```

Base class for writing documents.

### Context Manager

```python
with ix.writer() as writer:
    writer.add_document(title="Hello", content="World")
    # commit() called automatically
```

### Methods

#### `add_document()`

```python
writer.add_document(**fields)
```

Add a document to the index.

**Special kwargs:**
- `_stored_<fieldname>`: Alternate stored value
- `_<fieldname>_boost`: Field-specific boost
- `_boost`: Document-wide boost

---

#### `update_document()`

```python
writer.update_document(**fields)
```

Update/replace a document. Uses `unique` fields to find existing documents.

---

#### `delete_document()`

```python
writer.delete_document(docnum: int, delete: bool = True)
```

Delete by document number.

---

#### `delete_by_term()`

```python
writer.delete_by_term(fieldname: str, text: str) -> int
```

Delete all documents with term in field.

**Returns:**
- `int`: Number of documents deleted.

---

#### `delete_by_query()`

```python
writer.delete_by_query(q: Query, searcher=None) -> int
```

Delete documents matching query.

---

#### `commit()`

```python
writer.commit(
    mergetype=None,
    optimize=False,
    merge=True
)
```

Commit changes to disk.

**Args:**
- `mergetype`: Custom merge function
- `optimize`: Merge all segments into one
- `merge`: If False, don't merge existing segments

---

#### `cancel()`

```python
writer.cancel()
```

Cancel pending changes and release lock.

---

#### `add_field()`

```python
writer.add_field(fieldname: str, fieldtype, **kwargs)
```

Add a field to schema (before adding documents).

---

#### `remove_field()`

```python
writer.remove_field(fieldname: str)
```

Remove a field from schema.

---

#### `searcher()`

```python
searcher = writer.searcher(**kwargs)
```

Return a searcher (for reading during write session).

---

#### `reader()`

```python
reader = writer.reader(**kwargs)
```

Return a reader for the current state.

---

#### `group()`

```python
with writer.group():
    writer.add_document(kind="class", name="MyClass")
    writer.add_document(kind="method", name="my_method")
```

Context manager for grouping documents into one segment.

## SegmentWriter

Concrete implementation of `IndexWriter`.

### Constructor

```python
SegmentWriter(
    ix,
    poolclass=None,
    timeout=0.0,
    delay=0.1,
    _lk=True,
    limitmb=128,
    docbase=0,
    codec=None,
    compound=True,
    **kwargs
)
```

## AsyncWriter

Threaded writer that automatically retries on lock contention.

```python
from whoosh.writing import AsyncWriter

writer = AsyncWriter(
    index,
    delay=0.25,
    writerargs={}
)
```

## BufferedWriter

Buffers documents in memory and commits periodically.

```python
from whoosh.writing import BufferedWriter

writer = BufferedWriter(
    index,
    period=60,       # Max seconds between commits
    limit=100,       # Max documents per commit
    writerargs={}    # Extra args for writer
)
```

The `BufferedWriter` also provides `reader()` and `searcher()` methods for quasi-real-time search.

## Merge Policies

```python
from whoosh.writing import NO_MERGE, MERGE_SMALL, OPTIMIZE, CLEAR

writer.commit(mergetype=NO_MERGE)     # No merging
writer.commit(mergetype=MERGE_SMALL) # Merge small segments
writer.commit(mergetype=OPTIMIZE)    # Merge all into one
writer.commit(mergetype=CLEAR)       # Delete all existing segments
```

## PostingPool

Internal pool for sorting postings. Typically not used directly.

```python
class whoosh.writing.PostingPool
```

## Exceptions

### IndexingError

```python
class whoosh.writing.IndexingError(Exception)
```

Raised when an indexing operation fails.


## DOCUMENT: Auto Indexing

# Auto-Indexing

Whoosh-NG provides utilities for automatic schema discovery and data-source driven indexing.

## Schema Discovery

The `SchemaDiscovery` utility inspects a data source and auto-generates a Whoosh schema:

```python
from whoosh_modern.discovery import SchemaDiscovery

discovery = SchemaDiscovery(source=data_source)
schema = discovery.discover()
```

See [SearchView](/examples/search-view) and [Data Sources](/examples/data-sources) for usage examples.


## DOCUMENT: Backends

# Backends

Whoosh-NG supports pluggable storage backends through the Provider Architecture. The default backend stores data as files on disk, but you can use SQLite, PostgreSQL, S3, and more.

## Built-in Backends

| Backend | Class | Description |
|---------|-------|-------------|
| File (default) | `FileBackend` | Stores index as files on disk |
| SQLite | `SQLiteBackend` | Stores index in SQLite database |
| Memory | `MemoryBackend` | In-memory backend (testing only) |

## File Backend (Default)

```python
from whoosh.index import create_in

# Uses FileBackend by default
ix = create_in("indexdir", schema)
```

### Configuration

```python
from whoosh.backends.file import FileBackend

backend = FileBackend(
    storage=FileStorage("indexdir"),
    compound=True  # Use compound files
)
```

## SQLite Backend

```python
from whoosh.backends.sqlite import SQLiteBackend
from whoosh.store.sqlite import SQLiteStorage

storage = SQLiteStorage("index.db")
backend = SQLiteBackend(storage=storage)
```

### Advantages

- Single file index
- Better for transactional workloads
- Easier backups
- Supports concurrent reads

### Disadvantages

- Slower for large indexes
- Limited by SQLite performance

## Memory Backend

```python
from whoosh.backends.memory import MemoryBackend

backend = MemoryBackend()
# Useful for testing
```

## Custom Backend

Create a custom backend by subclassing `Backend`:

```python
from whoosh.backends.abc import Backend

class MyBackend(Backend):
    def create(self):
        """Create a new segment."""
        pass

    def open(self):
        """Open existing segment."""
        pass

    def close(self):
        """Close the backend."""
        pass

    def commit(self):
        """Commit changes."""
        pass
```

## Registering a Backend

```python
from whoosh.registry import BackendRegistry

BackendRegistry.register("my_backend", MyBackend, "my_package")
```

## Backend Selection

Choose a backend based on your use case:

| Use Case | Recommended Backend |
|----------|---------------------|
| Small to medium indexes | File (default) |
| Single-file deployment | SQLite |
| Testing | Memory |
| Distributed systems | Object storage (S3, MinIO) |
| High concurrency | SQLite or custom |

## Best Practices

1. **File backend for production**: Most battle-tested
2. **SQLite for single-file**: Easier deployment
3. **Memory for tests**: Fast, no cleanup needed
4. **Compound files**: Enable for reduced file count
5. **Backup strategy**: File backend = copy directory; SQLite = copy file


## DOCUMENT: Core Concepts

# Core Concepts

Whoosh-NG is a pure-Python search engine library. This guide explains the main concepts you need to understand to use it effectively.

## Architecture

Whoosh-NG follows a layered architecture:

![Architecture diagram: Application → Whoosh-NG Core (Search Engine, Schema, Plugin Manager, Registry, Middleware, Event Bus, Hooks) → Plugins (FastAPI, Vector Search, Autocomplete, PostgreSQL, S3, Monitoring, Admin UI)](/assets/architecture.svg)

## Key Components

### Index

An `Index` is the top-level container for your searchable documents. It manages one or more segments on disk.

```python
from whoosh.index import create_in, open_dir

# Create a new index
ix = create_in("indexdir", schema)

# Open an existing index
ix = open_dir("indexdir")
```

### Schema

The `Schema` defines the fields that documents in your index can have. Each field has a type that determines how it is indexed and stored.

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    content=TEXT,
    rating=NUMERIC(float, stored=True)
)
```

### Writer

An `IndexWriter` lets you add, update, and delete documents in the index.

```python
writer = ix.writer()
writer.add_document(title="Hello", content="World")
writer.commit()
```

### Searcher

A `Searcher` lets you query the index and retrieve results.

```python
with ix.searcher() as s:
    results = s.search("hello")
```

### Query Parser

The `QueryParser` converts a query string into a query object that the searcher can execute.

```python
from whoosh.qparser import QueryParser

qp = QueryParser("content", schema)
query = qp.parse("hello world")
```

## Modern Features

### Plugin System

Plugins extend Whoosh-NG without modifying the core. Plugins can:

- Register new vector providers
- Add FastAPI endpoints
- Provide custom analyzers
- Hook into the middleware pipeline

```python
from whoosh.plugins.manager import PluginManager

# Load plugins from entry points
PluginManager.load_plugins()

# Or register manually
PluginManager.register("my_plugin", MyPlugin())
```

### Middleware Pipeline

Middleware intercepts indexing and search operations:

```python
from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext):
        print(f"Searching: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext):
        print(f"Found: {len(context.results) if context.results else 0} results")
        return context
```

### Vector Search

Vector fields enable semantic search using embeddings:

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    content=TEXT,
    embedding=VectorField(dimensions=384)
)
```

### Event Bus

The event system allows loose coupling between components:

```python
from whoosh.event_bus import EventBus, DocumentIndexed

bus = EventBus()

@bus.subscribe
def on_document_indexed(event: DocumentIndexed):
    print(f"Document indexed: {event.docnum}")
```

## Data Flow

### Indexing Flow

1. Application calls `writer.add_document()`
2. Schema validates and analyzes fields
3. Middleware `before_index` hooks run
4. Document is written to segment
5. Middleware `after_index` hooks run
6. `DocumentIndexed` event is published
7. `commit()` merges segments and writes TOC

### Search Flow

1. Application calls `searcher.search(query)`
2. Query is parsed into query tree
3. Middleware `before_search` hooks run
4. Searcher executes query against segments
5. Results are scored and sorted
6. Middleware `after_search` hooks run
7. `SearchExecuted` event is published
8. Results are returned to application

## Design Principles

1. **Composability**: Components combine via `|` and `+` operators
2. **Zero-cost abstractions**: No middleware = no overhead
3. **Sync-first**: Core is synchronous; async is opt-in
4. **Plugin isolation**: Plugins cannot break the core
5. **Type safety**: Comprehensive type hints throughout


## DOCUMENT: Dates

# Dates and Numeric Ranges

This guide covers working with `DATETIME` and `NUMERIC` fields, including
range queries, range faceting, and date math.

## DATETIME Fields

`DATETIME` fields store Python `datetime` objects and can be queried with
range queries.

```python
from datetime import datetime
from whoosh import fields, index

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    published_date=fields.DATETIME(stored=True, sortable=True),
)
```

### Indexing Dates

```python
ix = index.create_in("indexdir", schema)
with ix.writer() as w:
    w.add_document(
        title="Article 1",
        published_date=datetime(2024, 6, 15, 14, 30),
    )
```

### Date Range Queries

Use `Range` or `QueryParser` syntax:

```python
from whoosh.qparser import QueryParser
from whoosh.query import Range, Every

# Using QueryParser syntax
qp = QueryParser("published_date", schema=ix.schema)
q = qp.parse("[2024-01-01 TO 2024-12-31]")

# Using Range query directly
from datetime import datetime
q = Range(
    "published_date",
    datetime(2024, 1, 1),
    datetime(2024, 12, 31),
)

with ix.searcher() as searcher:
    results = searcher.search(q)
```

### Sorting by Date

```python
from whoosh.sorting import FieldFacet

# Sort by date, most recent first
results = searcher.search(
    query,
    sortedby=FieldFacet("published_date", reverse=True),
)
```

## NUMERIC Fields

`NUMERIC` fields store integers and floating-point numbers.

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, stored=True, sortable=True),
    rating=fields.NUMERIC(float, stored=True),
)
```

### Numeric Range Queries

```python
from whoosh.query import NumericRange

q = NumericRange("price", 100, 500)

# Or with QueryParser
qp = QueryParser("price", schema=ix.schema)
q = qp.parse("[100 TO 500]")
```

### Numeric Faceting

Group results into numeric ranges using `RangeFacet`:

```python
from whoosh.sorting import RangeFacet

price_ranges = RangeFacet("price", 0, 1000, 100)
results = searcher.search(query, groupedby=price_ranges)

for groupname, docnums in results.groups("price").items():
    print(f"Price ${groupname}: {len(docnums)} results")
```

## Date Faceting

Group results by date intervals using `DateRangeFacet`:

```python
from datetime import datetime
from whoosh.sorting import DateRangeFacet

start = datetime(2020, 1, 1)
end = datetime(2026, 1, 1)
date_facet = DateRangeFacet(
    "published_date",
    start,
    end,
    relativedelta(years=1),  # Requires: from dateutil.relativedelta import relativedelta
)
results = searcher.search(query, groupedby=date_facet)

for year_range, docnums in results.groups("published_date").items():
    print(f"Year {year_range}: {len(docnums)} results")
```

## Sorting and Filtering by Numbers

### Sorting

```python
from whoosh.sorting import FieldFacet

# Sort by price ascending
results = searcher.search(query, sortedby=FieldFacet("price"))
```

### Filtering

```python
from whoosh.query import NumericRange

# Only results with price >= 50 and price < 200
filter_q = NumericRange("price", 50, 200)
results = searcher.search(query, filter=filter_q)
```

## Making Date/Numeric Fields Sortable

When defining a schema, set `sortable=True` on `NUMERIC` or `DATETIME` fields
to enable sorting by that field:

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, sortable=True),
    date=fields.DATETIME(sortable=True),
)
```

If you forgot to set `sortable=True`, you can add it after indexing:

```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    sorting.add_sortable(w, "price", sorting.FieldFacet("price"))
```


## DOCUMENT: Glossary

# Glossary

A glossary of key terms used in Whoosh.

## Analysis

The process of converting text into tokens (individual units like words or
terms) for indexing. Involves tokenization, normalization (lowercasing,
stemming), and filtering (stop word removal, etc.).

## Analyzer

A chain of `Tokenizer` and `Filter` objects that processes text into
tokens. Examples include `RegexTokenizer`, `NgramTokenizer`, `LowercaseFilter`,
`StopFilter`, and `StemmerFilter`.

## Compound File

A file format that combines multiple index segment files into a single
`.seg` file. This can improve performance on some filesystems by reducing
file handle usage. Configured via the codec's `should_assemble` setting.

## Document

A single record in the index, similar to a row in a database. A document
contains fields (analogous to columns).

## Field

A named attribute of a document. Fields have a type (defined by `FieldType`)
that determines how the field's value is indexed and stored.

## Field Type

The class (e.g., `TEXT`, `ID`, `NUMERIC`, `DATETIME`, `BOOLEAN`) that
defines how a field's value is tokenized, stored, indexed, and made
sortable/facetable.

## Filter

An `Analyzer` component that processes, transforms, or filters tokens
after tokenization. Examples: `LowercaseFilter`, `StopFilter`,
`StemmerFilter`.

## Format

A `Format` object controls how posting information (term frequency, positions,
character offsets) is encoded for each field in the inverted index.
Examples: `Existence`, `Frequency`, `Positions`, `Characters`.

## Fragmentation

The process of selecting text spans around matched terms for highlighting.

## Highlighter

The `whoosh.highlight` module, which provides formatters, fragmenters, and
scorers for highlighting search terms in documents.

## Index

The collection of segment files that store the inverted index, document
data, and metadata (the table of contents, or TOC).

## IndexWriter

The `IndexWriter` class is used to create and modify the index. It buffers
document additions and deletions and commits them to disk.

## Inverted Index

The core data structure of a search engine: for each unique term, it stores a
list of documents (and positions) where that term appears.

## Matcher

An object that iterates over matching documents in the postings list for a
query. Matchers can be combined (union, intersection, etc.) for compound
queries.

## Posting

A single entry in the inverted index: a (document ID, term frequency, value)
tuple for a given term.

## Schema

Defines the fields, their types, and indexing options for an index. A schema
is passed to `Storage.create_index()`.

## Scorer

An object that computes a relevance score for a document given a query and
term weights. Different weighting models (BM25, TF-IDF, etc.) use different
scorers.

## Segment

A self-contained portion of the inverted index. An index may consist of
multiple segments. Segments are merged periodically (during optimize or
merge operations) to improve performance.

## Sort Key

A value computed per-document (via a `FacetType` and its `Categorizer`)
used to order results during sorting and faceting.

## Stemming

The process of reducing words to their root form (e.g., "running" → "run",
"cats" → "cat") to improve recall by matching inflected forms.

## Stop Words

High-frequency, low-information words (e.g., "the", "a", "and") that are
typically filtered out during indexing.

## Term

A unique (field name, token text) pair in the inverted index.

## Term Vector

Optional per-document data structure storing the terms (and optionally
positions and character offsets) that appear in a document's field, enabling
features like highlighting and pseudo-relevance feedback.

## Tokenizer

An `Analyzer` component that splits input text into tokens. Examples:
`RegexTokenizer`, `PathTokenizer`, `NgramTokenizer`.

## Whoosh Query

Whoosh's own query syntax, parsed by `QueryParser`. Supports fielded
search, phrase queries, wildcards, ranges, and more.


## DOCUMENT: Indexing

# Indexing

This guide covers adding, updating, and deleting documents in your Whoosh-NG index.

## Opening a Writer

```python
from whoosh import index

ix = index.open_dir("indexdir")

# Basic writer
writer = ix.writer()

# Writer with custom options
writer = ix.writer(
    timeout=10.0,      # Lock acquisition timeout (seconds)
    delay=0.1,         # Delay between lock retries (seconds)
    limitmb=128,       # Posting pool run size (MiB)
    compound=True      # Use compound files
)
```

## Adding Documents

```python
with ix.writer() as writer:
    writer.add_document(
        title="First document",
        content="Hello world",
        path="/doc1",
        tags=["python", "search"]
    )
    writer.add_document(
        title="Second document",
        content="Goodbye world",
        path="/doc2",
        tags=["python", "tutorial"]
    )
    # commit() is called automatically on exit
```

### Multi-value Fields

Pass lists to add multiple values for multi-valued fields:

```python
writer.add_document(
    title="Document with multiple tags",
    content="Content here",
    tags=["python", "whoosh", "search", "tutorial"]
)
```

### Stored vs Indexed Values

For fields that are both indexed and stored, you can store a different value:

```python
writer.add_document(
    title="Title to be indexed",
    _stored_title="Display title to show in results"
)
```

### Field Boosts

Boost individual fields at document level:

```python
writer.add_document(
    title="Important title",
    _title_boost=2.0,   # Double weight for title terms
    content="Body content"
)
```

## Updating Documents

Use `update_document` to replace documents with matching unique fields:

```python
schema = Schema(path=ID(unique=True, stored=True), content=TEXT)
ix = index.create_in("indexdir", schema)

with ix.writer() as writer:
    writer.add_document(path="/doc1", content="Original content")
    writer.commit()

with ix.writer() as writer:
    # Replaces any document with path="/doc1"
    writer.update_document(path="/doc1", content="Updated content")
    writer.commit()
```

## Deleting Documents

```python
# Delete by document number
writer.delete_document(docnum=42)

# Delete by term in a field
writer.delete_by_term("path", "/doc1")

# Delete by query
from whoosh.query import Term
q = Term("tags", "deprecated")
writer.delete_by_query(q)

writer.commit()
```

## Commit and Merge Policies

### Basic Commit

```python
writer.commit()
```

### Optimize (Merge All)

```python
writer.commit(optimize=True)
```

### No Merge

```python
writer.commit(merge=False)
```

### Custom Merge Policy

```python
from whoosh.writing import NO_MERGE, MERGE_SMALL, OPTIMIZE

writer.commit(mergetype=NO_MERGE)
writer.commit(mergetype=MERGE_SMALL)
writer.commit(mergetype=OPTIMIZE)

# Custom function
def my_merge(writer, segments):
    # Custom merge logic
    return segments

writer.commit(mergetype=my_merge)
```

## BufferedWriter

For high-throughput scenarios where documents arrive one at a time:

```python
from whoosh.writing import BufferedWriter

# Buffers documents and commits periodically
buffered = BufferedWriter(
    ix,
    period=60,    # Max seconds between commits
    limit=100,    # Max documents per commit
    writerargs={} # Extra args for underlying writer
)

with buffered:
    buffered.add_document(title="Doc 1", content="Content")
    buffered.add_document(title="Doc 2", content="More")
# commit() called automatically on close
```

## AsyncWriter

For web applications where multiple processes may write:

```python
from whoosh.writing import AsyncWriter

# Automatically retries on lock contention
async_writer = AsyncWriter(ix, delay=0.25)

async_writer.add_document(title="Async doc", content="Content")
async_writer.commit()
```

## Middleware Integration

```python
from whoosh.middleware import MiddlewareChain, MetricsMiddleware, CacheMiddleware
from whoosh.middleware.integration import apply_middleware_to_writer

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware()
])

with apply_middleware_to_writer(ix.writer(), chain.middlewares) as writer:
    writer.add_document(title="Tracked", content="Content")
```

## Best Practices

1. **Use context managers**: `with ix.writer() as w:` ensures proper cleanup
2. **Batch commits**: Group many documents per commit for better performance
3. **Choose merge policy wisely**: `MERGE_SMALL` is usually fine; use `NO_MERGE` for bulk loads followed by `OPTIMIZE`
4. **Handle locks**: Use `BufferedWriter` or `AsyncWriter` in multi-process environments
5. **Don't forget to close**: Always call `commit()` or `cancel()` to release the write lock


## DOCUMENT: Installation

# Installation

## Requirements

- Python 3.10+
- No mandatory dependencies (pure Python)
- Optional extras for advanced features

## pip install

```bash
pip install whoosh-ng
```

## Extras

| Extra | Description |
|-------|-------------|
| `vector` | NumPy-based vector providers |
| `autocomplete` | Autocomplete plugin |
| `api` | FastAPI plugin |
| `metrics` | Prometheus metrics integration |
| `all` | Install everything |

```bash
pip install whoosh-ng[all]
```

## Development install

```bash
git clone https://github.com/your-org/whoosh-NG.git
cd whoosh-NG
uv sync --extra dev
```

## Verification

```bash
uv run pytest tests/ -q
uv run ruff check src/ tests/
uv run ruff format --check .
uv run mypy src/whoosh
```

## Next Steps

- [Quick Start](/core/quickstart)
- [Core Concepts](/core/core-concepts)


## DOCUMENT: Legacy Cleanup

# Legacy Code Cleanup Strategy

This guide explains how Whoosh-NG separates modern typed code from legacy code,
and how the legacy cleanup is progressing.

## Why a legacy boundary?

`whoosh-modern` is the new, fully typed surface of Whoosh-NG.
The original `whoosh` package still works at runtime, but it carries decades of
Python 2/3 compatibility patterns, dynamic metaprogramming, and untyped internals.
Trying to force strict types on all of it at once would block development.

The cleanup strategy is **incremental and opt-in**:

1. `src/whoosh_modern/` is typed and linted with `pyright` and `mypy` strict.
2. `src/whoosh/` is the legacy surface. It is split into:
   - **excluded modules** (documented in `pyrightconfig.json`) — code that is too
     dynamic or vendored for an economical type pass right now;
   - **cleanup candidates** — small, isolated files that are straightforward to
     annotate and verify.
3. Each sprint, a wave of candidates is typed, tested, and promoted out of the
   high-tolerance zone.

## Current pyright/mypy thresholds (Sprint 2)

| Checker | Scope | Threshold |
|---------|-------|-----------|
| `pyright` | `src/whoosh_modern/` | **0 errors** (strict) |
| `pyright` | legacy | **≤ 500 errors** (tolerant) |
| `mypy` | `src/` | **0 errors** (via overrides + `ignore_errors`) |

## Exclusion rationale (pyrightconfig.json)

The `exclude` list in `pyrightconfig.json` groups excluded files by theme:

- **Vendored / no stubs**: `pyparsing.py`, `relativedelta.py`
- **Migration shims**: `codec/whoosh2.py`, `codec/whoosh3.py`
- **Dynamic parsing**: `qparser/`, `query/`, `analysis/`, `automata/`
- **Large datastores**: `filedb/`, `reading/`, `writing/`
- **Heuristic / data-driven**: `lang/dmetaphone.py`, `lang/lovins.py`,
  `lang/phonetic.py`, `lang/wordnet.py`
- **Core dynamic objects**: `classify.py`, `index.py`, `locking.py`,
  `formats.py`, `middleware/`
- **Vendored low-level**: `support/bench.py`, `support/base85.py`,
  `support/bitstream.py`, `support/bitvector.py`, `support/charset.py`,
  `support/levenshtein.py`

## Sprint 2 cleanup plan

For Sprint 2, the focus is on small utility and support modules that have few
external dependencies and no heavy metaprogramming.

Candidate wave:

- `src/whoosh/util/varints.py`
- `src/whoosh/util/text.py`
- `src/whoosh/util/loading.py`
- `src/whoosh/support/bitstream.py`
- `src/whoosh/support/levenshtein.py`

For each file:

1. Remove the blanket `# type: ignore` (if present).
2. Add precise function signatures.
3. Run `pyright` and `mypy` to confirm **0 new errors**.
4. Move the file out of `pyrightconfig.json` excludes.
5. Add a regression test in `tests/test_legacy_cleanup.py`.

## Long-term goal

Eventually every file in `src/whoosh/` should be checkable by `mypy` and
`pyright` without blanket excludes. Until then, the exclude list is the
explicit ledger of debt, and each sprint chips away at it.


## DOCUMENT: Migration

# Migration Guide

This guide helps you migrate from Whoosh legacy or Whoosh-Reloaded 3.x to Whoosh-NG v3.0.0.
> **Next release**: v4.0.0.dev0 (in development) will add SchemaBuilder, enhanced middleware exception hierarchy, and more — see the [CHANGELOG](https://github.com/dorel14/whoosh-ng/blob/master/CHANGELOG.md) for details.

## From Whoosh 1.x/2.x (Legacy)

### Import Paths

| Legacy | Whoosh-NG |
|--------|-----------|
| `import whoosh` | `import whoosh` |
| `from whoosh.index import create_in` | `from whoosh.index import create_in` |
| `from whoosh.fields import Schema, TEXT` | `from whoosh.fields import Schema, TEXT` |
| `from whoosh.qparser import QueryParser` | `from whoosh.qparser import QueryParser` |

The core API is intentionally stable. Most existing code works unchanged.

### Spelling API

```python
# Legacy
from whoosh.spelling import SpellChecker
corrector = SpellChecker(ix.reader(), "content")

# Whoosh-NG
from whoosh.spelling import ReaderCorrector
corrector = ReaderCorrector(ix.searcher().reader(), "content", ix.schema["content"])
suggestions = corrector.suggest("helo", limit=5)
```

### Highlighting

```python
# Legacy API unchanged
results[0].highlights("content")
```

## From Whoosh-Reloaded 3.x

### No Breaking Changes

Whoosh-NG is a continuation of Whoosh-Reloaded. All existing code works as-is.

### Optional: Plugin Migration

If you used `whoosh_modern` directly:

```python
# Old
from whoosh_modern.vector.numpy_provider import NumpyProvider

# New (via registry)
from whoosh.vector import NumpyProvider
from whoosh.registry import VectorRegistry

VectorRegistry.register("numpy", NumpyProvider(), "my_app")
```

### Middleware (New in v4.0.0.dev0)

```python
# Optional migration: add middleware to existing code

from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context):
        print(f"Query: {context.query}")
        return context

# Wrap existing writer/searcher
writer = apply_middleware_to_writer(ix.writer(), [LoggingMiddleware()])
```

### SchemaBuilder (New in v4.0.0.dev0)

```python
# Old
schema = Schema(title=TEXT(stored=True), content=TEXT)

# New (fluent API)
from whoosh.fields import SchemaBuilder

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("content", TEXT)
    .build()
)
```

## Upgrade Checklist

1. **Update dependencies**:
   ```bash
   pip install --upgrade whoosh-ng
   ```

2. **Run tests**:
   ```bash
   uv run pytest tests/ -q
   ```

3. **Update optional deps** (if using plugins):
   ```bash
   pip install whoosh-ng[all]
   ```

4. **Review middleware**: Consider adding middleware for cross-cutting concerns

5. **Update config**: If using `whoosh.config`, review new options

## Deprecations

| Feature | Status | Replacement |
|---------|--------|-------------|
| `whoosh_modern.vector` | Deprecated | `whoosh.vector` |
| Raw `whoosh.store` | Deprecated | `whoosh.backends` |
| Direct `SegmentWriter` usage | Discouraged | Use `IndexWriter` |

## Breaking Changes

Whoosh-NG maintains backward compatibility. If you find a breaking change, please report it as an issue.

### Exception Hierarchy

New in v4.0.0.dev0: `MiddlewareError` and `StopOperation` in middleware:

```python
from whoosh.middleware.exceptions import MiddlewareError, StopOperation
```

## Getting Help

- [GitHub Issues](https://github.com/your-org/whoosh-NG/issues)
- [Documentation](/)
- [Migration Examples](https://github.com/dorel14/whoosh-ng/tree/master/docs/archive_jekyll/_en/examples)


## DOCUMENT: Nested

# Nested Documents

This guide covers indexing and searching hierarchical/nested document
structures (e.g., a parent document with multiple child documents) using
Whoosh's parent-child relationship features.

## Defining Nested Documents

You can index parent documents that contain child documents by using a
parent field and child fields:

```python
from whoosh import fields, index

schema = fields.Schema(
    type=fields.ID(sortable=True),
    title=fields.TEXT(stored=True),
    content=fields.TEXT,
    section_name=fields.ID,
    section_content=fields.TEXT,
)
```

The `type` field distinguishes parent documents from child documents.

## Indexing Nested Documents

Use `IndexWriter.add_all()` with a generator that yields parent and child
documents grouped together:

```python
writer = ix.writer()
writer.add_all([
    parent_doc,
    child_doc_1,
    child_doc_2,
    parent_doc_2,
    child_doc_3,
])
```

Parent documents have `type="parent"` and child documents have
`type="child"`.

## Searching Nested Documents

### Parent-Query Child-Search

Search within child documents and match their parents:

```python
from whoosh.query import Every, Term
from whoosh.sorting import NestedParent

# Match all parent documents
parents = NestedParent(Term("type", "parent"))
q = Every("section_content", "hello")
results = searcher.search(q, sortedby=parents)
```

### Child-Query Parent-Search

Search for parent documents whose children match:

```python
from whoosh.sorting import NestedChildren

# Match parent documents that have children matching the query
parent_results = searcher.search(child_query, groupedby=NestedChildren(parent_matcher, child_matcher))
```

## Parent-Child Relationships at Index Time

When writing documents, use the `parent` parameter to link children to
parents:

```python
writer.add_document(type="parent", title="Chapter 1", _key="chapter1")
writer.add_document(type="child", section_name="Section 1.1",
                    section_content="...", parent="chapter1")
writer.add_document(type="child", section_name="Section 1.2",
                    section_content="...", parent="chapter1")
```

## Accessing Nested Results

To retrieve child matches alongside parent results, use the `expand` method
on the results:

```python
results = searcher.search(parent_query)
expanded = results.expand_child("section")
```

## Nested Faceting

Combine parent-child relationships with faceting using `NestedParent` and
`NestedChildren` as facets:

```python
parent_facet = NestedParent(FieldFacet("type"))
results = searcher.search(query, groupedby=parent_facet)
```

## Performance Considerations

- Parent-child joins are more expensive than flat document searches
- Use `childperm` searcher option to limit the number of permutations
  examined
- Consider whether hierarchical structure is needed at query time, or
  whether documents can be flattened during indexing


## DOCUMENT: Query

# Query Language

Whoosh-NG provides a powerful query language similar to Lucene's, as well as a programmatic query API.

## QueryParser

The `QueryParser` converts a query string into a query tree:

```python
from whoosh.qparser import QueryParser

# Parse a query for a specific field
qp = QueryParser("content", schema)
query = qp.parse("hello world")
```

## Query Syntax

### Basic Terms

```
hello                    # Single term
hello world              # Multiple terms (default AND)
hello OR world           # Explicit OR
"hello world"            # Phrase
```

### Field-Specific

```
title:python             # Search only in title field
title:"Python Tutorial"  # Phrase in specific field
```

### Boolean Operators

```
python AND whoosh
python OR whoosh
python AND NOT java
python AND (whoosh OR lucene)
```

### Prefix and Wildcard

```
pyth*                    # Prefix query
pyth?n                   # Single character wildcard
```

### Range Queries

```
date:[2020 TO 2025]
price:[10 TO 50]
rating:[4.0 TO *]        # Open-ended range
```

### Fuzzy Search

```
python~2                 # Edit distance <= 2
lucene~1                 # Approximate match
```

### Proximity Search

```
"hello world"~5          # Within 5 terms
```

### Boosting

```
python^2.0 whoosh        # Boost python by 2x
(title:python)^3 content:python  # Boost title matches
```

## Query Classes

You can build queries programmatically:

```python
from whoosh.query import *

# Simple term
q = Term("content", "python")

# Multiple terms (AND)
q = And([Term("content", "python"), Term("content", "whoosh")])

# Multiple terms (OR)
q = Or([Term("content", "python"), Term("content", "lucene")])

# Phrase
q = Phrase("content", ["hello", "world"])

# Range
q = NumericRange("price", 10, 50)
q = DateRange("date", datetime(2020,1,1), datetime(2025,1,1))

# Prefix
q = Prefix("content", "pyth")

# Wildcard
q = Wildcard("content", "pyth?n")

# Fuzzy
q = FuzzyTerm("content", "python", maxdist=2)

# Boost
q = Boost(Term("title", "python"), 2.0) & Term("content", "python")
```

## QueryParser Plugins

Extend query parsing with plugins:

```python
from whoosh.qparser import QueryParserPlugin

class MyPlugin(QueryParserPlugin):
    def __init__(self):
        pass

    def evaluate(self, env, signode):
        # Custom evaluation logic
        return Term("custom_field", signode.content)
```

## Multifield Search

Search multiple fields with different boosts:

```python
from whoosh.qparser import MultifieldParser

qp = MultifieldParser(
    ["title", "content", "tags"],
    schema,
    fieldboosts={"title": 2.0, "tags": 1.5}
)
q = qp.parse("python search")
```

## Default Operator

```python
from whoosh.qparser import QueryParser, OrGroup

# Default AND
qp = QueryParser("content", schema)

# Default OR
qp = QueryParser("content", schema, group=OrGroup)
```

## Escaping Special Characters

```
title\:python              # Literal colon
path\:\/\/example          # Escape special chars
```

## Regex Queries

```
content:/p[ya]thon/        # Regex match
```

## Advanced: Custom Queries

```python
from whoosh.query import Query

class CustomQuery(Query):
    def __init__(self, fieldname, text):
        self.fieldname = fieldname
        self.text = text

    def __repr__(self):
        return f"CustomQuery({self.fieldname!r}, {self.text!r})"

    def __hash__(self):
        return hash((self.fieldname, self.text))

    def __eq__(self, other):
        return (
            isinstance(other, CustomQuery)
            and self.fieldname == other.fieldname
            and self.text == other.text
        )

    def __ne__(self, other):
        return not self.__eq__(other)

    def matcher(self, searcher, context=None):
        # Return a custom matcher
        return CustomMatcher(searcher, self)
```


## DOCUMENT: Quickstart

# Quick Start

## Installation

```bash
pip install whoosh-ng
uv pip install whoosh-ng
```

## Basic Example

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(id=ID(stored=True), content=TEXT())
ix = index.create_in("indexdir", schema)

with ix.writer() as w:
    w.add_document(id="1", content="hello world")
    w.add_document(id="2", content="goodbye world")

with ix.searcher() as s:
    results = s.search("world")
    for hit in results:
        print(hit["id"], hit.score)
```

## With Plugins

```bash
pip install whoosh-ng[vector,autocomplete,api]
```

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin

PluginManager.load_plugins()
```

## Data Sources

```bash
pip install whoosh-ng
```

```python
import sqlite3
from whoosh_modern.data_sources import SQLSource
from whoosh_modern.views import SearchView

# Use existing benchmark data
conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
)

view = SearchView(name="reuters", source=source)
ix = view.build("indexdir")
```


## DOCUMENT: Schema

# Schema Design

How to model documents with Whoosh-NG fields.

## Field types

| Type | Searchable | Stored |
|------|------------|--------|
| TEXT | Yes | Optional |
| ID | Yes | Optional |
| KEYWORD | Yes | Optional |
| STORED | No | Yes |
| NUMERIC | Yes | Optional |
| DATETIME | Yes | Optional |
| BOOLEAN | Yes | Optional |
| VectorField | Provider | Optional |

## Building a schema

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, STORED, NUMERIC, BOOLEAN, VectorField

schema = Schema(
    title=TEXT(stored=True),
    slug=ID(stored=True, unique=True),
    content=TEXT,
    tags=KEYWORD(lowercase=True, commas=True),
    published=NUMERIC(int, stored=True),
    featured=BOOLEAN(stored=True),
    embedding=VectorField(dimensions=384, metric="cosine")
)
```

## Multi-value fields

Pass lists for multiple values.

```python
writer.add_document(
    title="Multi-tag post",
    tags=["whoosh", "python", "search"],
    content="..."
)
```

## Per-field boost

Boost fields at write time.

```python
writer.add_document(
    title="Breaking News",
    title_boost=3.0,
    content="..."
)
```

## SchemaBuilder

```python
from whoosh.fields import SchemaBuilder, TEXT, ID, NUMERIC

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("path", ID(stored=True, unique=True))
    .field("rating", NUMERIC(float, stored=True))
    .build()
)
```

## Modifying fields

```python
writer.add_document(
    title="Multi-tag post",
    tags=["whoosh", "python", "search"],
    content="..."
)
```

## Search Models

Whoosh-NG can auto-map Python models (dataclasses, Pydantic, SQLAlchemy, SQLModel, msgspec) to a Whoosh `Schema` using `ModelIndex`.

### Level 1: Auto-mapping

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Book:
    title: str
    count: int
    tag: str | None = None

idx = ModelIndex(Book)
schema = idx.schema
```

`ModelIndex` inspects type annotations and maps them to Whoosh fields:

| Python type | Whoosh field |
|-------------|--------------|
| `str` | `TEXT` |
| `int` / `float` | `NUMERIC` |
| `bool` | `BOOLEAN` |
| `datetime` / `date` | `DATETIME` |
| `Decimal` | `NUMERIC(int, decimal_places=2)` |
| `Enum` | `KEYWORD` |
| `bytes` | `KEYWORD` (hex-encoded) |
| `list[str]` | `KEYWORD` |
| `Optional[T]` | mapped type or `STORED` |

ID fields are auto-detected: explicit `SearchOptions(id=True)` > field named `id`/`ID`/`_id` > first `str` field.

### Level 2: Explicit options

Use `SearchField` to override defaults:

```python
from whoosh_modern.models import SearchField, SearchOptions

class Book:
    title: str = SearchField(fulltext=True, stored=True, analyzer="Simple")
    count: int = SearchField(sortable=True)
    tags: list[str] = SearchField(multi=True)
```

### Level 3: Annotated types

Use `Annotated` to attach metadata directly to annotations:

```python
from typing import Annotated
from whoosh_modern.models import SearchField

class Book:
    title: Annotated[str, SearchField(fulltext=True, stored=True)]
```

### Integrations

#### Dataclass

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Article:
    title: str
    body: str
    published: datetime.datetime

idx = ModelIndex(Article)
```

#### Pydantic v2

```python
from pydantic import BaseModel
from whoosh_modern.models import register_model

class Article(BaseModel):
    title: str
    body: str
    published: datetime.datetime

    # Per-field search metadata via json_schema_extra
    model_config = {"json_schema_extra": {"search": {"fulltext": True}}}

idx = register_model(Article)
```

#### SQLAlchemy

```python
from sqlalchemy import Column, Integer, String, DateTime
from sqlalchemy.orm import DeclarativeBase
from whoosh_modern.models import register_model

class Base(DeclarativeBase):
    pass

class Article(Base):
    __tablename__ = "articles"
    id = Column(Integer, primary_key=True)
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    published = Column(DateTime, info={"search": {"sortable": True}})

idx = register_model(Article)
```

#### SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_model

class Article(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    published: datetime.datetime

idx = register_model(Article)
```

#### msgspec

```python
import msgspec
from whoosh_modern.models import register_model

class Article(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    published: datetime.datetime

idx = register_model(Article)
```

### Converting instances

```python
doc = idx.to_whoosh_document(book_instance)
writer.add_document(**doc)
```

`to_whoosh_document` handles:
- dataclass: `dataclasses.fields()` iteration
- Pydantic/SQLModel: `model_fields` iteration
- SQLAlchemy: `__mapper__.columns` iteration
- Enum values converted to `.value`
- `bytes` converted to hex string

## Best practices

1. **Minimal**: Only index what you search
2. **STORED sparingly**: Increases index size
3. **Unique fields**: Use `unique=True` for identifiers
4. **Field boost**: Boost important fields at schema level
5. **TEXT options**: Disable `phrase` if you don't need phrase search
6. **ID field**: Let `ModelIndex` auto-detect or explicitly mark with `SearchOptions(id=True)`


## DOCUMENT: Searching

# Searching

This guide covers executing searches, working with results, scoring, sorting, and filtering.

## Basic Search

```python
from whoosh.qparser import QueryParser

qp = QueryParser("content", ix.schema)
query = qp.parse("hello world")

with ix.searcher() as searcher:
    results = searcher.search(query)
    for hit in results:
        print(hit["title"], hit.score)
```

## The Searcher

The `Searcher` is the main interface for reading the index. It is lightweight and supports context management:

```python
# Always use context manager when possible
with ix.searcher() as searcher:
    results = searcher.search(query)

# Or manage manually
searcher = ix.searcher()
try:
    results = searcher.search(query)
finally:
    searcher.close()
```

### Searcher Options

```python
searcher = ix.searcher(
    weighting=None,       # Custom weighting model
    childperm=None,       # Permutations for nested documents
    fromindex=None        # Source index for cached readers
)
```

## QueryParser

Convert query strings into query objects:

```python
from whoosh.qparser import QueryParser, OrGroup

# Default: AND between terms
qp = QueryParser("content", schema)
q = qp.parse("hello world")  # Equivalent to: content:hello AND content:world

# Change default operator
qp = QueryParser("content", schema, group=OrGroup)
q = qp.parse("hello world")  # Equivalent to: content:hello OR content:world
```

## Search Methods

### search()

```python
results = searcher.search(
    query,
    limit=10,           # Max results (None for all)
    sortedby=None,      # Sort key(s)
    reverse=False,      # Reverse sort order
    terms=False,        # Collect matched terms
    filter=None,        # Allow only these docnums
    mask=None,          # Exclude these docnums
    collapse=None,      # Collapse facet
    collapse_limit=1    # Max docs per collapse key
)
```

### search_page()

```python
# Get page 1, 10 results per page (default)
results = searcher.search_page(query, 1)

# Get page 3, 20 results per page
results = searcher.search_page(query, 3, pagelen=20)
```

### search_with_collector()

For advanced result collection:

```python
from whoosh.collectors import FacetCollector

collector = FacetCollector(facets=[sorting.FieldFacet("date")])
searcher.search_with_collector(query, collector)
```

## Results Object

`Results` acts like a list of matched documents:

```python
results = searcher.search(query)

# Slice support
first_five = results[0:5]

# Length (may trigger recount)
total = len(results)

# Scored length (usually what was actually returned)
scored = results.scored_length()

# Iteration
for hit in results:
    print(hit["title"], hit.score)
```

### Hit Object

```python
for hit in results:
    # Stored fields
    title = hit["title"]
    path = hit["path"]

    # Score
    print(hit.score)

    # Highlighting
    highlights = hit.highlights("content", top=3)

    # Matched terms (if terms=True was used)
    if results.has_matched_terms():
        print(hit.matched_terms())
```

## Scoring

The default scoring model is BM25F:

```python
from whoosh import scoring

with ix.searcher(weighting=scoring.BM25F()) as s:
    results = s.search(query)
```

### Custom Scoring

```python
class MyScorer(scoring.WeightingModel):
    def scorer(self, searcher, fieldname, text, qf=1):
        return MyCustomScorer(searcher, fieldname, text, qf)

with ix.searcher(weighting=MyScorer()) as s:
    results = s.search(query)
```

## Sorting

Sort by a field or facet:

```python
from whoosh import sorting

# Sort by a single field
results = searcher.search(query, sortedby="date")

# Reverse sort
results = searcher.search(query, sortedby="date", reverse=True)

# Multi-field sort
results = searcher.search(query, sortedby=[
    sorting.FieldFacet("category"),
    sorting.ScoreFacet()
])
```

## Faceting

Used for grouping results:

```python
from whoosh import sorting

facet = sorting.FieldFacet("category")
with searcher.all_features() as features:
    facets = features.facet(facet)
    for cat, count in facets.most_common():
        print(f"{cat}: {count}")
```

## Filtering and Masking

```python
# Only show documents matching a subquery
filter_q = Term("published", True)
results = searcher.search(query, filter=filter_q)

# Exclude documents
mask_q = Term("draft", True)
results = searcher.search(query, mask=mask_q)
```

## Collapsing

Remove duplicates or limit per-group:

```python
from whoosh import sorting

# Collapse by hostname, keep top 3 per host
results = searcher.search(
    query,
    collapse=sorting.FieldFacet("hostname"),
    collapse_limit=3
)

# Collapse ordering (keep highest rated per type)
results = searcher.search(
    query,
    sortedby=sorting.FieldFacet("price", reverse=True),
    collapse=sorting.FieldFacet("type"),
    collapse_order=sorting.FieldFacet("rating", reverse=True)
)
```

## Highlighting

Get highlighted snippets for query terms:

```python
results = searcher.search(query, terms=True)

for hit in results:
    print(hit.highlights("content", top=2))

# Custom fragmenter
from whoosh.highlight import highlight

fragments = hit.highlights(
    "content",
    top=3,
    fragmenter=...,
    formatter=...
)
```

## Time-Limited Searches

```python
from whoosh.collectors import TimeLimitCollector

with ix.searcher() as s:
    c = s.collector(limit=None)
    tlc = TimeLimitCollector(c, timelimit=5.0)
    try:
        s.search_with_collector(query, tlc)
    except TimeLimit:
        print("Search aborted: too slow")
    results = tlc.results()
```

## Combining Results

```python
# Run two queries
best_bet_results = s.search(best_bet_query, limit=5)
main_results = s.search(main_query, limit=10)

# Merge: duplicates go to top, then append rest
best_bet_results.upgrade_and_extend(main_results)
```


## DOCUMENT: Sorting

# Sorting

The `whoosh.sorting` module provides facets and sort-key computation for ordering and grouping search results.

## Quick start

```python
from whoosh import sorting

# Sort by a field
results = searcher.search(query, sortedby="date")

# Sort descending
results = searcher.search(query, sortedby=sorting.FieldFacet("price", reverse=True))
```

For the full API reference, see [Sorting API](/api/sorting).


## DOCUMENT: Translation Status

# Translation Completion Tracking

- [x] EN quickstart
- [x] EN guides
- [x] EN API pages
- [x] EN examples
- [x] FR quickstart
- [x] FR guides
- [x] FR API pages
- [x] FR examples


## DOCUMENT: Autocomplete

# Autocomplete with Whoosh-NG

Whoosh-NG provides autocomplete functionality through the `whoosh_modern.autocomplete` module.

## Install

```bash
pip install "whoosh-ng[autocomplete]"
```

## Schema with Keyword Field for Terms

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, KEYWORD

schema = Schema(
    title=TEXT(stored=True),
    tags=KEYWORD(stored=True, commas=True),
)

ix = index.create_in("autocomplete_index", schema)
```

## Index Documents

```python
with ix.writer() as w:
    w.add_document(title="Python Programming", tags="python,programming,language")
    w.add_document(title="JavaScript Basics", tags="javascript,programming,web")
    w.add_document(title="Machine Learning", tags="ml,ai,data-science")
    w.add_document(title="Deep Learning", tags="ml,ai,neural-networks")
    w.commit()
```

## Basic Autocomplete

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create an autocomplete provider (supports "inverted" provider type)
provider = create_autocomplete("inverted")

# Add phrases to index
provider.add(["python", "programming", "javascript", "machine learning", "deep learning"])

# Search for suggestions
hits = provider.search("py", limit=5)
for hit in hits:
    print(hit.text, hit.score)
# Output: python 1.5, programming 0.2
```

## Real-time Suggestion Endpoint

```python
from fastapi import FastAPI
from whoosh_modern.autocomplete import create_autocomplete

app = FastAPI()
provider = create_autocomplete("inverted")

# Populate provider with terms from your index
# (typically done during indexing)
provider.add(["python", "programming", "javascript", "machine learning"])

@app.get("/suggest")
async def suggest(q: str, limit: int = 5):
    hits = provider.search(q, limit=limit)
    return {"suggestions": [hit.text for hit in hits]}
```

## Key Points

- Install with `pip install whoosh-ng[autocomplete]`.
- Use `KEYWORD` fields to store multi-value tags/terms.
- Use `create_autocomplete("inverted")` to create a provider.
- The `InvertedIndexAutocomplete` provider supports prefix matching with scoring.
- Each result is an `AutocompleteHit` with `text` and `score` attributes.


## DOCUMENT: Basic Indexing

# Basic Indexing

Examples for indexing documents in Whoosh‑NG. Each section is a self-contained, **runnable** script.

> **Real-world scenario**: You're building a blog search engine. You have a CSV file
> of articles (`blog_posts.csv`) with `title`, `url`, `tags`, and `body` columns.

## 1. Define a Production Schema

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC, DATETIME
from datetime import datetime

# Stored=True keeps the field value in the index so you can retrieve it
# in search results without querying an external DB.
schema = Schema(
    doc_id=ID(stored=True, unique=True),     # primary key
    title=TEXT(stored=True),                  # full-text searchable + retrievable
    url=ID(stored=True),                      # stored only, no full-text analysis
    tags=KEYWORD(stored=True, commas=True),   # multi-value: "python,search,guide"
    body=TEXT(stored=True, phrase=True),      # searchable text with phrase queries
    published_at=DATETIME(stored=True, sortable=True),
    word_count=NUMERIC(int, stored=True),
)
```

## 2. Build an Index from a CSV File

```python
import csv
import shutil
from whoosh import index

# Clean prior index (development only!)
shutil.rmtree("blog_index", ignore_errors=True)
ix = index.create_in("blog_index", schema)

# Simulate a CSV file with blog post data
# blog_posts.csv:
#   doc_id,title,url,tags,published_at,word_count,body
#   1,Building a Search Engine,/posts/1,python,search,2024-01-15,1200,"Learn how to build..."
#   2,Python Tips,/posts/2,python,tips,2024-02-20,800,"Ten tips for Python..."

with open("blog_posts.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    with ix.writer() as writer:
        for row in reader:
            writer.add_document(
                doc_id=row["doc_id"],
                title=row["title"],
                url=row["url"],
                tags=row["tags"],
                published_at=datetime.fromisoformat(row["published_at"]),
                word_count=int(row["word_count"]),
                body=row["body"],
            )
        writer.commit()
```

## 3. Incremental Update — Re-index Modified Documents

```python
# Suppose your CMS tells you which posts were updated since the last sync
updated_posts = [
    {"doc_id": "1", "title": "Building a Search Engine (Updated)", "body": "Updated content..."},
    {"doc_id": "3", "title": "New Post", "body": "Fresh content..."},
]

with ix.writer() as writer:
    for post in updated_posts:
        writer.update_document(
            doc_id=post["doc_id"],
            title=post["title"],
            url=f"/posts/{post['doc_id']}",
            tags="python,search",
            published_at=datetime(2024, 6, 1),
            word_count=len(post["body"].split()),
            body=post["body"],
        )
    writer.commit()
```

## 4. Delete Documents by Term

```python
from whoosh.query import Term

# Remove a post by its unique doc_id
with ix.writer() as writer:
    writer.delete_by_term("doc_id", "3")
    writer.commit()
```

## 5. Bulk Insert for Large Datasets (10k+ Documents)

```python
from whoosh.writing import BufferedWriter

# Use BufferedWriter for high-throughput indexing.
# It buffers documents and commits in batches.
buffered = BufferedWriter(ix, period=60, limit=500)

try:
    for doc in large_dataset:  # your generator/list of dicts
        with buffered:
            buffered.add_document(
                doc_id=doc["doc_id"],
                title=doc["title"],
                url=doc["url"],
                tags=",".join(doc["tags"]),
                published_at=doc["published_at"],
                word_count=doc["word_count"],
                body=doc["body"],
            )
finally:
    buffered.close()
```

## 6. Run a Search on the Indexed Data

```python
from whoosh.qparser import QueryParser

ix = index.open_dir("blog_index")

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    q = qp.parse("search engine")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"Title: {hit['title']}")
        print(f"URL:   {hit['url']}")
        print(f"Score: {hit.score:.3f}")
        print(f"Snippet: {hit.highlights('body')}")
        print("---")


## DOCUMENT: Data Sources

# Data Sources

Whoosh-NG provides a flexible data source layer for indexing documents from SQL databases, REST APIs, GraphQL APIs, file-based formats, and custom data providers.

## DataSource Protocol

All data sources implement the `DataSource` protocol, which defines the interface for querying, schema discovery, and metadata retrieval.

```python
from whoosh_modern.data_sources import DataSource

class DataSource(Protocol):
    @property
    def name(self) -> str: ...

    def discover_schema(self) -> Schema: ...
    def iter_documents(self) -> Iterator[Document]: ...
    def stream_batches(self, batch_size: int = 1000) -> Iterator[list[dict[str, Any]]]: ...
    def health_check(self) -> bool: ...
```

### Capability Protocols

| Protocol | Description |
|----------|-------------|
| `DataSource` | Base protocol: name, schema, iteration, metadata |
| `IncrementalDataSource` | Supports `iter_changes(since)` |
| `AsyncDataSource` | Async document streaming via `aiter_documents()` |
| `RefreshableDataSource` | `refresh()` support |
| `CountableDataSource` | `document_count()` |
| `MetadataDataSource` | `metadata()` |
| `ObservableDataSource` | Observer callbacks for document changes |

---

## SQLSource

`SQLSource` connects to SQL databases and yields documents from query results with automatic connection pooling.

### Basic Usage

```python
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("mydb.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM products",
)

# Discover schema from result-set metadata
schema = source.discover_schema()

# Iterate documents
for doc in source.iter_documents():
    print(doc["title"], doc["price"])

# Get metadata
meta = source.metadata()
# {"type": "sql", "query": "SELECT * FROM products", ...}

# Get document count
count = source.document_count()
```

### Connection Pooling (SQLSource)

Connection pooling is supported via `pool_size` for long-running processes:

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(
    connection="sqlite:///mydb.db",  # URL or connection object
    query="SELECT * FROM products",
    pool_size=10,          # Max connections in pool
)
```

### GROUP BY Aggregation

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT category, COUNT(*) as doc_count,
               AVG(price) as avg_price
        FROM products
        GROUP BY category
    """,
)

# Each aggregated row becomes a document
for doc in source.iter_documents():
    print(doc["category"], doc["doc_count"], doc["avg_price"])
```

### JOINs with Column Aliases

`SQLSource` uses result-set metadata for schema discovery. Always use column aliases in JOINs.

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT
            p.id AS product_id,
            p.name AS product_name,
            c.name AS category_name
        FROM products p
        JOIN categories c ON p.category_id = c.id
    """,
)
```

### Incremental Sync

```python
from datetime import datetime

source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

# Get documents changed since a timestamp
for doc in source.iter_changes(since=datetime(2025, 1, 1)):
    print(doc["id"], doc["updated_at"])
```

### SQLAlchemySource

For SQLAlchemy users, use `SQLAlchemySource` which supports engine-based connection management:

```python
from whoosh_modern.data_sources.sqlalchemy_ds import SQLAlchemySource
from sqlalchemy import create_engine

engine = create_engine("postgresql://user:pass@localhost/mydb")
source = SQLAlchemySource(
    engine=engine,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

schema = source.discover_schema()
```

### PeeweeSource

For Peewee ORM users:

```python
from whoosh_modern.data_sources.peewee_ds import PeeweeSource
from peewee import SqliteDatabase

db = SqliteDatabase("mydb.db")
source = PeeweeSource(
    database=db,
    model=MyArticleModel,
    fields=["id", "title", "content"],
)

schema = source.discover_schema()
```

### TortoiseSource

For Tortoise ORM users (async):

```python
from whoosh_modern.data_sources.tortoise_ds import TortoiseSource

source = TortoiseSource(
    model="myapp.models.Article",
    fields=["id", "title", "content"],
)

schema = source.discover_schema()
count = source.document_count()
```

---

## RESTSource

`RESTSource` fetches documents from REST APIs with pagination and authentication.

### Basic Usage

```python
from whoosh_modern.data_sources.rest import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    method="GET",
    headers={"Authorization": "Bearer your_token"},
    pagination="page",
    page_size=50,
)

# Discover schema from first page
schema = source.discover_schema()

# Iterate all documents (handles pagination automatically)
for doc in source.iter_documents():
    print(doc["name"], doc["price"])

# Get document count
count = source.document_count()
```

### Pagination Strategies

| Strategy | Parameters | Behavior |
|----------|-----------|----------|
| `page` | `?page=N&size=M` | Fetches page N with M items per page |
| `offset` | `?offset=N&limit=M` | Fetches M items starting at N |
| `cursor` | `?cursor=XYZ&size=M` | Follows `next_cursor` in response |

```python
# Page-based pagination
source = RESTSource(
    url="https://api.example.com/articles",
    pagination="page",
    page_size=50,
)

# Offset-based pagination
source = RESTSource(
    url="https://api.example.com/records",
    pagination="offset",
    page_size=100,
)

# Cursor-based pagination
source = RESTSource(
    url="https://api.example.com/feed",
    pagination="cursor",
    page_size=100,
)
```

### Authentication

```python
# Bearer token via headers
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": "Bearer your_token"},
)

# API key via headers
source = RESTSource(
    url="https://api.example.com/data",
    headers={"X-API-Key": "your_api_key"},
)

# Basic auth via headers
import base64
creds = base64.b64encode(b"user:pass").decode()
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": f"Basic {creds}"},
)
```

### Document Path

For nested API responses, use `document_path` to extract documents:

```python
# API returns: {"data": {"results": [...]}}
source = RESTSource(
    url="https://api.example.com/api/v2/products",
    document_path="data.results",
    pagination="page",
)
```

---

## GraphQLSource

`GraphQLSource` fetches documents from a GraphQL API endpoint:

```python
from whoosh_modern.data_sources.graphql import GraphQLSource

source = GraphQLSource(
    url="https://api.example.com/graphql",
    query="""
        query GetProducts($limit: Int!, $offset: Int!) {
            products(limit: $limit, offset: $offset) {
                id
                name
                price
                description
            }
        }
    """,
    pagination="offset",
    page_size=100,
    headers={"Authorization": "Bearer your_token"},
)

schema = source.discover_schema()
for doc in source.iter_documents():
    print(doc["id"], doc["name"])
```

---

## File-Based Data Sources

Whoosh-NG supports indexing from various file formats with optimized readers.

### FastCSVSource

High-performance CSV reader with configurable encoding and delimiter:

```python
from whoosh_modern.data_sources.fast_csv import FastCSVSource

source = FastCSVSource(
    file_path="data/products.csv",
    id_field="id",
    incremental_field="updated_at",
    delimiter=",",
    encoding="utf-8",
)

schema = source.discover_schema()
count = source.document_count()
for doc in source.iter_documents():
    print(doc)
```

### JSONSource

Index from JSON files or JSON Lines (.jsonl) files:

```python
from whoosh_modern.data_sources.json import JSONSource

# JSON array file
source = JSONSource(file_path="data/products.json")

# JSON Lines file (one JSON object per line)
source = JSONSource(
    file_path="data/logs.jsonl",
    format="jsonl",
)

schema = source.discover_schema()
```

### ParquetSource

Index from Parquet files using pyarrow or pandas backend:

```python
from whoosh_modern.data_sources.parquet_ds import ParquetSource

source = ParquetSource(
    file_path="data/large_dataset.parquet",
    engine="pyarrow",  # or "pandas"
    batch_size=1000,
)

schema = source.discover_schema()
```

### PandasSource

Index directly from a pandas DataFrame:

```python
from whoosh_modern.data_sources.pandas_ds import PandasSource
import pandas as pd

df = pd.read_csv("data/products.csv")
source = PandasSource(dataframe=df)

schema = source.discover_schema()
```

### PolarsSource

Index from a Polars DataFrame (faster, lazy evaluation):

```python
from whoosh_modern.data_sources.polars_ds import PolarsSource
import polars as pl

df = pl.read_csv("data/products.csv")
source = PolarsSource(dataframe=df)

schema = source.discover_schema()
```

---

## DataSourceConfig

For programmatic configuration, use `DataSourceConfig` to define data source properties:

```python
from whoosh_modern.data_sources.config import DataSourceConfig

config = DataSourceConfig(
    type="sql",
    connection=conn,
    query="SELECT * FROM products",
    id_field="id",
    incremental_field="updated_at",
)

source = config.create()
schema = source.discover_schema()
```

### Config File Support

Data source configurations can be loaded from dictionaries:

```python
from whoosh_modern.data_sources.config import DataSourceConfig

# From dict
config = DataSourceConfig.from_dict({
    "type": "rest",
    "url": "https://api.example.com/v2/products",
    "pagination": "page",
    "page_size": 50,
})
source = config.create()
```

Supported `type` values: `sql`, `sqlalchemy`, `rest`, `csv`, `json`, `graphql`,
`pydantic`, `pandas`, `polars`, `parquet`, `peewee`, `tortoise`.

### Available Data Sources

| Class | Source Type | Dependencies |
|-------|------------|--------------|
| `SQLSource` | SQLite, PostgreSQL, MySQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | Any SQLAlchemy-supported DB | `sqlalchemy` |
| `RESTSource` | REST APIs | none (stdlib `urllib`) |
| `GraphQLSource` | GraphQL APIs | none (stdlib `urllib`) |
| `FastCSVSource` | CSV files | none |
| `JSONSource` | JSON/JSONL files | none |
| `ParquetSource` | Parquet files | `pyarrow` or `pandas` |
| `PandasSource` | pandas DataFrames | `pandas` |
| `PolarsSource` | Polars DataFrames | `polars` |
| `PeeweeSource` | Peewee ORM | `peewee` |
| `TortoiseSource` | Tortoise ORM | `tortoise-orm` |
| `PydanticSource` | Pydantic models | `pydantic` |


## DOCUMENT: Facets

# Facet Manager

`FacetManager` manages facet configuration for a Whoosh `Schema`. It auto-discovers facetable fields and supports manual overrides.

## Basic Usage

```python
from whoosh.fields import Schema, TEXT, NUMERIC, BOOLEAN
from whoosh_modern.facets import FacetManager, TermsFacet, RangeFacet

schema = Schema(
    title=TEXT(stored=True),
    category=TEXT(sortable=True),
    price=NUMERIC(),
    active=BOOLEAN(),
)

manager = FacetManager(schema)
```

## Auto-Discovery

FacetManager automatically identifies facetable fields:

| Whoosh Field Type | Facet Type |
|-------------------|------------|
| `TEXT`, `KEYWORD`, `BOOLEAN`, `ID` | `TermsFacet` |
| `NUMERIC` | `RangeFacet` |
| `DATETIME` | `DateRangeFacet` |

```python
# Auto-discovered facets
facets = manager.get_facets()
# {"title": TermsFacet(limit=100), "category": TermsFacet(limit=100), "price": RangeFacet(), "active": TermsFacet(limit=100)}
```

## Manual Override

```python
from whoosh_modern.facets import TermsFacet

manager.set_manual_override("category", {
    "type": "terms",
    "limit": 50,
})
manager.set_manual_override("price", {
    "type": "range",
    "buckets": ["0-10", "10-50", "50-100", "100+"],
})
```

## Inspection

```python
# Is a field facetable?
manager.is_facetable("category")   # True
manager.is_facetable("title")      # False

# Get config for a field
config = manager.get_facet_config("category")
# {"type": "terms", "limit": 50}

# Get all configs
all_configs = manager.get_all_facet_configs()

# Get statistics
stats = manager.get_facet_stats()
# {
#     "total_fields": 4,
#     "auto_discovered_facets": 2,
#     "manual_overrides": 1,
#     "total_facets_configured": 3,
#     "facet_fields": ["category", "price", ...]
# }
```


## DOCUMENT: Fastapi Search

# FastAPI Integration

A complete, runnable FastAPI service exposing Whoosh‑NG search via HTTP.

## 1. Install

```bash
pip install whoosh-ng[api] fastapi uvicorn
```

## 2. Create the index

```python
# setup_index.py
import json
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
)

ix = index.create_in("docs_index", schema)

with ix.writer() as w:
    for doc in json.load(open("documents.json")):
        w.add_document(
            id=doc["id"],
            title=doc["title"],
            content=doc["content"],
        )
    w.commit()
```

## 3. REST API Service

```python
# main.py
from fastapi import FastAPI, Query
from typing import Optional
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh_fastapi import create_app

ix = index.open_dir("docs_index")

# Option A: Use the helper
app = create_app(ix, prefix="/api/v1")

# Option B: Manual endpoints
# app = FastAPI(title="Document Search API", version="1.0.0")
#
# @app.get("/api/v1/health")
# async def health():
#     return {"status": "ok"}
#
# @app.post("/api/v1/search")
# async def search(q: str = Query(...), limit: int = 10):
#     with ix.searcher() as s:
#         parser = QueryParser("content", ix.schema)
#         results = s.search(parser.parse(q), limit=limit)
#         return {"hits": [dict(h) for h in results], "total": len(results)}
#
# @app.get("/api/v1/documents/{doc_id}")
# async def get_doc(doc_id: str):
#     with ix.searcher() as s:
#         from whoosh.query import Term
#         results = s.search(Term("id", doc_id))
#         if results:
#             return dict(results[0])
#         return {"error": "not found"}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

## 4. Run the server

```bash
uvicorn main:app --reload --port 8000
```

## 5. Test the API

```bash
# Health check
curl http://localhost:8000/api/v1/health

# Search
curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{"q": "python search"}'

# Get document by ID
curl http://localhost:8000/api/v1/documents/doc1
```

## 6. Bulk Indexing Endpoint

```python
# Add to main.py for dynamic indexing
from fastapi import FastAPI
from whoosh.writing import BufferedWriter

@app.post("/api/v1/index")
async def index_docs(docs: list[dict]):
    with BufferedWriter(ix, period=30, limit=50) as w:
        for doc in docs:
            w.add_document(**doc)
    return {"indexed": len(docs)}
```

## 7. Alternative: WhooshFastAPI Class

For more control, use the `WhooshFastAPI` class directly:

```python
from fastapi import FastAPI
from whoosh import index
from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import QueryParser
from whoosh_fastapi import WhooshFastAPI

app = FastAPI()

schema = Schema(title=TEXT(stored=True), content=TEXT)
ix = index.create_in("indexdir", schema)

api = WhooshFastAPI(ix)
api.register_search_endpoint("/search", "content")
api.register_index_endpoint("/documents", schema)
```

## Key points

- `create_app()` from `whoosh_fastapi` provides `/health`, `/search`, and `/autocomplete` endpoints.
- All blocking calls run off the event loop via `run_sync`.
- Use `BufferedWriter` for high-throughput indexing via POST.
- `WhooshFastAPI` class offers per-endpoint registration for custom integrations.


## DOCUMENT: Middleware Pipeline

# Middleware Pipeline

The middleware pipeline wraps operations with cross-cutting concerns: retry, logging, etc.

## Architecture

```python
from whoosh_modern.middleware import Middleware, MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

# Chain middlewares
pipeline = MiddlewarePipeline(
    RetryMiddleware(attempts=3, backoff="exponential"),
    LoggingMiddleware(),
)

# Execute an operation through the chain
result = pipeline.execute(my_operation)
```

## RetryMiddleware

```python
from whoosh_modern.middleware import RetryMiddleware

retry = RetryMiddleware(attempts=3, backoff="exponential")

def flaky_operation():
    # Will retry up to 3 times on exception
    return fetch_data()

wrapped_op = retry.wrap(flaky_operation)
result = wrapped_op()
```

Backoff strategies:
- `"exponential"`: 1s, 2s, 4s, 8s...
- `"linear"`: 1s, 2s, 3s, 4s...

## LoggingMiddleware

```python
from whoosh_modern.middleware import LoggingMiddleware
import logging

logger = logging.getLogger("benchmark")
logging_mw = LoggingMiddleware(logger=logger, level=logging.INFO)

tracked_op = logging_mw.wrap(lambda: fetch_data())
result = tracked_op()
# Logs: "Operation wrapped completed in 0.123s"
# On error: "Operation wrapped failed after 0.123s: <error>"
```

## CacheMiddleware

```python
from whoosh_modern.middleware import CacheMiddleware

cache = CacheMiddleware(maxsize=128)

cached_op = cache.wrap(expensive_query)
result1 = cached_op(args)  # cache miss
result2 = cached_op(args)  # cache hit

print(cache.stats)  # {"hits": 1, "misses": 1, "size": 1}
cache.clear()
```

## Custom Middleware

```python
from whoosh_modern.middleware import Middleware

class TimingMiddleware(Middleware):
    def __init__(self):
        self.timings = []

    def wrap(self, operation):
        def wrapped(*args, **kwargs):
            start = time.time()
            try:
                result = operation(*args, **kwargs)
                self.timings.append(time.time() - start)
                return result
            except Exception:
                raise
        return wrapped
```


## DOCUMENT: Middleware

# Middleware Examples

Practical examples for building and using Whoosh-NG middleware.

## 1. Logging Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        print(f"[SEARCH] Query: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.results is not None:
            print(f"[RESULTS] Found {len(context.results)} hits")
        return context
```

## 2. Metrics Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MetricsMiddleware(Middleware):
    def __init__(self) -> None:
        self._metrics = {}

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["documents_indexed"] = self._metrics.get("documents_indexed", 0) + 1
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["searches_executed"] = self._metrics.get("searches_executed", 0) + 1
        return context

    def get_metrics(self) -> dict:
        return dict(self._metrics)
```

## 3. Cache Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class SearchCacheMiddleware(Middleware):
    def __init__(self) -> None:
        self._cache = {}

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and str(context.query) in self._cache:
            context.metadata["_cached_result"] = self._cache[str(context.query)]
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and context.results is not None:
            self._cache[str(context.query)] = context.results
        return context
```

## 4. Applying Middleware to an Index

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_searcher

# Create middleware chain
chain = MiddlewareChain([
    LoggingMiddleware(),
    MetricsMiddleware(),
])

# Apply to a searcher
with ix.searcher() as base_searcher:
    searcher = apply_middleware_to_searcher(base_searcher, chain.middlewares)
    results = searcher.search(query)
```

## 5. Middleware Lifecycle

```python
class LifecycleMiddleware(Middleware):
    def startup(self, context):
        print("Middleware initialized")

    def shutdown(self, context):
        print("Middleware shutting down")

    def on_error(self, context, exc):
        print(f"Error: {exc}")
        raise exc
```

## Key Hooks

| Hook | Phase | Context |
|------|-------|---------|
| `startup` | Init | Called once on middleware init |
| `shutdown` | Cleanup | Called once on teardown |
| `before_index` | Indexing | Before document added |
| `after_index` | Indexing | After document added |
| `before_delete` | Deletion | Before document deleted |
| `after_delete` | Deletion | After document deleted |
| `before_search` | Search | Before query executed |
| `after_search` | Search | After results returned |
| `on_error` | Error | When exception occurs |
| `on_commit` | Commit | After writer.commit() |


## DOCUMENT: Movie Search

# Movie Search Application

A complete, runnable example showing how to build a small **movie search** application with Whoosh‑NG: schema design, indexing from a JSON dataset, faceted search, highlighting, and filtering.

## 1. Schema

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    director=TEXT(stored=True),
    genre=KEYWORD(stored=True, commas=True, scorable=True),
    year=NUMERIC(int, stored=True),
    synopsis=TEXT,
)
```

## 2. Index the dataset

```python
import json
import shutil
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    director=TEXT(stored=True),
    genre=KEYWORD(stored=True, commas=True, scorable=True),
    year=NUMERIC(int, stored=True),
    synopsis=TEXT,
)

# Clean and create new index
shutil.rmtree("movies", ignore_errors=True)
ix = index.create_in("movies", schema)

movies = json.load(open("movies.json"))  # list of dicts

with ix.writer() as w:
    for m in movies:
        w.add_document(
            id=str(m["id"]),
            title=m["title"],
            director=m["director"],
            genre=",".join(m["genres"]),
            year=m["year"],
            synopsis=m["synopsis"],
        )
    w.commit()
```

Example `movies.json`:

```json
[
  {
    "id": 1,
    "title": "Blade Runner",
    "director": "Ridley Scott",
    "genres": ["sci-fi", "thriller"],
    "year": 1982,
    "synopsis": "A replicant hunter questions humanity in a rain-soaked future."
  },
  {
    "id": 2,
    "title": "Inception",
    "director": "Christopher Nolan",
    "genres": ["sci-fi", "action"],
    "year": 2010,
    "synopsis": "A thief who steals corporate secrets through dream-sharing technology."
  }
]
```

## 3. Search with facets and highlighting

```python
from whoosh import index
from whoosh.qparser import MultifieldParser
from whoosh.sorting import FieldFacet

ix = index.open_dir("movies")

qp = MultifieldParser(["title", "synopsis", "director"], ix.schema)

with ix.searcher() as s:
    q = qp.parse("future")

    # Search with sorting by year descending, grouped by genre
    results = s.search(
        q,
        sortedby=FieldFacet("year", reverse=True),
        groupedby=FieldFacet("genre", allow_overlap=True),
        limit=20,
    )

    for hit in results:
        print(hit["title"], hit["year"], "|", round(hit.score, 2))
        print("  ", hit.highlights("synopsis"))

    # Show genre facets
    print("\nGenres:", results.groups("genre"))
```

## 4. Filtering

Find sci-fi movies after 1990:

```python
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh.query import Term, And, NumericRange

ix = index.open_dir("movies")
qp = QueryParser("synopsis", ix.schema)

with ix.searcher() as s:
    user_q = qp.parse("dream")
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("year", 1990, None),
    ])
    results = s.search(user_q, filter=filters)
    for hit in results:
        print(hit["title"], hit["year"])
```

## 5. Key takeaways

- `KEYWORD(commas=True)` stores multi-value fields that can be faceted.
- `MultifieldParser` searches multiple fields with optional boosts.
- `FieldFacet` enables faceted grouping and sorting.
- `hit.highlights()` returns highlighted snippets ready for display.


## DOCUMENT: Plugin Dev

# Plugin Development

Complete guide to building, registering, and testing custom Whoosh-NG plugins.

## 1. Plugin Base Class

All plugins inherit from `whoosh.plugins.base.Plugin`:

```python
from whoosh.plugins.base import Plugin

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"
    depends_on = []  # Other plugins this requires
    conflicts_with = []  # Plugins that conflict
    priority = 0  # Load order (higher = later)
    middleware = []  # List of middleware class names

    def register(self, manager):
        """Called when plugin is loaded. Register your handlers here."""
        manager.register("my_handler", MyHandler())

    def register_hooks(self):
        """Register hooks using hookimpl decorator."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            # Hook logic here
            pass

        register_hook("on_search", hookimpl(on_search))
```

## 2. Registering a Plugin

### Manual Registration

```python
from whoosh.plugins.manager import PluginManager

plugin = MyPlugin()
PluginManager.register(plugin)
```

### Auto-Discovery via Entry Points

In `pyproject.toml`:

```toml
[project]
name = "whoosh-ng-my-plugin"

[project.entry-points."whoosh_ng.plugins"]
my_plugin = "my_package.plugin:MyPlugin"
```

Auto-load all registered plugins:

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Uses 'whoosh.plugins' group by default
```

## 3. Provider Plugin Example

A provider plugin registers a new implementation for a registry:

```python
from whoosh.plugins.base import Plugin
from whoosh.registry import VectorRegistry

class MyVectorProvider:
    def search(self, query_vector, k=10):
        # Your vector similarity logic
        return [{"doc_id": "1", "score": 0.95}]

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"

    def register(self, manager):
        provider = MyVectorProvider()
        VectorRegistry.register("my_vector", provider, self.name)
```

## 4. Creating a Custom Field Type

```python
from whoosh.fields import FieldType, TEXT
from whoosh.formats import Postings

class TagField(FieldType):
    scorable = True
    stored = True
    indexed = True
    format = Postings()

    def __init__(self, stored=True, scorable=True):
        super().__init__(format=Postings(), analyzer=None,
                        scorable=scorable, stored=stored)
```

## 5. Testing Your Plugin

```python
import pytest
from whoosh.plugins.manager import PluginManager
from whoosh.registry.base import Registry

class TestMyPlugin:
    def test_register(self):
        plugin = MyPlugin()
        manager = PluginManager()
        plugin.register(manager)
        assert "my_handler" in manager._plugins

    def test_entry_point(self):
        """Test that entry point loading works."""
        manager = PluginManager()
        manager.register(MyPlugin())
        assert "my_plugin" in manager.list_enabled()

    def test_conflict_detection(self):
        plugin1 = MyPlugin()
        plugin2 = ConflictingPlugin()
        manager = PluginManager()
        manager.register(plugin1)
        assert manager.detect_conflicts("my_plugin", "conflicting_plugin")
```

## 6. Async Plugin Methods

Plugins support async methods:

```python
from whoosh.plugins.base import Plugin
from typing import Awaitable

class AsyncPlugin(Plugin):
    name = "async_plugin"
    version = "1.0.0"

    async def register(self, manager):
        # Async initialization
        await some_async_setup()

    def register_hooks(self):
        from whoosh.hooks import hookimpl

        @hookimpl
        async def on_search(request, response):
            # Async hook
            await log_search_async(request)
```

## 7. Plugin Manager API

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()

# Register a plugin instance
manager.register(MyPlugin())

# Enable/disable
manager.enable("my_plugin")
manager.disable("my_plugin")

# Check status
manager.list_plugins()   # All registered
manager.list_enabled()   # Enabled plugins

# Get plugin
plugin = manager.get("my_plugin")

# Version checking
manager.validate_version("my_plugin", "1.0.0")
```

## 8. Built-in Plugins

Whoosh-NG includes several built-in plugins:

- `whoosh_modern.vector` - Vector similarity search (NumPy provider)
- `whoosh_modern.autocomplete` - Inverted index autocomplete
- `whoosh_fastapi` - FastAPI REST endpoints

Load them:

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

PluginManager.load_plugins()  # Auto-loads entry points
# Or manually:
manager = PluginManager()
manager.register(VectorPlugin())
manager.register(AutocompletePlugin())
```


## DOCUMENT: Schema Discovery

# Schema Discovery

Schema discovery infers Whoosh `Schema` from data source results. It operates on **actual result metadata and sample documents**, not SQL syntax.

## How It Works

### From Result Sets

```python
from whoosh_modern.schema_discovery import SchemaDiscovery
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
cursor = conn.cursor()
cursor.execute("PRAGMA table_info(reuters_articles)")
columns = [(row[1], row[2]) for row in cursor.fetchall()]
# [("id", "INTEGER"), ("article_date", "TEXT"), ("headline", "TEXT"), ...]

schema = SchemaDiscovery.from_result_set(columns)
```

### From Sample Documents

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles LIMIT 10",
)

# Get a few documents
docs = list(source.iter_documents())[:10]

# Infer schema from document values
schema = SchemaDiscovery.from_sample(docs)
```

### Detect ID Field

```python
id_field = SchemaDiscovery.detect_id_field(dict(schema))
# Returns "id" if an ID field is found, otherwise None
```

### Optimized Schema Discovery

```python
# Infer schema with optimization rules applied
# - drops non-searchable TEXT fields
# - converts TEXT fields ending in "id" to ID
# - converts boolean-like TEXT fields to BOOLEAN
schema = SchemaDiscovery.from_sample_optimized(
    docs, searchable_text=["title", "content"]
)
```

## SQL Type Mapping

| SQL Type | Whoosh Field |
|----------|-------------|
| `VARCHAR`, `TEXT`, `CHAR`, `STRING_AGG` | `TEXT` |
| `INTEGER`, `BIGINT`, `SMALLINT`, `COUNT`, `SUM`, `AVG`, `MIN`, `MAX` | `NUMERIC` |
| `FLOAT`, `DOUBLE`, `DECIMAL` | `NUMERIC` |
| `BOOLEAN` | `BOOLEAN` |
| `DATE`, `TIMESTAMP` | `DATETIME` |
| `UUID` | `ID` |
| `JSON`, `ENUM` | `KEYWORD` |

## Duplicate Column Detection

`from_result_set` raises `SchemaDiscoveryError` on duplicate column names.

```python
columns = [
    ("id", "INTEGER"),
    ("headline", "TEXT"),
    ("id", "INTEGER"),  # Duplicate!
]

try:
    schema = SchemaDiscovery.from_result_set(columns)
except SchemaDiscoveryError as e:
    print(f"Duplicate column: {e.field}")
```

Use explicit SQL aliases to avoid duplicates:

```sql
SELECT
    p.id AS product_id,
    c.id AS category_id
FROM products p
JOIN categories c ON p.category_id = c.id
```


## DOCUMENT: Search Models

# Search Models

Examples for auto-mapping Python models to Whoosh schemas.

## Dataclass

```python
from dataclasses import dataclass
from whoosh.fields import Schema, TEXT, NUMERIC
from whoosh_modern.models import register_dataclass_model
import tempfile
import shutil

@dataclass
class Book:
    title: str
    year: int
    tags: list[str] | None = None

idx = register_dataclass_model(Book)
print(idx.schema)
```

## Pydantic

```python
from pydantic import BaseModel
from whoosh_modern.models import register_pydantic_model

class BookModel(BaseModel):
    title: str
    year: int
    tags: list[str] | None = None

idx = register_pydantic_model(BookModel)
schema = idx.schema
```

## SQLAlchemy

```python
from sqlalchemy import Column, Integer, String
from whoosh_modern.models import register_sqlalchemy_model

class BookSQL:
    __tablename__ = "book"
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    year = Column(Integer, info={"search": {"sortable": True}})

idx = register_sqlalchemy_model(BookSQL)
schema = idx.schema
```

## SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_sqlmodel_model

class Book(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    year: int

idx = register_sqlmodel_model(Book)
schema = idx.schema
```

## msgspec

```python
import msgspec
from whoosh_modern.models import register_msgspec_model

class Book(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    year: int

idx = register_msgspec_model(Book)
schema = idx.schema
```

## Indexing documents

```python
from whoosh import index

tmp = tempfile.mkdtemp()
ix = index.create_in(tmp, schema)

with ix.writer() as w:
    book = Book(title="Whoosh Guide", year=2024, tags=["python", "search"])
    doc = idx.to_whoosh_document(book)
    w.add_document(**doc)
    w.commit()
```

## Auto-indexing with AutoIndexer

```python
from whoosh_modern.models import AutoIndexer

auto = AutoIndexer(ix, on_error="raise")
auto.register(Book)

# Index a single instance
book = Book(title="New Book", year=2024, tags=["python"])
auto.index(book)

# Remove by ID
auto.remove(book)

# Async versions
await auto.index_async(book)
await auto.remove_async(book)
```

For SQLAlchemy models, `AutoIndexer` automatically hooks into `after_insert`, `after_update`, and `after_delete` events.

## Cleanup

```python
shutil.rmtree(tmp)
```


## DOCUMENT: Search View

# SearchView

`SearchView` integrates a `DataSource` with Whoosh indexing. It discovers
schema, validates the source, builds the index, and supports incremental
refresh and schema evolution.

## Basic Usage

```python
from whoosh_modern.views import SearchView
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("data/articles.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

view = SearchView(
    name="articles",
    source=source,
)

# Build index (discovers schema, validates, populates)
ix = view.build("indexdir")
```

## Full Reindex vs Incremental Refresh

```python
# Full reindex (clears and rebuilds the entire index)
count = view.reindex()

# Incremental refresh (only changed documents since last sync)
count = view.refresh()
```

## Validation

```python
# Run validation before building
results = view.validate()
for result in results:
    print(f"Level {result.level}: {'PASS' if result.passed else 'FAIL'}")
    for error in result.errors:
        print(f"  ERROR: {error}")
    for warning in result.warnings:
        print(f"  WARNING: {warning}")
```

Validation levels:
1. **Structural** — DataSource availability, schema detection
2. **Search** — Indexable fields, term vectors, searchable analyzers
3. **Performance** — Performance warnings (e.g., TEXT fields on large datasets)
4. **Runtime** — Sample iteration, type conformance

## Field Overrides

Customize field types after schema discovery:

```python
from whoosh.fields import TEXT, NUMERIC, DATETIME

view = SearchView(
    name="custom",
    source=source,
    fields={
        "title": TEXT(stored=True, phrase=False),
        "price": NUMERIC(int, sortable=True),
        "published": DATETIME(sortable=True),
    },
)
```

## Facets

Configure facet settings:

```python
view = SearchView(
    name="faceted",
    source=source,
    facets={
        "category": {"type": "terms", "limit": 50},
        "price": {"type": "range", "buckets": ["0-100", "100-500", "500+"]},
    },
)
```

## Middleware

Attach middleware to the search pipeline:

```python
from whoosh_modern.views import SearchView
from whoosh_modern.middleware import LoggingMiddleware, RetryMiddleware

view = SearchView(
    name="with_middleware",
    source=source,
    middleware=[
        RetryMiddleware(attempts=3, backoff="exponential"),
        LoggingMiddleware(),
    ],
)
```

## Strict Mode

```python
view = SearchView(
    name="strict",
    source=source,
    strict=True,  # Raise ValidationError on any validation failure
)
```

## Schema Evolution

Add new fields to an existing index without a full reindex:

```python
view = SearchView(name="articles", source=source)
view.build("indexdir")

# Add a new field
view.evolve_schema({
    "new_field": TEXT(stored=True),
})
```

### Schema Version Checking

```python
view = SearchView(name="articles", source=source, schema_version="2.1")
view.build("indexdir")

# Check if the stored schema version matches
if not view.check_schema_version():
    print("Schema version mismatch, consider reindexing")
```


## DOCUMENT: Search

# Search Examples

Real, runnable search examples with Whoosh‑NG. Each section is a self-contained
script you can copy into a `.py` file and run.

> **Real-world scenario**: You built a book‑catalogue index (see
> `docs/_en/examples/basic-indexing.md`). Below are the search patterns
> you'll need for a production‑ready book search page.

## Prerequisites

The examples assume an index exists at `book_index/` with this schema:

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, NUMERIC, DATETIME

schema = Schema(
    isbn=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    author=TEXT(stored=True),
    content=TEXT,
    genre=KEYWORD(stored=True, commas=True),
    published_year=NUMERIC(int, stored=True, sortable=True),
    rating=NUMERIC(float, stored=True, sortable=True),
)
```

## 1. Basic Search — "Find books about Python"

```python
from whoosh import index
from whoosh.qparser import QueryParser

ix = index.open_dir("book_index")

with ix.searcher() as s:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("python")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} by {hit['author']} (ISBN: {hit['isbn']}) — score={hit.score:.2f}")
```

## 2. Multi-field Search with Boosts

Search across `title`, `author`, and `content` simultaneously. Title matches
are boosted 3× so they rank higher:

```python
from whoosh.qparser import MultifieldParser

ix = index.open_dir("book_index")
qp = MultifieldParser(
    ["title", "author", "content"],
    ix.schema,
    fieldboosts={"title": 3.0, "author": 2.0, "content": 1.0},
)

q = qp.parse("clean code")

with ix.searcher() as s:
    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} — {hit['author']}")
```

## 3. Pagination — "Page 3 of search results"

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("machine learning")

with ix.searcher() as s:
    page = s.search_page(q, 3, pagelen=15)  # Page 3, 15 results per page

    print(f"Page {page.number} / {page.pagecount}  ({page.total} results total)")
    for hit in page:
        print(f"  {hit['title']}")
```

## 4. Sort and Filter — "High-rated sci-fi books after 2010"

```python
from whoosh.query import Term, And, NumericRange
from whoosh.sorting import FieldFacet, ScoreFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("space")

with ix.searcher() as s:
    # Filter: genre must be "sci-fi" AND year >= 2010
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("published_year", 2010, None),
    ])

    results = s.search(
        q,
        filter=filters,
        sortedby=FieldFacet("rating", reverse=True),  # highest-rated first
        limit=20,
    )
    for hit in results:
        print(f"{hit['title']} ({hit['published_year']}) — rating: {hit['rating']}")
```

## 5. Highlighting — "Show users where their query matched"

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("neural networks")

with ix.searcher() as s:
    results = s.search(q, limit=5)

    for hit in results:
        snippet = hit.highlights("content", top=2)  # show 2 best fragments
        print(f"{hit['title']}:")
        print(f"  {snippet}")
        print()
```

## 6. Date / Numeric Range Search — "Books published in 2023"

```python
from whoosh.query import NumericRange

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = NumericRange("published_year", 2023, 2023)
    results = s.search(q)
    print(f"{results.total} books published in 2023")
```

## 7. Prefix Search — "All books starting with 'Deep'"

```python
from whoosh.query import Prefix

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = Prefix("title", "Deep")  # titles starting with "Deep"
    results = s.search(q)
    for hit in results:
        print(hit["title"])
```

## 8. Faceted Search — "Group results by genre"

```python
from whoosh.sorting import FieldFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("programming")

with ix.searcher() as s:
    results = s.search(q, groupedby=FieldFacet("genre"))

    # Show top genres alongside results
    for genre, group in results.groups("genre").items():
        print(f"{genre}: {len(group)} hits")
```

## Key points

- `QueryParser` parses a string into a `Query` object.
- `MultifieldParser` searches multiple fields with optional per-field boosts.
- `search_page()` handles pagination automatically.
- `filter` restricts results without affecting relevance scores.
- `sortedby` sorts by field value or relevance score.
- `hit.highlights()` returns highlighted snippets ready for display.
- `groupedby` enables faceted result grouping.


## DOCUMENT: Validation

# Validation Framework

The validation framework runs checks against a data source before indexing. It provides 4 distinct validation levels with different failure modes.

## Validation Levels

| Level | Method | Purpose |
|-------|--------|---------|
| **Level 1** | `validate_structural(source)` | DataSource availability, schema detection |
| **Level 2** | `validate_search(schema)` | Indexable fields, analyzer compatibility |
| **Level 3** | `validate_performance(schema, source)` | Performance warnings (TEXT fields, etc.) |
| **Level 4** | `validate_runtime(source, sample_size)` | Sample iteration, type validation |

## Basic Usage

```python
from whoosh_modern.validation import ValidationFramework, ValidationResult
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(connection=conn, query="SELECT * FROM reuters_articles")

validator = ValidationFramework()

# Run all 4 validation levels
results: list[ValidationResult] = validator.validate(source)

for result in results:
    level_name = f"Level {result.level}"
    status = "PASS" if result.passed else "FAIL"
    print(f"{level_name}: {status}")
    for error in result.errors:
        print(f"  ERROR: {error}")
    for warning in result.warnings:
        print(f"  WARN: {warning}")
```

## Individual Level Validation

```python
# Level 1: Structural
errors = validator.validate_structural(source)

# Level 2: Search
from whoosh.fields import Schema
schema = source.discover_schema()
errors = validator.validate_search(schema)

# Level 3: Performance
warnings = validator.validate_performance(schema, source)

# Level 4: Runtime
errors = validator.validate_runtime(source, sample_size=100)
```

## Validation Results

```python
@dataclass
class ValidationResult:
    level: int
    passed: bool
    warnings: list[str]
    errors: list[str]
```


## DOCUMENT: Vector Search

# Vector Search with Whoosh‑NG

This example shows how to enable **semantic/vector search** using the optional `vector` extra. We index document embeddings and perform k-nearest neighbour (k-NN) search.

## 1. Install Optional Dependencies

```bash
pip install "whoosh-ng[vector]" numpy
```

## 2. Schema with a Vector Field

```python
from whoosh.fields import Schema, TEXT, ID, VECTOR
from whoosh.vector import VectorField

schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VECTOR(stored=True, dim=128),  # 128-dimensional embedding
)
```

## 3. Indexing Vectors

```python
import numpy as np
from whoosh import index

shutil.rmtree("vector_index", ignore_errors=True)
ix = index.create_in("vector_index", schema)

# Simulate embeddings (in practice, use a model like SentenceTransformer)
documents = [
    {"doc_id": "doc1", "title": "Python Basics", "content": "Learn Python programming fundamentals."},
    {"doc_id": "doc2", "title": "Advanced Python", "content": "Deep dive into Python decorators and metaclasses."},
    {"doc_id": "doc3", "title": "Data Science", "content": "Pandas and NumPy for data analysis."},
]

# Generate random embeddings for demo
np.random.seed(42)
embeddings = {d["doc_id"]: np.random.rand(128).astype(np.float32) for d in documents}

with ix.writer() as w:
    for doc in documents:
        w.add_document(
            doc_id=doc["doc_id"],
            title=doc["title"],
            content=doc["content"],
            embedding=embeddings[doc["doc_id"]].tobytes(),
        )
    w.commit()
```

## 4. Vector Search with NumpyProvider

```python
from whoosh_modern.vector import VectorField
from whoosh_modern.vector.numpy_provider import NumpyProvider
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager

# Register the vector plugin
VectorPlugin().register(PluginManager())

# Create provider and add vectors
provider = NumpyProvider()
for doc_id, vec in embeddings.items():
    provider.add([(doc_id, vec.tolist())])

# Search: find 2 most similar docs to a query vector
query_vec = embeddings["doc1"]  # use doc1's embedding as query
hits = provider.search(query_vec, k=2)

for hit in hits:
    print(f"doc_id={hit.doc_id}, score={hit.score:.3f}")
```

## 5. Using VectorField for Serialization

```python
from whoosh.vector import VectorField

vf = VectorField(dimension=128, name="embedding")

# Convert list to bytes for storage
values = [0.1, 0.2, 0.3, 0.4] + [0.0] * 124  # 128 values
raw = vf.vector_to_bytes(values)

# Restore from bytes
restored = vf.bytes_to_vector(raw)
print(restored == tuple(values))  # True
```

## 6. Key Takeaways

- Install with `pip install whoosh-ng[vector]` to get `whoosh_modern.vector`.
- `VECTOR` field stores raw bytes; use `VectorField` to convert to/from Python lists.
- `NumpyProvider` implements cosine similarity via dot product.
- Register the plugin via `VectorPlugin().register(manager)` or use `PluginManager.load_plugins()`.
- Use `filter_ids` in `provider.search()` to restrict to a subset of documents.


## DOCUMENT: Autocomplete Sprint D

# Autocomplete Providers

Module: `whoosh_modern.autocomplete`
Version: 2.0.0

The autocomplete module provides multiple provider strategies for query suggestion and type-ahead search. All providers implement a common interface so you can swap strategies at runtime. Providers are registered via the `AutocompleteRegistry` and loaded through entry points.

## Module Overview

```text
whoosh_modern.autocomplete
    ├── provider.py   # AutocompleteHit, AutocompleteProvider (Protocol)
    ├── ngram.py      # NGramProvider (character n-gram based)
    ├── edge_ngram.py # InvertedIndexAutocomplete (inverted index prefix matching)
    ├── fuzzy.py      # FuzzySuggestProvider (approximate matching via rapidfuzz)
    ├── factory.py    # create_autocomplete() factory
    └── plugin.py     # AutocompletePlugin (entry-point plugin)
```

## AutocompleteProvider (Base Class)

Located in `whoosh_modern.autocomplete.provider`:

```python
from whoosh_modern.autocomplete.provider import AutocompleteProvider, AutocompleteHit

class MyProvider(AutocompleteProvider):
    def add(self, phrases: Iterable[str]) -> None:
        """Add phrases to the provider's index."""
        ...

    def search(self, prefix: str, limit: int = 10) -> list[AutocompleteHit]:
        """Return autocomplete suggestions for the given prefix."""
        ...
```

### AutocompleteHit

A simple result object returned by providers:

```python
class AutocompleteHit:
    def __init__(self, text: str, score: float) -> None:
        self.text = text    # The matched phrase
        self.score = score  # Relevance score (higher = better)
```

## Built-in Providers

### InvertedIndexAutocomplete

Located in `whoosh_modern.autocomplete.edge_ngram`. Uses simple prefix matching against an in-memory list:

```python
from whoosh_modern.autocomplete.edge_ngram import InvertedIndexAutocomplete

provider = InvertedIndexAutocomplete()
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("py", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output:
# python (score: 0.45)
# pyramid (score: 0.43)
# pytorch (score: 0.43)
```

**Scoring**: Exact prefix matches get a 1.5x bonus; base score is `1.0 / (len(phrase) + 1)`.

### NGramProvider

Located in `whoosh_modern.autocomplete.ngram`. Builds a character n-gram index for fuzzy substring matching:

```python
from whoosh_modern.autocomplete.ngram import NGramProvider

provider = NGramProvider(n=3)
provider.add(["python programming", "java development", "rust language"])

hits = provider.search("pyt", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
```

**Parameters:**

| Parameter | Type | Default | Description                          |
|-----------|------|---------|--------------------------------------|
| `n`       | `int` | `3`     | Size of character n-grams            |

**How it works**: N-grams are extracted from each phrase (lowercased). During search, n-grams from the prefix are matched against the index. Phrases with more matching n-gram occurrences receive higher scores.

### FuzzySuggestProvider

Located in `whoosh_modern.autocomplete.fuzzy`. Uses `rapidfuzz` for approximate string matching (typos, partial matches):

```python
from whoosh_modern.autocomplete.fuzzy import FuzzySuggestProvider

# Requires: pip install whoosh-ng[fuzzy]
provider = FuzzySuggestProvider(max_distance=2, score_cutoff=50.0)
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("pythn", limit=5)  # Typo in "python"
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output: python (score: 0.95), ...
```

**Parameters:**

| Parameter       | Type  | Default  | Description                              |
|-----------------|-------|----------|------------------------------------------|
| `max_distance`  | `int` | `2`      | Maximum edit distance (unused by rapidfuzz directly, reserved for future use) |
| `score_cutoff`  | `float` | `50.0` | Minimum similarity score (0-100 scale)   |

**Note**: Requires `rapidfuzz` (`pip install whoosh-ng[fuzzy]`). Falls back to `ImportError` if not installed.

## Factory Function

Located in `whoosh_modern.autocomplete.factory`:

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create any provider by name
provider = create_autocomplete("inverted")   # InvertedIndexAutocomplete
provider = create_autocomplete("ngram", n=3) # NGramProvider with custom n
provider = create_autocomplete("fuzzy", max_distance=2, score_cutoff=60.0)
```

**Available providers:**

| Name        | Class                    | Optional Dependency |
|-------------|--------------------------|---------------------|
| `"inverted"`| `InvertedIndexAutocomplete` | None              |
| `"ngram"`   | `NGramProvider`          | None               |
| `"fuzzy"`   | `FuzzySuggestProvider`   | `rapidfuzz`        |

## Registering with the AutocompleteRegistry

Providers are registered into `whoosh.registry.AutocompleteRegistry` (a `Registry` instance):

```python
from whoosh.registry import AutocompleteRegistry
from whoosh_modern.autocomplete import create_autocomplete

# Register a provider
provider = create_autocomplete("ngram", n=3)
AutocompleteRegistry.register("ngram-suggester", provider, owner="my_app")

# Retrieve it later
suggester = AutocompleteRegistry.get("ngram-suggester")

# List all registered providers
print(AutocompleteRegistry.list_keys())
```

## AutocompletePlugin (Entry Point)

Located in `whoosh_modern.autocomplete.plugin`, this is the built-in plugin registered via the `whoosh_ng.plugins` entry-point group:

```python
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

# Automatically loaded by PluginManager.load_plugins()
# Registers "inverted" provider in AutocompleteRegistry
```

### Entry Point Declaration

In `pyproject.toml`:

```toml
[project.entry-points."whoosh_ng.plugins"]
whoosh_autocomplete = "whoosh_modern.autocomplete.plugin:AutocompletePlugin"
```

### Plugin Details

```python
class AutocompletePlugin(Plugin):
    name = "whoosh_autocomplete"
    version = "3.0.0"

    def register(self, manager):
        # Registers InvertedIndexAutocomplete as "inverted"
        AutocompleteRegistry.register(
            "inverted", create_autocomplete("inverted"), self.name
        )

    def register_hooks(self):
        # Registers an on_search hook (currently a no-op)
        from whoosh.hooks import hookimpl, register_hook
        register_hook("on_search", hookimpl(on_search))
```

## Usage Examples

### Basic Usage

```python
from whoosh_modern.autocomplete import create_autocomplete

# Create and populate a provider
provider = create_autocomplete("inverted")
provider.add([
    "python programming",
    "python tutorial",
    "java tutorial",
    "javascript framework",
])

# Search for suggestions
hits = provider.search("py", limit=3)
for hit in hits:
    print(f"{hit.text}: {hit.score:.3f}")
```

### Using Fuzzy Matching with Typo Tolerance

```python
from whoosh_modern.autocomplete import create_autocomplete

provider = create_autocomplete("fuzzy", score_cutoff=70.0)
provider.add(["python", "pytorch", "tensorflow", "keras"])

# Even with a typo, relevant suggestions are returned
hits = provider.search("pyton", limit=5)
for hit in hits:
    print(hit.text, hit.score)
```

### Using N-Gram Matching for Partial Words

```python
from whoosh_modern.autocomplete import create_autocomplete

# Use 3-grams for better substring matching
provider = create_autocomplete("ngram", n=3)
provider.add(["machine learning", "deep learning", "neural networks"])

# Finds phrases containing the n-grams of "machin"
hits = provider.search("machin", limit=5)
```

### Integration with Search

```python
from whoosh_modern.autocomplete import create_autocomplete

# Build the autocomplete provider
provider = create_autocomplete("inverted")
provider.add(["python", "java", "javascript", "go", "rust"])

# Use in a search endpoint
def suggest(prefix: str, limit: int = 5):
    hits = provider.search(prefix, limit=limit)
    return [{"text": h.text, "score": h.score} for h in hits]

# In your FastAPI/REST endpoint:
# GET /api/suggest?q=py&limit=5
# Response: [{"text": "python", "score": 0.45}, ...]
```

## Comparison of Providers

| Provider              | Matching       | Strengths                    | Weaknesses                | Dependency    |
|-----------------------|----------------|------------------------------|---------------------------|---------------|
| `inverted`            | Prefix         | Simple, fast, no deps        | No typo tolerance         | None          |
| `ngram`               | N-gram overlap | Substring matching, flexible | Slower than prefix        | None          |
| `fuzzy`               | Edit distance  | Typo tolerance, flexible     | Requires rapidfuzz        | `rapidfuzz`   |

## Installation

```bash
# Core autocomplete (inverted + n-gram)
pip install whoosh-ng

# With fuzzy matching
pip install whoosh-ng[fuzzy]

# Full modern analysis
pip install whoosh-ng[modern]
```

## See Also

- [Plugin System Guide](plugins-sprint-c.md) — Plugin registration and discovery
- [Middleware Guide](middleware-sprint-c.md) — Middleware pipeline integration
- [API: Modern](../api/modern.md) — Full API reference for autocomplete extensions


## DOCUMENT: Autocomplete

# Autocomplete

An optional edge-ngram style autocomplete layer for Whoosh-NG.

## Install

```bash
pip install whoosh-ng[autocomplete]
```

## Minimal index

```python
from whoosh.fields import Schema, TEXT, AutocompleteField

schema = Schema(
    title=TEXT(stored=True),
    query=AutocompleteField()
)

with ix.writer() as writer:
    writer.add_document(title="Python Quickstart", query="python quickstart")
    writer.commit()
```

## Query autocomplete

```python
from whoosh_modern.autocomplete import AutocompleteProvider

provider = AutocompleteProvider(ix, "query")
suggestions = provider.suggest("py", limit=5)
print(suggestions)  # ["python", "pyramid", ...]
```

## Modern Autocomplete Providers (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces multiple autocomplete provider strategies: `InvertedIndexAutocomplete`, `NGramProvider`, and `FuzzySuggestProvider`. For full details on creating, registering, and switching providers, see the [Autocomplete Providers Guide](autocomplete-sprint-d.md).


## DOCUMENT: Linguistics Sprint D

# Synonyms & Linguistics

Module: `whoosh_modern.linguistics.synonyms`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

The linguistics module provides a comprehensive synonym expansion engine and language-specific text analyzers. It integrates with the middleware pipeline to expand queries and documents with synonyms at both index time and query time.

## Module Overview

```text
whoosh_modern.linguistics
    ├── synonyms/
    │   ├── provider.py       # SynonymProvider protocol + StaticSynonymProvider
    │   ├── yaml_provider.py  # YAMLSynonymProvider
    │   ├── json_provider.py  # JSONSynonymProvider
    │   ├── store.py          # SQLiteSynonymStore
    │   ├── compiler.py       # SynonymCompiler
    │   ├── manager.py        # SynonymManager
    │   ├── middleware.py     # SynonymExpansionMiddleware
    │   └── languages.py      # LANG_SYNONYMS (FR/EN/DE/ES/IT)
    └── stemmers/
        └── __init__.py       # Language-specific analyzers (FR/EN/DE/ES/IT)
```

## Synonym Providers

### SynonymProvider (Protocol)

The base protocol that all synonym providers implement:

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]:
        """Return synonyms for the given word."""
        ...

    def add_synonym(self, word: str, synonyms: list[str]) -> None:
        """Add synonyms for the given word."""
        ...

    def remove_synonym(self, word: str, synonym: str) -> None:
        """Remove a synonym for the given word."""
        ...
```

### StaticSynonymProvider

In-memory provider backed by a dictionary:

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"],
})

print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### YAMLSynonymProvider

Loads synonyms from a YAML file:

```yaml
# synonyms.yaml
car:
  - automobile
  - vehicle
  - auto
house:
  - home
  - residence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

# Requires: pip install pyyaml
provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### JSONSynonymProvider

Loads synonyms from a JSON file:

```json
{
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("car"))
```

### SQLiteSynonymStore

Persistent synonym store backed by SQLite:

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")

# CRUD operations
store.add_synonym("car", ["automobile", "vehicle"])
print(store.get_synonyms("car"))  # ['automobile', 'vehicle']
store.remove_synonym("car", "automobile")
print(store.get_synonyms("car"))  # ['vehicle']
store.close()
```

### SynonymCompiler

Precompiles raw synonym data into a fast lookup format:

```python
from whoosh_modern.linguistics.synonyms import SynonymCompiler

compiler = SynonymCompiler({"car": ["automobile", "vehicle"]})
compiler.add("house", ["home", "residence"])
compiler.merge({"book": ["publication", "work"]})

compiled = compiler.compile()
print(compiled)
# {'car': ['automobile', 'vehicle'], 'house': ['home', 'residence'], 'book': ['publication', 'work']}
```

## SynonymManager

The `SynonymManager` is the high-level interface for managing synonyms. It wraps a `StaticSynonymProvider` internally and supports import/export:

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"car": ["automobile", "vehicle"]})

# CRUD
manager.add_synonyms("house", ["home", "residence"])
print(manager.get_synonyms("house"))  # ['home', 'residence']
manager.remove_synonym("house", "home")

# Import from external sources
manager.import_yaml("synonyms.yaml")   # Requires PyYAML
manager.import_json("synonyms.json")

# Export
manager.export_json("output.json")
```

### Import/Export Workflow

```python
# Import from YAML
manager = SynonymManager()
manager.import_yaml("my_synonyms.yaml")

# Export to JSON (e.g., for migration or backup)
manager.export_json("backup.json")
```

## Prebuilt Language Synonyms

The `LANG_SYNONYMS` dictionary contains starter synonym mappings for five languages:

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

# Available languages: fr, en, de, es, it
french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']

# Bootstrap a SynonymManager with a language
manager = SynonymManager(LANG_SYNONYMS["fr"])
```

| Language | Code | Sample Entry                          |
|----------|------|---------------------------------------|
| French   | `fr` | `"voiture": ["automobile", "véhicule"]` |
| English  | `en` | `"car": ["automobile", "vehicle"]`    |
| German   | `de` | `"auto": ["wagen", "fahrzeug"]`       |
| Spanish  | `es` | `"coche": ["automóvil", "vehículo"]`  |
| Italian  | `it` | `"auto": ["automobile", "veicolo"]`   |

> **Note**: These are minimal starter dictionaries for demonstration. Production deployments should load from curated or domain-specific sources.

## SynonymExpansionMiddleware

Integrates synonym expansion into the middleware pipeline. It expands both search queries and indexed document fields:

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

# Create a manager with your synonyms
manager = SynonymManager({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})

# Create the middleware
middleware = SynonymExpansionMiddleware(manager)

# Register with the PluginManager or MiddlewareChain
from whoosh.plugins.manager import PluginManager
PluginManager._default.register_middleware("synonym", middleware)
```

### How It Works

- **`before_search`**: Expands `context.query` by appending synonyms for each token
- **`before_index`**: Expands string values in `context.document` by appending synonyms

```python
# Before: query = "car"
# After:  query = "car automobile vehicle"

# Before: document = {"title": "house for sale"}
# After:  document = {"title": "house for sale home residence"}
```

## Language-Specific Stemming Analyzers

Located in `whoosh_modern.linguistics.stemmers`, these analyzers combine tokenization, stemming, and stop-word removal:

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Each analyzer is callable and returns a list of tokens
analyzer = EnglishAnalyzer()
tokens = analyzer("The running cats")
# tokens are stemmed: ["run", "cat"] (stop words removed)
```

### Stemmer Backend Selection

Under the hood, the stemmers use `whoosh_modern.analysis.stemmer_providers`:

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    register_stemmer,
    list_available_backends,
)

# Auto-detect best available stemmer (PyStemmer preferred)
stemmer = get_stemmer("auto", "english")

# Explicit backend
stemmer = get_stemmer("internal", "english")   # Whoosh's built-in stemmer
stemmer = get_stemmer("pystemmer", "english")   # PyStemmer (faster)

# List available backends
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'available', ...}

# Register a custom stemmer
@register_stemmer("my_stemmer")
class MyStemmer:
    def stem(self, word: str) -> str:
        return word.lower()
```

| Backend       | Requires                          | Speed   |
|---------------|-----------------------------------|---------|
| `auto`        | None (falls back automatically)  | Fastest available |
| `internal`    | None (built-in Porter stemmer)   | Medium  |
| `pystemmer`   | `pip install whoosh-ng[fast-stemming]` | Fast |

## Integration Example: Full Pipeline

```python
from whoosh_modern.linguistics import (
    EnglishAnalyzer,
    LANG_SYNONYMS,
    SynonymExpansionMiddleware,
    SynonymManager,
)
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher

# 1. Build synonym manager with English synonyms
syn_manager = SynonymManager(LANG_SYNONYMS["en"])
syn_manager.add_synonyms("search", ["query", "find", "lookup"])

# 2. Create synonym expansion middleware
syn_middleware = SynonymExpansionMiddleware(syn_manager)

# 3. Build middleware chain
chain = MiddlewareChain([syn_middleware])

# 4. Wrap writer and searcher
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="How to search in Whoosh")

with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    # Query "search" is expanded to "search query find lookup"
    results = searcher.search("search")
```

## See Also

- [Stemming Guide](stemming-sprint-d.md) — Stemmer providers and language analyzers
- [Middleware Guide](middleware-sprint-c.md) — Middleware pipeline integration
- [API: Linguistics](../api/modern.md) — Full API reference


## DOCUMENT: Middleware Sprint C

# Middleware & Plugin Pipeline

Module: `whoosh.middleware`, `whoosh.middleware.chain`, `whoosh.middleware.context`, `whoosh_modern.middleware`
Version: 2.0.0

The middleware pipeline allows you to intercept and modify indexing and search operations. It is the primary extension mechanism for cross-cutting concerns like logging, caching, metrics, query rewriting, and security. Middleware can come from both the core `whoosh.middleware` package and from plugins loaded via the `PluginManager`.

## Architecture Overview

```text
Writer/Searcher  ───►  MiddlewareChain
                           ├── Middleware 1 (before hook)
                           ├── Middleware 2 (before hook)
                           ├── ─── core operation ───
                           ├── Middleware 2 (after hook, reverse)
                           └── Middleware 1 (after hook, reverse)
```

- **Before hooks** execute in registration order
- **After hooks** execute in reverse order (like a stack / onion)
- If a hook raises `StopOperation`, the pipeline aborts gracefully
- If `fail_open=False` (default), exceptions propagate immediately

## Core Middleware Classes

### Middleware (Base Class)

Located in `whoosh.middleware.base`. Subclasses implement lifecycle hooks:

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MyMiddleware(Middleware):
    def startup(self, context: MiddlewareContext) -> None:
        """Called once when middleware is initialized."""
        pass

    def shutdown(self, context: MiddlewareContext) -> None:
        """Called once when middleware is torn down."""
        pass

    def before_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called before a document is indexed. Modify context.document."""
        return context

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called after a document is indexed."""
        return context

    def before_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called before a document is deleted."""
        return context

    def after_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called after a document is deleted."""
        return context

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called before a search query is executed. Modify context.query."""
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Called after results are returned. Access context.results."""
        return context

    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        """Called when an exception occurs. Re-raise by default."""
        raise exc

    def on_commit(self, context: MiddlewareContext) -> None:
        """Called after a commit operation."""
        pass
```

### MiddlewareContext

Located in `whoosh.middleware.context`. The context object passed to every hook:

```python
class MiddlewareContext:
    def __init__(self, operation: str) -> None:
        self.operation: str           # e.g., "add_document", "search"
        self.index: Any = None        # The Index instance
        self.backend: Any = None       # The storage backend
        self.writer: Any = None        # The IndexWriter (if applicable)
        self.searcher: Any = None      # The Searcher (if applicable)
        self.document: dict[str, Any] | None  # Document being indexed
        self.query: str = ""           # The search query string
        self.collector: Any = None     # The collector (if applicable)
        self.results: Any = None       # Search results
        self.labels: dict[str, Any] = {}    # Arbitrary labels/key-value pairs
        self.metadata: dict[str, Any] = {} # Per-request metadata
```

Use `context.copy()` to create a shallow copy if you need to preserve state.

### MiddlewareChain

Located in `whoosh.middleware.chain`. Orchestrates middleware execution:

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.context import MiddlewareContext

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware(),
])

# Before hooks (in order)
context = MiddlewareContext("search")
context.query = "hello world"
context = chain.run_before("before_search", context)

# ... core search operation ...

# After hooks (in reverse order)
context = chain.run_after("after_search", context)
print(context.results)
```

**Async support**: Use `async_run_before()`, `async_run_after()`, `async_run_on_error()`, and `run_hook()` for async middleware.

### MiddlewareRegistry

Located in `whoosh.middleware.registry`. A class-level registry for named middleware:

```python
from whoosh.middleware.registry import MiddlewareRegistry

MiddlewareRegistry.register("my_mw", MyMiddleware(), owner="my_plugin")
mw = MiddlewareRegistry.get("my_mw")
MiddlewareRegistry.unregister("my_mw")
print(MiddlewareRegistry.list_all())  # ['my_mw', ...]
```

## Middleware Integration

### Wrappers: MiddlewareWriter & MiddlewareSearcher

Located in `whoosh.middleware.wrappers`. These wrap the core writer/searcher to automatically execute middleware hooks:

```python
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher
from whoosh.middleware.chain import MiddlewareChain

chain = MiddlewareChain([MetricsMiddleware(), CacheMiddleware()])

# Wrap a writer
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Hello", content="World")

# Wrap a searcher
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    results = searcher.search(query)
```

### Integration Helpers

Located in `whoosh.middleware.integration`:

```python
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

# Auto-loads middleware from PluginManager if chain is not provided
writer = apply_middleware_to_writer(ix.writer())
searcher = apply_middleware_to_searcher(ix.searcher())
```

## Built-in Middleware

### Core Middleware (`whoosh.middleware.base`)

| Class                  | Hooks              | Description                              |
|------------------------|--------------------|------------------------------------------|
| `CompressionMiddleware` | `before_index`    | Marks documents with `_compressed = True` |
| `EncryptionMiddleware`  | `before_index`    | Marks documents with `_encrypted = True`  |
| `MetricsMiddleware`     | `after_index`, `after_search` | Tracks indexed docs and search count |
| `CacheMiddleware`       | `before_search`, `after_search` | In-memory result caching |

### Observability (`whoosh.middleware.metrics`)

`PrometheusMiddleware` — exports metrics to Prometheus (requires `prometheus-client`):

```python
from whoosh.middleware.metrics import PrometheusMiddleware

# Requires: pip install whoosh-ng[metrics]
prom = PrometheusMiddleware()
# Exports: whoosh_searches_total, whoosh_documents_indexed_total, whoosh_search_duration_seconds
```

### Modern Middleware (`whoosh_modern.middleware`)

#### Resilience Pipeline (`whoosh_modern.middleware.pipeline`)

These use a **wrap-style** API (decorator pattern) rather than hooks:

| Class                  | Description                              |
|------------------------|------------------------------------------|
| `RetryMiddleware`      | Retries failed operations with exponential backoff |
| `LoggingMiddleware`    | Logs operation execution time and errors |
| `CacheMiddleware`      | Caches operation results (LRU eviction)  |
| `MiddlewarePipeline`   | Chains multiple wrap-style middlewares   |

```python
from whoosh_modern.middleware import MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

pipeline = MiddlewarePipeline(
    LoggingMiddleware(),
    RetryMiddleware(attempts=3, backoff="exponential", jitter=True),
)

result = pipeline.execute(lambda: my_index_operation())
```

#### Storage Middleware (`whoosh_modern.middleware.storage`)

| Class                  | Description                              |
|------------------------|------------------------------------------|
| `StorageMiddleware`    | Routes persistence through pluggable storage providers |
| `FileStorageProvider`  | Local filesystem storage                 |
| `SQLiteStorageProvider`| SQLite-backed blob storage               |
| `S3StorageProvider`    | S3 / S3-compatible cloud storage         |

```python
from whoosh_modern.middleware.storage import StorageMiddleware, FileStorageProvider

storage = StorageMiddleware(FileStorageProvider("/data/index"), name="primary")
```

#### Search Middleware (`whoosh_modern.middleware.search`)

| Class                      | Description                              |
|----------------------------|------------------------------------------|
| `QueryRewriteMiddleware`   | Rewrites `context.query` before search   |
| `RankingMiddleware`        | Re-ranks `context.results` after search  |

```python
from whoosh_modern.middleware.search import QueryRewriteMiddleware

def add_synonyms(query: str) -> str:
    # Expand query with synonyms before execution
    return query + " " + get_synonyms(query)

rewriter = QueryRewriteMiddleware(rewriter=add_synonyms)
```

#### Analyzer Middleware (`whoosh_modern.middleware.analyzer`)

| Class                  | Description                              |
|------------------------|------------------------------------------|
| `StemmingMiddleware`   | Applies a stemmer to document fields and query |
| `SynonymMiddleware`    | Expands text with synonyms (placeholder) |

## Creating Custom Middleware

### Hook-Based Middleware

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class RequestLoggingMiddleware(Middleware):
    """Log all search requests with timing."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        import time
        context.metadata["_start_time"] = time.time()
        logger.info(f"[SEARCH] Query: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        elapsed = time.time() - context.metadata.get("_start_time", time.time())
        result_count = len(context.results) if context.results is not None else 0
        logger.info(f"[RESULTS] Found {result_count} hits in {elapsed:.3f}s")
        return context
```

### Wrap-Style Middleware

```python
from whoosh_modern.middleware.pipeline import Middleware as WrapMiddleware

class RetryMiddleware(WrapMiddleware):
    """Retry failed operations with backoff."""

    def __init__(self, attempts: int = 3) -> None:
        self._attempts = attempts

    def wrap(self, operation):
        def wrapped(*args, **kwargs):
            last_exc = None
            for attempt in range(self._attempts):
                try:
                    return operation(*args, **kwargs)
                except Exception as e:
                    last_exc = e
                    if attempt < self._attempts - 1:
                        time.sleep(2 ** attempt)
            raise last_exc
        return wrapped
```

### Middleware with Plugin Integration

Register middleware via a plugin so it's automatically discovered:

```python
from whoosh.plugins.manager import Plugin

class LoggingPlugin(Plugin):
    name = "logging"
    version = "1.0.0"
    middleware = ["whoosh_modern.middleware.pipeline.LoggingMiddleware"]

    def register(self, manager):
        manager.register_middleware(
            "logging",
            LoggingMiddleware(),
        )
```

## Error Handling

### StopOperation

Abort a pipeline operation gracefully:

```python
from whoosh.middleware.exceptions import StopOperation

class RateLimitMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if not rate_limiter.allow(context):
            raise StopOperation("Rate limit exceeded")
        return context
```

### fail_open Behavior

```python
class ResilientMiddleware(Middleware):
    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        try:
            send_to_analytics(context.results)
        except Exception:
            # Log but don't fail the search
            logger.warning("Analytics failed", exc_info=True)
        # Middleware chain continues
```

## Middleware Discovery from Plugins

When `PluginManager.load_plugins()` is called, all plugins that declare a `middleware` list will have those middleware classes imported and instantiated. The `get_middleware_chain()` method builds a `MiddlewareChain` from all registered middleware:

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Discovers plugins and their middleware

manager = PluginManager._default
chain = manager.get_middleware_chain()
# chain is a MiddlewareChain ready for use
```

## Best Practices

1. **Statelessness**: Use `context.metadata` for per-request data, not instance attributes
2. **Lightweight hooks**: Keep `before_*` and `after_*` hooks fast; use async for I/O
3. **Order matters**: Place caching before metrics, authentication before routing
4. **Fail fast**: Only use `fail_open=True` for non-critical middleware
5. **Test isolation**: Mock the `MiddlewareContext` to test middleware independently
6. **Clean up**: Implement `shutdown()` for resources like connections and timers

## See Also

- [Plugin System Guide](plugins-sprint-c.md) — Plugin registration and entry points
- [Middleware Examples](../examples/middleware.md) — Practical middleware patterns
- [API: Middleware](../api/middleware.md) — Full API reference
- [API: Middleware Pipeline (modern)](../api/modern.md) — Modern middleware extensions


## DOCUMENT: Middleware

# Middleware

The middleware pipeline allows you to intercept and modify indexing and search operations. It is the primary extension mechanism for cross-cutting concerns like logging, caching, metrics, and security.

## Core Concepts

A middleware is a class that implements hooks into the indexing and search lifecycle:

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MyMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Modify context.query or context.metadata
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Access context.results
        return context
```

## Available Hooks

| Hook | When | Common Uses |
|------|------|-------------|
| `startup(context)` | Middleware initialized | Open connections, warm caches |
| `shutdown(context)` | Middleware torn down | Close connections, flush buffers |
| `before_index(context)` | Before document added | Validation, enrichment, compression flags |
| `after_index(context)` | After document added | Metrics, events, cache invalidation |
| `before_delete(context)` | Before document deleted | Audit logging, access control |
| `after_delete(context)` | After document deleted | Metrics, cache invalidation |
| `before_search(context)` | Before query executes | Query rewriting, caching, auth |
| `after_search(context)` | After results returned | Logging, metrics, result modification |
| `on_error(context, exc)` | On exception | Error handling, fallbacks |
| `on_commit(context)` | After commit | Metrics, notifications |

## Built-in Middlewares

### MetricsMiddleware

Tracks basic statistics:

```python
from whoosh.middleware import MetricsMiddleware

metrics = MetricsMiddleware()
# After operations:
stats = metrics.get_metrics()
# Returns: {"documents_indexed": N, "searches_executed": N}
```

### CacheMiddleware

Caches search results in memory:

```python
from whoosh.middleware import CacheMiddleware

cache = CacheMiddleware()

# Check cache
cached = cache.get_cached("user query string")

# Store manually
cache.set_cached("user query string", results)
```

### CompressionMiddleware

Marks documents for compression at the backend level:

```python
from whoosh.middleware import CompressionMiddleware

compression = CompressionMiddleware()
# Sets document["_compressed"] = True
```

### EncryptionMiddleware

Marks documents for encryption at the backend level:

```python
from whoosh.middleware import EncryptionMiddleware

encryption = EncryptionMiddleware()
# Sets document["_encrypted"] = True
```

## MiddlewareChain

Orchestrates middleware execution:

```python
from whoosh.middleware import MiddlewareChain

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware()
])

# Execute before hook
context = MiddlewareContext("search")
context.query = "test"
context = chain.run_before("before_search", context)

# ... core operation ...

# Execute after hook
context = chain.run_after("after_search", context)
```

### Execution Order

- `before_*` hooks run in registration order
- `after_*` hooks run in reverse order
- If a hook raises `StopOperation`, the pipeline aborts
- If `fail_open=False`, exceptions propagate immediately

## Integration

### With Writer

```python
from whoosh.middleware.integration import apply_middleware_to_writer

writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)

with writer:
    writer.add_document(title="Hello", content="World")
```

### With Searcher

```python
from whoosh.middleware.integration import apply_middleware_to_searcher

searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)
results = searcher.search("query")
```

### With PluginManager

```python
from whoosh.plugins.manager import PluginManager

# Plugins can provide middleware
PluginManager.load_plugins()
chain = PluginManager.get_middleware_chain()
```

## Custom Middleware Example

```python
class RequestLoggingMiddleware(Middleware):
    """Log all search requests."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        context.metadata["request_id"] = generate_request_id()
        logger.info(f"Search: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        logger.info(f"Found {len(context.results)} results")
        return context

class RateLimitMiddleware(Middleware):
    """Abort searches exceeding rate limit."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if not rate_limiter.allow(context):
            raise StopOperation("Rate limit exceeded")
        return context

class QueryEnrichmentMiddleware(Middleware):
    """Add synonyms to the query."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query:
            context.query += " " + get_synonyms(context.query)
        return context
```

## Error Handling

```python
class ResilientMiddleware(Middleware):
    """Continue on non-critical errors."""

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        try:
            send_to_analytics(context.results)
        except Exception:
            # Log but don't fail the search
            logger.warning("Analytics failed", exc_info=True)
        return context
```

## Best Practices

1. **Stateless**: Use `context.metadata` for per-request data
2. **Fail fast**: Only use `fail_open=True` for non-critical middleware
3. **Order matters**: Place caching before metrics, auth before routing
4. **Performance**: Keep hooks lightweight; use async for I/O
5. **Testing**: Mock the context object to test middleware in isolation

## Modern Middleware (Whoosh-NG 2.0)

Whoosh-NG 2.0 adds a modern middleware package (`whoosh_modern.middleware`) with wrap-style resilience middleware (retry, caching, logging) and hook-based middleware for storage, search, and analysis. For full details on the modern middleware architecture, plugin integration, and deployment, see the [Middleware & Plugin Pipeline Guide](middleware-sprint-c.md).


## DOCUMENT: Modern Indexing

# Modern Indexing API

Whoosh-NG provides an optimized indexing layer in `whoosh_modern.indexing` for high-throughput document ingestion. These utilities wrap the core Whoosh writer without modifying the library internals.

## BatchIndexWriter

`BatchIndexWriter` wraps a core Whoosh writer with optimizations for batch processing of large datasets.

### Key Optimizations

- Pre-computes schema field names for fast filtering (O(1) per-field validation)
- Skips fields not in the schema (avoiding per-document overhead)
- Uses `multisegment=True` to defer merging during indexing
- Supports configurable batch commits to reduce I/O pressure
- Accepts a callback for post-commit hooks

### Basic Usage

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh import index

ix = index.open_dir("indexdir")

writer = BatchIndexWriter(ix, batch_size=5000, commit_every=10)

for batch in source.stream_batches(batch_size=5000):
    writer.add_batch(batch)

writer.close()
```

### Context Manager

```python
with BatchIndexWriter(ix, batch_size=10000) as writer:
    for doc in documents:
        writer.add_document(doc)
```

### Parameters

| Parameter | Default | Description |
|-----------|---------|-------------|
| `batch_size` | 5000 | Number of documents per batch |
| `limitmb` | 512 | Memory limit for the writer (MB) |
| `commit_every` | None | Commit after N batches (None = no auto-commit during indexing) |
| `multisegment` | True | Use multisegment mode to defer merging |
| `callback` | None | Callback invoked after each commit |
| `**writer_kwargs` | None | Additional keyword args passed to `index.writer()` |

### With Commit Profiler

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import CommitProfilerV2

profiler = CommitProfilerV2()
with BatchIndexWriter(ix, batch_size=5000, commit_every=5, commit_profiler=profiler) as writer:
    for batch in source.stream_batches(batch_size=5000):
        writer.add_batch(batch)

print(profiler.report())
```

---

## AnalyzerCache

`AnalyzerCache` provides an LRU cache for analyzer results, avoiding redundant analysis work on repeated field values.

### Basic Usage

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    cache_key = f"title:{doc['title']}"
    tokens = cache.get(cache_key)
    if tokens is None:
        tokens = list(analyzer(doc['title']))
        cache.put(cache_key, tokens)
```

### With get_or_compute

```python
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    tokens = cache.get_or_compute(
        f"title:{doc['title']}",
        lambda: list(analyzer(doc['title']))
    )
```

### Cache Statistics

```python
cache = AnalyzerCache(maxsize=50000)
# ... use cache ...

print(f"Hit rate: {cache.hit_rate:.1%}")
print(f"Size: {cache.size}/{cache.maxsize}")
print(cache.report())
# Analyzer Cache Report
# ==================================================
#   Size: 4823/50000
#   Hits: 12543
#   Misses: 3421
#   Hit rate: 78.6%
```

### Sizing from Profiling Data

```python
from whoosh_modern.profiling import AnalyzerCache, CacheAnalyzer

analyzer = CacheAnalyzer()
analysis = analyzer.analyze(source.iter_documents())

cache = AnalyzerCache.from_profiling(analysis.to_dict())
# Creates an optimally sized cache based on field repetition ratios
```

---

## FieldAnalyzerCache

`FieldAnalyzerCache` wraps an analyzer and caches results per field, automatically generating cache keys from field name and value.

### Basic Usage

```python
from whoosh_modern.profiling import FieldAnalyzerCache

field_cache = FieldAnalyzerCache(
    analyzer=StandardAnalyzer(),
    fields=["Country", "City"],
    cache_size=50000,
)

for doc in docs:
    for field in ["Country", "City"]:
        tokens = field_cache.analyze(field, doc[field])
```

### Invalidating Cache Entries

```python
# Invalidate a specific entry
field_cache.invalidate("Country", "USA")

# Clear entire cache
field_cache.clear()
```

### Cache Statistics

```python
print(f"Hit rate: {field_cache.hit_rate:.1%}")
print(field_cache.report())
# Field Analyzer Cache Report
# ==================================================
#   Fields: ['City', 'Country']
#   Cache size: 4823/50000
#   Hit rate: 96.5%
#   Hits: 12543
#   Misses: 3421
```

---

## Available Data Sources

| Class | Type | Dependencies |
|-------|------|-------------|
| `SQLSource` | SQLite, PostgreSQL, MySQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | Any SQLAlchemy-supported DB | `sqlalchemy` |
| `RESTSource` | REST APIs | none (stdlib `urllib`) |
| `GraphQLSource` | GraphQL APIs | none (stdlib `urllib`) |
| `FastCSVSource` | CSV files | none |
| `JSONSource` | JSON/JSONL files | none |
| `ParquetSource` | Parquet files | `pyarrow` or `pandas` |
| `PandasSource` | pandas DataFrames | `pandas` |
| `PolarsSource` | Polars DataFrames | `polars` |
| `PeeweeSource` | Peewee ORM | `peewee` |
| `TortoiseSource` | Tortoise ORM | `tortoise-orm` |
| `PydanticSource` | Pydantic models | `pydantic` |


## DOCUMENT: Monitoring

# Monitoring

Whoosh-NG ships with hooks for observability and a Prometheus plugin for production use.

## Built-in Metrics

```python
from whoosh.middleware import MetricsMiddleware, MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

chain = MiddlewareChain([MetricsMiddleware()])
writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)
searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)

metrics = chain.get_metrics()
print(metrics)
```

## Prometheus

```bash
pip install whoosh-ng[metrics]
```

| Metric | Type | Description |
|--------|------|-------------|
| `whoosh_documents_indexed_total` | Counter | Total documents indexed |
| `whoosh_searches_executed_total` | Counter | Total searches executed |
| `whoosh_indexing_duration_seconds` | Histogram | Indexing latency |
| `whoosh_search_duration_seconds` | Histogram | Search latency |
| `whoosh_index_size_bytes` | Gauge | Current index size |

## Best practices

1. Add `MetricsMiddleware` early in your base chain.
2. Expose `/metrics` in production.
3. Use `/health` for load balancer health checks.
4. Emit `DocumentIndexed` and `SearchExecuted` events.


## DOCUMENT: Ngrams

# N-grams

This guide covers N-gram tokenization and analysis for substring matching,
prefix queries, and autocomplete functionality.

## What Are N-grams?

An N-gram is a contiguous sequence of N characters (or tokens) from a string.
For example, the 2-grams of "hello" are: "he", "el", "ll", "lo".

N-gram analysis is useful for:
- Substring search (finding "ell" within "hello")
- Autocomplete / typeahead suggestions
- Fuzzy matching without edit distance computation

## NgramTokenizer

The `NgramTokenizer` splits text into character-level N-grams:

```python
from whoosh.analysis import NgramTokenizer
from whoosh import fields

tokenizer = NgramTokenizer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=tokenizer),
)
```

### NgramTokenizer Parameters

- `minsize`: Minimum N-gram length (default `2`)
- `maxsize`: Maximum N-gram length (default `4`)

With the example above, the text "hello" produces these 2-4-grams:
`he, hel, hell, el, ell, ello, l, ll, llo, l, lo, o`

## NgramFilter

The `NgramFilter` creates word-level N-grams from tokenized text:

```python
from whoosh.analysis import RegexTokenizer, NgramFilter

analyzer = RegexTokenizer() | NgramFilter(maxsize=2)
```

This produces word-level grams: for "hello world", it produces ("hello",)
and ("hello", "world").

## NgramWordAnalyzer

A convenience analyzer that combines `NgramTokenizer` with `LowercaseFilter`:

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

## Use Cases

### Substring Search

With N-gram analysis, you can match substrings:

```python
from whoosh.qparser import QueryParser

# Index text with N-grams
# Searching for "ell" matches "hello" because "ell" is a substring
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("ell")
results = searcher.search(q)
```

### Prefix Matching

Set `maxsize` equal to a large value to effectively create prefix N-grams:

```python
from whoosh.analysis import NgramWordAnalyzer

# Create N-grams where each word's prefixes become searchable tokens
# e.g., "hello" -> "h", "he", "hel", "hell", "hello"
analyzer = NgramWordAnalyzer(minsize=1, maxsize=10)
```

### Autocomplete

N-gram indexes are commonly used for autocomplete/typeahead. For more
advanced autocomplete with edge n-grams, consider:

```python
from whoosh.analysis import RegexTokenizer, NgramFilter
from whoosh.query import Prefix

# Index with standard tokenization, then use Prefix queries for autocomplete
analyzer = RegexTokenizer()
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=analyzer),
    content=fields.TEXT(analyzer=analyzer),
)

# For autocomplete, query with Prefix
from whoosh.qparser import QueryParser
qp = QueryParser("title", schema=ix.schema)
q = Prefix("title", "hel")  # Find documents where title starts with "hel"
```

## Comparison with Edge N-grams

Some search engines support "edge n-grams" (only generating N-grams from the
beginning of words). This is more space-efficient for autocomplete:

- Full N-grams: "hello" → "he", "el", "ll", "lo", "hel", "ell", ...
- Edge N-grams: "hello" → "h", "he", "hel", "hell", "hello"

Whoosh's `NgramTokenizer` generates full (bidirectional) N-grams. For
edge-ngram-like behavior, use the `minsize` and `maxsize` parameters
strategically, or use `Prefix` queries against a standard tokenized field.

## Performance Considerations

- N-gram indexes are typically much larger than standard indexes
- Each original token produces multiple N-gram tokens, increasing index size
- Choose `minsize` and `maxsize` carefully to balance search quality against
  index size
- For autocomplete, consider using `Prefix` queries with a
  non-N-gram field for better performance


## DOCUMENT: Performance

# Performance Benchmarking

Whoosh-NG includes a comprehensive benchmarking toolkit in `whoosh_modern.profiling` for measuring and comparing analyzer performance. This guide explains how to use these tools and documents the optimizations shipped in 2.0.0.

## Quick Start

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

# Generate synthetic datasets for consistent benchmarks
gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Run tokenizer benchmark (P5.1)
run_p5_1(datasets)

# Run token creation benchmark (P5.2)
run_p5_2(token_count=100_000)

# Run stemmer benchmark
bench = StemmerBenchmark()
bench.run(gen.generate_dataset("A", 5000))
print(bench.report())
```

## Benchmarking Tools

### SyntheticDatasetGenerator

Generates deterministic text datasets of varying complexity:

```python
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator

gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Dataset A: 2 tokens/doc (short)
# Dataset B: 50 tokens/doc (medium)
# Dataset C: 500 tokens/doc (large)
# Dataset D: 1200 tokens/doc (very large)
for name, texts in datasets.items():
    print(f"{name}: {len(texts)} documents")
```

### P5.1: RegexTokenizer Benchmark

Compares different tokenizer implementations:

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1

results = run_p5_1(datasets)

# Compare:
# - Current Regex (whoosh default)
# - Compiled Global regex
# - Manual Python tokenizer
# - C extension (re2, if available)
```

### P5.2: Token Optimization Benchmark

Compares Token object implementations:

```python
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2

# Compare:
# - Current Token (dict-based)
# - __slots__ optimization
# - namedtuple
# - dataclass(slots=True)
results = run_p5_2(token_count=100_000)
```

### StemmerBenchmark

Compares stemmer backends:

```python
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

bench = StemmerBenchmark()
bench.run(texts, warmup=True)
print(bench.report())
# Output:
# Stemmer Benchmark
# ==================================================
# Stemmer         Tokens/s        Time (s)    Tokens
# ------------------------------------------------------
# StemFilter      1,004,172       0.1503      150,983
# PyStemmer       2,100,000+      0.0719+     150,983
```

## Performance Optimizations

### 2.0.0 Performance Summary

| Optimization | Component | Measurable Gain |
|---|---|---|
| `__slots__` on Token | `whoosh.analysis.acore` | +35% token creation |
| Global compiled regex | `RegexTokenizer` | +50% regex throughput |
| Compact postings (1-posting) | `W3TermInfo` / `W3PostingsWriter` | +35% commit speed |
| Compact postings (2-8 postings) | `W3TermInfo` / `W3PostingsWriter` | +35% commit speed |
| Field cache in add_postings | `whoosh.codec.base` | -93% write_block calls |
| Varint position encoding | `whoosh.formats` | reduced per-term overhead |
| Stemmer cache tuning | `whoosh.analysis.morph` | 96.5% hit rate, 4.12x on repetitive fields |
| Analyzer cache | `whoosh_modern.profiling.analyzer_cache` | 4.12x on high-repetition fields |
| Batch writer optimization | `whoosh_modern.indexing.batch_writer` | optimized filtered batches |
| Stopword setdefault optimization | `whoosh.formats` | reduced dict overhead |

### Benchmark Results: 20k Documents (`customers_csv`)

```
before:
  commit total      : 18.653s
  analyzing         : 8.641s  (51.5%)
  committing        : 10.012s (27.1%)
  write_postings    : 6.5s
  write_block calls : ~72612

after:
  commit total      : 6.806s   (-63.5%)
  analyzing         : ~2.7s    (-68%)
  committing        : 6.806s   (-32%)
  write_postings    : 6.5s -> reduced allocation
  write_block calls : 7565     (-93%)
  throughput        : 1275 docs/s
```

### Benchmark Results: Stemmer Backends (1.5M tokens)

| Stemmer | Throughput | Relative |
|---|---|---|
| StemFilter (internal) | 1,004,172 tokens/s | 1.0x |
| PyStemmer | ~2,100,000 tokens/s | ~2.1x |

### Benchmark Results: Regex Tokenizer

| Tokenizer | Throughput | Relative |
|---|---|---|
| Current regex | ~1,000,000 tokens/s | 1.0x |
| Compiled global | ~2,300,000 tokens/s | 2.3x |

### Benchmark Results: Token Object

| Implementation | Tokens/sec | Relative |
|---|---|---|
| Current (dict) | 1,000,000 | 1.0x |
| `__slots__` | ~1,350,000 | 1.35x |

## Profiling Tools

### IndexingPipelineProfiler

Profiles the complete indexing pipeline:

```python
from whoosh_modern.profiling.indexing_pipeline_profiler import IndexingPipelineProfiler

profiler = IndexingPipelineProfiler()
for doc in documents:
    profiler.before_tokenize(doc, analyzer)
    analyzer(doc)
    profiler.after_tokenize()

report = profiler.report()
print(report)
```

### CommitProfiler

Profiles commit performance including field writing and posting flushing:

```python
from whoosh_modern.profiling.commit_profiler_v2 import CommitProfiler

profiler = CommitProfiler()
# ... index documents ...
ix.commit()

report = profiler.report()
# Shows: analyze, convert_fields, write_postings, flush, commit breakdown
```

### FieldIndexProfiler

Profiles field conversion costs:

```python
from whoosh_modern.profiling.field_index_profiler import FieldIndexProfiler

profiler = FieldIndexProfiler()
# ... index documents ...
report = profiler.report()
# Identifies expensive field types and conversion bottlenecks
```

### IndexQualityAnalyzer

Analyzes index quality metrics including singleton terms:

```python
from whoosh_modern.profiling.index_quality_analyzer import IndexQualityAnalyzer

analyzer = IndexQualityAnalyzer(index_reader)
report = analyzer.analyze()
print(f"Singleton terms: {report['singleton_terms']}/{report['total_terms']} ({report['singleton_percent']}%)")
```

## Stemmer Provider System

Whoosh-NG provides a pluggable stemmer provider system:

```python
from whoosh_modern.analysis import get_stemmer, StemmingAnalyzer, list_available_backends

# Check available backends
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'not installed'}

# Use auto-detection (default)
analyzer = StemmingAnalyzer(stemmer="auto")

# Explicit internal stemmer
analyzer = StemmingAnalyzer(stemmer="internal")

# PyStemmer (requires: pip install whoosh-ng[fast-stemming])
analyzer = StemmingAnalyzer(stemmer="pystemmer")

# Custom stemmer provider
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def __init__(self, language="english"):
        self._lang = language

    def stem(self, word):
        return word.lower()

    @property
    def name(self):
        return "my_stemmer"

    @property
    def language(self):
        return self._lang

analyzer = StemmingAnalyzer(stemmer=MyStemmer())
```

## Performance Recommendations

1. **Use `StemmingAnalyzer`** from `whoosh_modern.analysis` for automatic PyStemmer selection
2. **Enable stemmer cache** for repetitive content (`cachesize=50000` by default)
3. **Minimize TEXT fields** — use KEYWORD or ID for low-cardinality fields
4. **Avoid stored positions/chars** unless highlighting requires them
5. **Use batch indexing** with larger segments for better throughput
6. **Monitor singleton terms** — reduce rare terms via stopword lists

## Running the Full Benchmark Suite

```bash
cd whoosh-ng

# Run all P5 benchmarks
uv run python -m pytest tests/test_regex_tokenizer_unicode.py tests/test_token_slots.py tests/test_stemmer_compatibility.py -v

# Run full test suite
uv run python -m pytest -q
```


## DOCUMENT: Plugins Sprint C

# Plugin System

Module: `whoosh.plugins.manager`
Version: 2.0.0

Whoosh-NG's plugin architecture enables external packages to extend the core indexing, search, and analysis pipeline. Plugins are discovered via Python [entry points](https://docs.python.org/3/library/importlib.metadata.html#entry-points) declared in `pyproject.toml` and managed by the `PluginManager`.

## Architecture Overview

```text
PluginManager (singleton)
    ├── load_plugins(group)          # Auto-discover from entry points
    ├── register(plugin)             # Manual registration
    ├── enable(name) / disable(name) # Toggle lifecycle
    ├── get(name) / list_plugins()   # Inspection
    ├── get_middleware_chain()       # Build MiddlewareChain from plugin middleware
    ├── register_datasource()        # Register a datasource provider
    ├── register_vector_provider()   # Register a vector provider
    ├── register_middleware()        # Register a middleware instance
    ├── register_embedding()         # Register an embedding provider
    ├── register_analyzer()          # Register a named analyzer
    └── register_query_rewriter()    # Register a query rewriter
```

## Plugin Base Classes

### Plugin (ABC)

The root plugin class. Subclasses set class-level attributes and implement `register()`.

```python
from whoosh.plugins.manager import Plugin, PluginMetadata

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"

    def register(self, manager: PluginManager) -> None:
        """Called when the plugin is loaded; register providers here."""
        manager.register_middleware("my_module.MyMiddleware", MyMiddleware())

    def register_hooks(self) -> None:
        """Register event hooks (optional)."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass
        register_hook("on_search", hookimpl(on_search))
```

### AnalyzerPlugin

For plugins that provide custom tokenizers/analyzers:

```python
from whoosh.plugins.manager import AnalyzerPlugin

class MyAnalyzerPlugin(AnalyzerPlugin):
    name = "my_analyzer"

    def register(self, manager):
        manager.register_analyzer("my_analyzer", MyTokenizer())
```

### QueryRewritePlugin

For plugins that transform queries before execution:

```python
from whoosh.plugins.manager import QueryRewritePlugin

class SynonymRewriterPlugin(QueryRewritePlugin):
    name = "synonym_rewriter"

    def rewrite(self, query, searcher):
        # Return modified query
        return query
```

## PluginMetadata

A dataclass describing plugin metadata:

| Field         | Type              | Description                            |
|---------------|-------------------|----------------------------------------|
| `name`        | `str`             | Unique plugin name                     |
| `version`     | `str`             | SemVer version string                  |
| `depends_on`  | `list[str]`       | Names of required plugins              |
| `priority`    | `int`             | Load ordering priority (higher = later)|
| `middleware`  | `list[str]`       | Dotted paths to middleware classes     |

## Entry Point Groups

The `PluginManager` discovers plugins from these standard entry-point groups:

| Group                | Purpose                              |
|----------------------|--------------------------------------|
| `whoosh.plugins`     | General plugins                      |
| `whoosh.datasources` | Data source providers                |
| `whoosh.vector.providers` | Vector similarity providers     |
| `whoosh.middlewares` | Middleware classes                    |
| `whoosh.embeddings`  | Embedding model providers            |
| `whoosh.language`    | Language-specific analyzers          |
| `whoosh.apps`        | App factories (FastAPI, admin, etc.) |

## Creating and Deploying a Plugin

### Step 1: Define the Plugin Class

```python
# my_plugin/plugin.py
from whoosh.plugins.manager import Plugin
from whoosh.registry import VectorRegistry

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Register a vector provider with the VectorRegistry."""
        provider = MyCustomVectorProvider()
        VectorRegistry.register("my_vector", provider, owner=self.name)

    def register_hooks(self):
        """Register optional hooks (e.g., on_search, on_index)."""
        pass
```

### Step 2: Declare the Entry Point

In your `pyproject.toml`:

```toml
[project]
name = "whoosh-ng-my-vector"
version = "1.0.0"
dependencies = ["whoosh-ng>=2.0"]

[project.entry-points."whoosh_ng.plugins"]
my_vector = "my_plugin.plugin:MyVectorPlugin"
```

### Step 3: Install and Verify

```bash
pip install -e .
```

```python
# Verify the plugin is registered
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Auto-discovers all entry points

manager = PluginManager._default
print(manager.list_plugins())
# ['whoosh_autocomplete', 'whoosh_vector', ..., 'my_vector']

# Check the registry
from whoosh.registry import VectorRegistry
print(VectorRegistry.list_keys())
# ['my_vector', 'numpy']
```

## Manual Registration (No Entry Point)

For testing or programmatic use:

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyVectorPlugin())
manager.enable("my_vector")
```

## Plugin Lifecycle

```
1. Entry point discovered  ───►  2. register() called  ───►  3. register_hooks()
   │                               │                            │
   └── load_plugins(group)          └── register provider/     └── register_hook()
                                      middleware/analyzer
```

### Enabling / Disabling

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager._default

manager.enable("my_vector")    # Activate a plugin
manager.disable("my_vector")   # Deactivate a plugin
print(manager.list_enabled())  # Only enabled plugins
```

### Version Validation

```python
# Check if a plugin meets a minimum version
ok = manager.validate_version("my_vector", "1.0.0")
print(ok)  # True if plugin version >= 1.0.0
```

### Conflict Detection

```python
# Check if two plugins conflict
if manager.detect_conflicts("plugin_a", "plugin_b"):
    print("These plugins cannot be loaded together")
```

## Plugin Manager API Reference

### `PluginManager.load_plugins(group=None)`

Load all plugins from entry-point groups. If `group` is `None`, loads from all standard groups (`STANDARD_GROUPS`).

### `PluginManager.register(plugin)`

Register a plugin instance. Calls `plugin.register(self)` and `plugin.register_hooks()`. Supports async `register()` via `asyncio`.

### `PluginManager.get_middleware_chain()`

Builds and returns a `MiddlewareChain` from all plugins that declare `middleware` entries. Middleware classes are imported and instantiated by dotted path.

### Registry Registration Methods

| Method                       | Description                          |
|------------------------------|--------------------------------------|
| `register_analyzer(name, analyzer)` | Register a named analyzer   |
| `register_datasource(name, datasource)` | Register a datasource  |
| `register_vector_provider(name, provider)` | Register a vector provider |
| `register_middleware(name, middleware)` | Register a middleware instance |
| `register_embedding(name, embedding)` | Register an embedding provider |
| `register_query_rewriter(plugin)` | Register a query rewriter plugin |

### Lookup Methods

| Method                       | Returns                          |
|------------------------------|----------------------------------|
| `get(name)`                  | `Plugin` instance                |
| `list_plugins()`             | All registered plugin names      |
| `list_enabled()`             | Enabled plugin names             |
| `get_analyzer(name)`         | Analyzer callable                |
| `list_analyzers()`           | Registered analyzer names        |
| `list_datasources()`         | Registered datasource names      |
| `list_vector_providers()`    | Registered vector provider names |
| `list_middlewares()`         | Registered middleware names      |
| `list_embeddings()`          | Registered embedding names       |
| `list_query_rewriters()`     | Registered query rewriter names  |

## Built-in Plugins

| Plugin            | Module                  | Entry-point Group       |
|-------------------|-------------------------|-------------------------|
| `whoosh_autocomplete` | `whoosh_modern.autocomplete.plugin` | `whoosh.plugins` |
| `whoosh_vector`   | `whoosh_modern.vector.plugin`      | `whoosh.plugins` |
| `whoosh_fastapi`  | `whoosh_fastapi`                  | `whoosh.apps` |
| `whoosh_observability` | `whoosh.middleware.metrics`  | `whoosh.middlewares` |
| `whoosh_admin`    | `whoosh_admin`                   | `whoosh.apps` |

## Best Practices

1. **Single responsibility**: One plugin, one feature
2. **Declare dependencies**: Use `depends_on` for required plugins
3. **Semantic versioning**: Increment version for API changes
4. **Graceful degradation**: Check for optional dependencies in `register()`
5. **Avoid side effects in `__init__`**: All setup in `register()`
6. **Clean up**: If applicable, provide teardown logic

## See Also

- [Middleware Guide](middleware-sprint-c.md) — Pipeline hooks and custom middleware
- [Plugin Development Example](../examples/plugin-dev.md) — Step-by-step plugin tutorial
- [API: Plugins](../api/plugins.md) — Full API reference


## DOCUMENT: Plugins

# Plugins

Whoosh-NG uses a plugin architecture to keep the core lightweight while enabling advanced features. Plugins are loaded via entry points and managed by the `PluginManager`.

## Plugin Architecture

```text
PluginManager
    ├── load_plugins()     # Auto-discover from entry points
    ├── register(name, plugin)  # Manual registration
    ├── enable(name)       # Enable a plugin
    ├── disable(name)      # Disable a plugin
    ├── get(name)          # Retrieve a plugin
    └── list_plugins()     # List all plugins
```

## Built-in Plugins

| Plugin | Package | Description |
|--------|---------|-------------|
| whoosh-ng-vector | `whoosh_modern.vector` | Vector search providers (NumPy, HNSW, Faiss) |
| whoosh-ng-autocomplete | `whoosh_modern.autocomplete` | Edge ngram autocomplete |
| whoosh-ng-fastapi | `whoosh_fastapi` | FastAPI app factory |
| whoosh-ng-observability | `whoosh_observability` | Prometheus metrics |
| whoosh-ng-admin | `whoosh_admin` | Admin UI |

## Creating a Plugin

Every plugin is a subclass of `BasePlugin`:

```python
from whoosh.plugins.base import BasePlugin

class MyPlugin(BasePlugin):
    name = "my_plugin"
    version = "1.0.0"
    dependencies = []

    def setup(self, registry):
        """Called when the plugin is enabled."""
        registry.register("my_provider", MyProvider())

    def teardown(self, registry):
        """Called when the plugin is disabled."""
        registry.unregister("my_provider")

    def middleware(self):
        """Return middleware to inject into the pipeline."""
        return [MyMiddleware()]

    def on_startup(self):
        """Called once at application startup."""
        pass

    def on_shutdown(self):
        """Called once at application shutdown."""
        pass
```

## Plugin Registration

### Via entry_points (pyproject.toml)

```toml
[project.entry-points."whoosh_ng.plugins"]
my_plugin = "my_package.plugin:MyPlugin"
```

### Programmatic

```python
from whoosh.plugins.manager import PluginManager

plugin = MyPlugin()
PluginManager.register("my_plugin", plugin)
PluginManager.enable("my_plugin")
```

## Plugin Lifecycle

```
register() -> setup() -> enable() -> middleware hooks -> teardown() -> disable()
```

## Plugin Dependencies

Plugins can declare dependencies on other plugins:

```python
class VectorPlugin(BasePlugin):
    name = "vector"
    version = "1.0.0"
    dependencies = ["metrics"]  # Requires metrics plugin
```

The `PluginManager` resolves dependency order and detects conflicts.

## Example: Vector Plugin

```python
from whoosh.plugins.base import BasePlugin
from whoosh.vector.base import VectorProvider, VectorField
from whoosh.vector.numpy_provider import NumpyProvider

class VectorPlugin(BasePlugin):
    name = "vector"
    version = "1.0.0"
    dependencies = []

    def setup(self, registry):
        provider = NumpyProvider()
        registry.register("numpy", provider, owner="vector")

    def middleware(self):
        from whoosh.middleware import MetricsMiddleware
        return [MetricsMiddleware()]

    def on_startup(self):
        print("Vector plugin loaded")
```

## Example: FastAPI Plugin

```python
from whoosh.plugins.base import BasePlugin

class FastAPIPlugin(BasePlugin):
    name = "fastapi"
    version = "1.0.0"

    def setup(self, registry):
        self.app = None

    def create_app(self, index, **kwargs):
        from whoosh_fastapi import create_app
        self.app = create_app(index=index, **kwargs)
        return self.app

    def middleware(self):
        from whoosh.middleware import CacheMiddleware
        return [CacheMiddleware()]
```

## Best Practices

1. **Keep plugins small**: One plugin, one responsibility
2. **Declare dependencies**: Help PluginManager resolve load order
3. **Handle conflicts**: Check for existing registrations before adding
4. **Clean up**: Implement `teardown()` to remove registries and middleware
5. **Version your plugin**: Semver for compatibility checking

## Modern Plugin System (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces an enhanced `PluginManager` with registry support for datasources, vector providers, embeddings, and middleware. For full details on the modern plugin architecture, entry point groups, and deployment, see the [Plugin System Guide](plugins-sprint-c.md).


## DOCUMENT: Stemming Sprint D

# Stemmer Providers

Module: `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.analysis.stemming_analyzer`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

The stemmer provider system gives you flexible control over which stemming backend is used for text analysis. It supports auto-detection, explicit backend selection, and custom stemmer registration—all with a clean plugin-style API.

## Module Overview

```text
whoosh_modern.analysis
    ├── stemmer_providers.py   # StemmerProvider protocol, Internal/PyStemmer backends, register_stemmer, get_stemmer
    └── stemming_analyzer.py   # Enhanced StemmingAnalyzer with plugin support

whoosh_modern.linguistics.stemmers
    └── __init__.py            # Language-specific analyzers (FR/EN/DE/ES/IT)
```

## StemmerProvider Protocol

Located in `whoosh_modern.analysis.stemmer_providers`:

```python
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def stem(self, word: str) -> str:
        """Stem a single word."""
        ...

    @property
    def name(self) -> str:
        """Return the stemmer name."""
        return "my_stemmer"

    @property
    def language(self) -> str:
        """Return the language code."""
        return "english"
```

## Getting a Stemmer

### Auto-Detection (Recommended)

The `get_stemmer("auto", language)` function automatically selects the best available backend:

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Auto-detect: prefers PyStemmer if installed, falls back to internal
stemmer = get_stemmer("auto", "english")
print(stemmer.stem("running"))  # "run"
print(stemmer.name)             # "pystemmer" or "internal"
```

**Priority order:**
1. **PyStemmer** (fastest, requires `pip install whoosh-ng[fast-stemming]`)
2. **Internal** stemmer (built-in Porter stemmer, always available)

### Explicit Backend Selection

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Force internal stemmer
stemmer = get_stemmer("internal", "english")

# Force PyStemmer (requires installation)
stemmer = get_stemmer("pystemmer", "english")
```

### List Available Backends

```python
from whoosh_modern.analysis.stemmer_providers import list_available_backends

backends = list_available_backends()
print(backends)
# {'internal': 'available', 'pystemmer': 'available', 'my_custom': 'registered'}
```

| Backend       | Status String    | Requires                          |
|---------------|------------------|-----------------------------------|
| `internal`    | `"available"`    | None (always bundled)             |
| `pystemmer`   | `"available"` / `"not installed"` | `pip install whoosh-ng[fast-stemming]` |
| Custom        | `"registered"`   | Registered via `@register_stemmer` |

## Built-in Stemmer Providers

### InternalStemmerProvider

Wraps Whoosh's built-in Porter stemmer. Always available (no extra dependencies):

```python
from whoosh_modern.analysis.stemmer_providers import InternalStemmerProvider

stemmer = InternalStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
print(stemmer.stem("running")) # "run"
```

### PyStemmerProvider

Wraps the `Stemmer` library for high-performance stemming. Supports all Snowball languages:

```python
from whoosh_modern.analysis.stemmer_providers import PyStemmerProvider

# Requires: pip install whoosh-ng[fast-stemming]
stemmer = PyStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
```

**Note**: This provider calls `self._stemmer.stemWord(word)` to stem words. Ensure PyStemmer is installed or auto-detection will fall back to the internal stemmer.

### IdentityStemmerProvider

A no-op stemmer for testing or when stemming is not desired:

```python
from whoosh_modern.analysis.stemmer_providers import IdentityStemmerProvider

stemmer = IdentityStemmerProvider()
print(stemmer.stem("anything"))  # "anything"
```

## Registering a Custom Stemmer

Use the `@register_stemmer` decorator:

```python
from whoosh_modern.analysis.stemmer_providers import register_stemmer

@register_stemmer("simple")
class SimpleStemmer:
    def stem(self, word: str) -> str:
        # Simple suffix stripping
        if word.endswith("s") and len(word) > 3:
            return word[:-1]
        return word

    @property
    def name(self) -> str:
        return "simple"

    @property
    def language(self) -> str:
        return "english"

# Now use it
from whoosh_modern.analysis.stemmer_providers import get_stemmer

stemmer = get_stemmer("simple", "english")
print(stemmer.stem("cats"))  # "cat"
```

## StemmingAnalyzer (Enhanced)

Located in `whoosh_modern.analysis.stemming_analyzer`, this is the main entry point for creating language-aware analyzers:

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Auto-detect best stemmer for English
analyzer = StemmingAnalyzer(stemmer="auto", language="english")

# Explicit internal stemmer
analyzer = StemmingAnalyzer(stemmer="internal", language="english")

# PyStemmer backend (if installed)
analyzer = StemmingAnalyzer(stemmer="pystemmer", language="french")

# Custom stemmer provider
analyzer = StemmingAnalyzer(stemmer=my_stemmer_instance)
```

### StemmingAnalyzer Parameters

| Parameter   | Type                          | Default                  | Description                      |
|-------------|-------------------------------|--------------------------|----------------------------------|
| `expression`| Regex pattern                 | default token pattern    | Tokenization regex              |
| `stoplist`  | Iterable of stop words        | `whoosh.analysis.STOP_WORDS` | Stop words to filter         |
| `minsize`   | `int`                         | `2`                      | Minimum token length            |
| `maxsize`   | `int \| None`                 | `None`                   | Maximum token length            |
| `gaps`      | `bool`                        | `False`                  | Split on expression vs. match  |
| `stemmer`   | `str \| StemmerProvider`      | `"auto"`                 | Stemmer backend                 |
| `language`  | `str`                         | `"english"`              | Language code                   |
| `ignore`    | `set[str] \| None`            | `None`                   | Words to skip                   |
| `cachesize` | `int`                         | `50000`                  | Stem cache size                 |

### Using with Field Types

```python
from whoosh_modern.analysis import StemmingAnalyzer
from whoosh.fields import Schema, TEXT

# English stemmer with stop words
en_analyzer = StemmingAnalyzer("auto", language="english")

# French stemmer
fr_analyzer = StemmingAnalyzer("auto", language="french")

schema = Schema(
    title=TEXT(stored=True),
    content_en=TEXT(analyzer=en_analyzer),
    content_fr=TEXT(analyzer=fr_analyzer),
)
```

## Language-Specific Analyzers

Pre-built analyzers for five languages, available in `whoosh_modern.linguistics.stemmers`:

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Each is callable and returns a list of tokens
en = EnglishAnalyzer()
tokens = en("The quick brown foxes")
# tokens are stemmed: ["quick", "brown", "fox"] (stop words like "the" removed)
```

### Available Language Analyzers

| Class             | Language  | Module                              |
|-------------------|-----------|-------------------------------------|
| `EnglishAnalyzer` | English   | `whoosh_modern.linguistics.stemmers` |
| `FrenchAnalyzer`  | French    | `whoosh_modern.linguistics.stemmers` |
| `GermanAnalyzer`  | German    | `whoosh_modern.linguistics.stemmers` |
| `SpanishAnalyzer` | Spanish   | `whoosh_modern.linguistics.stemmers` |
| `ItalianAnalyzer` | Italian   | `whoosh_modern.linguistics.stemmers` |

Each internally uses `get_stemmer("auto", language)` to select the best available backend and applies language-specific stop words.

## Stemmer Compatibility Validation

Validate that a stemmer provider works correctly with a set of test words:

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    validate_stemmer_compatibility,
)

stemmer = get_stemmer("auto", "english")
report = validate_stemmer_compatibility(stemmer, ["running", "cats", "jumps", "houses"])

print(report["total_words"])   # 4
print(report["successful"])    # 4 (or fewer if errors)
print(report["failed"])        # 0
print(report["results"])       # [{'word': 'running', 'stemmed': 'run', 'success': True}, ...]
```

### Compatibility Report Structure

| Field          | Type       | Description                          |
|----------------|------------|--------------------------------------|
| `provider`     | `str`      | Stemmer provider name                |
| `language`     | `str`      | Language code                        |
| `total_words`  | `int`      | Total test words                     |
| `successful`   | `int`      | Words stemmed successfully           |
| `failed`       | `int`      | Words that failed                    |
| `results`      | `list[dict]` | Per-word results with `word`, `stemmed`, `success` |

## Integration with StemmingMiddleware

The stemmer providers can be used with the `StemmingMiddleware` from `whoosh_modern.middleware.analyzer`:

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer
from whoosh_modern.middleware.analyzer import StemmingMiddleware

stemmer = get_stemmer("auto", "english")
middleware = StemmingMiddleware(
    stemmer=stemmer.stem,
    fields=["title", "content"],  # Only stem these fields
    stem_query=True,              # Also stem the search query
)
```

## Migration from Classic Whoosh

### Old API (Whoosh 1.x/2.x)

```python
from whoosh.analysis import StemmingAnalyzer as OldAnalyzer
analyzer = OldAnalyzer("en")  # Hardcoded to "english"
```

### New API (Whoosh-NG 2.0)

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Auto-detect backend (preferred)
analyzer = StemmingAnalyzer("auto", language="en")

# Or use a language-specific analyzer
from whoosh_modern.linguistics.stemmers import EnglishAnalyzer
analyzer = EnglishAnalyzer()
```

> **Note**: The old `StemmingAnalyzer("en")` hardcoded the language to `"english"`. The new `StemmingAnalyzer(stemmer, language)` parameter is explicit and supports all Snowball languages via PyStemmer.

## Installation

```bash
# Without PyStemmer (uses internal stemmer, slower)
pip install whoosh-ng

# With PyStemmer (recommended, faster)
pip install whoosh-ng[fast-stemming]

# Full modern analysis
pip install whoosh-ng[modern]
```

## See Also

- [Stemming and Stop Words Guide](stemming.md) — Classic Whoosh stemming guide
- [Synonyms & Linguistics Guide](linguistics-sprint-d.md) — Synonym expansion engine
- [API: Modern](../api/modern.md) — Full API reference for analysis extensions


## DOCUMENT: Stemming

# Stemming and Stop Words

This guide covers using stemmers, stop-word filters, and language-specific
text analysis with Whoosh.

## Stemmers

A stemmer reduces words to their root form (e.g., "running" → "run",
"cats" → "cat"), so that different forms of the same word match in
searches.

### Using StemmerFilter

```python
from whoosh.analysis import RegexTokenizer, StemmerFilter
from whoosh.lang.porter import stem
from whoosh import fields

# English Porter stemmer
stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    content=fields.TEXT(analyzer=stem_analyzer),
)
```

### Snowball Stemmers

Whoosh includes Snowball stemmers for multiple languages:

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.snowball import EnglishStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=EnglishStemmer().stem)
```

### Language-Aware Stemmer Selection

```python
from whoosh.lang import stemmer_for_language, StemmerFilter
from whoosh.analysis import RegexTokenizer

stem = stemmer_for_language("en")
analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

# Or use the analysis StemmingAnalyzer:
from whoosh.analysis import StemmingAnalyzer

analyzer = StemmingAnalyzer("en")
```

### Available Languages

```python
from whoosh.lang import languages, has_stemmer, has_stopwords

print(languages)  # ('ar', 'da', 'nl', 'en', 'fi', 'fr', ...)
print(has_stemmer("en"))  # True
print(has_stopwords("en"))  # True
```

## Stop Words

Stop words are common words (like "the", "a", "and") that are typically
filtered out during indexing since they appear in too many documents to be
useful for ranking.

### Using StopFilter

```python
from whoosh.analysis import RegexTokenizer, StopFilter
from whoosh.lang import stopwords_for_language

# English stop words
stop_words = set(stopwords_for_language("en"))
stop_analyzer = RegexTokenizer() | StopFilter(stoplist=stop_words)

schema = fields.Schema(
    content=fields.TEXT(analyzer=stop_analyzer),
)
```

### Combining Stemming and Stop Words

```python
from whoosh.analysis import StemmingAnalyzer

# StemmingAnalyzer automatically loads stemmer and stopwords for the language
analyzer = StemmingAnalyzer("en")

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

### Custom Stop Words

```python
from whoosh.analysis import RegexTokenizer, StopFilter

# Custom stop words list
custom_stops = frozenset(["the", "a", "an", "foo", "bar"])
analyzer = RegexTokenizer() | StopFilter(stoplist=custom_stops)
```

## StemmingAnalyzer (Recommended)

The `StemmingAnalyzer` combines tokenizer, stemming, and stop word filtering:

```python
from whoosh.analysis import StemmingAnalyzer

# Automatically uses the correct stemmer and stop words for the language
analyzer = StemmingAnalyzer("en")

# You can override defaults
analyzer = StemmingAnalyzer("en",
                            use_stopwords=True,
                            use_stems=True)
```

### StemmingAnalyzer Options

- `lang`: Language code (e.g., `"en"`, `"fr"`, `"de"`)
- `use_stopwords`: Whether to load and apply stop words (default `True`)
- `use_stems`: Whether to apply stemming (default `True`)
- `args`: Arguments passed to the tokenizer
- `kwargs`: Keyword arguments for the stemmer or stopwords

## Language-Specific Considerations

### Arabic (ISRI Stemmer)

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.isri import ISRIStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=ISRIStemmer().stem)
```

### Double Metaphone for Phonetic Matching

```python
from whoosh.analysis import RegexTokenizer, DoubleMetaphoneFilter

analyzer = RegexTokenizer() | DoubleMetaphoneFilter()
```

## Query-Side Stemming

The analyzer is applied at both index time and query time (via the query
parser), so stemming is automatically applied to search terms:

```python
from whoosh.qparser import QueryParser

# If the index uses stemming, queries are stemmed too
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("running cats")  # Will match "run", "cat", etc.
```

## N-gram Analysis

For substring and prefix matching, use N-gram analyzers:

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)
schema = fields.Schema(content=fields.TEXT(analyzer=analyzer))
```

See the [N-grams Guide](ngrams.md) for more details.

## Modern Stemmer Providers (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces a plugin-style stemmer provider system with auto-detection, PyStemmer support, and language-specific analyzers. For full details, see the [Stemmer Providers Guide](stemming-sprint-d.md).


## DOCUMENT: Storage Providers

# Storage Providers

Whoosh-NG provides pluggable storage backends through the
`SyncStorageProvider` / `AsyncStorageProvider` contracts. This allows the
index to be persisted on local disk, SQLite, S3, or a hybrid cache + remote
setup without changing the writer or the index.

## Architecture Overview

### Level 1: SnapshotStorage (Simple)

```
Writer → Local FS → Commit → Upload Segment → S3
Reader → Download Segment → Open locally
```

Very simple to maintain. Use `SnapshotStorage` when you want S3 as a simple
backup/restore target without the complexity of a local cache.

### Level 2: CachedObjectStorage (Recommended for Production)

```
+----------+
|  MinIO   |
+----------+
     ^
     |
 Sync |
     v
+-----------+   Cache Layer   +-----------+
| Searcher  |<--------------->| Writer    |
+-----------+                 +-----------+
        |
        v
 Local SSD
```

- Index lives on SSD
- S3 serves as replication
- Segments are pushed after commit
- Restoration possible at any moment

This is what many modern distributed search systems do.

## Available providers

| Provider | Type | Backend | Use Case |
|----------|------|---------|----------|
| `FileStorage` | sync | local filesystem | Single-node, no cloud |
| `AsyncFileStorage` | async | local filesystem | Single-node async |
| `S3Storage` | sync | S3-compatible | Direct S3 access |
| `SnapshotStorage` | sync | S3-compatible | Simple backup/restore |
| `HybridStorage` | sync | local cache + remote | **Production** (alias: `CachedObjectStorage`) |
| `AsyncHybridStorage` | async | local cache + remote | Production async |

All providers are importable from `whoosh_modern.storage`.

## FileStorage

Local filesystem storage. Keys are relative paths under `root`.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

## AsyncFileStorage

Async variant of `FileStorage`. All operations run on a worker thread via
`asyncio.to_thread` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## S3Storage

S3-compatible blob storage. `boto3` is imported lazily, so it is an optional
dependency. A `client` can be injected for testing.

```python
from whoosh_modern.storage import S3Storage

# Default client (requires boto3 installed and configured)
storage = S3Storage(bucket="my-index-bucket", prefix="segments")

# Or inject a client for testing / custom configuration
storage = S3Storage(
    bucket="my-index-bucket",
    prefix="segments",
    client=my_boto3_client,
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

Install the optional dependency:

```bash
pip install whoosh-ng[s3]
```

## SnapshotStorage

Simple S3 snapshot storage without local cache. This is the simplest
S3-backed storage strategy:

- Write: upload segment directly to S3
- Read: download segment from S3 to local temporary file

Use this when you want S3 as a simple backup/restore target without the
complexity of a local cache.

```python
from whoosh_modern.storage import SnapshotStorage

storage = SnapshotStorage(
    local_path="./index",
    bucket="my-index-bucket",
    prefix="snapshots",
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
```

## HybridStorage / CachedObjectStorage

`HybridStorage` composes a local cache and a remote backend. The remote is
the source of truth; the local cache is a write-through performance layer.

`CachedObjectStorage` is an alias for `HybridStorage` that better conveys
the intent: a local object cache synchronized with S3.

This is the recommended architecture for production deployments with repeated
read patterns.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

# Write-through: remote is source of truth, cache is updated on success
storage.write("segment_1.dat", b"data")

# First read: cache miss → fetch from S3, write-through into cache
data = storage.read("segment_1.dat")

# Second read: cache hit → served from local disk, zero network
data = storage.read("segment_1.dat")

# Force refresh from remote
storage.invalidate("segment_1.dat")

# Warm cache proactively
storage.prefetch(["segment_2.dat", "segment_3.dat"])
```

### Read path

1. local cache hit → return immediately
2. cache miss → read from remote, write-through into cache, return

### Write path

- `remote.write(key, data)` (source of truth)
- on success → `local_cache.write(key, data)`
- on failure → raise before polluting cache

### Cache eviction

The local cache is bounded by `max_cache_size_mb` (default 1024 MB). When
the limit is reached, the oldest entries are evicted using an LRU policy.

### `list_keys`

`list_keys()` uses the remote as source of truth because the cache is only
partial. Pass `include_cache=True` to return the union of remote and cache
keys.

## AsyncHybridStorage

Async variant of `HybridStorage`. Remote operations are executed on a worker
thread via `asyncio.to_thread` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = AsyncHybridStorage(local_cache="./cache", remote=remote)

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")
    keys = await storage.alist_keys()

asyncio.run(main())
```

## Using storage with SearchApplication

```python
from whoosh_modern import SearchApplication, SQLSource
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

app = SearchApplication(
    source=SQLSource(query="SELECT * FROM products", connection=engine),
    storage=storage,
)
app.build()
results = app.index.search("laptop")
```

## Performance Benchmarks

Benchmarks were run against a local MinIO instance using a 28.89 MB Whoosh
index (2 segment files). Results are indicative of relative performance
between strategies on S3-compatible storage.

| Strategy | Backup (MB/s) | Restore (MB/s) | Notes |
|----------|---------------|----------------|-------|
| `1_obj_per_segment` | 39.44 | 139.72 | Best restore throughput; simplest |
| `compressed_zstd` | 31.56 | 133.74 | Lower bandwidth, CPU overhead |
| `hybrid_cache_s3` | 44.97 | 133.61 | Best backup; excellent warm-cache reads |
| `1_obj_per_posting_list` | 0.28 | 4.79 | **Avoid**: millions of small objects kill S3 |

### Recommendations

- **Default**: `S3Storage` with 1 object per segment file. It offers the
  best restore throughput and is the simplest to operate.
- **Production with repeated reads**: `HybridStorage(local_cache, S3Storage)`.
  After the first read, subsequent reads are served from local disk at
  ~133 MB/s.
- **Avoid**: 1 object per posting list. S3 is not optimized for millions of
  tiny objects; latency and cost explode.
- **Compression**: ZSTD reduces transfer size by ~20-30% at the cost of CPU.
  Use it when network bandwidth is the bottleneck, not when CPU is.

### Running the benchmarks

```bash
# Start MinIO
docker run -d --name minio-benchmark -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio:latest server /data --console-address ":9001"

# Run synthetic benchmark
python benchmark/s3_storage_benchmark.py

# Run real Whoosh index benchmark (requires customers CSV)
python benchmark/s3_storage_benchmark_real.py
```


## DOCUMENT: Vector

# Vector Search

Whoosh-NG supports semantic search through vector embeddings. This guide covers setting up and using vector fields.

## Concept

Vector search lets you find documents based on semantic similarity rather than exact keyword matches. You embed documents and queries into a high-dimensional space, then find nearest neighbors.

```
Query embedding  ----\
                      >--- Cosine Similarity ---> Ranked results
Document embedding ---/
```

## Setup

### Define Schema

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VectorField(dimensions=384)  # e.g., all-MiniLM-L6-v2
)
```

## Providers

Whoosh-NG includes multiple vector backends:

| Provider | Description | Use Case |
|----------|-------------|----------|
| `NumpyProvider` | Pure NumPy, cosine similarity | Small to medium indexes |
| `HNSWProvider` | Hierarchical navigable small world | Large indexes, fast ANN |
| `FaissProvider` | Facebook AI Similarity Search | Very large indexes |
| `QdrantProvider` | Qdrant vector DB | Distributed |

### NumpyProvider (Default)

```python
from whoosh.vector import NumpyProvider

provider = NumpyProvider()
provider.add_vector(doc_id, embedding)
results = provider.search(query_embedding, limit=10)
```

## Indexing with Vectors

### Generate Embeddings

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")

embeddings = model.encode([
    "First document content",
    "Second document content"
])
```

### Write Documents

```python
with ix.writer() as writer:
    writer.add_document(
        title="Doc 1",
        content="Python is great",
        embedding=embeddings[0].tolist()
    )
    writer.commit()
```

## Searching with Vectors

### Hybrid Search (Keyword + Vector)

```python
from whoosh.searching import Searcher
from whoosh.vector import VectorProvider

with ix.searcher() as searcher:
    # Semantic search component
    query_embedding = model.encode(["Python tutorial"])[0]
    vector_results = searcher.vector_search(
        "embedding", query_embedding, limit=20
    )

    # Keyword search component
    keyword_query = QueryParser("content", schema).parse("Python")
    keyword_results = searcher.search(keyword_query, limit=20)

    # Combine (e.g., RRF fusion)
    final_results = fuse_results(vector_results, keyword_results)
```

### Pure Vector Search

```python
with ix.searcher() as searcher:
    query_embedding = model.encode(["search query"])[0]
    results = searcher.vector_search(
        "embedding",
        query_embedding,
        limit=10,
        metric="cosine"  # or "euclidean", "dot"
    )
```

## VectorField Options

```python
embedding_field = VectorField(
    dimensions=384,      # Required: embedding dimension
    metric="cosine",     # Similarity metric: cosine, euclidean, dot
    provider="hnsw"      # Provider name from registry
)
```

## Indexing Stream

```python
from whoosh.vector.indexing import VectorIndexer

indexer = VectorIndexer(ix)
indexer.add_document(
    title="Doc",
    content="Content",
    embedding=embedding.tolist()
)
indexer.commit()
```

## Similarity Metrics

| Metric | Description | Range |
|--------|-------------|-------|
| `cosine` | Cosine similarity | [0, 1] (higher is more similar) |
| `euclidean` | Euclidean distance | [0, inf) (lower is more similar) |
| `dot` | Dot product | [-inf, inf] (higher is more similar) |

## Best Practices

1. **Normalize embeddings**: Use cosine similarity with normalized vectors
2. **Choose provider wisely**: Numpy for &lt;100k vectors, HNSW/Faiss for larger
3. **Hybrid search**: Combine vector and keyword search for best results
4. **Cache embeddings**: Pre-compute and store to avoid recomputation
5. **Batch indexing**: Index vectors in batches for efficiency


## DOCUMENT (FR): Analysis

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Analysis API

Classes and functions for turning text into indexable "tokens" (usually words).
Analysis is the first step in the indexing pipeline: an analyzer tokenizes text
and applies zero or more filters to the resulting token stream.

## Overview

Three general categories of objects make up the analysis pipeline:

- **Tokenizers** split text into individual tokens (words, n-grams, identifiers).
  Every tokenizer is callable: `tokenizer(text) -> iterator of Token objects`.
- **Filters** transform one token stream into another. Common operations include
  lowercasing, stop-word removal, stemming, and synonym expansion. Every filter
  is callable: `filter(token_generator) -> token_generator`.
- **Analyzers** compose a tokenizer and zero or more filters into a single unit.
  Every analyzer is callable and can be used directly as a field's `analyzer`
  argument.

Tokenizers and filters are combined using the `|` operator:

```python
my_analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

The first item must be a tokenizer; subsequent items must be filters.

## Composition

### Composable

```python
class whoosh.analysis.Composable
```

Base class for tokenizers and filters, providing `|` composition.

**Attributes:**
- `is_morph (bool)`: Whether this object performs morphological transformation
  (e.g. stemming). Defaults to `False`.

**Methods:**

#### `__or__(self, other)`

Combines this object with `other` using `CompositeAnalyzer`.

```python
analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
```

### CompositeAnalyzer

```python
class whoosh.analysis.CompositeAnalyzer
```

Composed analyzer created by chaining a tokenizer and filters with `|`.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, LowercaseFilter, StopFilter

analyzer = RegexTokenizer() | LowercaseFilter() | StopFilter()
tokens = list(analyzer("Hello world, this is a test"))
```

## Token

```python
class whoosh.analysis.Token
```

Represents a single token (usually a word) extracted from source text.
Tokenizers yield the **same** `Token` object repeatedly (for performance), so
consumers must not hold references between iterations.

**Slots:**

| Attribute | Type | Description |
|-----------|------|-------------|
| `text` | `str` | The text of this token |
| `pos` | `int` | Token position (if `positions=True`) |
| `startchar` | `int` | Start character offset (if `chars=True`) |
| `endchar` | `int` | End character offset (if `chars=True`) |
| `original` | `str` | Original text before filters (if `keeporiginal=True`) |
| `positions` | `bool` | Whether position info was requested |
| `chars` | `bool` | Whether character offsets were requested |
| `stopped` | `bool` | Set by `StopFilter` |
| `boost` | `float` | Token boost factor (default `1.0`) |
| `removestops` | `bool` | Whether stop words should be removed |
| `mode` | `str` | `'index'` or `'query'` |
| `boosts` | `dict` | Per-position boost values (if requested) |
| `tokenize` | `bool` | Whether tokenization should proceed |
| `matched` | `bool` | Used during highlighting |
| `fieldname` | `str` | Field name for this token |

**Methods:**

#### `copy()`

Returns a new `Token` with the same attribute values. Use this if you need to
retain a token between iterations.

```python
def remove_duplicates(stream):
    last = None
    for t in stream:
        if last != t.text:
            yield t
        last = t.text
```

## Utility Functions

### entoken

```python
whoosh.analysis.entoken(
    textstream,
    positions=False,
    chars=False,
    start_pos=0,
    start_char=0,
    **kwargs
) -> Iterator[Token]
```

Converts a sequence of strings into a stream of `Token` objects.

### unstopped

```python
whoosh.analysis.unstopped(tokenstream) -> Iterator[Token]
```

Removes tokens where `token.stopped` is `True`.

## Analyzers

### Analyzer (Base)

```python
class whoosh.analysis.Analyzer
```

Abstract base class for all analyzers. Subclasses implement `__call__`.

### CompositeAnalyzer

Created automatically when you use `|` to compose tokenizers and filters.

### Predefined Analyzers

#### IDAnalyzer

```python
whoosh.analysis.IDAnalyzer(lowercase=False) -> Analyzer
```

Yields the entire input as a single token. Deprecated; use `IDTokenizer` directly.

- `lowercase (bool)`: If True, add a `LowercaseFilter`.

#### KeywordAnalyzer

```python
whoosh.analysis.KeywordAnalyzer(
    lowercase=False,
    commas=False
) -> Analyzer
```

Splits on whitespace or commas. Suitable for field values that are lists of
keywords.

- `lowercase (bool)`: Lowercase each token.
- `commas (bool)`: Split on commas instead of whitespace.

**Example:**
```python
from whoosh.analysis import KeywordAnalyzer

an = KeywordAnalyzer(lowercase=True, commas=True)
list(an("Hello, WORLD, test"))
# => ["hello", "world", "test"]
```

#### RegexAnalyzer

```python
whoosh.analysis.RegexAnalyzer(
    expression=r"\w+(\.?\w+)*",
    gaps=False
) -> Analyzer
```

Deprecated; use `RegexTokenizer` directly.

#### SimpleAnalyzer

```python
whoosh.analysis.SimpleAnalyzer(
    expression=default_pattern,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer` with `LowercaseFilter`.

- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = SimpleAnalyzer()
list(an("Hello there, this is a TEST"))
# => ["hello", "there", "this", "is", "a", "test"]
```

#### StandardAnalyzer

```python
whoosh.analysis.StandardAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and optional `StopFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length (default `None`, no limit).
- `gaps`: If True, split on the expression instead of matching it.

**Example:**
```python
an = StandardAnalyzer()
list(an("Testing is testing and testing"))
# => ["testing", "testing", "testing"]
```

#### StemmingAnalyzer

```python
whoosh.analysis.StemmingAnalyzer(
    expression=default_pattern,
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    gaps=False,
    stemfn=stem,
    ignore=None,
    cachesize=50000
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, optional `StopFilter`, and
`StemFilter`.

- `expression`: Regex pattern for tokens.
- `stoplist`: Words to remove (set to `None` to disable).
- `minsize`: Minimum token length (default `2`).
- `maxsize`: Maximum token length.
- `gaps`: If True, split on the expression instead of matching it.
- `stemfn`: Stemming function (default: Porter stemmer for English).
- `ignore`: Words to not stem (set).
- `cachesize`: Stem cache size (default `50000`). Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
an = StemmingAnalyzer()
list(an("Testing is testing and testing"))
# => ["test", "test", "test"]
```

#### FancyAnalyzer

```python
whoosh.analysis.FancyAnalyzer(
    expression=r"\s+",
    stoplist=STOP_WORDS,
    minsize=2,
    gaps=True,
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
) -> Analyzer
```

Composes `RegexTokenizer`, `IntraWordFilter`, `LowercaseFilter`, and `StopFilter`.
Splits on whitespace and breaks compound words into subwords.

**Example:**
```python
an = FancyAnalyzer()
list(an("Should I call getInt or get_real?"))
# => ["should", "call", "get", "int", "get", "real"]
```

#### LanguageAnalyzer

```python
whoosh.analysis.LanguageAnalyzer(
    lang,
    expression=default_pattern,
    gaps=False,
    cachesize=50000
) -> Analyzer
```

Configures a language-specific analyzer with `LowercaseFilter`, `StopFilter`,
and `StemFilter`.

- `lang`: Language code (e.g., `"en"`, `"es"`, `"fr"`).
- `expression`: Regex pattern for tokens.
- `gaps`: If True, split on the expression instead of matching it.
- `cachesize`: Stem cache size.

Available languages: `ar`, `da`, `nl`, `en`, `fi`, `fr`, `de`, `hu`, `it`,
`no`, `pt`, `ro`, `ru`, `es`, `sv`, `tr`.

See `whoosh.lang` for `has_stemmer()` and `has_stopwords()` helper functions.

## Tokenizers

All tokenizers inherit from `Tokenizer`.

### Tokenizer

```python
class whoosh.analysis.Tokenizer
```

Base class for tokenizers. Each tokenizer is callable and yields `Token`
objects.

### RegexTokenizer

```python
class whoosh.analysis.RegexTokenizer(
    expression=default_pattern,
    gaps=False
)
```

Uses a regular expression to extract tokens from text. Each match of the
expression equals one token; group 0 (the entire match) is used as the text.

- `expression`: Compiled regex or pattern string.
- `gaps`: If True, split on the expression rather than matching it.

**Example:**
```python
from whoosh.analysis import RegexTokenizer

rext = RegexTokenizer()
list(rext("hi there 3.141 big-time under_score"))
# => ["hi", "there", "3.141", "big", "time", "under_score"]
```

### IDTokenizer

```python
class whoosh.analysis.IDTokenizer
```

Yields the entire input string as a single token. Used for indexed but
untokenized fields (e.g., document paths).

### CharsetTokenizer

```python
class whoosh.analysis.CharsetTokenizer(charmap)
```

Tokenizes and translates text according to a character mapping dictionary.
Characters that map to `None` are treated as token break characters.

- `charmap`: Mapping from integer character codes to unicode characters
  (as used by `unicode.translate()`).

### PathTokenizer

```python
class whoosh.analysis.PathTokenizer(expression="[^/]+")
```

Tokenizes path strings into hierarchical prefixes. Given `"/a/b/c"`, yields
`["/a", "/a/b", "/a/b/c"]`.

### NgramTokenizer

```python
class whoosh.analysis.NgramTokenizer(minsize, maxsize=None)
```

Splits input text into N-grams instead of words. Unlike `RegexTokenizer`, this
tokenizer does not use a regex, so grams may include whitespace and punctuation.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).

**Example:**
```python
from whoosh.analysis import NgramTokenizer

ngt = NgramTokenizer(4)
list(ngt("hi there"))
# => ["hi t", "i th", " the", "ther", "here"]
```

### CachedRegexTokenizer

```python
class whoosh.analysis.CachedRegexTokenizer(
    expression=default_pattern,
    gaps=False,
    maxsize=8192
)
```

A `RegexTokenizer` wrapper that caches tokenization results for repeated
strings, trading memory for speed.

- `expression`: Regex pattern.
- `gaps`: If True, split on the expression.
- `maxsize`: Maximum cache size (LRU eviction when exceeded).

### SpaceSeparatedTokenizer

```python
whoosh.analysis.SpaceSeparatedTokenizer() -> RegexTokenizer
```

Returns a `RegexTokenizer` that splits on whitespace.

### CommaSeparatedTokenizer

```python
whoosh.analysis.CommaSeparatedTokenizer() -> CompositeAnalyzer
```

Returns a composed analyzer that splits on commas and strips whitespace.

## Filters

All filters inherit from `Filter`.

### Filter

```python
class whoosh.analysis.Filter
```

Base class for filters. Subclasses implement `__call__(self, tokens)` which
takes a token generator and returns a token generator.

- `is_morph (bool)`: Set to `True` for morphological filters (e.g., stemming).
  This allows the filter to be bypassed during query analysis if desired.

### STOP_WORDS

```python
whoosh.analysis.STOP_WORDS
```

A frozenset of common English stop words: `"a"`, `"an"`, `"and"`, `"the"`, etc.
Used as the default stoplist for `StopFilter` and `StandardAnalyzer`.

### url_pattern

```python
whoosh.analysis.url_pattern
```

A compiled regex useful for URL filtering.

### LowercaseFilter

```python
class whoosh.analysis.LowercaseFilter
```

Lowercases token text using `unicode.lower()`.

**Example:**
```python
rext = RegexTokenizer() | LowercaseFilter()
list(rext("This is a TEST"))
# => ["this", "is", "a", "test"]
```

### StopFilter

```python
class whoosh.analysis.StopFilter(
    stoplist=STOP_WORDS,
    minsize=2,
    maxsize=None,
    renumber=True,
    lang=None
)
```

Marks and optionally removes stop words from the token stream.

- `stoplist`: Set of words to filter out (defaults to `STOP_WORDS`).
- `minsize`: Minimum token length; shorter tokens are removed (default `2`).
- `maxsize`: Maximum token length; longer tokens are removed (default `None`).
- `renumber`: Renumber positions to account for removed tokens (default `True`).
- `lang`: If set, loads stop words for the given language code.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StopFilter

stopper = RegexTokenizer() | StopFilter()
list(stopper("this is a test"))
# => ["test"]
```

### StripFilter

```python
class whoosh.analysis.StripFilter
```

Calls `unicode.strip()` on each token's text.

### CharsetFilter

```python
class whoosh.analysis.CharsetFilter(charmap)
```

Translates token text using `unicode.translate()` with the given character map.
Useful for case folding and accent folding.

- `charmap`: Dictionary mapping character ordinals to unicode characters.

**Example:**
```python
from whoosh.support.charset import accent_map

rext = RegexTokenizer() | CharsetFilter(accent_map)
list(rext("café"))
# => ["cafe"]
```

### DelimitedAttributeFilter

```python
class whoosh.analysis.DelimitedAttributeFilter(
    delimiter="^",
    attribute="boost",
    default=1.0,
    type=float
)
```

Looks for delimiter characters in token text and extracts data after the
delimiter into a named token attribute.

- `delimiter`: Separator character (default `"^"`).
- `attribute`: Attribute name on the token (default `"boost"`).
- `default`: Default value if no delimiter is found (default `1.0`).
- `type`: Type to cast the extracted value (default `float`).

**Example:**
```python
from whoosh.analysis import RegexTokenizer, DelimitedAttributeFilter

daf = DelimitedAttributeFilter()
an = RegexTokenizer(r"\S+") | daf
for t in an(u"image 3.14^2 render"):
    print(t.text, t.boost)
# image 1.0
# 3.14 2.0
# render 1.0
```

### SubstitutionFilter

```python
class whoosh.analysis.SubstitutionFilter(pattern, replacement)
```

Performs regex substitution on token text using `re.sub()`.

- `pattern`: Pattern string or compiled regex.
- `replacement`: Replacement text.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, SubstitutionFilter

# Remove hyphens
ana = RegexTokenizer(r"\S+") | SubstitutionFilter("-", "")
```

### MultiFilter

```python
class whoosh.analysis.MultiFilter(**kwargs)
```

Selects between two or more sub-filters based on the `mode` attribute of the
token stream. Useful for using different filters during indexing vs. querying.

- Keyword arguments map mode names to filter instances.

**Example:**
```python
from whoosh.analysis import MultiFilter, IntraWordFilter

iwf_index = IntraWordFilter(mergewords=True, mergenums=True)
iwf_query = IntraWordFilter(mergewords=False, mergenums=False)
mf = MultiFilter(index=iwf_index, query=iwf_query)
```

### TeeFilter

```python
class whoosh.analysis.TeeFilter(*filters)
```

Interleaves the results of two or more filter chains. Requires at least two
filters. Note: this filter is slow because it creates token copies.

**Example:**
```python
# Lowercase in one branch, reverse in another
f1 = LowercaseFilter()
f2 = ReverseTextFilter()
ana = RegexTokenizer(r"\S+") | TeeFilter(f1, f2)
```

### ReverseTextFilter

```python
class whoosh.analysis.ReverseTextFilter
```

Reverses the text of each token.

**Example:**
```python
an = RegexTokenizer() | ReverseTextFilter()
list(an("hello there"))
# => ["olleh", "ereht"]
```

### PassFilter

```python
class whoosh.analysis.PassFilter
```

Identity filter; passes tokens through unchanged.

### LoggingFilter

```python
class whoosh.analysis.LoggingFilter(logger=None)
```

Prints debug log entries for every token that passes through.

- `logger`: Logger instance (defaults to `whoosh.analysis` logger).

## Intraword Filters

### IntraWordFilter

```python
class whoosh.analysis.IntraWordFilter(
    delims="-_'\"()!@#$%^&*[]{}<>\\|;:,./?`~+=",
    splitwords=True,
    splitnums=True,
    mergewords=False,
    mergenums=False
)
```

Splits words into subwords and performs optional merging. Based on
WordDelimiterFilter in Solr.

- `delims`: String of delimiter characters.
- `splitwords`: Split at case transitions (e.g., `PowerShot` → `Power`, `Shot`).
- `splitnums`: Split at letter-number transitions (e.g., `SD500` → `SD`, `500`).
- `mergewords`: Merge consecutive alphabetic subwords.
- `mergenums`: Merge consecutive numeric subwords.

### CompoundWordFilter

```python
class whoosh.analysis.CompoundWordFilter(wordset, keep_compound=True)
```

Breaks compound tokens into their constituent parts if they match words in the
given wordset. Useful for agglutinative languages and trademarks.

- `wordset`: A set (or any `__contains__` object) of known words.
- `keep_compound`: If True, keep the original compound token in the stream.

### BiWordFilter

```python
class whoosh.analysis.BiWordFilter(sep="-")
```

Merges adjacent tokens into bigram tokens. Useful for pseudo-phrase searching.

- `sep`: Separator string for bigrams.

### ShingleFilter

```python
class whoosh.analysis.ShingleFilter(size=2, sep="-")
```

Merges N adjacent tokens into multi-word tokens (shingles).

- `size`: Number of tokens to combine.
- `sep`: Separator string.

**Note:** For `size=2`, `BiWordFilter` is faster.

## Morphological Filters

### StemFilter

```python
class whoosh.analysis.StemFilter(
    stemfn=stem,
    lang=None,
    ignore=None,
    cachesize=50000
)
```

Stems tokens using the Porter stemming algorithm (or a language-specific
stemmer if `lang` is specified).

- `stemfn`: Stemming function (default: Porter stemmer).
- `lang`: Language code to override `stemfn` with a Snowball stemmer.
- `ignore`: Set of words to not stem (defaults to stemming all words).
- `cachesize`: Cache size for stemmed words. Use `-1` for unbounded,
  `None` for no cache.

**Example:**
```python
from whoosh.analysis import RegexTokenizer, StemFilter

stemmer = RegexTokenizer() | StemFilter()
list(stemmer("fundamentally willows"))
# => ["fundament", "willow"]
```

### PyStemmerFilter

```python
class whoosh.analysis.PyStemmerFilter(
    lang="english",
    ignore=None,
    cachesize=10000
)
```

Subclass of `StemFilter` that uses the third-party `py-stemmer` library.
Requires the py-stemmer package to be installed.

**Methods:**
- `algorithms()`: Returns available stemming algorithms from py-stemmer.

### DoubleMetaphoneFilter

```python
class whoosh.analysis.DoubleMetaphoneFilter(
    primary_boost=1.0,
    secondary_boost=0.5,
    combine=False
)
```

Encodes tokens using Lawrence Philips's Double Metaphone algorithm. Useful
for phonetic matching of names and places.

- `primary_boost`: Boost factor for the primary code token.
- `secondary_boost`: Boost factor for the secondary code token.
- `combine`: If True, keep the original token alongside the encoded tokens.

## N-gram Filters and Analyzers

### NgramFilter

```python
class whoosh.analysis.NgramFilter(minsize, maxsize=None, at=None)
```

Splits token text into N-grams of varying sizes.

- `minsize`: Minimum N-gram size.
- `maxsize`: Maximum N-gram size (defaults to `minsize`).
- `at`: `'start'` for prefix grams, `'end'` for suffix grams, or `None`
  for all position grams.

### NgramAnalyzer

```python
whoosh.analysis.NgramAnalyzer(minsize, maxsize=None) -> Analyzer
```

Composes `NgramTokenizer` with `LowercaseFilter`.

### NgramWordAnalyzer

```python
whoosh.analysis.NgramWordAnalyzer(
    minsize,
    maxsize=None,
    tokenizer=None,
    at=None
) -> Analyzer
```

Composes `RegexTokenizer`, `LowercaseFilter`, and `NgramFilter`. Use this
when you want sub-word n-grams (without whitespace) rather than raw
character n-grams.


## DOCUMENT (FR): Automata

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Automata API

Module for constructing and manipulating finite state automata (FSAs),
including NFAs, DFAs, finite state transducers (FSTs), Levenshtein
automata, and regular expression automata. Used internally for spelling
correction, fuzzy term queries, and term dictionary operations.

The automata module is a refactored package with submodules. All classes
and functions are importable directly from `whoosh.automata`.

## Module Functions

### `parse_glob`

```python
whoosh.automata.parse_glob(pattern, _glob_multi="*", _glob_single="?", _glob_range1="[", _glob_range2="]") -> NFA
```

Parses a glob-style pattern string and returns an NFA that matches strings
matching the pattern.

**Parameters:**
- `pattern`: Glob pattern string (`*` matches any sequence, `?` matches any
  single character).
- `_glob_multi`, `_glob_single`: Override the wildcard characters.
- `_glob_range1`, `_glob_range2`: Override the range syntax brackets.

### `glob_automaton`

```python
whoosh.automata.glob_automaton(pattern) -> NFA
```

Convenience function that parses a glob pattern and returns an NFA.

## FSA (Finite State Automaton) Classes

### `FSA`

```python
class whoosh.automata.FSA(initial)
```

Base class for finite state automata.

**Constructor:**
- `initial`: The initial state.

**Attributes:**
- `initial`: Initial state.
- `transitions`: Dict mapping source states to dicts mapping labels to
  target states.
- `final_states`: Set of accepting (final) states.

**Methods:**
- `__eq__(other)`: Compares initial state, final states, and transitions.
- `all_states()`: Returns a set of all states reachable in the automaton.
- `all_labels()`: Returns a set of all transition labels.
- `get_labels(src)`: Yields all labels leaving state `src`.
- `generate_all(state=None, sofar="")`: Yields all strings accepted by the
  automaton.
- `move(state, label)`: Returns the state reached by following `label` from
  `state`, or `None`.
- `moves(state, labels)`: Yields `(label, next_state)` pairs.
- `next(state)`: Yields target states reachable from `state` via any label.
- `is_final(state)`: Returns `True` if `state` is a final state.
- `start()`: Returns the initial state.
- `has_path_to(target)`: Returns `True` if there is a path to `target`.

### `Marker`

```python
class whoosh.automata.Marker(name)
```

Marker object used as a special transition label in NFAs (e.g., `ANY`,
`EPSILON`).

### `EPSILON`

```python
whoosh.automata.EPSILON = Marker("EPSILON")
```

Special marker representing an epsilon transition (no input consumed).

### `ANY`

```python
whoosh.automata.ANY = Marker("ANY")
```

Special marker representing a transition that matches any input character.

### `NFA`

```python
class whoosh.automata.NFA(initial)
```

Nondeterministic Finite Automaton. Extends `FSA` with epsilon transitions
and NFA-specific construction methods.

**Methods:**
- `add_transition(src, label, dst)`: Adds a transition from `src` to `dst`
  consuming `label`.
- `add_final_state(state, final=True)`: Marks `state` as a final/accepting
  state.
- `epsilon_closure(state)`: Returns the set of states reachable from `state`
  via epsilon transitions.
- `to_dfa()`: Converts this NFA to an equivalent DFA and returns it.

### `DFA`

```python
class whoosh.automata.DFA(initial)
```

Deterministic Finite Automaton. Extends `FSA` with DFA-specific operations.

**Methods:**
- `next_valid_string(string)`: Finds the lexicographically smallest string
  accepted by the DFA that is greater than or equal to `string`.
- `to_dfa()`: Returns self (already a DFA).

### `renumber_dfa`

```python
whoosh.automata.renumber_dfa(dfa, base=0) -> DFA
```

Renumerates the states of a DFA to integers starting at `base`.

### `u_to_utf8`

```python
whoosh.automata.u_to_utf8(dfa, base=0) -> DFA
```

Converts a Unicode DFA to a UTF-8 DFA.

### `find_all_matches`

```python
whoosh.automata.find_all_matches(dfa, lookup_func, first=unull)
```

Yields all strings accepted by the DFA, using `lookup_func` to determine
which strings exist in the dictionary.

**Parameters:**
- `dfa`: A deterministic finite automaton.
- `lookup_func`: Function called with each candidate string; returns the
  string if found in the dictionary.
- `first`: First string to start matching from (default `chr(0)`).

### `reverse_nfa`

```python
whoosh.automata.reverse_nfa(n) -> NFA
```

Returns the reverse of an NFA (reversed transitions, swapped initial
and final states).

### `product`

```python
whoosh.automata.product(dfa1, op, dfa2) -> DFA
```

Computes the product of two DFAs using a binary operation.

**Parameters:**
- `dfa1`, `dfa2`: Input DFAs.
- `op`: A function `(set1, set2) -> set` computing the output final states
  from the two input final state sets.

### `intersection`

```python
whoosh.automata.intersection(dfa1, dfa2) -> DFA
```

Returns the intersection of two DFAs.

### `union`

```python
whoosh.automata.union(dfa1, dfa2) -> DFA
```

Returns the union of two DFAs.

### `epsilon_nfa`

```python
whoosh.automata.epsilon_nfa() -> NFA
```

Returns an NFA that accepts only the empty string.

### `dot_nfa`

```python
whoosh.automata.dot_nfa() -> NFA
```

Returns an NFA that accepts any single character.

### `basic_nfa`

```python
whoosh.automata.basic_nfa(label) -> NFA
```

Returns an NFA that accepts exactly the string `label`.

### `charset_nfa`

```python
whoosh.automata.charset_nfa(labels) -> NFA
```

Returns an NFA that accepts any single character in `labels`.

### `string_nfa`

```python
whoosh.automata.string_nfa(string) -> NFA
```

Returns an NFA that accepts exactly `string`.

### `choice_nfa`

```python
whoosh.automata.choice_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts strings accepted by either `n1` or `n2`.

### `concat_nfa`

```python
whoosh.automata.concat_nfa(n1, n2) -> NFA
```

Returns an NFA that accepts the concatenation of `n1` and `n2`.

### `star_nfa`

```python
whoosh.automata.star_nfa(n) -> NFA
```

Returns an NFA that accepts zero or more repetitions of `n`.

### `plus_nfa`

```python
whoosh.automata.plus_nfa(n) -> NFA
```

Returns an NFA that accepts one or more repetitions of `n`.

### `optional_nfa`

```python
whoosh.automata.optional_nfa(n) -> NFA
```

Returns an NFA that accepts zero or one occurrence of `n`.

### `strings_dfa`

```python
whoosh.automata.strings_dfa(strings) -> DFA
```

Constructs a minimal DFA that accepts exactly the given strings.

### `add_suffix`

```python
whoosh.automata.add_suffix(dfa, nodes, last, downto, seen)
```

Internal function for adding suffixes to a trie during DFA construction.

## Levenshtein Automata

### `levenshtein_automaton`

```python
whoosh.automata.levenshtein_automaton(term, k, prefix=0) -> NFA
```

Constructs an NFA that matches all strings within edit distance `k` of
`term`. This is the core function for fuzzy term queries and spelling
suggestions.

**Parameters:**
- `term`: The reference string to compute edit distance from.
- `k`: Maximum edit distance (number of insertions, deletions, or
  substitutions).
- `prefix`: If positive, require matched strings to share this length of
  prefix with `term` (speeds up matching significantly).

**Returns:** An NFA that can be converted to a DFA via `.to_dfa()`.

```python
from whoosh.automata import levenshtein_automaton

nfa = levenshtein_automaton("hello", k=1, prefix=0)
dfa = nfa.to_dfa()
```

## RegEx

### `parse`

```python
whoosh.automata.parse(pattern) -> NFA
```

Parses a regular expression pattern string and returns an NFA.

**Parameters:**
- `pattern`: A regex pattern string (Python `re`-style syntax).

### `RegexBuilder`

```python
class whoosh.automata.RegexBuilder(pattern)
```

Helper class for building NFAs from regex patterns.

## FST (Finite State Transducer) Classes

### `Values`

```python
class whoosh.automata.Values
```

Abstract base class for value types stored in FST arcs.

### `IntValues`

```python
class whoosh.automata.IntValues
```

Stores integer values in FST arcs.

### `SequenceValues`

```python
class whoosh.automata.SequenceValues
```

Base class for value types that store sequences of values.

### `BytesValues`

```python
class whoosh.automata.BytesValues
```

Stores byte string values in FST arcs.

### `ArrayValues`

```python
class whoosh.automata.ArrayValues
```

Stores arrays of values in FST arcs.

### `IntListValues`

```python
class whoosh.automata.IntListValues
```

Stores lists of integers in FST arcs.

### `Node`

```python
class whoosh.automata.Node
```

Base class for nodes in an FST.

### `ComboNode`

```python
class whoosh.automata.ComboNode
```

Base class for nodes that combine multiple sub-nodes (intersection, union).

### `UnionNode`

```python
class whoosh.automata.UnionNode
```

A node that represents the union of multiple sub-nodes.

### `IntersectionNode`

```python
class whoosh.automata.IntersectionNode
```

A node that represents the intersection of multiple sub-nodes.

### `BaseCursor`

```python
class whoosh.automata.BaseCursor
```

Base class for cursors that iterate over FST contents.

### `Cursor`

```python
class whoosh.automata.Cursor
```

Concrete cursor for iterating over an FST, supporting `next()`, `find()`,
`text()`, and other navigation methods.

### `UncompiledNode`

```python
class whoosh.automata.UncompiledNode
```

Represents an FST node that has not yet been compiled into a binary
representation. Used during FST construction.

### `Arc`

```python
class whoosh.automata.Arc
```

Represents a single arc in an FST, with a label, target node, and associated
value.

### `GraphWriter`

```python
class whoosh.automata.GraphWriter
```

Writes an FST to a binary file on disk or to an in-memory buffer.

### `BaseGraphReader`

```python
class whoosh.automata.BaseGraphReader
```

Base class for reading FSTs from disk.

### `GraphReader`

```python
class whoosh.automata.GraphReader
```

Concrete reader for FSTs stored on disk. Supports `find()`, `next()`, and
`text()` for navigating the graph.

### `to_labels`

```python
whoosh.automata.to_labels(key)
```

Converts a key (string, int, etc.) into a list of FST arc labels.

### `within`

```python
whoosh.automata.within(graph, text, k=1, prefix=0, address=None)
```

Uses a pre-built FST and a Levenshtein automaton to find all keys in the
graph within edit distance `k` of `text`.

**Parameters:**
- `graph`: A `GraphReader` instance.
- `text`: The search term.
- `k`: Maximum edit distance.
- `prefix`: Required shared prefix length.
- `address`: Optional starting address in the graph.

### `dump_graph`

```python
whoosh.automata.dump_graph(graph, address=None, tab=0, out=None)
```

Debug utility that prints the structure of an FST to stdout or a file.

### `FileVersionError`

```python
class whoosh.automata.FileVersionError
```

Raised when reading an FST file with an incompatible version.

### `InactiveCursor`

```python
class whoosh.automata.InactiveCursor
```

Raised when operating on a cursor that is not at a valid position.


## DOCUMENT (FR): Backends

# API Backends

Architecture de stockage pliable via backends.

## FileBackend (défaut)

```python
class whoosh.backends.file.FileBackend
```

Backend par défaut stockant les segments comme fichiers sur disque.

### Options

| Paramètre | Description |
|-----------|-------------|
| `storage` | Instance de storage (FileStorage par défaut) |
| `limitmb` | Taille maximum des segments (MiB) |

**Exemple:**
```python
from whoosh import index

ix = index.create_in("indexdir", schema)
# Utilise FileBackend implicitement
```

## SQLiteBackend

```python
class whoosh.backends.sqlite.SQLiteBackend
```

Stocke l'index entier dans une base de données SQLite.

### Options

| Paramètre | Description |
|-----------|-------------|
| `storage` | `SQLiteStorage(path)` |
| `writethrough` | Écriture synchrone |

**Exemple:**
```python
from whoosh.backends.sqlite import SQLiteStorage, SQLiteBackend

storage = SQLiteStorage("mon_index.db")
backend = SQLiteBackend(storage=storage)

ix = backend.create_index(schema)
```

## MemoryBackend

```python
class whoosh.backends.memory.MemoryBackend
```

Backend en mémoire (tests uniquement, données perdues au redémarrage).

## Classes Storage

### FileStorage

```python
class whoosh.store.FileStorage
```

Gère les fichiers sur disque.

### SQLiteStorage

```python
class whoosh.store.SQLiteStorage(db_path)
```

Gère le stockage SQLite.

## ProviderRegistry

```python
class whoosh.registry.ProviderRegistry
```

Registre pour les providers de stockage:

```python
from whoosh.registry import ProviderRegistry

ProviderRegistry.register("sqlite", SQLiteBackend(), "mon_app")
```


## DOCUMENT (FR): Classify

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Classify API

Classes and functions for classifying and extracting information from
documents. This module provides query expansion models, similarity
functions (shingling, simhash), and clustering algorithms.

## Expansion Models

### `ExpansionModel`

```python
class whoosh.classify.ExpansionModel(doc_count, field_length)
```

Abstract base class for query expansion models. Subclass to implement custom
expansion scoring.

**Constructor:**
- `doc_count`: Total number of documents in the collection.
- `field_length`: Total length of the field across all documents.

**Computed Attributes:**
- `N`: Document count.
- `collection_total`: Total field length.
- `mean_length`: Average field length (`collection_total / N`).

**Methods:**
- `normalizer(maxweight, top_total)`: Returns a normalization factor.
- `score(weight_in_top, weight_in_collection, top_total)`: Returns the
  expansion score for a term.

### `Bo1Model`

```python
class whoosh.classify.Bo1Model(doc_count, field_length)
```

Bayesian One-Poisson expansion model. One of the standard query expansion
models.

### `Bo2Model`

```python
class whoosh.classify.Bo2Model(doc_count, field_length)
```

Bayesian Two-Poisson expansion model. Another standard query expansion model.

### `KLModel`

```python
class whoosh.classify.KLModel(doc_count, field_length)
```

Kullback-Leibler divergence-based expansion model.

## Expander

### `Expander`

```python
class whoosh.classify.Expander(
    ixreader,
    fieldname,
    model=Bo1Model
)
```

Uses an `ExpansionModel` to expand the set of query terms based on the top N
result documents.

**Constructor:**
- `ixreader`: An `IndexReader` object.
- `fieldname`: The name of the field to expand terms from.
- `model`: An `ExpansionModel` class or instance. Defaults to `Bo1Model`.

**Methods:**

#### `add(vector)`

Adds forward-index information about one of the "top N" documents.

- `vector`: A series of `(text, weight)` tuples, such as is returned by
  `Reader.vector_as("weight", docnum, fieldname)`.

#### `add_document(docnum)`

Adds a document's term vector to the expander. If the field has a term vector,
uses it; otherwise falls back to stored field text.

#### `add_text(string)`

Adds a text string by indexing it with the field's analyzer.

#### `expanded_terms(number, normalize=True)`

Returns the N most important terms in the vectors added so far, ranked by
the expansion model's score.

- `number`: Number of terms to return.
- `normalize`: Whether to normalize weights.
- Returns: List of `(term, weight)` tuples, sorted by weight descending.

```python
from whoosh.classify import Expander, Bo1Model

expander = Expander(ix.reader(), "content")
for docnum in results.ids()[:10]:
    expander.add_document(docnum)

for word, weight in expander.expanded_terms(5):
    print(word, weight)
```

## Similarity Functions

### `shingles`

```python
whoosh.classify.shingles(input, size=2) -> iterable
```

Generates `(shingle, frequency)` pairs from a string by sliding a window of
the given size over the input.

**Parameters:**
- `input`: The input string.
- `size`: The shingle size (default `2`).

```python
from whoosh.classify import shingles

for shingle, freq in shingles("hello world", size=2):
    print(shingle, freq)
```

### `simhash`

```python
whoosh.classify.simhash(features, hashbits=32) -> int
```

Computes a simhash (perceptual hash) from a sequence of weighted features.
Simhashes that are similar produce similar hash values, allowing fast
near-duplicate detection via Hamming distance.

**Parameters:**
- `features`: Iterable of `(feature, weight)` tuples.
- `hashbits`: Number of bits in the hash (default `32`).
- Returns: An integer hash value.

```python
from whoosh.classify import shingles, simhash

h1 = simhash(shingles(text1))
h2 = simhash(shingles(text2))
from whoosh.classify import hamming_distance
dist = hamming_distance(h1, h2)
```

### `hamming_distance`

```python
whoosh.classify.hamming_distance(first_hash, other_hash, hashbits=32) -> int
```

Computes the Hamming distance between two hash values. A small distance
indicates high similarity.

**Parameters:**
- `first_hash`: First hash integer.
- `other_hash`: Second hash integer.
- `hashbits`: Number of bits in the hashes (default `32`).

## Clustering

### `kmeans`

```python
whoosh.classify.kmeans(
    data,
    k,
    t=0.0001,
    distfun=None,
    maxiter=50,
    centers=None
) -> (labels, centroids)
```

One-dimensional K-means clustering. Assigns each data point to the nearest
of `k` centroids and returns cluster labels and final centroids.

**Parameters:**
- `data`: List of data points (numeric values).
- `k`: Number of clusters.
- `t`: Tolerance; stops if centroid changes are below this value.
- `distfun`: Optional distance function (unused if `None`).
- `maxiter`: Maximum iterations (default `50`).
- `centers`: Optional list of initial centroids. If `None`, selects `k`
  random points from `data`.

**Returns:** A tuple `(labels, centroids)` where `labels` is a list of
cluster assignments per data point and `centroids` is the list of final
centroid positions.

### `two_pass_variance`

```python
whoosh.classify.two_pass_variance(data) -> float
```

Computes the sample variance of a data list using the two-pass algorithm
(first pass computes the mean, second pass accumulates squared deviations).

### `weighted_incremental_variance`

```python
whoosh.classify.weighted_incremental_variance(data_weight_pairs) -> float
```

Computes the weighted variance incrementally from a sequence of
`(value, weight)` pairs.

### `swin`

```python
whoosh.classify.swin(data, size) -> list
```

Sliding window clustering that groups data points where the range (max - min)
within a window of `size` is below a threshold. Uses variance for ranking.

**Parameters:**
- `data`: Sorted list of data points.
- `size`: Maximum window range (max - min) for clustering.

**Returns:** A list of `(left, right, count, variance)` tuples representing
clusters, sorted by count descending then by variance ascending.


## DOCUMENT (FR): Codecs

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Codecs API

Classes and interfaces for how Whoosh writes and reads the inverted index,
postings, and per-document values. The codecs module is a refactored package
exposing the same public API as the former monolithic module.

## Module Functions

### `default_codec`

```python
whoosh.codec.default_codec(*args, **kwargs) -> Codec
```

Returns the default codec used by the index. Currently returns a
`W3Codec` instance.

```python
from whoosh.codec import default_codec
codec = default_codec()
```

## Exceptions

### `OutOfOrderError`

```python
whoosh.codec.OutOfOrderError
```

Raised when documents are added to a field out of order. Fields must
receive documents in ascending docnum order.

## Base Classes

### `Codec`

```python
class whoosh.codec.Codec
```

Abstract base class for index codecs. Subclasses implement methods for
writing and reading the index format.

**Class Attributes:**
- `length_stats (bool)`: If `True`, the codec stores per-document field
  length statistics. Default `True`.

**Methods:**

#### `per_document_writer(storage, segment)`

Abstract. Returns a `PerDocumentWriter` for writing per-document values
(columns, term vectors) to the given segment.

#### `field_writer(storage, segment)`

Abstract. Returns a `FieldWriter` for writing postings to the given segment.

#### `postings_writer(dbfile, byteids=False)`

Abstract. Returns a `PostingsWriter` for writing posting lists to `dbfile`.

#### `postings_reader(dbfile, terminfo, format_, term=None, scorer=None)`

Abstract. Returns a `Matcher` for reading postings from `dbfile`.

#### `automata(storage, segment)`

Returns an `Automata` instance for spelling correction using automata-based
edit distance. Default returns a base `Automata()` object.

#### `terms_reader(storage, segment)`

Abstract. Returns a `TermsReader` for reading the term dictionary and
postings of the given segment.

#### `per_document_reader(storage, segment)`

Abstract. Returns a `PerDocumentReader` for reading per-document values
from the given segment.

#### `new_segment(storage, indexname)`

Abstract. Creates and returns a new `Segment` object for the given storage
and index name.

### `WrappingCodec`

```python
class whoosh.codec.WrappingCodec(child)
```

A `Codec` that delegates all operations to a child codec. Useful for
creating codec wrappers that modify or intercept specific operations.

**Constructor:**
- `child`: The underlying `Codec` instance to wrap.

All methods delegate to the child codec:
`per_document_writer()`, `field_writer()`, `postings_writer()`,
`postings_reader()`, `automata()`, `terms_reader()`, `per_document_reader()`,
`new_segment()`.

## Writer Classes

### `PerDocumentWriter`

```python
class whoosh.codec.PerDocumentWriter
```

Abstract base class for writing per-document values (columns, term vectors).

**Methods:**

#### `start_doc(docnum)`

Abstract. Called when starting to write a new document.

#### `add_field(fieldname, fieldobj, value, length)`

Abstract. Adds a field value to the current document.

#### `add_column_value(fieldname, columnobj, value)`

Abstract. Adds a column value. Raises `NotImplementedError` if the codec
doesn't support columns.

#### `add_vector_items(fieldname, fieldobj, items)`

Abstract. Adds term vector items.

#### `add_vector_matcher(fieldname, fieldobj, vmatcher)`

Convenience method that reads items from a `Matcher` and calls
`add_vector_items()`.

#### `finish_doc()`

Called when finishing a document. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `FieldWriter`

```python
class whoosh.codec.FieldWriter
```

Abstract base class for writing postings (inverted index) data.

**Methods:**

#### `add_postings(schema, lengths, items)`

Translates a generator of `(fieldname, btext, docnum, weight, vbytes)`
tuples into calls to `start_field()`, `start_term()`, `add()`,
`finish_term()`, and `finish_field()`.

**Parameters:**
- `schema`: The `Schema` object.
- `lengths`: Optional `FieldLengthTable` for document field lengths.
- `items`: Iterable of posting tuples.

#### `start_field(fieldname, fieldobj)`

Abstract. Called when starting a new field.

#### `start_term(text)`

Abstract. Called when starting a new term within a field.

#### `add(docnum, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `add_spell_word(fieldname, text)`

Called to add a word to the spelling index. Default does nothing.

#### `finish_term()`

Abstract. Called when finishing a term.

#### `finish_field()`

Called when finishing a field. Default does nothing.

#### `close()`

Called when done writing. Default does nothing.

### `PostingsWriter`

```python
class whoosh.codec.PostingsWriter
```

Abstract base class for writing posting lists (the inverted index).

**Methods:**

#### `start_postings(format_, terminfo)`

Abstract. Starts writing postings for a new term.

#### `add_posting(id_, weight, vbytes, length=None)`

Abstract. Adds a posting to the current term.

#### `finish_postings(allow_compact=True)`

Called when finished writing postings. Default does nothing.

#### `written()`

Abstract. Returns `True` if this writer has already written to disk.

## Reader Classes

### `FieldCursor`

```python
class whoosh.codec.FieldCursor
```

Abstract base class for iterating over terms in a field.

**Methods:**
- `first()`: Move to the first term.
- `find(string)`: Find a term matching or closest to `string`.
- `next()`: Move to the next term.
- `term()`: Returns the current term's text.

### `EmptyCursor`

```python
class whoosh.codec.EmptyCursor
```

A `FieldCursor` representing an empty field. All methods return `None` or
`False`.

### `TermsReader`

```python
class whoosh.codec.TermsReader
```

Abstract base class for reading the term dictionary and postings of a
segment.

**Methods:**
- `__contains__(term)`: Returns `True` if the term exists.
- `cursor(fieldname, fieldobj)`: Returns a `FieldCursor`.
- `terms()`: Yields `(fieldname, text)` tuples for all terms.
- `terms_from(fieldname, prefix)`: Yields terms from `fieldname` starting
  with `prefix`.
- `items()`: Yields `((fieldname, text), TermInfo)` tuples.
- `items_from(fieldname, prefix)`: Like `items()` but filtered by prefix.
- `term_info(fieldname, text)`: Returns a `TermInfo` for the term.
- `frequency(fieldname, text)`: Returns the total frequency.
- `doc_frequency(fieldname, text)`: Returns the document frequency.
- `matcher(fieldname, text, format_, scorer=None)`: Returns a `Matcher`.
- `indexed_field_names()`: Yields names of indexed fields.
- `close()`: Close the reader.

### `PerDocumentReader`

```python
class whoosh.codec.PerDocumentReader
```

Abstract base class for reading per-document values (columns, term vectors,
stored fields).

**Methods:**
- `close()`: Close the reader.
- `doc_count()`: Returns number of non-deleted documents.
- `doc_count_all()`: Returns total document count (including deleted).
- `has_deletions()`: Returns `True` if any documents are deleted.
- `is_deleted(docnum)`: Returns `True` if docnum is deleted.
- `deleted_docs()`: Yields docnums of deleted documents.
- `all_doc_ids()`: Yields docnums of all non-deleted documents.
- `supports_columns()`: Returns `True` if column storage is supported.
- `has_column(fieldname)`: Returns `True` if field has a column.
- `list_columns()`: Yields names of available columns.
- `column_reader(fieldname, column)`: Returns a column reader.
- `doc_field_length(docnum, fieldname)`: Returns field length for docnum.
- `field_length(fieldname)`: Returns total field length.
- `min_field_length(fieldname)`: Returns minimum field length.
- `max_field_length(fieldname)`: Returns maximum field length.
- `has_vector(docnum, fieldname)`: Returns `True` if docnum has a vector.
- `vector(docnum, fieldname, format_)`: Returns a `Matcher` for the vector.
- `stored_fields(docnum)`: Returns dict of stored field values.
- `all_stored_field()`: Yields stored fields for all documents.

### `MultiPerDocumentReader`

```python
class whoosh.codec.MultiPerDocumentReader(readers, offset=0)
```

Combines multiple `PerDocumentReader` instances into one for multi-segment
indices.

**Constructor:**
- `readers`: List of `PerDocumentReader` instances.
- `offset`: Base document offset (usually `0`).

## Automata

### `Automata`

```python
class whoosh.codec.Automata
```

Provides static methods for automata-based term matching, used by the
spelling corrector.

**Static Methods:**

#### `levenshtein_dfa(uterm, maxdist, prefix=0)`

Returns a deterministic finite automaton (DFA) that matches all edit-distance
variants of `uterm` within `maxdist` edits, optionally requiring a minimum
shared prefix of length `prefix`.

#### `find_matches(dfa, cur)`

Given a DFA and a `FieldCursor`, yields all matching terms.

**Methods:**

#### `terms_within(fieldcur, uterm, maxdist, prefix=0)`

Returns an iterator of matching terms within the given edit distance of
`uterm`.

## Segment

### `Segment`

```python
class whoosh.codec.Segment
```

Represents a segment of the index. Instances are pickled into the TOC file
to describe on-disk files.

**Class Attributes:**
- `COMPOUND_EXT = ".seg"`: Extension for compound segment files.

**Instance Attributes:**
- `indexname`: Base name of the segment.
- `segid`: Random unique ID string.
- `compound (bool)`: Whether this segment uses compound file format.

**Methods:**
- `make_filename(ext)`: Returns `f"{segment_id()}{ext}"`.
- `list_files(storage)`: Lists all files belonging to this segment.
- `create_file(storage, ext, **kwargs)`: Creates a new file for this segment.
- `open_file(storage, ext, **kwargs)`: Opens a file for this segment.
- `create_compound_file(storage)`: Combines all segment files into a
  compound `.seg` file.
- `open_compound_file(storage)`: Opens the compound segment file.
- `doc_count_all()`: Abstract. Returns total document count.
- `doc_count()`: Returns non-deleted document count.
- `set_doc_count(doccount)`: Sets the document count.
- `has_deletions()`: Returns `True` if any documents are deleted.
- `deleted_count()`: Abstract. Returns number of deleted documents.
- `deleted_docs()`: Abstract. Yields docnums of deleted documents.
- `delete_document(docnum, delete=True)`: Abstract. Deletes/undeletes a
  document.
- `is_deleted(docnum)`: Abstract. Returns `True` if docnum is deleted.
- `should_assemble()`: Returns `True` by default. Override to control
  compound file behavior.
- `validate(storage)`: Checks on-disk integrity of this segment.
- `segment_id()`: Returns the unique segment identifier string.
- `is_compound()`: Returns `True` if this segment uses compound file format.

### `WrappingSegment`

```python
class whoosh.codec.WrappingSegment(child)
```

A `Segment` that delegates all operations to a child segment.

**Constructor:**
- `child`: The underlying `Segment` instance to wrap.

## W3 Codec (Default)

The `W3` codec ("Whoosh 3") is the default index format, storing postings in
compressed blocks for efficient reading and skipping.

### `W3Codec`

```python
class whoosh.codec.whoosh3.W3Codec(blocklimit=128, compression=3, inlinelimit=1)
```

The default codec. Uses compressed blocks and term inlining for efficient
storage and fast lookups.

**Constructor:**
- `blocklimit`: Number of postings per block (default `128`).
- `compression`: zlib compression level (default `3`, `0` = no compression).
- `inlinelimit`: Maximum number of postings to inline directly in the term
  info (default `1`).

**File Extensions:**
- `.trm`: Term dictionary
- `.pst`: Postings
- `.vps`: Vector postings
- `.col`: Per-document value columns

### `W3PerDocWriter`

Writer for per-document values using the W3 format. Handles columns,
stored fields, term vectors, and field lengths.

### `W3FieldWriter`

Writer for the inverted term index using the W3 format. Uses a
`OrderedHashWriter` for the term dictionary and posts to a postings file.

### `W3LeafMatcher`

```python
class whoosh.codec.whoosh3.W3LeafMatcher(postfile, startoffset, length, format_, term=None, byteids=None, scorer=None)
```

Reads on-disk postings from the postings file and presents the
`Matcher` interface. Supports block-level skipping and lazy block loading.

**Optimization methods:**
- `block_min_id()`: Returns the first doc ID in the current block.
- `block_max_id()`: Returns the last doc ID in the current block.
- `block_min_length()`: Returns the minimum field length in the current block.
- `block_max_length()`: Returns the maximum field length in the current block.
- `block_max_weight()`: Returns the maximum weight in the current block.
- `skip_to_quality(minquality)`: Skips blocks exceeding a quality threshold.

### `W3TermsReader`

Reader for the term dictionary using the W3 format. Uses an
`OrderedHashReader` for fast lookups.

### `W3TermInfo`

```python
class whoosh.codec.whoosh3.W3TermInfo
```

Stores term statistics and posting location information. Supports inlining
small posting sets directly in the term dictionary for fast lookups.

**Flags:**
- `_FLAG_OFFSET` (0): Postings stored at an offset in the postings file.
- `_FLAG_INLINE_PICKLE` (1): Postings inlined as a pickled tuple.
- `_FLAG_INLINE_COMPACT` (2): Single posting compactly inlined.
- `_FLAG_INLINE_COMPACT_SHORT` (3): Multiple postings compactly inlined.

**Methods:**
- `add_block(block)`: Merges block statistics into this term info.
- `set_extent(offset, length)`: Sets offset and length of postings in file.
- `extent()`: Returns `(offset, length)`.
- `set_inlined(ids, weights, values)`: Sets inlined posting data.
- `set_compact_inline(id_, weight, value)`: Sets single inlined posting.
- `set_compact_short_inline(ids, weights, values)`: Sets multiple compact
  inlined postings.
- `is_inlined()`: Returns `True` if postings are inlined.
- `inlined_postings()`: Returns `(ids, weights, values)` tuples for inlined
  postings.
- `to_bytes()` / `from_bytes()`: Serialize/deserialize.

### `W3Segment`

```python
class whoosh.codec.whoosh3.W3Segment(codec, indexname, doccount=0, segid=None, deleted=None)
```

Segment class for the W3 codec. Stores a reference to the codec, document
count, and deleted document set.

## Plain Text Codec (Debugging)

### `PlainTextCodec`

```python
class whoosh.codec.plaintext.PlainTextCodec
```

A codec that stores the index as human-readable plain text. Intended for
debugging and manual inspection, not for production use.

**Class Attributes:**
- `length_stats = False`

**File extensions:**
- `.dcs`: Document (stored fields, columns, vectors)
- `.trm`: Term dictionary (plain text)

### `PlainPerDocWriter`

Plain text writer for per-document values.

### `PlainPerDocReader`

Plain text reader for per-document values.

### `PlainFieldWriter`

Plain text writer for the inverted index.

### `PlainTermsReader`

Plain text reader for the term dictionary.

### `PlainSegment`

```python
class whoosh.codec.plaintext.PlainSegment(indexname)
```

Segment class for the plain text codec. Does not support compound files
(`should_assume()` returns `False`).

## Memory Codec

### `MemoryCodec`

```python
class whoosh.codec.memory.MemoryCodec
```

An in-memory-only codec for testing. Stores all data in Python objects
rather than on disk.

**Class Attributes:**
- `storage`: A `RamStorage` instance.
- `segment`: A `MemSegment` instance.

**Methods:**
- `writer(schema)`: Returns a `MemWriter`.
- `reader(schema)`: Returns a `SegmentReader`.

### `MemWriter`

```python
class whoosh.codec.memory.MemWriter
```

A `SegmentWriter` subclass that commits immediately without merging.

### `MemPerDocWriter`

In-memory writer for per-document values.

### `MemPerDocReader`

In-memory reader for per-document values.

### `MemFieldWriter`

In-memory writer for the inverted index.

### `MemTermsReader`

In-memory reader for the term dictionary.

### `MemSegment`

```python
class whoosh.codec.memory.MemSegment(codec, indexname)
```

In-memory segment storing all data in Python dictionaries (inverted index,
stored fields, lengths, vectors, term infos). Uses a `Lock` for thread-safe
access.


## DOCUMENT (FR): Collectors

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Collectors API

Classes and functions for gathering search results. Collectors are used
internally by `Searcher.search()` to collect matching documents and build
`Results` objects. The collectors module is a refactored package exposing the
same public API as the former monolithic module.

## Overview

A `Collector` iterates over matching documents in an index, collects
information about them, and produces a `Results` object. The base `Collector`
class defines the interface; specialized subclasses implement different
collection strategies (top-N, unlimited, sorting, filtering, faceting, etc.).

## Core Classes

### `Collector`

```python
class whoosh.collectors.Collector
```

Abstract base class for all collectors. Subclasses must implement `collect()`
and `results()`.

**Methods:**

#### `prepare(top_searcher, q, context)`

Called before a search begins. Sets up `self.top_searcher`, `self.q`,
`self.context`, `self.starttime`, and `self.docset`.

#### `run()`

Iterates over sub-searchers, calling `set_subsearcher()` and
`collect_matches()` for each, then calls `finish()`.

#### `set_subsearcher(subsearcher, offset)`

Called when moving to a new sub-searcher. Sets `self.subsearcher`,
`self.offset`, and `self.matcher`.

#### `collect(sub_docnum)`

Called for every matched document. Must add the document to results and
return a sort key. Subclasses must implement this.

- `sub_docnum`: Segment-relative document number. Add `self.offset` to get
  the top-level document number.

#### `sort_key(sub_docnum)`

Returns a sort key for the current match without the side effect of adding
the document to results. Subclasses must implement this.

#### `collect_matches()`

Calls `matches()` and then `collect()` for each matched document.

#### `matches()`

Yields segment-relative document numbers for matches in the current
sub-searcher.

#### `count()`

Returns the total number of matching documents.

#### `all_ids()`

Returns a sequence of docnums matched in this collector.

#### `computes_count()`

Returns `True` if the collector naturally computes the exact count of
matching documents.

#### `finish()`

Called after the search completes. Sets `self.runtime`.

#### `remove(global_docnum)`

Removes a document from the collector using its global docnum.

#### `results()`

Returns a `Results` object. Subclasses must implement this.

### `ilen`

```python
whoosh.collectors.ilen(iterator) -> int
```

Counts the number of items in an iterator without loading it all into memory.

## Scored Collectors

### `ScoredCollector`

```python
class whoosh.collectors.ScoredCollector(replace=10)
```

Base class for collectors that sort by document score.

**Constructor:**
- `replace`: Number of matches between attempts to replace the matcher with
  a more efficient version.

### `TopCollector`

```python
class whoosh.collectors.TopCollector(
    limit=10,
    usequality=True,
    **kwargs
)
```

A collector that returns only the top N scored results.

**Constructor:**
- `limit`: Maximum number of results to return.
- `usequality`: Whether to use block-quality optimizations for faster
  search. Can be set to `False` for debugging.

**Notes:**
- When `usequality=True`, `computes_count()` returns `False` and
  `all_ids()` requires re-searching.
- Uses a min-heap to efficiently track the top N documents.

### `UnlimitedCollector`

```python
class whoosh.collectors.UnlimitedCollector(reverse=False)
```

A collector that returns **all** scored results. Sorts by score (descending
by default).

**Constructor:**
- `reverse`: If `True`, sort results in ascending order (lowest scores first).

### `UnsortedCollector`

```python
class whoosh.collectors.UnsortedCollector
```

A collector that returns results in document order (no sorting). Used when
the search weighting is `None`.

## Wrapping Collectors

### `WrappingCollector`

```python
class whoosh.collectors.WrappingCollector(child)
```

Base class for collectors that wrap other collectors. Delegates most
operations to the child collector while adding additional behavior.

**Constructor:**
- `child`: The collector to wrap.

**Methods** (all delegated to child):
`top_searcher`, `context`, `prepare`, `set_subsearcher`, `all_ids`,
`count`, `collect_matches`, `sort_key`, `collect`, `remove`, `matches`,
`finish`, `results()`

### `SortingCollector`

```python
class whoosh.collectors.SortingCollector(
    sortedby,
    limit=10,
    reverse=False
)
```

A collector that returns results sorted by a `FacetType` object.

**Constructor:**
- `sortedby`: A `FacetType` or field name to sort by.
- `limit`: Maximum number of results (0 for no limit).
- `reverse`: If `True`, reverse the overall sort order.

### `FilterCollector`

```python
class whoosh.collectors.FilterCollector(
    child,
    allow=None,
    restrict=None
)
```

A collector that allows and/or restricts certain document numbers in
results.

A document is discarded if:
- `allow` is set and the docnum is not in the allowed set, or
- `restrict` is set and the docnum is in the restricted set.

**Constructor:**
- `child`: The collector to wrap.
- `allow`: A query, `Results` object, or set-like of allowed docnums.
  `None` means everything is allowed.
- `restrict`: A query, `Results` object, or set-like of disallowed docnums.
  `None` means nothing is disallowed.

**Attributes:**
- `filtered_count`: Number of documents filtered out.

### `FacetCollector`

```python
class whoosh.collectors.FacetCollector(child, groupedby, maptype=None)
```

A collector that creates groups of documents based on facet objects. Used
when `groupedby` is specified in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `groupedby`: A field name, `FacetType`, dict, or `Facets` object.
- `maptype`: Default `FacetMap` class for facets that don't specify one.

**Attributes:**
- `facetmaps`: Dictionary of facet name to `FacetMap` objects.

### `CollapseCollector`

```python
class whoosh.collectors.CollapseCollector(
    child,
    keyfacet,
    limit=1,
    order=None
)
```

A collector that eliminates all but the top N results sharing the same facet
key. Useful for "dedup" or grouped result views.

**Constructor:**
- `child`: The collector to wrap.
- `keyfacet`: A `FacetType` to collapse on. All but the top N documents
  sharing a key are eliminated.
- `limit`: Maximum documents to keep per key (default `1`).
- `order`: Optional `FacetType` to determine which documents are "top" within
  each group. Defaults to the results order (e.g., highest score).

**Attributes:**
- `collapsed_counts`: Dictionary mapping keys to the number of documents
  eliminated.

### `TimeLimitCollector`

```python
class whoosh.collectors.TimeLimitCollector(
    child,
    timelimit,
    greedy=False,
    use_alarm=True
)
```

A collector that raises a `TimeLimit` exception if the search exceeds a
time limit. Partial results are still available via `results()`.

**Constructor:**
- `child`: The collector to wrap.
- `timelimit`: Maximum search time in seconds.
- `greedy`: If `True`, finish adding the current hit before raising.
- `use_alarm`: If `True` (default), use `signal.SIGALRM` on Unix for
  immediate interruption. On Windows, time is only checked between
  documents.

```python
from whoosh.searching import TimeLimit

uc = collectors.UnlimitedCollector()
tlc = TimeLimitCollector(uc, timelimit=5.8)
try:
    searcher.search_with_collector(myquery, tlc)
except TimeLimit:
    print("Search timed out!")
# Still get partial results:
print(tlc.results())
```

### `TermsCollector`

```python
class whoosh.collectors.TermsCollector(child, settype=set)
```

A collector that records which terms appeared in which matched documents.
Used when `terms=True` in `Searcher.search()`.

**Constructor:**
- `child`: The collector to wrap.
- `settype`: Set type to use for docnum collections (default `set`).

**Attributes:**
- `termdocs`: Dict mapping `(fieldname, text)` tuples to arrays of docnums.
- `docterms`: Dict mapping docnums to lists of `(fieldname, text)` tuples.

## Exceptions

### `TimeLimit`

```python
from whoosh.searching import TimeLimit
```

Raised by `TimeLimitCollector` when the search exceeds the time limit.
Partial results are still available from the collector.


## DOCUMENT (FR): Columns

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Columns API

Classes for storing per-document values (column-oriented storage) used for
fast sorting, faceting, and filtering. Columns are the mechanism by which
Whoosh stores field values alongside the inverted index, in a column-oriented
layout for efficient range access.

The default column type for most fields is `VarBytesColumn`, although numeric
and date fields use `NumericColumn`. Expert users may use other column types
that may be faster or more storage-efficient based on the field contents.

A `Column` object stores configuration information and provides two important
methods: `writer()` to return a `ColumnWriter` and `reader()` to return a
`ColumnReader`.

## Module Functions

### `bytes_column`

```python
whoosh.columns.bytes_column
```

A default `VarBytesColumn` instance used as the column type for string fields.

### `numeric_column`

```python
whoosh.columns.numeric_column
```

A default `NumericColumn` instance used as the column type for numeric fields.

## Base Classes

### `Column`

```python
class whoosh.columns.Column
```

Base class for all column types.

**Class Attributes:**
- `reversible (bool)`: Whether values can be reversed for descending sort.
  Default `False`.

**Methods:**
- `writer(dbfile)`: Returns a `ColumnWriter` for this column type.
- `reader(dbfile, basepos, length, doccount)`: Returns a `ColumnReader` for
  this column type.
- `default_value(reverse=False)`: Returns the default value for documents
  without a column value at index time.
- `stores_lists()`: Returns `True` if the column stores a list of values per
  document instead of a single value.

### `ColumnWriter`

```python
class whoosh.columns.ColumnWriter(dbfile)
```

Base class for writing column values to disk.

**Constructor:**
- `dbfile`: The `StructFile` to write to.

**Methods:**
- `fill(docnum)`: Fills any gap in docnums up to `docnum` with default values.
- `add(docnum, value)`: Adds a value for the given docnum.
- `finish(docnum)`: Called when done writing. Default does nothing.

### `ColumnReader`

```python
class whoosh.columns.ColumnReader(dbfile, basepos, length, doccount)
```

Base class for reading column values from disk.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `basepos`: The offset within the file at which the column starts.
- `length`: The length in bytes the column occupies in the file.
- `doccount`: The number of rows (documents) in the column.

**Methods:**
- `__getitem__(docnum)`: Returns the value for the given docnum.
- `sort_key(docnum)`: Returns the value for sorting (defaults to
  `__getitem__`).
- `__iter__()`: Yields values for all documents.
- `load()`: Returns a list of all values.
- `set_reverse()`: Prepares the reader for reverse iteration.

## Concrete Column Types

### `VarBytesColumn`

```python
class whoosh.columns.VarBytesColumn(
    allow_offsets=True,
    write_offsets_cutoff=2**15
)
```

Stores variable-length byte strings. The default value for documents without
a value is `b''` (empty bytes).

**Constructor:**
- `allow_offsets`: Whether to write offsets for faster lookup when there are
  many rows. Default `True`.
- `write_offsets_cutoff`: Write offsets when there are more than this many
  rows (default `2**15`).

### `FixedBytesColumn`

```python
class whoosh.columns.FixedBytesColumn(blocksize, default=emptybytes)
```

Stores fixed-length byte strings, saving space by not storing the length of
each value.

**Constructor:**
- `blocksize`: Fixed size of each value in bytes.
- `default`: Default value for documents without a value.

### `RefBytesColumn`

```python
class whoosh.columns.RefBytesColumn(
    cachesize=1000,
    stable=True,
    default=emptybytes
)
```

Stores references to unique values rather than the values themselves, saving
space when the field has few unique values. Uses a `DocIdSet` to track which
documents contain each value.

**Constructor:**
- `cachesize`: Size of the LRU cache for value lookups (default `1000`).
- `stable`: Whether to use a stable sort of references (default `True`).
- `default`: Default value for missing documents.

### `NumericColumn`

```python
class whoosh.columns.NumericColumn(
    typecode,
    default=None,
    nullable=False
)
```

Stores numbers (int, float, datetime) encoded as binary values. Extends
`FixedBytesColumn`.

**Constructor:**
- `typecode`: A `struct` typecode string (e.g., `"I"` for unsigned int,
  `"q"` for long, `"d"` for float).
- `default`: Default numeric value (None for the type's zero value).
- `nullable`: Whether `None` values are allowed.

### `BitColumn`

```python
class whoosh.columns.BitColumn
```

Stores boolean values as a bitmap. Each value is either `True` (1) or
`False` (0). Uses a `BitSet` internally.

### `CompressedBytesColumn`

```python
class whoosh.columns.CompressedBytesColumn(default=emptybytes)
```

Wraps a `VarBytesColumn` with zlib compression for the value bytes.

### `CompressedBlockColumn`

```python
class whoosh.columns.CompressedBlockColumn
```

Stores values with block-level zlib compression. More efficient for large
columns.

### `StructColumn`

```python
class whoosh.columns.StructColumn(struct, name)
```

Wraps a `FixedBytesColumn` to store structured binary data (e.g., tuples
encoded with `struct`).

**Constructor:**
- `struct`: A `struct.Struct` object defining the format.
- `name`: Field name for error messages.

### `EmptyColumnReader`

```python
class whoosh.columns.EmptyColumnReader(default, doccount)
```

A `ColumnReader` that returns a constant default value for every document.
Used when a field has no column.

### `MultiColumnReader`

```python
class whoosh.columns.MultiColumnReader(readers)
```

Combines multiple `ColumnReader` instances into one for multi-segment indices.

**Constructor:**
- `readers`: List of `ColumnReader` instances (one per segment).

### `TranslatingColumnReader`

```python
class whoosh.columns.TranslatingColumnReader(child, translator)
```

Wraps a `ColumnReader` to apply a translation function to the values.

**Constructor:**
- `child`: The underlying `ColumnReader`.
- `translator`: Function that maps sort keys to human-readable values.

### `WrappedColumn`

```python
class whoosh.columns.WrappedColumn(child)
```

Base class for column wrappers that adapt another column type.

### `WrappedColumnWriter`

```python
class whoosh.columns.WrappedColumnWriter(child)
```

Base class for column writer wrappers.

### `WrappedColumnReader`

```python
class whoosh.columns.WrappedColumnReader(child)
```

Base class for column reader wrappers.

### `ClampedNumericColumn`

```python
class whoosh.columns.ClampedNumericColumn(child, clampfn)
```

Wraps a `NumericColumn` to clamp values to a valid range before sorting.

**Constructor:**
- `child`: The wrapped `NumericColumn`.
- `clampfn`: Function that clamps a value to the valid range.

### `PickleColumn`

```python
class whoosh.columns.PickleColumn(child, ...)
```

Wraps another column to store pickled Python objects.

### `ListColumn`

```python
class whoosh.columns.ListColumn(child)
```

Base class for columns that store multiple values per document.

### `ListColumnReader`

```python
class whoosh.columns.ListColumnReader(child)
```

Reader for list-valued columns.

### `VarBytesListColumn`

```python
class whoosh.columns.VarBytesListColumn
```

A `ListColumn` variant of `VarBytesColumn` that stores lists of byte strings.

### `FixedBytesListColumn`

```python
class whoosh.columns.FixedBytesListColumn(blocksize)
```

A `ListColumn` variant of `FixedBytesColumn` that stores lists of fixed-size
byte strings.


## DOCUMENT (FR): Core

# API Core

Gestion des indexes via les fonctions et classes du module `whoosh.index`.

## Fonctions

### create_in

```python
def create_in(dirname, schema, indexname="MAIN", create=True, **kwargs) -> FileIndex
```

Crée un nouvel index dans le répertoire donné.

**Args:**
- `dirname (str)`: Chemin du répertoire.
- `schema (Schema)`: Objet Schema définissant les champs.
- `indexname (str)`: Nom de l'index.
- `create (bool)`: Si True, crée même si existe (efface l'existant).

**Retourne:**
- `FileIndex`: Objet index.

**Exemple:**
```python
from whoosh.index import create_in
index = create_in("indexdir", schema)
```

### open_dir

```python
def open_dir(dirname, indexname="MAIN", readonly=False, **kwargs) -> FileIndex
```

Ouvre un index existant.

**Exemple:**
```python
index = open_dir("indexdir")
```

### exists_in

```python
def exists_in(dirname, indexname="MAIN", **kwargs) -> bool
```

Vérifie si un index valide existe dans le répertoire.

## Classes

### Index

Classe de base pour les objets index.

#### Méthodes principales

| Méthode | Description |
|---------|-------------|
| `writer(**kwargs)` | Retourne un IndexWriter |
| `searcher(**kwargs)` | Retourne un Searcher |
| `reader()` | Retourne un IndexReader |
| `commit()` | Commit via un writer temporaire |
| `optimize()` | Fusionne tous les segments |
| `add_field()` | Ajoute un champ au schéma |
| `remove_field()` | Supprime un champ du schéma |
| `doc_count()` | Nombre de documents |
| `doc_count_all()` | Nombre total (y compris supprimés) |

## Exceptions

### LockError

Levée quand l'index est verrouillé par un autre writer.

```python
from whoosh.index import LockError

try:
    writer = ix.writer(timeout=5.0)
except LockError:
    print("Index verrouillé, réessayez plus tard")
```

### IndexMissingError

Levée quand l'index n'existe pas.


## DOCUMENT (FR): Events

# API Events

Système d'événements pour un couplage lâche entre les composants.

## EventBus

```python
class whoosh.event_bus.EventBus
```

Registre central des événements et subscribers.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `bus.publish(event)` | Publie un événement |
| `bus.subscribe(fn)` | Abonne un handler |
| `bus.unsubscribe(fn)` | Désabonne un handler |
| `bus.clear()` | Supprime tous les abonnés |

**Exemple:**
```python
from whoosh.event_bus import EventBus

bus = EventBus()

@bus.subscribe
def on_index(event: DocumentIndexed):
    print(f"Indexé: {event.docnum}")

# Publier
bus.publish(DocumentIndexed(docnum=42))
```

## Événements intégrés

### DocumentIndexed

```python
class DocumentIndexed
    docnum: int           # Numéro de document
    schema: Schema        # Schéma utilisé
    timestamp: datetime   # Horodatage
    metadata: dict        # Métadonnées
```

### SearchExecuted

```python
class SearchExecuted
    query: str            # Requête originale
    result_count: int     # Nombre de résultats
    duration_ms: float    # Durée en ms
    user: str | None      # Utilisateur (si auth)
```

### IndexOptimized

```python
class IndexOptimized
    segments_before: int
    segments_after: int
    size_bytes: int
```

## Utilisation avec FastAPI

```python
from fastapi import FastAPI
from whoosh.event_bus import EventBus, DocumentIndexed

app = FastAPI()
bus = EventBus()

@app.on_event("startup")
def startup():
    bus.subscribe(on_index)

@app.post("/documents")
def add_document(doc: dict):
    # Indexation...
    bus.publish(DocumentIndexed(docnum=doc["id"]))
```

## Gestion d'erreurs

```python
@bus.subscribe
def on_error(event: SearchExecuted):
    if event.result_count == 0:
        logger.warning(f"Recherche vide: {event.query}")
```


## DOCUMENT (FR): Fields

# API Champs

Définissez la structure de votre index avec les types de champs.

## Schema

```python
class whoosh.fields.Schema
```

Définit les champs disponibles dans l'index.

### Méthodes

#### `add()`

```python
schema.add(fieldname, fieldtype, glob=False, **kwargs)
```

Ajoute un champ. Si `glob=True`, le nom est traité comme un pattern glob.

#### `remove()`

```python
schema.remove(fieldname, **kwargs)
```

Supprime un champ.

#### `items()`

```python
for name, field in schema.items():
    print(name, field)
```

Retourne les paires (nom, objet champ).

## Types de champs

### TEXT

```python
TEXT(
    stored=False,
    unique=False,
    phrase=True,
    analyzer=None,
    field_boost=1.0
)
```

Texte libre avec tokenisation et recherche de phrase optionnelle.

**Exemple:**
```python
titre = TEXT(stored=True)
corps = TEXT(analyzer=StemmingAnalyzer(), phrase=False)
```

### ID

```python
ID(stored=False, unique=False, field_boost=1.0)
```

Identifiant non tokenisé. Stocke la valeur entière comme terme unique.

**Exemple:**
```python
chemin = ID(stored=True, unique=True)
slug = ID(stored=True)
```

### KEYWORD

```python
KEYWORD(
    stored=False,
    lowercase=False,
    commas=False,
    scorable=False,
    field_boost=1.0
)
```

Mots-clés séparés par espace ou virgule.

**Exemple:**
```python
tags = KEYWORD(lowercase=True, commas=True, stored=True)
```

### STORED

```python
STORED(stored=True)
```

Champ stocké uniquement, non indexé ni searchable.

### NUMERIC

```python
NUMERIC(numtype=int, stored=False, unique=False, field_boost=1.0)
```

Champ numérique (entier ou flottant).

### DATETIME

```python
DATETIME(stored=False, unique=False, field_boost=1.0)
```

Champ date/heure.

### BOOLEAN

```python
BOOLEAN(stored=False, unique=False, field_boost=1.0)
```

Champ booléen. Searchable avec `oui`, `non`, `vrai`, `faux`, `1`, `0`, `t`, `f`.

### VectorField

```python
VectorField(
    dimensions: int,
    metric: str = "cosine",
    provider: str = "numpy",
    stored: bool = False
)
```

Champ pour embeddings vectoriels.

**Exemple:**
```python
embedding = VectorField(dimensions=384, metric="cosine", stored=True)
```

## SchemaBuilder

API fluent pour construire des schémas :

```python
from whoosh.fields import SchemaBuilder, TEXT, ID, NUMERIC

schema = (
    SchemaBuilder()
    .field("titre", TEXT(stored=True))
    .field("chemin", ID(stored=True, unique=True))
    .field("contenu", TEXT)
    .field("note", NUMERIC(float, stored=True))
    .build()
)
```

## Attributs de FieldType

| Attribut | Type | Description |
|----------|------|-------------|
| `format` | `Format` | Définit l'indexation |
| `vector` | `Format` | Format vectoriel optionnel |
| `scorable` | `bool` | Stocke la longueur pour BM25F |
| `stored` | `bool` | Stocke la valeur |
| `unique` | `bool` | Identifie les documents de façon unique |


## DOCUMENT (FR): Filedb Storage

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# File DB / Storage API

Classes for storing and retrieving index data on disk or in memory. The
`Storage` class is the main entry point for persisting an index.

## Storage Classes

### `Storage`

```python
class whoosh.filedb.filestore.Storage(path=None)
```

Abstract base class for storage backends. A `Storage` manages a filesystem-
or memory-based location where index files can be created, read, and
manipulated.

**Constructor:**
- `path`: Optional path string. Subclasses may use this to set the storage
  location.

**Methods:**

#### `create_file(name, **kwargs)`

Creates and returns a file object for writing.

#### `open_file(name, **kwargs)`

Opens and returns a file object for reading.

#### `list()`

Returns a list of all filenames in this storage.

#### `exists(name)`

Returns `True` if a file/named item exists in the storage.

#### `file_exists(name)`

Alias for `exists()`.

#### `file_length(name)`

Returns the length of file `name` in bytes.

#### `rename(src, dst)`

Renames a file from `src` to `dst`.

#### `delete_file(name)`

Deletes file `name` from storage.

#### `destroy()`

Deletes all files and the storage itself.

#### `temp_storage()`

Creates and returns a temporary isolated `Storage` for scratch space.

#### `supports_mmap`

Returns `True` if this storage supports memory-mapped file access.

**Properties:**
- `schema`: The `Schema` for this storage (if it holds an index).
- `lock`: The lock object used for this storage.

### `FileStorage`

```python
class whoosh.filedb.filestore.FileStorage(
    path,
    cachesize_limit=40,
    supports_mmap=None,
    **kwargs
)
```

A `Storage` subclass that uses the operating system's filesystem.

**Constructor:**
- `path`: A `Path` (or string path) to the directory where files are stored.
- `cachesize_limit`: Maximum number of open file handles to cache.
- `supports_mmap`: If `None`, auto-detected; otherwise force enable/disable.

**Methods:** All `Storage` methods plus:
- `create_index(schema, indexname="index", ...)`: Creates and returns a new
  `Index` object.
- `open_index(indexname="index", ...)`: Opens an existing `Index`.
- `lock(name)`: Returns a lock object for the given lock name.

### `RamStorage`

```python
class whoosh.filedb.filestore.RamStorage(cachesize_limit=10)
```

A `Storage` subclass that keeps all files in memory as bytes. Useful for
testing and small indexes.

**Constructor:**
- `cachesize_limit`: Maximum number of files to cache as decoded objects.

**Methods:** All `Storage` methods plus:
- `create_index(schema, ...)`: Creates an in-memory `Index`.
- `save_to_file(filename, ...)`: Saves the entire storage to a file.
- `load_from_file(filename, ...)`: Loads storage contents from a file.

### `OverlayStorage`

```python
class whoosh.filedb.filestore.OverlayStorage(base, overlay)
```

A `Storage` wrapper that presents two storage layers: a base and an overlay.
Files in the overlay take precedence over the base.

**Constructor:**
- `base`: The base `Storage` (e.g., read-only original).
- `overlay`: The overlay `Storage` (e.g., writable copy).

## Storage Exceptions

### `StorageError`

```python
class whoosh.filedb.filestore.StorageError
```

Base exception for storage-related errors.

### `ReadOnlyError`

```python
class whoosh.filedb.filestore.ReadOnlyError(StorageError)
```

Raised when attempting to write to a read-only storage.

## File Tables

### `HashWriter`

```python
class whoosh.filedb.filetables.HashWriter(dbfile, keycoder=None, keydecoder=None, data_encoder=None, data_decoder=None, **kwargs)
```

Writes key-value pairs to a file, with optional indexing by key.

**Constructor:**
- `dbfile`: The `StructFile` to write to.
- `keycoder`: Function to encode keys for storage.
- `keydecoder`: Function to decode keys from storage.
- `data_encoder`: Function to encode values.
- `data_decoder`: Function to decode values.

### `HashReader`

```python
class whoosh.filedb.filetables.HashReader(dbfile, length, keycoder=None, keydecoder=None, data_decoder=None, **kwargs)
```

Reads key-value pairs from a file written by `HashWriter`.

**Constructor:**
- `dbfile`: The `StructFile` to read from.
- `length`: Length of the data section.
- `keycoder`/`keydecoder`/`data_decoder`: Same as `HashWriter`.

**Methods:**
- `__getitem__(key)`: Returns the value for `key`.
- `keys()`: Yields all keys.
- `values()`: Yields all values.
- `items()`: Yields `(key, value)` pairs.
- `keys_from(prefixbytes)`: Yields keys starting at `prefixbytes`.
- `items_from(prefixbytes)`: Yields `(key, value)` pairs starting at prefix.
- `closest_key_pos(key)`: Returns the position of the closest matching key.
- `range_for_key(key)`: Returns `(startpos, endpos)` for a key range.

### `OrderedHashWriter`

```python
class whoosh.filedb.filetables.OrderedHashWriter(HashWriter)
```

A `HashWriter` that maintains keys in sorted order.

### `OrderedHashReader`

```python
class whoosh.filedb.filetables.OrderedHashReader(HashReader)`

A `HashReader` for reading data written by `OrderedHashWriter`. Preserves
key ordering for efficient prefix iteration.

### `FieldedOrderedHashWriter`

```python
class whoosh.filedb.filetables.FieldedOrderedHashWriter(HashWriter)
```

An `OrderedHashWriter` that stores an extra "fieldmap" in the extras dict,
mapping field names to numeric IDs.

### `FieldedOrderedHashReader`

```python
class whoosh.filedb.filetables.FieldedOrderedHashReader(HashReader)
```

Reader for data written by `FieldedOrderedHashWriter`.

## Struct File

### `StructFile`

```python
class whoosh.filedb.structfile.StructFile(name, source, cachesize_limit=40)
```

Wraps a file object and adds methods for reading/writing packed binary
values, arrays, varints, and pickle objects.

**Methods include:**
- `read_int()`, `write_int(n)`: Read/write a 4-byte signed integer.
- `read_long()`, `write_long(n)`: Read/write a 8-byte signed integer.
- `read_uint()`, `write_uint(n)`: Read/write unsigned int.
- `read_ulong()`, `write_ulong(n)`: Read/write unsigned long.
- `read_float()`, `write_float(n)`: Read/write a float.
- `read_ushort()`, `write_ushort(n)`: Read/write unsigned short.
- `read_byte()`, `write_byte(b)`: Read/write a single byte.
- `write_array(arr)`: Write an array of values.
- `get_array(offset, typecode, length)`: Read an array from offset.
- `write_pickle(obj)`: Pickle and write an object.
- `read_pickle()`: Read and unpickle an object.
- `get(offset, length)`: Read `length` bytes from `offset`.
- `get_int()`, `get_uint()`, `get_long()`, `get_float()`, `get_byte()`:
  Read a single value from the given offset.

### `BufferFile`

```python
class whoosh.filedb.structfile.BufferFile
```

A `StructFile` that wraps an in-memory byte buffer.

### `ChecksumFile`

```python
class whoosh.filedb.structfile.ChecksumFile(dbfile)
```

A `StructFile` wrapper that computes a checksum as data is written, for
integrity verification.

## Compound Storage

### `CompoundStorage`

```python
class whoosh.filedb.compound.CompoundStorage(dbfile, use_mmap=True)
```

Treats a single file as a container for multiple sub-files. Used for compound
segment files.

**Methods:**
- `create_file(name)`: Create a sub-file within the compound file.
- `open_file(name)`: Open a sub-file for reading.
- `list()`: List all sub-file names.
- `close()`: Close the compound storage.

### `SubFile`

```python
class whoosh.filedb.compound.SubFile
```

A file-like object representing a sub-file within a `CompoundStorage`.

### `CompoundWriter`

```python
class whoosh.filedb.compound.CompoundWriter(storage)
```

Writes a compound file by assembling multiple files from a storage.

**Methods:**
- `create_file(name)`: Reserve a filename in the compound file.
- `save_as_files(dest_storage, fn_generator)`: Assemble the compound file
  from source files into the destination storage.

## Storage Utility Functions

### `copy_storage`

```python
whoosh.filedb.filestore.copy_storage(sourcestore, deststore)
```

Copies all files from one storage to another.

### `copy_to_ram`

```python
whoosh.filedb.filestore.copy_to_ram(storage)
```

Reads all files from a storage into a `RamStorage` and returns it.


## DOCUMENT (FR): Formats

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Formats API

Classes that control how posting information (frequencies, positions,
character offsets, and weights) is encoded and stored for each field in the
index. The `Format` object is a factory and encoder/decoder for the
value strings stored alongside each posting.

## Module Functions

### `tokens`

```python
whoosh.formats.tokens(value, analyzer, kwargs)
```

Takes a text `value` and an `analyzer`, runs the analyzer on the value, and
returns the resulting token generator (wrapped with `unstopped()` to ignore
`STOP` tokens). Used internally by `Format.word_values()`.

## Format Classes

All format classes accept a `field_boost` parameter (default `1.0`) that
scales the score of all queries matching terms in that field.

### `Format`

```python
class whoosh.formats.Format(field_boost=1.0, **options)
```

Abstract base class for all posting formats. Format objects are
field-level objects: one is created per `Field` and shared across all
postings for that field.

**Attributes:**
- `posting_size (int)`: Fixed byte size of encoded postings, or `None`/`-1`
  if variable-size.
- `textual (bool)`: Whether this format expects string tokens (vs. bytes).
  Default `True`.

**Methods:**

#### `word_values(value, analyzer, **kwargs)`

Abstract. Takes a text value, runs it through the analyzer, and yields
`(tokentext, frequency, weight, valuestring)` tuples.

#### `encode(value)`

Abstract. Encodes raw posting data into the value string bytes.

#### `decode_frequency(valuestring)`

Abstract. Decodes the frequency (term count in document) from the value
string.

#### `decode_weight(valuestring)`

Abstract. Decodes the weight (total boost contribution) from the value string.

#### `combine(valuestrings)`

Abstract. Combines multiple value strings (from overlapping segments) into
a single value string.

#### `supports(name)`

Returns `True` if this format supports interpreting its postings as `name`
(e.g., `"frequency"`, `"positions"`, `"characters"`, `"position_boosts"`,
`"character_boosts"`). Equivalent to `hasattr(self, "decode_" + name)`.

#### `decoder(name)`

Returns the `decode_<name>` method for the given attribute name.

#### `decode_as(astype, valuestring)`

Calls the appropriate `decode_<astype>` method on `valuestring` and returns
the result.

#### `fixed_value_size()`

Returns `self.posting_size` if positive, otherwise `None`.

#### `__eq__(other)`

Returns `True` if `other` is the same class with equal `__dict__`.

### `Existence`

```python
class whoosh.formats.Existence(field_boost=1.0, **options)
```

Indexes only whether a term occurred in a document—not its frequency or
positions. Useful for non-scorable fields like paths.

- `posting_size = 0`
- Supports: `frequency` (always 1), `weight` (always `field_boost`)
- `encode()` returns empty bytes

### `Frequency`

```python
class whoosh.formats.Frequency(field_boost=1.0, boost_as_freq=False, **options)
```

Stores term frequency information (term count per document) for each posting.

- `posting_size = _INT_SIZE` (4 bytes)
- Supports: `frequency`, `weight`
- `encode()` encodes the count as a packed unsigned int
- `boost_as_freq`: If `True`, boosts are interpreted as frequency boosts

```python
from whoosh.formats import Frequency
fmt = Frequency(field_boost=1.0)
```

### `Positions`

```python
class whoosh.formats.Positions(field_boost=1.0, **options)
```

Stores position information (term offsets within the document) in each
posting, enabling phrase queries and "near" queries.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode(poslist)` encodes positions using variable-length delta encoding
- Positions are stored as delta-encoded variable-length integers

```python
from whoosh.formats import Positions
fmt = Positions()
```

### `Characters`

```python
class whoosh.formats.Characters(field_boost=1.0, **options)
```

Extends `Positions` to also store character start and end offsets for each
term occurrence, enabling character-precise highlighting.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`
- `encode()` encodes (position, startchar, endchar) triples with delta
  encoding

### `PositionBoosts`

```python
class whoosh.formats.PositionBoosts(field_boost=1.0, **options)
```

Extends `Positions` to store per-position boost values in addition to
positions.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`
- `encode()` encodes `(position, boost)` pairs

### `CharacterBoosts`

```python
class whoosh.formats.CharacterBoosts(field_boost=1.0, **options)
```

Extends `Characters` to store per-position boost values along with
character offsets.

- Supports: `frequency`, `weight`, `positions`, `position_boosts`,
  `characters`, `character_boosts`
- `encode()` encodes `(position, startchar, endchar, boost)` tuples


## DOCUMENT (FR): Highlight

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Highlight API

Classes and functions for highlighting matches in search result fragments.
The highlight module is a refactored package exposing the same public API as
the former monolithic module.

## Overview

The highlighting system has four components:

- **Fragmenters** split text into fragments.
- **Fragment Scorers** score fragments to determine which to display.
- **Formatters** render fragments as output (HTML, plain text, etc.).
- **Highlighter** ties these together and is used by `Searcher.highlights()`.

## Module-level Functions

### `highlight`

```python
whoosh.highlight.highlight(
    text: str,
    terms: list[str],
    analyzer,
    fragmenter,
    formatter,
    top: int = 3,
    scorer=None,
    minscore: int = 1,
    order=SCORE,
    mode: str = "query"
) -> str
```

Highlights the matched terms in `text` and returns a formatted string.

- `text`: The text to highlight.
- `terms`: A list of matched terms (strings).
- `analyzer`: The analyzer for the field.
- `fragmenter`: A `Fragmenter` instance or class.
- `formatter`: A `Formatter` instance or class.
- `top`: Maximum number of fragments to return.
- `scorer`: Optional fragment scorer (defaults to `BasicFragmentScorer`).
- `minscore`: Minimum score for a fragment to be included.
- `order`: Sort order for fragments (`FIRST`, `SCORE`, `LONGER`, `SHORTER`).
- `mode`: Analysis mode, typically `"query"` or `"index"`.

### `mkfrag`

```python
whoosh.highlight.mkfrag(
    text: str,
    tokens,
    startchar=None,
    endchar=None,
    charsbefore: int = 0,
    charsafter: int = 0
) -> Fragment
```

Returns a `Fragment` object based on `Token` objects in `tokens`.

### `get_text`

```python
whoosh.highlight.get_text(
    original: str,
    token,
    replace: bool
) -> str
```

Returns the text to use for a match when formatting. If `replace` is `False`,
returns the original text between `token.startchar` and `token.endchar`. If
`True`, returns `token.text`.

### `set_matched_filter`

```python
whoosh.highlight.set_matched_filter(
    tokens,
    termset: frozenset
) -> Iterator[Token]
```

Marks tokens as matched if their `text` attribute is in `termset`. Used for
phrase-agnostic highlighting.

### `set_matched_filter_phrases`

```python
whoosh.highlight.set_matched_filter_phrases(
    tokens,
    text: str,
    terms,
    phrases
) -> Iterator[Token]
```

Marks tokens as matched using phrase-aware logic. Highlights only tokens that
are part of matched phrases.

### `top_fragments`

```python
whoosh.highlight.top_fragments(
    fragments,
    count: int,
    scorer,
    order,
    minscore: int = 1
) -> list[Fragment]
```

Returns the best `count` fragments sorted by `order`, filtered by `minscore`.

## Constants

### `DEFAULT_CHARLIMIT`

```python
whoosh.highlight.DEFAULT_CHARLIMIT = 2**15
```

Default character limit for fragments.

### Sort Order Constants

```python
whoosh.highlight.FIRST   # Sort passages from earlier in the document first
whoosh.highlight.SCORE   # Sort higher scored passages first
whoosh.highlight.LONGER  # Sort longer passages first
whoosh.highlight.SHORTER # Sort shorter passages first
```

## Formatters

### `Formatter`

```python
class whoosh.highlight.Formatter
```

Base class for formatters. Subclasses implement `format_token()` to define
how matched tokens are rendered.

**Methods:**

- `format_token(text, token, replace=False)`: Returns formatted text for a
  matched token.
- `format_fragment(fragment, replace=False)`: Returns formatted text for a
  `Fragment`.
- `format(fragments, replace=False)`: Returns formatted text for a list of
  fragments, joined by `between`.

**Attributes:**
- `between`: String inserted between formatted fragments (default `"..."`).

### `NullFormatter`

```python
class whoosh.highlight.NullFormatter(Formatter)
```

A formatter that does not modify the string. Returns fragments unformatted.

### `UppercaseFormatter`

```python
class whoosh.highlight.UppercaseFormatter(between="...")
```

Formats matched terms in uppercase.

### `HtmlFormatter`

```python
class whoosh.highlight.HtmlFormatter(
    tagname="strong",
    between="...",
    classname="match",
    termclass="term",
    maxclasses=5,
    attrquote='"'
)
```

Wraps matched terms in HTML tags with CSS class names. Two classes are
applied to each match: `classname` (same for all matches) and `termclass`
(different for each term, e.g. `term0`, `term1`).

- `tagname`: The HTML tag to wrap matches (default `"strong"`).
- `between`: Text inserted between fragments.
- `classname`: CSS class applied to all matched term tags.
- `termclass`: CSS class prefix for per-term classes.
- `maxclasses`: Maximum number of distinct per-term class numbers.
- `attrquote`: Quote character for attribute values.

**Methods:**
- `clean()`: Clears the internal term-to-classname mapping dictionary.

### `GenshiFormatter`

```python
class whoosh.highlight.GenshiFormatter(qname="strong", between="...")
```

Formats matched terms as Genshi event streams (requires the Genshi library).

## Fragmenters

### `Fragmenter`

```python
class whoosh.highlight.Fragmenter
```

Base class for fragmenters. Subclasses implement `fragment_tokens()` and/or
`fragment_matches()`.

**Methods:**
- `must_retokenize()`: Returns `True` if this fragmenter needs to re-tokenize
  the text (calls `fragment_tokens` with all tokens). Returns `False` if it can
  work from matched token positions alone (calls `fragment_matches`).

### `WholeFragmenter`

```python
class whoosh.highlight.WholeFragmenter(charlimit=DEFAULT_CHARLIMIT)
```

Does not fragment text. Returns the entire text as one fragment. Useful for
highlighting short fields.

```python
results.fragmenter = WholeFragmenter()
```

### `SentenceFragmenter`

```python
class whoosh.highlight.SentenceFragmenter(
    maxchars: int = 200,
    sentencechars=".!?",
    charlimit=DEFAULT_CHARLIMIT
)
```

Breaks text at sentence-ending punctuation (`.`, `!`, `?`).

- `maxchars`: Maximum characters per fragment.
- `sentencechars`: Characters that indicate sentence boundaries.
- `charlimit`: Maximum character position to process.

**Note:** Should be used with an analyzer that does not remove stop words.

### `ContextFragmenter`

```python
class whoosh.highlight.ContextFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    charlimit=DEFAULT_CHARLIMIT
)
```

The default fragmenter. Finds matched terms and includes `surround` characters
of context before and after each match.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters to include around matches.
- `charlimit`: Maximum character position to process.

### `PinpointFragmenter`

```python
class whoosh.highlight.PinpointFragmenter(
    maxchars: int = 200,
    surround: int = 20,
    autotrim: bool = False,
    charlimit=DEFAULT_CHARLIMIT
)
```

A non-retokenizing fragmenter that builds fragments from character positions of
matched terms. Faster than `ContextFragmenter` because it doesn't need to
re-tokenize text.

- `maxchars`: Maximum characters per fragment.
- `surround`: Number of context characters around matches.
- `autotrim`: If `True`, trims fragments to the nearest spaces.
- `charlimit`: Maximum character position to process.

### `NullFragmeter`

Alias for `WholeFragmenter`.

### `Fragment`

```python
class whoosh.highlight.Fragment(
    text: str,
    matches,
    startchar: int = 0,
    endchar: int = -1
)
```

Represents a fragment (excerpt) from a hit document. Stores the start and end
character offsets and the list of matched term objects.

**Attributes:**
- `text`: The original source text.
- `matches`: List of objects with `startchar` and `endchar` attributes.
- `startchar`: Start index of the fragment.
- `endchar`: End index of the fragment.
- `matched_terms`: Set of text values of matched terms.

**Methods:**
- `overlaps(fragment)`: Returns `True` if this fragment overlaps the given one.
- `overlapped_length(fragment)`: Returns the combined length of overlapping
  fragments.

### `FragmentScorer`

```python
class whoosh.highlight.FragmentScorer
```

Base class for fragment scoring objects. Subclasses implement `__call__()`
to score a `Fragment`.

### `BasicFragmentScorer`

```python
class whoosh.highlight.BasicFragmentScorer
```

Scores fragments by summing the boosts of matched terms, then multiplying by
the number of distinct matched terms (favors diversity).

## Highlighter

### `Highlighter`

```python
class whoosh.highlight.Highlighter(
    fragmenter=None,
    scorer=None,
    formatter=None,
    always_retokenize: bool = False,
    order=SCORE
)
```

Main highlighter object used by `Searcher.highlights()`.

- `fragmenter`: Fragmenter instance (defaults to `ContextFragmenter`).
- `scorer`: Fragment scorer (defaults to `BasicFragmentScorer`).
- `formatter`: Formatter instance (defaults to `HtmlFormatter(tagname="b")`).
- `always_retokenize`: If `True`, always re-tokenize text instead of using
  character offsets from postings.
- `order`: Sort order for fragments.

**Methods:**
- `highlight_hit(hitobj, fieldname, top=3, minscore=1, strict_phrase=False)`:
  Returns the highlighted string for a single hit in a given field.
- `can_load_chars(results, fieldname)`: Returns `True` if the field supports
  "pinpoint" highlighting using stored character offsets.


## DOCUMENT (FR): Idsets

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Idsets API

Specialized set implementations for storing sorted lists of positive
integers (document IDs). These are more memory-efficient than the built-in
`set` for certain use cases, though they are slower for most operations since
they are pure Python.

## Overview

The `DocIdSet` class is the abstract base class. Concrete implementations
include `BitSet`, `OnDiskBitSet`, `SortedIntSet`, `RoaringIdSet`, and
`MultiIdSet`. The `AutoIdSet` function selects the best implementation
based on the contents.

## Module Functions

### `autoset`

```python
whoosh.idsets.autoset
```

A factory that creates an appropriate `DocIdSet` subclass based on the
contents of a given iterable. If all integers in the set are below 10,000,
returns a `BitSet`; otherwise returns a `SortedIntSet`.

## `DocIdSet`

```python
class whoosh.idsets.DocIdSet
```

Abstract base class for set implementations specialized toward storing sorted
lists of positive integers.

**Inheritance:** Inherits from `set`-like interface.

**Methods:**
- `__eq__(other)`: Compares two `DocIdSet` instances by iterating.
- `__len__()`: Returns the number of elements. Override in subclasses.
- `__iter__()`: Yields elements in sorted order. Override in subclasses.
- `__contains__(i)`: Returns `True` if `i` is in the set.
- `__or__(other)`: Returns `self.union(other)`.
- `__and__(other)`: Returns `self.intersection(other)`.
- `__sub__(other)`: Returns `self.difference(other)`.
- `copy()`: Returns a copy of this set.
- `add(n)`: Adds `n` to the set.
- `discard(n)`: Removes `n` from the set (no error if absent).
- `update(other)`: Adds all elements from `other`.
- `intersection_update(other)`: Removes elements not in `other`.
- `difference_update(other)`: Removes all elements in `other`.
- `invert_update(size)`: In-place inversion over the range `[0, size)`.
- `intersection(other)`: Returns a new set with elements in both.
- `union(other)`: Returns a new set with elements from both.
- `difference(other)`: Returns a new set with elements in self but not other.
- `invert(size)`: Returns a new set that is the inversion over `[0, size)`.
- `isdisjoint(other)`: Returns `True` if no elements are shared.
- `before(i)`: Returns the previous integer in the set before `i`, or `None`.
- `after(i)`: Returns the next integer in the set after `i`, or `None`.
- `first()`: Returns the first (lowest) integer.
- `last()`: Returns the last (highest) integer.

## `BaseBitSet`

```python
class whoosh.idsets.BaseBitSet(DocIdSet)
```

Base class for bitmap-backed `DocIdSet` implementations. Uses a bytes-based
bitmap where each bit represents membership of an integer.

**Abstract Methods to Override:**
- `byte_count()`: Returns the number of bytes in the bitmap.
- `_get_byte(i)`: Returns the byte at index `i`.
- `_iter_bytes()`: Yields all bytes in the bitmap.

**Inherited Methods:** All `DocIdSet` methods with efficient bitmap
implementations of `__len__`, `__iter__`, `__contains__`, `first`, and
`last`.

## `OnDiskBitSet`

```python
class whoosh.idsets.OnDiskBitSet(file, doc_count)
```

A `BaseBitSet` that reads the bitmap from a file on disk, using `mmap` for
memory efficiency.

**Constructor:**
- `file`: A file-like object (opened in binary mode) containing the bitmap.
- `doc_count`: Total number of documents (bits) represented.

```python
from whoosh.idsets import OnDiskBitSet

with open("deletions.dat", "rb") as f:
    bs = OnDiskBitSet(f, doc_count=10000)
    if 42 in bs:
        print("Document 42 is deleted")
```

## `BitSet`

```python
class whoosh.idsets.BitSet
```

A `BaseBitSet` that stores the bitmap in memory as a `bytearray`. Fast for
membership tests and set operations on small ranges of integers.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import BitSet

bs = BitSet([0, 5, 10, 15])
print(5 in bs)  # True
print(bs.first())  # 0
print(len(bs))   # 4
```

**Methods:**
- `from_blob(data)`: Create a `BitSet` from raw bytes.
- `tostring()`: Returns the bitmap as a `bytes` string.
- `set_reverse()`: Prepares the set for reverse iteration.

## `SortedIntSet`

```python
class whoosh.idsets.SortedIntSet
```

A `DocIdSet` that stores integers as a sorted list of Python `int` objects.
More memory-efficient than `BitSet` for sparse sets but slower for membership
tests.

**Constructor:**
- Optional initial iterable of integers.

```python
from whoosh.idsets import SortedIntSet

sis = SortedIntSet([100, 500, 999])
print(500 in sis)  # True
print(sis.after(200))  # 500
```

## `ReverseIdSet`

```python
class whoosh.idsets.ReverseIdSet(child)
```

Wraps another `DocIdSet` to reverse the interpretation of integers. Instead
of representing membership directly, the set represents the *complement* of
the inner set. Useful for representing deleted documents.

**Constructor:**
- `child`: The `DocIdSet` to reverse.

**Example:** If `child` represents documents `{3, 7, 9}`, then
`ReverseIdSet(child)` represents all documents *except* `{3, 7, 9}`.

## `RoaringIdSet`

```python
class whoosh.idsets.RoaringIdSet
```

A `DocIdSet` that partitions integers into 16-bit buckets and uses `BitSet`
within each bucket. More memory-efficient than a single flat `BitSet` for
large, sparse sets of integers.

**Constructor:**
- Optional initial iterable of integers.

**Methods:**
- `from_bytes(data)`: Deserialize from bytes.
- `to_bytes()`: Serialize to bytes.
- `to_bytes_list()`: Returns a list of `(bucket, bytes)` pairs.

## `MultiIdSet`

```python
class whoosh.idsets.MultiIdSet(readers, offsets=None)
```

Combines multiple `DocIdSet` instances into one, handling document ID offsets
automatically. Used for combining deletions across multiple segments.

**Constructor:**
- `readers`: List of `DocIdSet` instances (one per segment).
- `offsets`: Optional list of base docnum offsets for each reader. If
  omitted, offsets are computed automatically.

**Methods:**
- `__contains__(i)`: Checks the appropriate sub-set based on offsets.
- `__iter__()`: Iterates over all integers in all sub-sets.
- `__len__()`: Returns the total count across all sub-sets.


## DOCUMENT (FR): Lang

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Language Support API

Language detection helpers, stemmer selection, stop-word lists, and
language-specific modules (Snowball stemmers, ISRI stemmer, Soundex,
Double Metaphone, etc.).

## Module Overview

The `whoosh.lang` package provides functions for detecting and selecting
language-specific resources (stemmers, stop words) and submodules containing
stemmers for various languages.

## Supported Languages

```python
whoosh.lang.languages = ("ar", "da", "nl", "en", "fi", "fr", "de", "hu",
                         "it", "no", "pt", "ro", "ru", "es", "sv", "tr")
```

Two-letter ISO 639-1 language codes for which stemmers or stop-word lists
are available.

## Language Aliases

```python
whoosh.lang.aliases = { ... }
```

A dictionary mapping alternate language identifiers to their canonical
two-letter codes. Includes ISO 639-3 three-letter codes, English names,
and native-language names.

## Exceptions

### `NoStemmer`

```python
class whoosh.lang.NoStemmer
```

Raised by `stemmer_for_language()` when no stemmer is available for the
given language.

### `NoStopWords`

```python
class whoosh.lang.NoStopWords
```

Raised by `stopwords_for_language()` when no stop-word list is available for
the given language.

## Language Functions

### `two_letter_code`

```python
whoosh.lang.two_letter_code(name) -> str or None
```

Converts a language identifier to its canonical two-letter code. Accepts
two-letter codes, ISO 639-3 codes, English names, and native-language names.

```python
from whoosh.lang import two_letter_code

code = two_letter_code("french")   # 'fr'
code = two_letter_code("deutsch")  # 'de'
code = two_letter_code("español")  # 'es'
```

### `has_stemmer`

```python
whoosh.lang.has_stemmer(lang) -> bool
```

Returns `True` if a stemmer is available for the given language.

### `has_stopwords`

```python
whoosh.lang.has_stopwords(lang) -> bool
```

Returns `True` if a stop-word list is available for the given language.

### `stemmer_for_language`

```python
whoosh.lang.stemmer_for_language(lang) -> callable
```

Returns a stemmer function for the given language. Raises `NoStemmer` if
no stemmer is available.

**Supported languages and stemmers:**
- `"en"` / `"en_porter"`: Original Porter stemmer (`whoosh.lang.porter`)
- `"ar"`: ISRI Arabic stemmer (`whoosh.lang.isri`)
- `"da"`: Danish Snowball stemmer
- `"nl"`: Dutch Snowball stemmer
- `"en"`: English Snowball stemmer
- `"fi"`: Finnish Snowball stemmer
- `"fr"`: French Snowball stemmer
- `"de"`: German Snowball stemmer
- `"hu"`: Hungarian Snowball stemmer
- `"it"`: Italian Snowball stemmer
- `"no"`: Norwegian Snowball stemmer
- `"pt"`: Portuguese Snowball stemmer
- `"ro"`: (no stemmer currently)
- `"ru"`: Russian Snowball stemmer
- `"es"`: Spanish Snowball stemmer
- `"sv"`: Swedish Snowball stemmer
- `"tr"`: (no stemmer currently)

```python
from whoosh.lang import stemmer_for_language

stem = stemmer_for_language("en")
print(stem("running"))  # 'run'
```

### `stopwords_for_language`

```python
whoosh.lang.stopwords_for_language(lang) -> list
```

Returns the stop-word list for the given language. Raises `NoStopWords` if
no stop-word list is available.

```python
from whoosh.lang import stopwords_for_language

stops = stopwords_for_language("en")
```

## Snowball Stemmers

The `whoosh.lang.snowball` subpackage contains stemmers implementing the
Snowball stemming algorithms for various languages.

### Available Stemmers

| Module | Class | Language |
|--------|-------|----------|
| `snowball.english` | `EnglishStemmer` | English |
| `snowball.dutch` | `DutchStemmer` | Dutch |
| `snowball.finnish` | `FinnishStemmer` | Finnish |
| `snowball.french` | `FrenchStemmer` | French |
| `snowball.german` | `GermanStemmer` | German |
| `snowball.hungarian` | `HungarianStemmer` | Hungarian |
| `snowball.italian` | `ItalianStemmer` | Italian |
| `snowball.norwegian` | `NorwegianStemmer` | Norwegian |
| `snowball.portugese` | `PortugueseStemmer` | Portuguese |
| `snowball.russian` | `RussianStemmer` | Russian |
| `snowball.romanian` | `RomanianStemmer` | Romanian |
| `snowball.spanish` | `SpanishStemmer` | Spanish |
| `snowball.swedish` | `SwedishStemmer` | Swedish |
| `snowball.danish` | `DanishStemmer` | Danish |

### Base Classes

```python
class whoosh.lang.snowball.bases._ScandinavianStemmer
class whoosh.lang.snowball.bases._StandardStemmer
```

Internal base classes for Snowball stemmers. User code should use the
language-specific stemmer classes directly.

### `classes`

```python
whoosh.lang.snowball.classes = {"da": DanishStemmer, "nl": DutchStemmer, ...}
```

Dictionary mapping two-letter language codes to Snowball stemmer classes.

## Porter Stemmer

### `whoosh.lang.porter`

The original Porter stemming algorithm, faster but less accurate than
Snowball English stemmer.

#### `stem`

```python
whoosh.lang.porter.stem(w) -> str
```

Stems a single English word using the Porter algorithm.

## ISRI Stemmer

### `whoosh.lang.isri.ISRIStemmer`

```python
class whoosh.lang.isri.ISRIStemmer
```

Arabic stemmer based on the Information Science Research Institute (ISRI)
algorithm. Does not use a root dictionary.

#### `stem`

```python
def ISRIStemmer.stem(word) -> str
```

Stems an Arabic word.

## Double Metaphone

### `whoosh.lang.dmetaphone.double_metaphone`

```python
whoosh.lang.dmetaphone.double_metaphone(text) -> tuple
```

Returns a tuple of `(primary, secondary)` metaphone codes for the given
text, using the Double Metaphone algorithm.

## Soundex

### `whoosh.lang.phonetic`

Soundex implementations for phonetic matching.

#### `soundex_en`

```python
whoosh.lang.phonetic.soundex_en(word) -> str
```

English Soundex encoding.

#### `soundex_esp`

```python
whoosh.lang.phonetic.soundex_esp(word) -> str
```

Spanish Soundex encoding.

#### `soundex_ar`

```python
whoosh.lang.phonetic.soundex_ar(word) -> str
```

Arabic Soundex encoding.

## WordNet Thesaurus

### `whoosh.lang.wordnet.Thesaurus`

```python
class whoosh.lang.wordnet.Thesaurus
```

Provides synonym expansion based on WordNet-style data.

**Methods:**
- `synonyms(word)`: Returns the set of synonyms for `word`.
- `__contains__(word)`: Returns `True` if `word` is in the thesaurus.

### Functions

```python
whoosh.lang.wordnet.parse_file(f) -> dict
whoosh.lang.wordnet.make_index(storage, indexname, word2nums, num2words)
whoosh.lang.wordnet.synonyms(word2nums, num2words, word) -> set
```

## Lovins Stemmer

### `whoosh.lang.lovins`

A suffix-stripping stemmer by Lovins. Functions include:
- `stem(word)`: Main stemming function.
- `remove_ending(word)`: Removes suffixes.
- `fix_ending(word)`: Fixes the word ending after stemming.

## Paice-Husk Stemmer

### `whoosh.lang.paicehusk.PaiceHuskStemmer`

```python
class whoosh.lang.paicehusk.PaiceHuskStemmer(rules)
```

A rule-based stemmer using Paice-Husk rules.

#### `stem`

```python
def PaiceHuskStemmer.stem(word) -> str
```

Stems a word using the Paice-Husk algorithm.

**Usage note:** The module also exposes a pre-configured stemmer:
```python
whoosh.lang.paicehusk.stem = PaiceHuskStemmer(defaultrules).stem
```


## DOCUMENT (FR): Matching

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Matching API

Classes and functions for iterating over and combining result sets during
searching. The matching module is a refactored package exposing the same
public API as the former monolithic module.

## Overview

When you search an index, Whoosh creates `Matcher` objects representing the
postings (document IDs and scores) produced by query objects. Matchers can
be combined (e.g., union, intersection) to build compound queries. The
matching module provides the core `Matcher` class hierarchy, utility
functions, and concrete implementations for various query types.

## Core Matcher Classes

### `Matcher`

```python
class whoosh.matching.Matcher
```

Abstract base class for all matchers. Concrete subclasses implement
`__init__()` and the `_set()` and `_maybe_values()` methods.

**Methods:**

#### `init = property(is_active)`

Property that returns whether the matcher is "active" (at top of segment
postings, not exhausted).

#### `init(view, docnum, score)`

Called when the matcher is initialized.

#### `set(matcher)`

Replaces this matcher with another one.

#### `copy()`

Returns a copy of this matcher.

#### `all_ids()`

Returns a list of docnums matched by this matcher.

#### `matches(matcher)`

Returns `True` if any of the current matches in `self` also match in
`matcher`.

#### `skip_to(docid)`

Advances the matcher to the first match at or after `docid`.

#### `skip_to_intersect(matcher)`

Moves this matcher to the earliest matching docnum that is also matched in
`matcher`.

#### `next()`

Advances the matcher to the next match.

#### `next_in_segment()`

Advances to the next match in the current segment.

#### `next_segment(matcher)`

Advances to the next segment in the context of `matcher`.

#### `is_active(in_segment=False)`

Returns `True` if this matcher has more matches to process.

#### `all_matching_segments()`

Generates `(segment_num, matcher)` pairs for all matching segments.

#### `doc()`

Returns the current document number of this matcher. May advance to next
document if not already on one.

#### `docnum()`

Returns the current docnum (segment-relative) of the matcher.

#### `score()`

Returns the current match's score.

#### `value()`

Returns the current match's value (e.g., the decoded stored value of the
term).

#### `supports()`

Returns `True` if `value()` is supported.

#### `value_matches()`

Returns the value at the current match.

#### `all_values()`

Returns a list of all values in this matcher.

#### `supports_lee()`

Returns `True` if the matcher uses lazy evaluation.

#### `lee`

Returns the current "lazy evaluation extension" value (for term vectors).

#### `spans()`

If the postings include positions, returns a list of `Position` objects for
the current match.

#### `spans()`

Returns the spans (positions) of the match in the current document.

#### `next_type()`

Returns the type of the next match.

#### `copy()`

Returns a shallow copy of this matcher.

### `Child`

```python
class whoosh.matching.Child
```

Mixin class for matchers that wrap other matchers.

### `FilterMixin`

```python
class whoosh.matching.FilterMixin
```

Mixin for matchers used as filters (boolean scoring, no relevance).

### `Custom`

```python
class whoosh.matching.Custom
```

Mixin for matchers that return a custom score from `score()` rather than 1.

### `Constant`

```python
class whoosh.matching.Constant
```

Mixin for matchers whose score is always the same value.

### `Coord`

```python
class whoosh.matching.Coord
```

Mixin for matchers that compute coordination factor (for phrase and other
queries that benefit from it).

## Concrete Matcher Classes

### `ListUnion`

```python
class whoosh.matching.ListUnion(matcher, items, maptype=None)
```

Base class for matchers that combine multiple matchers with a list of keys.

#### `filter`

```python
class whoosh.matching.filter
```

Decorator for creating filter matchers (boolean matchers with no relevance).

### `Union`

```python
class whoosh.matching.Union(matcher, items)
```

Base class for the `OR` operator.

### `Intersection`

```python
class whoosh.matching.Intersection(matcher, items)
```

The `AND` operator. A document matches only if it appears in all the child
matchers.

#### `IntersectionFilter`

```python
class whoosh.matching.IntersectionFilter(matcher, items)
```

A filter (no scoring) version of intersection.

### `And`

```python
class whoosh.matching.And(matcher, items)
```

Alias for `Intersection`.

### `Or`

```python
class whoosh.matching.Or(matcher, items)
```

Alias for `Union`.

### `Not`

```python
class whoosh.matching.Not(matcher, a, b)
```

The `NOT` operator. Matches all documents in `a` that are not in `b`.

### `Require`

```python
class whoosh.matching.Require(matcher, a, b)
```

Matches documents in `a` only if they also appear in `b`, but does not add
`b`'s score.

### `AndNot`

```python
class whoosh.matching.AndNot(matcher, a, b)
```

Matches documents in `a` that are not in `b`.

#### `AndMaybe`

```python
class whoosh.matching.AndMaybe(matcher, a, b)
```

Matches documents in `a`, adding `b`'s score if present.

### `BinaryUnion`

```python
class whoosh.matching.BinaryUnion(items)
```

Efficient intersection of exactly two matchers.

#### `BinaryUnion2`

```python
class whoosh.matching.BinaryUnion2
```

Optimized binary union for two items.

### `TreeMatcher`

```python
class whoosh.matching.TreeMatcher
```

A matcher that wraps a `Tree` object for combining results.

### `NestedParent`

```python
class whoosh.matching.NestedParent(parent, child, bools=False)
```

Matches parent documents that have at least one child document matched by
the child matcher. Used for nested document queries.

### `NestedChildren`

```python
class who which.matching.NestedChildren(parentmatch, child)
```

Matches child documents for a given parent document.

### `LengthMatcher`

```python
class whoosh.matching.LengthMatcher(child, q, polarity=False)
```

Matches documents based on field length (used by `Every` query).

### `Filter`

```python
class whoosh.matching.Filter(matcher)
```

Converts any matcher into a filter (no scoring).

### `AlwaysFilter`

```python
class whoosh.matching.AlwaysFilter
```

A filter that matches all documents.

### `NeverFilter`

```python
class whoosh.matching.NeverFilter
```

A filter that matches no documents.

### `PseudoMatcher`

```python
class whoosh.matching.PseudoMatcher
```

Base class for pseudo-matchers used in span queries.

## Matching Utilities

### `current_spans`

```python
whoosh.matching.current_spans(matcher) -> list
```

Returns a list of `Span` objects for the current match in `matcher`, or an
empty list if the matcher doesn't support positions.

### `disjunction_score`

```python
whoosh.matching.disjunction_score(matcher) -> float
```

Returns the sum of `matcher.score()` and the scores of all child matchers of
type `Union`.

### `intersection_score`

```python
whoosh.matching.intersection_score(matcher) -> float
```

Returns the sum of `matcher.score()` and all child matchers of type
`Intersection`.

### `child_count`

```python
whoosh.matching.child_count(matcher) -> int
```

Returns the number of child matchers in `matcher`.

### `has_quality`

```python
whoosh.matching.has_quality(matcher) -> bool
```

Returns `True` if `matcher` has a `query` attribute (i.e., is a
`QueryMatcher`-derived object, or a combination of such matchers).

### `has_untranslated`

```python
whoosh.matching.has_untranslated(matcher) -> bool
```

Returns `True` if the matcher has an `untranslated` attribute (set by
certain wrapper matchers like `TimeLimited`).

### `wrap`

```python
whoosh.matching.wrap(matcher)
```

Returns `matcher` if it has a `.copy()` method, otherwise wraps it in an
`AutoMatcher`.

### `wrap2`

```python
whoosh.matching.wrap2(a, b, m)
```

Returns either a `BinaryUnion2` or an `AutoMatcher` depending on whether `a`
and `b` are list-compatible.

### `unified`

```python
whoosh.matching.unified(matcher)
```

Returns `matcher` if it has an `untranslated` attribute, otherwise returns
`None`.

### `deletion`

```python
whoosh.matching.deletion(matcher)
```

If `matcher` has a `parent` attribute, returns the parent, otherwise returns
`None`.

### `AutoMatcher`

```python
class whoosh.matching.AutoMatcher(m, **kwargs)
```

A general-purpose matcher that wraps arbitrary objects and adds default
behavior for scoring, docnums, and other features. Created by `wrap()`.

### `MatchingTimeLimit`

```python
class whoosh.matching.MatchingTimeLimit
```

A lightweight exception raised when a query matcher exceeds a time limit.

### `TimeLimited`

```python
class whoosh.matching.TimeLimited(child, maxsteps=100, timeout=None, currenttime=None)
```

Wrapper that wraps a `Matcher` to enforce a time limit. Raises
`MatchingTimeLimit` if the time limit is exceeded.

**Parameters:**
- `child`: The matcher to wrap.
- `maxsteps`: Check time every N documents (default `100`).
- `timeout`: Maximum time in seconds (default `None`, no limit).
- `currenttime`: Optional function to use for getting the current time.

### `TermMatcher`

```python
class whoosh.matching.TermMatcher(postings, text, qname, scorer=None, boost=1.0)
```

Matches documents containing a specific term.

**Constructor:**
- `postings`: A `Postings` object from the index reader.
- `text`: The term text.
- `qname`: The query name for this term.
- `scorer`: Optional `Scorer` object.
- `boost`: Boost factor for this term's score.

### `MultiScorer`

```python
class whoosh.matching.MultiScorer(numgroups, start_i=0)
```

A `Scorer` that combines the scores from multiple scorers into one, weighted
across groups of segments.

### `RangeMatcher`

```python
class whoosh.matching.RangeMatcher(start_matcher, end_matcher, query)
```

Matches documents within a range of term values.

### `RegexMatcher`

```python
class whoosh.matching.RegexMatcher(regex, qname, boost=1.0)
```

Matches documents whose terms match a compiled regex.

### `SpanMatcher`

```python
class whoosh.matching.SpanMatcher(matcher, order=0, end=0)
```

Matches spans (positions) within documents.

### `SpanOverlap`

```python
class whoosh.matching.SpanOverlap(l, r)
```

Matches overlapping spans from two matchers.

### `SpanNear`

```python
class whoosh.matching.SpanNear(l, r, slop=1, ordered=True)
```

Matches spans that are near each other within a document.

### `SpanCondition`

```python
class whoosh.matching.SpanCondition(l, r)
```

Matches a condition on spans.

### `SpanBefore`

```python
class whoosh.matching.SpanBefore(l, r, end=0)
```

Matches spans before a given position.

### `SpanAfter`

```python
class whoosh.matching.SpanAfter(l, r, end=0)
```

Matches spans after a given position.

### `SpanOutside`

```python
class whoosh.matching.SpanOutside(l, r, end=0)
```

Matches spans outside a given range.

### `SpanFirst`

```python
class whoosh.matching.SpanFirst(l, start=0, end=1)
```

Matches spans at the beginning of a document.

### `SpanNot`

```python
class whoosh.matching.SpanNot(l, r)
```

Matches spans in `l` that are not in `r`.

### `SpanOr`

```python
class whoosh.matching.SpanOr(items)
```

Logical OR for span matchers.

### `SpanAnd`

```python
class whoosh.matching.SpanAnd(l, r)
```

Logical AND for span matchers.


## DOCUMENT (FR): Middleware

# API Middleware

Pipeline de middleware pour les opérations d'indexation et de recherche.

## Classes principales

### Middleware

```python
class whoosh.middleware.base.Middleware
```

Classe de base pour tous les middlewares.

#### Méthodes

| Hook | Signature | Appelé quand |
|------|-----------|--------------|
| startup | (context) -> context | Initialisation writer/searcher |
| shutdown | (context) -> context | Nettoyage writer/searcher |
| before_index | (context) -> context | Avant l'ajout d'un document |
| after_index | (context) -> context | Après l'ajout d'un document |
| before_delete | (context) -> context | Avant la suppression |
| after_delete | (context) -> context | Après la suppression |
| before_search | (context) -> context | Avant la recherche |
| after_search | (context) -> context | Après les résultats |
| on_error | (context, exc) -> None | Sur exception |
| on_commit | (context) -> None | Après le commit |

### MiddlewareContext

```python
class whoosh.middleware.context.MiddlewareContext(
    operation: str,
    metadata: dict | None = None
)
```

Attributs:

| Attribut | Type | Description |
|----------|------|-------------|
| `operation` | str | `"index"`, `"search"`, `"delete"`, `"commit"` |
| `query` | Any | Requête (pour `search`) |
| `results` | Any | Résultats de la recherche |
| `document` | dict | Document à indexer (pour `index`) |
| `docnum` | int | Numéro du document (pour `delete`) |
| `metadata` | dict | Données par requête (request_id, trace_id) |

### MiddlewareChain

```python
class whoosh.middleware.base.MiddlewareChain(middlewares)
```

Orchestre l'exécution des middlewares dans l'ordre.

#### Méthodes

| Méthode | Description |
|---------|-------------|
| `chain.add(middleware)` | Ajoute un middleware |
| `chain.run_before(hook, context)` | Exécute les hooks before |
| `chain.run_after(hook, context)` | Exécute les hooks after |
| `chain.run_before_all(hook, context)` | Exécute tous les hooks before |
| `chain.run_after_all(hook, context)` | Exécute tous les hooks after |

### SettingGuard

```python
class whoosh.middleware.base.SettingGuard(
    field: str | None = None,
    default: bool = False
)
```

Vérifie et réinitialise les settings de middleware.

### Skip

```python
class whoosh.middleware.base.Skip(metadata)
```

Exception pour sauter une opération tout en la commitant.

## Intégration

### apply_middleware_to_writer

```python
def apply_middleware_to_writer(
    writer: IndexWriter,
    middlewares: list[Middleware]
) -> IndexWriter
```

Retourne un writer enveloppé par les middlewares.

### apply_middleware_to_searcher

```python
def apply_middleware_to_searcher(
    searcher: Searcher,
    middlewares: list[Middleware]
) -> Searcher
```

Retourne un searcher enveloppé par les middlewares.

## Exceptions

```python
class MiddlewareError(Exception)
class StopOperation(Exception)
class SkipOperation(Skip)
```


## DOCUMENT (FR): Modern

# Fournisseurs de stockage

Whoosh-NG fournit des backends de stockage pluggables via les contrats
`SyncStorageProvider` / `AsyncStorageProvider`. Cela permet de persister
l'index sur disque local, SQLite, S3, ou une configuration hybride cache +
distant sans modifier le writer ni l'index.

## Vue d'ensemble de l'architecture

### Niveau 1 : SnapshotStorage (Simple)

```
Writer → FS local → Commit → Upload Segment → S3
Reader → Download Segment → Open local
```

Très simple à maintenir. Utilisez `SnapshotStorage` quand vous voulez S3
comme cible de sauvegarde/restauration simple sans la complexité d'un cache
local.

### Niveau 2 : CachedObjectStorage (Recommandé pour la production)

```
+----------+
|  MinIO   |
+----------+
     ^
     |
 Sync |
     v
+-----------+   Couche Cache   +-----------+
| Searcher  |<--------------->| Writer    |
+-----------+                 +-----------+
        |
        v
 Local SSD
```

- L'index vit sur SSD
- S3 sert de réplication
- Les segments sont poussés après commit
- Restauration possible à tout moment

C'est exactement ce que font beaucoup de systèmes de recherche distribués
modernes.

## Fournisseurs disponibles

| Fournisseur | Type | Backend | Cas d'usage |
|-------------|------|---------|-------------|
| `FileStorage` | sync | système de fichiers local | Single-node, pas de cloud |
| `AsyncFileStorage` | async | système de fichiers local | Single-node async |
| `S3Storage` | sync | compatible S3 | Accès S3 direct |
| `SnapshotStorage` | sync | compatible S3 | Sauvegarde/restauration simple |
| `HybridStorage` | sync | cache local + distant | **Production** (alias : `CachedObjectStorage`) |
| `AsyncHybridStorage` | async | cache local + distant | Production async |

Tous les fournisseurs sont importables depuis `whoosh_modern.storage`.

## FileStorage

Stockage local sur système de fichiers. Les clés sont des chemins relatifs
sous ``root``.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

## AsyncFileStorage

Variante async de ``FileStorage``. Toutes les opérations s'exécutent dans
un thread de travail via ``asyncio.to_thread``.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## S3Storage

Stockage blobs compatible S3. ``boto3`` est requis uniquement lorsque ce
fournisseur est utilisé ; il est importé paresseusement pour que le reste
de Whoosh-NG n'en dépende pas. Un ``client`` peut être injecté pour les tests.

```python
from whoosh_modern.storage import S3Storage

storage = S3Storage(bucket="mon-bucket", prefix="segments")
storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

Installer la dépendance optionnelle :

```bash
pip install whoosh-ng[s3]
```

## SnapshotStorage

Stockage snapshot S3 simple sans cache local. C'est la stratégie la plus
simple :

- Écriture : upload du segment directement vers S3
- Lecture : download du segment depuis S3 vers un fichier temporaire local

Utilisez ceci quand vous voulez S3 comme cible de sauvegarde/restauration
simple sans la complexité d'un cache local.

```python
from whoosh_modern.storage import SnapshotStorage

storage = SnapshotStorage(
    local_path="./index",
    bucket="mon-bucket",
    prefix="snapshots",
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
```

## HybridStorage / CachedObjectStorage

`HybridStorage` compose un cache local avec un backend distant. Le distant
est la source de vérité ; le cache local est une couche de performance
write-through.

`CachedObjectStorage` est un alias de `HybridStorage` qui exprime mieux
l'intention : un cache d'objets local synchronisé avec S3.

C'est l'architecture recommandée pour les déploiements production avec des
motifs de lecture répétés.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

distant = S3Storage(bucket="mon-bucket", prefix="segments")
stockage = HybridStorage(local_cache="./cache", remote=distant)

# Write-through : le distant est la source de vérité, le cache est mis à jour
stockage.write("segment_1.dat", b"data")

# Première lecture : miss cache → fetch depuis S3, write-through dans le cache
data = stockage.read("segment_1.dat")

# Deuxième lecture : hit cache → servi depuis le disque local, zéro réseau
data = stockage.read("segment_1.dat")

# Forcer le rafraîchissement depuis le distant
stockage.invalidate("segment_1.dat")

# Pré-chauffer le cache
stockage.prefetch(["segment_2.dat", "segment_3.dat"])
```

### Chemin de lecture

1. hit cache local → retour immédiat
2. miss → lecture depuis le distant, write-through dans le cache, retour

### Chemin d'écriture

- ``distant.write(key, data)`` (source de vérité)
- en cas de succès → ``local_cache.write(key, data)``
- en cas d'échec → lever l'erreur avant de polluer le cache

### Éviction du cache

Le cache local est limité par `max_cache_size_mb` (défaut 1024 Mo). Quand
la limite est atteinte, les entrées les plus anciennes sont évincées selon
une politique LRU.

### `list_keys`

`list_keys()` utilise le distant comme source de vérité car le cache n'est
que partiel. Passez `include_cache=True` pour retourner l'union des clés
distant et cache.

## AsyncHybridStorage

Variante async de ``HybridStorage``. Les opérations distantes s'exécutent
dans un thread de travail via ``asyncio.to_thread`` pour ne jamais bloquer
la boucle d'événements.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

distant = S3Storage(bucket="mon-bucket", prefix="segments")
stockage = AsyncHybridStorage(local_cache="./cache", remote=distant)

async def main() -> None:
    await stockage.awrite("segment_1.dat", b"data")
    data = await stockage.aread("segment_1.dat")
    await stockage.adelete("segment_1.dat")
    cles = await stockage.alist_keys()

asyncio.run(main())
```

## Utilisation avec SearchApplication

```python
from whoosh_modern import SearchApplication, SQLSource
from whoosh_modern.storage import HybridStorage, S3Storage

distant = S3Storage(bucket="mon-bucket", prefix="segments")
stockage = HybridStorage(local_cache="./cache", remote=distant)

app = SearchApplication(
    source=SQLSource(query="SELECT * FROM produits", connection=engine),
    storage=stockage,
)
app.build()
resultats = app.index.search("laptop")
```

## Benchmarks de performance

Les benchmarks ont été exécutés contre une instance MinIO locale en utilisant
un index Whoosh de 28,89 Mo (2 fichiers de segment). Les résultats indiquent
les performances relatives entre les stratégies sur un stockage compatible S3.

| Stratégie | Sauvegarde (Mo/s) | Restauration (Mo/s) | Notes |
|-----------|-------------------|---------------------|-------|
| `1_obj_per_segment` | 39.44 | 139.72 | Meilleur débit de restauration ; le plus simple |
| `compressed_zstd` | 31.56 | 133.74 | Bande passante réduite, overhead CPU |
| `hybrid_cache_s3` | 44.97 | 133.61 | Meilleure sauvegarde ; lectures cache chaud excellentes |
| `1_obj_per_posting_list` | 0.28 | 4.79 | **À éviter** : millions de petits objets tuent S3 |

### Recommandations

- **Par défaut** : `S3Storage` avec 1 objet par fichier de segment. Il offre
  le meilleur débit de restauration et est le plus simple à exploiter.
- **Production avec lectures répétées** : `HybridStorage(cache_local, S3Storage)`.
  Après le premier accès, les lectures suivantes sont servies depuis le disque
  local à ~133 Mo/s.
- **À éviter** : 1 objet par posting list. S3 n'est pas optimisé pour des
  millions de petits objets ; la latence et le coût explosent.
- **Compression** : ZSTD réduit la taille des transferts de ~20-30% au prix
  d'un overhead CPU. À utiliser quand la bande passante réseau est le
  goulot, pas quand le CPU l'est.

### Exécution des benchmarks

```bash
# Démarrer MinIO
docker run -d --name minio-benchmark -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio:latest server /data --console-address ":9001"

# Lancer le benchmark synthétique
python benchmark/s3_storage_benchmark.py

# Lancer le benchmark avec un vrai index Whoosh (nécessite le CSV customers)
python benchmark/s3_storage_benchmark_real.py
```


## DOCUMENT (FR): Overview

# Vue d'ensemble API

Cette section fournit une référence complète de l'API publique de Whoosh-NG.

## Modules

| Module | Description |
|--------|-------------|
| `whoosh.index` | Gestion des indexes |
| `whoosh.fields` | Types de champs et schéma |
| `whoosh.writing` | Writers et politiques de fusion |
| `whoosh.searching` | Searcher, Results, collectors |
| `whoosh.query` | Classes de requêtes |
| `whoosh.qparser` | Analyseur de requêtes |
| `whoosh.analysis` | Tokenizers, filtres, analyseurs |
| `whoosh.highlight` | Surbrillance des résultats |
| `whoosh.spelling` | Correction orthographique |
| `whoosh.sorting` | Facettes et tri |
| `whoosh.event_bus` | Système d'événements |
| `whoosh.hooks` | Système de hooks |
| `whoosh.middleware` | Pipeline de middleware |
| `whoosh.plugins` | Système de plugins et registres |
| `whoosh.backends` | Backends de stockage |
| `whoosh.vector` | Providers de recherche vectorielle |
| `whoosh_modern.autocomplete` | Providers d'autocomplétion |
| `whoosh_fastapi` | Intégration FastAPI |

## Référence rapide

### Cycle de vie d'un index

```python
from whoosh.index import create_in, open_dir, exists_in

ix = create_in("indexdir", schema)
ix = open_dir("indexdir")
exists = exists_in("indexdir")
```

### Écriture

```python
with ix.writer() as writer:
    writer.add_document(champ1=val1, champ2=val2)
```

### Lecture

```python
from whoosh.qparser import QueryParser

with ix.searcher() as searcher:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("requête")
    results = searcher.search(q)
```

### Schéma

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    titre=TEXT(stored=True),
    chemin=ID(stored=True, unique=True),
    compte=NUMERIC(int, stored=True)
)
```


## DOCUMENT (FR): Plugins

# API Plugins

Système de plugins, registres de providers et discovery.

## PluginManager

```python
class whoosh.plugins.manager.PluginManager
```

Point d'entrée unique pour gérer tous les plugins.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `PluginManager.load_plugins()` | Auto-découvre et charge les entry points |
| `PluginManager.register(name, plugin)` | Enregistre un plugin |
| `PluginManager.enable(name)` | Active un plugin |
| `PluginManager.disable(name)` | Désactive un plugin |
| `PluginManager.get(name)` | Retourne le plugin par nom |
| `PluginManager.list_plugins()` | Liste des plugins enregistrés |

**Exemple:**
```python
from whoosh.plugins.manager import PluginManager

# Programmatique
PluginManager.register("mon_plugin", MonPlugin())
PluginManager.enable("mon_plugin")

# Entry points (dans pyproject.toml)
# [project.entry-points."whoosh_ng.plugins"]
# mon_plugin = "mon_package.plugin:MonPlugin"
PluginManager.load_plugins()
```

## BasePlugin

```python
class whoosh.plugins.base.BasePlugin
```

Classe de base pour tous les plugins.

### Méthodes à implémenter

| Méthode | Appelée quand |
|---------|---------------|
| `setup(registry)` | Plugin activé |
| `teardown(registry)` | Plugin désactivé |
| `on_startup()` | Démarrage application |
| `on_shutdown()` | Arrêt application |

### Attributs

| Attribut | Description |
|----------|-------------|
| `name` | Nom unique |
| `version` | Version du plugin |
| `dependencies` | Liste des noms de plugins requis |

## Registre

```python
class whoosh.registry.Registry
```

Registre global pour les providers de tous types.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `registry.register(name, provider, category)` | Enregistre un provider |
| `registry.get(name, category)` | Récupère un provider |
| `registry.unregister(name, category)` | Supprime un provider |
| `registry.list_providers(category)` | Liste tous les providers d'une catégorie |

**Catégories courantes:**
- `"vector_provider"`
- `"autocomplete_provider"`
- `"storage_provider"`
- `"middleware"`

## VectorRegistry

```python
from whoosh.registry import VectorRegistry

VectorRegistry.register("numpy", NumpyProvider(), "mon_app")
provider = VectorRegistry.get("numpy", "mon_app")
```

## Exception

```python
class PluginNotFoundError(Exception)
```


## DOCUMENT (FR): Query

# API Requêtes

Construisez des requêtes complexes avec l'API de Whoosh-NG.

## Classes principales

### Query

Classe de base pour toutes les requêtes.

```python
class whoosh.query.Query
```

#### Méthodes

| Méthode | Description |
|---------|-------------|
| `q.all(methodname, *args)` | Applique une méthode à tous les termes |
| `q.normalize()` | Normalise la représentation textuelle |
| `q.replace(fieldname, old, new)` | Remplace un terme |
| `q.exclude(term)` | Exclut un terme spécifique |
| `q.fieldname` | Champ principal de la requête |
| `q.children()` | Requêtes enfants (pour opérateurs) |

## Requêtes booléennes

### And

```python
And(require, boost=1.0)
# Tous doivent matcher
```

### Or

```python
Or(require, boost=1.0)
# Un seul doit matcher
```

### Not

```python
Not(require, exclude=None)
```

### Requête Term

```python
Term(fieldname, text, boost=1.0)
```

## Plages

### NumericRange

```python
NumericRange(fieldname, start, end, startexcl=False, endexcl=False)
```

### DateRange

```python
DateRange(fieldname, start, end, startexcl=False, endexcl=False)
```

## Phrase et proximité

### Phrase

```python
Phrase(fieldname, words, slop=1, boost=1.0)
```

### Distance / Near

### Prefix

```python
Prefix(fieldname, text, boost=1.0)
```

## Combinateurs avancés

### Every

```python
Every(fieldname, boost=1.0)
```

### Null

```python
NullQuery
```

## Opérateurs de requête globaux

| Requête | Description |
|---------|-------------|
| `Term` | Égalité exacte (pas d'analyse) |
| `Variations` | Variantes lexicales |
| `FuzzyTerm` | Recherche floue |
| `Wildcard` | Jokers (lent sur grands corpus) |
| `Regex` | Expression régulière |

## Construction manuelle

```python
from whoosh.query import (
    Term, And, Or, Not, Phrase, NumericRange, DateRange
)

q = And([
    Term("status", "published"),
    NumericRange("date", 2020, 2025),
    Or([Term("tags", "python"), Term("tags", "recherche")]),
    Phrase("content", ["tutoriel", "whoosh"])
])
```


## DOCUMENT (FR): Reading

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Reading API

Classes and functions for reading from an index. The reading module is a
refactored package exposing the same public API as the former monolithic
module.

## Overview

The reading module provides classes for accessing documents, terms, and
postings in an index. The main entry points are `IndexReader` objects obtained
from a searcher. These readers allow you to enumerate terms, access stored
fields, iterate postings, and get term frequencies.

## Core Classes

### `IndexReader`

```python
class whoosh.reading.IndexReader
```

Abstract base class for reading index data. Concrete subclasses include
`SegmentReader` and `MultiReader` (which wraps multiple segment readers).

### `MultiReader`

```python
class whoosh.reading.MultiReader(readers, base=None)
```

Combines multiple `IndexReader` instances into one. All docnums are treated
as relative to the combined index.

**Constructor:**
- `readers`: A list of `IndexReader` instances.
- `base`: Optional list of cumulative document count offsets for each reader.

**Methods:**

#### `doc_frequency(fieldname, text)`

Returns the total number of documents that have the given term in the given
field across all sub-readers.

#### `documents()`

Yields dictionaries of stored fields for each document in the index.

#### `stored_fields(docnum)`

Returns a dictionary of stored field values for the given document number
(index-wide docnum).

```python
r = my_index.reader()
print(r.stored_fields(20))
```

#### `all_stored_fields()`

Yields a `(docnum, stored_fields)` tuple for each document in the index.

#### `terms(fieldname)`

Yields `(fieldname, text)` tuples for every term in the given field.

#### `terms_from(segmentreader,fieldnameprefix)`

Low-level method for multi-reader.

#### `has_termvector(docnum, fieldname)`

Returns `True` if the document has a term vector for the given field.

#### `term_vector(docnum, fieldname)`

Returns a `TermVector` for the given document and field.

#### `is_deleted(docnum)`

Returns `True` if the given document (index-wide docnum) is deleted.

#### `all_doc_ids()`

Returns a sorted array of non-deleted document IDs.

#### `min_spam(fieldname)`

Returns the minimum spam value for the given field.

#### `set_spam(fieldname)`

Returns the set spam value for the given field.

#### `has_exact_length(docnum)`

Returns `True` if the exact length is known for `docnum`.

#### `doc_field_length(docnum, fieldname=None, default=1)`

Returns the length of the given field in the given document.

```python
r = my_index.reader()
length = r.doc_field_length(20, "content")
```

#### `max_field_length(fieldname)`

Returns the maximum length of the given field across all documents.

#### `iter_fieldname`

Low-level method for multi-reader.

#### `lexicon(fieldname)`

Returns an array of all unique terms in the given field, sorted.

#### `expanded_lexicon(fieldname)`

Low-level method that yields terms without the overhead of building an array.

#### `term_info(fieldname, text)`

Returns a `TermInfo` object for the given term, or `None` if the term does
not appear in the index.

#### `terminfos(fieldname)`

Yields `(text, TermInfo)` pairs for the given field.

#### `postings(fieldname, text, stype=None)`

Returns a `Matcher` for the postings list of the given term.

```python
r = my_index.reader()
m = r.postings("content", "whoosh")
for docnum, score in m:
    print("doc %d has term" % docnum)
```

#### `_all_postings(fieldname)`

Low-level. Yields `(text, matcher)` pairs for all terms in a field.

#### `_posting_fragments()`

Low-level.

#### `has_vector(docnum, fieldname)`

Returns `True` if the given field has a term vector in the given document.

#### `vectors(docnum)`

Yields `(fieldname, TermVector)` pairs for all term vectors in the document.

#### `all_items(fieldname)`

Yields `(term, weight, docfreq)` tuples for every term in the given field.

#### `frequency(fieldname, text)`

Returns the total frequency of the term across all documents.

#### `idf(term)`

Returns an iterator of `(docnum, idf)` pairs for the given term.

#### `spelling`

Returns a `SpellingAnalyzer` for the given field.

#### `doc_term(slicenum, fieldname, word)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

#### `doc_diff(slicenum, fieldname, text, num)`

Returns `(df, weight)` for `word` in `fieldname` in segment `slicenum`.

### `SegmentReader`

```python
class whoosh.reading.SegmentReader(segment, schema, storage, base=True)
```

Reader for a single segment of the index.

**Constructor:**
- `segment`: The `Segment` object.
- `schema`: The `Schema` object.
- `storage`: The `Storage` instance.
- `base`: Base document number offset (usually `True`, meaning compute it).

### `MultiID3Reader`

```python
class whoosh.reading.MultiID3Reader(readers, base)
```

Combines multiple readers that have ID3 codec.

### `TermInfo`

```python
class whoosh.reading.TermInfo(
    df=0,
    weight=0,
    minlength=0,
    maxlen=0,
    maxnum=0,
    numdocs=0,
    scorable=True
)
```

Information about a term in the index.

**Attributes:**
- `df`: Document frequency (number of documents containing the term).
- `weight`: Total term frequency across all documents.
- `minlength`: Minimum document length where the term appears.
- `maxlength`: Maximum document length where the term appears. This is `0`
  if lengths are not stored.
- `maxnum`: Maximum number of occurrences per document.
- `numdocs`: Number of documents where the term has a non-zero contribution
  to the score.
- `scorable`: Whether this term is scorable.

## Term Vector

### `TermVector`

```python
class whoosh.reading.TermVector(docnum, fieldname, format_, terms, store_term_vector)
```

Represents the term vector for a single document/field pair.

**Methods:**

#### `tokens(text=None)`

Yields `(t, w, v, p)` tuples for terms in this field.

- `t`: The term string.
- `w`: The term weight (frequency in this document).
- `v`: The list of positions where the term occurs. (`None` if positions
  are not stored.)
- `p`: The list of characters where the term occurs. (`None` if character
  vectors are not stored.)

#### `items(text=None)`

Like `tokens()` but includes term strings in the result.

```python
tv = my_index.reader().term_vector(0, "content")
for token, frequency, positions, chars in tv.tokens():
    print(token, frequency, positions)
```

**Parameters:**
- `text`: Optional `Bytes` object. If given, only yield terms starting with
  this text (used for multi-byte tokenization).

## Reader Utilities

### `get_storage`

```python
whoosh.reading.get_storage(searcher) -> Storage
```

Returns the storage object associated with the searcher.

### `get_index_schema`

```python
whoosh.reading.get_index_schema(searcher) -> Schema
```

Returns the schema object associated with the searcher.

### `load_termdocs`

```python
whoosh.reading.load_termdocs(reader, fieldname, text) -> list
```

Returns a list of document numbers that have the given term.

### `read_pattern`

```python
whoosh.reading.read_pattern(reader, fieldname, expression) -> list
```

Returns sorted term list from `reader.lexicon(fieldname)` filtered to those
matching `expression`.

### `read_terminfo`

```python
whoosh.reading.read_terminfo(reader, fieldname, text) -> TermInfo or None
```

Returns a `TermInfo` for the given term, or `None` if not found.


## DOCUMENT (FR): Searching

# API Recherche

Exécuter des requêtes et récupérer les résultats.

## Searcher

```python
class whoosh.searching.Searcher
```

Interface principale pour lire l'index.

### Méthodes

| Méthode | Description |
|---------|-------------|
| `search(query, limit=10)` | Exécute une requête |
| `search_page(query, pagenum, pagelen=10)` | Récupère une page de résultats |
| `search_with_collector(query, collector)` | Recherche avancée avec collector |
| `find(fieldname, text)` | Recherche dans un champ |
| `documents(**kwargs)` | Documents stockés correspondants |
| `document(**kwargs)` | Un document stocké |
| `lexicon(fieldname)` | Liste des termes d'un champ |
| `all_stored_fields()` | Itère sur tous les champs stockés |

## Results

```python
class whoosh.searching.Results
```

Conteneur de résultats (semblable à une liste).

### Méthodes

| Méthode/attribut | Description |
|------------------|-------------|
| `len(results)` | Nombre total de correspondances |
| `results.scored_length()` | Nombre de résultats scorés |
| `results[0]` | Premier résultat |
| `results[0:10]` | Slice de résultats |
| `results.has_matched_terms()` | Vérifie si les termes matchés sont collectés |
| `results.filtered_count` | Nombre de documents filtrés |
| `results.collapsed_counts` | Comptage par clé d'effondrement |

## Hit

```python
class whoosh.searching.Hit
```

Un document matché.

### Attributs

- `hit["champ"]`: Valeur du champ stocké
- `hit.score`: Score de pertinence
- `hit.docnum`: Numéro interne du document

### Méthodes

| Méthode | Description |
|---------|-------------|
| `hit.highlights("content", top=3)` | Extrait surlignés |
| `hit.matched_terms()` | Termes matchés (si `terms=True`) |

## Highlight

```python
from whoosh.highlight import highlight, Fragment

snippets = hit.highlights("content", top=3)
```

## Collectors

```python
from whoosh.collectors import FacetCollector, TimeLimitCollector
```

## Tri et facettes

```python
from whoosh import sorting

facet = sorting.FieldFacet("categorie")
results = searcher.search(query, sortedby="date")
```


## DOCUMENT (FR): Sorting

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Sorting API

Classes and functions for faceting and sorting search results. The sorting
module is a refactored package exposing the same public API as the former
monolithic module.

## Overview

Sorting and faceting use `FacetType` objects to compute sort keys for documents.
A `FacetType` creates a `Categorizer` that computes a key for each document.
The key is used for sorting and grouping. `FacetMap` objects hold the
results of grouping documents by a facet.

## Facet Types

### `FacetType`

```python
class whoosh.sorting.FacetType
```

Base class for "facets" — aspects that can be sorted and/or faceted.

**Attributes:**
- `maptype`: Default `FacetMap` class to use for this facet.

**Methods:**

#### `categorizer(global_searcher)`

Returns a `Categorizer` corresponding to this facet.

- `global_searcher`: A parent searcher for global document ID references.

#### `map(default=None)`

Returns a `FacetMap` instance for holding facet results.

#### `default_name()`

Returns the default name for this facet (default `"facet"`).

### `Categorizer`

```python
class whoosh.sorting.Categorizer
```

Base class for objects that compute a key value for a document for sorting and
faceting. Created by `FacetType` objects via `categorizer()`.

**Attributes:**
- `allow_overlap (bool)`: If `True`, use `keys_for()` to allow overlapping
  groups. Default `False`.
- `needs_current (bool)`: If `True`, the categorizer needs the matcher to be
  in a valid state when `key_for()` is called. Default `False`.

**Methods:**

#### `set_searcher(segment_searcher, docoffset)`

Called when the collector moves to a new segment. Sets up segment-specific
data.

- `segment_searcher`: The atomic sub-searcher for the current segment.
- `docoffset`: Offset of the segment's docnums relative to the full index.

#### `key_for(matcher, segment_docnum)`

Returns a sort key for the current match.

- `matcher`: A `Matcher` object. If `needs_current` is `False`, do not use
  this object as it may be inconsistent.
- `segment_docnum`: Segment-relative document number.

#### `keys_for(matcher, segment_docnum)`

Yields multiple keys for the current match. Called instead of `key_for()`
when `allow_overlap` is `True`.

#### `key_to_name(key)`

Translates the sort key into a human-readable representation for facet
group names (e.g., converts an integer date sort key to a `datetime`).

### `FieldFacet`

```python
class whoosh.sorting.FieldFacet(
    fieldname,
    reverse=False,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets by the contents of a field.

**Constructor:**
- `fieldname`: Name of the field to sort/facet on.
- `reverse`: If `True`, reverse the sort order.
- `allow_overlap`: If `True`, allow documents to appear in multiple groups
  when they have multiple terms in the field.
- `maptype`: `FacetMap` class for holding results.

```python
paths = FieldFacet("path", reverse=True)
tags = FieldFacet("tag")
results = searcher.search(myquery, sortedby=paths, groupedby=tags)
```

### `ColumnCategorizer`

Categorizer that reads values from a column for sorting. Used when a field
has a column type.

### `ReversedColumnCategorizer`

Categorizer that reverses column values for fields that are not naturally
reversible.

### `OverlappingCategorizer`

```python
class whoosh.sorting.OverlappingCategorizer
```

Categorizer used when `allow_overlap=True`. A single document can belong to
multiple facet groups.

### `PostingCategorizer`

```python
class whoosh.sorting.PostingCategorizer
```

Categorizer for fields without column values. Builds an array caching the
order of all documents. Used as a fallback; prefer setting
`sortable=True` on fields.

### `QueryFacet`

```python
class whoosh.sorting.QueryFacet(
    querydict: dict,
    other=None,
    allow_overlap=False,
    maptype=None
)
```

Sorts/facets based on the results of a series of queries.

**Constructor:**
- `querydict`: Dictionary mapping keys to `Query` objects.
- `other`: Key to use for documents matching no queries.

### `RangeFacet`

```python
class whoosh.sorting.RangeFacet(
    fieldname,
    start,
    end,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on numeric ranges. Ranges are inclusive at the start and
exclusive at the end.

```python
prices = RangeFacet("price", 0, 1000, 100)
results = searcher.search(myquery, groupedby=prices)
```

- `fieldname`: The numeric field to facet on.
- `start`: Start of the entire range.
- `end`: End of the entire range.
- `gap`: Size of each bucket (can be a sequence for progressive gaps).
- `hardend`: If `True`, clamp the last bucket to `end`.

### `DateRangeFacet`

```python
class whoosh.sorting.DateRangeFacet(
    fieldname,
    startdate,
    enddate,
    gap,
    hardend=False,
    maptype=None
)
```

Sorts/facets based on date ranges. Extends `RangeFacet` but uses
`datetime` objects for start/end and `timedelta`/`relativedelta` for gaps.
Generates `DateRange` queries instead of `TermRange` queries.

```python
from datetime import datetime
from whoosh.support.relativedelta import relativedelta

startdate = datetime(1920, 1, 1)
enddate = datetime.now()
gap = relativedelta(years=5)
bdays = DateRangeFacet("birthday", startdate, enddate, gap)
```

### `ScoreFacet`

```python
class whoosh.sorting.ScoreFacet
```

Uses a document's relevance score as a sorting criterion.

```python
tag_score = MultiFacet(["tag", ScoreFacet()])
results = searcher.search(myquery, sortedby=tag_score)
```

### `FunctionFacet`

```python
class whoosh.sorting.FunctionFacet(fn)
```

Lets you pass an arbitrary function that computes the sort key. The function
is called with `(searcher, docid)` where `docid` is an absolute index
document number.

```python
fn = lambda s, docid: s.doc_field_length(docid, "content")
lengths = FunctionFacet(fn)
```

### `TranslateFacet`

```python
class whoosh.sorting.TranslateFacet(fn, *facets)
```

Applies a custom function to the key generated by one or more wrapped facets.
Useful for custom collation, such as Unicode Collation Algorithm (UCA) sorting.

```python
from pyuca import Collator

c = Collator("allkeys.txt")
facet = FieldFacet("name")
facet = TranslateFacet(c.sort_key, facet)
results = searcher.search(myquery, sortedby=facet)
```

**Constructor:**
- `fn`: Function applied to the computed key values.
- `*facets`: One or more `FacetType` objects whose keys are passed to `fn`.

### `StoredFieldFacet`

```python
class whoosh.sorting.StoredFieldFacet(
    fieldname,
    allow_overlap=False,
    split_fn=None,
    maptype=None
)
```

Sorts/groups using the value in an unindexed, stored field (e.g., `STORED`).
Usually slower than using an indexed field.

**Constructor:**
- `fieldname`: Name of the stored field.
- `allow_overlap`: If `True`, when grouping, allow documents to appear in
  multiple groups when they have multiple values (split by `split_fn` or
  `string.split()`).
- `split_fn`: Custom function to split a stored field value into multiple
  facet values (only used when `allow_overlap=True`).

### `MultiFacet`

```python
class whoosh.sorting.MultiFacet(items=None, maptype=None)
```

Sorts/facets by the combination of multiple sub-facets.

```python
facet = MultiFacet([FieldFacet("tag"), FieldFacet("path")])
results = searcher.search(myquery, sortedby=facet)
```

Strings in the items list are treated as field names:

```python
facet = MultiFacet(["tag", "path"])
```

**Methods:**
- `from_sortedby(sortedby)`: Class method that creates a `MultiFacet` from
  a field name, facet, or list thereof.
- `add_field(fieldname, reverse=False)`: Add a `FieldFacet`.
- `add_query(querydict, other=None, allow_overlap=False)`: Add a `QueryFacet`.
- `add_score()`: Add a `ScoreFacet`.
- `add_facet(facet)`: Add an arbitrary `FacetType`.

### `Facets`

```python
class whoosh.sorting.Facets(x=None)
```

Maps facet names to `FacetType` objects for creating multiple independent
groupings of documents.

```python
facets = Facets()
facets.add_field("tag")
facets.add_facet("price", RangeFacet("price", 0, 1000, 100))
results = searcher.search(myquery, groupedby=facets)

tag_groups = results.groups("tag")
price_groups = results.groups("price")
```

**Class Methods:**
- `from_groupedby(groupedby)`: Creates a `Facets` object from a field name,
  `FacetType`, dict, list, or another `Facets` object.

**Methods:**
- `names()`: Returns an iterator of facet names.
- `items()`: Returns a list of `(name, facet)` tuples.
- `add_field(fieldname, **kwargs)`: Adds a `FieldFacet`.
- `add_query(name, querydict, **kwargs)`: Adds a `QueryFacet`.
- `add_facet(name, facet)`: Adds a `FacetType` under the given name.
- `add_facets(facets, replace=True)`: Adds the contents of a `Facets` or
  `dict` to this object.

## Facet Maps

### `FacetMap`

```python
class whoosh.sorting.FacetMap
```

Base class for objects holding the results of grouping search results by a
facet. Use `as_dict()` to access results.

```python
myfacet = FieldFacet("size", maptype=OrderedList)
myfacet = FieldFacet("size", maptype=Count)
```

**Methods:**
- `add(groupname, docid, sortkey)`: Adds a document to the facet results.
- `as_dict()`: Returns a dictionary mapping group names to values.

### `OrderedList`

```python
class whoosh.sorting.OrderedList
```

Stores a list of document numbers for each group, in sorted order.

### `UnorderedList`

```python
class whoosh.sorting.UnorderedList
```

Stores a list of document numbers for each group in arbitrary order. Slightly
faster and more memory-efficient than `OrderedList` when ordering doesn't
matter.

### `Count`

```python
class whoosh.sorting.Count
```

Stores the count of documents in each group.

### `Best`

```python
class whoosh.sorting.Best
```

Stores the "best" (highest sort key) document in each group.

## Sorting Utilities

### `add_sortable`

```python
whoosh.sorting.add_sortable(
    writer,
    fieldname,
    facet,
    column=None
)
```

Adds a per-document value column to an existing field, making it sortable.
Useful for retrofitting fields that were created without `sortable=True`.

**Example:**
```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    facet = sorting.FieldFacet("price")
    sorting.add_sortable(w, "price", facet)
```

**Parameters:**
- `writer`: An `IndexWriter` object.
- `fieldname`: Name of the field to add sortable values to.
- `facet`: A `FacetType` object to generate per-document values.
- `column`: Optional `ColumnType` to store the values. If omitted, uses the
  field's default column type.


## DOCUMENT (FR): Spelling

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Spelling API

Functions and classes for correcting typos in user queries using edit-distance
(Damerau-Levenshtein) matching against the terms in the index.

## Corrector Objects

### `Corrector`

```python
class whoosh.spelling.Corrector
```

Base class for spelling correction objects. Concrete subclasses implement the
`_suggestions()` method.

**Methods:**

#### `suggest(text, limit=5, maxdist=2, prefix=0)`

Returns a list of suggested corrections for `text`, ranked by edit distance
then by frequency.

- `text`: The text to check. Will **not** be added to suggestions even if it
  appears in the index.
- `limit`: Maximum number of suggestions to return.
- `maxdist`: Maximum edit distance to look at (values > 2 are inefficient).
- `prefix`: Require suggestions to share this length of prefix with `text`.
  Increasing to even `1` dramatically speeds up suggestions.

#### `_suggestions(text, maxdist, prefix)`

Low-level method yielding `(score, suggestion)` tuples. Subclasses must
implement this.

### `ReaderCorrector`

```python
class whoosh.spelling.ReaderCorrector(reader, fieldname, fieldobj)
```

Suggests corrections based on terms in a specific field of an `IndexReader`.

**Ranks suggestions by edit distance, then by highest to lowest frequency.**

**Constructor:**
- `reader`: An `IndexReader` object.
- `fieldname`: The name of the field to get suggestions from.
- `fieldobj`: The `FieldType` for the field.

### `ListCorrector`

```python
class whoosh.spelling.ListCorrector(wordlist)
```

Suggests corrections based on a sorted list of strings.

**Constructor:**
- `wordlist`: A sorted list of words to match against.

### `MultiCorrector`

```python
class whoosh.spelling.MultiCorrector(correctors, op)
```

Merges suggestions from a list of sub-correctors.

**Constructor:**
- `correctors`: List of `Corrector` objects.
- `op`: A function (e.g., `max` or `operator.add`) to combine scores from
  multiple correctors for the same suggestion.

## Query Correction

### `Correction`

```python
class whoosh.spelling.Correction(q, qstring, corr_q, tokens)
```

Represents the corrected version of a user query string.

**Attributes:**
- `query`: The corrected `Query` object.
- `string`: The corrected user query string.
- `original_query`: The original `Query` object.
- `original_string`: The original user query string.
- `tokens`: List of token objects representing corrected words.

**Methods:**

#### `format_string(formatter)`

Highlights corrected words in the original query string using the given
`Formatter`.

```python
from whoosh import highlight

correction = searcher.correct_query(q, qstring)
hf = highlight.HtmlFormatter(classname="change")
html = correction.format_string(hf)
```

- `formatter`: A `Formatter` instance (or class, which will be instantiated).
- Returns: Formatted string, typically with corrections emphasized.

### `QueryCorrector`

```python
class whoosh.spelling.QueryCorrector(fieldname)
```

Base class for objects that correct words in a user query.

**Constructor:**
- `fieldname`: The default field name for corrections.

**Methods:**

#### `correct_query(q, qstring)`

Returns a `Correction` object representing the corrected form of the given
query.

- `q`: The original `Query` tree to be corrected.
- `qstring`: The original user query string (may be `None`).
- Returns: A `Correction` object.

#### `field()`

Returns the field name this corrector operates on.

### `SimpleQueryCorrector`

```python
class whoosh.spelling.SimpleQueryCorrector(
    correctors: dict,
    terms: list,
    aliases=None,
    prefix: int = 0,
    maxdist: int = 2
)
```

A simple query corrector based on a mapping of field names to `Corrector`
objects, and a list of `(fieldname, text)` tuples to correct.

**Constructor:**
- `correctors`: Dictionary mapping field names to `Corrector` objects.
- `terms`: Sequence of `(fieldname, text)` tuples representing terms to be
  corrected.
- `aliases`: Dictionary mapping field names in the query to field names for
  spelling suggestions.
- `prefix`: Suggested replacement words must share this number of initial
  characters. Default `0`.
- `maxdist`: Maximum edit distance for suggestions. Values > 2 may be slow.


## DOCUMENT (FR): Writing

# API Écriture

Écrire, mettre à jour et supprimer des documents via l'interface `IndexWriter`.

## IndexWriter

```python
class whoosh.writing.IndexWriter
```

Classe de base pour l'écriture de documents.

### Contexte manager

```python
with ix.writer() as writer:
    writer.add_document(title="Bonjour", content="Monde")
    # commit() appelé automatiquement
```

### Méthodes

#### `add_document(**fields)`

Ajoute un document.

**Kwargs spéciaux:**
- `_stored_<nom_champ>`: Valeur stockée alternative
- `_<nom_champ>_boost`: Boost spécifique au champ
- `_boost`: Boost global du document

---

#### `update_document(**fields)`

Met à jour/remplace un document. Utilise les champs `unique` pour trouver les documents existants.

---

#### `delete_document(docnum, delete=True)`

Supprime par numéro de document.

---

#### `delete_by_term(fieldname, text) -> int`

Supprime tous les documents avec le terme dans le champ.

**Retourne:**
- `int`: Nombre de documents supprimés.

---

#### `delete_by_query(q, searcher=None) -> int`

Supprime les documents correspondant à la requête.

---

#### `commit(mergetype=None, optimize=False, merge=True)`

Commit les changements sur disque.

**Args:**
- `mergetype`: Fonction de fusion personnalisée
- `optimize`: Fusionner tous les segments en un seul
- `merge`: Si False, ne pas fusionner les segments existants

---

#### `cancel()`

Annule les changements et libère le verrou.

---

#### `add_field(fieldname, fieldtype, **kwargs)`

Ajoute un champ (avant d'ajouter des documents).

---

#### `remove_field(fieldname)`

Supprime un champ du schéma.

---

#### `searcher(**kwargs) -> Searcher`

Retourne un searcher (pour lecture pendant la session d'écriture).

---

#### `reader(**kwargs) -> IndexReader`

Retourne un reader pour l'état actuel.

---

#### `group()`

Context manager pour grouper des documents dans un segment.

## SegmentWriter

Implémentation concrète d'IndexWriter.

## AsyncWriter

Writer threaded qui réessaie automatiquement en cas de contention.

```python
from whoosh.writing import AsyncWriter

writer = AsyncWriter(index, delay=0.25, writerargs={})
```

## BufferedWriter

Buffer les documents en mémoire et commit périodiquement.

```python
from whoosh.writing import BufferedWriter

writer = BufferedWriter(
    index,
    period=60,      # Max secondes entre commits
    limit=100       # Max documents par commit
)
```

## Politiques de fusion

```python
from whoosh.writing import NO_MERGE, MERGE_SMALL, OPTIMIZE, CLEAR

writer.commit(mergetype=NO_MERGE)
writer.commit(mergetype=MERGE_SMALL)
writer.commit(mergetype=OPTIMIZE)
writer.commit(mergetype=CLEAR)
```

## Exceptions

### IndexingError

```python
class whoosh.writing.IndexingError(Exception)
```

Levée quand une opération d'indexation échoue.


## DOCUMENT (FR): Auto Indexing

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Auto-Indexing

Whoosh-NG provides utilities for automatic schema discovery and data-source driven indexing.

## Schema Discovery

The `SchemaDiscovery` utility inspects a data source and auto-generates a Whoosh schema:

```python
from whoosh_modern.discovery import SchemaDiscovery

discovery = SchemaDiscovery(source=data_source)
schema = discovery.discover()
```

See [SearchView](/examples/search-view) and [Data Sources](/examples/data-sources) for usage examples.


## DOCUMENT (FR): Backends

# Backends

Whoosh-NG supporte des backends de stockage pluggables via l'architecture Provider. Le backend par défaut stocke les données comme fichiers sur disque, mais vous pouvez utiliser SQLite, PostgreSQL, S3, et plus encore.

## Backends intégrés

| Backend | Description |
|---------|-------------|
| Fichier (défaut) | Stocke l'index comme fichiers sur disque |
| SQLite | Stocke l'index dans une base SQLite |
| Mémoire | Backend en mémoire (tests uniquement) |

## Backend Fichier (défaut)

```python
from whoosh.index import create_in

# Utilise FileBackend par défaut
ix = create_in("indexdir", schema)
```

## Backend SQLite

```python
from whoosh.backends.sqlite import SQLiteBackend
from whoosh.store.sqlite import SQLiteStorage

storage = SQLiteStorage("index.db")
backend = SQLiteBackend(storage=storage)
```

### Avantages

- Index en un seul fichier
- Meilleur pour les charges transactionnelles
- Sauvegardes plus faciles
- Supporte les lectures concurrentes

## Backend Mémoire

```python
from whoosh.backends.memory import MemoryBackend

backend = MemoryBackend()  # Utile pour les tests
```

## Bonnes pratiques

1. **File backend pour production**: Le plus éprouvé
2. **SQLite pour déploiement mono-fichier**: Plus facile à déployer
3. **Mémoire pour les tests**: Rapide, pas de nettoyage nécessaire
4. **Fichiers composés**: Activez pour réduire le nombre de fichiers
5. **Stratégie de sauvegarde**: File = copier le répertoire; SQLite = copier le fichier


## DOCUMENT (FR): Core Concepts

# Concepts fondamentaux

Whoosh-NG est une bibliothèque de recherche purement Python. Ce guide explique les principaux concepts pour l'utiliser efficacement.

## Architecture

Whoosh-NG suit une architecture en couches :

```text
Application
    ▼
┌─────────────────────────────┐
│       Whoosh-NG Core        │
├─────────────────────────────┤
│ Schema                      │
│ Search Engine               │
│ Plugin Manager              │
│ Registry System             │
│ Middleware Pipeline         │
│ Event Bus                   │
│ Hook System                 │
└─────────────────────────────┘
       ▼
┌─────────────────────────────┐
│           Plugins           │
├─────────────────────────────┤
│ FastAPI                     │
│ Autocomplete                │
│ Vector Search               │
│ PostgreSQL                  │
│ S3                          │
│ Monitoring                  │
│ Admin UI                    │
└─────────────────────────────┘
```

## Composants clés

### Index

Un `Index` est le conteneur de vos documents. Il gère un ou plusieurs segments sur disque.

```python
from whoosh.index import create_in, open_dir

ix = create_in("indexdir", schema)
ix = open_dir("indexdir")
```

### Schema

Le `Schema` définit les champs des documents. Chaque champ a un type qui détermine son indexation et stockage.

```python
from whoosh.fields import Schema, TEXT, ID, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    content=TEXT,
    rating=NUMERIC(float, stored=True)
)
```

### Writer

Un `IndexWriter` permet d'ajouter, modifier et supprimer des documents.

```python
writer = ix.writer()
writer.add_document(title="Bonjour", content="Monde")
writer.commit()
```

### Searcher

Un `Searcher` interroge l'index et retourne des résultats.

```python
with ix.searcher() as s:
    results = s.search("bonjour")
```

### QueryParser

Convertit une chaîne de requête en objet Query.

```python
from whoosh.qparser import QueryParser

qp = QueryParser("content", schema)
query = qp.parse("bonjour monde")
```

## Fonctionnalités modernes

### Système de plugins

Les plugins étendent Whoosh-NG sans modifier le core. Ils peuvent :

- Enregistrer de nouveaux providers vectoriels
- Ajouter des endpoints FastAPI
- Fournir des analyseurs personnalisés
- S'intégrer au pipeline de middleware

```python
from whoosh.plugins.manager import PluginManager

# Auto-découverte depuis les entry points
PluginManager.load_plugins()
```

### Pipeline de middleware

Le middleware intercepte les opérations d'indexation et de recherche :

```python
from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext):
        print(f"Recherche: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext):
        print(f"Trouvé: {len(context.results) if context.results else 0} résultats")
        return context
```

### Recherche vectorielle

Permet la recherche sémantique via des embeddings :

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    content=TEXT,
    embedding=VectorField(dimensions=384)
)
```

### Event Bus

Système d'événements pour un couplage lâche :

```python
from whoosh.event_bus import EventBus, DocumentIndexed

bus = EventBus()

@bus.subscribe
def on_document_indexed(event: DocumentIndexed):
    print(f"Document indexé: {event.docnum}")
```

## Principes de conception

1. **Composabilité**: Les composants se combinent via les opérateurs `|` et `+`
2. **Abstractions sans coût**: Pas de middleware = pas de surcoût
3. **Sync-first**: Le core est synchrone; async est optionnel
4. **Isolation des plugins**: Les plugins ne peuvent pas casser le core
5. **Sécurité des types**: Typage complet avec annotations


## DOCUMENT (FR): Dates

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Dates and Numeric Ranges

This guide covers working with `DATETIME` and `NUMERIC` fields, including
range queries, range faceting, and date math.

## DATETIME Fields

`DATETIME` fields store Python `datetime` objects and can be queried with
range queries.

```python
from datetime import datetime
from whoosh import fields, index

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    published_date=fields.DATETIME(stored=True, sortable=True),
)
```

### Indexing Dates

```python
ix = index.create_in("indexdir", schema)
with ix.writer() as w:
    w.add_document(
        title="Article 1",
        published_date=datetime(2024, 6, 15, 14, 30),
    )
```

### Date Range Queries

Use `Range` or `QueryParser` syntax:

```python
from whoosh.qparser import QueryParser
from whoosh.query import Range, Every

# Using QueryParser syntax
qp = QueryParser("published_date", schema=ix.schema)
q = qp.parse("[2024-01-01 TO 2024-12-31]")

# Using Range query directly
from datetime import datetime
q = Range(
    "published_date",
    datetime(2024, 1, 1),
    datetime(2024, 12, 31),
)

with ix.searcher() as searcher:
    results = searcher.search(q)
```

### Sorting by Date

```python
from whoosh.sorting import FieldFacet

# Sort by date, most recent first
results = searcher.search(
    query,
    sortedby=FieldFacet("published_date", reverse=True),
)
```

## NUMERIC Fields

`NUMERIC` fields store integers and floating-point numbers.

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, stored=True, sortable=True),
    rating=fields.NUMERIC(float, stored=True),
)
```

### Numeric Range Queries

```python
from whoosh.query import NumericRange

q = NumericRange("price", 100, 500)

# Or with QueryParser
qp = QueryParser("price", schema=ix.schema)
q = qp.parse("[100 TO 500]")
```

### Numeric Faceting

Group results into numeric ranges using `RangeFacet`:

```python
from whoosh.sorting import RangeFacet

price_ranges = RangeFacet("price", 0, 1000, 100)
results = searcher.search(query, groupedby=price_ranges)

for groupname, docnums in results.groups("price").items():
    print(f"Price ${groupname}: {len(docnums)} results")
```

## Date Faceting

Group results by date intervals using `DateRangeFacet`:

```python
from datetime import datetime
from whoosh.sorting import DateRangeFacet

start = datetime(2020, 1, 1)
end = datetime(2026, 1, 1)
date_facet = DateRangeFacet(
    "published_date",
    start,
    end,
    relativedelta(years=1),  # Requires: from dateutil.relativedelta import relativedelta
)
results = searcher.search(query, groupedby=date_facet)

for year_range, docnums in results.groups("published_date").items():
    print(f"Year {year_range}: {len(docnums)} results")
```

## Sorting and Filtering by Numbers

### Sorting

```python
from whoosh.sorting import FieldFacet

# Sort by price ascending
results = searcher.search(query, sortedby=FieldFacet("price"))
```

### Filtering

```python
from whoosh.query import NumericRange

# Only results with price >= 50 and price < 200
filter_q = NumericRange("price", 50, 200)
results = searcher.search(query, filter=filter_q)
```

## Making Date/Numeric Fields Sortable

When defining a schema, set `sortable=True` on `NUMERIC` or `DATETIME` fields
to enable sorting by that field:

```python
schema = fields.Schema(
    title=fields.TEXT(stored=True),
    price=fields.NUMERIC(int, sortable=True),
    date=fields.DATETIME(sortable=True),
)
```

If you forgot to set `sortable=True`, you can add it after indexing:

```python
from whoosh import index, sorting

ix = index.open_dir("indexdir")
with ix.writer() as w:
    sorting.add_sortable(w, "price", sorting.FieldFacet("price"))
```


## DOCUMENT (FR): Glossary

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Glossary

A glossary of key terms used in Whoosh.

## Analysis

The process of converting text into tokens (individual units like words or
terms) for indexing. Involves tokenization, normalization (lowercasing,
stemming), and filtering (stop word removal, etc.).

## Analyzer

A chain of `Tokenizer` and `Filter` objects that processes text into
tokens. Examples include `RegexTokenizer`, `NgramTokenizer`, `LowercaseFilter`,
`StopFilter`, and `StemmerFilter`.

## Compound File

A file format that combines multiple index segment files into a single
`.seg` file. This can improve performance on some filesystems by reducing
file handle usage. Configured via the codec's `should_assemble` setting.

## Document

A single record in the index, similar to a row in a database. A document
contains fields (analogous to columns).

## Field

A named attribute of a document. Fields have a type (defined by `FieldType`)
that determines how the field's value is indexed and stored.

## Field Type

The class (e.g., `TEXT`, `ID`, `NUMERIC`, `DATETIME`, `BOOLEAN`) that
defines how a field's value is tokenized, stored, indexed, and made
sortable/facetable.

## Filter

An `Analyzer` component that processes, transforms, or filters tokens
after tokenization. Examples: `LowercaseFilter`, `StopFilter`,
`StemmerFilter`.

## Format

A `Format` object controls how posting information (term frequency, positions,
character offsets) is encoded for each field in the inverted index.
Examples: `Existence`, `Frequency`, `Positions`, `Characters`.

## Fragmentation

The process of selecting text spans around matched terms for highlighting.

## Highlighter

The `whoosh.highlight` module, which provides formatters, fragmenters, and
scorers for highlighting search terms in documents.

## Index

The collection of segment files that store the inverted index, document
data, and metadata (the table of contents, or TOC).

## IndexWriter

The `IndexWriter` class is used to create and modify the index. It buffers
document additions and deletions and commits them to disk.

## Inverted Index

The core data structure of a search engine: for each unique term, it stores a
list of documents (and positions) where that term appears.

## Matcher

An object that iterates over matching documents in the postings list for a
query. Matchers can be combined (union, intersection, etc.) for compound
queries.

## Posting

A single entry in the inverted index: a (document ID, term frequency, value)
tuple for a given term.

## Schema

Defines the fields, their types, and indexing options for an index. A schema
is passed to `Storage.create_index()`.

## Scorer

An object that computes a relevance score for a document given a query and
term weights. Different weighting models (BM25, TF-IDF, etc.) use different
scorers.

## Segment

A self-contained portion of the inverted index. An index may consist of
multiple segments. Segments are merged periodically (during optimize or
merge operations) to improve performance.

## Sort Key

A value computed per-document (via a `FacetType` and its `Categorizer`)
used to order results during sorting and faceting.

## Stemming

The process of reducing words to their root form (e.g., "running" → "run",
"cats" → "cat") to improve recall by matching inflected forms.

## Stop Words

High-frequency, low-information words (e.g., "the", "a", "and") that are
typically filtered out during indexing.

## Term

A unique (field name, token text) pair in the inverted index.

## Term Vector

Optional per-document data structure storing the terms (and optionally
positions and character offsets) that appear in a document's field, enabling
features like highlighting and pseudo-relevance feedback.

## Tokenizer

An `Analyzer` component that splits input text into tokens. Examples:
`RegexTokenizer`, `PathTokenizer`, `NgramTokenizer`.

## Whoosh Query

Whoosh's own query syntax, parsed by `QueryParser`. Supports fielded
search, phrase queries, wildcards, ranges, and more.


## DOCUMENT (FR): Indexing

# Indexation

Guide pour ajouter, mettre à jour et supprimer des documents.

## Ouvrir un writer

```python
from whoosh import index

ix = index.open_dir("indexdir")

# Writer basique
writer = ix.writer()

# Writer avec options
writer = ix.writer(
    timeout=10.0,
    delay=0.1,
    limitmb=128,
    compound=True
)
```

## Ajouter des documents

```python
with ix.writer() as writer:
    writer.add_document(
        title="Premier document",
        content="Bonjour le monde",
        path="/doc1",
        tags=["python", "recherche"]
    )
    writer.commit()
```

## Mettre à jour

```python
with ix.writer() as writer:
    writer.update_document(
        path="/doc1",
        content="Contenu mis à jour"
    )
```

## Supprimer

```python
# Par numéro de document
writer.delete_document(docnum=42)

# Par terme
writer.delete_by_term("path", "/doc1")

# Par requête
from whoosh.query import Term
q = Term("tags", "deprecated")
writer.delete_by_query(q)

writer.commit()
```

## Bonnes pratiques

- Utilisez `with ix.writer() as writer:` pour le nettoyage automatique
- Commutez par lots pour de meilleures performances
- Utilisez `BufferedWriter` en environnement multi-processus
- Libérez toujours le verrou avec `commit()` ou `cancel()`


## DOCUMENT (FR): Installation

# Installation

## Prérequis

- Python 3.10+
- Aucune dépendance obligatoire (pur Python)
- Extras optionnels pour les fonctionnalités avancées

## pip install

```bash
pip install whoosh-ng
```

## Extras

| Extra | Description |
|-------|-------------|
| `vector` | Providers de recherche vectorielle (NumPy, HNSW, Faiss) |
| `autocomplete` | Plugin d'autocomplétion |
| `api` | Plugin FastAPI |
| `metrics` | Intégration Prometheus |
| `all` | Installer tout |

```bash
pip install whoosh-ng[all]
```

## Installation pour le développement

```bash
git clone https://github.com/your-org/whoosh-NG.git
cd whoosh-NG
uv sync --extra dev
```

## Vérification

```bash
uv run pytest tests/ -q
uv run ruff check src/ tests/
uv run ruff format --check .
uv run mypy src/whoosh
```

## Prochaines étapes

- [Démarrage rapide](/core/quickstart)
- [Concepts fondamentaux](/core/core-concepts)


## DOCUMENT (FR): Legacy Cleanup

# Stratégie de nettoyage du code legacy

Ce guide explique comment Whoosh-NG sépare le code moderne typé du code legacy,
et comment le nettoyage progressif est organisé.

## Pourquoi une frontière legacy ?

`whoosh-modern` est la nouvelle surface de Whoosh-NG, entièrement typée.
Le package `whoosh` original fonctionne toujours au runtime, mais il contient
des décennies de motifs de compatibilité Python 2/3, de métaprogrammation
dynamique et d'internes non typés. Forcer des types stricts sur l'ensemble
d'un coup bloquerait le développement.

La stratégie de nettoyage est **incrémentale et opt-in** :

1. `src/whoosh_modern/` est typé et vérifié avec `pyright` et `mypy` en mode strict.
2. `src/whoosh/` est la surface legacy. Elle est divisée en :
   - **modules exclus** (documentés dans `pyrightconfig.json`) — code trop
     dynamique ou vendu pour justifier un passage de types rentable maintenant ;
   - **candidats au nettoyage** — petits fichiers isolés, faciles à annoter et
     à vérifier.
3. Chaque sprint, une vague de candidats est typée, testée, puis sortie de la
   zone de tolérance élevée.

## Seuils actuels (Sprint 2)

| Vérificateur | Portée | Seuil |
|--------------|--------|-------|
| `pyright` | `src/whoosh_modern/` | **0 erreur** (strict) |
| `pyright` | legacy | **≤ 500 erreurs** (tolérant) |
| `mypy` | `src/` | **0 erreur** (via overrides + `ignore_errors`) |

## Justification des exclusions (`pyrightconfig.json`)

La liste `exclude` de `pyrightconfig.json` regroupe les fichiers par thème :

- **Vendu / sans stubs** : `pyparsing.py`, `relativedelta.py`
- **Shims de migration** : `codec/whoosh2.py`, `codec/whoosh3.py`
- **Parsing dynamique** : `qparser/`, `query/`, `analysis/`, `automata/`
- **Stockage fichiers** : `filedb/`, `reading/`, `writing/`
- **Heuristique / data-driven** : `lang/dmetaphone.py`, `lang/lovins.py`,
  `lang/phonetic.py`, `lang/wordnet.py`
- **Objets dynamiques** : `classify.py`, `index.py`, `locking.py`,
  `formats.py`, `middleware/`
- **Bas niveau vendu** : `support/bench.py`, `support/base85.py`,
  `support/bitstream.py`, `support/bitvector.py`, `support/charset.py`,
  `support/levenshtein.py`

## Plan Sprint 2

Pour le Sprint 2, l'accent est mis sur les petits modules utilitaires et de
support, sans dépendances externes ni métaprogrammation lourde.

Vague de candidats :

- `src/whoosh/util/varints.py`
- `src/whoosh/util/text.py`
- `src/whoosh/util/loading.py`
- `src/whoosh/support/bitstream.py`
- `src/whoosh/support/levenshtein.py`

Pour chaque fichier :

1. Supprimer le `# type: ignore` global (si présent).
2. Ajouter des signatures de fonctions précises.
3. Lancer `pyright` et `mypy` pour confirmer **0 nouvelle erreur**.
4. Retirer le fichier des exclusions de `pyrightconfig.json`.
5. Ajouter un test de régression dans `tests/test_legacy_cleanup.py`.

## Objectif long terme

Chaque fichier de `src/whoosh/` doit finir par être vérifiable par `mypy` et
`pyright` sans exclusion globale. D'ici là, la liste d'exclusion est le
registre explicite de la dette, et chaque sprint la réduit.


## DOCUMENT (FR): Migration

# Guide de migration

Ce guide vous aide à migrer depuis Whoosh legacy ou Whoosh-Reloaded 3.x vers Whoosh-NG v3.0.0.
> **Prochaine version** : v4.0.0.dev0 (en développement) ajoute `SchemaBuilder`, la hiérarchie d'exceptions middleware, et plus encore — voir le [CHANGELOG](https://github.com/dorel14/whoosh-ng/blob/master/CHANGELOG.md).

## Depuis Whoosh 1.x/2.x (Legacy)

### Chemins d'import

| Legacy | Whoosh-NG |
|--------|-----------|
| `import whoosh` | `import whoosh` |
| `from whoosh.index import create_in` | `from whoosh.index import create_in` |
| `from whoosh.fields import Schema, TEXT` | `from whoosh.fields import Schema, TEXT` |

L'API core est intentionnellement stable. La plupart du code existant fonctionne sans modification.

## Depuis Whoosh-Reloaded 3.x

Aucun changement cassant. Whoosh-NG est une continuation de Whoosh-Reloaded.

### Migration optionnelle des plugins

```python
# Ancien
from whoosh_modern.vector.numpy_provider import NumpyProvider

# Nouveau (via registre)
from whoosh.vector import NumpyProvider
from whoosh.registry import VectorRegistry

VectorRegistry.register("numpy", NumpyProvider(), "mon_app")
```

### SchemaBuilder (nouveau dans v4.0.0.dev0)

```python
# Ancien
schema = Schema(title=TEXT(stored=True), content=TEXT)

# Nouveau (API fluent)
from whoosh.fields import SchemaBuilder

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("content", TEXT)
    .build()
)
```

## Méthode de migration middleware (nouveau dans v4.0.0.dev0)

```python
from whoosh.middleware import Middleware, MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext):
        print(f"Query: {context.query}")
        return context

# Envelopper le writer/searcher existant
writer = apply_middleware_to_writer(ix.writer(), [LoggingMiddleware()])
```

## Liste de vérification

1. **Mettre à jour les dépendances**:
   ```bash
   pip install --upgrade whoosh-ng
   ```

2. **Exécuter les tests**:
   ```bash
   uv run pytest tests/ -q
   ```

3. **Mettre à jour les dépendances optionnelles** (si plugins utilisés):
   ```bash
   pip install whoosh-ng[all]
   ```

4. **Revoir le middleware**: Envisagez d'ajouter du middleware pour les préoccupations transverses

## Dépréciations

| Fonctionnalité | Statut | Remplacement |
|----------------|--------|--------------|
| `whoosh_modern.vector` | Déprécié | `whoosh.vector` |
| `whoosh.store` brut | Déprécié | `whoosh.backends` |
| Utilisation directe de `SegmentWriter` | Découragé | Utilisez `IndexWriter` |

## Compatibilité

Whoosh-NG maintient la compatibilité ascendante. Si vous trouvez un changement cassant, signalez-le comme une issue.


## DOCUMENT (FR): Nested

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Nested Documents

This guide covers indexing and searching hierarchical/nested document
structures (e.g., a parent document with multiple child documents) using
Whoosh's parent-child relationship features.

## Defining Nested Documents

You can index parent documents that contain child documents by using a
parent field and child fields:

```python
from whoosh import fields, index

schema = fields.Schema(
    type=fields.ID(sortable=True),
    title=fields.TEXT(stored=True),
    content=fields.TEXT,
    section_name=fields.ID,
    section_content=fields.TEXT,
)
```

The `type` field distinguishes parent documents from child documents.

## Indexing Nested Documents

Use `IndexWriter.add_all()` with a generator that yields parent and child
documents grouped together:

```python
writer = ix.writer()
writer.add_all([
    parent_doc,
    child_doc_1,
    child_doc_2,
    parent_doc_2,
    child_doc_3,
])
```

Parent documents have `type="parent"` and child documents have
`type="child"`.

## Searching Nested Documents

### Parent-Query Child-Search

Search within child documents and match their parents:

```python
from whoosh.query import Every, Term
from whoosh.sorting import NestedParent

# Match all parent documents
parents = NestedParent(Term("type", "parent"))
q = Every("section_content", "hello")
results = searcher.search(q, sortedby=parents)
```

### Child-Query Parent-Search

Search for parent documents whose children match:

```python
from whoosh.sorting import NestedChildren

# Match parent documents that have children matching the query
parent_results = searcher.search(child_query, groupedby=NestedChildren(parent_matcher, child_matcher))
```

## Parent-Child Relationships at Index Time

When writing documents, use the `parent` parameter to link children to
parents:

```python
writer.add_document(type="parent", title="Chapter 1", _key="chapter1")
writer.add_document(type="child", section_name="Section 1.1",
                    section_content="...", parent="chapter1")
writer.add_document(type="child", section_name="Section 1.2",
                    section_content="...", parent="chapter1")
```

## Accessing Nested Results

To retrieve child matches alongside parent results, use the `expand` method
on the results:

```python
results = searcher.search(parent_query)
expanded = results.expand_child("section")
```

## Nested Faceting

Combine parent-child relationships with faceting using `NestedParent` and
`NestedChildren` as facets:

```python
parent_facet = NestedParent(FieldFacet("type"))
results = searcher.search(query, groupedby=parent_facet)
```

## Performance Considerations

- Parent-child joins are more expensive than flat document searches
- Use `childperm` searcher option to limit the number of permutations
  examined
- Consider whether hierarchical structure is needed at query time, or
  whether documents can be flattened during indexing


## DOCUMENT (FR): Query

# Langage de requête

Whoosh-NG fournit un langage de requête puissant similaire à Lucene, ainsi qu'une API de requêtes programmatique.

## Syntaxe de requête

### Termes de base

```
bonjour                    # Terme unique
bonjour monde              # Termes multiples (AND par défaut)
bonjour OU monde           # OR explicite
"bonjour monde"            # Phrase
```

### Spécification de champ

```
titre:python               # Recherche dans le champ titre
titre:"Tutoriel Python"    # Phrase dans un champ spécifique
```

### Opérateurs booléens

```
python AND whoosh
python OR whoosh
python AND NOT java
python AND (whoosh OR lucene)
```

### Préfixe et jokers

```
pyth*                     # Requête préfixe
pyth?n                    # Joker caractère unique
```

### Requêtes de plage

```
date:[2020 TO 2025]
prix:[10 TO 50]
rating:[4.0 TO *]         # Plage ouverte
```

### Recherche floue

```
python~2                  # Distance d'édition <= 2
lucene~1                  # Correspondance approximative
```

### Recherche de proximité

```
"bonjour monde"~5         # Dans un rayon de 5 termes
```

### Boost

```
python^2.0 whoosh         # Booster python par 2x
(titre:python)^3 content:python  # Booster les matches dans titre
```

## Classes de requêtes

Construisez des requêtes programmatiquement :

```python
from whoosh.query import *

# Terme simple
q = Term("content", "python")

# AND
q = And([Term("content", "python"), Term("content", "whoosh")])

# OR
q = Or([Term("content", "python"), Term("content", "lucene")])

# Phrase
q = Phrase("content", ["bonjour", "monde"])

# Plage
q = NumericRange("prix", 10, 50)
q = DateRange("date", datetime(2020,1,1), datetime(2025,1,1))

# Préfixe
q = Prefix("content", "pyth")
```

## MultifieldParser

Recherchez plusieurs champs avec des boosts différents :

```python
from whoosh.qparser import MultifieldParser

qp = MultifieldParser(
    ["titre", "content", "tags"],
    schema,
    fieldboosts={"titre": 2.0, "tags": 1.5}
)
q = qp.parse("python recherche")
```

## Échappement des caractères spéciaux

```
titre\:python              # Deux-points littéral
chemin\:\/\/exemple        # Échapper les caractères spéciaux
```


## DOCUMENT (FR): Quickstart

# Démarrage rapide

## Installation

```bash
pip install whoosh-ng
uv pip install whoosh-ng
```

## Exemple basique

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(id=ID(stored=True), content=TEXT())
ix = index.create_in("indexdir", schema)

with ix.writer() as w:
    w.add_document(id="1", content="hello world")
    w.add_document(id="2", content="goodbye world")

with ix.searcher() as s:
    results = s.search("world")
    for hit in results:
        print(hit["id"], hit.score)
```

## Avec plugins

```bash
pip install whoosh-ng[vector,autocomplete,api]
```

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin

PluginManager.load_plugins()
```

## Sources de données

```bash
pip install whoosh-ng
```

```python
import sqlite3
from whoosh_modern.data_sources import SQLSource
from whoosh_modern.views import SearchView

# Utiliser les données de benchmark existantes
conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
)

vue = SearchView(name="reuters", source=source)
ix = vue.build("indexdir")
```


## DOCUMENT (FR): Schema

# Conception de schéma

Le schéma définit la structure des documents dans votre index. Il spécifie les champs existants, leur indexation et leur stockage.

## Types de champs

| Type | Description | Indexé | Stocké |
|------|-------------|--------|--------|
| `TEXT` | Texte libre, tokenisé | Oui | Optionnel |
| `ID` | Identifiant non tokenisé | Oui | Optionnel |
| `KEYWORD` | Mots-clés séparés par espace/virgule | Oui | Optionnel |
| `STORED` | Stocké uniquement, non searchable | Non | Oui |
| `NUMERIC` | Entier ou flottant | Oui | Optionnel |
| `DATETIME` | Dates et heures | Oui | Optionnel |
| `BOOLEAN` | Booléen | Oui | Optionnel |
| `NGRAM` | N-grammes de caractères | Oui | Optionnel |
| `NGRAMWORDS` | N-grammes de mots | Oui | Optionnel |
| `VectorField` | Vecteur d'embedding | Personnalisé | Optionnel |

## Créer un schéma

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, STORED, NUMERIC

schema = Schema(
    title=TEXT(stored=True),
    path=ID(stored=True, unique=True),
    content=TEXT,
    tags=KEYWORD(lowercase=True),
    published=NUMERIC(int, stored=True),
    is_published=BOOLEAN,
    icon=STORED
)
```

## Options des champs

### TEXT

```python
content = TEXT(
    stored=False,        # Stocker le texte original ?
    unique=False,        # Utiliser pour remplacer des documents ?
    phrase=True,         # Indexer les positions pour recherche de phrases
    analyzer=None,       # Analyseur personnalisé
    field_boost=1.0      # Boost pour le scoring
)
```

### ID

```python
path = ID(
    stored=True,         # Stocker le chemin
    unique=True          # Utiliser pour remplacement de documents
)
```

### KEYWORD

```python
tags = KEYWORD(
    stored=False,
    lowercase=True,      # Minusculiser automatiquement
    commas=True,         # Séparer par virgules
    scorable=True        # Stocker la longueur pour scoring
)
```

## SchemaBuilder

Whoosh-NG v4.0.0.dev0 (en développement) introduit `SchemaBuilder` pour une API fluide :

```python
from whoosh.fields import SchemaBuilder, TEXT, ID, NUMERIC

schema = (
    SchemaBuilder()
    .field("title", TEXT(stored=True))
    .field("path", ID(stored=True, unique=True))
    .field("content", TEXT)
    .field("rating", NUMERIC(float, stored=True))
    .build()
)
```

## Champs dynamiques

Utilisez des patterns glob pour associer des types :

```python
# Tout champ finissant par "_date" est un DATETIME
schema.add("*_date", DATETIME(stored=True), glob=True)

# Tout champ finissant par "_id" est un ID
schema.add("*_id", ID(stored=True), glob=True)
```

## Modifier le schéma

Ajoutez ou supprimez des champs après création :

```python
writer = ix.writer()

# Ajouter un champ
writer.add_field("description", TEXT(stored=True))

# Supprimer un champ
writer.remove_field("legacy_field")

writer.commit()
```

> Note: Supprimer un champ ne fait que le retirer du schéma. Les données ne sont libérées qu'à l'optimisation.

## Modèles de recherche

Whoosh-NG peut mapper automatiquement des modèles Python (dataclasses, Pydantic, SQLAlchemy, SQLModel, msgspec) vers un `Schema` Whoosh via `ModelIndex`.

### Niveau 1 : Auto-mapping

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Book:
    title: str
    count: int
    tag: str | None = None

idx = ModelIndex(Book)
schema = idx.schema
```

`ModelIndex` inspecte les annotations de type et les mappe vers des champs Whoosh :

| Type Python | Champ Whoosh |
|-------------|--------------|
| `str` | `TEXT` |
| `int` / `float` | `NUMERIC` |
| `bool` | `BOOLEAN` |
| `datetime` / `date` | `DATETIME` |
| `Decimal` | `NUMERIC(int, decimal_places=2)` |
| `Enum` | `KEYWORD` |
| `bytes` | `KEYWORD` (stockage hexadécimal) |
| `list[str]` | `KEYWORD` |
| `Optional[T]` | type mappé ou `STORED` |

Les champs ID sont auto-détectés : `SearchOptions(id=True)` explicite > nom `id`/`ID`/`_id` > premier champ `str`.

### Niveau 2 : Options explicites

Utilisez `SearchField` pour remplacer les valeurs par défaut :

```python
from whoosh_modern.models import SearchField, SearchOptions

class Book:
    title: str = SearchField(fulltext=True, stored=True)
    count: int = SearchField(sortable=True)
    tags: list[str] = SearchField(multi=True)
```

### Niveau 3 : Types annotés

Utilisez `Annotated` pour attacher des métadonnées directement aux annotations :

```python
from typing import Annotated
from whoosh_modern.models import SearchField

class Book:
    title: Annotated[str, SearchField(fulltext=True, stored=True)]
```

### Intégrations

#### Dataclass

```python
from dataclasses import dataclass
from whoosh_modern.models import ModelIndex

@dataclass
class Article:
    title: str
    body: str
    published: datetime.datetime

idx = ModelIndex(Article)
```

#### Pydantic v2

```python
from pydantic import BaseModel
from whoosh_modern.models import register_model

class Article(BaseModel):
    title: str
    body: str
    published: datetime.datetime

    # Métadonnées de recherche par champ via json_schema_extra
    model_config = {"json_schema_extra": {"search": {"fulltext": True}}}

idx = register_model(Article)
```

#### SQLAlchemy

```python
from sqlalchemy import Column, Integer, String, DateTime
from sqlalchemy.orm import DeclarativeBase
from whoosh_modern.models import register_model

class Base(DeclarativeBase):
    pass

class Article(Base):
    __tablename__ = "articles"
    id = Column(Integer, primary_key=True)
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    published = Column(DateTime, info={"search": {"sortable": True}})

idx = register_model(Article)
```

#### SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_model

class Article(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    published: datetime.datetime

idx = register_model(Article)
```

#### msgspec

```python
import msgspec
from whoosh_modern.models import register_model

class Article(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    published: datetime.datetime

idx = register_model(Article)
```

### Conversion d'instances

```python
doc = idx.to_whoosh_document(book_instance)
writer.add_document(**doc)
```

`to_whoosh_document` gère :
- dataclass : itération via `dataclasses.fields()`
- Pydantic/SQLModel : itération via `model_fields`
- SQLAlchemy : itération via `__mapper__.columns`
- Valeurs Enum converties en `.value`
- `bytes` convertis en chaîne hexadécimale

## Bonnes pratiques

1. **Minimal** : N'indexez que ce que vous cherchez
2. **STORED avec parcimonie** : Augmente la taille de l'index
3. **Champs uniques** : Utilisez `unique=True` pour les identifiants
4. **Boost de champ** : Boostez les champs importants au niveau schéma
5. **TEXT options** : Désactivez `phrase` si vous n'avez pas besoin de recherche de phrase
6. **Champ ID** : Laissez `ModelIndex` auto-détecter ou marquez explicitement avec `SearchOptions(id=True)`


## DOCUMENT (FR): Searching

# Recherche

Guide pour exécuter des recherches, travailler avec les résultats, le scoring et le tri.

## Recherche basique

```python
from whoosh.qparser import QueryParser

with ix.searcher() as searcher:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("bonjour monde")
    results = searcher.search(q)
    for hit in results:
        print(hit["title"], hit.score)
```

## Le Searcher

Le `Searcher` est l'interface principale pour lire l'index.

```python
# Toujours utiliser le context manager
with ix.searcher() as searcher:
    results = searcher.search(query)

# Ou gestion manuelle
searcher = ix.searcher()
try:
    results = searcher.search(query)
finally:
    searcher.close()
```

## QueryParser

Convertit une chaîne de requête en objet Query :

```python
from whoosh.qparser import QueryParser, OrGroup

# AND par défaut entre termes
qp = QueryParser("content", schema)
q = qp.parse("bonjour monde")  # content:bonjour AND content:monde

# Changer l'opérateur par défaut
qp = QueryParser("content", schema, group=OrGroup)
q = qp.parse("bonjour monde")  # content:bonjour OR content:monde
```

## Méthodes de recherche

### search()

```python
results = searcher.search(
    query,
    limit=10,           # Max résultats (None pour tout)
    sortedby=None,      # Clé(s) de tri
    reverse=False,      # Tri inversé
    terms=False,        # Collecter les termes matchés
    filter=None,        # Autoriser seulement ces docnums
    mask=None,          # Exclure ces docnums
    collapse=None       # Facette d'effondrement
)
```

### search_page()

```python
# Page 1, 10 résultats par page (défaut)
results = searcher.search_page(query, 1)

# Page 3, 20 résultats par page
results = searcher.search_page(query, 3, pagelen=20)
```

## Résultats

`Results` agit comme une liste de documents matchés :

```python
results = searcher.search(query)

# Support de slice
first_five = results[0:5]

# Longueur (peut déclencher un recompte)
total = len(results)

# Longueur scorée (ce qui est réellement retourné)
scored = results.scored_length()
```

### Objet Hit

```python
for hit in results:
    # Champs stockés
    title = hit["title"]
    path = hit["path"]

    # Score
    print(hit.score)

    # Surbrillance
    highlights = hit.highlights("content", top=3)
```

## Scoring

Le modèle de scoring par défaut est BM25F :

```python
from whoosh import scoring

with ix.searcher(weighting=scoring.BM25F()) as s:
    results = s.search(query)
```

### Scoring personnalisé

```python
class MyScorer(scoring.WeightingModel):
    def scorer(self, searcher, fieldname, text, qf=1):
        return MyCustomScorer(searcher, fieldname, text, qf)

with ix.searcher(weighting=MyScorer()) as s:
    results = s.search(query)
```

## Tri

```python
from whoosh import sorting

# Tri par champ unique
results = searcher.search(query, sortedby="date")

# Tri inversé
results = searcher.search(query, sortedby="date", reverse=True)

# Tri multi-champs
results = searcher.search(query, sortedby=[
    sorting.FieldFacet("category"),
    sorting.ScoreFacet()
])
```

## Facettes

```python
from whoosh import sorting

facet = sorting.FieldFacet("category")
with searcher.all_features() as features:
    facets = features.facet(facet)
    for cat, count in facets.most_common():
        print(f"{cat}: {count}")
```

## Filtrage et masquage

```python
from whoosh.query import Term

# Autoriser seulement les documents publiés
filter_q = Term("published", True)
results = searcher.search(query, filter=filter_q)

# Exclure les brouillons
mask_q = Term("draft", True)
results = searcher.search(query, mask=mask_q)
```

## Surbrillance

```python
results = searcher.search(query, terms=True)

for hit in results:
    print(hit.highlights("content", top=2))
```

## Recherches à temps limité

```python
from whoosh.collectors import TimeLimitCollector

with ix.searcher() as s:
    c = s.collector(limit=None)
    tlc = TimeLimitCollector(c, timelimit=5.0)
    try:
        s.search_with_collector(query, tlc)
    except TimeLimit:
        print("Recherche annulée: trop lente")
    results = tlc.results()
```


## DOCUMENT (FR): Sorting

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Sorting

The `whoosh.sorting` module provides facets and sort-key computation for ordering and grouping search results.

## Quick start

```python
from whoosh import sorting

# Sort by a field
results = searcher.search(query, sortedby="date")

# Sort descending
results = searcher.search(query, sortedby=sorting.FieldFacet("price", reverse=True))
```

For the full API reference, see [Sorting API](/api/sorting).


## DOCUMENT (FR): Translation Status

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Translation Completion Tracking

- [x] EN quickstart
- [x] EN guides
- [x] EN API pages
- [x] EN examples
- [x] FR quickstart
- [x] FR guides
- [x] FR API pages
- [x] FR examples


## DOCUMENT (FR): Autocomplete

# Autocomplétion avec Whoosh‑NG

Cet exemple démontre la fonctionnalité **autocomplete/suggestion** avec le plugin `whoosh_modern.autocomplete`.

## 1. Installation

```bash
pip install "whoosh-ng[autocomplete]"
```

## 2. Schéma avec champ Keyword pour les termes

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, KEYWORD
from whoosh_modern.autocomplete.plugin import AutocompletePlugin
from whoosh.plugins.manager import PluginManager

schema = Schema(
    title=TEXT(stored=True),
    tags=KEYWORD(stored=True, commas=True),
)

ix = index.create_in("autocomplete_index", schema)
```

## 3. Enregistrer le plugin

```python
AutocompletePlugin().register(PluginManager())
```

## 4. Indexer les documents

```python
with ix.writer() as w:
    w.add_document(title="Python Programming", tags="python,programming,language")
    w.add_document(title="JavaScript Basics", tags="javascript,programming,web")
    w.add_document(title="Machine Learning", tags="ml,ai,data-science")
    w.commit()
```

## 5. Utiliser l’index inversé pour les suggestions

```python
from whoosh_modern.autocomplete.factory import create_autocomplete
from whoosh.registry import AutocompleteRegistry

provider = AutocompleteRegistry.get("inverted")

with ix.searcher() as s:
    for term in s.lexicon("tags"):
        provider.add_term(term, s.doc_count_all())

suggestions = provider.suggest("py", maxdist=1, limit=5)
print(suggestions)  # ['python', 'programming']
```

## Points clés

- Installez avec `pip install whoosh-ng[autocomplete]`.
- Utilisez des champs `KEYWORD` pour les tags/mots-clés.
- Enregistrez `AutocompletePlugin` pour activer les suggestions.
- Le provider inverted supporte les correspondances floues (`maxdist`).


## DOCUMENT (FR): Basic Indexing

# Indexation de base

Exemples pour indexer des documents dans Whoosh‑NG. Chaque section est un script
autonome **exécutable**.

> **Scénario concret** : Vous construisez un moteur de recherche de blog. Vous avez
> un fichier CSV d'articles (`blog_posts.csv`) avec les colonnes `title`, `url`,
> `tags`, `body` et `published_at`.

## 1. Schéma de production

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC, DATETIME
from datetime import datetime

schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    url=ID(stored=True),
    tags=KEYWORD(stored=True, commas=True),
    body=TEXT(stored=True, phrase=True),
    published_at=DATETIME(stored=True, sortable=True),
    word_count=NUMERIC(int, stored=True),
)
```

## 2. Créer l'index

```python
from whoosh import index
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC, DATETIME
import shutil

shutil.rmtree("blog_index", ignore_errors=True)
ix = index.create_in("blog_index", schema)
```

## 3. Indexer depuis un CSV

```python
import csv
from datetime import datetime

with open("blog_posts.csv", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)
    with ix.writer() as writer:
        for row in reader:
            writer.add_document(
                doc_id=row["doc_id"],
                title=row["title"],
                url=row["url"],
                tags=row["tags"],
                published_at=datetime.fromisoformat(row["published_at"]),
                word_count=int(row["word_count"]),
                body=row["body"],
            )
        writer.commit()
```

## 4. Mise à jour incrémentale

```python
updated_posts = [
    {"doc_id": "1", "title": "Titre mis à jour", "body": "Nouveau contenu..."},
]

with ix.writer() as writer:
    for post in updated_posts:
        writer.update_document(
            doc_id=post["doc_id"],
            title=post["title"],
            url=f"/posts/{post['doc_id']}",
            tags="python,search",
            published_at=datetime(2024, 6, 1),
            word_count=len(post["body"].split()),
            body=post["body"],
        )
    writer.commit()
```

## 5. Suppression

```python
from whoosh.query import Term

with ix.writer() as writer:
    writer.delete_by_term("doc_id", "3")
    writer.commit()
```

## 6. Indexation en bloc (10k+ documents)

```python
from whoosh.writing import BufferedWriter

buffered = BufferedWriter(ix, period=60, limit=500)
try:
    for doc in large_dataset:
        with buffered:
            buffered.add_document(**doc)
finally:
    buffered.close()
```

## 7. Recherche sur les données indexées

```python
from whoosh.qparser import QueryParser

ix = index.open_dir("blog_index")

with ix.searcher() as s:
    qp = QueryParser("body", ix.schema)
    q = qp.parse("moteur de recherche")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} | {hit['url']} | score={hit.score:.3f}")
```


## DOCUMENT (FR): Data Sources

# Sources de données

Whoosh-NG fournit une couche de sources de données flexible pour l'indexation de documents à partir de bases de données SQL, d'API REST, d'API GraphQL, de fichiers et d'autres fournisseurs.

## Protocole DataSource

Toutes les sources implémentent le protocole `DataSource` :

```python
from whoosh_modern.data_sources import DataSource

class DataSource(Protocol):
    @property
    def name(self) -> str: ...

    def discover_schema(self) -> Schema: ...
    def iter_documents(self) -> Iterator[Document]: ...
    def document_count(self) -> int: ...
    def metadata(self) -> Mapping[str, Any]: ...
```

### Protocoles de capacités

| Protocole | Description |
|----------|-------------|
| `DataSource` | Protocole de base : nom, schéma, itération, métadonnées |
| `IncrementalDataSource` | Supporte `iter_changes(since)` |
| `AsyncDataSource` | Diffusion asynchrone via `aiter_documents()` |
| `RefreshableDataSource` | Support de `refresh()` |
| `CountableDataSource` | `document_count()` |
| `MetadataDataSource` | `metadata()` |
| `ObservableDataSource` | Callbacks d'observation pour les changements de documents |

---

## SQLSource

`SQLSource` se connecte aux bases de données SQL et restitue les documents depuis les résultats de requête, avec pooling de connexions automatique.

### Utilisation de base

```python
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("mydb.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM products",
)

schema = source.discover_schema()
for doc in source.iter_documents():
    print(doc["title"], doc["price"])

count = source.document_count()
```

### Pooling de connexions

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(
    connection="sqlite:///mydb.db",
    query="SELECT * FROM products",
    pool_size=10,
    pool_recycle=3600,
)
```

### GROUP BY

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT category, COUNT(*) as doc_count,
               AVG(price) as avg_price
        FROM products GROUP BY category
    """,
)
```

### JOINs avec alias

```python
source = SQLSource(
    connection=conn,
    query="""
        SELECT p.id AS product_id, p.name AS product_name,
               c.name AS category_name
        FROM products p
        JOIN categories c ON p.category_id = c.id
    """,
)
```

### Synchronisation incrémentale

```python
from datetime import datetime

source = SQLSource(
    connection=conn,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)

for doc in source.iter_changes(since=datetime(2025, 1, 1)):
    print(doc["id"], doc["updated_at"])
```

### SQLAlchemySource

```python
from whoosh_modern.data_sources.sqlalchemy_ds import SQLAlchemySource
from sqlalchemy import create_engine

engine = create_engine("postgresql://user:pass@localhost/mydb")
source = SQLAlchemySource(
    engine=engine,
    query="SELECT * FROM articles",
    incremental_field="updated_at",
    id_field="id",
)
```

### PeeweeSource

```python
from whoosh_modern.data_sources.peewee_ds import PeeweeSource
from peewee import SqliteDatabase

db = SqliteDatabase("mydb.db")
source = PeeweeSource(
    database=db,
    model=MyArticleModel,
    fields=["id", "title", "content"],
)
```

### TortoiseSource (async)

```python
from whoosh_modern.data_sources.tortoise_ds import TortoiseSource

source = TortoiseSource(
    model="myapp.models.Article",
    fields=["id", "title", "content"],
)
```

---

## RESTSource

`RESTSource` récupère les documents depuis des API REST avec pagination et authentification.

### Utilisation de base

```python
from whoosh_modern.data_sources.rest import RESTSource

source = RESTSource(
    url="https://api.example.com/v2/products",
    method="GET",
    headers={"Authorization": "Bearer your_token"},
    pagination="page",
    page_size=50,
)

schema = source.discover_schema()
for doc in source.iter_documents():
    print(doc["name"], doc["price"])
```

### Stratégies de pagination

| Stratégie | Paramètres |
|----------|-----------|
| `page` | `?page=N&size=M` |
| `offset` | `?offset=N&limit=M` |
| `cursor` | `?cursor=XYZ&size=M` |

### Authentification

```python
# Bearer token
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": "Bearer your_token"},
)

# API key
source = RESTSource(
    url="https://api.example.com/data",
    headers={"X-API-Key": "your_api_key"},
)

# Basic auth
import base64
creds = base64.b64encode(b"user:pass").decode()
source = RESTSource(
    url="https://api.example.com/data",
    headers={"Authorization": f"Basic {creds}"},
)
```

### Document Path

Pour les réponses API imbriquées :

```python
source = RESTSource(
    url="https://api.example.com/api/v2/products",
    document_path="data.results",
    pagination="page",
)
```

---

## GraphQLSource

```python
from whoosh_modern.data_sources.graphql import GraphQLSource

source = GraphQLSource(
    url="https://api.example.com/graphql",
    query="""
        query GetProducts($limit: Int!, $offset: Int!) {
            products(limit: $limit, offset: $offset) {
                id
                name
                price
            }
        }
    """,
    pagination="offset",
    page_size=100,
    headers={"Authorization": "Bearer your_token"},
)
```

---

## Sources de fichiers

### FastCSVSource

```python
from whoosh_modern.data_sources.fast_csv import FastCSVSource

source = FastCSVSource(
    file_path="data/products.csv",
    id_field="id",
    incremental_field="updated_at",
)
```

### JSONSource

```python
from whoosh_modern.data_sources.json import JSONSource

source = JSONSource(file_path="data/products.json")
# ou fichier JSONL
source = JSONSource(file_path="data/logs.jsonl", format="jsonl")
```

### ParquetSource

```python
from whoosh_modern.data_sources.parquet_ds import ParquetSource

source = ParquetSource(
    file_path="data/large_dataset.parquet",
    engine="pyarrow",
    batch_size=1000,
)
```

### PandasSource

```python
from whoosh_modern.data_sources.pandas_ds import PandasSource
import pandas as pd

df = pd.read_csv("data/products.csv")
source = PandasSource(dataframe=df)
```

### PolarsSource

```python
from whoosh_modern.data_sources.polars_ds import PolarsSource
import polars as pl

df = pl.read_csv("data/products.csv")
source = PolarsSource(dataframe=df)
```

---

## DataSourceConfig

Pour une configuration programmatique :

```python
from whoosh_modern.data_sources.config import DataSourceConfig

config = DataSourceConfig(
    source_type="sql",
    connection="sqlite:///mydb.db",
    query="SELECT * FROM products",
    id_field="id",
    incremental_field="updated_at",
    mapping={"db_title": "title"},
    exclude=["description_long"],
)

source = config.create_source()
```

### Sources de données disponibles

| Classe | Type | Dépendances |
|-------|------|-------------|
| `SQLSource` | Bases SQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | SQLAlchemy | `sqlalchemy` |
| `RESTSource` | API REST | aucune (stdlib `urllib`) |
| `GraphQLSource` | API GraphQL | aucune (stdlib `urllib`) |
| `FastCSVSource` | Fichiers CSV | aucune |
| `JSONSource` | JSON/JSONL | aucune |
| `ParquetSource` | Parquet | `pyarrow` ou `pandas` |
| `PandasSource` | DataFrames pandas | `pandas` |
| `PolarsSource` | DataFrames Polars | `polars` |
| `PeeweeSource` | ORM Peewee | `peewee` |
| `TortoiseSource` | ORM Tortoise | `tortoise-orm` |
| `PydanticSource` | Modèles Pydantic | `pydantic` |


## DOCUMENT (FR): Facets

# Gestionnaire de facettes

`FacetManager` gère la configuration des facettes pour un schéma Whoosh.

## Usage basique

```python
from whoosh.fields import Schema, TEXT, NUMERIC, BOOLEAN
from whoosh_modern.facets import FacetManager, TermsFacet, RangeFacet

schema = Schema(
    title=TEXT(stored=True),
    category=TEXT(sortable=True),
    price=NUMERIC(),
    active=BOOLEAN(),
)

manager = FacetManager(schema)
```

## Auto-découverte

| Type Whoosh | Facette auto-découverte |
|-------------|-------------------------|
| `KEYWORD`, `BOOLEAN`, `ID` | `TermsFacet` |
| `NUMERIC` | `RangeFacet` |
| `DATETIME` | `DateRangeFacet` |

```python
facets = manager.get_facets()
manager.is_facetable("category")  # True
manager.is_facetable("title")     # False
```

## Remplacement manuel

```python
manager.set_manual_override("category", {"type": "terms", "limit": 50})
manager.set_manual_override("price", {"type": "range", "buckets": ["0-10", "10-50"]})
```

## Statistiques

```python
stats = manager.get_facet_stats()
# {"total_fields": 4, "auto_discovered_facets": 2, ...}
```


## DOCUMENT (FR): Fastapi Search

# Intégration FastAPI

Un service FastAPI complet exposant la recherche Whoosh-NG via HTTP.

## 1. Installation

```bash
pip install "whoosh-ng[api]" fastapi uvicorn
```

## 2. Créer l'index

```python
# setup_index.py
import json
from whoosh import index
from whoosh.fields import Schema, TEXT, ID

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
)

ix = index.create_in("docs_index", schema)

with ix.writer() as w:
    for doc in json.load(open("documents.json")):
        w.add_document(
            id=doc["id"],
            title=doc["title"],
            content=doc["content"],
        )
    w.commit()
```

## 3. Service REST

```python
# main.py
from fastapi import FastAPI, Query
from typing import Optional
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh_fastapi import create_app

ix = index.open_dir("docs_index")

# Option A: Utiliser l'aide
app = create_app(ix, prefix="/api/v1")

# Option B: endpoints manuels
# app = FastAPI(title="Document Search API", version="1.0.0")
#
# @app.get("/api/v1/health")
# async def health():
#     return {"status": "ok"}
#
# @app.post("/api/v1/search")
# async def search(q: str = Query(...), limit: int = 10):
#     with ix.searcher() as s:
#         parser = QueryParser("content", ix.schema)
#         results = s.search(parser.parse(q), limit=limit)
#         return {"hits": [dict(h) for h in results], "total": len(results)}

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8000)
```

## 4. Démarrer le serveur

```bash
uvicorn main:app --reload --port 8000
```

## 5. Tester l'API

```bash
# Vérification de santé
curl http://localhost:8000/api/v1/health

# Recherche
curl -X POST http://localhost:8000/api/v1/search \
  -H "Content-Type: application/json" \
  -d '{"q": "python recherche"}'

# Get document by ID
curl http://localhost:8000/api/v1/documents/doc1

# Autocomplétion
curl "http://localhost:8000/api/v1/autocomplete?q=py"
```

## 6. Indexation en lot

```python
# Ajouter à main.py pour l'indexation dynamique
from fastapi import FastAPI
from whoosh.writing import BufferedWriter

@app.post("/api/v1/index")
async def index_docs(docs: list[dict]):
    with BufferedWriter(ix, period=30, limit=50) as w:
        for doc in docs:
            w.add_document(**doc)
    return {"indexed": len(docs)}
```

## Points clés

- `create_app()` de `whoosh_fastapi` fournit les endpoints `/health`, `/search` et `/autocomplete`.
- Tous les appels bloquants s'exécutent hors boucle d'événements via `run_sync`.
- Utilisez `BufferedWriter` pour l'indexation en masse.
- `WhooshFastAPI` classe offre une enregistrement par endpoint pour les intégrations personnalisées.


## DOCUMENT (FR): Middleware Pipeline

# Pipeline de middleware

Le pipeline de middleware enveloppe les opérations avec des préoccupations transversales : nouvelle tentative, journalisation, etc.

## Architecture

```python
from whoosh_modern.middleware import Middleware, MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

pipeline = MiddlewarePipeline(
    RetryMiddleware(attempts=3, backoff="exponential"),
    LoggingMiddleware(),
)

result = pipeline.execute(my_operation)
```

## RetryMiddleware

```python
from whoosh_modern.middleware import RetryMiddleware

retry = RetryMiddleware(attempts=3, backoff="exponential")

@retry.wrap
def operation():
    return fetch_data()
```

Stratégies de backoff :
- `"exponential"` : 1s, 2s, 4s, 8s...
- `"linear"` : 1s, 2s, 3s, 4s...

## LoggingMiddleware

```python
from whoosh_modern.middleware import LoggingMiddleware
import logging

logger = logging.getLogger("benchmark")
logging_mw = LoggingMiddleware(logger=logger)

@logging_mw.wrap
def tracked():
    return fetch_data()
```


## DOCUMENT (FR): Middleware

# Exemples de Middleware

Des exemples pratiques pour construire et utiliser les middleware de Whoosh-NG.

## 1. Middleware de Logging

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class LoggingMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        print(f"[RECHERCHE] Requête: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.results is not None:
            print(f"[RÉSULTATS] {len(context.results)} résultats trouvés")
        return context
```

## 2. Middleware de Metrics

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MetricsMiddleware(Middleware):
    def __init__(self) -> None:
        self._metrics = {}

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["documents_indexés"] = self._metrics.get("documents_indexés", 0) + 1
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        self._metrics["recherches_executées"] = self._metrics.get("recherches_executées", 0) + 1
        return context

    def get_metrics(self) -> dict:
        return dict(self._metrics)
```

## 3. Middleware de Cache

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class SearchCacheMiddleware(Middleware):
    def __init__(self) -> None:
        self._cache = {}

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and str(context.query) in self._cache:
            context.metadata["_résultat_cache"] = self._cache[str(context.query)]
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if context.query and context.results is not None:
            self._cache[str(context.query)] = context.results
        return context
```

## 4. Appliquer un Middleware

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_searcher

chain = MiddlewareChain([
    LoggingMiddleware(),
    MetricsMiddleware(),
])

with ix.searcher() as base_searcher:
    searcher = apply_middleware_to_searcher(base_searcher, chain.middlewares)
    results = searcher.search(query)
```

## Points clés

| Hook | Phase | Description |
|------|-------|-------------|
| `startup` | Init | Appelé une fois à l'initialisation |
| `shutdown` | Nettoyage | Appelé à la fermeture |
| `before_index` | Indexation | Avant l'ajout d'un document |
| `after_index` | Indexation | Après l'ajout d'un document |
| `before_delete` | Suppression | Avant la suppression |
| `after_delete` | Suppression | Après la suppression |
| `before_search` | Recherche | Avant l'exécution de la requête |
| `after_search` | Recherche | Après le retour des résultats |
| `on_error` | Erreur | En cas d'exception |
| `on_commit` | Commit | Après writer.commit() |


## DOCUMENT (FR): Movie Search

# Application de Recherche de Films

Exemple complet montrant comment créer une petite **application de recherche de films** avec Whoosh‑NG : conception du schéma, indexation à partir d’un fichier JSON, recherche facettée, mise en évidence et filtrage.

## 1. Schéma

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    director=TEXT(stored=True),
    genre=KEYWORD(stored=True, commas=True, scorable=True),
    year=NUMERIC(int, stored=True),
    synopsis=TEXT,
)
```

## 2. Indexer les documents

```python
import json
import shutil
from whoosh import index

shutil.rmtree("movies", ignore_errors=True)
ix = index.create_in("movies", schema)

movies = json.load(open("movies.json"))

with ix.writer() as w:
    for m in movies:
        w.add_document(
            id=str(m["id"]),
            title=m["title"],
            director=m["director"],
            genre=",".join(m["genres"]),
            year=m["year"],
            synopsis=m["synopsis"],
        )
    w.commit()
```

Fichier `movies.json` :

```json
[
  {
    "id": 1,
    "title": "Blade Runner",
    "director": "Ridley Scott",
    "genres": ["sci-fi", "thriller"],
    "year": 1982,
    "synopsis": "Un chasseur de replicants questionne l’humanité dans un futur pluvieux."
  }
]
```

## 3. Recherche avec facettes et mise en évidence

```python
from whoosh import index
from whoosh.qparser import MultifieldParser
from whoosh.sorting import FieldFacet

ix = index.open_dir("movies")

qp = MultifieldParser(["title", "synopsis", "director"], ix.schema)

with ix.searcher() as s:
    q = qp.parse("sci-fi")

    results = s.search(
        q,
        sortedby=FieldFacet("year", reverse=True),
        groupedby=FieldFacet("genre", allow_overlap=True),
        limit=20,
    )

    for hit in results:
        print(hit["title"], hit["year"], "|", round(hit.score, 2))
        print("  ", hit.highlights("synopsis"))

    print("\nGenres:", results.groups("genre"))
```

## 4. Filtrage

Filtrer les films de science-fiction après 1990 :

```python
from whoosh import index
from whoosh.qparser import QueryParser
from whoosh.query import Term, And, NumericRange

ix = index.open_dir("movies")
qp = QueryParser("synopsis", ix.schema)

with ix.searcher() as s:
    user_q = qp.parse("future")
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("year", 1990, None),
    ])
    results = s.search(user_q, filter=filters)
    for hit in results:
        print(hit["title"], hit["year"])
```

## Points clés

- `KEYWORD(commas=True)` stocke des champs multi-valeurs facetables.
- `MultifieldParser` recherche sur plusieurs champs avec des boosts optionnels.
- `FieldFacet` permet les facettes et le tri.
- `hit.highlights()` renvoie des fragments mis en évidence prêts à afficher.


## DOCUMENT (FR): Plugin Dev

# Développement de Plugins

Guide complet pour créer, enregistrer et tester vos propres plugins Whoosh-NG.

## 1. Classe de base des Plugins

Tous les plugins héritent de `whoosh.plugins.base.Plugin` :

```python
from whoosh.plugins.base import Plugin

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Appelé quand le plugin est chargé."""
        manager.register("my_handler", MyHandler())

    def register_hooks(self):
        """Enregistrer les hooks avec le décorateur hookimpl."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass

        register_hook("on_search", hookimpl(on_search))
```

## 2. Enregistrement d’un Plugin

### Enregistrement manuel

```python
from whoosh.plugins.manager import PluginManager

plugin = MyPlugin()
PluginManager.register(plugin)
```

### Auto-découverte via Entry Points

Dans `pyproject.toml` :

```toml
[project]
name = "whoosh-ng-my-plugin"

[project.entry-points."whoosh_ng.plugins"]
my_plugin = "my_package.plugin:MyPlugin"
```

Auto-chargement :

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()
```

## 3. Exemple de Plugin Provider

```python
from whoosh.plugins.base import Plugin
from whoosh.registry import VectorRegistry

class MyVectorProvider:
    def search(self, query_vector, k=10):
        return [{"doc_id": "1", "score": 0.95}]

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"

    def register(self, manager):
        provider = MyVectorProvider()
        VectorRegistry.register("my_vector", provider, self.name)
```

## 4. Tester son Plugin

```python
import pytest
from whoosh.plugins.manager import PluginManager

class TestMyPlugin:
    def test_register(self):
        plugin = MyPlugin()
        manager = PluginManager()
        plugin.register(manager)
        assert "my_handler" in manager._plugins

    def test_entry_point(self):
        manager = PluginManager()
        manager.register(MyPlugin())
        assert "my_plugin" in manager.list_enabled()
```

## 5. API du PluginManager

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyPlugin())
manager.enable("my_plugin")
manager.disable("my_plugin")
manager.list_plugins()
manager.list_enabled()
plugin = manager.get("my_plugin")
```

## 6. Plugins Intégrés

- `whoosh_modern.vector` - Recherche vectorielle (NumPy)
- `whoosh_modern.autocomplete` - Autocomplétion par index inversé
- `whoosh_fastapi` - Endpoints REST FastAPI

```python
from whoosh.plugins.manager import PluginManager
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

PluginManager.load_plugins()
```


## DOCUMENT (FR): Schema Discovery

# Découverte de schéma

La découverte de schéma infère un schéma Whoosh à partir des métadonnées de résultats ou d'échantillons de documents.

## Depuis les métadonnées de colonnes

```python
from whoosh_modern.schema_discovery import SchemaDiscovery
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
cursor = conn.cursor()
cursor.execute("PRAGMA table_info(reuters_articles)")
columns = [(row[1], row[2]) for row in cursor.fetchall()]

schema = SchemaDiscovery.from_result_set(columns)
```

## Depuis des échantillons de documents

```python
from whoosh_modern.data_sources.sql import SQLSource

source = SQLSource(connection=conn, query="SELECT * FROM reuters_articles LIMIT 10")
docs = list(source.iter_documents())[:10]
schema = SchemaDiscovery.from_sample(docs)
```

## Détection du champ ID

```python
id_field = SchemaDiscovery.detect_id_field(dict(schema))
```

## Détection des doublons

`from_result_set` lève `SchemaDiscoveryError` sur les noms de colonnes dupliqués.


## DOCUMENT (FR): Search Models

# Modèles de recherche

Exemples de mapping automatique de modèles Python vers des schémas Whoosh.

## Dataclass

```python
from dataclasses import dataclass
from whoosh.fields import Schema, TEXT, NUMERIC
from whoosh_modern.models import ModelIndex
import tempfile
import shutil

@dataclass
class Book:
    title: str
    year: int
    tags: list[str] | None = None

idx = ModelIndex(Book)
print(idx.schema)
```

## Pydantic

```python
from pydantic import BaseModel
from whoosh_modern.models import register_model

class BookModel(BaseModel):
    title: str
    year: int
    tags: list[str] | None = None

idx = register_model(BookModel)
schema = idx.schema
```

## SQLAlchemy

```python
from sqlalchemy import Column, Integer, String
from whoosh_modern.models import register_model

class BookSQL:
    __tablename__ = "book"
    title = Column(String, info={"search": {"fulltext": True, "stored": True}})
    year = Column(Integer, info={"search": {"sortable": True}})

idx = register_model(BookSQL)
schema = idx.schema
```

## SQLModel

```python
from sqlmodel import SQLModel, Field
from whoosh_modern.models import register_model

class Book(SQLModel, table=True):
    id: int = Field(primary_key=True)
    title: str = Field(sa_column_kwargs={"info": {"search": {"fulltext": True}}})
    year: int

idx = register_model(Book)
schema = idx.schema
```

## msgspec

```python
import msgspec
from whoosh_modern.models import register_model

class Book(msgspec.Struct):
    title: str = msgspec.field(metadata={"search": {"fulltext": True}})
    year: int

idx = register_model(Book)
schema = idx.schema
```

## Indexation de documents

```python
from whoosh import index

tmp = tempfile.mkdtemp()
ix = index.create_in(tmp, schema)

with ix.writer() as w:
    book = Book(title="Guide Whoosh", year=2024, tags=["python", "recherche"])
    doc = idx.to_whoosh_document(book)
    w.add_document(**doc)
    w.commit()
```

## Auto-indexation avec AutoIndexer

```python
from whoosh_modern.models import AutoIndexer

auto = AutoIndexer(ix, on_error="raise")
auto.register(Book)

# Indexer une instance unique
book = Book(title="Nouveau livre", year=2024, tags=["python"])
auto.index(book)

# Supprimer par ID
auto.remove(book)

# Versions asynchrones
await auto.index_async(book)
await auto.remove_async(book)
```

Pour les modèles SQLAlchemy, `AutoIndexer` se connecte automatiquement aux événements `after_insert`, `after_update` et `after_delete`.

## Nettoyage

```python
shutil.rmtree(tmp)
```


## DOCUMENT (FR): Search View

# SearchView

`SearchView` intègre une source de données avec l'indexation Whoosh.

## Usage basique

```python
from whoosh_modern.views import SearchView
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(
    connection=conn,
    query="SELECT * FROM reuters_articles",
    incremental_field="article_date",
    id_field="id",
)

view = SearchView(
    name="reuters",
    source=source,
)

# Créer l'index
ix = view.build("indexdir")
```

## Rafraîchissement incrémental

```python
# Réindexation complète
count = view.reindex()

# Rafraîchissement incrémental
count = view.refresh()
```

## Validation

```python
results = view.validate()
for result in results:
    print(f"Niveau {result.level}: {'PASS' if result.passed else 'FAIL'}")
```

## Mode strict

```python
view = SearchView(
    name="strict",
    source=source,
    strict=True,  # Lever ValidationError en cas d'échec
)
```


## DOCUMENT (FR): Search

# Recherche

Exemples concrets d'interrogation d'un index Whoosh-NG.

> **Scénario** : Vous avez indexé un catalogue de livres (voir `basic-indexing.md`).
> Les exemples ci-dessous montrent les patterns de recherche produit.

## Prérequis

L'index `book_index/` contient ce schéma :

```python
from whoosh.fields import Schema, TEXT, ID, KEYWORD, NUMERIC

schema = Schema(
    isbn=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    author=TEXT(stored=True),
    content=TEXT,
    genre=KEYWORD(stored=True, commas=True),
    published_year=NUMERIC(int, stored=True, sortable=True),
    rating=NUMERIC(float, stored=True, sortable=True),
)
```

## 1. Recherche basique — « Livres sur Python »

```python
from whoosh import index
from whoosh.qparser import QueryParser

ix = index.open_dir("book_index")

with ix.searcher() as s:
    qp = QueryParser("content", ix.schema)
    q = qp.parse("python")

    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} par {hit['author']} — score={hit.score:.2f}")
```

## 2. Recherche multi-champs avec boosts

```python
from whoosh.qparser import MultifieldParser

ix = index.open_dir("book_index")
qp = MultifieldParser(
    ["title", "author", "content"],
    ix.schema,
    fieldboosts={"title": 3.0, "author": 2.0, "content": 1.0},
)

q = qp.parse("clean code")

with ix.searcher() as s:
    results = s.search(q, limit=10)
    for hit in results:
        print(f"{hit['title']} — {hit['author']}")
```

## 3. Pagination — « Page 3 des résultats »

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("machine learning")

with ix.searcher() as s:
    page = s.search_page(q, 3, pagelen=15)

    print(f"Page {page.number} / {page.pagecount} ({page.total} résultats)")
    for hit in page:
        print(f"  {hit['title']}")
```

## 4. Tri et filtres — « Sci-fi noté ≥4 après 2010 »

```python
from whoosh.query import Term, And, NumericRange
from whoosh.sorting import FieldFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("space")

with ix.searcher() as s:
    filters = And([
        Term("genre", "sci-fi"),
        NumericRange("published_year", 2010, None),
    ])

    results = s.search(
        q,
        filter=filters,
        sortedby=FieldFacet("rating", reverse=True),
        limit=20,
    )
    for hit in results:
        print(f"{hit['title']} ({hit['published_year']}) — note: {hit['rating']}")
```

## 5. Mise en évidence — « Où la requête correspond »

```python
ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("réseaux de neurones")

with ix.searcher() as s:
    results = s.search(q, limit=5)

    for hit in results:
        snippet = hit.highlights("content", top=2)
        print(f"{hit['title']}:")
        print(f"  {snippet}")
```

## 6. Recherche par plage — « Livres publiés en 2023 »

```python
from whoosh.query import NumericRange

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = NumericRange("published_year", 2023, 2023)
    results = s.search(q)
    print(f"{results.total} livres publiés en 2023")
```

## 7. Recherche par préfixe — « Titres commençant par "Deep" »

```python
from whoosh.query import Prefix

ix = index.open_dir("book_index")

with ix.searcher() as s:
    q = Prefix("title", "Deep")
    results = s.search(q)
    for hit in results:
        print(hit["title"])
```

## 8. Recherche facetée — « Grouper par genre »

```python
from whoosh.sorting import FieldFacet

ix = index.open_dir("book_index")
qp = QueryParser("content", ix.schema)
q = qp.parse("programming")

with ix.searcher() as s:
    results = s.search(q, groupedby=FieldFacet("genre"))

    for genre, group in results.groups("genre").items():
        print(f"{genre}: {len(group)} résultats")
```

## Points clés

- `QueryParser` analyse une chaîne en objet `Query`.
- `MultifieldParser` recherche plusieurs champs avec des boosts.
- `search_page()` gère la pagination.
- `filter` restreint les résultats sans affecter le score.
- `sortedby` trie par valeur de champ ou par score de pertinence.
- `hit.highlights()` renvoie des extraits mis en surbrillance.


## DOCUMENT (FR): Validation

# Framework de validation

Le framework de validation exécute 4 niveaux de vérifications sur une source de données avant l'indexation.

## Les 4 niveaux

| Niveau | Méthode | Objectif |
|-------|---------|---------|
| **Niveau 1** | `validate_structural(source)` | Disponibilité de la source, détection de schéma |
| **Niveau 2** | `validate_search(schema)` | Champs indexables, compatibilité des analyseurs |
| **Niveau 3** | `validate_performance(schema, source)` | Avertissements de performance (TEXT, etc.) |
| **Niveau 4** | `validate_runtime(source, sample_size)` | Itération d'échantillon, validation de types |

## Usage basique

```python
from whoosh_modern.validation import ValidationFramework, ValidationResult
from whoosh_modern.data_sources.sql import SQLSource
import sqlite3

conn = sqlite3.connect("benchmark/benchmark_data.db")
source = SQLSource(connection=conn, query="SELECT * FROM reuters_articles")

validator = ValidationFramework()
results: list[ValidationResult] = validator.validate(source)

for result in results:
    status = "PASS" if result.passed else "FAIL"
    print(f"Niveau {result.level}: {status}")
```

## Validation individuelle

```python
errors = validator.validate_structural(source)
errors = validator.validate_search(schema)
warnings = validator.validate_performance(schema, source)
errors = validator.validate_runtime(source, sample_size=100)
```


## DOCUMENT (FR): Vector Search

# Recherche Vectorielle avec Whoosh‑NG

Cet exemple montre comment activer la **recherche sémantique/vectorielle** avec l’option `vector` supplémentaire. Nous indexons des embeddings et effectuons une recherche de plus proches voisins (k-NN).

## 1. Installer les dépendances optionnelles

```bash
pip install "whoosh-ng[vector]" numpy
```

## 2. Schéma avec champ Vectoriel

```python
from whoosh.fields import Schema, TEXT, ID, VECTOR

schema = Schema(
    doc_id=ID(stored=True, unique=True),
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VECTOR(stored=True, dim=128),
)
```

## 3. Indexer les vecteurs

```python
import numpy as np
from whoosh import index
import shutil

shutil.rmtree("vector_index", ignore_errors=True)
ix = index.create_in("vector_index", schema)

documents = [
    {"doc_id": "doc1", "title": "Python Basics", "content": "Learn Python programming."},
    {"doc_id": "doc2", "title": "Advanced Python", "content": "Deep dive into decorators."},
    {"doc_id": "doc3", "title": "Data Science", "content": "Pandas and NumPy."},
]

np.random.seed(42)
embeddings = {d["doc_id"]: np.random.rand(128).astype(np.float32) for d in documents}

with ix.writer() as w:
    for doc in documents:
        w.add_document(
            doc_id=doc["doc_id"],
            title=doc["title"],
            content=doc["content"],
            embedding=embeddings[doc["doc_id"]].tobytes(),
        )
    w.commit()
```

## 4. Recherche vectorielle avec NumpyProvider

```python
from whoosh_modern.vector.numpy_provider import NumpyProvider
from whoosh_modern.vector.plugin import VectorPlugin
from whoosh.plugins.manager import PluginManager

VectorPlugin().register(PluginManager())

provider = NumpyProvider()
for doc_id, vec in embeddings.items():
    provider.add([(doc_id, vec.tolist())])

query_vec = embeddings["doc1"]
hits = provider.search(query_vec, k=2)

for hit in hits:
    print(f"doc_id={hit.doc_id}, score={hit.score:.3f}")
```

## 5. Utiliser VectorField pour la sérialisation

```python
from whoosh.vector import VectorField

vf = VectorField(dimension=128, name="embedding")

values = [0.1, 0.2, 0.3, 0.4] + [0.0] * 124
raw = vf.vector_to_bytes(values)
restored = vf.bytes_to_vector(raw)
print(restored == tuple(values))  # True
```

## Points clés

- Installez avec `pip install whoosh-ng[vector]`.
- `VECTOR` stocke les octets bruts ; utilisez `VectorField` pour convertir.
- `NumpyProvider` implémente la similarité cosinus.
- Enregistrez le plugin via `VectorPlugin().register(manager)`.
- Utilisez `filter_ids` dans `provider.search()` pour restreindre les documents.


## DOCUMENT (FR): Autocomplete Sprint D

# Fournisseurs d'Autocomplétion

Module: `whoosh_modern.autocomplete`
Version: 2.0.0

Le module d'autocomplétion fournit plusieurs stratégies de fournisseurs pour la suggestion de requêtes et la recherche en tapant. Tous les fournisseurs implémentent une interface commune afin de pouvoir changer de stratégie à l'exécution. Les fournisseurs sont enregistrés via le `AutocompleteRegistry` et chargés via des entry points.

## Vue d'ensemble du module

```text
whoosh_modern.autocomplete
    ├── provider.py   # AutocompleteHit, AutocompleteProvider (Protocole)
    ├── ngram.py      # NGramProvider (basé sur des n-grammes de caractères)
    ├── edge_ngram.py # InvertedIndexAutocomplete (correspondance de préfixe par indice inversé)
    ├── fuzzy.py      # FuzzySuggestProvider (correspondance approximative via rapidfuzz)
    ├── factory.py    # create_autocomplete()
    └── plugin.py     # AutocompletePlugin (plugin via entry point)
```

## AutocompleteProvider (Classe de Base)

Située dans `whoosh_modern.autocomplete.provider` :

```python
from whoosh_modern.autocomplete.provider import AutocompleteProvider, AutocompleteHit

class MyProvider(AutocompleteProvider):
    def add(self, phrases: Iterable[str]) -> None:
        """Ajouter des phrases à l'index du fournisseur."""
        ...

    def search(self, prefix: str, limit: int = 10) -> list[AutocompleteHit]:
        """Retourner les suggestions d'autocomplétion pour le préfixe donné."""
        ...
```

### AutocompleteHit

Un objet de résultat simple retourné par les fournisseurs :

```python
class AutocompleteHit:
    def __init__(self, text: str, score: float) -> None:
        self.text = text    # La phrase correspondante
        self.score = score  # Score de pertinence (plus haut = mieux)
```

## Fournisseurs Intégrés

### InvertedIndexAutocomplete

Situé dans `whoosh_modern.autocomplete.edge_ngram`. Utilise une correspondance simple de préfixe contre une liste en mémoire :

```python
from whoosh_modern.autocomplete.edge_ngram import InvertedIndexAutocomplete

provider = InvertedIndexAutocomplete()
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("py", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output:
# python (score: 0.45)
# pyramid (score: 0.43)
# pytorch (score: 0.43)
```

**Score** : Les correspondances exactes de préfixe obtiennent un bonus de 1.5x ; le score de base est `1.0 / (len(phrase) + 1)`.

### NGramProvider

Situé dans `whoosh_modern.autocomplete.ngram`. Construit un index de n-grammes de caractères pour une correspondance de sous-chaîne souple :

```python
from whoosh_modern.autocomplete.ngram import NGramProvider

provider = NGramProvider(n=3)
provider.add(["python programming", "java development", "rust language"])

hits = provider.search("pyt", limit=5)
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
```

**Paramètres :**

| Paramètre | Type | Défaut | Description                          |
|-----------|------|---------|--------------------------------------|
| `n`       | `int` | `3`     | Taille des n-grammes de caractères   |

**Fonctionnement** : Les n-grammes sont extraits de chaque phrase (en minuscules). Lors de la recherche, les n-grammes du préfixe sont comparés à l'index. Les phrases avec plus de n-grammes correspondants obtiennent des scores plus élevés.

### FuzzySuggestProvider

Situé dans `whoosh_modern.autocomplete.fuzzy`. Utilise `rapidfuzz` pour une correspondance approximative (fautes de frappe, correspondances partielles) :

```python
from whoosh_modern.autocomplete.fuzzy import FuzzySuggestProvider

# Nécessite: pip install whoosh-ng[fuzzy]
provider = FuzzySuggestProvider(max_distance=2, score_cutoff=50.0)
provider.add(["python", "pyramid", "pytorch", "java", "javascript"])

hits = provider.search("pythn", limit=5)  # Faute de frappe dans "python"
for hit in hits:
    print(f"{hit.text} (score: {hit.score})")
# Output: python (score: 0.95), ...
```

**Paramètres :**

| Paramètre       | Type  | Défaut  | Description                              |
|-----------------|-------|----------|------------------------------------------|
| `max_distance`  | `int` | `2`      | Distance d'édition maximale (réservé pour une utilisation future) |
| `score_cutoff`  | `float` | `50.0` | Score de similarité minimum (échelle 0-100)   |

**Note** : Nécessite `rapidfuzz` (`pip install whoosh-ng[fuzzy]`). Retourne `ImportError` si non installé.

## Fonction d'Usine

Située dans `whoosh_modern.autocomplete.factory` :

```python
from whoosh_modern.autocomplete import create_autocomplete

# Créer n'importe quel fournisseur par nom
provider = create_autocomplete("inverted")   # InvertedIndexAutocomplete
provider = create_autocomplete("ngram", n=3) # NGramProvider avec n personnalisé
provider = create_autocomplete("fuzzy", max_distance=2, score_cutoff=60.0)
```

**Fournisseurs disponibles :**

| Nom         | Classe                    | Dépendance Optionnelle |
|-------------|---------------------------|------------------------|
| `"inverted"`| `InvertedIndexAutocomplete` | Aucune               |
| `"ngram"`   | `NGramProvider`           | Aucune                |
| `"fuzzy"`   | `FuzzySuggestProvider`    | `rapidfuzz`           |

## Enregistrement dans AutocompleteRegistry

Les fournisseurs sont enregistrés dans `whoosh.registry.AutocompleteRegistry` (une instance de `Registry`) :

```python
from whoosh.registry import AutocompleteRegistry
from whoosh_modern.autocomplete import create_autocomplete

# Enregistrer un fournisseur
provider = create_autocomplete("ngram", n=3)
AutocompleteRegistry.register("ngram-suggester", provider, owner="my_app")

# Le récupérer plus tard
suggester = AutocompleteRegistry.get("ngram-suggester")

# Lister tous les fournisseurs enregistrés
print(AutocompleteRegistry.list_keys())
```

## AutocompletePlugin (Entry Point)

Situé dans `whoosh_modern.autocomplete.plugin`, c'est le plugin intégré enregistré via le groupe d'entry points `whoosh_ng.plugins` :

```python
from whoosh_modern.autocomplete.plugin import AutocompletePlugin

# Automatiquement chargé par PluginManager.load_plugins()
# Enregistre le fournisseur "inverted" dans AutocompleteRegistry
```

### Déclaration d'Entry Point

Dans `pyproject.toml` :

```toml
[project.entry-points."whoosh_ng.plugins"]
whoosh_autocomplete = "whoosh_modern.autocomplete.plugin:AutocompletePlugin"
```

### Détails du Plugin

```python
class AutocompletePlugin(Plugin):
    name = "whoosh_autocomplete"
    version = "3.0.0"

    def register(self, manager):
        # Enregistre InvertedIndexAutocomplete comme "inverted"
        AutocompleteRegistry.register(
            "inverted", create_autocomplete("inverted"), self.name
        )

    def register_hooks(self):
        # Enregistre un hook on_search (actuellement un no-op)
        from whoosh.hooks import hookimpl, register_hook
        register_hook("on_search", hookimpl(on_search))
```

## Exemples d'Utilisation

### Utilisation de Base

```python
from whoosh_modern.autocomplete import create_autocomplete

# Créer et peupler un fournisseur
provider = create_autocomplete("inverted")
provider.add([
    "python programming",
    "python tutorial",
    "java tutorial",
    "javascript framework",
])

# Rechercher des suggestions
hits = provider.search("py", limit=3)
for hit in hits:
    print(f"{hit.text}: {hit.score:.3f}")
```

### Correspondance Floue avec Tolérance aux Fautes

```python
from whoosh_modern.autocomplete import create_autocomplete

provider = create_autocomplete("fuzzy", score_cutoff=70.0)
provider.add(["python", "pytorch", "tensorflow", "keras"])

# Même avec une faute, les suggestions pertinentes sont retournées
hits = provider.search("pyton", limit=5)
for hit in hits:
    print(hit.text, hit.score)
```

### Correspondance par N-grammes pour les Mots Partiels

```python
from whoosh_modern.autocomplete import create_autocomplete

# Utiliser des n-grammes de taille 3 pour une meilleure correspondance de sous-chaînes
provider = create_autocomplete("ngram", n=3)
provider.add(["machine learning", "deep learning", "neural networks"])

# Trouve les phrases contenant les n-grammes de "machin"
hits = provider.search("machin", limit=5)
```

### Intégration avec la Recherche

```python
from whoosh_modern.autocomplete import create_autocomplete

# Construire le fournisseur d'autocomplétion
provider = create_autocomplete("inverted")
provider.add(["python", "java", "javascript", "go", "rust"])

# Utiliser dans un endpoint de recherche
def suggest(prefix: str, limit: int = 5):
    hits = provider.search(prefix, limit=limit)
    return [{"text": h.text, "score": h.score} for h in hits]

# Dans votre endpoint FastAPI/REST :
# GET /api/suggest?q=py&limit=5
# Response: [{"text": "python", "score": 0.45}, ...]
```

## Comparaison des Fournisseurs

| Fournisseur              | Correspondance       | Forces                    | Faiblesses                | Dépendance    |
|--------------------------|----------------------|---------------------------|---------------------------|---------------|
| `inverted`               | Préfixe              | Simple, rapide, pas de deps | Pas de tolérance aux fautes | Aucune          |
| `ngram`                  | Chevauchement n-gramme | Correspondance de sous-chaînes, flexible | Plus lent que préfixe     | Aucune        |
| `fuzzy`                  | Distance d'édition   | Tolérance aux fautes, flexible | Nécessite rapidfuzz    | `rapidfuzz`   |

## Installation

```bash
# Autocomplétion core (inverted + n-gram)
pip install whoosh-ng

# Avec correspondance floue
pip install whoosh-ng[fuzzy]

# Analyse moderne complète
pip install whoosh-ng[modern]
```

## Voir Aussi

- [Guide Système de Plugins](plugins-sprint-c.md) — Enregistrement et découverte de plugins
- [Guide Middleware](middleware-sprint-c.md) — Intégration du pipeline de middleware
- [API: Moderne](../api/modern.md) — Référence complète de l'API pour les extensions d'autocomplétion


## DOCUMENT (FR): Autocomplete

# Autocomplétion

Couche optionnelle d'autocomplétion par edge-ngram pour Whoosh-NG.

## Installer

```bash
pip install whoosh-ng[autocomplete]
```

## Index minimal

```python
from whoosh.fields import Schema, TEXT, AutocompleteField

schema = Schema(
    titre=TEXT(stored=True),
    query=AutocompleteField()
)

with ix.writer() as writer:
    writer.add_document(titre="Démarrage Python", query="demarrage python")
    writer.commit()
```

## Requête d'autocomplétion

```python
from whoosh_modern.autocomplete import AutocompleteProvider

provider = AutocompleteProvider(ix, "query")
suggestions = provider.suggest("de", limit=5)
print(suggestions)  # ["demarrage python", ...]
```


## DOCUMENT (FR): Linguistics Sprint D

# Synonymes & Linguistique

Module: `whoosh_modern.linguistics.synonyms`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

Le module de linguistique fournit un moteur complet d'expansion de synonymes et des analyseurs linguistiques spécifiques à chaque langue. Il s'intègre au pipeline de middleware pour étendre les requêtes et les documents avec des synonymes à la fois à l'indexation et au moment de la recherche.

## Vue d'ensemble du module

```text
whoosh_modern.linguistics
    ├── synonyms/
    │   ├── provider.py       # Protocole SynonymProvider + StaticSynonymProvider
    │   ├── yaml_provider.py  # YAMLSynonymProvider
    │   ├── json_provider.py  # JSONSynonymProvider
    │   ├── store.py          # SQLiteSynonymStore
    │   ├── compiler.py       # SynonymCompiler
    │   ├── manager.py        # SynonymManager
    │   ├── middleware.py     # SynonymExpansionMiddleware
    │   └── languages.py      # LANG_SYNONYMS (FR/EN/DE/ES/IT)
    └── stemmers/
        └── __init__.py       # Analyseurs linguistiques (FR/EN/DE/ES/IT)
```

## Fournisseurs de Synonymes

### SynonymProvider (Protocole)

Le protocole de base que tous les fournisseurs de synonymes implémentent :

```python
from whoosh_modern.linguistics.synonyms import SynonymProvider

class MyProvider(SynonymProvider):
    def get_synonyms(self, word: str) -> list[str]:
        """Retourner les synonymes pour le mot donné."""
        ...

    def add_synonym(self, word: str, synonyms: list[str]) -> None:
        """Ajouter des synonymes pour le mot donné."""
        ...

    def remove_synonym(self, word: str, synonym: str) -> None:
        """Supprimer un synonyme pour le mot donné."""
        ...
```

### StaticSynonymProvider

Fournisseur de synonymes en mémoire basé sur un dictionnaire :

```python
from whoosh_modern.linguistics.synonyms import StaticSynonymProvider

provider = StaticSynonymProvider({
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"],
})

print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### YAMLSynonymProvider

Charge les synonymes depuis un fichier YAML :

```yaml
# synonyms.yaml
car:
  - automobile
  - vehicle
  - auto
house:
  - home
  - residence
```

```python
from whoosh_modern.linguistics.synonyms import YAMLSynonymProvider

# Nécessite: pip install pyyaml
provider = YAMLSynonymProvider("synonyms.yaml")
print(provider.get_synonyms("car"))  # ['automobile', 'vehicle', 'auto']
```

### JSONSynonymProvider

Charge les synonymes depuis un fichier JSON :

```json
{
    "car": ["automobile", "vehicle", "auto"],
    "house": ["home", "residence"]
}
```

```python
from whoosh_modern.linguistics.synonyms import JSONSynonymProvider

provider = JSONSynonymProvider("synonyms.json")
print(provider.get_synonyms("car"))
```

### SQLiteSynonymStore

Magasin de synonymes persistant basé sur SQLite :

```python
from whoosh_modern.linguistics.synonyms import SQLiteSynonymStore

store = SQLiteSynonymStore("synonyms.db")

# Opérations CRUD
store.add_synonym("car", ["automobile", "vehicle"])
print(store.get_synonyms("car"))  # ['automobile', 'vehicle']
store.remove_synonym("car", "automobile")
print(store.get_synonyms("car"))  # ['vehicle']
store.close()
```

### SynonymCompiler

Précompile les données de synonymes brutes en un format de recherche rapide :

```python
from whoosh_modern.linguistics.synonyms import SynonymCompiler

compiler = SynonymCompiler({"car": ["automobile", "vehicle"]})
compiler.add("house", ["home", "residence"])
compiler.merge({"book": ["publication", "work"]})

compiled = compiler.compile()
print(compiled)
# {'car': ['automobile', 'vehicle'], 'house': ['home', 'residence'], 'book': ['publication', 'work']}
```

## SynonymManager

Le `SynonymManager` est l'interface de haut niveau pour gérer les synonymes. Il encapsule un `StaticSynonymProvider` en interne et prend en charge l'import/export :

```python
from whoosh_modern.linguistics.synonyms import SynonymManager

manager = SynonymManager({"car": ["automobile", "vehicle"]})

# CRUD
manager.add_synonyms("house", ["home", "residence"])
print(manager.get_synonyms("house"))  # ['home', 'residence']
manager.remove_synonym("house", "home")

# Import depuis des sources externes
manager.import_yaml("synonyms.yaml")   # Nécessite PyYAML
manager.import_json("synonyms.json")

# Export
manager.export_json("output.json")
```

### Flux de Travail d'Import/Export

```python
# Import depuis YAML
manager = SynonymManager()
manager.import_yaml("my_synonyms.yaml")

# Export vers JSON (ex: pour migration ou sauvegarde)
manager.export_json("backup.json")
```

## Synonymes Linguistiques Prédéfinis

Le dictionnaire `LANG_SYNONYMS` contient des mappings de synonymes de démarrage pour cinq langues :

```python
from whoosh_modern.linguistics.synonyms import LANG_SYNONYMS

# Langues disponibles : fr, en, de, es, it
french_syns = LANG_SYNONYMS["fr"]
print(french_syns["voiture"])  # ['automobile', 'véhicule']

english_syns = LANG_SYNONYMS["en"]
print(english_syns["car"])  # ['automobile', 'vehicle']

# Initialiser un SynonymManager avec une langue
manager = SynonymManager(LANG_SYNONYMS["fr"])
```

| Langue   | Code | Exemple                               |
|----------|------|---------------------------------------|
| Français | `fr` | `"voiture": ["automobile", "véhicule"]` |
| Anglais  | `en` | `"car": ["automobile", "vehicle"]`    |
| Allemand | `de` | `"auto": ["wagen", "fahrzeug"]`       |
| Espagnol | `es` | `"coche": ["automóvil", "vehículo"]`  |
| Italien  | `it` | `"auto": ["automobile", "veicolo"]`   |

> **Note** : Ce sont des dictionnaires de démarrage minimaux pour la démonstration et les tests. Les déploiements de production devraient charger depuis des sources élaborées ou spécifiques au domaine.

## SynonymExpansionMiddleware

Intègre l'expansion de synonymes dans le pipeline de middleware. Elle étend à la fois les requêtes de recherche et les champs de documents indexés :

```python
from whoosh_modern.linguistics.synonyms import (
    SynonymManager,
    SynonymExpansionMiddleware,
)

# Créer un gestionnaire avec vos synonymes
manager = SynonymManager({
    "car": ["automobile", "vehicle"],
    "house": ["home", "residence"],
})

# Créer le middleware
middleware = SynonymExpansionMiddleware(manager)

# L'enregistrer auprès du PluginManager ou MiddlewareChain
from whoosh.plugins.manager import PluginManager
PluginManager._default.register_middleware("synonym", middleware)
```

### Fonctionnement

- **`before_search`** : Étend `context.query` en ajoutant les synonymes de chaque token
- **`before_index`** : Étend les valeurs de type chaîne dans `context.document` en ajoutant les synonymes

```python
# Avant : query = "car"
# Après :  query = "car automobile vehicle"

# Avant : document = {"title": "house for sale"}
# Après :  document = {"title": "house for sale home residence"}
```

## Analyseurs Linguistiques Spécifiques

Situés dans `whoosh_modern.linguistics.stemmers`, ces analyseurs combinent tokenisation, stemme et suppression des mots vides :

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Chaque analyseur est appelable et retourne une liste de tokens
analyzer = EnglishAnalyzer()
tokens = analyzer("The running cats")
# tokens sont stemmés: ["run", "cat"] (mots vides supprimés)
```

## Voir Aussi

- [Guide Stemmers](stemming-sprint-d.md) — Fournisseurs de stemmers et analyseurs linguistiques
- [Guide Middleware](middleware-sprint-c.md) — Intégration du pipeline de middleware
- [API: Linguistique](../api/modern.md) — Référence complète de l'API


## DOCUMENT (FR): Middleware Sprint C

# Middleware & Pipeline de Plugins

Module: `whoosh.middleware`, `whoosh.middleware.chain`, `whoosh.middleware.context`, `whoosh_modern.middleware`
Version: 2.0.0

Le pipeline de middleware permet d'intercepter et de modifier les opérations d'indexation et de recherche. C'est le mécanisme d'extension principal pour les préoccupations transverses comme la journalisation, la mise en cache, les métriques, la réécriture de requêtes et la sécurité. Le middleware peut provenir à la fois du package core `whoosh.middleware` et des plugins chargés via le `PluginManager`.

## Vue d'ensemble de l'architecture

```text
Writer/Searcher  ───►  MiddlewareChain
                           ├── Middleware 1 (hook before)
                           ├── Middleware 2 (hook before)
                           ├── ─── opération core ───
                           ├── Middleware 2 (hook after, inverse)
                           └── Middleware 1 (hook after, inverse)
```

- Les **hooks `before_*`** s'exécutent dans l'ordre d'enregistrement
- Les **hooks `after_*`** s'exécutent dans l'ordre inverse (comme une pile / oignon)
- Si un hook lève `StopOperation`, le pipeline s'arrête gracieusement
- Si `fail_open=False` (défaut), les exceptions se propagent immédiatement

## Classes de Base du Middleware

### Middleware (Classe de Base)

Localisée dans `whoosh.middleware.base`. Les sous-classes implémentent les hooks du cycle de vie :

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MyMiddleware(Middleware):
    def startup(self, context: MiddlewareContext) -> None:
        """Appelé une fois quand le middleware est initialisé."""
        pass

    def shutdown(self, context: MiddlewareContext) -> None:
        """Appelé une fois quand le middleware est détruit."""
        pass

    def before_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé avant qu'un document soit indexé. Modifier context.document."""
        return context

    def after_index(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé après qu'un document a été indexé."""
        return context

    def before_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé avant la suppression d'un document."""
        return context

    def after_delete(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé après la suppression d'un document."""
        return context

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé avant l'exécution d'une requête. Modifier context.query."""
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        """Appelé après le retour des résultats. Accéder à context.results."""
        return context

    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        """Appelé quand une exception survient. Re-raise par défaut."""
        raise exc

    def on_commit(self, context: MiddlewareContext) -> None:
        """Appelé après une opération de commit."""
        pass
```

### MiddlewareContext

Localisé dans `whoosh.middleware.context`. L'objet contexte passé à chaque hook :

```python
class MiddlewareContext:
    def __init__(self, operation: str) -> None:
        self.operation: str           # ex: "add_document", "search"
        self.index: Any = None        # L'instance Index
        self.backend: Any = None       # Le backend de stockage
        self.writer: Any = None        # L'IndexWriter (si applicable)
        self.searcher: Any = None      # Le Searcher (si applicable)
        self.document: dict[str, Any] | None  # Document à indexer
        self.query: str = ""           # La chaîne de requête de recherche
        self.collector: Any = None     # Le collecteur (si applicable)
        self.results: Any = None       # Résultats de recherche
        self.labels: dict[str, Any] = {}    # Paires clé-valeur arbitraires
        self.metadata: dict[str, Any] = {} # Métadonnées par requête
```

Utilisez `context.copy()` pour créer une copie superficielle si vous devez préserver l'état.

### MiddlewareChain

Localisé dans `whoosh.middleware.chain`. Ordonnance l'exécution des middleware :

```python
from whoosh.middleware.chain import MiddlewareChain
from whoosh.middleware.context import MiddlewareContext

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware(),
])

# Hooks before (dans l'ordre)
context = MiddlewareContext("search")
context.query = "hello world"
context = chain.run_before("before_search", context)

# ... opération de recherche core ...

# Hooks after (dans l'ordre inverse)
context = chain.run_after("after_search", context)
print(context.results)
```

**Support asynchrone** : Utilisez `async_run_before()`, `async_run_after()`, `async_run_on_error()` et `run_hook()` pour un middleware asynchrone.

### MiddlewareRegistry

Localisé dans `whoosh.middleware.registry`. Un registre au niveau de la classe pour les middleware nommés :

```python
from whoosh.middleware.registry import MiddlewareRegistry

MiddlewareRegistry.register("my_mw", MyMiddleware(), owner="my_plugin")
mw = MiddlewareRegistry.get("my_mw")
MiddlewareRegistry.unregister("my_mw")
print(MiddlewareRegistry.list_all())  # ['my_mw', ...]
```

## Intégration du Middleware

### Wrappers: MiddlewareWriter & MiddlewareSearcher

Localisés dans `whoosh.middleware.wrappers`. Ces wrappers enveloppent le writer/searcher core pour exécuter automatiquement les hooks de middleware :

```python
from whoosh.middleware.wrappers import MiddlewareWriter, MiddlewareSearcher
from whoosh.middleware.chain import MiddlewareChain

chain = MiddlewareChain([MetricsMiddleware(), CacheMiddleware()])

# Envelopper un writer
with MiddlewareWriter(ix.writer(), chain) as writer:
    writer.add_document(title="Hello", content="World")

# Envelopper un searcher
with MiddlewareSearcher(ix.searcher(), chain) as searcher:
    results = searcher.search(query)
```

### Assistants d'Intégration

Localisés dans `whoosh.middleware.integration` :

```python
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

# Charge automatiquement le middleware depuis PluginManager si chain non fournie
writer = apply_middleware_to_writer(ix.writer())
searcher = apply_middleware_to_searcher(ix.searcher())
```

## Middleware Intégrés

### Middleware Core (`whoosh.middleware.base`)

| Classe                  | Hooks              | Description                              |
|------------------------|--------------------|------------------------------------------|
| `CompressionMiddleware` | `before_index`    | Marque les documents avec `_compressed = True` |
| `EncryptionMiddleware`  | `before_index`    | Marque les documents avec `_encrypted = True`  |
| `MetricsMiddleware`     | `after_index`, `after_search` | Compte les documents indexés et les recherches |
| `CacheMiddleware`       | `before_search`, `after_search` | Mise en cache en mémoire des résultats |

### Observabilité (`whoosh.middleware.metrics`)

`PrometheusMiddleware` — exporte des métriques vers Prometheus (nécessite `prometheus-client`) :

```python
from whoosh.middleware.metrics import PrometheusMiddleware

# Nécessite: pip install whoosh-ng[metrics]
prom = PrometheusMiddleware()
# Exporte: whoosh_searches_total, whoosh_documents_indexed_total, whoosh_search_duration_seconds
```

### Middleware Moderne (`whoosh_modern.middleware`)

#### Pipeline de Résilience (`whoosh_modern.middleware.pipeline`)

Ces middleware utilisent un **API de type wrapper** (pattern décorateur) plutôt que des hooks :

| Classe                  | Description                              |
|------------------------|------------------------------------------|
| `RetryMiddleware`      | Réessaie les opérations échouées avec backoff exponentiel |
| `LoggingMiddleware`    | Journalise le temps d'exécution et les erreurs |
| `CacheMiddleware`      | Met en cache les résultats d'opérations (éviction LRU) |
| `MiddlewarePipeline`   | Enchaîne plusieurs middleware de type wrapper |

```python
from whoosh_modern.middleware import MiddlewarePipeline, RetryMiddleware, LoggingMiddleware

pipeline = MiddlewarePipeline(
    LoggingMiddleware(),
    RetryMiddleware(attempts=3, backoff="exponential", jitter=True),
)

result = pipeline.execute(lambda: my_index_operation())
```

#### Middleware de Stockage (`whoosh_modern.middleware.storage`)

| Classe                  | Description                              |
|------------------------|------------------------------------------|
| `StorageMiddleware`    | Redirige la persistance vers des fournisseurs de stockage pluginables |
| `FileStorageProvider`  | Stockage sur système de fichiers local   |
| `SQLiteStorageProvider`| Stockage blob SQLite                    |
| `S3StorageProvider`    | Stockage cloud S3 / S3-compatible       |

```python
from whoosh_modern.middleware.storage import StorageMiddleware, FileStorageProvider

storage = StorageMiddleware(FileStorageProvider("/data/index"), name="primary")
```

#### Middleware de Recherche (`whoosh_modern.middleware.search`)

| Classe                      | Description                              |
|----------------------------|------------------------------------------|
| `QueryRewriteMiddleware`   | Réécrit `context.query` avant la recherche |
| `RankingMiddleware`        | Re-classe `context.results` après la recherche |

```python
from whoosh_modern.middleware.search import QueryRewriteMiddleware

def add_synonyms(query: str) -> str:
    # Étendre la requête avec des synonymes avant l'exécution
    return query + " " + get_synonyms(query)

rewriter = QueryRewriteMiddleware(rewriter=add_synonyms)
```

#### Middleware d'Analyse (`whoosh_modern.middleware.analyzer`)

| Classe                  | Description                              |
|------------------------|------------------------------------------|
| `StemmingMiddleware`   | Applique un stemmer aux champs de document et à la requête |
| `SynonymMiddleware`    | Étend le texte avec des synonymes (placeholder) |

## Créer un Middleware Personnalisé

### Middleware Basé sur des Hooks

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class RequestLoggingMiddleware(Middleware):
    """Journaliser toutes les recherches avec le timing."""

    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        import time
        context.metadata["_start_time"] = time.time()
        logger.info(f"[RECHERCHE] Requête: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        elapsed = time.time() - context.metadata.get("_start_time", time.time())
        result_count = len(context.results) if context.results is not None else 0
        logger.info(f"[RÉSULTATS] Trouvé {result_count} résultats en {elapsed:.3f}s")
        return context
```

### Middleware de Type Wrapper

```python
from whoosh_modern.middleware.pipeline import Middleware as WrapMiddleware

class RetryMiddleware(WrapMiddleware):
    """Réessaie les opérations échouées avec backoff."""

    def __init__(self, attempts: int = 3) -> None:
        self._attempts = attempts

    def wrap(self, operation):
        def wrapped(*args, **kwargs):
            last_exc = None
            for attempt in range(self._attempts):
                try:
                    return operation(*args, **kwargs)
                except Exception as e:
                    last_exc = e
                    if attempt < self._attempts - 1:
                        time.sleep(2 ** attempt)
            raise last_exc
        return wrapped
```

### Middleware avec Intégration de Plugin

Enregistrer du middleware via un plugin pour qu'il soit automatiquement découvert :

```python
from whoosh.plugins.manager import Plugin

class LoggingPlugin(Plugin):
    name = "logging"
    version = "1.0.0"
    middleware = ["whoosh_modern.middleware.pipeline.LoggingMiddleware"]

    def register(self, manager):
        manager.register_middleware(
            "logging",
            LoggingMiddleware(),
        )
```

## Gestion des Erreurs

### StopOperation

Abandonner une opération de pipeline gracieusement :

```python
from whoosh.middleware.exceptions import StopOperation

class RateLimitMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        if not rate_limiter.allow(context):
            raise StopOperation("Limite de taux dépassée")
        return context
```

### Comportement fail_open

```python
class ResilientMiddleware(Middleware):
    def on_error(self, context: MiddlewareContext, exc: Exception) -> None:
        try:
            send_to_analytics(context.results)
        except Exception:
            # Journaliser mais ne pas échouer la recherche
            logger.warning("Analytics failed", exc_info=True)
        # La chaîne de middleware continue
```

## Découverte de Middleware depuis les Plugins

Quand `PluginManager.load_plugins()` est appelé, tous les plugins qui déclarent une liste `middleware` auront leurs classes de middleware importées et instanciées. La méthode `get_middleware_chain()` construit une `MiddlewareChain` à partir de tous les middleware enregistrés :

```python
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Découvre les plugins et leurs middleware

manager = PluginManager._default
chain = manager.get_middleware_chain()
# chain est une MiddlewareChain prête à l'emploi
```

## Bonnes Pratiques

1. **Sans état** : Utilisez `context.metadata` pour les données par requête, pas les attributs d'instance
2. **Hooks légers** : Gardez les hooks `before_*` et `after_*` rapides ; utilisez async pour les E/S
3. **L'ordre compte** : Placez le cache avant les métriques, l'authentification avant le routage
4. **Fail fast** : N'utilisez `fail_open=True` que pour les middleware non critiques
5. **Testabilité** : Mockez le `MiddlewareContext` pour tester le middleware indépendamment
6. **Nettoyage** : Implémentez `shutdown()` pour les ressources comme les connexions et les minuteurs

## Voir Aussi

- [Guide Système de Plugins](plugins-sprint-c.md) — Enregistrement et entry points des plugins
- [Exemples: Middleware](../examples/middleware.md) — Patterns de middleware pratiques
- [API: Middleware](../api/middleware.md) — Référence complète de l'API
- [API: Middleware Pipeline (moderne)](../api/modern.md) — Extensions middleware modernes


## DOCUMENT (FR): Middleware

# Middleware

Le pipeline de middleware permet d'intercepter et modifier les opérations d'indexation et de recherche. C'est le mécanisme d'extension principal pour les préoccupations transverses comme le logging, le cache, les métriques et la sécurité.

## Concepts de base

Un middleware est une classe qui implémente des hooks dans le cycle de vie :

```python
from whoosh.middleware.base import Middleware
from whoosh.middleware.context import MiddlewareContext

class MonMiddleware(Middleware):
    def before_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Modifier context.query ou context.metadata
        return context

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        # Accéder à context.results
        return context
```

## Hooks disponibles

| Hook | Quand | Utilisations courantes |
|------|------|------------------------|
| `startup(context)` | Initialisation | Ouvrir connexions, remplir caches |
| `shutdown(context)` | Nettoyage | Fermer connexions, flush buffers |
| `before_index(context)` | Avant indexation | Validation, enrichissement, flags compression |
| `after_index(context)` | Après indexation | Métriques, événements, invalidation cache |
| `before_delete(context)` | Avant suppression | Journalisation audit, contrôle d'accès |
| `after_delete(context)` | Après suppression | Métriques, invalidation cache |
| `before_search(context)` | Avant recherche | Réécriture de requête, cache, auth |
| `after_search(context)` | Après résultats | Logging, métriques, modification résultats |
| `on_error(context, exc)` | Sur exception | Gestion d'erreur, fallbacks |
| `on_commit(context)` | Après commit | Métriques, notifications |

## Classes intégrées

### MetricsMiddleware

```python
from whoosh.middleware import MetricsMiddleware

metrics = MetricsMiddleware()
# Après opérations:
stats = metrics.get_metrics()
# Retourne: {"documents_indexed": N, "searches_executed": N}
```

### CacheMiddleware

```python
from whoosh.middleware import CacheMiddleware

cache = CacheMiddleware()
cached = cache.get_cached("requête utilisateur")
cache.set_cached("requête utilisateur", results)
```

## MiddlewareChain

```python
from whoosh.middleware import MiddlewareChain

chain = MiddlewareChain([
    MetricsMiddleware(),
    CacheMiddleware()
])

# Exécuter un hook before
context = MiddlewareContext("search")
context.query = "test"
context = chain.run_before("before_search", context)

# ... opération core ...

# Exécuter un hook after
context = chain.run_after("after_search", context)
```

## Intégration

### Avec Writer

```python
from whoosh.middleware.integration import apply_middleware_to_writer

writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)

with writer:
    writer.add_document(title="Bonjour", content="Monde")
```

### Avec Searcher

```python
from whoosh.middleware.integration import apply_middleware_to_searcher

searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)
results = searcher.search("query")
```

## Exemple: middleware personnalisé

```python
class RequestLoggingMiddleware(Middleware):
    """Journaliser toutes les recherches."""

    def before_search(self, context: MiddlewareContext):
        context.metadata["request_id"] = generate_request_id()
        logger.info(f"Recherche: {context.query}")
        return context

    def after_search(self, context: MiddlewareContext):
        logger.info(f"Trouvé: {len(context.results)} résultats")
        return context

class RateLimitMiddleware(Middleware):
    """Abandonner les recherches dépassant la limite."""

    def before_search(self, context: MiddlewareContext):
        if not rate_limiter.allow(context):
            raise StopOperation("Limite de taux dépassée")
        return context
```

## Gestion des erreurs

```python
class ResilientMiddleware(Middleware):
    """Continuer malgré les erreurs non critiques."""

    def after_search(self, context: MiddlewareContext) -> MiddlewareContext:
        try:
            send_to_analytics(context.results)
        except Exception:
            logger.warning("Analytics failed", exc_info=True)
        return context
```

## Bonnes pratiques

1. **Sans état**: Utilisez `context.metadata` pour les données par requête
2. **Fail fast**: Utilisez `fail_open=True` uniquement pour middleware non critique
3. **L'ordre compte**: Placez le cache avant les métriques, l'auth avant le routage
4. **Performance**: Gardez les hooks légers; utilisez async pour les I/O
5. **Testabilité**: Mockez le contexte pour tester le middleware isolément

## Middleware Moderne (Whoosh-NG 2.0)

Whoosh-NG 2.0 ajoute un package middleware moderne (`whoosh_modern.middleware`) avec un middleware de résilience de type wrapper (réessaissance, cache, journalisation) et un middleware basé sur des hooks pour le stockage, la recherche et l'analyse. Pour plus de détails sur l'architecture moderne du middleware, l'intégration de plugins et le déploiement, consultez le [Guide Middleware & Pipeline de Plugins](middleware-sprint-c.md).


## DOCUMENT (FR): Modern Indexing

# Indexation Moderne

Whoosh-NG fournit une couche d'indexation optimisée dans `whoosh_modern.indexing` pour l'absorption de grands volumes de documents. Ces utilitaires encapsulent l'écrivain Whoosh de base sans modifier les internals de la bibliothèque.

## BatchIndexWriter

`BatchIndexWriter` encapsule un écrivain Whoosh avec les optimisations suivantes pour le traitement par lots de grands ensembles de données.

### Optimisations clés

- Pré-calcule les noms de champs du schéma pour un filtrage rapide (O(1) par champ)
- Ignore les champs non présents dans le schéma (évitant le surcoût par document)
- Utilise `multisegment=True` pour reporter les fusions pendant l'indexation
- Supporte des commits par lots configurables pour réduire la pression I/O
- Accepte un callback pour les hooks post-commit

### Utilisation de base

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh import index

ix = index.open_dir("indexdir")

writer = BatchIndexWriter(ix, batch_size=5000, commit_every=10)

for batch in source.stream_batches(batch_size=5000):
    writer.add_batch(batch)

writer.close()
```

### Context Manager

```python
with BatchIndexWriter(ix, batch_size=10000) as writer:
    for doc in documents:
        writer.add_document(doc)
```

### Paramètres

| Paramètre | Par défaut | Description |
|-----------|---------|-------------|
| `batch_size` | 5000 | Nombre de documents par lot |
| `limitmb` | 512 | Limite mémoire pour l'écrivain (Mo) |
| `commit_every` | None | Commit après N lots (None = pas de commit auto) |
| `multisegment` | True | Utiliser le mode multisegment |
| `callback` | None | Callback invoqué après chaque commit |
| `**writer_kwargs` | None | Arguments supplémentaires pour `index.writer()` |

### Avec Commit Profiler

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import CommitProfilerV2

profiler = CommitProfilerV2()
with BatchIndexWriter(ix, batch_size=5000, commit_every=5, commit_profiler=profiler) as writer:
    for batch in source.stream_batches(batch_size=5000):
        writer.add_batch(batch)

print(profiler.report())
```

---

## AnalyzerCache

`AnalyzerCache` fournit un cache LRU pour les résultats d'analyse, évitant les travaux d'analyse redondants sur des valeurs de champ répétées.

### Utilisation de base

```python
from whoosh_modern.indexing import BatchIndexWriter
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    cache_key = f"title:{doc['title']}"
    tokens = cache.get(cache_key)
    if tokens is None:
        tokens = list(analyzer(doc['title']))
        cache.put(cache_key, tokens)
```

### Avec get_or_compute

```python
from whoosh_modern.profiling import AnalyzerCache

cache = AnalyzerCache(maxsize=50000)
analyzer = StandardAnalyzer()

for doc in docs:
    tokens = cache.get_or_compute(
        f"title:{doc['title']}",
        lambda: list(analyzer(doc['title']))
    )
```

### Statistiques du cache

```python
print(f"Hit rate: {cache.hit_rate:.1%}")
print(f"Size: {cache.size}/{cache.maxsize}")
print(cache.report())
```

### Dimensionnement à partir de données de profiling

```python
from whoosh_modern.profiling import AnalyzerCache, CacheAnalyzer

analyzer = CacheAnalyzer()
analysis = analyzer.analyze(source.iter_documents())

cache = AnalyzerCache.from_profiling(analysis.to_dict())
```

---

## FieldAnalyzerCache

`FieldAnalyzerCache` encapsule un analyseur et met en cache les résultats par champ.

### Utilisation de base

```python
from whoosh_modern.profiling import FieldAnalyzerCache

field_cache = FieldAnalyzerCache(
    analyzer=StandardAnalyzer(),
    fields=["Country", "City"],
    cache_size=50000,
)

for doc in docs:
    for field in ["Country", "City"]:
        tokens = field_cache.analyze(field, doc[field])
```

### Invalidation du cache

```python
# Invalider une entrée spécifique
field_cache.invalidate("Country", "USA")

# Vider le cache entier
field_cache.clear()
```

### Statistiques du cache

```python
print(f"Hit rate: {field_cache.hit_rate:.1%}")
print(field_cache.report())
```

---

## Sources de données disponibles

| Classe | Type | Dépendances |
|-------|------|-------------|
| `SQLSource` | Bases SQL | `sqlite3` (stdlib) |
| `SQLAlchemySource` | SQLAlchemy | `sqlalchemy` |
| `RESTSource` | API REST | aucune (stdlib `urllib`) |
| `GraphQLSource` | API GraphQL | aucune (stdlib `urllib`) |
| `FastCSVSource` | Fichiers CSV | aucune |
| `JSONSource` | JSON/JSONL | aucune |
| `ParquetSource` | Parquet | `pyarrow` ou `pandas` |
| `PandasSource` | DataFrames pandas | `pandas` |
| `PolarsSource` | DataFrames Polars | `polars` |
| `PeeweeSource` | ORM Peewee | `peewee` |
| `TortoiseSource` | ORM Tortoise | `tortoise-orm` |
| `PydanticSource` | Modèles Pydantic | `pydantic` |


## DOCUMENT (FR): Monitoring

# Monitoring

Whoosh-NG inclut des hooks d'observabilité intégrés et un plugin Prometheus pour le monitoring en production.

## Métriques intégrées

### MetricsMiddleware

```python
from whoosh.middleware import MetricsMiddleware, MiddlewareChain
from whoosh.middleware.integration import apply_middleware_to_writer, apply_middleware_to_searcher

chain = MiddlewareChain([MetricsMiddleware()])

writer = apply_middleware_to_writer(ix.writer(), chain.middlewares)
searcher = apply_middleware_to_searcher(ix.searcher(), chain.middlewares)

# Obtenir les métriques
metrics = chain.get_metrics()
```

## Plugin Prometheus

### Installation

```bash
pip install whoosh-ng[metrics]
```

### Métriques exposées

| Métrique | Type | Description |
|----------|------|-------------|
| `whoosh_documents_indexed_total` | Counter | Total documents indexés |
| `whoosh_searches_executed_total` | Counter | Total recherches exécutées |
| `whoosh_indexing_duration_seconds` | Histogram | Temps d'indexation |
| `whoosh_search_duration_seconds` | Histogram | Temps de recherche |
| `whoosh_index_size_bytes` | Gauge | Taille actuelle de l'index |
| `whoosh_cache_hits_total` | Counter | Cache hits |
| `whoosh_cache_misses_total` | Counter | Cache misses |

## Event Bus pour monitoring

```python
from whoosh.event_bus import EventBus, DocumentIndexed, SearchExecuted

bus = EventBus()

@bus.subscribe
def on_indexed(event: DocumentIndexed):
    stats.increment("documents.indexed")

@bus.subscribe
def on_searched(event: SearchExecuted):
    stats.timing("search.duration", event.duration)
```

## Bonnes pratiques

1. **Ajoutez MetricsMiddleware tôt**: Incluez-le dans votre chaîne de base
2. **Exportez via Prometheus**: En production, exposez l'endpoint `/metrics`
3. **Endpoint health**: Utilisez `/health` pour les health checks load balancer
4. **Logging structuré**: Corrélez les événements search/index avec des request IDs
5. **Alerting**: Définissez des alertes sur les taux d'erreur et les latences


## DOCUMENT (FR): Ngrams

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# N-grams

This guide covers N-gram tokenization and analysis for substring matching,
prefix queries, and autocomplete functionality.

## What Are N-grams?

An N-gram is a contiguous sequence of N characters (or tokens) from a string.
For example, the 2-grams of "hello" are: "he", "el", "ll", "lo".

N-gram analysis is useful for:
- Substring search (finding "ell" within "hello")
- Autocomplete / typeahead suggestions
- Fuzzy matching without edit distance computation

## NgramTokenizer

The `NgramTokenizer` splits text into character-level N-grams:

```python
from whoosh.analysis import NgramTokenizer
from whoosh import fields

tokenizer = NgramTokenizer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=tokenizer),
)
```

### NgramTokenizer Parameters

- `minsize`: Minimum N-gram length (default `2`)
- `maxsize`: Maximum N-gram length (default `4`)

With the example above, the text "hello" produces these 2-4-grams:
`he, hel, hell, el, ell, ello, l, ll, llo, l, lo, o`

## NgramFilter

The `NgramFilter` creates word-level N-grams from tokenized text:

```python
from whoosh.analysis import RegexTokenizer, NgramFilter

analyzer = RegexTokenizer() | NgramFilter(maxsize=2)
```

This produces word-level grams: for "hello world", it produces ("hello",)
and ("hello", "world").

## NgramWordAnalyzer

A convenience analyzer that combines `NgramTokenizer` with `LowercaseFilter`:

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

## Use Cases

### Substring Search

With N-gram analysis, you can match substrings:

```python
from whoosh.qparser import QueryParser

# Index text with N-grams
# Searching for "ell" matches "hello" because "ell" is a substring
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("ell")
results = searcher.search(q)
```

### Prefix Matching

Set `maxsize` equal to a large value to effectively create prefix N-grams:

```python
from whoosh.analysis import NgramWordAnalyzer

# Create N-grams where each word's prefixes become searchable tokens
# e.g., "hello" -> "h", "he", "hel", "hell", "hello"
analyzer = NgramWordAnalyzer(minsize=1, maxsize=10)
```

### Autocomplete

N-gram indexes are commonly used for autocomplete/typeahead. For more
advanced autocomplete with edge n-grams, consider:

```python
from whoosh.analysis import RegexTokenizer, NgramFilter
from whoosh.query import Prefix

# Index with standard tokenization, then use Prefix queries for autocomplete
analyzer = RegexTokenizer()
schema = fields.Schema(
    title=fields.TEXT(stored=True, analyzer=analyzer),
    content=fields.TEXT(analyzer=analyzer),
)

# For autocomplete, query with Prefix
from whoosh.qparser import QueryParser
qp = QueryParser("title", schema=ix.schema)
q = Prefix("title", "hel")  # Find documents where title starts with "hel"
```

## Comparison with Edge N-grams

Some search engines support "edge n-grams" (only generating N-grams from the
beginning of words). This is more space-efficient for autocomplete:

- Full N-grams: "hello" → "he", "el", "ll", "lo", "hel", "ell", ...
- Edge N-grams: "hello" → "h", "he", "hel", "hell", "hello"

Whoosh's `NgramTokenizer` generates full (bidirectional) N-grams. For
edge-ngram-like behavior, use the `minsize` and `maxsize` parameters
strategically, or use `Prefix` queries against a standard tokenized field.

## Performance Considerations

- N-gram indexes are typically much larger than standard indexes
- Each original token produces multiple N-gram tokens, increasing index size
- Choose `minsize` and `maxsize` carefully to balance search quality against
  index size
- For autocomplete, consider using `Prefix` queries with a
  non-N-gram field for better performance


## DOCUMENT (FR): Performance

# Performance et Benchmarking

Whoosh-NG inclut un ensemble complet d'outils de benchmarking dans `whoosh_modern.profiling` pour mesurer et comparer les performances des analyseurs. Ce guide explique comment utiliser ces outils et documente les optimisations livrées dans la version 2.0.0.

## Démarrage rapide

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

# Générer des datasets synthétiques pour des benchmarks cohérents
gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Exécuter le benchmark du tokeniseur (P5.1)
run_p5_1(datasets)

# Exécuter le benchmark de création de tokens (P5.2)
run_p5_2(token_count=100_000)

# Exécuter le benchmark du stemmer
bench = StemmerBenchmark()
bench.run(gen.generate_dataset("A", 5000))
print(bench.report())
```

## Outils de benchmarking

### SyntheticDatasetGenerator

Génère des datasets de texte déterministes de complexité variable :

```python
from whoosh_modern.profiling.synthetic_datasets import SyntheticDatasetGenerator

gen = SyntheticDatasetGenerator(seed=42)
datasets = gen.generate_all(count=5000)

# Dataset A: 2 tokens/doc (court)
# Dataset B: 50 tokens/doc (moyen)
# Dataset C: 500 tokens/doc (grand)
# Dataset D: 1200 tokens/doc (très grand)
for name, texts in datasets.items():
    print(f"{name}: {len(texts)} documents")
```

### P5.1 : Benchmark du RegexTokenizer

Compare différentes implémentations de tokeniseur :

```python
from whoosh_modern.profiling.benchmarks.regex_tokenizer import run_p5_1

results = run_p5_1(datasets)

# Compare :
# - Current Regex (whoosh par défaut)
# - Compiled Global regex
# - Manual Python tokenizer
# - C extension (re2, si disponible)
```

### P5.2 : Benchmark d'optimisation des tokens

Compare différentes implémentations d'objets Token :

```python
from whoosh_modern.profiling.benchmarks.token_optimization import run_p5_2

# Compare :
# - Current Token (dict-based)
# - __slots__ optimization
# - namedtuple
# - dataclass(slots=True)
results = run_p5_2(token_count=100_000)
```

### StemmerBenchmark

Compare les backends de stemming :

```python
from whoosh_modern.profiling.stemmer_benchmark import StemmerBenchmark

bench = StemmerBenchmark()
bench.run(texts, warmup=True)
print(bench.report())
# Résultats :
# Stemmer         Tokens/s        Time (s)    Tokens
# ------------------------------------------------------
# StemFilter      1,004,172       0.1503      150,983
# PyStemmer       ~2,100,000+     0.0719+     150,983
```

## Optimisations de performance

### Résumé des gains 2.0.0

| Optimisation | Composant | Gain mesurable |
|---|---|---|
| `__slots__` sur Token | `whoosh.analysis.acore` | +35% création de tokens |
| Regex globale compilée | `RegexTokenizer` | +50% débit regex |
| Postings compactés (1 posting) | `W3TermInfo` / `W3PostingsWriter` | +35% vitesse de commit |
| Postings compactés (2-8 postings) | `W3TermInfo` / `W3PostingsWriter` | +35% vitesse de commit |
| Cache de champ dans add_postings | `whoosh.codec.base` | -93% appels write_block |
| Encodage varint des positions | `whoosh.formats` | réduction de l'overhead par terme |
| Cache de stemmer | `whoosh.analysis.morph` | taux de hit 96,5 %, 4,12x sur champs répétitifs |
| Cache d'analyseur | `whoosh_modern.profiling.analyzer_cache` | 4,12x sur champs hautement répétitifs |
| Écrivain par lots | `whoosh_modern.indexing.batch_writer` | lots filtrés optimisés |
| Optimisation setdefault stopwords | `whoosh.formats` | réduction de l'overhead dict |

### Résultats de benchmark : 20 000 documents (`customers_csv`)

```
Avant :
  commit total      : 18,653 s
  analyzing         : 8,641 s  (51,5 %)
  committing        : 10,012 s  (27,1 %)
  write_postings    : 6,5 s
  write_block calls : ~72 612

Après :
  commit total      : 6,806 s  (-63,5 %)
  analyzing         : ~2,7 s   (-68 %)
  committing        : 6,806 s  (-32 %)
  write_postings    : 6,5 s -> allouations réduites
  write_block calls : 7 565   (-93 %)
  débit             : 1 275 docs/s
```

### Résultats de benchmark : Backends de stemming (1,5M de tokens)

| Stemmer | Débit | Relatif |
|---|---|---|
| StemFilter (interne) | 1 004 172 tokens/s | 1,0x |
| PyStemmer | ~2 100 000 tokens/s | ~2,1x |

### Résultats de benchmark : Tokeniseur regex

| Tokeniseur | Débit | Relatif |
|---|---|---|
| Current regex | ~1 000 000 tokens/s | 1,0x |
| Compiled global | ~2 300 000 tokens/s | 2,3x |

### Résultats de benchmark : Objet Token

| Implémentation | Tokens/s | Relatif |
|---|---|---|
| Current (dict) | 1 000 000 | 1,0x |
| `__slots__` | ~1 350 000 | 1,35x |

## Outils de profiling

### IndexingPipelineProfiler

Profile le pipeline d'indexation complet :

```python
from whoosh_modern.profiling.indexing_pipeline_profiler import IndexingPipelineProfiler

profiler = IndexingPipelineProfiler()
for doc in documents:
    profiler.before_tokenize(doc, analyzer)
    analyzer(doc)
    profiler.after_tokenize()

report = profiler.report()
print(report)
```

### CommitProfiler

Profile les performances de commit :

```python
from whoosh_modern.profiling.commit_profiler_v2 import CommitProfiler

profiler = CommitProfiler()
# ... indexer des documents ...
ix.commit()

report = profiler.report()
# Affiche : analyze, convert_fields, write_postings, flush, commit
```

### FieldIndexProfiler

Profile les coûts de conversion des champs :

```python
from whoosh_modern.profiling.field_index_profiler import FieldIndexProfiler

profiler = FieldIndexProfiler()
# ... indexer des documents ...
report = profiler.report()
```

### IndexQualityAnalyzer

Analyse les métriques de qualité de l'index :

```python
from whoosh_modern.profiling.index_quality_analyzer import IndexQualityAnalyzer

analyzer = IndexQualityAnalyzer(index_reader)
report = analyzer.analyze()
print(f"Termes singletons: {report['singleton_terms']}/{report['total_terms']} ({report['singleton_percent']}%)")
```

## Système de fournisseurs de stemmer

Whoosh-NG propose un système de fournisseurs de stemmer :

```python
from whoosh_modern.analysis import get_stemmer, StemmingAnalyzer, list_available_backends

# Vérifier les backends disponibles
print(list_available_backends())
# {'internal': 'available', 'pystemmer': 'not installed'}

# Utiliser la détection automatique (par défaut)
analyzer = StemmingAnalyzer(stemmer="auto")

# Stemmer interne explicite
analyzer = StemmingAnalyzer(stemmer="internal")

# PyStemmer (nécessite: pip install whoosh-ng[fast-stemming])
analyzer = StemmingAnalyzer(stemmer="pystemmer")
```

## Recommandations de performance

1. **Utilisez** `StemmingAnalyzer` de `whoosh_modern.analysis` pour une sélection automatique de PyStemmer
2. **Activez le cache de stemmer** pour le contenu répétitif (`cachesize=50000` par défaut)
3. **Minimisez les champs TEXT** — utilisez KEYWORD ou ID pour les champs à faible cardinalité
4. **Évitez les positions/chars stockés** sauf si la mise en évidence l'exige
5. **Utilisez l'indexation par lots** avec des segments plus grands pour de meilleurs débits
6. **Surveillez les termes singletons** — réduisez les termes rares via des listes de stopwords

## Exécution de la suite complète de benchmarks

```bash
cd whoosh-ng

# Exécuter tous les tests P5
uv run python -m pytest tests/test_regex_tokenizer_unicode.py tests/test_token_slots.py tests/test_stemmer_compatibility.py -v

# Suite de tests complète
uv run python -m pytest -q
```


## DOCUMENT (FR): Plugins Sprint C

# Système de Plugins

Module: `whoosh.plugins.manager`
Version: 2.0.0

L'architecture à plugins de Whoosh-NG permet aux paquets externes d'étendre le moteur d'indexation, de recherche et d'analyse de texte. Les plugins sont découverts via les [entry points](https://docs.python.org/3/library/importlib.metadata.html#entry-points) Python déclarés dans `pyproject.toml` et gérés par le `PluginManager`.

## Vue d'ensemble de l'architecture

```text
PluginManager (singleton)
    ├── load_plugins(group)          # Auto-découverte depuis entry points
    ├── register(plugin)             # Enregistrement manuel
    ├── enable(name) / disable(name) # Activer/désactiver le cycle de vie
    ├── get(name) / list_plugins()   # Inspection
    ├── get_middleware_chain()       # Construire MiddlewareChain depuis middleware des plugins
    ├── register_datasource()        # Enregistrer un fournisseur de source de données
    ├── register_vector_provider()   # Enregistrer un fournisseur de vecteurs
    ├── register_middleware()        # Enregistrer une instance de middleware
    ├── register_embedding()         # Enregistrer un fournisseur d'embeddings
    ├── register_analyzer()          # Enregistrer un analyseur nommé
    └── register_query_rewriter()    # Enregistrer un réécrivant de requête
```

## Classes de Base des Plugins

### Plugin (ABC)

La classe de base pour tous les plugins. Les sous-classes définissent les attributs de classe et implémentent `register()` :

```python
from whoosh.plugins.manager import Plugin, PluginMetadata

class MyPlugin(Plugin):
    name = "my_plugin"
    version = "1.0.0"

    def register(self, manager: PluginManager) -> None:
        """Appelé quand le plugin est chargé ; enregistrer les fournisseurs ici."""
        manager.register_middleware("my_module.MyMiddleware", MyMiddleware())

    def register_hooks(self) -> None:
        """Enregistrer les hooks d'événements (optionnel)."""
        from whoosh.hooks import hookimpl, register_hook

        @hookimpl
        def on_search(request, response):
            pass
        register_hook("on_search", hookimpl(on_search))
```

### AnalyzerPlugin

Pour les plugins qui fournissent des tokenizers/analyseurs personnalisés :

```python
from whoosh.plugins.manager import AnalyzerPlugin

class MyAnalyzerPlugin(AnalyzerPlugin):
    name = "my_analyzer"

    def register(self, manager):
        manager.register_analyzer("my_analyzer", MyTokenizer())
```

### QueryRewritePlugin

Pour les plugins qui transforment les requêtes avant l'exécution :

```python
from whoosh.plugins.manager import QueryRewritePlugin

class SynonymRewriterPlugin(QueryRewritePlugin):
    name = "synonym_rewriter"

    def rewrite(self, query, searcher):
        # Retourner la requête modifiée
        return query
```

## PluginMetadata

Un dataclass décrivant les métadonnées du plugin :

| Champ         | Type              | Description                            |
|---------------|-------------------|----------------------------------------|
| `name`        | `str`             | Nom unique du plugin                   |
| `version`     | `str`             | Version sémantique                     |
| `depends_on`  | `list[str]`       | Noms des plugins requis                |
| `priority`    | `int`             | Priorité d'ordre de chargement (plus haut = plus tard) |
| `middleware`  | `list[str]`       | Chemins pointés vers les classes de middleware |

## Groupes d'Entry Points

Le `PluginManager` découvre les plugins depuis ces groupes d'entry points standards :

| Groupe                      | Utilisation                          |
|-----------------------------|--------------------------------------|
| `whoosh.plugins`            | Plugins généraux                     |
| `whoosh.datasources`        | Fournisseurs de sources de données   |
| `whoosh.vector.providers`   | Fournisseurs de similarité vectorielle |
| `whoosh.middlewares`        | Classes de middleware                 |
| `whoosh.embeddings`         | Fournisseurs de modèles d'embedding  |
| `whoosh.language`           | Analyseurs linguistiques             |
| `whoosh.apps`               | Usines d'applications (FastAPI, admin, etc.) |

## Créer et Déployer un Plugin

### Étape 1 : Définir la Classe du Plugin

```python
# my_plugin/plugin.py
from whoosh.plugins.manager import Plugin
from whoosh.registry import VectorRegistry

class MyVectorPlugin(Plugin):
    name = "my_vector"
    version = "1.0.0"
    depends_on = []
    conflicts_with = []
    priority = 0
    middleware = []

    def register(self, manager):
        """Enregistrer un fournisseur de vecteurs dans le VectorRegistry."""
        provider = MyCustomVectorProvider()
        VectorRegistry.register("my_vector", provider, owner=self.name)

    def register_hooks(self):
        """Enregistrer les hooks optionnels (ex: on_search, on_index)."""
        pass
```

### Étape 2 : Déclarer l'Entry Point

Dans votre `pyproject.toml` :

```toml
[project]
name = "whoosh-ng-my-vector"
version = "1.0.0"
dependencies = ["whoosh-ng>=2.0"]

[project.entry-points."whoosh_ng.plugins"]
my_vector = "my_plugin.plugin:MyVectorPlugin"
```

### Étape 3 : Installer et Vérifier

```bash
pip install -e .
```

```python
# Vérifier que le plugin est bien enregstré
from whoosh.plugins.manager import PluginManager

PluginManager.load_plugins()  # Découvre tous les entry points

manager = PluginManager._default
print(manager.list_plugins())
# ['whoosh_autocomplete', 'whoosh_vector', ..., 'my_vector']

# Vérifier le registre
from whoosh.registry import VectorRegistry
print(VectorRegistry.list_keys())
# ['my_vector', 'numpy']
```

## Enregistrement Manuel (Sans Entry Point)

Pour les tests ou un usage programmatique :

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager()
manager.register(MyVectorPlugin())
manager.enable("my_vector")
```

## Cycle de Vie d'un Plugin

```
1. Entry point découvert  ───►  2. register() appelé  ───►  3. register_hooks()
   │                               │                            │
   └── load_plugins(group)          └── register provider/     └── register_hook()
                                      middleware/analyzer
```

### Activation / Désactivation

```python
from whoosh.plugins.manager import PluginManager

manager = PluginManager._default

manager.enable("my_vector")    # Activer un plugin
manager.disable("my_vector")   # Désactiver un plugin
print(manager.list_enabled())  # Seuls les plugins activés
```

### Validation de Version

```python
# Vérifie si un plugin respecte une version minimale
ok = manager.validate_version("my_vector", "1.0.0")
print(ok)  # True si la version du plugin >= 1.0.0
```

### Détection de Conflits

```python
# Vérifie si deux plugins entrent en conflit
if manager.detect_conflicts("plugin_a", "plugin_b"):
    print("Ces plugins ne peuvent pas être chargés ensemble")
```

## API du PluginManager

### `PluginManager.load_plugins(group=None)`

Charge tous les plugins depuis les groupes d'entry points. Si `group` est `None`, charge tous les groupes standards (`STANDARD_GROUPS`).

### `PluginManager.register(plugin)`

Enregistre une instance de plugin. Appelle `plugin.register(self)` et `plugin.register_hooks()`. Supporte les méthodes `register()` asynchrones via `asyncio`.

### `PluginManager.get_middleware_chain()`

Construit et retourne une `MiddlewareChain` à partir de tous les plugins qui déclarent une liste `middleware`. Les classes de middleware sont importées et instanciées par chemin pointé.

### Méthodes d'Enregistrement dans les Registres

| Méthode                       | Description                          |
|-------------------------------|--------------------------------------|
| `register_analyzer(name, analyzer)` | Enregistrer un analyseur nommé   |
| `register_datasource(name, datasource)` | Enregistrer une source de données  |
| `register_vector_provider(name, provider)` | Enregistrer un fournisseur de vecteurs |
| `register_middleware(name, middleware)` | Enregistrer une instance de middleware |
| `register_embedding(name, embedding)` | Enregistrer un fournisseur d'embeddings |
| `register_query_rewriter(plugin)` | Enregistrer un plugin réécrivant des requêtes |

### Méthodes de Recherche

| Méthode                       | Retourne                          |
|-------------------------------|----------------------------------|
| `get(name)`                   | Instance `Plugin`                |
| `list_plugins()`              | Noms de tous les plugins enregistrés |
| `list_enabled()`              | Noms des plugins activés         |
| `get_analyzer(name)`          | Analyseur callable               |
| `list_analyzers()`            | Noms des analyseurs enregistrés  |
| `list_datasources()`          | Noms des sources de données enregistrées |
| `list_vector_providers()`     | Noms des fournisseurs de vecteurs enregistrés |
| `list_middlewares()`          | Noms des middleware enregistrés  |
| `list_embeddings()`           | Noms des fournisseurs d'embeddings enregistrés |
| `list_query_rewriters()`      | Noms des réécrivants enregistrés |

## Plugins Intégrés

| Plugin            | Module                  | Groupe d'Entry Point       |
|-------------------|-------------------------|----------------------------|
| `whoosh_autocomplete` | `whoosh_modern.autocomplete.plugin` | `whoosh.plugins` |
| `whoosh_vector`   | `whoosh_modern.vector.plugin`      | `whoosh.plugins` |
| `whoosh_fastapi`  | `whoosh_fastapi`                  | `whoosh.apps` |
| `whoosh_observability` | `whoosh.middleware.metrics`  | `whoosh.middlewares` |
| `whoosh_admin`    | `whoosh_admin`                   | `whoosh.apps` |

## Bonnes Pratiques

1. **Responsabilité unique** : Un plugin, une fonctionnalité
2. **Déclarez les dépendances** : Utilisez `depends_on` pour les plugins requis
3. **Version sémantique** : Incrémentez la version pour les changements d'API
4. **Degradation gracieuse** : Vérifiez les dépendances optionnelles dans `register()`
5. **Pas d'effets de bord dans `__init__`** : Toute l'initialisation dans `register()`
6. **Nettoyage** : Si applicable, fournissez une logique de teardown

## Voir Aussi

- [Guide Middleware](middleware-sprint-c.md) — Pipeline hooks et middleware personnalisé
- [Exemple: Développement de Plugin](../examples/plugin-dev.md) — Tutoriel pas à pas
- [API: Plugins](../api/plugins.md) — Référence complète de l'API


## DOCUMENT (FR): Plugins

# Plugins

Whoosh-NG utilise une architecture à plugins pour garder le core léger tout en permettant des fonctionnalités avancées. Les plugins sont chargés via des entry points et gérés par le `PluginManager`.

## Architecture des plugins

```text
PluginManager
    ├── load_plugins()           # Auto-découvrir depuis entry points
    ├── register(name, plugin)   # Enregistrement manuel
    ├── enable(name)            # Activer un plugin
    ├── disable(name)           # Désactiver un plugin
    ├── get(name)               # Récupérer un plugin
    └── list_plugins()          # Lister tous les plugins
```

## Plugins intégrés

| Plugin | Description |
|--------|-------------|
| whoosh-ng-vector | Recherche vectorielle (NumPy, HNSW, Faiss) |
| whoosh-ng-autocomplete | Autocomplétion par edge n-gram |
| whoosh-ng-fastapi | Factory d'app FastAPI |
| whoosh-ng-observability | Métriques Prometheus |
| whoosh-ng-admin | Interface d'administration |

## Créer un plugin

Tout plugin hérite de `BasePlugin` :

```python
from whoosh.plugins.base import BasePlugin

class MonPlugin(BasePlugin):
    name = "mon_plugin"
    version = "1.0.0"
    dependencies = []

    def setup(self, registry):
        """Appelé quand le plugin est activé."""
        registry.register("mon_provider", MonProvider())

    def teardown(self, registry):
        """Appelé quand le plugin est désactivé."""
        registry.unregister("mon_provider")

    def middleware(self):
        """Middleware à injecter dans le pipeline."""
        return [MonMiddleware()]

    def on_startup(self):
        """Appelé une fois au démarrage."""
        pass

    def on_shutdown(self):
        """Appelé une fois à l'arrêt."""
        pass
```

## Enregistrement de plugin

### Via entry_points (pyproject.toml)

```toml
[project.entry-points."whoosh_ng.plugins"]
mon_plugin = "mon_package.plugin:MonPlugin"
```

### Programmatique

```python
from whoosh.plugins.manager import PluginManager

plugin = MonPlugin()
PluginManager.register("mon_plugin", plugin)
PluginManager.enable("mon_plugin")
```

## Cycle de vie d'un plugin

```
register() -> setup() -> enable() -> hooks middleware -> teardown() -> disable()
```

## Dépendances entre plugins

```python
class VectorPlugin(BasePlugin):
    name = "vector"
    dependencies = ["metrics"]  # Requiert le plugin metrics
```

Le `PluginManager` résout l'ordre de chargement et détecte les conflits.

## Bonnes pratiques

1. **Un plugin, une responsabilité**: Gardez les plugins petits et focalisés
2. **Déclarez les dépendances**: Aidez PluginManager à résoudre l'ordre
3. **Nettoyez bien**: Implémentez `teardown()` pour supprimer les registres

## Système de Plugins Moderne (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduit un `PluginManager` amélioré avec support de registres pour les sources de données, les fournisseurs de vecteurs, les embeddings et le middleware. Pour plus de détails sur l'architecture moderne des plugins, les groupes d'entry points et le déploiement, consultez le [Guide Système de Plugins](plugins-sprint-c.md).


## DOCUMENT (FR): Stemming Sprint D

# Fournisseurs de Stemmers

Module: `whoosh_modern.analysis.stemmer_providers`, `whoosh_modern.analysis.stemming_analyzer`, `whoosh_modern.linguistics.stemmers`
Version: 2.0.0

Le système de fournisseurs de stemmers donne un contrôle flexible sur le backend de stemming utilisé pour l'analyse de texte. Il prend en charge la détection automatique, la sélection explicite du backend et l'enregistrement de stemmers personnalisés — le tout avec une API propre de type plugin.

## Vue d'ensemble du module

```text
whoosh_modern.analysis
    ├── stemmer_providers.py   # Protocole StemmerProvider, fournisseurs Internal/PyStemmer, register_stemmer, get_stemmer
    └── stemming_analyzer.py   # StemmingAnalyzer amélioré avec support plugin

whoosh_modern.linguistics.stemmers
    └── __init__.py            # Analyseurs linguistiques (FR/EN/DE/ES/IT)
```

## Protocole StemmerProvider

Situé dans `whoosh_modern.analysis.stemmer_providers` :

```python
from whoosh_modern.analysis.stemmer_providers import StemmerProvider

class MyStemmer(StemmerProvider):
    def stem(self, word: str) -> str:
        """Réduire un mot à sa racine."""
        ...

    @property
    def name(self) -> str:
        """Retourner le nom du stemmer."""
        return "my_stemmer"

    @property
    def language(self) -> str:
        """Retourner le code de langue."""
        return "english"
```

## Obtenir un Stemmer

### Détection Automatique (Recommandé)

La fonction `get_stemmer("auto", language)` sélectionne automatiquement le meilleur backend disponible :

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Détection automatique : préfère PyStemmer si installé, sinon fallback interne
stemmer = get_stemmer("auto", "english")
print(stemmer.stem("running"))  # "run"
print(stemmer.name)             # "pystemmer" ou "internal"
```

**Ordre de priorité :**
1. **PyStemmer** (le plus rapide, nécessite `pip install whoosh-ng[fast-stemming]`)
2. **Stemmer interne** (Porter, toujours disponible)

### Sélection Explicite du Backend

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer

# Forcer le stemmer interne
stemmer = get_stemmer("internal", "english")

# Forcer PyStemmer (nécessite l'installation)
stemmer = get_stemmer("pystemmer", "english")
```

### Lister les Backends Disponibles

```python
from whoosh_modern.analysis.stemmer_providers import list_available_backends

backends = list_available_backends()
print(backends)
# {'internal': 'available', 'pystemmer': 'available', 'my_custom': 'registered'}
```

| Backend       | Chaîne de statut    | Nécessite                          |
|---------------|---------------------|-------------------------------------|
| `internal`    | `"available"`       | Aucun (toujours inclus)             |
| `pystemmer`   | `"available"` / `"not installed"` | `pip install whoosh-ng[fast-stemming]` |
| Personnalisé  | `"registered"`      | Enregistré via `@register_stemmer` |

## Fournisseurs de Stemmers Intégrés

### InternalStemmerProvider

Enveloppe le stemmer Porter intégré de Whoosh. Toujours disponible (aucune dépendance externe) :

```python
from whoosh_modern.analysis.stemmer_providers import InternalStemmerProvider

stemmer = InternalStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
print(stemmer.stem("running")) # "run"
```

### PyStemmerProvider

Enveloppe la bibliothèque `Stemmer` pour un stemming haute performance. Supporte toutes les langues Snowball :

```python
from whoosh_modern.analysis.stemmer_providers import PyStemmerProvider

# Nécessite: pip install whoosh-ng[fast-stemming]
stemmer = PyStemmerProvider("english")
print(stemmer.stem("cats"))    # "cat"
```

**Note** : Ce fournisseur appelle `self._stemmer.stemWord(word)` pour réduire les mots. Assurez-vous que PyStemmer est installé ou la détection automatique basculera vers le stemmer interne.

### IdentityStemmerProvider

Un stemmer sans opération pour les tests ou lorsque le stemming n'est pas souhaité :

```python
from whoosh_modern.analysis.stemmer_providers import IdentityStemmerProvider

stemmer = IdentityStemmerProvider()
print(stemmer.stem("anything"))  # "anything"
```

## Enregistrer un Stemmer Personnalisé

Utilisez le décorateur `@register_stemmer` :

```python
from whoosh_modern.analysis.stemmer_providers import register_stemmer

@register_stemmer("simple")
class SimpleStemmer:
    def stem(self, word: str) -> str:
        # Suppression simple de suffixe
        if word.endswith("s") and len(word) > 3:
            return word[:-1]
        return word

    @property
    def name(self) -> str:
        return "simple"

    @property
    def language(self) -> str:
        return "english"

# Maintenant l'utiliser
from whoosh_modern.analysis.stemmer_providers import get_stemmer

stemmer = get_stemmer("simple", "english")
print(stemmer.stem("cats"))  # "cat"
```

## StemmingAnalyzer (Amélioré)

Situé dans `whoosh_modern.analysis.stemming_analyzer`, c'est le point d'entrée principal pour créer des analyseurs linguistiques :

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Détection automatique du meilleur stemmer pour l'anglais
analyzer = StemmingAnalyzer(stemmer="auto", language="english")

# Stemmer interne explicite
analyzer = StemmingAnalyzer(stemmer="internal", language="english")

# Backend PyStemmer (si installé)
analyzer = StemmingAnalyzer(stemmer="pystemmer", language="french")

# Instance de fournisseur de stemmer personnalisé
analyzer = StemmingAnalyzer(stemmer=my_stemmer_instance)
```

### Paramètres de StemmingAnalyzer

| Paramètre   | Type                          | Défaut                   | Description                      |
|-------------|-------------------------------|--------------------------|----------------------------------|
| `expression`| Motif regex                   | motif de token par défaut | Tokenisation regex             |
| `stoplist`  | Itérable de mots vides        | `whoosh.analysis.STOP_WORDS` | Mots vides à filtrer         |
| `minsize`   | `int`                         | `2`                      | Longueur minimale du token       |
| `maxsize`   | `int \| None`                 | `None`                   | Longueur maximale du token       |
| `gaps`      | `bool`                        | `False`                  | Diviser sur l'expression vs correspondre |
| `stemmer`   | `str \| StemmerProvider`      | `"auto"`                 | Backend de stemmer               |
| `language`  | `str`                         | `"english"`              | Code de langue                   |
| `ignore`    | `set[str] \| None`            | `None`                   | Mots à ignorer                   |
| `cachesize` | `int`                         | `50000`                  | Taille du cache de stemming      |

### Utilisation avec les Types de Champs

```python
from whoosh_modern.analysis import StemmingAnalyzer
from whoosh.fields import Schema, TEXT

# Stemmer anglais avec mots vides
en_analyzer = StemmingAnalyzer("auto", language="english")

# Stemmer français
fr_analyzer = StemmingAnalyzer("auto", language="french")

schema = Schema(
    title=TEXT(stored=True),
    content_en=TEXT(analyzer=en_analyzer),
    content_fr=TEXT(analyzer=fr_analyzer),
)
```

## Analyseurs Linguistiques Spécifiques

Des analyseurs prêts à l'emploi pour cinq langues, disponibles dans `whoosh_modern.linguistics.stemmers` :

```python
from whoosh_modern.linguistics.stemmers import (
    EnglishAnalyzer,
    FrenchAnalyzer,
    GermanAnalyzer,
    SpanishAnalyzer,
    ItalianAnalyzer,
)

# Chaque analyseur est appelable et retourne une liste de tokens
en = EnglishAnalyzer()
tokens = en("The quick brown foxes")
# les tokens sont stemmés: ["quick", "brown", "fox"] (mots vides comme "the" supprimés)
```

### Analyseurs Linguistiques Disponibles

| Classe             | Langue    | Module                              |
|-------------------|-----------|-------------------------------------|
| `EnglishAnalyzer` | Anglais   | `whoosh_modern.linguistics.stemmers` |
| `FrenchAnalyzer`  | Français  | `whoosh_modern.linguistics.stemmers` |
| `GermanAnalyzer`  | Allemand  | `whoosh_modern.linguistics.stemmers` |
| `SpanishAnalyzer` | Espagnol  | `whoosh_modern.linguistics.stemmers` |
| `ItalianAnalyzer` | Italien   | `whoosh_modern.linguistics.stemmers` |

Chaque analyseur utilise en interne `get_stemmer("auto", language)` pour sélectionner le meilleur backend disponible et applique des mots vides spécifiques à la langue.

## Validation de la Compatibilité des Stemmers

Validez qu'un fournisseur de stemmer fonctionne correctement avec un ensemble de mots de test :

```python
from whoosh_modern.analysis.stemmer_providers import (
    get_stemmer,
    validate_stemmer_compatibility,
)

stemmer = get_stemmer("auto", "english")
report = validate_stemmer_compatibility(stemmer, ["running", "cats", "jumps", "houses"])

print(report["total_words"])   # 4
print(report["successful"])    # 4 (ou moins si erreurs)
print(report["failed"])        # 0
print(report["results"])       # [{'word': 'running', 'stemmed': 'run', 'success': True}, ...]
```

### Structure du Rapport de Compatibilité

| Champ          | Type       | Description                          |
|----------------|------------|--------------------------------------|
| `provider`     | `str`      | Nom du fournisseur de stemmer        |
| `language`     | `str`      | Code de langue                       |
| `total_words`  | `int`      | Nombre total de mots de test         |
| `successful`   | `int`      | Mots stemmés avec succès             |
| `failed`       | `int`      | Mots qui ont échoué                  |
| `results`      | `list[dict]` | Résultats par mot avec `word`, `stemmed`, `success` |

## Intégration avec StemmingMiddleware

Les fournisseurs de stemmers peuvent être utilisés avec le `StemmingMiddleware` de `whoosh_modern.middleware.analyzer` :

```python
from whoosh_modern.analysis.stemmer_providers import get_stemmer
from whoosh_modern.middleware.analyzer import StemmingMiddleware

stemmer = get_stemmer("auto", "english")
middleware = StemmingMiddleware(
    stemmer=stemmer.stem,
    fields=["title", "content"],  # Ne stemmer que ces champs
    stem_query=True,              # Also stemmer la requête de recherche
)
```

## Migration depuis Whoosh Classique

### Ancienne API (Whoosh 1.x/2.x)

```python
from whoosh.analysis import StemmingAnalyzer as OldAnalyzer
analyzer = OldAnalyzer("en")  # Codé en dur sur "english"
```

### Nouvelle API (Whoosh-NG 2.0)

```python
from whoosh_modern.analysis import StemmingAnalyzer

# Détection automatique du backend (recommandé)
analyzer = StemmingAnalyzer("auto", language="en")

# Ou utiliser un analyseur linguistique
from whoosh_modern.linguistics.stemmers import EnglishAnalyzer
analyzer = EnglishAnalyzer()
```

> **Note** : L'ancienne `StemmingAnalyzer("en")` était codée en dur sur la langue `"english"`. La nouvelle `StemmingAnalyzer(stemmer, language)` est explicite et prend en charge toutes les langues Snowball via PyStemmer.

## Installation

```bash
# Sans PyStemmer (utilise le stemmer interne, plus lent)
pip install whoosh-ng

# Avec PyStemmer (recommandé, plus rapide)
pip install whoosh-ng[fast-stemming]

# Analyse moderne complète
pip install whoosh-ng[modern]
```

## Voir Aussi

- [Guide Stemming et Mots Vides](stemming.md) — Guide classique de stemming de Whoosh
- [Guide Synonymes & Linguistique](linguistics-sprint-d.md) — Moteur d'expansion de synonymes
- [API: Moderne](../api/modern.md) — Référence complète de l'API pour les extensions d'analyse


## DOCUMENT (FR): Stemming

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Stemming and Stop Words

This guide covers using stemmers, stop-word filters, and language-specific
text analysis with Whoosh.

## Stemmers

A stemmer reduces words to their root form (e.g., "running" → "run",
"cats" → "cat"), so that different forms of the same word match in
searches.

### Using StemmerFilter

```python
from whoosh.analysis import RegexTokenizer, StemmerFilter
from whoosh.lang.porter import stem
from whoosh import fields

# English Porter stemmer
stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

schema = fields.Schema(
    title=fields.TEXT(stored=True),
    content=fields.TEXT(analyzer=stem_analyzer),
)
```

### Snowball Stemmers

Whoosh includes Snowball stemmers for multiple languages:

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.snowball import EnglishStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=EnglishStemmer().stem)
```

### Language-Aware Stemmer Selection

```python
from whoosh.lang import stemmer_for_language, StemmerFilter
from whoosh.analysis import RegexTokenizer

stem = stemmer_for_language("en")
analyzer = RegexTokenizer() | StemmerFilter(stemfn=stem)

# Or use the analysis StemmingAnalyzer:
from whoosh.analysis import StemmingAnalyzer

analyzer = StemmingAnalyzer("en")
```

### Available Languages

```python
from whoosh.lang import languages, has_stemmer, has_stopwords

print(languages)  # ('ar', 'da', 'nl', 'en', 'fi', 'fr', ...)
print(has_stemmer("en"))  # True
print(has_stopwords("en"))  # True
```

## Stop Words

Stop words are common words (like "the", "a", "and") that are typically
filtered out during indexing since they appear in too many documents to be
useful for ranking.

### Using StopFilter

```python
from whoosh.analysis import RegexTokenizer, StopFilter
from whoosh.lang import stopwords_for_language

# English stop words
stop_words = set(stopwords_for_language("en"))
stop_analyzer = RegexTokenizer() | StopFilter(stoplist=stop_words)

schema = fields.Schema(
    content=fields.TEXT(analyzer=stop_analyzer),
)
```

### Combining Stemming and Stop Words

```python
from whoosh.analysis import StemmingAnalyzer

# StemmingAnalyzer automatically loads stemmer and stopwords for the language
analyzer = StemmingAnalyzer("en")

schema = fields.Schema(
    content=fields.TEXT(analyzer=analyzer),
)
```

### Custom Stop Words

```python
from whoosh.analysis import RegexTokenizer, StopFilter

# Custom stop words list
custom_stops = frozenset(["the", "a", "an", "foo", "bar"])
analyzer = RegexTokenizer() | StopFilter(stoplist=custom_stops)
```

## StemmingAnalyzer (Recommended)

The `StemmingAnalyzer` combines tokenizer, stemming, and stop word filtering:

```python
from whoosh.analysis import StemmingAnalyzer

# Automatically uses the correct stemmer and stop words for the language
analyzer = StemmingAnalyzer("en")

# You can override defaults
analyzer = StemmingAnalyzer("en",
                            use_stopwords=True,
                            use_stems=True)
```

### StemmingAnalyzer Options

- `lang`: Language code (e.g., `"en"`, `"fr"`, `"de"`)
- `use_stopwords`: Whether to load and apply stop words (default `True`)
- `use_stems`: Whether to apply stemming (default `True`)
- `args`: Arguments passed to the tokenizer
- `kwargs`: Keyword arguments for the stemmer or stopwords

## Language-Specific Considerations

### Arabic (ISRI Stemmer)

```python
from whoosh.analysis import StemmerFilter
from whoosh.lang.isri import ISRIStemmer

stem_analyzer = RegexTokenizer() | StemmerFilter(stemfn=ISRIStemmer().stem)
```

### Double Metaphone for Phonetic Matching

```python
from whoosh.analysis import RegexTokenizer, DoubleMetaphoneFilter

analyzer = RegexTokenizer() | DoubleMetaphoneFilter()
```

## Query-Side Stemming

The analyzer is applied at both index time and query time (via the query
parser), so stemming is automatically applied to search terms:

```python
from whoosh.qparser import QueryParser

# If the index uses stemming, queries are stemmed too
qp = QueryParser("content", schema=ix.schema)
q = qp.parse("running cats")  # Will match "run", "cat", etc.
```

## N-gram Analysis

For substring and prefix matching, use N-gram analyzers:

```python
from whoosh.analysis import NgramWordAnalyzer

analyzer = NgramWordAnalyzer(minsize=2, maxsize=4)
schema = fields.Schema(content=fields.TEXT(analyzer=analyzer))
```

See the [N-grams Guide](ngrams.md) for more details.

## Modern Stemmer Providers (Whoosh-NG 2.0)

Whoosh-NG 2.0 introduces a plugin-style stemmer provider system with auto-detection, PyStemmer support, and language-specific analyzers. For full details, see the [Stemmer Providers Guide](stemming-sprint-d.md).


## DOCUMENT (FR): Storage Providers

> **Note de traduction** : Cette page n'est pas encore traduite en français.
> Le contenu anglais est affiché ci-dessous en attendant la traduction.

<!-- Creez une version francaise de ce fichier et supprimez ce message. -->


# Storage Providers

Whoosh-NG provides pluggable storage backends through the
`SyncStorageProvider` / `AsyncStorageProvider` contracts. This allows the
index to be persisted on local disk, SQLite, S3, or a hybrid cache + remote
setup without changing the writer or the index.

## Architecture Overview

### Level 1: SnapshotStorage (Simple)

```
Writer → Local FS → Commit → Upload Segment → S3
Reader → Download Segment → Open locally
```

Very simple to maintain. Use `SnapshotStorage` when you want S3 as a simple
backup/restore target without the complexity of a local cache.

### Level 2: CachedObjectStorage (Recommended for Production)

```
+----------+
|  MinIO   |
+----------+
     ^
     |
 Sync |
     v
+-----------+   Cache Layer   +-----------+
| Searcher  |<--------------->| Writer    |
+-----------+                 +-----------+
        |
        v
 Local SSD
```

- Index lives on SSD
- S3 serves as replication
- Segments are pushed after commit
- Restoration possible at any moment

This is what many modern distributed search systems do.

## Available providers

| Provider | Type | Backend | Use Case |
|----------|------|---------|----------|
| `FileStorage` | sync | local filesystem | Single-node, no cloud |
| `AsyncFileStorage` | async | local filesystem | Single-node async |
| `S3Storage` | sync | S3-compatible | Direct S3 access |
| `SnapshotStorage` | sync | S3-compatible | Simple backup/restore |
| `HybridStorage` | sync | local cache + remote | **Production** (alias: `CachedObjectStorage`) |
| `AsyncHybridStorage` | async | local cache + remote | Production async |

All providers are importable from `whoosh_modern.storage`.

## FileStorage

Local filesystem storage. Keys are relative paths under `root`.

```python
from whoosh_modern.storage import FileStorage

storage = FileStorage("indexdir")
storage.write("segment_1.dat", b"data")
assert storage.read("segment_1.dat") == b"data"
assert storage.exists("segment_1.dat") is True
storage.delete("segment_1.dat")
keys = storage.list_keys()
```

## AsyncFileStorage

Async variant of `FileStorage`. All operations run on a worker thread via
`asyncio.to_thread` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncFileStorage

storage = AsyncFileStorage("indexdir")

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")

asyncio.run(main())
```

## S3Storage

S3-compatible blob storage. `boto3` is imported lazily, so it is an optional
dependency. A `client` can be injected for testing.

```python
from whoosh_modern.storage import S3Storage

# Default client (requires boto3 installed and configured)
storage = S3Storage(bucket="my-index-bucket", prefix="segments")

# Or inject a client for testing / custom configuration
storage = S3Storage(
    bucket="my-index-bucket",
    prefix="segments",
    client=my_boto3_client,
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
keys = storage.list_keys()
```

Install the optional dependency:

```bash
pip install whoosh-ng[s3]
```

## SnapshotStorage

Simple S3 snapshot storage without local cache. This is the simplest
S3-backed storage strategy:

- Write: upload segment directly to S3
- Read: download segment from S3 to local temporary file

Use this when you want S3 as a simple backup/restore target without the
complexity of a local cache.

```python
from whoosh_modern.storage import SnapshotStorage

storage = SnapshotStorage(
    local_path="./index",
    bucket="my-index-bucket",
    prefix="snapshots",
)

storage.write("segment_1.dat", b"data")
data = storage.read("segment_1.dat")
```

## HybridStorage / CachedObjectStorage

`HybridStorage` composes a local cache and a remote backend. The remote is
the source of truth; the local cache is a write-through performance layer.

`CachedObjectStorage` is an alias for `HybridStorage` that better conveys
the intent: a local object cache synchronized with S3.

This is the recommended architecture for production deployments with repeated
read patterns.

```python
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

# Write-through: remote is source of truth, cache is updated on success
storage.write("segment_1.dat", b"data")

# First read: cache miss → fetch from S3, write-through into cache
data = storage.read("segment_1.dat")

# Second read: cache hit → served from local disk, zero network
data = storage.read("segment_1.dat")

# Force refresh from remote
storage.invalidate("segment_1.dat")

# Warm cache proactively
storage.prefetch(["segment_2.dat", "segment_3.dat"])
```

### Read path

1. local cache hit → return immediately
2. cache miss → read from remote, write-through into cache, return

### Write path

- `remote.write(key, data)` (source of truth)
- on success → `local_cache.write(key, data)`
- on failure → raise before polluting cache

### Cache eviction

The local cache is bounded by `max_cache_size_mb` (default 1024 MB). When
the limit is reached, the oldest entries are evicted using an LRU policy.

### `list_keys`

`list_keys()` uses the remote as source of truth because the cache is only
partial. Pass `include_cache=True` to return the union of remote and cache
keys.

## AsyncHybridStorage

Async variant of `HybridStorage`. Remote operations are executed on a worker
thread via `asyncio.to_thread` so the event loop is never blocked.

```python
import asyncio
from whoosh_modern.storage import AsyncHybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = AsyncHybridStorage(local_cache="./cache", remote=remote)

async def main() -> None:
    await storage.awrite("segment_1.dat", b"data")
    data = await storage.aread("segment_1.dat")
    await storage.adelete("segment_1.dat")
    keys = await storage.alist_keys()

asyncio.run(main())
```

## Using storage with SearchApplication

```python
from whoosh_modern import SearchApplication, SQLSource
from whoosh_modern.storage import HybridStorage, S3Storage

remote = S3Storage(bucket="my-index-bucket", prefix="segments")
storage = HybridStorage(local_cache="./cache", remote=remote)

app = SearchApplication(
    source=SQLSource(query="SELECT * FROM products", connection=engine),
    storage=storage,
)
app.build()
results = app.index.search("laptop")
```

## Performance Benchmarks

Benchmarks were run against a local MinIO instance using a 28.89 MB Whoosh
index (2 segment files). Results are indicative of relative performance
between strategies on S3-compatible storage.

| Strategy | Backup (MB/s) | Restore (MB/s) | Notes |
|----------|---------------|----------------|-------|
| `1_obj_per_segment` | 39.44 | 139.72 | Best restore throughput; simplest |
| `compressed_zstd` | 31.56 | 133.74 | Lower bandwidth, CPU overhead |
| `hybrid_cache_s3` | 44.97 | 133.61 | Best backup; excellent warm-cache reads |
| `1_obj_per_posting_list` | 0.28 | 4.79 | **Avoid**: millions of small objects kill S3 |

### Recommendations

- **Default**: `S3Storage` with 1 object per segment file. It offers the
  best restore throughput and is the simplest to operate.
- **Production with repeated reads**: `HybridStorage(local_cache, S3Storage)`.
  After the first read, subsequent reads are served from local disk at
  ~133 MB/s.
- **Avoid**: 1 object per posting list. S3 is not optimized for millions of
  tiny objects; latency and cost explode.
- **Compression**: ZSTD reduces transfer size by ~20-30% at the cost of CPU.
  Use it when network bandwidth is the bottleneck, not when CPU is.

### Running the benchmarks

```bash
# Start MinIO
docker run -d --name minio-benchmark -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio:latest server /data --console-address ":9001"

# Run synthetic benchmark
python benchmark/s3_storage_benchmark.py

# Run real Whoosh index benchmark (requires customers CSV)
python benchmark/s3_storage_benchmark_real.py
```


## DOCUMENT (FR): Vector

# Recherche vectorielle

Whoosh-NG supporte la recherche sémantique via des embeddings vectoriels. Ce guide couvre la configuration et l'utilisation des champs vectoriels.

## Concept

La recherche vectorielle permet de trouver des documents par similarité sémantique plutôt que par correspondance exacte de mots-clés.

```
Embedding requête  ----\
                       >--- Similarité cosinus ---> Résultats classés
Embedding document ---/
```

## Configuration

```python
from whoosh.fields import Schema, TEXT, VectorField

schema = Schema(
    title=TEXT(stored=True),
    content=TEXT,
    embedding=VectorField(dimensions=384)  # ex: all-MiniLM-L6-v2
)
```

## Providers

| Provider | Description | Cas d'usage |
|----------|-------------|-------------|
| `NumpyProvider` | NumPy pur, similarité cosinus | Petits/moyens indexes |
| `HNSWProvider` | Hierarchical Navigable Small World | Gros indexes, ANN rapide |
| `FaissProvider` | Facebook AI Similarity Search | Très gros indexes |
| `QdrantProvider` | Qdrant vector DB | Distribué |

## Indexation avec vecteurs

```python
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = model.encode([
    "Premier document",
    "Deuxième document"
])

with ix.writer() as writer:
    writer.add_document(
        title="Doc 1",
        content="Python est génial",
        embedding=embeddings[0].tolist()
    )
    writer.commit()
```

## Recherche hybride (mots-clés + vecteur)

```python
with ix.searcher() as searcher:
    # Composante sémantique
    query_embedding = model.encode(["Tutoriel Python"])[0]
    vector_results = searcher.vector_search(
        "embedding", query_embedding, limit=20
    )

    # Composante mots-clés
    keyword_query = QueryParser("content", schema).parse("Python")
    keyword_results = searcher.search(keyword_query, limit=20)

    # Combiner (ex: fusion RRF)
    final_results = fuse_results(vector_results, keyword_results)
```

## Bonnes pratiques

1. **Normalisez les embeddings**: Utilisez la similarité cosinus avec des vecteurs normalisés
2. **Choisissez le provider wisely**: Numpy pour &lt;100k vecteurs, HNSW/Faiss pour plus
3. **Recherche hybride**: Combinez vecteur et mots-clés pour de meilleurs résultats
4. **Cachez les embeddings**: Pré-calculez et stockez pour éviter de recalculer
5. **Indexation par lots**: Indexez les vecteurs en lots pour l'efficacité


# Code Examples
