Metadata-Version: 2.4
Name: zvec-db
Version: 0.9.0
Summary: Utility suite for sparse vectorization and document reranking using zvec
Author: Charles Condevaux
License: MIT
Project-URL: Homepage, https://github.com/ccdv-ai/zvec-db
Project-URL: Repository, https://github.com/ccdv-ai/zvec-db.git
Project-URL: Issues, https://github.com/ccdv-ai/zvec-db/issues
Keywords: search,ranking,BM25,TF-IDF,sparse,vectors,embeddings
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: zvec>=0.5.0
Requires-Dist: scikit-learn
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: httpx
Requires-Dist: requests
Requires-Dist: sentence_transformers
Requires-Dist: openai
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: cloudpickle
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
Requires-Dist: nltk>=3.8.0; extra == "test"
Provides-Extra: preprocessing
Requires-Dist: nltk>=3.8.0; extra == "preprocessing"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == "docs"
Requires-Dist: sphinx-math-dollar>=1.2.0; extra == "docs"
Provides-Extra: dev
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: black>=25.9.0; extra == "dev"
Requires-Dist: isort; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Provides-Extra: build
Requires-Dist: build; extra == "build"
Requires-Dist: twine; extra == "build"
Requires-Dist: wheel; extra == "build"

# zvec-db

[![Version](https://img.shields.io/pypi/v/zvec-db)](https://pypi.org/project/zvec-db/)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/ccdv-ai/zvec-db/blob/main/LICENSE)
[![Documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://ccdv-ai.github.io/zvec-db/)

Sparse/dense vectorization and document reranking toolkit for [zvec](https://github.com/alibaba/zvec).

---

## Quick Start

```bash
pip install zvec-db
```

### 1. Basic hybrid search with `search()`

```python
from zvec_db import search, BM25Embedder, SentenceTransformersEmbedder
import zvec

collection = zvec.open("./my_collection")
bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# Hybrid search: sparse + dense + FTS
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    fusion="auto",
    topk=10,
)
```

### 2. Two-stage retrieval with cross-encoder

```python
from zvec_db.rerankers import SentenceTransformerReranker

reranker = SentenceTransformerReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
    topk=10,
)

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    fusion="auto",
    reranker=reranker,
    topk=50,  # Fetch 50 candidates, rerank to 10
)
```

### 3. Custom fusion reranker (e.g., RRF)

```python
from zvec_db.rerankers import RrfReranker

fusion_model = RrfReranker(topk=50, rank_constant=60)

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fusion=fusion_model,  # Pre-instantiated fusion reranker
    topk=50,
)
```

**Parameters:** `collection`, `query`, `sparse_fields`, `dense_fields`, `fts_fields`, `fusion` (`"auto"` | `dict` | reranker instance | `None`), `weights`, `reranker`, `topk`, `output_fields`.

**Note:** Configure `topk` and `cross_encoder_weight` directly on the reranker instance.

---

## Embedders

### Sparse (lexical search)

| Embedder | Description |
|----------|-------------|
| `BM25Embedder` | **Recommended** — Standard BM25 scoring |
| `BM25LEmbedder` | BM25L for documents with variable lengths |
| `BM25PlusEmbedder` | BM25+ with delta smoothing |
| `TfidfEmbedder` | TF-IDF weighting |
| `CountEmbedder` | Simple term counts |
| `DisMaxEmbedder` | Multi-field search (maximum score) |

**Example:**

```python
from zvec_db import BM25Embedder

bm25 = BM25Embedder(max_features=4096, k1=1.2, b=0.75)
bm25.fit(documents)

# Embed query
query_vector = bm25.embed("machine learning")

# Save/load
bm25.save("bm25.joblib")
bm25_loaded = BM25Embedder().load("bm25.joblib")
```

### Dense (semantic search)

| Embedder | Description |
|----------|-------------|
| `SentenceTransformersEmbedder` | Local models (e.g., `all-MiniLM-L6-v2`) |
| `OpenAIEmbedder` | OpenAI API (`text-embedding-3-small`) |

**Example:**

```python
from zvec_db import SentenceTransformersEmbedder

dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")
vector = dense.embed("machine learning")
```

---

## Rerankers

### Fusion (combine multiple sources)

| Reranker | Description |
|----------|-------------|
| `WeightedReranker` | Weighted score fusion with normalization |
| `RrfReranker` | Reciprocal Rank Fusion (rank-based) |
| `MultiFieldWeightedReranker` | Field-based weighted fusion |

**Example — WeightedReranker:**

```python
from zvec_db.rerankers import WeightedReranker

reranker = WeightedReranker(
    schema=collection.schema,
    weights={"sparse": 0.4, "dense": 0.6},
    normalize="auto",
    topk=10,
)
results = reranker.rerank({"sparse": sparse_docs, "dense": dense_docs})
```

**Example — RrfReranker:**

```python
from zvec_db.rerankers import RrfReranker

reranker = RrfReranker(topk=10, rank_constant=60)
results = reranker.rerank({"sparse": sparse_docs, "dense": dense_docs})
```

### Cross-Encoder (query + document scoring)

| Reranker | Description |
|----------|-------------|
| `SentenceTransformerReranker` | Local cross-encoder models |
| `ClassificationReranker` | Multi-class classification |
| `OpenAIReranker` | OpenAI API-based reranking |

**Example:**

```python
from zvec_db.rerankers import SentenceTransformerReranker

reranker = SentenceTransformerReranker(
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
    topk=10,
    cross_encoder_weight=0.8,  # Blend: 80% cross-encoder, 20% fusion
)
results = reranker.rerank({"bm25": docs}, query="machine learning")
```

### Pipeline (chain multiple rerankers)

```python
from zvec_db.rerankers import PipelineReranker, RrfReranker, SentenceTransformerReranker

pipeline = PipelineReranker(
    rerankers=[
        RrfReranker(topk=32, rank_constant=60),  # RRF fusion
        SentenceTransformerReranker(topk=10),    # Cross-encoder
    ],
    topk=10,
    query="neural networks",
)
results = pipeline.rerank({"sparse": sparse_docs, "dense": dense_docs})
```

---

## Preprocessing

Improve sparse embedding quality with text normalization:

```python
from zvec_db.embedders import BM25Embedder
from zvec_db.preprocessing import NormalizationConfig

config = NormalizationConfig.aggressive(language="english")
bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
bm25.fit(documents)
```

Install: `pip install "zvec-db[preprocessing]"`

Utilities: `normalize_text()`, `stem_word()`, `remove_stopwords()`

---

## Advanced Usage

### Manual query building

For full control, use `multi_field_queries()`:

```python
from zvec_db import multi_field_queries

queries = multi_field_queries(
    query_text="neural networks",
    sparse_fields={"sparse_title": bm25, "sparse_content": bm25},
    dense_fields={"dense_title": dense, "dense_content": dense},
    fts_fields=["title", "content"],
)

results = collection.query(queries=queries, topk=10, output_fields=["text"])
```

### FTS with phrase matching

```python
from zvec_db import fts_query

q = fts_query(field_name="title", query="deep learning", match_string="deep learning")
results = collection.query(queries=[q], topk=10)
```

---

## Documentation

For detailed examples and API reference:

- [User Guide](https://ccdv-ai.github.io/zvec-db/guide.html) — Hybrid search, FTS, reranking
- [API Reference](https://ccdv-ai.github.io/zvec-db/api.html) — Complete API docs

---

## Development

```bash
git clone https://github.com/ccdv-ai/zvec-db.git
cd zvec-db
pip install -e ".[dev,test,docs,preprocessing]"

make test   # Run tests
make lint   # black, isort, flake8, mypy
make docs   # Build documentation
```

---

## License

MIT License
