Metadata-Version: 2.4
Name: zvec-db
Version: 0.14.0
Summary: Utility suite for sparse vectorization and document reranking using zvec
Author: Charles Condevaux
License: MIT
Project-URL: Homepage, https://github.com/ccdv-ai/zvec-db
Project-URL: Repository, https://github.com/ccdv-ai/zvec-db.git
Project-URL: Issues, https://github.com/ccdv-ai/zvec-db/issues
Keywords: search,ranking,BM25,TF-IDF,sparse,vectors,embeddings
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: zvec>=0.5.0
Requires-Dist: scikit-learn
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: httpx
Requires-Dist: requests
Requires-Dist: sentence_transformers
Requires-Dist: openai
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: cloudpickle
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
Requires-Dist: nltk>=3.8.0; extra == "test"
Provides-Extra: preprocessing
Requires-Dist: nltk>=3.8.0; extra == "preprocessing"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == "docs"
Requires-Dist: sphinx-math-dollar>=1.2.0; extra == "docs"
Provides-Extra: dev
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: black>=25.9.0; extra == "dev"
Requires-Dist: isort; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Provides-Extra: build
Requires-Dist: build; extra == "build"
Requires-Dist: twine; extra == "build"
Requires-Dist: wheel; extra == "build"

# zvec-db

[![Version](https://img.shields.io/pypi/v/zvec-db)](https://pypi.org/project/zvec-db/)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/ccdv-ai/zvec-db/blob/main/LICENSE)
[![Documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://ccdv-ai.github.io/zvec-db/)
[![Tests](https://img.shields.io/badge/tests-464+ passed-brightgreen.svg)](https://github.com/ccdv-ai/zvec-db/actions)

Sparse/dense vectorization and document reranking toolkit for [zvec](https://github.com/alibaba/zvec).

**zvec-db** provides:
- 🚀 **Hybrid search** - Combine sparse (BM25) and dense (semantic) vectors with full-text search
- 🔄 **Score fusion** - Weighted fusion and Reciprocal Rank Fusion (RRF)
- 🎯 **Cross-encoder reranking** - Two-stage retrieval with local or API-based rerankers
- 📦 **6 sparse embedders** - BM25, BM25L, BM25+, TF-IDF, Count, DisMax
- 🔧 **Normalization** - Bayesian, minmax, percentile, and arctan-based normalization
- ⚡ **Native fast path** - C++ acceleration for fusion rerankers

---

## Quick Start

```bash
pip install zvec-db
```

### Complete Example: Hybrid Search with RerankPipeline

```python
from zvec_db import search, RerankPipeline, ScoreNormalizer, WeightedReranker
from zvec_db.rerankers import SentenceTransformerReranker
from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder
import zvec

# 1. Open collection and initialize embedders
collection = zvec.open("./my_collection")
bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# 2. Create rerank pipeline with auto-detection from schema
reranker = RerankPipeline(
    normalizer=ScoreNormalizer(
        method="bayes",
        schema=collection.schema,  # Auto-detect metrics (COSINE, L2, IP)
    ),
    fusion=WeightedReranker(
        weights={"sparse": 0.5, "dense": 0.3, "text": 0.2},
        topk=16,  # Top 16 after fusion
    ),
    cross_encoder=SentenceTransformerReranker(
        model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
        topk=8,  # Top 8 after cross-encoder
    ),
)

# 3. Search with full pipeline
results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    fts_fields="text",
    reranker=reranker,
    topk=32,  # Fetch 32 candidates
)

for doc in results:
    print(f"{doc.fields['text']} (score: {doc.score:.4f})")
```

### Simple Search (No Reranking)

```python
from zvec_db import search

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    topk=10,
)
```

### Custom Fusion Only

```python
from zvec_db.rerankers import WeightedReranker

# List weights enable C++ fast path (2-5x faster)
reranker = WeightedReranker(weights=[0.7, 0.3])

results = search(
    collection=collection,
    query="neural networks",
    sparse_fields={"sparse": bm25},
    dense_fields={"dense": dense},
    reranker=reranker,
)
```

---

## API Reference

### `search()` Parameters

| Parameter | Type | Description |
|-----------|------|-------------|
| `collection` | `zvec.Collection` | Collection to search |
| `query` | `str` | Query text |
| `sparse_fields` | `dict[str, Embedder]` | Sparse fields (e.g., `{"sparse": bm25}`) |
| `dense_fields` | `dict[str, Embedder]` | Dense fields (e.g., `{"dense": dense}`) |
| `fts_fields` | `str \| list[str]` | FTS fields |
| `reranker` | `Reranker` | Reranker instance |
| `topk` | `int` | Results to return |
| `output_fields` | `list[str]` | Fields to include |

### Reranker Types

| Type | Description |
|------|-------------|
| `WeightedReranker(weights={...})` | Weighted fusion |
| `RrfReranker()` | Reciprocal Rank Fusion |
| `SentenceTransformerReranker()` | Cross-encoder |
| `RerankPipeline(normalizer=..., fusion=..., cross_encoder=...)` | Full pipeline |

---

## Embedders

**Sparse (lexical):** `BM25Embedder` (recommended), `BM25LEmbedder`, `BM25PlusEmbedder`, `TfidfEmbedder`, `CountEmbedder`, `DisMaxEmbedder`

**Dense (semantic):** `SentenceTransformersEmbedder`, `OpenAIEmbedder`, `VllmEmbedder`

```python
from zvec_db import BM25Embedder, SentenceTransformersEmbedder

bm25 = BM25Embedder(max_features=4096).load("./bm25.joblib")
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

bm25.save("bm25.joblib")
```

## Rerankers

**Fusion:** `WeightedReranker` (weighted sum), `RrfReranker` (rank-based)

**Cross-Encoder:** `SentenceTransformerReranker`, `OpenAIReranker`

**Pipeline:** `RerankPipeline` (normalization + fusion + cross-encoder)

```python
from zvec_db import RerankPipeline, ScoreNormalizer, WeightedReranker
from zvec_db.rerankers import SentenceTransformerReranker

# Full pipeline with schema auto-detection
pipeline = RerankPipeline(
    normalizer=ScoreNormalizer(method="bayes", schema=collection.schema),
    fusion=WeightedReranker(weights={"sparse": 0.7, "dense": 0.3}),
    cross_encoder=SentenceTransformerReranker(topk=10),
)
```

**Performance tip:** Use list weights for C++ fast path: `WeightedReranker(weights=[0.7, 0.3])`

---

## Documentation

- [User Guide](https://ccdv-ai.github.io/zvec-db/guide.html) — Complete examples
- [API Reference](https://ccdv-ai.github.io/zvec-db/api.html) — Full API docs

---

## License

MIT License
