Metadata-Version: 2.4
Name: zvec-db
Version: 0.8.0
Summary: Utility suite for sparse vectorization and document reranking using zvec
Author: Charles Condevaux
License: MIT
Project-URL: Homepage, https://github.com/ccdv-ai/zvec-db
Project-URL: Repository, https://github.com/ccdv-ai/zvec-db.git
Project-URL: Issues, https://github.com/ccdv-ai/zvec-db/issues
Keywords: search,ranking,BM25,TF-IDF,sparse,vectors,embeddings
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.12
Description-Content-Type: text/markdown
Requires-Dist: zvec>=0.4.0
Requires-Dist: scikit-learn
Requires-Dist: numpy
Requires-Dist: scipy
Requires-Dist: httpx
Requires-Dist: requests
Requires-Dist: sentence_transformers
Requires-Dist: openai
Requires-Dist: aiohttp>=3.9.0
Requires-Dist: cloudpickle
Provides-Extra: test
Requires-Dist: pytest>=7.0.0; extra == "test"
Requires-Dist: pytest-cov>=4.0.0; extra == "test"
Requires-Dist: pytest-asyncio>=0.23.0; extra == "test"
Requires-Dist: nltk>=3.8.0; extra == "test"
Provides-Extra: preprocessing
Requires-Dist: nltk>=3.8.0; extra == "preprocessing"
Provides-Extra: docs
Requires-Dist: sphinx>=7.0.0; extra == "docs"
Requires-Dist: sphinx-rtd-theme>=2.0.0; extra == "docs"
Requires-Dist: sphinx-math-dollar>=1.2.0; extra == "docs"
Provides-Extra: dev
Requires-Dist: pre-commit; extra == "dev"
Requires-Dist: black>=25.9.0; extra == "dev"
Requires-Dist: isort; extra == "dev"
Requires-Dist: flake8; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Provides-Extra: build
Requires-Dist: build; extra == "build"
Requires-Dist: twine; extra == "build"
Requires-Dist: wheel; extra == "build"

# zvec-db

[![Version](https://img.shields.io/pypi/v/zvec-db)](https://pypi.org/project/zvec-db/)
[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
[![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/ccdv-ai/zvec-db/blob/main/LICENSE)
[![Documentation](https://img.shields.io/badge/docs-latest-brightgreen.svg)](https://ccdv-ai.github.io/zvec-db/)

Sparse/dense vectorization and document reranking toolkit for [zvec](https://github.com/alibaba/zvec).

---

## Quick Start

```bash
pip install zvec-db
```

---

## Available Embedders & Rerankers

### Embedders

**Sparse (lexical search):**
| Embedder | Description |
|----------|-------------|
| `BM25Embedder` | Standard BM25 scoring (recommended for most use cases) |
| `BM25LEmbedder` | BM25L variant for documents with highly variable lengths |
| `BM25PlusEmbedder` | BM25+ with delta smoothing to avoid zero scores |
| `TfidfEmbedder` | TF-IDF weighting with optional sublinear TF |
| `CountEmbedder` | Simple term counts (binary option available) |
| `DisMaxEmbedder` | Multi-field search using maximum score |

**Dense (semantic search):**
| Embedder | Description |
|----------|-------------|
| `SentenceTransformersEmbedder` | Local sentence-transformers models (e.g., `all-MiniLM-L6-v2`) |
| `OpenAIEmbedder` | OpenAI API embeddings (e.g., `text-embedding-3-small`) |

### Rerankers

**Fusion (combine multiple sources):**
| Reranker | Description |
|----------|-------------|
| `WeightedReranker` | Weighted score fusion with automatic metric handling |
| `RrfReranker` | Reciprocal Rank Fusion (rank-based, robust to score scale differences) |
| `MultiFieldWeightedReranker` | Weighted fusion for multi-field schemas |

**Cross-Encoder (query + document scoring):**
| Reranker | Description |
|----------|-------------|
| `SentenceTransformerReranker` | Local cross-encoder models (e.g., `ms-marco-MiniLM-L-6-v2`) |
| `ClassificationReranker` | Multi-class classification reranking |
| `OpenAIReranker` | OpenAI API-based reranking |
| `OpenAIDecoderReranker` | OpenAI decoder-style reranking |

**Utilities:**
| Utility | Description |
|---------|-------------|
| `PipelineReranker` | Chain multiple rerankers in sequence |
| `Normalize` | Score normalization (bayes, minmax, rank, percentile) |

---

## Quick Start (5 minutes)

### 1. Index documents

```python
from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder

documents = [
    "Machine learning is a subset of AI",
    "Deep learning uses neural networks",
    "NLP helps computers understand text",
]

# Create embedders
bm25 = BM25Embedder(max_features=4096)
bm25.fit(documents)

dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# Encode documents
for doc in documents:
    sparse_vec = bm25.embed(doc)   # dict: {index: score}
    dense_vec = dense.embed(doc)   # numpy array
```

### BM25 variants and CountVectorizer parameters

**Standard BM25 (unigrams only - FTS):**
```python
bm25 = BM25Embedder(
    max_features=4096,
    k1=1.2,  # Term frequency saturation
    b=0.75,  # Length normalization
)
```

**BM25 with trigrams (character n-grams for fuzzy matching):**
```python
bm25_trigram = BM25Embedder(
    max_features=8192,
    k1=1.2,
    b=0.75,
    analyzer="char_wb",      # Character-level n-grams (or "char")
    ngram_range=(3, 3),      # Trigrams only
    min_df=2,                # Minimum document frequency
)
```

**BM25 with mixed unigrams + trigrams:**
```python
bm25_mixed = BM25Embedder(
    max_features=16384,
    analyzer="word",
    ngram_range=(1, 1),      # Word unigrams only
)

# Or combine word + char n-grams by using char_wb:
bm25_fuzzy = BM25Embedder(
    max_features=16384,
    analyzer="char_wb",      # Character n-grams at word boundaries
    ngram_range=(2, 4),      # Bi-grams, trigrams, 4-grams
)
```

**Other CountVectorizer parameters you can use:**
```python
bm25 = BM25Embedder(
    max_features=4096,
    # Tokenization
    tokenizer=lambda x: x.split(),  # Custom tokenizer
    token_pattern=r"(?u)\b\w+\b",   # Regex pattern for tokens
    # Vocabulary filtering
    min_df=2,            # Minimum document frequency
    max_df=0.8,          # Maximum document frequency (removes common terms)
    max_features=10000,  # Max vocabulary size
    # N-grams
    ngram_range=(1, 2),  # Unigrams + bigrams
    analyzer="word",     # "word", "char", or "char_wb"
    # Preprocessing
    lowercase=True,      # Convert to lowercase
    stop_words="english",# Remove English stopwords
)
```

### 2. Search with hybrid + reranking

```python
from zvec.model.doc import Doc
from zvec_db.rerankers import WeightedReranker
from zvec.typing import MetricType

query = "neural networks"

# Simulated search results from different retrievers
bm25_results = [
    Doc(id="0", score=1.2),
    Doc(id="1", score=0.9),
    Doc(id="2", score=0.6),
]
dense_results = [
    Doc(id="1", score=0.85),
    Doc(id="0", score=0.75),
    Doc(id="3", score=0.65),
]

# Option 1: With explicit metrics
reranker = WeightedReranker(
    weights={"bm25": 0.4, "dense": 0.6},
    metrics={"bm25": MetricType.IP, "dense": MetricType.COSINE},
    normalize=True,  # Smart default: COSINE->/2, others->bayes
)

# Option 2: With schema auto-detection (recommended with zvec)
# import zvec
# collection = zvec.open("./my_collection")
# reranker = WeightedReranker(
#     schema=collection.schema,  # Auto-detect metrics from schema
#     weights={"bm25": 0.4, "dense": 0.6},
#     normalize=True,
# )

results = reranker.rerank({
    "bm25": bm25_results,
    "dense": dense_results,
})

print(results[0].id)  # Most relevant document
```

---

## Table of Contents

- [Key Concepts](#key-concepts)
- [Advanced Example](#advanced-example-hybrid-search-with-zvec)
- [Reranking](#reranking)
- [Pipeline Example](#pipeline-example)
- [Cross-Encoder Rerankers](#cross-encoder-rerankers)
- [Preprocessing](#preprocessing)
- [Model Persistence](#model-persistence)
- [Development](#development)

---

## Key Concepts

### Distance vs Similarity Metrics

Vector databases store **distances** (smaller = more similar), but fusion algorithms assume **similarities** (larger = more relevant). The `metrics` parameter handles conversion automatically:

| Metric   | Type       | Range     | Conversion      | Usage                                      |
|----------|------------|-----------|-----------------|--------------------------------------------|
| COSINE   | Distance   | [0, 2]    | `(2 - score) / 2` | Normalized embeddings (Qdrant, zvec)       |
| L2       | Distance   | [0, ∞)    | `-score`          | Euclidean distance                         |
| IP       | Similarity | (-∞, ∞)   | None              | Inner product, BM25 scores (already similar) |

**Default**: `MetricType.COSINE` (main use case with zvec/Qdrant).

### Choosing a Sparse Embedder

| Embedder           | Use case                                      |
|--------------------|-----------------------------------------------|
| `BM25Embedder`     | **Recommended** - standard lexical search     |
| `TfidfEmbedder`    | TF-IDF weighting with sublinear TF option     |
| `CountEmbedder`    | Simple term counts (binary option available)  |
| `BM25LEmbedder`    | Documents with highly variable lengths        |
| `BM25PlusEmbedder` | Avoid zero scores with delta smoothing        |
| `DisMaxEmbedder`   | Multi-field search (takes maximum score)      |

---

## Advanced Example: Hybrid Search with zvec

```python
import zvec
from zvec.model.doc import Doc
from zvec_db.embedders import BM25Embedder, SentenceTransformersEmbedder
from zvec_db.rerankers import WeightedReranker

documents = [
    "Machine learning is a subset of AI",
    "Deep learning uses neural networks",
    "NLP helps computers understand text",
]

# Create embedders
bm25 = BM25Embedder(max_features=4096)
bm25.fit(documents)
dense = SentenceTransformersEmbedder(model_name="all-MiniLM-L6-v2")

# Create collection
schema = zvec.CollectionSchema(
    name="docs",
    fields=[zvec.FieldSchema("text", zvec.DataType.STRING)],
    vectors=[
        zvec.VectorSchema(name="sparse", data_type=zvec.DataType.SPARSE_VECTOR_FP32, dimension=4096),
        zvec.VectorSchema(
            name="dense", 
            data_type=zvec.DataType.VECTOR_FP32, 
            dimension=384,
            index_param=zvec.FlatIndexParam(metric_type=zvec.MetricType.COSINE)
        ),
    ]
)
collection = zvec.create_and_open("./my_db", schema)

# Index documents
for i, doc in enumerate(documents):
    collection.insert(Doc(
        id=str(i),
        fields={"text": doc},
        vectors={
            "sparse": bm25.embed(doc),
            "dense": dense.embed(doc),
        }
    ))

# Search with auto-detected metrics from schema
reranker = WeightedReranker(
    topn=3,
    schema=collection.schema,  # Auto-detect metrics: sparse->None, dense->COSINE
    weights={"sparse": 0.4, "dense": 0.6},
    normalize=True,  # Smart default: sparse->bayes, dense->/2
)

query = "neural networks"
results = collection.query(
    vectors=[
        zvec.VectorQuery(field_name="sparse", vector=bm25.embed(query)),
        zvec.VectorQuery(field_name="dense", vector=dense.embed(query)),
    ],
    topk=10,
    reranker=reranker,
)

print("Top results:")
for i, doc in enumerate(results[:3]):
    print(f"  {i+1}. {doc.fields['text']} (score: {doc.score:.4f})")
```

---

## Reranking

### Normalization

The `normalize` parameter controls score normalization:

| Value                                | Effect                                              |
|--------------------------------------|-----------------------------------------------------|
| `True`                               | **Smart default**: COSINE → no-op, others → `"bayes"` |
| `"bayes"`                            | Bayesian sigmoid calibration (robust to outliers)   |
| `"minmax"`                           | Min-max: `(x - min) / (max - min)`                  |
| `"rank"` / `"percentile"`            | Rank-based (very robust to outliers)                |
| `"cosine"`                           | No-op (identity). COSINE scores already in [0, 1]   |
| `{"sparse": "bayes", "dense": "cosine"}` | Per-source configuration                        |
| `None` / `False`                     | No normalization                                    |

**Note**: `normalize=True` requires `schema` or `metrics` to auto-detect the metric per source.

COSINE is already normalized to [0, 1] by the conversion formula `(2 - score) / 2`, so `normalize="cosine"` is a no-op (identity). Use it for explicit API consistency when you want to document that no additional normalization is applied.

### WeightedReranker

Weighted fusion of multiple sources:

```python
from zvec_db.rerankers import WeightedReranker
from zvec.typing import MetricType

# With explicit metrics
reranker = WeightedReranker(
    weights={"bm25": 0.4, "dense": 0.6},
    metrics={"bm25": MetricType.IP, "dense": MetricType.COSINE},
    normalize="bayes",
)

# With schema auto-detection (recommended)
import zvec
collection = zvec.open("./my_collection")
reranker = WeightedReranker(
    schema=collection.schema,
    weights={"sparse": 0.4, "dense": 0.6},
    normalize=True,
)

results = reranker.rerank({"bm25": bm25_docs, "dense": dense_docs})
```

### RrfReranker (Reciprocal Rank Fusion)

Rank-based fusion (robust to score scale differences):

```python
from zvec_db.rerankers import RrfReranker
from zvec.model.doc import Doc
import zvec

# Search results from different retrievers
bm25_results = [Doc(id="1", score=1.5), Doc(id="2", score=1.2)]
dense_results = [Doc(id="2", score=0.9), Doc(id="3", score=0.8)]

# With schema auto-detection (recommended)
collection = zvec.open("./my_collection")
reranker = RrfReranker(
    topn=10,
    rank_constant=60,
    schema=collection.schema,  # Auto-detect metrics
)
results = reranker.rerank({
    "bm25": bm25_results,
    "dense": dense_results,
})

# With custom weights
reranker = RrfReranker(
    topn=10,
    rank_constant=60,
    weights={"dense": 0.7, "bm25": 0.3},
    schema=collection.schema,
)
```

**Note**: RRF uses ranks, not scores. The `normalize` parameter has no effect.

---

## Cross-Encoder Rerankers

Cross-encoders recalculate scores using both query and document. **Require a `query` parameter.**

```python
from zvec_db.rerankers import SentenceTransformerReranker

reranker = SentenceTransformerReranker(
    query="machine learning",
    model_name="cross-encoder/ms-marco-MiniLM-L-6-v2",
    topn=10,
)
results = reranker.rerank({"bm25": docs})
```

Other cross-encoders: `ClassificationReranker` (multi-class), `OpenAIReranker` (API).

---

## Pipeline Example

Chain multiple rerankers (RRF fusion + Cross-Encoder refinement):

```python
from zvec_db.rerankers import PipelineReranker, RrfReranker, SentenceTransformerReranker

pipeline = PipelineReranker(
    rerankers=[
        RrfReranker(
            topn=32, 
            rank_constant=60, 
            schema=collection.schema
        ),  # First: RRF fusion
        SentenceTransformerReranker(
            topn=16,
            model_name="cross-encoder/ms-marco-MiniLM-L-6-v2"
        )  # Then: Cross-Encoder re-scoring
    ],
    topn=10
)

results = pipeline.rerank({
    "bm25": bm25_results,
    "dense": dense_results,
}, query="neural networks")
```

---

## Preprocessing

Preprocessing improves sparse embedding quality:

```python
from zvec_db.embedders import BM25Embedder
from zvec_db.preprocessing import NormalizationConfig

config = NormalizationConfig.aggressive(language="english")
bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
bm25.fit(documents)

# Utility functions
from zvec_db.preprocessing import normalize_text, stem_word, remove_stopwords
normalize_text("  HELLO WORLD  ", lowercase=True, stem=True)  # "hello world"
```

Install: `pip install "zvec-db[preprocessing]"`

---

## Model Persistence

```python
from zvec_db.embedders import BM25Embedder

# Save
bm25 = BM25Embedder(max_features=4096, preprocessing_config=config)
bm25.fit(documents)
bm25.save("models/bm25_model.joblib")

# Load
bm25_loaded = BM25Embedder()
bm25_loaded.load("models/bm25_model.joblib")
```

---

## Development

```bash
# Clone and install
git clone https://github.com/ccdv-ai/zvec-db.git
cd zvec-db
pip install -e ".[dev,test,docs,preprocessing]"

# Run tests
make test

# Lint
make lint      # black, isort, flake8, mypy

# Build docs
make docs
```

---

## License

MIT License
