Metadata-Version: 2.4
Name: llama-index-vector-stores-velesdb
Version: 2.0.0
Summary: LlamaIndex VectorStore for VelesDB: The Local AI Memory Database. Microsecond RAG retrieval.
Project-URL: Homepage, https://github.com/cyberlife-coder/VelesDB
Project-URL: Documentation, https://github.com/cyberlife-coder/VelesDB#readme
Author-email: VelesDB Team <contact@wiscale.fr>
License-Expression: MIT
License-File: LICENSE
Keywords: ai-memory,embeddings,gpt,llamaindex,llm,local-first,rag,semantic-search,vector-store,velesdb
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Requires-Dist: llama-index-core<1.0.0,>=0.10.0
Requires-Dist: velesdb-common>=1.14.0
Requires-Dist: velesdb>=1.14.0
Provides-Extra: dev
Requires-Dist: pytest-asyncio<1.0.0,>=0.21; extra == 'dev'
Requires-Dist: pytest<9.0,>=7.0; extra == 'dev'
Description-Content-Type: text/markdown

# LlamaIndex VelesDB Integration

[![PyPI](https://img.shields.io/pypi/v/llama-index-vector-stores-velesdb)](https://pypi.org/project/llama-index-vector-stores-velesdb/)
[![License](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)

VelesDB vector store integration for [LlamaIndex](https://www.llamaindex.ai/).

## Features

- 🚀 **Sub-millisecond search** — SIMD-optimized vector retrieval
- 📦 **Self-contained** — Single VelesDB binary, no external services required
- 🔒 **Local-first** — All data stays on your machine
- 🧠 **RAG-ready** — Built for Retrieval-Augmented Generation
- 🔀 **Multi-Query Fusion** — Native MQG support with RRF/Weighted strategies
- 🌊 **Streaming inserts** — Backpressure-aware streaming ingestion through VelesDB's bounded-channel API

## Installation

```bash
pip install llama-index-vector-stores-velesdb
```

## Quick Start

```python
from llama_index.core import SimpleDirectoryReader, StorageContext, VectorStoreIndex
from llamaindex_velesdb import VelesDBVectorStore

# Create vector store
vector_store = VelesDBVectorStore(
    path="./velesdb_data",
    collection_name="my_docs",
    metric="cosine",
)

# Wrap it in a StorageContext. This step is required: from_documents()
# ignores a bare vector_store= keyword and would silently index into an
# in-memory store, leaving the VelesDB collection empty.
storage_context = StorageContext.from_defaults(vector_store=vector_store)

# Load and index documents — chunks are written to VelesDB
documents = SimpleDirectoryReader("./data").load_data()
index = VectorStoreIndex.from_documents(
    documents,
    storage_context=storage_context,
)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("What is VelesDB?")
print(response)
```

## Usage with Existing Index

```python
from llama_index.core import VectorStoreIndex
from llamaindex_velesdb import VelesDBVectorStore

# Connect to existing data
vector_store = VelesDBVectorStore(path="./existing_data")
index = VectorStoreIndex.from_vector_store(vector_store)

# Query
query_engine = index.as_query_engine()
response = query_engine.query("Summarize the key points")
```

## API Reference

### VelesDBVectorStore

```python
VelesDBVectorStore(
    path: str = "./velesdb_data",      # Database directory
    collection_name: str = "llamaindex", # Collection name
    metric: str = "cosine",             # Distance metric
    storage_mode: str = "full",         # Storage / quantization mode
    search_quality: str = None,         # Quality preset (see below)
)
```

**Parameters:**

| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `path` | `str` | `"./velesdb_data"` | Path to database directory |
| `collection_name` | `str` | `"llamaindex"` | Name of the collection |
| `metric` | `str` | `"cosine"` | Distance metric: `cosine`, `euclidean`, `dot` (aliases: `dotproduct`, `inner`, `ip`), `hamming`, `jaccard` |
| `storage_mode` | `str` | `"full"` | Storage mode: `full`/`f32`, `sq8`/`int8` (4× compression), `binary`/`bit` (32× compression), `pq` (8-32× compression), `rabitq` (32× with scalar correction) |
| `search_quality` | `str \| None` | `None` | Quality preset: `fast`, `balanced`, `accurate`, `perfect`, `autotune`, `custom:N`, `adaptive:MIN:MAX` |

**Methods:**

| Method | Description |
|--------|-------------|
| **Core Operations** | |
| `add(nodes)` | Add nodes with embeddings |
| `add_bulk(nodes)` | Bulk insert (2-3x faster for large batches) |
| `delete(ref_doc_id)` | Delete by document ID |
| `get_nodes(node_ids)` | Retrieve nodes by their IDs |
| `flush()` | Flush pending changes to disk |
| **Search** | |
| `query(query)` | Query with vector |
| `batch_query(queries)` | Batch query multiple vectors in parallel |
| `multi_query_search(query_embeddings, ...)` | **Multi-query fusion search** ⭐ NEW |
| `hybrid_query(query_str, query_embedding, ...)` | Hybrid vector+BM25 search |
| `text_query(query_str, ...)` | Full-text BM25 search |
| `velesql(query_str, params)` | Execute VelesQL query |
| **Utilities** | |
| `get_collection_info()` | Get collection metadata |
| `is_empty()` | Check if collection is empty |
| `scroll(batch_size)` | Iterate all points in stable batches without a query vector |

## Advanced Features

### Multi-Query Fusion (MQG)
Search with multiple query embeddings and fuse results using various strategies.
Perfect for RAG pipelines using Multiple Query Generation (MQG).

```python
from llamaindex_velesdb import VelesDBVectorStore

vector_store = VelesDBVectorStore(path="./velesdb_data")

# Basic usage with RRF (Reciprocal Rank Fusion)
results = vector_store.multi_query_search(
    query_embeddings=[emb1, emb2, emb3],  # Multiple query reformulations
    similarity_top_k=10,
    fusion="rrf",
    fusion_params={"k": 60}
)

# With weighted fusion (like SearchXP's scoring)
results = vector_store.multi_query_search(
    query_embeddings=[emb1, emb2],
    similarity_top_k=10,
    fusion="weighted",
    fusion_params={
        "avg_weight": 0.6,   # Average score weight
        "max_weight": 0.3,   # Maximum score weight  
        "hit_weight": 0.1,   # Hit ratio weight
    }
)

for node in results.nodes:
    print(f"{node.metadata}: {node.text[:50]}...")
```

**Fusion Strategies:**
- `"rrf"` - Reciprocal Rank Fusion (default, robust to score scale differences)
- `"average"` - Mean score across all queries
- `"maximum"` - Maximum score from any query
- `"weighted"` - Custom combination of avg, max, and hit ratio
- `"relative_score"` - Linear blend of dense and sparse scores

```python
# Relative Score Fusion
results = vector_store.multi_query_search(
    query_embeddings=[emb1, emb2],
    similarity_top_k=10,
    fusion="relative_score",
    fusion_params={"dense_weight": 0.7, "sparse_weight": 0.3}
)
```

### Advanced Search

#### `search_quality` — Quality Presets

Control the recall/latency trade-off for all queries with a single parameter set
at construction time or overridden per-call via `query()` kwargs.

```python
# Set once on the store — applies to every query() call
vector_store = VelesDBVectorStore(
    path="./velesdb_data",
    search_quality="accurate",   # higher recall at the cost of latency
)

q = VectorStoreQuery(query_embedding=embedding, similarity_top_k=10)
results = vector_store.query(q)

# Override per-call via kwargs
results = vector_store.query(q, search_quality="fast")
```

Accepted values:

| Value | Description |
|-------|-------------|
| `"fast"` | Lowest latency, reduced recall |
| `"balanced"` | Balanced latency/recall |
| `"accurate"` | Higher recall, higher latency |
| `"perfect"` | Exhaustive search, maximum recall |
| `"autotune"` | Runtime-adaptive quality |
| `"custom:N"` | Explicit ef_search (e.g. `"custom:256"`) |
| `"adaptive:MIN:MAX"` | Adaptive ef range (e.g. `"adaptive:32:512"`) |

#### `query_with_ef(query_embedding, ef_search, top_k)`

Search with an explicit HNSW `ef_search` parameter to trade query latency for recall.

```python
# Higher ef_search = better recall, slower query
results = vector_store.query_with_ef(
    query_embedding=embedding,
    ef_search=256,
    top_k=10
)
```

#### `query_ids(query_embedding, top_k)`

Search returning only node IDs and scores — no payloads transferred.
Faster than `query()` when only IDs are needed (e.g., for post-processing pipelines).

```python
hits = vector_store.query_ids(query_embedding=embedding, top_k=50)
# [{"id": "abc", "score": 0.92}, ...]
```

### Hybrid Search (Vector + BM25)

```python
from llamaindex_velesdb import VelesDBVectorStore

vector_store = VelesDBVectorStore(path="./velesdb_data")

# Hybrid search combining semantic and keyword matching
results = vector_store.hybrid_query(
    query_str="machine learning optimization",
    query_embedding=embedding_model.get_query_embedding("machine learning optimization"),
    similarity_top_k=10,
    vector_weight=0.7  # 70% vector, 30% BM25
)
for node in results.nodes:
    print(node.text)
```

### Full-Text Search (BM25)

```python
# Pure keyword-based search without embeddings
results = vector_store.text_query(
    query_str="VelesDB performance",
    similarity_top_k=5
)
```

### Cross-Collection MATCH

The `velesql()` method runs single-collection VelesQL/MATCH queries against the
vector store's own collection:

```python
results = vector_store.velesql(
    "MATCH (p:Product)-[:STORED_IN]->(w:Warehouse) RETURN p.name, w.city LIMIT 20"
)
for row in results:
    print(row["p.name"], row["w.city"])
```

> **Cross-collection `@collection` MATCH is not available through the LlamaIndex
> integration.** `velesql()` delegates to a single `velesdb.Collection`, which
> cannot resolve `@collection`-annotated nodes from *other* collections — that
> requires `Database`-level routing. For cross-collection MATCH, use the core
> `velesdb.Database` API or the [REST server](https://github.com/cyberlife-coder/VelesDB)
> directly. (Tracked in `docs/reference/ECOSYSTEM_PARITY.md`, action item #7.)

## Graph Retrieval

Three graph helpers ship with the package: `GraphLoader` (build a knowledge
graph over stored nodes), `GraphRetriever` (vector seed + graph expansion)
and `GraphQARetriever` (adds depth-based re-ranking). Native mode talks to
the VelesDB graph layer directly — no server required. Create the graph
collection once with the core API, then link the **VelesDB point IDs** of
stored nodes (the hash of each LlamaIndex node id, via
`velesdb_common.ids.stable_hash_id`):

```python
from llamaindex_velesdb import VelesDBVectorStore, GraphLoader

vector_store = VelesDBVectorStore(path="./velesdb_data", collection_name="docs")
vector_store._get_db().create_graph_collection("kg")  # once

loader = GraphLoader(vector_store, graph_collection_name="kg")
loader.add_edge(edge_id=1, source=source_id, target=target_id, label="RELATES_TO")
edges = loader.get_edges(label="RELATES_TO")
```

`GraphRetriever` finds seed nodes by vector search, then expands the context
through the graph:

```python
from llama_index.core import VectorStoreIndex
from llamaindex_velesdb import GraphRetriever

index = VectorStoreIndex.from_vector_store(vector_store)
retriever = GraphRetriever(
    index=index,
    mode="native",
    graph_collection_name="kg",
    seed_k=3,        # initial vector hits
    expand_k=10,     # max nodes after expansion
    max_depth=2,     # BFS traversal depth
)
nodes = retriever.retrieve("What is VelesDB?")
```

`GraphQARetriever` re-ranks results by graph depth (seeds first) for Q&A:

```python
from llamaindex_velesdb import GraphQARetriever

retriever = GraphQARetriever(
    index=index,
    mode="native",
    graph_collection_name="kg",
    rerank_by_depth=True,
)
nodes = retriever.retrieve("How are these concepts related?")
```

Pass `low_latency=True` to either retriever to skip graph expansion and
return vector-only seeds (no graph collection required).

## Performance

Measured on CI runners (ubuntu-latest, 2-core). Local hardware will be faster.
Source: `benchmarks/baseline.json`.

| Operation | Latency | Notes |
|-----------|---------|-------|
| Search (10K / 128D, k=10) | ~0.34 ms | HNSW + SIMD cosine |
| Hybrid (vector + filter) | ~0.27 ms | Filtered vector search |
| Batch insert (10K / 128D) | ~9 s total | Sequential HNSW build |

## Agent Memory (optional)

`llamaindex-velesdb` re-exports three agent-memory wrappers around VelesDB's
native memory subsystems. They are imported lazily — if the underlying
`velesdb` native extension isn't installed, the import becomes a no-op and
the symbols are exposed as `None`.

```python
from llamaindex_velesdb import (
    VelesDBSemanticMemory,       # long-term knowledge store
    VelesDBEpisodicMemory,       # time-sequenced event recall
    VelesDBProceduralMemory,     # learned procedures with reinforcement
)
```

See `llamaindex_velesdb/memory.py` for the full per-class API. The
companion `GraphLoader`, `GraphRetriever`, and `GraphQARetriever` exports
(top of the public namespace) are documented in the Graph Retrieval
section above.

## Comparison with Other Stores

| Feature | VelesDB | Chroma | Pinecone |
|---------|---------|--------|----------|
| **Deployment** | Local binary | Docker | Cloud |
| **Cost** | Free | Free | $$$  |
| **Offline** | ✅ | ✅ | ❌ |

## License

MIT License (this integration)

VelesDB Core and Server are licensed under VelesDB Core License 1.0 (source-available). See the [root LICENSE](../../LICENSE) for details.
