Metadata-Version: 2.4
Name: microvec
Version: 0.2.0
Summary: A lightweight, production-grade in-memory vector database
Project-URL: Homepage, https://github.com/huolter/microVector
Project-URL: Repository, https://github.com/huolter/microVector
Project-URL: Bug Tracker, https://github.com/huolter/microVector/issues
Project-URL: Changelog, https://github.com/huolter/microVector/blob/main/CHANGELOG.md
Author: huolter
License: MIT
Keywords: database,embeddings,rag,search,similarity,vector
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Requires-Dist: numpy>=1.21.0
Provides-Extra: dev
Requires-Dist: black>=23.0; extra == 'dev'
Requires-Dist: mypy>=1.5; extra == 'dev'
Requires-Dist: pytest-cov>=4.1; extra == 'dev'
Requires-Dist: pytest>=7.4; extra == 'dev'
Requires-Dist: ruff>=0.1.0; extra == 'dev'
Description-Content-Type: text/markdown

# MicroVector

A lightweight, production-grade in-memory vector database for Python.

```
pip install microvector
```

No external services. No complex setup. Just numpy and your embeddings.

---

## Features

- **Fast similarity search** — cosine, euclidean, and dot product metrics
- **Rich results** — every search result includes the document text and metadata, not just an index
- **Batch operations** — insert thousands of vectors in one call
- **Metadata filtering** — filter candidates before scoring with any Python predicate
- **Safe persistence** — JSON + numpy format (no pickle, no security risk)
- **Full type hints** — works great with mypy and IDEs
- **Zero required dependencies** beyond numpy

---

## Quick Start

microvec stores and searches vectors — you bring the embeddings from any model you like.

### With OpenAI

```python
import numpy as np
from openai import OpenAI
from microvector import MicroVectorDB

client = OpenAI()  # uses OPENAI_API_KEY env var

def embed(text: str) -> np.ndarray:
    result = client.embeddings.create(input=text, model="text-embedding-3-small")
    return np.array(result.data[0].embedding, dtype=np.float32)

db = MicroVectorDB()  # dimension inferred from first insert
db.add_node(embed("Paris is the capital of France"), "Paris is the capital of France")
db.add_node(embed("The Eiffel Tower is in Paris"),   "The Eiffel Tower is in Paris")
db.add_node(embed("Python is a programming language"), "Python is a programming language")

results = db.search_top_k(embed("What city is the Eiffel Tower in?"), k=2)
for r in results:
    print(f"{r.score:.3f}  {r.document}")
# 0.921  The Eiffel Tower is in Paris
# 0.887  Paris is the capital of France
```

### With a local model (no API key)

```python
import numpy as np
from sentence_transformers import SentenceTransformer
from microvector import MicroVectorDB

model = SentenceTransformer("all-MiniLM-L6-v2")  # pip install sentence-transformers

def embed(text: str) -> np.ndarray:
    return model.encode(text, normalize_embeddings=True)

db = MicroVectorDB()  # dimension inferred from first insert
db.add_node(embed("Paris is the capital of France"), "Paris is the capital of France")

results = db.search_top_k(embed("What is the capital of France?"), k=1)
print(results[0].document)  # Paris is the capital of France
```

### Save and reload

```python
db.save("my_knowledge_base")
db = MicroVectorDB.load("my_knowledge_base")
```

---

## Installation

**Requires Python 3.9+ and numpy >= 1.21.**

```bash
pip install microvector
```

Or from source:

```bash
git clone https://github.com/huolter/microVector
cd microVector
pip install -e ".[dev]"
```

---

## API Reference

### `MicroVectorDB(dimension=None)`

Create a new database. If `dimension` is omitted, it is inferred automatically from the first vector inserted and locked for all subsequent inserts.

```python
db = MicroVectorDB()           # recommended — dimension inferred from first insert
db = MicroVectorDB(dimension=512)  # or set explicitly
```

---

### `add_node(vector, document, metadata=None, num_bits=None) → int`

Add a single vector. Returns the assigned index.

```python
idx = db.add_node(
    vector=np.array([...]),
    document="The text this vector represents",
    metadata={"source": "wikipedia", "date": "2024-01"},
    num_bits=8,   # optional: apply 8-bit scalar quantization
)
```

| Parameter | Type | Description |
|-----------|------|-------------|
| `vector` | `np.ndarray` | 1-D array matching the database dimension |
| `document` | `str` | Text or identifier associated with this vector |
| `metadata` | `dict` | Optional key-value pairs stored alongside the vector |
| `num_bits` | `int` | Optional quantization (1–16 bits). Reduces memory at the cost of precision. |

---

### `add_nodes(vectors, documents, metadata=None) → list[int]`

Batch insert. All inputs are validated before any insertion occurs.

```python
import numpy as np

vectors = np.random.rand(1000, 512)
docs = [f"Document {i}" for i in range(1000)]
indices = db.add_nodes(vectors, docs)
```

---

### `search_top_k(query_vector, k, metric='cosine', filter_fn=None) → list[SearchResult]`

Find the top-k most similar vectors. Returns results sorted by score descending.

```python
results = db.search_top_k(query, k=5)
results = db.search_top_k(query, k=5, metric="euclidean")

# Filter before scoring — only consider documents tagged "news"
results = db.search_top_k(
    query, k=5,
    filter_fn=lambda node: node.metadata.get("type") == "news"
)
```

**Metrics:**

| Metric | Range | Notes |
|--------|-------|-------|
| `cosine` | [-1, 1] | Best for text embeddings. Direction-based, scale-invariant. |
| `euclidean` | (0, 1] | `1 / (1 + dist)`. Identical vectors = 1.0. |
| `dot` | (-∞, +∞) | Fastest. Best for pre-normalized unit vectors. |

**`SearchResult` fields:**

```python
result.index     # int   — node's assigned index
result.document  # str   — the document text
result.score     # float — similarity score (higher = more similar)
result.metadata  # dict  — metadata stored with this node
```

---

### `search_by_threshold(query_vector, min_score, metric='cosine') → list[SearchResult]`

Return all nodes with score >= `min_score`, sorted descending.

```python
results = db.search_by_threshold(query, min_score=0.8)
```

---

### `get_node(index) → Node`

Retrieve a node by index.

```python
node = db.get_node(42)
print(node.document, node.metadata)
```

---

### `update_node(index, vector=None, document=None, metadata=None)`

Update fields of an existing node. Omit fields to leave them unchanged.

```python
db.update_node(42, document="Updated text")
db.update_node(42, metadata={"status": "reviewed"})
```

---

### `remove_node(index)`

Remove a node. Its index is permanently retired (never reused).

```python
db.remove_node(42)
```

---

### `save(path)` / `MicroVectorDB.load(path)`

Persist to and restore from a `.mvdb/` directory. Safe format: JSON + numpy binary.

```python
db.save("my_db")              # creates my_db.mvdb/
db = MicroVectorDB.load("my_db")
```

The `.mvdb/` directory contains:
- `index.json` — node metadata and documents (human-readable)
- `vectors.npy` — all vectors as a 2-D numpy array

---

### Utility methods

```python
len(db)           # number of nodes
42 in db          # check if index exists
repr(db)          # MicroVectorDB(dimension=512, nodes=1000, next_index=1000)
db.stats()        # {'count': 1000, 'dimension': 512, 'next_index': 1000, ...}
```

---

## Exceptions

All exceptions inherit from `MicroVectorError`.

```python
from microvector import (
    MicroVectorError,
    DimensionMismatchError,  # wrong vector dimension
    EmptyDatabaseError,      # search on empty database
    NodeNotFoundError,       # get/update/remove non-existent index
)
```

---

## Design Notes

**Why in-memory?** MicroVector is designed for small-to-medium datasets (up to ~100k vectors) where the simplicity of pure-Python outweighs the need for a dedicated service. It starts instantly, needs no configuration, and is trivially embeddable in any Python application.

**Why not pickle?** Pickle can execute arbitrary code when loading untrusted files. MicroVector uses JSON + numpy binary format which is safe, portable, and inspectable with any text editor.

**Index monotonicity.** Deleted indexes are never reused. This prevents stale external references from silently pointing to new data.

---

## Roadmap

- [ ] Approximate nearest neighbor (HNSW)
- [ ] Voronoi cell indexing
- [ ] FAISS optional backend for large datasets
- [ ] Async search support
- [ ] Benchmarks

---

## License

MIT
