## rustvdb: A High-Performance Embedding Management and Search Library for Python

**rustvdb** is a Python module implemented in Rust, designed to efficiently manage and search through high-dimensional embeddings. It leverages Rust's performance capabilities while providing a user-friendly Python interface. This library supports various operations, including adding, updating, deleting embeddings, performing brute-force and cluster-based searches, and evaluating search algorithm recall.

### Features

- **Embedding Management:** Add, update, and delete embeddings with associated content and metadata.
- **Efficient Search:** Perform brute-force search or more efficient cluster-based search for finding similar embeddings.
- **Clustering:** Supports K-means clustering of embeddings for optimized search.
- **Recall Testing:** Evaluate the effectiveness of search algorithms with recall tests.
- **Vertex AI Integration (Optional):** Generate embeddings using Google Cloud Vertex AI.
- **Memory-Mapped File Storage:** Efficiently save and load embeddings to/from disk.

### Installation

**Prerequisites:**

- Rust and Cargo installed on your system.

**Installation:**

1. Clone the repository:
Use code with caution.
﻿
bash
git clone https://github.com/your-username/rustvdb.git
﻿
2. Build and install the Python package:
Use code with caution.
﻿
bash
cd rustvdb
pip install .
﻿
### Usage

**Basic Usage:**
Use code with caution.

```python
import rustvdb

# Create an EmbeddingManager instance
manager = rustvdb.EmbeddingManager()

# Add an embedding
manager.add_embedding("id1", "content1", [0.1, 0.2, 0.3], {"meta1": "value1"})

# Perform brute-force search
results = manager.brute_force_search([0.1, 0.2, 0.3], 5, "full")

# Create clusters
manager.create_clusters(3, 100)

# Perform cluster-based search
cluster_results = manager.cluster_search([0.1, 0.2, 0.3], 2, 5, "full", False)

# Evaluate recall
recall_score = manager.recal_test(20, 2, 5)

# Save embeddings to file
manager.save_to_file()

# Load embeddings from file
manager.load_from_file()
```
**Vertex AI Integration (Optional):**

1. Initialize Vertex AI environment:
Use code with caution.

```python
manager.init_vertex_embedding(project_id="your-project-id", location="us-central1", qps=10)
```
2. Generate and add embeddings using Vertex AI:
Use code with caution.

```python
embedding = manager.vertex_embedding("id2", "title2", "content2", {"meta2": "value2"}, "retrieve")
```
### Configuration

Logging is configured through the `env_logger` crate. Set the `RUST_LOG` environment variable to control logging verbosity:
Use code with caution.

```bash
export RUST_LOG=info
```

### Contributing

Contributions are welcome! Please submit pull requests with any enhancements, bug fixes, or documentation improvements.

### License

Apache 2.0

**This README was generated by Gemini 1.5 pro.**