Welcome to HyperStreamDB

HyperStreamDB is a serverless, hybrid-search database optimized for high-performance vector and scalar queries directly on data lakes (S3, GCS, Azure, Local).

Built on Rust with Apache Arrow and DataFusion, it provides ultra-fast indexing and retrieval without the overhead of traditional database servers.

Key Features

• Hybrid Vector Search: Approximate Nearest Neighbor (ANN) search with HNSW-IVF.

• Vectorized SQL: Full SQL support with pgvector-compatible operators.

• Storage-Native: Native support for Iceberg and Parquet formats.

• Hardware Acceleration: Blazing fast search using CUDA, Metal, ROCm, and AVX-512.

• Transactional Snapshots: ACID-compliant updates via Optimistic Concurrency Control.

• Multi-Catalog Support: Seamless integration with AWS Glue, Nessie, and Hive Metastore.

Getting Started

• Installation
  • GPU Acceleration (Optional)
  • pgvector SQL Compatibility
  • Basic Usage
• 🔄 Fluent Query API
  • Python Fluent API
  • Rust Fluent API
  • Benefits
  • Python Vector Distance API with GPU Acceleration
• SQL queries (full DataFusion support with pgvector syntax)
• Optional: Enable GPU acceleration for SQL queries
• Simple SQL
• Vector similarity search with pgvector operators (GPU-accelerated)
• Joins (uses Index Nested Loop Join optimization)
• Maintenance
  • Phase 2: Nessie Integration (Next)
  • Phase 3: Production Hardening
  • 🏗️ Architecture
  • 🔌 Connectors
  • 🔨 Building Connectors
  • 🧪 Development
  • 📈 Roadmap
  • 🤝 Contributing
  • 📄 License
  • 🙏 Acknowledgments
• HyperStreamDB Comprehensive Guide
  • 1. Architecture Overview
  • 2. Getting Started
  • 3. Key Features
  • 3.3 Multi-Catalog Support
  • 4. Operational Tooling
  • 5. Performance Tips
  • 6. Roadmap Status
  • 7. Partitioning Strategy

User Guides

• Python Vector Distance API Reference
  • Overview
  • Supported Distance Metrics
  • GPU Backend Support
  • Installation
  • Quick Start
  • API Reference
  • Performance Optimization
  • Integration with SQL
  • Troubleshooting
  • Examples
  • See Also
• GPU Setup Guide for HyperStreamDB
  • Overview
  • Installation
  • AMD ROCm Setup
  • Apple Metal (MPS) Setup
  • Intel XPU Setup
  • Troubleshooting
  • Performance Benchmarks
  • Best Practices
  • See Also
• Configuration Guide
  • Environment Variables
  • The hyperstream.toml File
  • Storage Credentials
  • Query Configuration (QueryConfig)
• Concurrency and Atomic Commits
  • Optimistic Concurrency Control (OCC)
  • Catalog-Level Locking
  • Read Isolation

API Reference

• Python API Reference
  • Table
  • Query
  • Session
  • Embedding Registry
• Rust API Reference

Roadmap

• Project Roadmap