HyperStreamDB Comprehensive Guide¶

Version: 0.1.10 (Alpha)
Last Updated: 2026-04-03

HyperStreamDB is a serverless, hybrid-search database optimized for high-performance vector and scalar queries directly on data lakes (S3, GCS, Azure, Local).

1. Architecture Overview¶

HyperStreamDB decouples compute from storage, allowing for infinite scaling and zero-copy integration with data lakes.

  • Storage-Native: Data and indices are stored as Parquet and custom index files in your object store.

  • Serverless: No long-running daemons required. Operations spin up, execute, and spin down.

  • Hybrid Indices:

    • HNSW-IVF: For approximate nearest neighbor (vector) search.

    • Inverted Index: For ultra-fast scalar filtering on high-cardinality columns.

    • Roaring Bitmaps: For boolean and categorical filtering.

  • Engine: Built on Rust with Apache Arrow and DataFusion for vectorized query execution.

2. Getting Started¶

Installation¶

Prerequisites: Rust toolchain (latest stable).

# Build from source
cargo build --release
pip install .

Basic Usage (Python)¶

import hyperstreamdb as hdb
import pyarrow as pa
import pandas as pd
import numpy as np

# 1. Create a Table with AG News Schema
schema = pa.schema([
    ('id', pa.int32()), 
    ('label', pa.int32()),   # 1:World, 2:Sports, 3:Business, 4:Sci/Tech
    ('title', pa.string()),
    ('description', pa.string()),
    ('embedding', pa.list_(pa.float32(), 384)) # SBERT/all-MiniLM-L6-v2 size
])

table = hdb.Table.create("file:///tmp/ag_news", schema)

# 2. Ingest Real Data (Example: AG News Sample)
df = pd.DataFrame({
    'id': [1, 2],
    'label': [3, 4],
    'title': ["Wall St. Bears Claw Back", "SpaceX Launches New Falcon"],
    'description': ["Stocks fell today as inflation concerns...", "The private space company successfully..."],
    'embedding': [np.random.rand(384).tolist() for _ in range(2)]
})

table.write(df)
table.commit()

# 3. Hybrid Search (Scalar + Vector)
# Search for "Space" related news in "Sci/Tech" category (label=4)
query_vec = np.random.rand(384).tolist()
results = table.search(
    vector_column="embedding",
    query_vector=query_vec,
    k=5,
    filter="label = 4 AND description LIKE '%Space%'"
)
print(results.to_pandas()[['title', 'description']])

3. Key Features¶

3.1 SQL Support¶

HyperStreamDB integrates with Apache DataFusion to support full SQL queries with pgvector-compatible syntax.

Basic SQL Queries¶

session = hdb.Session()
session.register_table("my_table", table)

df = session.sql("""
    SELECT id, content 
    FROM my_table 
    WHERE id > 500 
    ORDER BY id DESC 
    LIMIT 10
""")

pgvector SQL Operators¶

HyperStreamDB provides full pgvector compatibility for vector operations:

# Vector similarity search with distance operators
results = session.sql("""
    SELECT id, content,
           embedding <-> '[0.1, 0.2, 0.3]'::vector AS l2_distance,
           embedding <=> '[0.1, 0.2, 0.3]'::vector AS cosine_distance
    FROM documents
    WHERE category = 'science'
    ORDER BY l2_distance
    LIMIT 10
""")

Supported Distance Operators:

  • <-> L2 (Euclidean) distance

  • <=> Cosine distance

  • <#> Inner product

  • <+> L1 (Manhattan) distance

  • <~> Hamming distance

  • <%> Jaccard distance

Vector Aggregations:

# Compute category centroids
results = session.sql("""
    SELECT category, 
           vector_avg(embedding) AS centroid,
           COUNT(*) AS doc_count
    FROM documents
    GROUP BY category
""")

Configuration Parameters:

# Tune search accuracy/speed tradeoff
session.set_config("hnsw.ef_search", 128)  # Higher = more accurate
session.set_config("ivf.probes", 20)       # Higher = more accurate

See pgvector SQL Guide for complete documentation.

3.2 Hardware Acceleration¶

The indexing engine supports hardware acceleration for multiple backends:

  • CUDA: NVIDIA GPUs (Linux, Windows via WSL2)

  • Metal: Apple Silicon (MPS)

  • ROCm: AMD GPUs

  • Intel: AVX-512 optimizations

Enable via Cargo.toml features or environment detection.

3.3 Multi-Catalog Support¶

HyperStreamDB is designed to integrate seamlessly with standard data catalogs to provide discovery, cross-table atomicity, and consistent metadata across the enterprise. We support a variety of industry-standard protocols.

Below is a detailed example using the Hive Metastore, followed by short-form examples for other supported catalogs. Full integration guides for each will be provided in future updates.

Hive Metastore (Detailed Example)¶

Connecting to a Hive Metastore allows you to resolve table names to storage locations automatically.

import hyperstreamdb as hdb

# Load a table from Hive Metastore
table = hdb.Table.from_hive(
    address="thrift://localhost:9083", 
    namespace="default", 
    table="my_analytics_table"
)

# Any writes will now be atomically committed back to Hive
df = table.to_pandas(filter="status = 'active'")

AWS Glue, Nessie, and REST Catalogs¶

HyperStreamDB also provides native support for cloud-modern catalogs. These can be configured similarly to the Hive example:

# AWS Glue (Native AWS Integration)
table = hdb.Table.from_glue(namespace="prod", table="users")

# Project Nessie (Git-like Versioning)
table = hdb.Table.from_nessie(nessie_url, namespace="dev", table="experiments")

# Iceberg REST Catalog (Standard API)
table = hdb.Table.from_rest(rest_url, namespace="marketing", table="campaigns")

# Unity Catalog (Databricks Ecosystem)
table = hdb.Table.from_unity(unity_url, namespace="main", table="gold_data")

For more details on advanced configurations and authentication (Kerberos, SASL, IAM), see the Configuration Guide or the Catalog Usage Guide.

4. Operational Tooling¶

A dedicated CLI hdb is provided for management.

CLI Commands¶

# Inspect table metadata/stats
hdb table inspect --uri s3://bucket/table

# Compaction (Merge small files)
hdb table compact --uri s3://bucket/table

# Vacuum (Cleanup old files)
hdb table vacuum --uri s3://bucket/table --older-than-days 7

Metrics & Tracing¶

  • Metrics: Prometheus endpoint compatible. Tracks ingestion rates, query latency, and compaction times.

  • Tracing: OpenTelemetry (OTLP) support.

    • Enable: export JAEGER_ENABLED=true

    • Sends traces to localhost:4317 (OTLP/gRPC).

5. Performance Tips¶

  1. Index Everything: Ensure vector columns have HNSW indices and scalar filter columns have Inverted indices.

    • table.create_index("col", index_type="hnsw")

  2. Projection: Always specify columns=[...] in read operations to avoid reading unused large embeddings.

  3. Compaction: Run compaction regularly to keep file count low and query performance high.

6. Roadmap Status¶

Feature

Status

Core Vector Search

✅

Hybrid Filtering

✅

Native Partitioning

✅

Hardware Accel (GPU)

✅

SQL Engine

✅

Catalogs (Glue/Unity/etc)

✅

CLI & Observability

✅

Spark/Trino Connectors

🚧 (APIs ready)

7. Partitioning Strategy¶

HyperStreamDB supports coarse-grained pruning via table partitioning. This allows the query planner to skip entire directories of data without reading file headers.

Creating a Partitioned Table¶

spec = {
    "fields": [
        {"name": "category", "transform": "identity"}
    ]
}
table = hdb.Table.create_partitioned("s3://bucket/table", schema, spec)

Benefits¶

  • Massive Scale: Scan millions of files in milliseconds by pruning based on high-level keys.

  • S3 Lifecycle: Easily move old partitions (e.g., date=2023) to colder storage classes.

  • Isolation: Writes to different partitions never conflict, enabling high-concurrency ingestion.