API Reference¶

This page provides an overview of HyperStreamDB APIs across different languages and interfaces.

Python API¶

Table Operations¶

import hyperstreamdb as hdb

# Create/open table
table = hdb.Table("s3://bucket/my-table")

# Write data
table.write_pandas(df)
table.write_arrow(arrow_table)

# Read data
df = table.to_pandas()
df = table.to_pandas(filter="id > 100")

# Vector search with default L2 distance
df = table.to_pandas(
    vector_filter={
        "column": "embedding",
        "query": query_vec,
        "k": 10
    }
)

# Vector search with custom metric and index parameters
df = table.to_pandas(
    vector_filter={
        "column": "embedding",
        "query": query_vec,
        "k": 10,
        "metric": "cosine",      # l2, cosine, inner_product, l1, hamming, jaccard
        "ef_search": 200,        # HNSW parameter (higher = more accurate, slower)
        "probes": 10             # IVF parameter (higher = more accurate, slower)
    }
)

# Hybrid query (scalar + vector)
df = table.to_pandas(
    filter="category = 'science'",
    vector_filter={
        "column": "embedding",
        "query": query_vec,
        "k": 10,
        "metric": "cosine"
    }
)

# Maintenance
table.compact()
table.expire_snapshots(retain_last=10)

Vector Distance API¶

See Python Vector API Documentation for complete reference.

import hyperstreamdb as hdb
import numpy as np

# Single-pair distance
distance = hdb.l2_distance(vec1, vec2)
distance = hdb.cosine_distance(vec1, vec2)

# Batch operations with GPU acceleration
ctx = hdb.GPUContext.auto_detect()
distances = hdb.l2_distance_batch(query, database, context=ctx)

# Sparse vectors
sparse = hdb.SparseVector(indices, values, dim)
distance = hdb.l2_distance_sparse(sparse1, sparse2)

# Binary vectors
distance = hdb.hamming_distance_packed(binary1, binary2)

Supported Distance Metrics:

l2_distance() - Euclidean distance
cosine_distance() - Cosine distance
inner_product() - Inner product
l1_distance() - Manhattan distance
hamming_distance() - Hamming distance
jaccard_distance() - Jaccard distance

GPU Backends:

CUDA (NVIDIA)
ROCm (AMD)
Metal/MPS (Apple Silicon)
Intel XPU (via WGPU)
CPU (fallback)

SQL API¶

import hyperstreamdb as hdb

# Create session
session = hdb.Session()
session.register("my_table", table)

# Execute SQL with pgvector operators
results = session.sql("""
    SELECT id, content,
           embedding <-> '[0.1, 0.2, 0.3]'::vector AS distance
    FROM my_table
    WHERE category = 'science'
    ORDER BY distance
    LIMIT 10
""")

# Enable GPU acceleration for SQL
ctx = hdb.GPUContext.auto_detect()
hdb.set_thread_gpu_context(ctx)

See pgvector SQL Guide for SQL syntax reference.

Iceberg V2/V3 API¶

import hyperstreamdb as hdb

table = hdb.Table("s3://bucket/table")

# Sort orders (V2)
table.set_sort_order(["timestamp", "user_id"], ascending=[False, True])

# Partition evolution (V2)
table.set_partition_spec([
    {"source_id": 1, "field_id": 1000, "name": "date", "transform": "day"}
])

# Row lineage (V3) - automatic when format_version >= 3
# Adds _row_id and _last_updated_sequence_number columns

# Standard Iceberg operations
table.update_spec(new_spec)
table.replace_sort_order(sort_order)
table.rewrite_data_files(filter_expr)
table.rollback_to_snapshot(snapshot_id)

See Iceberg V2/V3 API Guide for complete reference.

Spark Connector¶

// Read
val df = spark.read
  .format("hyperstream")
  .option("path", "s3://bucket/table")
  .load()

// Write
df.write
  .format("hyperstream")
  .option("path", "s3://bucket/table")
  .save()

// Vector search
df.createOrReplaceTempView("documents")
spark.sql("""
  SELECT id, content,
         embedding <-> array(0.1, 0.2, 0.3) AS distance
  FROM documents
  ORDER BY distance
  LIMIT 10
""")

Trino Connector¶

-- Query table
SELECT * FROM hyperstream.default.my_table
WHERE id > 100;

-- Vector search with pgvector operators
SELECT id, content,
       embedding <-> ARRAY[0.1, 0.2, 0.3] AS distance
FROM hyperstream.default.documents
WHERE category = 'science'
ORDER BY distance
LIMIT 10;

Configuration¶

GPU Context Configuration¶

import hyperstreamdb as hdb

# Auto-detect best backend
ctx = hdb.GPUContext.auto_detect()

# Specify backend explicitly
ctx = hdb.GPUContext("cuda", device_id=0)
ctx = hdb.GPUContext("rocm", device_id=0)
ctx = hdb.GPUContext("mps")
ctx = hdb.GPUContext("intel")

# List available backends
backends = ctx.list_available_backends()

# Performance monitoring
stats = ctx.get_stats()
print(f"GPU time: {stats['total_gpu_time_ms']}ms")
print(f"Kernel launches: {stats['kernel_launches']}")
ctx.reset_stats()

See GPU Setup Guide for installation and configuration.

Vector Index Configuration¶

import hyperstreamdb as hdb

table = hdb.Table("s3://bucket/table")

# TurboQuant 8-bit (4x compression)
table.add_index("embedding", "hnsw_tq8")

# TurboQuant 4-bit (8x compression)
table.add_index("embedding", "hnsw_tq4")

# Custom HNSW configuration
table.add_index(
    column="embedding",
    index_config={
        "type": "hnsw",
        "complexity": 16, # connections per node
        "quality": 200     # construction search width
    }
)

See Vector Configuration Guide for tuning parameters.

Error Handling¶

Python Exceptions¶

import hyperstreamdb as hdb

try:
    distance = hdb.l2_distance(vec1, vec2)
except ValueError as e:
    # Dimension mismatch, NaN/inf values, invalid input
    print(f"Invalid input: {e}")
except TypeError as e:
    # Invalid input types
    print(f"Type error: {e}")
except RuntimeError as e:
    # GPU errors, backend not available
    print(f"Runtime error: {e}")
except MemoryError as e:
    # GPU out of memory
    print(f"Memory error: {e}")

GPU Fallback¶

import hyperstreamdb as hdb

# GPU operations automatically fall back to CPU on error
ctx = hdb.GPUContext.auto_detect()
try:
    distances = hdb.l2_distance_batch(query, database, context=ctx)
except RuntimeError as e:
    print(f"GPU error, falling back to CPU: {e}")
    distances = hdb.l2_distance_batch(query, database)  # No context = CPU

Performance Tips¶

Use GPU for large batches: 10,000+ vectors for best speedup
Reuse GPU context: Don’t create new context for each operation
Use float32: Better GPU performance than float64
Batch operations: Process multiple queries together
Use sparse vectors: For high-dimensional sparse data
Use binary vectors: For binary features (SimHash, LSH)
Profile workload: Use ctx.get_stats() to monitor GPU usage