Python API Reference

The HyperStreamDB Python package provides a high-level, Pandas-compatible interface for managing tables, executing hybrid searches, and integrating with data catalogs.

Table

class hyperstreamdb.Table(uri, inner_table=None, device=None, index_all=False, primary_key=None, explain=False)

Bases: object

HyperStreamDB Table — Apache Iceberg/Parquet-compatible columnar vector store.

Default behaviour (v0.4.1+)

• index_all = False — Vector indexes are not built automatically. Call table.index_all = True or table.add_index(column, 'hnsw') to enable indexing for a specific session or column.

• autocommit = False — Writes accumulate in an in-memory buffer. Call table.commit() (or await table.commit_async()) to persist data to Parquet and advance the Iceberg snapshot.

These defaults exist for performance: automatic indexing previously caused silent 15-18 s HNSW build latency on every commit() for tables with vector columns, even when the user had not requested an index.

Parameters:

• uri (str) – Table location (file:///path or cloud URI).

• inner_table (Table | None) – Internal — do not pass directly.

• device (Any | None) – Optional compute device for GPU-accelerated index builds.

• index_all (bool) – Enable automatic indexing of all compatible columns. Defaults to False. Set True to restore legacy behaviour.

• primary_key (str | None) – Column name (or list) to use as primary key.

• explain (bool) – If True, return query plans instead of results.

classmethod create(uri, schema, device=None)

Create a new table with an explicit schema.

Parameters:

• uri (str)

• device (Any | None)

Return type:

Table

classmethod create_partitioned(uri, schema, partition_spec, device=None)

Create a new table with an explicit schema and partitioning.

Parameters:

• uri (str)

• partition_spec (Dict[str, Any])

• device (Any | None)

Return type:

Table

classmethod register_external(uri, iceberg_metadata_uri, device=None)

Register an existing Iceberg table.

Parameters:

• uri (str)

• iceberg_metadata_uri (str)

• device (Any | None)

Return type:

Table

define_embedding(column, function, vector_column=None)

Link a source column to an embedding function for automatic vectorization.

Parameters:

• column (str) – The source text column.

• function (str | EmbeddingFunction) – Registered function name or EmbeddingFunction instance.

• vector_column (str | None) – Target vector column name (defaults to {column}_vector).

write(data, device=None, mode='append')

Write data to the table, automatically generating embeddings for configured columns.

Parameters:

• data (Any) – pandas.DataFrame, pyarrow.Table, polars.DataFrame, numpy.ndarray, torch.Tensor, or List[Dict].

• device (Any | None) – Optional Device for GPU acceleration.

• mode (str) – ‘append’ (default) or ‘overwrite’ (clears table first).

insert(data, device=None)

Alias for write() for compatibility with common vector DB APIs.

Parameters:

• data (Any)

• device (Any | None)

write_pandas(df, device=None)

High-level Pandas ingestion with auto-vectorization.

Parameters:

• df (DataFrame)

• device (Any | None)

write_arrow(table, device=None)

High-level Arrow ingestion with auto-vectorization.

Parameters:

• table (Table)

• device (Any | None)

upsert(data, key_column, mode='merge_on_read', device=None)

Update or insert data using a key column (or list of columns) to avoid duplicates.

Parameters:

• data (Any)

• key_column (str | List[str])

• mode (str)

• device (Any | None)

commit()

Commit temporary segments to the table.

truncate()

Clear all data from the table while keeping the schema.

vacuum(retention_versions=1)

Physically delete unreferenced data and manifest files to reclaim space.

Parameters:

retention_versions (int) – Number of snapshots to keep (default 1).

property autocommit: bool

Get or set the autocommit state of the table.

wait_for_background_tasks()

Wait for all background tasks (like index building) to complete.

delete(filter)

Delete rows matching the filter expression.

Parameters:

filter (str)

to_pandas(filter=None, vector_filter=None, columns=None, device=None, **kwargs)

Read table to Pandas with auto-vectorization of search queries and flexible parameters.

Parameters:

• filter (str | None) – Optional scalar WHERE clause (e.g., “category = ‘news’”)

• vector_filter (Dict[str, Any] | List[float] | None) – Dict with vector search params: - column: str (required) - vector column name - query: list (required) - query vector - k: int (required) - number of results - metric: str (optional) - ‘l2’|’cosine’|’innerproduct’|’l1’|’hamming’|’jaccard’ (default: l2) - ef_search: int (optional) - HNSW ef parameter for tuning - probes: int (optional) - IVF probes parameter for tuning

• columns (List[str] | None) – Optional list of column names to select

• device (Any | None) – Optional compute device (GPU/CPU)

• **kwargs – Extra params (merged into vector_filter if present)

Example:

# Vector search with cosine metric
df = table.to_pandas(vector_filter={
    "column": "embedding",
    "query": [1.0, 2.0, 3.0],
    "k": 5,
    "metric": "cosine",
    "ef_search": 200  # Tune HNSW search quality
})

to_arrow(filter=None, vector_filter=None, columns=None, device=None, **kwargs)

Read table to Arrow Table with auto-vectorization of search queries and flexible parameters.

Parameters:

• filter (str | None) – Optional scalar WHERE clause (e.g., “category = ‘news’”)

• vector_filter (Dict[str, Any] | List[float] | None) – Dict with vector search params: - column: str (required) - vector column name - query: list (required) - query vector - k: int (required) - number of results - metric: str (optional) - ‘l2’|’cosine’|’innerproduct’|’l1’|’hamming’|’jaccard’ (default: l2) - ef_search: int (optional) - HNSW ef parameter for tuning - probes: int (optional) - IVF probes parameter for tuning

• columns (List[str] | None) – Optional list of column names to select

• device (Any | None) – Optional compute device (GPU/CPU)

• **kwargs – Extra params (merged into vector_filter if present)

sql(query)

Execute a SQL query against the table. The table is registered as ‘t’.

Parameters:

query (str)

Return type:

Any

query()

Start a fluent query.

Return type:

Query

read(filter=None, vector_filter=None, columns=None, device=None, **kwargs)

Read table to Arrow Table (alias for to_arrow).

Parameters:

• filter (str | None)

• vector_filter (Dict[str, Any] | List[float] | None)

• columns (List[str] | None)

• device (Any | None)

vector_search(column, query, k=10, filter=None, columns=None, device=None, **kwargs)

Backward compatibility alias for to_pandas with vector filter.

Parameters:

• column (str)

• query (List[float])

• k (int)

• filter (str | None)

• columns (List[str] | None)

• device (Any | None)

search(column, query, k=10, filter=None, columns=None, device=None, **kwargs)

Alias for vector_search.

Parameters:

• column (str)

• query (List[float])

• k (int)

• filter (str | None)

• columns (List[str] | None)

• device (Any | None)

filter(expr=None, vector_filter=None, **kwargs)

Start a fluent query or apply immediate filters.

Parameters:

• expr (str | None)

• vector_filter (Dict[str, Any] | List[float] | None)

Return type:

Query

property primary_key

Get the current primary key column.

property index_all

Whether to build HNSW/BM25 indexes for all compatible columns on commit.

Defaults to False (opt-in). Setting this to True triggers background index builds after every commit() call — useful when you want fast ANN search but be aware of the additional commit latency (~15 s per 100 K rows with 768-dim vectors on CPU).

For selective indexing, prefer table.add_index(column, 'hnsw').

property row_count: int

Get total row count in the table.

property statistics

Get full table statistics.

add_index_columns(columns, tokenizer=None)

Add columns to the indexing configuration.

Parameters:

• columns (List[str]) – List of column names to index.

• tokenizer (str | None) – Optional tokenizer name from the registry.

set_index_config(column, enabled=True, tokenizer=None, device=None)

Set indexing configuration for a specific column. (Legacy compatibility wrapper)

Parameters:

• column (str)

• enabled (bool)

• tokenizer (str | None)

• device (str | None)

set_index_columns(config)

Update indexing specifications for multiple columns at once. Supports both simple strings and advanced configuration dictionaries.

Example:

table.set_index_columns({
    "embedding": IndexType.HNSW,
    "content": ["hnsw", "bm25"],
    "category": "bitmap"
})

Parameters:

config (Dict[str, str | List[str | Dict[str, Any]] | Dict[str, Any]])

add_index(column, algorithm='hnsw', **kwargs)

Add an indexing strategy to a column.

Parameters:

• column (str)

• algorithm (str | Dict[str, Any])

drop_index(column)

Remove all indexing strategies from a column.

Parameters:

column (str)

add_primary_key(column)

Atomically add a column to the primary key. This performs a validation check for duplicates across all existing data. If validation fails, the change is NOT committed.

Parameters:

column (str)

drop_primary_key(column)

Atomically remove a column from the primary key.

Parameters:

column (str)

set_sort_order(columns, ascending)

Set the table’s default sort order for future data writes.

Parameters:

• columns (List[str])

• ascending (List[bool])

set_partition_spec(spec)

Update the table’s partition specification.

Parameters:

spec (List[Dict[str, Any]]) – List of partition fields, each being a dict with: - source_id: int (or source_ids: List[int]) - name: str - transform: str - field_id: int (optional)

Query

class hyperstreamdb.Query(table, filter_expr=None)

Bases: object

Fluent Query interface for HyperStreamDB.

Parameters:

filter_expr (str | None)

filter(expr)

Apply a SQL-like filter expression.

Parameters:

expr (str)

Return type:

Query

vector_search(query, column=None, k=10, **kwargs)

Apply a vector search filter.

Parameters:

• query (List[float] | str) – The query vector (list of floats) or a string to be vectorized.

• column (str | None) – The vector column to search against.

• k (int) – Number of nearest neighbors to return.

• **kwargs – Additional parameters (e.g., n_probe).

Return type:

Query

select(columns)

Select specific columns to return.

Parameters:

columns (List[str])

Return type:

Query

to_pandas(device=None)

Execute the query and return results as a Pandas DataFrame.

Parameters:

device (Any | None)

to_arrow(device=None)

Execute the query and return results as an Arrow Table.

Parameters:

device (Any | None)

execute(device=None, to_arrow=False)

Execute the query and return results as a Pandas DataFrame (default) or Arrow Table.

Parameters:

• device (Any | None)

• to_arrow (bool)

Session

class hyperstreamdb.Session(memory_mb=None)

Bases: object

HyperStreamDB Query Session with integration for Python Table objects.

Parameters:

memory_mb (int | None)

register(name, table)

Register a table in the session for SQL queries.

Parameters:

• name (str)

• table (Table | Table)

sql(query)

Execute a SQL query against the table (registered as ‘t’).

Parameters:

query (str)

Return type:

Any

Embedding Registry

class hyperstreamdb.embeddings.EmbeddingFunction

Bases: ABC

Abstract base class for all embedding functions.

class hyperstreamdb.embeddings.HuggingFaceFunction(model_name='all-MiniLM-L6-v2', device='cpu', **kwargs)

Bases: EmbeddingFunction

Local embedding function using Sentence Transformers (supports all Hugging Face models). Examples: ‘all-MiniLM-L6-v2’, ‘BAAI/bge-large-en-v1.5’, ‘Qwen/Qwen-7B-Chat’ (if supported by ST)

Parameters:

• model_name (str)

• device (str)

class hyperstreamdb.embeddings.OpenAIEmbeddingFunction(model_name='text-embedding-3-small', api_key=None, **kwargs)

Bases: EmbeddingFunction

Embedding function using the OpenAI API.

Parameters:

• model_name (str)

• api_key (str | None)

class hyperstreamdb.embeddings.AnthropicEmbeddingFunction(model_name='voyage-2', api_key=None, **kwargs)

Bases: EmbeddingFunction

Embedding function using Anthropic/Claude (placeholder as Anthropic doesn’t have a direct embedding API yet). Often used in conjunction with Voyage AI or similar.

Parameters:

• model_name (str)

• api_key (str | None)

class hyperstreamdb.embeddings.GeminiEmbeddingFunction(model_name='models/embedding-001', api_key=None, **kwargs)

Bases: EmbeddingFunction

Embedding function using Google’s Gemini API.

Parameters:

• model_name (str)

• api_key (str | None)

class hyperstreamdb.embeddings.EmbeddingRegistry

Bases: object

Registry to manage and retrieve embedding functions.

register(name, func)

Register a new embedding function.

Parameters:

• name (str)

• func (EmbeddingFunction)

get(name)

Retrieve a registered embedding function.

Parameters:

name (str)

Return type:

EmbeddingFunction | None

hyperstreamdb.embeddings.get_registry()

Access the global embedding registry.