In-Memory Component Analysis: Does It Matter for Benchmarking?¶

Date: January 25, 2026
Purpose: Understand the in-memory component and its impact on benchmarking/positioning

What Is the In-Memory Component?¶

HyperStreamDB has two types of in-memory components:

1. Write Buffer + InMemoryVectorIndex (Temporary)¶

  • Purpose: Buffer writes before flush to disk

  • Lifetime: Temporary (cleared after flush)

  • Storage: RAM only (not persisted)

  • Use Case: Query buffered data immediately after write (before commit)

2. LRU Caches (Performance Optimization)¶

  • Purpose: Cache frequently accessed indexes/manifests

  • Lifetime: Evicted based on LRU policy

  • Storage: RAM (optional optimization)

  • Use Case: Reduce S3 reads for hot data

In-Memory Component Details¶

Write Buffer Architecture¶

// From src/table.rs
pub struct Table {
    write_buffer: Arc<RwLock<Vec<RecordBatch>>>,  // Buffered writes
    memory_index: Arc<RwLock<Option<InMemoryVectorIndex>>>,  // In-memory vector index
    // ...
}

How It Works:

  1. Write Path: Data is buffered in memory (write_buffer)

  2. Indexing: In-memory vector index built for buffered data (InMemoryVectorIndex)

  3. Query Path: Queries search both:

    • Disk: Persistent indexes on S3 (HNSW-IVF)

    • Memory: In-memory index for buffered data

  4. Flush: When buffer is flushed, memory index is cleared

Key Code:

// From src/table.rs:513-521
// 2. Search Memory
let memory_hits = {
    let idx = self.memory_index.read().unwrap();
    if let Some(mem_idx) = idx.as_ref() {
        mem_idx.search(&vs_params.query, vs_params.k)  // Brute-force search
    } else {
        vec![]
    }
};

InMemoryVectorIndex Implementation¶

// From src/index/memory.rs
pub struct InMemoryVectorIndex {
    vectors: Vec<f32>,  // Flat storage (not HNSW)
    count: usize,
    dim: usize,
}

impl InMemoryVectorIndex {
    pub fn search(&self, query: &[f32], k: usize) -> Vec<(usize, f32)> {
        // Brute-force L2 search using SIMD (AVX2/NEON)
        // Parallelized with Rayon
    }
}

Characteristics:

  • Brute-force search (not HNSW - too expensive for temporary buffer)

  • SIMD-accelerated (AVX2/NEON for distance calculations)

  • Parallelized (Rayon for multi-threading)

  • Temporary (cleared after flush)

Does It Matter for Benchmarking?¶

YES - But in a Specific Way¶

1. Write Responsiveness ⭐ Important¶

Impact:

  • Queries can return results immediately after write (before commit)

  • No need to wait for flush to query buffered data

  • Better user experience for streaming workloads

Benchmarking:

  • Write-then-query latency: Measure time from write to query result

  • Buffered query performance: Query performance on buffered data

  • Comparison: Qdrant has this too (always in-memory), but HyperStreamDB has it as a buffer

Key Message:

“Query data immediately after write (before commit) - same responsiveness as in-memory databases”

2. Hybrid Architecture ⭐ Important for Positioning¶

Impact:

  • Not the same as Qdrant: Qdrant keeps everything in memory permanently

  • Different model: HyperStreamDB uses memory as a temporary buffer, S3 for persistence

  • Best of both worlds: Responsiveness of in-memory + durability of object storage

Benchmarking:

  • Positioning: “Hybrid architecture - in-memory responsiveness + S3 durability”

  • Comparison: Show that buffered queries are fast (like Qdrant), but data is also durable (unlike Qdrant)

Key Message:

“In-memory responsiveness for buffered writes, S3 durability for persisted data”

3. Query Performance ⚠ Limited Impact¶

Impact:

  • Only affects queries on buffered data (before flush)

  • After flush, queries use persistent indexes on S3

  • Most production queries hit persistent indexes, not memory

Benchmarking:

  • Most benchmarks should test persistent indexes (after commit)

  • Buffered queries are a bonus, not the primary use case

  • Fair comparison: Compare persistent indexes vs competitors’ persistent storage

Key Message:

“Buffered queries are fast, but primary performance comes from persistent indexes”

Comparison to Competitors¶

Qdrant (In-Memory Database)¶

Aspect

Qdrant

HyperStreamDB

Primary Storage

RAM (in-memory)

S3 (object storage)

Memory Usage

All data in memory

Buffer only (temporary)

Durability

Requires persistence layer

Native S3 durability

Query Performance

Sub-ms (in-memory)

~50ms (S3 I/O)

Buffered Queries

N/A (always in-memory)

Fast (in-memory buffer)

Scale

RAM-limited

Petabyte-scale

Key Difference:

  • Qdrant: Everything in memory (primary storage)

  • HyperStreamDB: Memory as buffer, S3 as primary storage

Deep Lake / LanceDB (Data Lakes)¶

Aspect

Deep Lake/LanceDB

HyperStreamDB

Memory Component

Cache only

Write buffer + cache

Buffered Queries

No

Yes (immediate query after write)

Query Performance

Disk-based

Disk-based (with memory buffer)

Write Responsiveness

Wait for flush

Immediate (buffered)

Key Difference:

  • Deep Lake/LanceDB: No in-memory query capability

  • HyperStreamDB: Can query buffered data immediately

Strategic Positioning¶

How to Position In-Memory Component¶

Option 2: “Write Responsiveness”¶

Message:

“Query data immediately after write - no need to wait for commit.

HyperStreamDB maintains an in-memory index for buffered writes, allowing instant queries on recently written data while maintaining S3 durability for persisted data.”

Benefits:

  • Highlights user experience benefit

  • Shows competitive with in-memory databases for write responsiveness

  • Doesn’t overstate (it’s a buffer, not primary storage)

Option 3: “Don’t Emphasize” (If Confusing)¶

Message:

Focus on persistent indexes and S3 durability

In-memory component is an implementation detail, not a core differentiator

Benefits:

  • Simpler messaging

  • Focus on core value prop (indexes + S3)

  • Avoids confusion with in-memory databases

Benchmarking Recommendations¶

What to Benchmark¶

✅ Do Benchmark:¶

  1. Persistent Index Performance (Primary)

    • Query performance after commit (using S3 indexes)

    • This is the core value proposition

    • Compare to competitors’ persistent storage

  2. Write Responsiveness (Secondary)

    • Time from write to query result (buffered)

    • Shows in-memory buffer benefit

    • Compare to competitors that require flush

  3. Hybrid Query Performance (Unique)

    • Query that spans both buffered and persisted data

    • Shows seamless integration

    • Unique capability

⚠ Don’t Over-Emphasize:¶

  1. Buffered Query Performance Alone

    • This is not the primary use case

    • Most production queries hit persistent indexes

    • Could be misleading if over-emphasized

  2. In-Memory vs In-Memory Comparison

    • Don’t compare HyperStreamDB’s buffer to Qdrant’s primary storage

    • Different architectures (buffer vs primary storage)

    • Unfair comparison

Fair Benchmarking Methodology¶

For Persistent Index Benchmarks:¶

  1. Commit data first (flush buffer)

  2. Query persistent indexes (S3-based)

  3. Compare to competitors’ persistent storage

  4. Fair comparison: Same storage model (disk/object storage)

For Write Responsiveness Benchmarks:¶

  1. Write data (buffered)

  2. Query immediately (before commit)

  3. Measure latency (write → query result)

  4. Compare to competitors that require flush

  5. Note: This is a bonus feature, not primary use case

Key Insights¶

1. It’s a Buffer, Not Primary Storage¶

Important Distinction:

  • Qdrant: Memory is primary storage (all data in RAM)

  • HyperStreamDB: Memory is a buffer (temporary, cleared after flush)

  • Different architectures: Don’t compare directly

Positioning:

  • “In-memory responsiveness for buffered writes”

  • “S3 durability for persisted data”

  • “Best of both worlds”

2. Write Responsiveness is a Real Benefit¶

User Experience:

  • Can query data immediately after write

  • No need to wait for commit

  • Better for streaming workloads

Competitive Advantage:

  • Deep Lake/LanceDB: Must wait for flush

  • HyperStreamDB: Immediate query capability

  • Qdrant: Always in-memory (but different model)

3. Most Queries Hit Persistent Indexes¶

Reality:

  • In-memory buffer is temporary (cleared after flush)

  • Most production queries hit persistent indexes on S3

  • Buffered queries are a bonus, not primary use case

Benchmarking:

  • Focus on persistent index performance

  • Buffered queries are secondary benefit

  • Don’t over-emphasize in-memory component

Recommendations¶

For Benchmarking Strategy:¶

  1. Primary Benchmarks: Test persistent indexes (after commit)

    • This is the core value proposition

    • Fair comparison with competitors

    • Shows S3-based performance

  2. Secondary Benchmarks: Test write responsiveness (buffered)

    • Shows in-memory buffer benefit

    • Differentiates from Deep Lake/LanceDB

    • But note it’s a bonus feature

  3. Positioning: “Hybrid Architecture”

    • In-memory responsiveness + S3 durability

    • Best of both worlds

    • Not the same as pure in-memory databases

For Competitive Positioning:¶

Against Qdrant:

  • ✅ “In-memory responsiveness for buffered writes (like Qdrant)”

  • ✅ “But with S3 durability and petabyte scale (unlike Qdrant)”

  • ❌ Don’t say “same as Qdrant” (different architecture)

Against Deep Lake/LanceDB:

  • ✅ “Query data immediately after write (they can’t)”

  • ✅ “In-memory buffer for write responsiveness”

  • ✅ “Plus persistent indexes for durability”

Against Iceberg/Delta:

  • ✅ “In-memory buffer for write responsiveness”

  • ✅ “Plus persistent indexes for query performance”

  • ✅ “They have neither”

Conclusion¶

Does the In-Memory Component Matter?¶

YES, but strategically:

  1. For User Experience: ✅ Important

    • Write responsiveness is a real benefit

    • Better than Deep Lake/LanceDB

    • Competitive with Qdrant (for buffered queries)

  2. For Positioning: ✅ Important

    • “Hybrid architecture” differentiator

    • Best of both worlds messaging

    • But don’t over-emphasize

  3. For Benchmarking: ⚠ Secondary

    • Primary benchmarks should test persistent indexes

    • Buffered queries are a bonus feature

    • Don’t compare buffer to Qdrant’s primary storage

Key Takeaway¶

Position it as:

“Hybrid architecture: In-memory responsiveness for buffered writes + S3 durability for persisted data. Best of both worlds - fast writes and durable storage.”

Don’t position it as:

“Same as Qdrant” (it’s not - different architecture) “Primary storage in memory” (it’s a buffer, not primary storage)

Last Updated: January 25, 2026