Open-source · C++20 · CPU · GGUF

MemVanta — Low-Memory CPU LLM Inference for GGUF

A memory-first C++ LLM inference runtime for quantized GGUF models on CPUs with limited RAM. MemVanta uses mmap-backed model access, paged KV cache, Q4/Q8 kernels, and bounded adaptive prefetching.

Run a GGUF model View on GitHub See 7B benchmark

{{RSS_REDUCTION}}% lower peak RSS in the published OpenLLaMA 7B v2 Q4_0 same-model CPU A/B test: {{MEMVANTA_RSS_GIB}} GiB for MemVanta vs {{LLAMA_RSS_GIB}} GiB for pinned llama.cpp. llama.cpp is substantially faster in this test.

OpenLLaMA 7B memory benchmark vs llama.cpp

{{MEMVANTA_RSS_GIB}} GiB
MemVanta peak RSS
{{LLAMA_RSS_GIB}} GiB
pinned llama.cpp peak RSS
{{RSS_REDUCTION}}%
peak-RSS reduction in this test
MetricMemVantapinned llama.cpp
OpenLLaMA 7B v2 Q4_0 peak RSS{{MEMVANTA_RSS_GIB}} GiB{{LLAMA_RSS_GIB}} GiB
Prompt processing{{MEMVANTA_PP}} tok/s{{LLAMA_PP}} tok/s
Token generation{{MEMVANTA_TG}} tok/s{{LLAMA_TG}} tok/s

Results are scoped to the tested model, settings, host, and pinned comparison runtime. Read the benchmark interpretation · Raw evidence

Run your first GGUF model

Build the project, then use memvanta_real for trained-model text generation with a supported Llama-family GGUF file.

git clone https://github.com/sauravsingla/MemVanta.git
cd MemVanta
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

./build/memvanta_real \
  --model /path/to/model.gguf \
  --prompt "Hello from MemVanta" \
  --n 64 --threads 4 --ctx 2048 --temperature 0

memvanta_real is the text-generation CLI. memvanta run <file> is a separate mapped streaming/cache telemetry path.

Getting-started guide

Memory-efficient GGUF inference on CPU

MemVanta is built for local AI, edge systems, and constrained machines where resident memory is a primary bottleneck. The project focuses on a transparent memory/performance trade-off rather than claiming throughput leadership.

Learn how the low-memory design works or explore the runtime architecture.

Current model support

Trained-model execution currently supports GGUF models with general.architecture=llama. The GGUF parser also validates pinned Qwen2 files, but Qwen2 inference is not implemented.

MemVanta is an active research prototype with trained-model evidence up to 7B. Independent reproduction is welcome.

Evidence and reproducibility

Published benchmark numbers are generated from machine-readable repository evidence. The methodology requires matched model artifacts and workloads, a pinned comparison runtime, repeated runs, and throughput reporting alongside memory.

Reproduction guideCite MemVanta