Before benchmarking: verify real inference
If you have not used MemVanta before, first build the project and run memvanta_real against a supported Llama-family GGUF model. This separates basic model-compatibility problems from benchmark-methodology problems.
What to measure
- Peak resident set size across repeated measured runs.
- Prompt-processing throughput.
- Token-generation throughput.
- Where practical on Linux, success or failure under enforced cgroup-v2 memory ceilings with swap disabled.
The goal is not to find only the most favorable run. Keep warm-up runs separate, retain every measured run, and preserve failed or OOM-killed attempts in constrained-memory sweeps.
Keep the comparison controlled
Use the identical GGUF file for MemVanta and the comparison runtime. Record the model filename, byte size, and SHA-256 digest. Pin the exact comparison-runtime commit and keep CPU-only execution, thread count, prompt length, generated-token count, context size, batch size, and KV-cache precision aligned whenever both runtimes support the same setting.
For peak-RSS comparisons, the project protocol calls for at least one warm-up and five measured runs.
Evidence to publish
- MemVanta commit SHA
- Comparison-runtime commit SHA
- Model filename and SHA-256
- CPU model and core counts
- Operating system and kernel
- Total RAM and swap state
- Compiler and build type
- Complete command lines
- Context, batch, and token counts
- Raw outputs for every run
- Machine-readable CSV or JSON summary
- cgroup configuration when used
How to report disagreement
A reproduction is valuable whether it confirms, narrows, or contradicts the current result. If you observe a materially different memory or throughput profile, publish it with the same evidence rather than filtering it out. Methodology corrections are welcome.
For constrained-memory testing, report the actual tested ceilings and outcomes. Do not convert the lowest successful tested ceiling into an exact minimum physical-RAM requirement.
Reference result
The current canonical OpenLLaMA 7B v2 Q4_0 comparison reports {{MEMVANTA_RSS_GIB}} GiB peak RSS for MemVanta and {{LLAMA_RSS_GIB}} GiB for pinned llama.cpp, with llama.cpp substantially faster in throughput. Use the raw repository evidence as the reference rather than copying numbers from screenshots or third-party summaries.