Metadata-Version: 2.4
Name: memvanta
Version: 0.8.3
Summary: Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU
Keywords: llm,gguf,cpu-inference,local-llm,low-memory,quantization,kv-cache,edge-ai,llama,inference
Author: Saurav Singla
License-Expression: Apache-2.0
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: C++
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Project-URL: Homepage, https://sauravsingla.github.io/MemVanta/
Project-URL: Repository, https://github.com/sauravsingla/MemVanta
Project-URL: Issues, https://github.com/sauravsingla/MemVanta/issues
Project-URL: Documentation, https://sauravsingla.github.io/MemVanta/
Project-URL: Changelog, https://github.com/sauravsingla/MemVanta/releases
Requires-Python: >=3.9
Description-Content-Type: text/markdown

# MemVanta

**Low-memory C++20 LLM inference runtime for quantized GGUF models on CPU.**

MemVanta is a memory-first local LLM runtime for running quantized Llama-family GGUF models on CPUs with limited RAM. It uses mmap-backed model access, paged KV cache, Q4/Q8 kernels, and bounded adaptive prefetching, with reproducible memory and throughput benchmarks against pinned `llama.cpp`.

[Website](https://sauravsingla.github.io/MemVanta/) · [Getting started](https://sauravsingla.github.io/MemVanta/getting-started/) · [Latest release](https://github.com/sauravsingla/MemVanta/releases/tag/v0.8.3) · [PyPI](https://pypi.org/project/memvanta/) · [7B benchmark](https://sauravsingla.github.io/MemVanta/benchmark/) · [DOI](https://doi.org/10.5281/zenodo.22886357) · [Reproduce](https://sauravsingla.github.io/MemVanta/reproduce/) · [Contributing](CONTRIBUTING.md) · [Discussions](https://github.com/sauravsingla/MemVanta/discussions)

## Why MemVanta?

MemVanta explores a specific systems trade-off: **how much resident memory can CPU LLM inference avoid while still executing a real quantized GGUF model correctly?**

It is designed for experiments where RAM pressure matters more than maximum token throughput, including constrained developer machines, edge systems, and research into memory-aware local inference.

The project is intentionally transparent about the cost of that trade-off. MemVanta is **memory-first**; it does not claim to be faster than `llama.cpp`.

## 7B memory benchmark vs llama.cpp

<!-- BEGIN_CANONICAL_7B_BENCHMARK -->
| Metric | MemVanta | pinned `llama.cpp` |
|---|---:|---:|
| OpenLLaMA 7B v2 Q4_0 peak RSS | **3.80 GiB** | 7.24 GiB |
| Prompt processing | 2.78 ± 0.00 tok/s | **12.86 ± 0.01 tok/s** |
| Token generation | 1.81 ± 0.00 tok/s | **8.14 ± 0.01 tok/s** |
| Peak-RSS reduction | **47.54%** | baseline |

Source of truth: [`results/openllama-7b-v2-ab/summary.json`](results/openllama-7b-v2-ab/summary.json). The README table is generated from that file; do not edit its numbers by hand.
<!-- END_CANONICAL_7B_BENCHMARK -->

The result above is a repeated same-model CPU A/B test on OpenLLaMA 7B v2 Q4_0. It applies to the tested model, workload, host, and pinned comparison runtime; it is not a universal memory-reduction claim.

A separate cgroup-v2 experiment also measured execution under tight memory limits. It is systems evidence, **not a physical-RAM requirement**.

[Benchmark details](https://sauravsingla.github.io/MemVanta/benchmark/) · [Raw evidence](results/openllama-7b-v2-ab/) · [Methodology](docs/MEMORY_BENCHMARKING.md)

## Install from PyPI (Linux x86-64)

MemVanta is also distributed on PyPI as a native Linux x86-64 CLI package. The initial PyPI package is a distribution shim for the C++ runtime; it is **not a separate Python inference implementation or Python API**.

```bash
python -m pip install memvanta
memvanta-real --help
```

The package exposes these console commands:

```text
memvanta
memvanta-real
memvanta-tokenize
memvanta-gguf-inspect
```

PyPI releases are built as manylinux wheels with `MEMVANTA_NATIVE=OFF`, installed and smoke-tested before publication, and published from GitHub Actions through PyPI Trusted Publishing rather than a long-lived API token.

## Download prebuilt release (Linux x86-64)

The `v0.8.3` pre-release includes a portable Linux x86-64 build (`MEMVANTA_NATIVE=OFF`) plus a SHA-256 checksum. Download, verify, and extract it with:

```bash
curl -fLO https://github.com/sauravsingla/MemVanta/releases/download/v0.8.3/memvanta-v0.8.3-linux-x86_64.tar.gz
curl -fLO https://github.com/sauravsingla/MemVanta/releases/download/v0.8.3/memvanta-v0.8.3-linux-x86_64.sha256
sha256sum -c memvanta-v0.8.3-linux-x86_64.sha256
tar -xzf memvanta-v0.8.3-linux-x86_64.tar.gz
cd memvanta-v0.8.3-linux-x86_64
./bin/memvanta_real --help
```

Then run trained-model text generation with a supported Llama-family GGUF model that you are licensed to use:

```bash
./bin/memvanta_real \
  --model /path/to/model.gguf \
  --prompt "Hello from MemVanta" \
  --n 64 \
  --threads 4 \
  --ctx 2048 \
  --temperature 0
```

[Download MemVanta v0.8.3](https://github.com/sauravsingla/MemVanta/releases/tag/v0.8.3)

## Build from source and run a GGUF model

Build the project:

```bash
git clone https://github.com/sauravsingla/MemVanta.git
cd MemVanta
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j
ctest --test-dir build --output-on-failure
```

Then run trained-model text generation with a supported Llama-family GGUF model that you are licensed to use:

```bash
./build/memvanta_real \
  --model /path/to/model.gguf \
  --prompt "Hello from MemVanta" \
  --n 64 \
  --threads 4 \
  --ctx 2048 \
  --temperature 0
```

`memvanta_real` is the trained-model inference CLI. The separate `memvanta run <file>` command exercises mapped streaming/cache behavior and reports memory telemetry; it is not the text-generation command.

[Full getting-started guide](https://sauravsingla.github.io/MemVanta/getting-started/)

## How low-memory inference works

MemVanta's runtime is organized around explicit memory ownership and bounded data movement:

- **mmap-backed GGUF access** avoids requiring an unconditional full-model copy in a separate heap buffer.
- **Bounded tensor slices and caching** keep model access under explicit memory policy.
- **Paged KV cache** manages attention state with defined bounds.
- **Q4/Q8 quantized CPU kernels** provide compact execution paths for supported tensors.
- **Byte-bounded adaptive prefetching** can change look-ahead behavior without silently expanding the memory budget.
- **Runtime CPU dispatch and AVX2/FMA paths** improve hot paths while portability and correctness remain independently tested.

[Low-memory inference guide](https://sauravsingla.github.io/MemVanta/low-memory-llm-inference/) · [Architecture](https://sauravsingla.github.io/MemVanta/architecture/)

## Current model scope

Trained-model execution currently supports GGUF models with:

```text
general.architecture=llama
```

The GGUF parser also validates pinned Qwen2 files, but **Qwen2 inference is not implemented**. Parser/container compatibility should not be interpreted as trained-model execution support.

The project currently has trained-model evidence up to 7B and remains an active research / engineering prototype rather than a drop-in replacement for a mature general-purpose inference runtime.

## Validation and reproducibility

MemVanta's validation stack includes:

- Release and Debug correctness checks
- AddressSanitizer / UndefinedBehaviorSanitizer and ThreadSanitizer lanes
- parser limits and fuzz smoke
- deterministic trained-model checks
- x86 portability and runtime-dispatch validation
- AVX2/FMA paths
- ARM64 cross-build and QEMU validation
- repeated same-machine A/B memory and throughput measurements

Published benchmark methodology requires the identical GGUF artifact for both runtimes, a pinned comparison-runtime revision, matched workload parameters, warm-up plus repeated measured runs, and throughput reporting beside memory results.

Independent results that confirm, narrow, or contradict the current measurements are useful. Reproduction reports should include model hashes, runtime commits, machine metadata, commands, and raw outputs.

[Reproduction guide](https://sauravsingla.github.io/MemVanta/reproduce/) · [Memory benchmarking protocol](docs/MEMORY_BENCHMARKING.md)

## Contributing and external reproductions

Outside systems contributors are welcome. Good first contributions include tooling that improves reproducibility, local build/smoke workflows, platform-validation documentation, and narrowly scoped runtime fixes with deterministic tests. Start with [`CONTRIBUTING.md`](CONTRIBUTING.md) or the open [`good first issue`](https://github.com/sauravsingla/MemVanta/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) tasks.

If you reproduce MemVanta on different CPUs, compilers, RAM limits or model files, please share the full environment and raw outputs even when the result differs from the current measurements. Use [GitHub Discussions](https://github.com/sauravsingla/MemVanta/discussions) for benchmark reproduction, hardware results, model compatibility and systems-design questions.

The first stable release is intentionally evidence-gated rather than date-gated; see [release-readiness issue #56](https://github.com/sauravsingla/MemVanta/issues/56).

## Project links

- [Project website](https://sauravsingla.github.io/MemVanta/)
- [Getting started](https://sauravsingla.github.io/MemVanta/getting-started/)
- [Latest release: v0.8.3](https://github.com/sauravsingla/MemVanta/releases/tag/v0.8.3)
- [PyPI package](https://pypi.org/project/memvanta/)
- [7B benchmark](https://sauravsingla.github.io/MemVanta/benchmark/)
- [Architecture](docs/ARCHITECTURE.md)
- [Results](results/)
- [Citation](CITATION.cff)
- [Zenodo DOI](https://doi.org/10.5281/zenodo.22886357)
- [Contributing](CONTRIBUTING.md)
- [License](LICENSE)
