MemVanta — Low-Memory CPU LLM Inference for GGUF

OpenLLaMA 7B memory benchmark vs llama.cpp

Results are scoped to the tested model, settings, host, and pinned comparison runtime. Read the benchmark interpretation · Raw evidence

Run your first GGUF model

Build the project, then use memvanta_real for trained-model text generation with a supported Llama-family GGUF file.

git clone https://github.com/sauravsingla/MemVanta.git
cd MemVanta
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

./build/memvanta_real \
  --model /path/to/model.gguf \
  --prompt "Hello from MemVanta" \
  --n 64 --threads 4 --ctx 2048 --temperature 0

memvanta_real is the text-generation CLI. memvanta run <file> is a separate mapped streaming/cache telemetry path.

Memory-efficient GGUF inference on CPU

MemVanta is built for local AI, edge systems, and constrained machines where resident memory is a primary bottleneck. The project focuses on a transparent memory/performance trade-off rather than claiming throughput leadership.

• mmap-backed GGUF model access
• Paged KV cache with explicit bounds
• Q4/Q8 quantized CPU kernels
• Adaptive, byte-bounded prefetching
• AVX2/FMA optimized paths
• ARM64 cross-build and QEMU validation
• Repeated performance and memory A/B gates
• Reproducible llama.cpp comparisons

Learn how the low-memory design works or explore the runtime architecture.

Current model support

Trained-model execution currently supports GGUF models with general.architecture=llama. The GGUF parser also validates pinned Qwen2 files, but Qwen2 inference is not implemented.

MemVanta is an active research prototype with trained-model evidence up to 7B. Independent reproduction is welcome.

Evidence and reproducibility

Published benchmark numbers are generated from machine-readable repository evidence. The methodology requires matched model artifacts and workloads, a pinned comparison runtime, repeated runs, and throughput reporting alongside memory.