Before benchmarking: verify real inference
If you have not used MemVanta before, first build the project and run memvanta_real against a supported Llama-family GGUF model. This separates basic model-compatibility problems from benchmark-methodology problems.
What to measure
- Peak resident set size across repeated measured runs.
- Prompt-processing throughput.
- Token-generation throughput.
- Where practical on Linux, success or failure under enforced cgroup-v2 memory ceilings with swap disabled.
The goal is not to find only the most favorable run. Keep warm-up runs separate, retain every measured run, and preserve failed or OOM-killed attempts in constrained-memory sweeps.
Keep the comparison controlled
Use the identical GGUF file for MemVanta and the comparison runtime. Record the model filename, byte size, and SHA-256 digest. Pin the exact comparison-runtime commit and keep CPU-only execution, thread count, prompt length, generated-token count, context size, batch size, and KV-cache precision aligned whenever both runtimes support the same setting.
For peak-RSS comparisons, the project protocol calls for at least one warm-up and five measured runs.
Evidence to publish
- MemVanta commit SHA
- Comparison-runtime commit SHA
- Model filename and SHA-256
- CPU model and core counts
- Operating system and kernel
- Total RAM and swap state
- Compiler and build type
- Complete command lines
- Context, batch, and token counts
- Raw outputs for every run
- Machine-readable CSV or JSON summary
- cgroup configuration when used
How to report disagreement
A reproduction is valuable whether it confirms, narrows, or contradicts the current result. If you observe a materially different memory or throughput profile, publish it with the same evidence rather than filtering it out. Methodology corrections are welcome.
For constrained-memory testing, report the actual tested ceilings and outcomes. Do not convert the lowest successful tested ceiling into an exact minimum physical-RAM requirement.
Reference result
The current canonical OpenLLaMA 7B v2 Q4_0 comparison reports 3.80 GiB peak RSS for MemVanta and 7.24 GiB for pinned llama.cpp, with llama.cpp substantially faster in throughput. Use the raw repository evidence as the reference rather than copying numbers from screenshots or third-party summaries.