GreenBoost Extends NVIDIA VRAM with System RAM and NVMe for Larger LLMs
An open-source Linux kernel module named GreenBoost adds system RAM and NVMe storage as a CUDA-accessible extension to NVIDIA GPU memory, enabling larger AI models by caching data outside of VRAM. It uses a kernel component to pin memory and export it as DMA-BUF, and a CUDA shim (LD_PRELOAD) to intercept and redirect large allocations to the extended memory pool, while small allocations pass through. The setup is complementary to NVIDIA’s drivers and includes a watchdog to monitor RAM/NVMe pressure; it could allow running models bigger than VRAM (e.g., a 31.8GB model on a 12GB RTX) but requires careful handling of symbol resolution for certain frameworks like Ollama.