Multi-model llama.cpp router
Serve many GGUF models from one endpoint. The router loads and switches models on demand and queues incoming requests during a swap instead of failing them.
NVIDIA Llama Manager · CUDA
Run open-source models on your own silicon.
Llama Manager turns an NVIDIA GPU box into a self-contained LLM server — the same multi-model llama.cpp router, built against CUDA — shipped as an Ubuntu 24.04 appliance ISO and a signed APT repository.
GPG-signed releasesNVIDIA builds in progressFully offlineUbuntu 24.04 Desktop
- Target hardware
- NVIDIA RTX / CUDA GPUs
- Compute stack
- CUDA
- Memory
- GPU VRAM + system-RAM offload
- Base OS
- Ubuntu 24.04 Desktop
What's inside
One appliance, everything local.
The same Llama Manager appliance, built for NVIDIA GPUs and the CUDA stack.
CUDA-tuned engines
Beyond stock llama.cpp, dedicated inference engines are built against CUDA and selected automatically per model — the same router architecture, tuned for NVIDIA GPUs.
Appliance & kiosk
Boots straight into a full-screen local dashboard. Manage models, presets, and routing from the browser, with a local System Login escape hatch for shell access.
Fully offline
Everything runs on the box. No cloud dependency and no telemetry required — models and inference stay entirely local to your hardware.
Signed APT + reproducible ISO
Ships with a signed APT repository preconfigured, so updates install in place with standard apt tooling and GPG verification against the archive key.
VRAM + smart offload
Models are split between GPU VRAM and system RAM automatically, so a single box can serve models larger than its VRAM budget alone.
Experimental
GPG signedLlama Manager 24.04.4
Preview appliance image for the NVIDIA DGX Spark (GB10).
Experimental build for DGX Spark — not production-supported; flash and test at your own risk.
SHA-256
bdbf554966bdee78963efdbe5c7847978132d12f078dc4bb7609373933d5e5dbDownload imagearm64 · experimental preview
Llama Manager Flasher
Flash it in one step.
Download the Flasher, plug in a USB drive or microSD, and it fetches the latest NVIDIA DGX Spark (experimental) image — checksum-verified — and writes it for you. No installation: one portable file.
These builds are not yet code-signed. On macOS, right-click the app and choose Open on first launch (or run xattr -cr on the downloaded .dmg). On Windows, choose More info → Run anyway at the SmartScreen prompt.
Coming to NVIDIA hardware
The NVIDIA appliance image is in the works. Watch the GitHub project for build progress, or explore the AMD Ryzen release available today.