Local AI

How to run an LLM locally on Linux (2026)

Run a private LLM on Linux in 2026 - the easy path with Ollama, the flexible path with llama.cpp, and the fast path with vLLM, plus GPU requirements.

Updated July 13, 2026

Linux is the natural home for running LLMs, with three tools covering three needs: Ollama for the easiest start, llama.cpp when you want to build and tune it yourself, and vLLM when you need to serve a model fast to many users. This guide covers all three and their GPU requirements. For picking the model, see the best local models by VRAM tier; for the wider case, our run AI locally privacy guide.

The easy path: Ollama

Ollama uses the same one-line install as elsewhere:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma4:12b

It detects NVIDIA CUDA automatically; AMD GPUs need a current ROCm v7 driver on Linux, per Ollama’s GPU docs. For most people, Ollama is all they need.

The flexible path: llama.cpp

llama.cpp is the C/C++ inference engine that most of the ecosystem is built on. Building it yourself lets you target your exact hardware. It is CMake-based:

# CPU only
cmake -B build && cmake --build build --config Release

# NVIDIA (CUDA toolkit must be installed)
cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release

For AMD, llama.cpp’s build docs cover a HIP/ROCm build (-DGGML_HIP=ON with your GPU target, for example gfx1030). Its native format is GGUF, with quantization from 1.5-bit up to 8-bit, so you can trade quality for memory precisely.

The high-throughput path: vLLM

vLLM is for serving - fast inference to multiple concurrent users, with the full model context. Install it with uv:

uv pip install vllm --torch-backend=auto

Requirements: Python 3.10 to 3.13, and an NVIDIA GPU with compute capability 7.5 or higher (T4, RTX 20-series and newer, A100, L4, H100, and up). The default binaries are built against CUDA 12.9, with Blackwell cards (B200/GB200) needing at least CUDA 12.8. vLLM also has AMD ROCm, Intel XPU, and CPU-only builds. Reach for vLLM only when you are building a service; for a personal setup, Ollama or llama.cpp is simpler.

Models for single-GPU Linux serving

  • Qwen3.6-35B-A3B - a mixture-of-experts model (35B total, 3B active) with a very long context window; the 4-bit GGUF is around 22GB for llama.cpp, or serve the full weights via vLLM.
  • Gemma 4 31B (~20GB) - a solid dense single-GPU target with a 256K context.
  • gpt-oss:120b - OpenAI’s larger open-weight model (117B total, ~5B active) that fits a single 80GB data-centre GPU, if you have one.

Quick start

# Easiest: Ollama
ollama pull qwen3.6:27b
ollama run qwen3.6:27b

# Or build llama.cpp with CUDA and run a GGUF model you downloaded

For a web interface on top of Ollama, pair it with Open WebUI. Everything runs on your own hardware - private, free per query, and offline once downloaded.