An open-weights model is a file on disk. The runner is the separate piece of software that loads that file, decides how much of it goes on the GPU, applies the right chat template, and exposes it over HTTP so other programs can use it. Choosing badly here is how people end up concluding that local AI is slow, when the model was fine and the runner was spilling half of it into system RAM.
This is a survey of the runners that are actually maintained, what engine each one uses underneath, and what each one cannot do.
How the facts below were checked. Every licence is read from the GitHub API
license.spdx_id field or from the project’s own LICENSE file, not from memory. Archive
status and the last push to the default branch were checked for every repository. Feature
claims come from each project’s own documentation. All of it was pulled on 2026-08-07.
The comparison
| Runner | Licence | Engine underneath | Platforms | Who it is for | The honest catch |
|---|---|---|---|---|---|
| Ollama | MIT | Vendored llama.cpp, plus its own MLX runner on Apple Silicon | macOS 14+, Linux, Windows, Docker | Anyone who wants other software to talk to a local model | Parallel requests are off by default; the library has no reranker |
| LM Studio | Proprietary (Element Labs), free at home and at work | llama.cpp for GGUF, Apple MLX on Apple Silicon | Apple Silicon macOS, x64/ARM64 Windows, x64 Linux | First-timers who want a GUI and a model browser | Closed source; the docs list Linux support as x64 only |
| Jan | Apache 2.0, with a request for attribution | llama.cpp for GGUF, MLX on Apple Silicon (experimental) | macOS, Windows, Linux | People who want an open-source desktop app | The MLX path is documented as experimental and cannot do embeddings |
| llama.cpp | MIT | It is the engine | Metal, CUDA, HIP, MUSA, Vulkan, SYCL, CPU | Anyone who wants every flag exposed | You assemble the setup yourself; there is no hand-holding |
| vLLM | Apache 2.0 | Its own CUDA/HIP kernel stack with PagedAttention | NVIDIA, AMD and Intel GPUs, x86/ARM/PowerPC CPUs | Serving many concurrent users | Wrong tool for single-user chat; real setup cost |
| LocalAI | MIT | A small core that pulls llama.cpp, vLLM, MLX and others as backends | Linux, macOS, Windows, containers | Self-hosters who want one API over many model types | Backend sprawl; more moving parts than a desktop app |
The engine question, which most coverage gets wrong
The common shorthand is that Ollama and LM Studio are “wrappers around llama.cpp”. That was true and is now only half true.
llama.cpp is still the GGUF engine inside both. Ollama’s repository root carries a
LLAMA_CPP_VERSION pin and a llama/ tree that fetches, patches and builds that exact
revision. LM Studio’s documentation states plainly that it “supports running LLMs on Mac,
Windows, and Linux using llama.cpp”.
Both now ship a second engine as well. Ollama’s repository root also pins MLX_VERSION and
MLX_C_VERSION, and contains an x/mlxrunner/ tree built against Apple’s MLX framework.
Its release notes describe the two side by side: v0.32.6, published 2026-08-04, lists
“Updated the MLX and llama.cpp engines”, and v0.32.4 added a model “on Apple GPUs via the
MLX engine”. LM Studio’s docs say that “on Apple Silicon Macs, LM Studio also supports
running LLMs using Apple’s MLX”, and its MLX engine is published separately under the MIT
licence. Jan has an MLX provider too, though its own documentation labels it experimental.
If that split matters to you, the trade-offs between formats are covered in
which quantization to use, and the
Apple-specific setup in
running LLMs on Apple Silicon.
vLLM is not in this category at all. It is a serving engine with its own kernel stack, built around PagedAttention and continuous batching, with tensor, pipeline, data, expert and context parallelism. Those features exist to keep a GPU saturated across many simultaneous requests. For one person chatting in one window, almost none of that pays off.
What each one can and cannot do
OpenAI-compatible APIs. All five expose one. Ollama’s compatibility page documents
/v1/chat/completions, /v1/completions, /v1/models, /v1/models/{model},
/v1/embeddings and /v1/responses. LM Studio documents /v1/models, /v1/responses,
/v1/chat/completions, /v1/embeddings and /v1/completions, plus an Anthropic-compatible
Messages endpoint. Jan’s server defaults to loopback on port 1337, with a configurable path
prefix and an optional API key. llama.cpp’s own server covers the OpenAI routes plus an
Anthropic-compatible /v1/messages. vLLM does the same and adds gRPC.
Model library versus bring-your-own file. Ollama and LM Studio both curate. Ollama
pulls by name from its own library, and imports a local file through a Modelfile with
FROM /path/to/file.gguf. LM Studio and Jan both search Hugging Face directly; Jan can
link an existing GGUF in place without copying it. llama.cpp and vLLM assume you already
know which weights you want.
GPU and CPU split. This is llama.cpp’s quiet advantage and it propagates to everything
built on it. Its README describes “CPU+GPU hybrid inference to partially accelerate models
larger than the total VRAM capacity”, controlled by -ngl for the number of layers held in
VRAM and -sm for how a model splits across several GPUs. If you are still deciding what
fits, start from local AI by VRAM.
Serving more than one person. Here the gap is wide. Ollama’s FAQ documents
OLLAMA_NUM_PARALLEL, and gives its default as 1, so a fresh install handles one request at
a time per model until you change that. llama.cpp exposes server slots through
-np, --parallel and a /slots monitoring endpoint. vLLM is built for this case and
nothing else. LM Studio and Jan are desktop applications first; both can bind to 0.0.0.0,
which is a network convenience rather than a multi-tenant design.
Reranking. A reranker is the second-stage model that reorders retrieved documents, and
it is what usually separates a working local RAG setup from a disappointing one. llama.cpp
has a /reranking endpoint, enabled with --rerank and a reranker model. vLLM implements
the Cohere Rerank API at /rerank, /v1/rerank and /v2/rerank, plus a Score API, and
lists supported cross-encoder architectures. LocalAI documents a reranker API.
Ollama is the outlier. Its model search page offers capability filters for cloud, embedding, vision, tools and thinking, with no reranking filter, and the string “rerank” does not appear anywhere in that page’s HTML. Its OpenAI-compatibility page lists the six endpoints above and no rerank route. So the library ships embedding models but nothing to rerank with, which is a real constraint if retrieval is your reason for running locally.
Also checked
LocalAI (MIT) describes itself as “a small core, not a bundle”: backends wrapping llama.cpp, vLLM, whisper.cpp, MLX and others are pulled on demand as separate images. It released v4.8.2 on 2026-08-07 and offers OpenAI, Anthropic and ElevenLabs-compatible APIs over the lot. Good fit for a home server, heavier than a desktop app.
Msty is a closed-source desktop workspace that drives Ollama, llama.cpp and MLX rather than replacing them. Msty Studio 2.9.6 is the newest changelog entry, dated 2026-07-24. No source repository is linked from the product or pricing pages checked here. There is a free tier, with paid tiers above it.
text-generation-webui now lives at oobabooga/textgen and is licensed AGPL-3.0. Its
latest release is v4.9 from 2026-05-20, with the last commit to the default branch on
2026-06-01. Maintained, but on a slower cadence than the rest of this list.
GPT4All (MIT) is the one to skip for new work. The repository is not archived, but its last release is v3.10.0 from 2025-02-25 and the last commit to its default branch is dated 2025-05-27. Everything else here shipped within the last ten weeks.
What to install first
Install LM Studio. One installer, a built-in model browser, both engines handled for you on Apple Silicon, and an OpenAI-compatible server behind a toggle when you want it. The licence is proprietary, but the terms permit use at home and at work at no cost, and for a first local model that trade is worth making over spending an evening on build flags.
Switch to Ollama the moment something else needs to reach the model, because it is what editors, agents and scripts assume by default. Reach for llama.cpp directly when a default is costing you performance and you want the flag that fixes it. Reach for vLLM only when more than one person is waiting on the same GPU.