Local LLM Prompt Caching: Reuse Explained (October 2026)
How llama.cpp, vLLM and LM Studio decide whether to reuse your prompt's KV cache, what each reports about tokens saved, and where the reuse stops.
Tag
#vllm
Why Your Local LLM Gives a Different Answer Each Time
Temperature, top-k, top-p and seed decide how random your local model is. What they do in Ollama, llama.cpp and vLLM, and how to get repeatable answers.
Local LLM Privacy: What Each Runtime Actually Sends Home
Side-by-side look at what Ollama, LM Studio, llama.cpp, vLLM, AnythingLLM, and Hugging Face TGI collect by default, and how to switch each one off.
Speculative Decoding: Drafts, EAGLE-3, n-gram (October 2026)
Speculative decoding strategies for local LLMs: how draft models, EAGLE-3, DFlash, and n-gram matching speed up llama.cpp and vLLM, and when each one helps.
Structured Output and Tool Use with Local LLMs (October 2026)
JSON mode, GBNF grammar-constrained decoding, and OpenAI-style tool calling across Ollama, llama-server, LM Studio, and vLLM.
Best Local Models by Context Window (October 2026)
Local LLMs ranked by maximum context length: 32K, 128K, 256K, and 1M+ tiers, with the model that fits each tier and the VRAM each one needs.
What Thinking Mode Actually Costs You (September 2026)
Reasoning tokens, latency, and KV cache cost of thinking mode in local LLMs. How to toggle it per runner and when it is the wrong choice.
Local LLM Backends Compared: Ollama, LM Studio, llama.cpp, vLLM
How Ollama, LM Studio, llama.cpp, and vLLM differ on model format, OpenAI-compatible API, hardware support, and which to pick for a home server.
Local AI When One GPU Isn't Enough (September 2026)
Multi-GPU options for self-hosted AI when one card runs out of room: how Ollama, llama.cpp and vLLM split models, and what the interconnects actually cost.
Serving a Local Model to Your Household Without Exposing It
None of the mainstream inference servers ask for a password, and vLLM binds every interface by default. The safe pattern for household serving.
Best Local LLM Runners (September 2026)
Ollama, LM Studio, Jan, llama.cpp and vLLM compared: licences, engines, OpenAI-compatible APIs, multi-user serving, and which to install first.
GGUF vs AWQ vs GPTQ vs MLX: Which Quantization to Use
Quantization formats compared with real file sizes and bits per weight. Why Q4 does not halve a model, and why low quants generate faster.
Kimi K3 Weights Are Public: 2.8T Params, 64-GPU Reality
Moonshot released the full Kimi K3 weights on July 27, 2026: 2.8T params, 1M context, MXFP4. Read the license before you plan a deployment.
Open-Source AI Wins: NVIDIA Goes All-In on Open Models, Lightricks Ships 4K Video, and Local Inference Matures
GTC 2026's biggest announcements were open-source. Nemotron 3 Super runs locally on RTX PCs, LTX 2.3 generates 4K video with audio, and vLLM hits production grade.
Open-Source AI Wins: OLMo Hybrid Rewrites Efficiency, Karpathy Ships Research Agents, and Local Tools Level Up
This week's open-source highlights: AI2's hybrid architecture proves transformers need help, autoresearch automates ML experiments overnight, and local inference gets serious upgrades.
Critical vLLM Flaw Lets Attackers Take Over AI Servers
A CVSS 9.8 flaw in vLLM allows unauthenticated remote code execution through malicious video URLs. Patch now if you run multimodal models.