What Thinking Mode Actually Costs You (September 2026)
Reasoning tokens, latency, and KV cache cost of thinking mode in local LLMs. How to toggle it per runner and when it is the wrong choice.
Tag
#llama-cpp
Local LLM Backends Compared: Ollama, LM Studio, llama.cpp, vLLM
How Ollama, LM Studio, llama.cpp, and vLLM differ on model format, OpenAI-compatible API, hardware support, and which to pick for a home server.
Serve a Local Model to Your Team (September 2026)
Auth, reverse proxy, HTTPS, concurrency and audit: how a small team shares one local model without exposing it to the public internet.
Local AI When One GPU Isn't Enough (September 2026)
Multi-GPU options for self-hosted AI when one card runs out of room: how Ollama, llama.cpp and vLLM split models, and what the interconnects actually cost.
Run an LLM Locally on a Raspberry Pi (September 2026)
Raspberry Pi 4 and Pi 5 can run small open-weight models on CPU alone. What fits on 2 GB, 4 GB, 8 GB, or 16 GB of RAM, and what speed is realistic.
Best Used GPUs for Local AI in 2026 (September 2026)
The RTX 30 and 40 series cards that still clear current Ollama and llama.cpp floors, what VRAM tier each opens, and which variants are worth the used premium.
Serve a Local Model to Your Household (August 2026)
Practical patterns for letting two to six people in one home share one local model on one machine, with the right chat UI, bind address, and overlay network.
What Quantization Costs You in Quality (August 2026)
How much quality a smaller quant actually loses. Real MMLU and KL Divergence numbers across Q2_K through Q8_0, plus how to measure on your own workload.
Local AI on AMD and Intel GPUs (August 2026)
What works on AMD Radeon and Intel Arc hardware today, where the official ROCm and SYCL paths stop, and how Vulkan fills the gap.
Running Local LLMs on AMD and Intel GPUs in 2026
AMD's ROCm support splits by card and by OS, Intel archived both of its own LLM libraries this year, and llama.cpp is the path still standing.
What Quantization Actually Costs You in Quality
8-bit measures close to lossless. 4-bit ranges from a 0.6% gain to a 59% drop on one benchmark, depending on the model. The figures, fully attributed.
Local AI Without a GPU: What Actually Runs (August 2026)
Which open models run on CPU and RAM alone, how to size them, why mixture-of-experts helps, and what a machine with no discrete GPU cannot do.
Best Local LLM Runners (August 2026)
Ollama, LM Studio, Jan, llama.cpp and vLLM compared: licences, engines, OpenAI-compatible APIs, multi-user serving, and which to install first.
Local AI by VRAM: Which Models Fit Your GPU (September 2026)
Find your VRAM tier, then the current open-weight pick for chat, coding, vision, speech, translation or agents. Eighteen guides, one index.
Run an LLM Locally on Android and iPhone (September 2026)
Apple caps its on-device model at 4096 tokens per session, and Google's own tables put Gemma 4 E2B at 25.0 decode tokens/sec on an iPhone 17 Pro CPU.
Run LLMs Locally on a Mac: What Actually Fits (September 2026)
Apple Silicon has no discrete VRAM, so tier guides mislead Mac owners. The real ceilings are bandwidth and the GPU-usable slice of unified memory.
GGUF vs AWQ vs GPTQ vs MLX: Which Quantization to Use
Quantization formats compared with real file sizes and bits per weight. Why Q4 does not halve a model, and why low quants generate faster.
MiniMax-M3 Lands in llama.cpp: Sparse Attention and Vision
Two back-to-back merges add MiniMax Sparse Attention and a Qwen2.5-VL style vision tower to llama.cpp, but every existing MiniMax-M3 GGUF must be regenerated.
VKUE: Can a 34.7B Reasoner Run on a Laptop Without a GPU?
VIDRAFT_LAB posts Ourbox-35B-JGOS to Hugging Face: 20 tok/s on an 8GB laptop GPU, ~17 tok/s on a CPU-only server, 86.4% on GPQA Diamond.
llama.cpp Joins Hugging Face: What It Means for Local AI's Future
Georgi Gerganov's team is now at Hugging Face, unifying the model hub with the inference engine that powers Ollama, LM Studio, and the entire local AI ecosystem.
Open-Source AI Wins: OLMo Hybrid Rewrites Efficiency, Karpathy Ships Research Agents, and Local Tools Level Up
This week's open-source highlights: AI2's hybrid architecture proves transformers need help, autoresearch automates ML experiments overnight, and local inference gets serious upgrades.
The Week Local AI Grew Up: Ollama 0.17 and llama.cpp's New Home
Ollama delivers 40% faster inference while llama.cpp finds a permanent home at Hugging Face. Two developments that secure the future of running AI on your own hardware.
Open-Source AI Wins: ggml Joins Hugging Face, GLM-5 Goes MIT, NanoClaw Hits 14K Stars
This week's biggest open-source AI developments: llama.cpp finds a permanent home, China releases a 744B parameter model under MIT license, and a secure WhatsApp AI assistant goes viral
llama.cpp Joins Hugging Face, Stays Open and Independent
llama.cpp's creators join Hugging Face to keep the project open and sustainable while local AI competes with cloud inference.