A reasoning model answers “What is 9.9 minus 9.11?” with 8,000 tokens of internal deliberation and then writes “0.79”. That trace is the cost of thinking mode, and it is real on three axes at once: tokens, latency, and KV cache. This page is the version you can actually price against your hardware. For model selection, the reasoning models hub ranks what is out there; for the KV-cache math, the context-window hub covers it in detail.
What thinking mode is, mechanically
A thinking model emits a block of “reasoning tokens” before the final answer, then closes the block and emits the answer. The block is delimited by tags inside the model’s own output. Three tag conventions are in live use:
| Family | Tag | Default on | Toggle |
|---|---|---|---|
| Qwen3 (including the 235B-A22B line), DeepSeek-R1, Phi-4-mini-reasoning | <think>...</think> | enable_thinking=True by default; soft switch /think and /no_think per prompt | API flag, soft tag, or parser |
| EXAONE-Deep | <thought>...</thought> | Reasoning on by default | ”Ensure the model starts with <thought>\n” per the EXAONE-Deep card |
| gpt-oss (OpenAI) | No literal tags; reasoning streams as a structured field | On by default | System prompt “Reasoning: low/medium/high” |
Three consequences fall out. First, the trace is part of the model’s output, not a side channel: it goes through the same sampler and the same KV cache as the final answer. Second, the tag convention is family-specific; a runner that only knows <think> will treat EXAONE-Deep’s <thought> as plain text and double-count the trace against the answer. Third, some models are deliberately shipped without thinking at all: Qwen3-235B-A22B-Instruct-2507 “supports only non-thinking mode and does not generate <think></think> blocks”.
What it costs in tokens
The published generation budgets are not hints, they are the floor.
| Model | Default max_new_tokens | Notes |
|---|---|---|
| Qwen3-8B | 32,768 (recommend 38,912 for hard problems) | “Reserving 32,768 tokens for outputs and 8,192 tokens for typical prompts” |
| DeepSeek-R1 | 32,768 (eval); 128K context | ”Maximum generation length is set to 32,768 tokens for evaluations” |
| EXAONE-Deep-32B | 32,768 | ”Reasoning steps … usually have lots of tokens” |
| Phi-4-mini-reasoning | 32,768 | Same default at benchmark time |
| Magistral-Small-2506 | 40,960 | ”Performance might degrade past 40k” |
Three rules follow. First, a query that would take 80 output tokens in non-thinking mode can easily spend 2,000 to 10,000 in thinking mode before answering. Second, a long chain that overruns max_new_tokens is cut mid-thought and the answer is whatever the runner managed to emit; there is no automatic “skip the trace and just answer” fallback. Third, the trace is held in the KV cache for the rest of the turn, so a multi-turn conversation that starts thinking will keep paying the cache cost of the original trace until the conversation is trimmed.
What it costs in latency and KV cache
A 32B reasoning model on a 24 GB GPU at 30 tokens per second means a 5,000-token trace takes about 170 seconds before the answer starts. On a 13B at the same speed, the same trace is about 100 seconds. That is not interactive. For chat-style use it is unusable.
The KV-cache cost is the one people miss. The reasoning trace and the final answer live in the same cache. Two consequences:
- Context window burns fast. A 32K output trace on a 40,960-token trained context (the Qwen3-8B default) leaves less than 9,000 tokens for the prompt. On the 262K-context Qwen3.5-9B the budget looks generous, but the VRAM arithmetic is the real limit; reasoning traces that long will OOM before they finish.
- Quantization interacts oddly. KV-cache quantization in llama.cpp (
--cache-type-k q8_0,--cache-type-v q4_0) reduces the trace’s VRAM cost but does not reduce its token cost. Capping the trace is the only real lever.
llama.cpp server is the runner with a real cap. The --reasoning-budget flag takes a number of tokens, with -1 meaning “no budget” (the default, which lets the model think until max_tokens runs out), and 0 meaning “force the model to answer immediately”. --reasoning-effort accepts default / minimal / low / medium / high / xhigh / max. Ollama and vLLM do not expose a per-token budget field; you cap by setting the model’s max_tokens and accept that a long trace will overrun it.
How to toggle thinking per runner
The flag name and the supported families change every few months. What follows is what the runner docs say today.
Ollama. The think field on /api/chat and /api/generate accepts a boolean for most models (true/false) and one of low, medium, high, max for models that expose levels; GPT-OSS specifically requires one of low, medium, or high and ignores booleans. CLI flags are --think and --think=false; --hidethinking suppresses the trace display but still runs it. Interactive sessions use /set think and /set nothink.
llama.cpp server. --reasoning-format (auto default, none, deepseek, deepseek-legacy) decides where thoughts land in the JSON response. --reasoning (on/off/auto) is the master switch. --reasoning-effort takes the seven levels above. --reasoning-budget sets a token cap. The thinking content lands in message.reasoning_content and the final answer in message.content.
vLLM. --reasoning-parser picks the family; the documented parsers include deepseek_r1, qwen3, gemma4, granite, glm45, ernie45, hunyuan_a13b, minimax_m2_append_think, and several others. Reasoning defaults vary per model: Qwen3 is on by default, IBM Granite 3.2 and DeepSeek-V3.1 are off and require thinking=True. The output field is message.reasoning (renamed from reasoning_content); the docs flag reasoning_content as deprecated and ask users to migrate client code.
Hugging Face transformers. Set enable_thinking=False in tokenizer.apply_chat_template(...) to suppress thinking; this is the hard switch that overrides any /think or /no_think tag in the user prompt. With it True (the default) the soft tags work per turn.
A practical hazard: the toggle is not symmetric across runners. --reasoning-format deepseek-legacy keeps <think> tags inside message.content while also populating reasoning_content. A UI that strips <think> blocks but reads from content will accidentally display the trace. A UI that reads from reasoning_content but the runner is configured none will display the trace as the final reply and lose the answer.
When thinking helps and when it does not
The published comparisons are stark on the upside. The Qwen3-235B-A22B-Instruct-2507 card benchmarks the updated non-thinking variant against the prior non-thinking Qwen3-235B-A22B: GPQA Diamond moves 62.9 to 77.5, AIME 2025 moves 24.7 to 70.3, HMMT 2025 moves 10.0 to 55.4, ARC-AGI moves 4.3 to 41.8. The non-thinking variant of that family already beats the prior non-thinking one; the same family with thinking on goes substantially higher on the math/code/planning axes the benchmarks measure.
Three rules for when not to use it:
- Simple lookups and short chat. “Rewrite this email politely” does not need 5,000 tokens of internal monologue. A non-reasoning model is faster, cheaper, and indistinguishable in output quality.
- Latency-bound interfaces. Autocomplete, voice agents, IDE inline suggestions. A 30-second pause is not a feature.
- KV-cache-starved runs. A 32K trace on a 24 GB card plus 8 GB of weights plus a large prompt is the configuration that OOMs. Either shorten the trace with
--reasoning-budget, drop the trace to KV-q4 (--cache-type-k q8_0 --cache-type-v q4_0), or pick a smaller reasoning model from the VRAM hub.
The middle path is what most local setups actually want: cap the trace at a few thousand tokens and let the model think briefly. --reasoning-budget 2048 on llama.cpp server is a reasonable starting point; tune from there.
The bottom line
Thinking mode is a chain-of-thought block emitted into the model’s own output. It costs tokens (often 1,000 to 10,000 per query), latency (seconds to minutes at consumer speeds), and KV cache (the trace and the answer share it). It pays off on math, multi-step planning, and hard reasoning, and it is the wrong choice for short chat, latency-bound interfaces, and KV-cache-tight setups. Toggle per runner: Ollama’s think field, llama.cpp’s --reasoning-budget and --reasoning-effort, vLLM’s --reasoning-parser plus chat-template kwargs. The tag convention is family-specific, so do not assume <think> is universal; EXAONE-Deep uses <thought>, gpt-oss has no literal tags at all.