Plain text out of a local LLM is not what an app wants. An agent wants a JSON object whose fields parse; an IDE plugin wants a tool call it can dispatch. The four local runtimes this cluster already covers - Ollama, llama-server, LM Studio, vLLM - all expose those patterns, but they expose them under different names with different reliability guarantees. This page walks through what each one supports, where grammar-constrained decoding beats JSON mode, and how tool-calling parsers map to specific model families.
JSON mode versus grammar-constrained decoding
Three tiers of reliability sit between “I asked for JSON” and “I got JSON that parses”:
- Plain text + JSON mode. The model is told to emit JSON; nothing prevents it emitting a trailing comma. Ollama’s
formatparameter on/api/chatand/api/generateaccepts either the literal stringjsonor a JSON Schema object (Ollama HTTP API). - Schema-constrained sampling. The sampler constrains the next-token distribution so only tokens that keep the output on a valid schema path are allowed. llama-server calls this
--grammar(inline GBNF string) and--grammar-file(path), with-j, --json-schemaand-jf, --json-schema-fileas the JSON-Schema-front doors (llama-server README). vLLM calls it “guided decoding” ---structured-outputs-config.backendselects amongxgrammar,guidance,outlines, andlm-format-enforcer, withautoas the default (vLLM structured outputs). - Library-level structured generation. A Python library compiles a regex, grammar, or JSON schema to a state machine and patches the inference loop. Outlines and Guidance work this way; LM Studio uses Outlines on the MLX engine path (LM Studio structured output).
The reliability jump from tier 1 to tier 2 is the one that matters in production. A 7B-class model asked for “JSON” will hallucinate keys and drop closing braces. Constrained decoding forces the structure, and the cognitive load moves from the model onto the grammar.
The GBNF grammar files shipped with llama.cpp
The grammars/ directory in llama.cpp ships eight reference GBNF grammars: arithmetic.gbnf, c.gbnf, chess.gbnf, english.gbnf, japanese.gbnf, json.gbnf, json_arr.gbnf, and list.gbnf (llama.cpp grammars). The json.gbnf grammar starts with root ::= object and a value production that covers object, array, string, number, and the literals true/false/null; strings permit the standard \", \\, \b, \f, \n, \r, \t, and \uXXXX escapes (json.gbnf source).
For most users the JSON Schema front door is what gets used. llama-server’s --json-schema flag compiles a JSON Schema to GBNF at request time; the same is true of vLLM’s guided_json (llama.cpp grammars README). The README notes a real catch: complex schemas (recursive refs, anyOf chains) can produce grammars that the engine rejects or compiles inefficiently, so a hand-written GBNF is sometimes faster than the auto-converted one.
What each runtime actually accepts
The four runtimes overlap on /v1/chat/completions (how to call a local LLM API), but the structured-output knobs differ:
| Runtime | JSON mode | Schema-constrained | Grammar file | Tool calling | Tool parser |
|---|---|---|---|---|---|
| Ollama | format: "json" or JSON Schema on /api/chat (Ollama API) | Backend-internal, no flag | No | OpenAI-style tools array on /api/chat | Built-in |
| llama-server | response_format: {"type":"json_object"} on /v1/chat/completions (llama-server README) | response_format: {"type":"json_schema","schema":{...}} plus --grammar/--grammar-file | Yes (GBNF) | --jinja (default on) with OpenAI-style schema; --chat-template-file for non-aligned templates (llama.cpp function-calling) | Generic fallback plus per-model overrides |
| LM Studio | response_format JSON mode on /v1/chat/completions (LM Studio structured output) | response_format: {"type":"json_schema","json_schema":{"strict":true,"schema":{...}}} | No (engine-internal) | OpenAI-style tools (Tool Use docs page) | Engine-internal |
| vLLM | guided_choice / guided_regex / guided_grammar / StructuredOutputsParams(choice=...) (vLLM structured outputs) | --structured-outputs-config.backend among xgrammar (default), guidance, outlines, lm-format-enforcer | Yes (EBNF via guided_grammar) | --enable-auto-tool-choice plus --tool-call-parser <name> (vLLM tool calling) | hermes, llama3_json, llama4_pythonic, mistral, granite, granite4, granite-20b-fc, internlm, jamba, xlam, deepseek_v3, deepseek_v31, openai, kimi_k2, hunyuan_a13b, cohere_command3, cohere_command4, longcat, glm45, glm47, functiongemma, qwen3_xml, mimo, step3p5, olmo3, gigachat3, apertus, pythonic, hf |
XGrammar, the default vLLM backend since December 2024, advertises “100% structural correctness of the output” for general context-free grammars including JSON, regex, and custom CFGs (XGrammar). The other three backends - guidance, outlines, lm-format-enforcer - differ on regex dialect: xgrammar, guidance, and outlines use Rust-style regex, lm-format-enforcer uses Python’s re (vLLM structured outputs).
Tool calling, parser by parser
Tool calling is the harder of the two. Models trained on OpenAI-style tool schemas emit something in a specific dialect (Hermes/Qwen use <tool_call>...</tool_call> XML; Llama 3 uses {"name":...,"parameters":...} JSON; Llama 4 uses Pythonic [func1(arg=val)] syntax), and the parser has to convert that dialect back into a structured tool_calls array the client code can dispatch.
On llama-server the catch is template fidelity. The docs name Qwen2.5-7B-Instruct, Mistral-Nemo, and Llama-3.3-70B-Instruct as working out of the box under --jinja; DeepSeek R1 distills, Functionary v3.2, Hermes 2/3 Pro Llama-3-8B, Firefunction v2, and c4ai Command R7B each need a --chat-template-file override because their upstream templates are either buggy or non-standard (llama.cpp function-calling). When no template matches, the README’s fallback is --chat-template chatml, which works on many models but consumes more tokens.
The llama.cpp docs also name a quantisation-specific catch: “Extreme KV quantizations (e.g. -ctk q4_0)… can substantially degrade the model’s tool calling performance” (llama.cpp function-calling). The structured output’s grammar always wins - the model cannot emit anything the grammar forbids - but choosing which token is the right one is still the model’s problem, and KV-cache quantisation distorts the logits it ranks with.
vLLM exposes the parser list explicitly because each parser is a separate Python class. vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --reasoning-parser deepseek_r1 and vllm serve meta-llama/Llama-3.1-8B-Instruct --tool-call-parser llama3_json --enable-auto-tool-choice are the two patterns the docs use (vLLM tool calling). Adding a new model family is a matter of writing a new ToolParser and registering it via --tool-parser-plugin.
What to use, for what
For a single-agent app pulling structured data into a script, Ollama’s format parameter is the path of least resistance on /api/chat, and it accepts a JSON Schema directly. For production systems where parsing failures are unacceptable, llama-server’s --grammar flag or vLLM’s guided_json removes the failure mode entirely at the cost of a request-time grammar compile. For a desktop install with no command line, LM Studio wires grammar sampling into the GGUF engine and Outlines into the MLX engine without the user touching a flag.
For tool calling specifically, the practical rule is: pick the model first, then pick the parser. The [best local coding models by VRAM](/articles/best-local-coding-models-by-vram/) page covers the model half; the parser half lives in vLLM’s --tool-call-parser list, llama-server’s --chat-template-file overrides, and Ollama’s built-in parser. The same model can produce structurally valid JSON via grammar and yet still call the wrong tool - reliability of the call is a function of the model’s training data, the parser’s tolerance, and (on llama.cpp) the KV-cache quantisation, in that order.
Bottom line
If you only need JSON and don’t care about the schema, every runtime does it. If you need enforced schema - an enum, nested objects, required fields - use llama-server’s --grammar/--json-schema, vLLM’s guided_json, or LM Studio’s json_schema mode, in that order of documentation depth. If you need tool calls, name the model, name the parser, and test it against a known-good prompt; the parser list on vLLM is the most explicit, llama-server’s --chat-template-file overrides are the most surgical, and Ollama’s built-in parser is the path of least resistance for models that the Ollama team has shipped a chat template for. The serving side - exposing the endpoint to a household, putting it behind auth, monitoring it - is the serve-a-local-model-to-your-household page’s job, not this one.