Structured Output and Tool Use with Local LLMs (October 2026)

JSON mode, GBNF grammar-constrained decoding, and OpenAI-style tool calling across Ollama, llama-server, LM Studio, and vLLM.

Updated October 13, 2026

Plain text out of a local LLM is not what an app wants. An agent wants a JSON object whose fields parse; an IDE plugin wants a tool call it can dispatch. The four local runtimes this cluster already covers - Ollama, llama-server, LM Studio, vLLM - all expose those patterns, but they expose them under different names with different reliability guarantees. This page walks through what each one supports, where grammar-constrained decoding beats JSON mode, and how tool-calling parsers map to specific model families.

JSON mode versus grammar-constrained decoding

Three tiers of reliability sit between “asked for JSON” and “got JSON that parses”:

  • Prompt-only JSON. The model is told in the prompt to emit JSON; nothing prevents it emitting a trailing comma. Ollama’s format parameter on /api/chat and /api/generate goes further: it accepts either the literal string json or a JSON Schema object, and the docs state that with json “the output will always be a well-formed JSON object” (Ollama HTTP API).
  • Schema-constrained sampling. The sampler constrains the next-token distribution so only tokens that keep the output on a valid schema path are allowed. llama-server calls this --grammar (inline GBNF string) and --grammar-file (path), with -j, --json-schema and -jf, --json-schema-file as the JSON-Schema-front doors (llama-server README). vLLM calls it structured outputs - --structured-outputs-config.backend picks the backend, with auto as the default, and the docs name xgrammar, guidance, outlines, and lm-format-enforcer when describing regex support (vLLM structured outputs).
  • Library-level structured generation. A Python library compiles a regex, grammar, or JSON schema to a state machine and patches the inference loop. Outlines and Guidance work this way; LM Studio uses Outlines on the MLX engine path (LM Studio structured output).

The reliability jump from tier 1 to tier 2 is the one that matters in production. A small model asked for JSON in the prompt alone can drop closing braces or invent keys; LM Studio’s docs warn that not all models are capable of structured output, “particularly LLMs below 7B parameters” (LM Studio structured output). Constrained decoding forces the structure, and the cognitive load moves from the model onto the grammar.

The GBNF grammar files shipped with llama.cpp

The grammars/ directory in llama.cpp ships eight reference GBNF grammars: arithmetic.gbnf, c.gbnf, chess.gbnf, english.gbnf, japanese.gbnf, json.gbnf, json_arr.gbnf, and list.gbnf (llama.cpp grammars). The json.gbnf grammar starts with root ::= object and a value production that covers object, array, string, number, and the literals true/false/null; strings permit the standard \", \\, \b, \f, \n, \r, \t, and \uXXXX escapes (json.gbnf source).

For most users the JSON Schema front door is what gets used. llama.cpp converts a supported subset of JSON Schema to GBNF, either ahead of time or at each request (llama-server’s --json-schema flag, or the json_schema / response_format body fields) (llama.cpp grammars README). The README lists real catches: unsupported features are skipped silently, nested $refs are broken, and properties cannot be mixed with anyOf/oneOf in the same type, so it advises inspecting and testing the generated grammar.

What each runtime actually accepts

The four runtimes overlap on /v1/chat/completions (how to call a local LLM API), but the structured-output knobs differ:

RuntimeJSON modeSchema-constrainedGrammar fileTool callingTool parser
Ollamaformat: "json" or JSON Schema on /api/chat (Ollama API)JSON Schema object in formatNotools array on /api/chatBuilt-in
llama-serverresponse_format: {"type":"json_object"} on /v1/chat/completions (llama-server README)response_format: {"type":"json_schema","schema":{...}} plus --grammar/--grammar-fileYes (GBNF)--jinja (default on) with OpenAI-style schema; --chat-template-file for non-aligned templates (llama.cpp function-calling)Generic fallback plus per-model overrides
LM StudioNo schemaless JSON mode documented; JSON Schema via response_format on /v1/chat/completions (LM Studio structured output)response_format: {"type":"json_schema","json_schema":{"strict":true,"schema":{...}}}No (engine-internal)OpenAI-style tools (Tool Use docs page)Engine-internal
vLLMstructured_outputs extra parameter with choice / regex / json / grammar, or StructuredOutputsParams offline; the old guided_* fields were removed in v0.12.0 (vLLM structured outputs)json schema; --structured-outputs-config.backend defaults to autoYes (EBNF via grammar)--enable-auto-tool-choice plus --tool-call-parser <name> (vLLM tool calling)hermes, llama3_json, llama4_pythonic, mistral, granite, granite4, granite-20b-fc, internlm, jamba, xlam, deepseek_v3, deepseek_v31, openai, kimi_k2, hunyuan_a13b, cohere_command3, cohere_command4, longcat, glm45, glm47, functiongemma, qwen3_xml, mimo, step3p5, olmo3, gigachat3, apertus, pythonic, hf

XGrammar, integrated into vLLM in December 2024 and described by its maintainers as the default structured generation backend for most inference engines including vLLM, claims (vendor-reported) “100% structural correctness of the output” for general context-free grammars including JSON, regex, and custom CFGs (XGrammar). The other three backends - guidance, outlines, lm-format-enforcer - differ on regex dialect: xgrammar, guidance, and outlines use Rust-style regex, lm-format-enforcer uses Python’s re (vLLM structured outputs).

Tool calling, parser by parser

Tool calling is the harder of the two. Models trained on OpenAI-style tool schemas emit something in a specific dialect (Hermes/Qwen use <tool_call>...</tool_call> XML; Llama 3 uses {"name":...,"parameters":...} JSON; Llama 4 uses Pythonic [func1(arg=val)] syntax), and the parser has to convert that dialect back into a structured tool_calls array the client code can dispatch.

On llama-server the catch is template fidelity. The docs name Qwen2.5-7B-Instruct, Mistral-Nemo, and Llama-3.3-70B-Instruct as working out of the box under --jinja; DeepSeek R1 distills, Functionary v3.2, Hermes 2/3 Pro Llama-3-8B, Firefunction v2, and c4ai Command R7B each get a --chat-template-file override in the examples: the docs call DeepSeek R1’s official template buggy, and say the others need the right template for their GGUFs (llama.cpp function-calling). If there is no official tool_use template, the docs suggest --chat-template chatml as “a default that works with many models (YMMV!)”; templates no native handler recognises fall back to a generic format that “may consume more tokens”.

The llama.cpp docs also name a quantisation-specific catch: “Beware of extreme KV quantizations (e.g. -ctk q4_0), they can substantially degrade the model’s tool calling performance” (llama.cpp function-calling). The structured output’s grammar always wins - the model cannot emit anything the grammar forbids - but choosing which token is the right one is still the model’s problem, and KV-cache quantisation distorts the logits it ranks with.

vLLM exposes the parser list explicitly; each parser is a ToolParser implementation. The quickstart pattern is vllm serve meta-llama/Llama-3.1-8B-Instruct --enable-auto-tool-choice --tool-call-parser llama3_json --chat-template examples/tool_chat_template_llama3.1_json.jinja (vLLM tool calling). Adding a new model family is a matter of writing a new ToolParser and registering it via --tool-parser-plugin.

What to use, for what

For a single-agent app pulling structured data into a script, Ollama’s format parameter is the path of least resistance on /api/chat, and it accepts a JSON Schema directly. For production systems where parsing failures are unacceptable, llama-server’s --grammar flag or vLLM’s json structured output removes the structural failure mode at the cost of a request-time grammar compile. For a desktop install with no command line, LM Studio wires grammar sampling into the GGUF engine and Outlines into the MLX engine without the user touching a flag.

For tool calling specifically, the practical rule is: pick the model first, then pick the parser. The best local coding models by VRAM page covers the model half; the parser half lives in vLLM’s --tool-call-parser list, llama-server’s --chat-template-file overrides, and Ollama’s built-in parser. The same model can produce structurally valid JSON via grammar and yet still call the wrong tool - reliability of the call is a function of the model’s training data, the parser’s tolerance, and (on llama.cpp) the KV-cache quantisation.

Bottom line

If you only need JSON and don’t care about the schema, Ollama’s format: "json" and llama-server’s json_object do it directly, and the others accept a permissive schema. If you need enforced schema - an enum, nested objects, required fields - use llama-server’s --grammar/--json-schema, vLLM’s json structured output, Ollama’s schema-valued format, or LM Studio’s json_schema mode. If you need tool calls, name the model, name the parser, and test it against a known-good prompt; the parser list on vLLM is the most explicit, llama-server’s --chat-template-file overrides are the most surgical, and Ollama’s built-in parser is the path of least resistance for models that the Ollama team has shipped a chat template for. The serving side - exposing the endpoint to a household, putting it behind auth, monitoring it - is the serve-a-local-model-to-your-household page’s job, not this one.