Local AI by Editorial Team

Local LLM guardrails compared (October 2026)

Llama Guard, Prompt Guard, NeMo Guardrails, guardrails-ai, and llm-guard: a feature-by-feature comparison of options for adding guardrails to a self-hosted LLM.

Updated October 16, 2026

Anyone running a self-hosted LLM eventually has to decide what it is allowed to say, and what to do when a prompt is malicious or off-topic. Cloud providers resolve this for you; on a self-hosted stack it is your problem. The open-source ecosystem now offers five approaches: purpose-built classifier models (Llama Guard, Llama Prompt Guard), programmable policy engines (NeMo Guardrails, guardrails-ai), and a filter pipeline of pre-trained detectors (ProtectAI llm-guard). They are not interchangeable, the trade-offs are specific, and the licence terms move quickly.

What “guardrails” actually means at a self-hosted LLM

A guardrail is anything that sits between the user prompt and the model’s response, or between the response and the caller. The NeMo Guardrails README defines five categories that the rest of the ecosystem has converged on:

“Input rails: applied to the input from the user; an input rail can reject the input, stopping any additional processing, or alter the input (e.g., to mask potentially sensitive data, to rephrase).” “Output rails: applied to the output generated by the LLM; an output rail can reject the output, preventing it from being returned to the user, or alter it (e.g., removing sensitive data).”

NeMo also names three more: dialog rails (steer the conversation itself), retrieval rails (filter RAG chunks), and execution rails (filter tool calls and their results). Most self-hosters only need input and output rails, which is why classifier-only tools like Llama Guard cover the majority of cases without a separate framework.

The Llama Guard family: classifiers you can run alongside your model

Meta publishes four distinct safety models on Hugging Face, all under the meta-llama org and all gated: manual (you must accept the licence on HF before downloading):

Modelsafetensors.totalcardData.license / license_nameRoleLast modified
Llama-Guard-3-1B1,498,482,688 B (~1.5 GB)llama3.2, gated manualOutput classifier (14 S-categories)2024-09-26
Llama-Guard-3-8B8,030,261,248 B (~8.03 GB)llama3.1, gated manualOutput classifier (14 S-categories, higher accuracy)2024-10-11
Llama-Guard-4-12B12,001,097,216 B (~12.0 GB)other / llama4, gated manualOutput classifier (input + output)2025-04-29
Llama-Prompt-Guard-2-86M278,810,882 B (~278.8 MB)other / llama4, gated manualInput-side jailbreak / injection classifier2025-04-29
Llama-Prompt-Guard-2-22M70,830,722 B (~70.8 MB)other / llama4, gated manualInput-side, smaller variant2025-04-29

The prompt-side and output-side models are separate checkpoints. Prompt Guard is a small encoder (86M or 22M parameters) that flags an incoming message as safe or a jailbreak; it runs next to an Ollama installation without a second GPU. The Guard 3/4 series is the response classifier called after the model generates, returning a structured verdict listing which of 14 S-categories (Violence, Self-Harm, Sexual Content, etc.) tripped.

The catch: every checkpoint in this family carries a Meta Llama Community License clause, not Apache or MIT. The 700M-MAU clause from Llama 4 (which Llama-Guard-4-12B and Prompt Guard 2 inherit via the license_name: llama4 field) is fine for personal use but matters if you build a hosted product on top of them. The Hugging Face cardData.license: other flag is your signal to read the upstream LICENSE file before shipping.

Programmable engines vs filter pipelines

The two non-classifier paths are very different animals. Their GitHub metadata, fetched on 2026-10-16:

ProjectStarsForksOpen issuesLicense (spdx_id)Last pushArchived?
NVIDIA-NeMo/Guardrails7,310869270NOASSERTION (the GitHub API returns “Other” - see repo LICENSE)2026-10-15No
guardrails-ai/guardrails7,52672272Apache-2.02026-10-15No
protectai/llm-guard3,21646638MIT2026-07-08Yes

NeMo Guardrails is a programmable toolkit: you write Colang or YAML policies that describe what the model is allowed to talk about, and a parallel LLM call decides if a message fits. The README lists built-in guards including “jailbreak_detection”, “content_safety”, “topic_safety”, and integrations with NVIDIA safety models. This is the most flexible option but the most complex - you pay for a second LLM call per turn, doubling latency.

guardrails-ai takes the opposite tack: you define structured validators (“the response must be valid JSON”, “no SSN in the response”, “the model must mention a disclaimer”) and the library wraps an LLM call in a try/except until validators pass. Apache-2.0, no second model in the loop, fewer open issues than NeMo. The trade-off is that it is opinionated about output structure and less helpful for free-form chat moderation.

ProtectAI’s llm-guard is a different shape: a pipeline of pre-trained scanners (anonymise PII, ban-topics, detect prompt injection, redact emails) running synchronously on the request and response. Fastest of the three at runtime and the only one with no LLM call in the loop. The 2026-10-16 GitHub API returns "archived": true and pushed_at: 2026-07-08 - the maintainers have not pushed in over three months. Still usable and MIT-licensed; not the default choice for new deployments, but if your stack runs a single Python service and needs PII redaction at line-rate, it is the cleanest option to drop in.

When the guard itself is “local” and when it isn’t

Four hosting patterns come up in practice, and only the first three keep every byte on your machine:

  • Model + classifier on the same box. Llama Guard 3 1B and Prompt Guard 2 86M together add roughly 1.8 GB of weights; on a 16 GB box they coexist with your main model. Llama Guard 4 12B is too large to load side-by-side with a 70B-class model on a single card.
  • Classifier as a separate HTTP service. Run the guard on a different port (llama-server --port 8081) or container; your application calls it before and after the main LLM. This is the pattern best-local-llm-runners documents for Ollama and llama-server side-by-side.
  • Filter pipeline as middleware. llm-guard slots between your app and the LLM as a small HTTP proxy, closest to a classic API gateway.
  • Programmable engine in-process. NeMo Guardrails and guardrails-ai run as part of your application; they hold conversation state and need to persist somewhere. Cloud moderation endpoints like /v1/moderations are out of scope for this page but are the comparison most readers are coming from.

Self-host-specific catches that are easy to miss

Three things only bite you on a self-hosted setup:

  1. Latency compounds. A chain of input classifier + main LLM + output classifier on consumer hardware typically adds 200-900 ms to first-token time. Drop the input classifier if you only need output moderation.
  2. The classifier is itself a model you operate. Llama Guard 3 8B at FP16 wants ~16 GB VRAM; Llama Guard 4 12B wants ~24 GB. Running one alongside a 32B-class reasoning model is not practical on a single card.
  3. Gated weights are a deployment gotcha. gated: manual on HF means your container needs an HF token at build time. Once downloaded, weights serve without that token, but air-gapped installs need pre-downloaded mirrors - a separate workflow from Ollama’s default pull.

A common misconfig: routing the model’s output through Llama Guard and then sending the Guard’s verdict back through the same model for a “polite rewrite”. The guard is a classifier, not a rewriter - send its verdict to your application code, not back into the LLM.

The Bottom line

Pick by what you actually need:

  • One classifier, no framework. Llama Guard 3 1B on the output, Prompt Guard 2 86M on the input. Both run alongside Ollama on a 16 GB box; both are gated but download in a few minutes.
  • Programmable topic and tone controls. NeMo Guardrails, accepting the cost of a second LLM call per turn. Note the repo migrated from NVIDIA/NeMo-Guardrails to NVIDIA-NeMo/Guardrails (the GitHub API URL must point at the new location).
  • Structured-output validation around an existing model. guardrails-ai, Apache-2.0, no second LLM call, lower issue count than NeMo.
  • Synchronous filter pipeline (PII redaction, banned topics, prompt-injection detection). llm-guard, MIT, archived upstream but functional; consider a maintained fork for ongoing updates.

There is no single “best” guardrail. Self-hosted stacks commonly get this wrong by treating it as a model-only problem and shipping with no guard at all; the safe move is to name the role (input vs output vs topic vs structure) before picking the tool.