/defending-llms-with-guardrails

name: defending-llms-with-guardrails
description: Deploys Llama Guard 3 safety classification, NeMo Guardrails programmable dialogue rails, and LLM Guard input/output scanner pipelines as complementary runtime defenses that inspect and constrain LLM prompts and responses. Use when adding a production runtime safety layer to an LLM, RAG, or agent application to block jailbreaks, prompt injection (OWASP LLM01), toxic content, or sensitive-data leakage before it reaches or leaves the model.
domain: cybersecurity
subdomain: ai-security
tags:
- ai-security
- llm-guardrails
- llama-guard
- nemo-guardrails
- llm-guard
- prompt-injection
- content-moderation
- runtime-defense
version: '1.0'
author: mahipal
license: Apache-2.0
nist_ai_rmf:
- MANAGE-2.1
atlas_techniques:
- AML.T0054

Defending LLMs with Guardrails

> **Defensive scope:** This skill describes runtime defenses for production LLM applications. The example jailbreak/injection payloads exist only to validate that guardrails block them. Test against systems you own or are authorized to assess.

Overview

Large language model (LLM) applications are exposed to adversarial input (jailbreaks, prompt injection, toxic content) and can emit unsafe, biased, or sensitive output. A guardrail is a runtime control that inspects and constrains the data flowing into and out of an LLM. Three production-grade, open-source guardrail systems dominate the ecosystem and are complementary rather than mutually exclusive:

• **Llama Guard 3** (Meta) — a Llama-3.1-8B model fine-tuned as a *safety classifier*. Given a prompt or a response, it emits `safe` or `unsafe` plus the violated MLCommons hazard categories (S1–S14). It is the strongest *semantic* content-safety classifier of the three and supports prompt classification, response classification, and tool-call/code-interpreter classification across 8 languages.
• **NeMo Guardrails** (NVIDIA) — a *programmable* dialogue-rail framework. You define `input`, `output`, `dialog`, `retrieval`, and `execution` rails in a `config.yml` plus Colang (`.co`) flows. It can call external models (including Llama Guard) as actions, enforce topical boundaries, and add fact-checking/jailbreak-detection rails.
• **LLM Guard** (Protect AI) — a *scanner pipeline* with 15 input scanners and 20 output scanners (PromptInjection, Toxicity, Anonymize/Deanonymize, Secrets, BanTopics, Sensitive, Regex, etc.). It returns a sanitized string, a validity flag, and a risk score per scanner, making it ideal for a deterministic pre/post pipeline.

This skill maps to MITRE ATLAS **AML.T0054 — LLM Jailbreak**: the guardrail layer is the mitigation that detects and blocks jailbreak/injection attempts before they reach (or after they leave) the model.

When to Use

• When deploying an LLM/RAG/agent application to production and needing a runtime safety layer.
• When you must block jailbreaks and prompt injection (OWASP LLM01) before they reach the model.
• When you must moderate model output for toxicity, PII leakage, secrets, or off-topic responses.
• When validating that a guardrail configuration actually blocks a corpus of known-bad payloads.
• When layering defense-in-depth: a deterministic scanner (LLM Guard) plus a semantic classifier (Llama Guard) plus dialog rails (NeMo).

Prerequisites

• Python 3.9+ (LLM Guard requires 3.9+; Llama Guard via transformers requires `transformers>=4.43`).
• GPU recommended for Llama Guard 3 8B (CPU works for the 1B variant or quantized builds).
• A Hugging Face account with accepted Meta Llama license to download `meta-llama/Llama-Guard-3-8B`.

# LLM Guard
python -m pip install llm-guard

# NeMo Guardrails
python -m pip install nemoguardrails

# Llama Guard via Hugging Face transformers
python -m pip install "transformers>=4.43" torch accelerate huggingface_hub
huggingface-cli login   # accept the Meta Llama license first on the model page

Objectives

• Run Llama Guard 3 as a prompt and response safety classifier and parse its category output.
• Build an LLM Guard input/output scanner pipeline with PromptInjection, Toxicity, Secrets, and Anonymize scanners.
• Author a NeMo Guardrails `config.yml` plus Colang flows with input/output/jailbreak rails.
• Wire Llama Guard into NeMo as a content-safety check.
• Validate the combined stack against a corpus of jailbreak and injection payloads.

MITRE ATT&CK Mapping

| ID | Tactic | Official Technique Name | Role in this skill | |----|--------|-------------------------|--------------------| | AML.T0054 | ATLAS: Defense Evasion / Impact | LLM Jailbreak | Guardrails detect and block the jailbreak attempt this technique describes | | AML.T0051 | ATLAS: Initial Access | LLM Prompt Injection | Input rails / PromptInjection scanner block direct injection | | AML.T0051.001 | ATLAS: Initial Access | LLM Prompt Injection: Indirect | Retrieval/input scanning blocks injection in retrieved content | | AML.T0057 | ATLAS: Exfiltration | LLM Data Leakage | Output scanners (Sensitive, Secrets, Deanonymize) block leakage |

Workflow

Step 1: Classify prompts and responses with Llama Guard 3

Llama Guard takes a chat-format conversation and returns `safe` or `unsafe\nS<n>`. Use the `apply_chat_template` helper which builds the MLCommons-taxonomy prompt for you.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "meta-llama/Llama-Guard-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)

def moderate(chat):
    input_ids = tokenizer.apply_chat_template(chat, return_tensors="pt").to(model.device)
    output = model.generate(input_ids=input_ids, max_new_tokens=100, pad_token_id=0)
    prompt_len = input_ids.shape[-1]
    return tokenizer.decode(output[0][prompt_len:], skip_special_tokens=True)

# Classify a user prompt (role 'user' = prompt classification)
print(moderate([{"role": "user