/ai-observability-promptfoo

name: ai-observability-promptfoo
description: Testing and evaluation framework for LLM prompts and applications -- promptfooconfig.yaml, assertions, model-graded evals, red teaming, CI/CD integration, custom providers, and comparative evaluation

Promptfoo Patterns

> **Quick Guide:** Use promptfoo for systematic LLM evaluation. Define prompts, providers, and test cases in `promptfooconfig.yaml`. Use assertion types (`contains`, `is-json`, `llm-rubric`, `similar`, `cost`, `latency`) to validate outputs. Use `promptfoo eval` to run (exits with code 100 on test failures), `promptfoo view` for results UI. Use model-graded assertions (`llm-rubric`, `factuality`) for subjective quality. Use `promptfoo redteam run` for security scanning. Use `--share` flag or `promptfoo share` to share results. All provider API keys come from environment variables -- never hardcode them.

---

<critical_requirements>

CRITICAL: Before Using This Skill

**(You MUST define test cases with explicit `assert` arrays -- tests without assertions only capture output without validating it)**

**(You MUST use `llm-rubric` for subjective quality evaluation -- do NOT rely solely on deterministic assertions for natural language output)**

**(You MUST set `threshold` on similarity and model-graded assertions -- omitting thresholds uses defaults that may not match your quality bar)**

**(You MUST use environment variables for all API keys -- never hardcode keys in promptfooconfig.yaml or provider configs)**

**(You MUST verify `promptfoo eval` exit code in CI pipelines -- it returns exit code 100 on test failures, exit code 1 on other errors)**

</critical_requirements>

---

**Auto-detection:** promptfoo, promptfooconfig, promptfooconfig.yaml, promptfoo eval, promptfoo view, promptfoo redteam, llm-rubric, model-graded-closedqa, promptfoo share, promptfoo cache, assertion type, LLM evaluation, prompt testing, red teaming, PROMPTFOO_CONFIG

**When to use:**

• Writing or evaluating LLM prompts across one or more providers
• Setting up automated test suites for LLM-powered features
• Comparing model outputs side-by-side (GPT vs Claude vs Gemini)
• Running model-graded evaluations (LLM-as-a-judge)
• Red teaming LLM applications for security vulnerabilities
• Integrating LLM quality gates into CI/CD pipelines
• Validating structured output (JSON, function calls) from LLMs

**Key patterns covered:**

• `promptfooconfig.yaml` structure (prompts, providers, tests, defaultTest)
• Assertion types (deterministic, model-graded, performance)
• Custom TypeScript providers
• Red teaming configuration (plugins, strategies)
• CI/CD integration with GitHub Actions
• Programmatic API (`evaluate()` function)
• Result sharing and caching

**When NOT to use:**

• Unit testing application code (use your test runner)
• Load testing / benchmarking API throughput (use a load testing tool)
• Runtime monitoring of production LLM calls (use observability tooling)

---

Examples Index

• [Core: Config & Assertions](examples/core.md) -- promptfooconfig.yaml structure, providers, prompts, test cases, assertion types
• [Model-Graded & Advanced Assertions](examples/model-graded.md) -- llm-rubric, factuality, similar, context evaluation, custom assertions
• [Red Teaming](examples/red-teaming.md) -- Security scanning, plugins, strategies, presets
• [Custom Providers & Programmatic API](examples/custom-providers.md) -- TypeScript providers, evaluate() function, CI/CD integration
• [Quick API Reference](reference.md) -- CLI commands, assertion type table, provider IDs, red team plugins

---

<philosophy>

Philosophy

Promptfoo brings **test-driven development to LLM applications**. Instead of manually checking outputs, you define expected behaviors as assertions and run them systematically across prompts and providers.

**Core principles:**

1. **Declarative test definitions** -- YAML config over imperative test scripts. Define prompts, providers, test cases, and assertions in `promptfooconfig.yaml`. No code required for standard evaluations. 2. **Assertion-driven validation** -- Every test case should have assertions. Deterministic assertions (`contains`, `is-json`, `equals`) for structured output; model-graded assertions (`llm-rubric`, `factuality`) for subjective quality. 3. **Comparative evaluation** -- Run the same tests across multiple providers or prompt variants simultaneously. The results matrix shows which combination performs best. 4. **Shift-left LLM testing** -- Catch prompt regressions in CI before they reach production. `promptfoo eval` exits with code 100 on test failures, making it a natural CI quality gate. 5. **Red teaming as a first-class concern** -- Security scanning for prompt injection, PII leakage, harmful content, and jailbreak vulnerabilities is built in, not bolted on.

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Basic Configuration

Every promptfoo project starts with `promptfooconfig.yaml`. Three required sections: `prompts`, `providers`, `tests`.

# promptfooconfig.yaml
description: "Translation quality evaluation"

prompts:
  - "Convert the following to {{language}}: {{input}}"

providers:
  - openai:gpt-4o
  - anthropic:messages:claude-sonnet-4-6

tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: icontains
        value: "bonjour"
      - type: llm-rubric
        value: "Output is a natural French translation, not word-for-word"

**Why good:** Declarative config, multi-provider comparison, both deterministic and model-graded assertions

# BAD: Tests without assertions
tests:
  - vars:
      language: French
      input: Hello world
  # No assert array -- output is captured but never validated

**Why bad:** Tests without assertions only log output, they never fail -- you lose the entire point of automated evaluation

**See:** [examples/core.md](examples/core.md) for prompts from files, provider config, defaultTest, variable lo