Testing
Plugin
promptfoo
promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps. Website · Getting Started · Red Teaming · Documentation · Discord Promptfoo is now part of OpenAI.
Plugin Signal77/100Solid
maintained100
capable43
documented75
adopted93
Install on Claude Code
> /plugin marketplace add promptfoo/promptfoo> /plugin install promptfoo@promptfoo
Stats
24,102
Stars
2,172
Forks
Active
Maintenance
TypeScript
Language
MIT
License
15h ago
Last commit
3y ago
Created
Repo: promptfoo/promptfoo
What's inside
Skills
7/promptfoo-evalsWrite, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for/redteam-plugin-developmentStandards for creating redteam plugins and graders. Use when creating new plugins, writing graders, or modifying attack templates./search-paramsURL search param and hash state management. Use when adding or modifying URL search params, working with useSearchParams, setSearchParams, useSearchParamState, or navigate() with query strings or hash fragments, or fixing browser back/forward button issues./promptfoo-evalsWrite, refine, run, and QA non-redteam promptfoo eval suites after the target or provider already works: prompts, vars, test cases, assertions, model-graded rubrics, transforms, datasets, output exports, filters, and CI gates. Use for regression tests and eval-suite authoring./promptfoo-provider-setupConfigure promptfoo providers or redteam targets for hosted models, live HTTP APIs, Python/JavaScript local scripts, agent SDKs, or multi-input systems. Use when connecting promptfoo to the system under test, mapping vars, auth env vars, request bodies, response transforms, or/promptfoo-redteam-runRun, rerun, inspect, and QA promptfoo redteam scans from generated redteam YAML or an existing redteam setup config. Use when executing `promptfoo redteam eval` or `promptfoo redteam run`, exporting results, triaging attack success rate, grader failures, target errors,
Badges
5
README
1Promptfoo: LLM evals & red teaming
Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.
Quick Start
Requires Node.js >=22.22.0 for npm and npx usage. Node.js 24 LTS
is recommended; see the runtime support guide.
npm install -g promptfoo
promptfoo init --example getting-started
Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.
Most LLM providers require an API key. Set yours as an environment variable:
export OPENAI_API_KEY=sk-abc123
Once you're in the example directory, run an eval and view results:
cd getting-started
promptfoo eval
promptfoo view
See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.
What can you do with Promptfoo?
- Test your prompts and models with automated evaluations
- Secure your LLM apps with red teaming and vulnerability scanning
- Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
- Automate checks in CI/CD
- Review pull requests for LLM-related security and compliance issues with code scanning
- Share results with your team
Here's what it looks like in action:
It works on the command line too:
It also can generate security vulnerability reports:
Why Promptfoo?
- Developer-first: Fast, with features like live reload and caching
- Private: LLM evals run 100% locally - your prompts never leave your machine
- Flexible: Works with any LLM API or programming language
- Battle-tested: Powers LLM apps serving 10M+ users in production
- Data-driven: Make decisions based on metrics, not gut feel
- Open source: MIT licensed, with an active community
Learn More
- Getting Started
- Full Documentation
- Red Teaming Guide
- CLI Usage
- Node.js Package
- Supported Models
- Code Scanning Guide
Contributing
We welcome contributions! Check out our contributing guide to get started.
Join our Discord community for help and discussion.
File tree
1500promptfoo/promptfooView on GitHub
.agents/
AGENTS.md
plugins/
marketplace.json
skills/
redteam-plugin-development/
SKILL.md
search-params/
SKILL.md
.biomeignore
.claude/
.claude-plugin/
marketplace.json
settings.json
skills/
promptfoo-evals/
references/
cheatsheet.md
SKILL.md
redteam-plugin-development/
SKILL.md
search-params/
SKILL.md
.cursor/
mcp.json
.devcontainer/
devcontainer.json
docker-compose.yml
Dockerfile.dev
.dockerignore
.git-blame-ignore-revs
.github/
AGENTS.md
assets/
promptfooconfig.yaml
CODEOWNERS
FUNDING.yml
ISSUE_TEMPLATE/
bug_report.md
feature_request.md
promptfoo-code-scan.yaml
scripts/
determine-review-scope.sh
workflows/
deploy-launcher.yml
docker.yml
image-actions.yml
main.yml
promptfoo-code-scan.yml
release-please-format.yml
release-please-sha-drift.yml
release-please.yml
tusk-test-runner-app-vitest-unit-tests.yml
tusk-test-runner-vitest-unit-tests.yml
validate-pr-title.yml
validate-renovate-config.yml
.gitignore
.husky/
pre-commit
.mailmap
.npmignore
.npmrc
.nvmrc
.prettierignore
.prettierrc.yaml
.release-please-manifest.json
.rubocop.yml
.ruff.toml
.vscode/
extensions.json
settings.json
AGENTS.md
architecture/
edge-baseline.json
layers.json
biome.jsonc
CHANGELOG.md
CITATION.cff
CLAUDE.md
CODE_OF_CONDUCT.md
code-scan-action/
.github/
workflows/
test-event.json
test-scan.yml
action.yml
AGENTS.md
CHANGELOG.md
package-lock.json
package.json
README.md
src/
auth.ts
config.ts
github.ts
main.ts
tsconfig.json
tsconfig.tests.json
codecov.yml
CONTRIBUTING.md
Dockerfile
docs/
agents/
AGENTS.md
codex-app-server-provider-notes.md
coding-agent-provider-taxonomy.md
database-security.md
dependency-management.md
git-workflow.md
logging.md
pr-conventions.md
python.md
architecture/
packages.md
plans/
2026-01-08-plugins-state-management-refactor.md
2026-05-02-multi-package-system-proposal.md
eng-1770.md
smoke-tests.md
scheduler-architecture.md
drizzle/
drizzle.config.ts
0000_lush_hellion.sql
0001_wide_calypso.sql
0002_tidy_juggernaut.sql
0003_lively_naoko.sql
0004_minor_peter_quill.sql
0005_silky_millenium_guard.sql
0006_harsh_caretaker.sql
0007_cloudy_wong.sql
0008_broad_boomer.sql
0009_strong_marten_broadcloak.sql
0010_needy_bishop.sql
0011_moaning_millenium_guard.sql
0012_late_marten_broadcloak.sql
0013_previous_dormammu.sql
0014_lazy_captain_universe.sql
0015_zippy_wallop.sql
0016_jazzy_zemo.sql
0017_reflective_praxagora.sql
0018_fat_vanisher.sql
0019_new_clint_barton.sql
0020_skinny_maverick.sql
0021_mysterious_madelyne_pryor.sql
0022_sleepy_ultimo.sql
0023_wooden_mandrill.sql
0024_repair_eval_redteam_flags.sql
AGENTS.md
CLAUDE.md
meta/
_journal.json
0000_snapshot.json
0001_snapshot.json
0002_snapshot.json
0005_snapshot.json
0006_snapshot.json
0007_snapshot.json
0008_snapshot.json
0009_snapshot.json
0010_snapshot.json
0011_snapshot.json
0012_snapshot.json
0013_snapshot.json
0014_snapshot.json
0015_snapshot.json
0016_snapshot.json
0018_snapshot.json
0019_snapshot.json
0020_snapshot.json
0021_snapshot.json
0022_snapshot.json
0023_snapshot.json
0024_snapshot.json
examples/
AGENTS.md
amazon-bedrock/
agents/
promptfooconfig.multi-agent.yaml
promptfooconfig.yaml
README.md
models/
assets/
hello.wav
weather.wav
cat.jpg
converse_image_prompt.json
llama_vision_prompt.json
nova_multimodal_prompt.json
nova_sonic_prompt.js
promptfooconfig.a21.yaml
promptfooconfig.claude.yaml
promptfooconfig.converse-comprehensive.yaml
promptfooconfig.converse-images.yaml
promptfooconfig.converse-mcp.yaml
promptfooconfig.converse-system.yaml
promptfooconfig.converse-tools.yaml
promptfooconfig.converse.yaml
promptfooconfig.deepseek.yaml
promptfooconfig.grok.yaml
promptfooconfig.inference-profiles-simple.yaml
promptfooconfig.inference-profiles.yaml
promptfooconfig.kb.yaml
promptfooconfig.llama-vision.yaml
promptfooconfig.llama.yaml
promptfooconfig.mantle.yaml
promptfooconfig.mistral.yaml
promptfooconfig.nova-2-converse.yaml
promptfooconfig.nova-2-reasoning.yaml
promptfooconfig.nova-sonic.yaml
promptfooconfig.nova.multimodal.yaml
promptfooconfig.nova.tool.yaml
promptfooconfig.nova.yaml
promptfooconfig.openai-compatible.yaml
promptfooconfig.openai-frontier.yaml
promptfooconfig.openai.yaml
promptfooconfig.qwen.yaml
promptfooconfig.yaml
prompts.txt
README.md
README.md
video/
promptfooconfig.luma-ray.yaml
promptfooconfig.nova-reel.yaml
README.md
anthropic/
claude-code-session/
promptfooconfig.yaml
README.md
fable-5-coding/
promptfooconfig.yaml
README.md
mcp/
promptfooconfig.yaml
README.md
memory-tool/
promptfooconfig.yaml
README.md
opus-4-6-coding/
promptfooconfig.yaml
README.md
opus-4-8-coding/
promptfooconfig.yaml
README.md
sonnet-5/
promptfooconfig.yaml
README.md
structured-outputs/
customer_schema.json
nested_format.json
output_format.json
promptfooconfig.yaml
README.md
web-tools/
promptfooconfig.yaml
README.md
azure/
azure-mai/
promptfooconfig.vision-judge.yaml
promptfooconfig.yaml
README.md
assistant/
callbacks/
weather.js
promptfooconfig-file-search.yaml
promptfooconfig-function.yaml
promptfooconfig-multi-tool.yaml
README.md
tools/
multiple-tools.json
weather-function.json
claude/
promptfooconfig.yaml
README.md
comparison/
promptfooconfig.yaml
README.md
deepseek/
promptfooconfig.yaml
README.md
foundry-agent/
promptfooconfig.yaml
README.md
llama/
promptfooconfig.yaml
README.md
mistral/
promptfooconfig.yaml
README.md
openai/
assets/
earth.jpg
prompt.vision.json
promptfooconfig.vision.yaml
promptfooconfig.yaml
README.md
README.md
claude-agent-sdk/
advanced/
advanced-options/
promptfooconfig.yaml
README.md
.gitignore
hooks.js
promptfooconfig.yaml
workspace/
auth.js
package.json
ask-user-question/
promptfooconfig.yaml
basic/
promptfooconfig.tracing.yaml
promptfooconfig.yaml
cyber-espionage/
agent-prompt.txt
promptfooconfig.yaml
workspace/
placeholder
mcp/
promptfooconfig.yaml
plugins/
promptfooconfig.yaml
sample-plugin/
.claude-plugin/
plugin.json
skills/
standards-check/
SKILL.md
sample-project/
src/
index.ts
README.md
skill-comparison/
fixtures/
v1/
.claude/
skills/
review-standards/
SKILL.md
src/
auth.ts
v2/
.claude/
skills/
review-standards/
SKILL.md
src/
auth.ts
promptfooconfig.yaml
README.md
skills/
promptfooconfig.yaml
sample-project/
.claude/
skills/
code-review/
SKILL.md
src/
auth.ts
structured-output/
promptfooconfig.yaml
working-dir/
promptfooconfig.yaml
sample-project/
auth.ts
calculator.py
index.js
main.py
package.json
claude-thinking/
promptfooconfig.yaml
prompts.txt
README.md
claude-vision/
prompt.json
promptfooconfig.yaml
README.md
CLAUDE.md
compare-agentic-sdks/
promptfooconfig.yaml
README.md
test-codebase/
payment_processor.py
README.md
user_service.py
compare-claude-vs-gpt/
compare-claude-vs-gpt-image/
prompt.js
prompt.py
promptfooconfig.yaml
README.md
prompt.yaml
promptfooconfig.yaml
README.md
compare-deepseek-r1-vs-openai-o1/
promptfooconfig.yaml
README.md
compare-gpt-model-tiers/
compare-gpt-model-tiers-mmlu-pro/
promptfooconfig.yaml
README.md
... 1600 moreShowing a partial view of a very large repo.
FAQ
promptfoo is a Claude Code plugin with 7 hand-picked skills for testing work, indexed on Flowy. Install it with the command on its page. It includes promptfoo-evals, redteam-plugin-development, search-params. Its skills do not fire on their own yet. Request auto-invocation to have Flowy route them as you prompt. Free and open source.
- What is promptfoo?
- promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps. Website · Getting Started · Red Teaming · Documentation · Discord Promptfoo is now part of OpenAI.
- How do I install promptfoo?
- Run these in Claude Code: /plugin marketplace add promptfoo/promptfoo then /plugin install promptfoo@promptfoo. Then prompt normally.
- Does promptfoo auto-invoke its skills?
- Not yet. It is indexed on Flowy as a plain plugin. Request auto-invocation on its page and Flowy will route its skills for you as you prompt.
- Is promptfoo free?
- Yes. Flowy is free and open source, with nothing gated. You can read every skill in full before you install.