Testing
Plugin
promptfoo
promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps. Website · Getting Started · Red Teaming · Documentation · Discord Promptfoo is now part of OpenAI.
Plugin Signal76/100Solid
maintained100
capable41
documented75
adopted94
Install on Claude Code
> /plugin marketplace add promptfoo/promptfoo> /plugin install promptfoo@promptfoo
Stats
25,336
Stars
2,351
Forks
Active
Maintenance
TypeScript
Language
MIT
License
1h ago
Last commit
3y ago
Created
Repo: promptfoo/promptfoo
What's inside
Skill(6 skills)
/promptfoo-evalsWrite, refine, run, and QA promptfoo evaluation suites: promptfooconfig.yaml, prompts, providers, vars, tests, assertions, model-graded rubrics, transforms, datasets, exports, and CI gates. Use for non-redteam eval coverage, regression tests, or new eval matrices. Do not use for/redteam-plugin-developmentStandards for creating redteam plugins and graders. Use when creating new plugins, writing graders, or modifying attack templates./search-paramsURL search param and hash state management. Use when adding or modifying URL search params, working with useSearchParams, setSearchParams, useSearchParamState, or navigate() with query strings or hash fragments, or fixing browser back/forward button issues.
Badges
README
Promptfoo: LLM evals & red teaming
Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.
Quick Start
Requires Node.js >=22.22.0 for npm and npx usage. Node.js 24 LTS
is recommended; see the runtime support guide.
npm install -g promptfoo
promptfoo init --example getting-started
Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.
Most LLM providers require an API key. Set yours as an environment variable:
export OPENAI_API_KEY=sk-abc123
Once you're in the example directory, run an eval and view results:
cd getting-started
promptfoo eval
promptfoo view
See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.
What can you do with Promptfoo?
- Test your prompts and models with automated evaluations
- Secure your LLM apps with red teaming and vulnerability scanning
- Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
- Automate checks in CI/CD
- Review pull requests for LLM-related security and compliance issues with code scanning
- Share results with your team
Here's what it looks like in action:
It works on the command line too:
It also can generate security vulnerability reports:
Why Promptfoo?
- Developer-first: Fast, with features like live reload and caching
- Private: LLM evals run 100% locally - your prompts never leave your machine
- Flexible: Works with any LLM API or programming language
- Battle-tested: Powers LLM apps serving 10M+ users in production
- Data-driven: Make decisions based on metrics, not gut feel
- Open source: MIT licensed, with an active community
Learn More
- Getting Started
- Full Documentation
- Red Teaming Guide
- CLI Usage
- Node.js Package
- Supported Models
- Code Scanning Guide
Contributing
We welcome contributions! Check out our contributing guide to get started.
Join our Discord community for help and discussion.
File tree
Showing a partial view of a very large repo.
FAQ
promptfoo is a Claude Code plugin with 6 hand-picked skills for testing work, indexed on Flowy. Install it with the command on its page. It includes promptfoo-evals, redteam-plugin-development, search-params. Its skills do not fire on their own yet. Request auto-invocation to have Flowy route them as you prompt. Free and open source.
- What is promptfoo?
- promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps. Website · Getting Started · Red Teaming · Documentation · Discord Promptfoo is now part of OpenAI.
- How do I install promptfoo?
- Run these in Claude Code: /plugin marketplace add promptfoo/promptfoo then /plugin install promptfoo@promptfoo. Then prompt normally.
- Does promptfoo auto-invoke its skills?
- Not yet. It is indexed on Flowy as a plain plugin. Request auto-invocation on its page and Flowy will route its skills for you as you prompt.
- Is promptfoo free?
- Yes. Flowy is free and open source, with nothing gated. You can read every skill in full before you install.
Is this plugin yours?
Claim it with GitHubSubmit a pluginPromote it