/prompt-evaluation-runner

name: prompt-evaluation-runner
description: Use when evaluating prompts, LLM outputs, red-team suites, or model behavior with local eval configs and safe provider/cost controls.

Prompt Evaluation Runner

When to use

Use when you need to evaluate an LLM app, test a prompt systematically, or run red-team/vulnerability scans against a target model or application.

Requirements / Checks

1. Check if an evaluation tool is defined in project deps, scripts, lockfiles, or local toolchain (e.g., `promptfoo`, `evals`, `braintrust`). 2. Do not run unvetted remote runners without checking the project's toolchain first (e.g., avoid `npx promptfoo@latest` if `promptfoo` is already installed locally). 3. If no runner exists, ask before adding a dev dependency or using an ephemeral runner. 4. Confirm expected cost, provider, API keys, and network target before any execution.

Workflow

1. **Define risk** — state target behavior, failure mode, provider(s), and budget limits before writing any config.

2. **Choose assertions** — prefer deterministic checks first:

| Assertion type | When to use | |---|---| | `contains` / `not-contains` | Output must include/exclude specific text | | `regex` | Structured output pattern (e.g., JSON key present) | | `json-schema` | Output must conform to a schema | | `cost` | Must stay under a token/dollar budget | | `latency` | Must respond within N ms | | `javascript` / `python` | Custom logic when simpler types don't fit | | Model grader | Last resort — only for subjective quality checks |

3. **Use model graders sparingly** — pin the grader model and provider explicitly; document the cost and non-determinism risk.

4. **Minimal config structure**:

   description: "Test that the summarizer stays under 200 words"
   providers:
     - id: openai:gpt-4o-mini
       config:
         temperature: 0
   prompts:
     - "Summarize: {{input}}"
   defaultTest:
     assert:
       - type: javascript
         value: output.split(' ').length < 200
   tests:
     - vars:
         input: "{{env.TEST_DOCUMENT}}"

5. **Handle env safely** — use `{{env.VAR_NAME}}` for all secrets and inputs. Never hardcode API keys or sensitive data in config files.

6. **Execute locally** — run the smallest suite first. Ask before running long, paid, red-team, or production-targeted suites.

7. **Analyze failures** — classify before fixing:

• Prompt failure (model output is wrong)
• Provider variance (non-deterministic model)
• Flaky grader (model grader is inconsistent)
• Bad fixture (test input is unrealistic)
• Config mistake (assertion logic error)

Safety Constraints

• Do NOT log, echo, or store API keys in configuration files or chat output.
• Do NOT run evaluations against production endpoints without user consent.
• Do not execute arbitrary remote code or unvetted plugins during evaluation.

Validation / Done Criteria

• Eval config is valid, minimal, and uses `{{env.VAR}}` references for secrets.
• Deterministic assertions exist where possible; model grader use is documented and justified.
• Run scope, provider, and estimated cost are reported before execution.
• Results are summarized without leaking sensitive input data.

References

• `references/eval-config-patterns.md`