/maplibre-running-evals

name: maplibre-running-evals
description: The step-by-step procedure for running this repo's Promptfoo evals with a generator and a judge over provider APIs. Use when starting a new skill and needing a baseline before writing content, when a run stalls or returns no data, when a provider rate-limits, or when deciding whether an edit needs a retest.
status: process

Running Promptfoo Evals

Every eval here runs two roles over provider APIs: a **generator**, the model under test, and a **judge**, the model that grades each `llm-rubric` assertion. [evals/README.md](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md) covers what to test and how to write prompts and rubrics; this is the procedure for running them. Work the phases in order — do not start at phase 5 because content already exists.

1. Set up and budget the run

Budget before anything else. These providers are used on free tiers, and a run you cannot finish teaches nothing.

• **Confirm a generator key and a judge key are set — presence only, never the value:** `echo "generator: ${GROQ_API_KEY:+set}"` and `echo "judge: ${GOOGLE_API_KEY:+set}"`. If either is missing, stop and report; do not extrapolate a result from a run you did not make.
• **Both roles are pinned:** the generator in [`evals/prompts/lib/providers.yaml`](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/prompts/lib/providers.yaml), the judge in `package.json`'s `eval:graded` script. Use the pinned pair by default. The one hard requirement is that they are different models — never grade a generator with itself — and any result you report names both.
• **Send every recorded run through `npm run eval:graded`,** the only place `--grader`, `--delay`, and `-j` are set. Do not hand-type them, and do not copy a value out of an older results doc: that value documents that run, not this one. Do not pass `--providers` either: it replaces the pinned generator with a bare one, so the run is no longer pinned and its results are mislabeled — [evals/README.md, "Writing eval prompts"](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md#writing-eval-prompts) has the detail. `--providers echo` is the probe case: it returns the rendered prompt in place of a generator answer, so you see exactly what the generator would be sent — but every `llm-rubric` still calls the judge, so even a probe spends judge calls.
• **Learn the provider's limits before launching:** read its rate-limit documentation for the pinned model's tier, then make one minimal request to the endpoint with response headers shown. On an OpenAI-compatible endpoint the `x-ratelimit-*` headers carry the per-minute token window (limit, remaining, reset) and the per-day request count; a daily token cap has no header of its own, so it surfaces only when a `429` arrives — `retry-after` gives the wait, and the body names which limit tripped. Header names and what they count are provider-specific and change, so confirm them against the provider's own page.

# Example against the current generator's OpenAI-compatible endpoint; the URL and model id come from providers.yaml.
curl -s -X POST https://api.groq.com/openai/v1/chat/completions \
  -H "Authorization: Bearer $GROQ_API_KEY" -H "Content-Type: application/json" \
  --data '{"model":"openai/gpt-oss-120b","messages":[{"role":"user","content":"hi"}],"max_tokens":1}' \
  -D - | grep -iE 'ratelimit|retry-after|"error"'

• **Do the arithmetic before launching.** Tokens ≈ bytes / 4, so `wc -c skills/<skill>/SKILL.md` gives the injection cost per test; a with-skill run costs roughly tests × (that + the generator's configured `max_tokens`), plus one judge call per rubric, against whatever the day has left. A baseline run drops the injection. If the total does not fit the remaining budget, do not launch it whole.
• **Run sequentially, one config per command.** `-j 1` is pinned, and never run two graded evals at once — across terminals or across teammates. Limits are per account, not per process, so a `429` on either role kills both runs; chain them instead, leaving a full per-minute window between: `<run one> && sleep 60 && <run two>`. Each config already wires `lib/skill-prompt.mjs` and `lib/providers.yaml`, so there is nothing to assemble by hand; if you do write a Promptfoo command yourself, it still names exactly one `--config` — given several, Promptfoo merges their `defaultTest` blocks and the last `skillFile` wins.
• **`--delay` paces requests; it does not create budget.** A rate-limit failure that survives correct pacing with no concurrency is exhausted capacity — stop, take the reset time from the direct call above, and do not loop.
• **When the remaining window or day is smaller than the run, run fewer tests per invocation — never a larger delay.** `--filter-pattern '<regex against the test description>'` runs only the matching tests (quote it; `npm run` passes it through a shell), and it is also how you probe one test of a run type before committing the whole config. `--filter-failing <the run's .json output>` re-runs only what failed or errored — phase 8 writes that file.

2. Write the tests before the content

• Copy [`evals/prompts/TEMPLATE.yaml`](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/prompts/TEMPLATE.yaml) to `evals/prompts/<skill>.yaml` and write four tests — explicit, implicit, anti-pattern, negative — before the skill has any content. [evals/README.md, "Writing eval prompts"](https://github.com/maplibre/maplibre-agent-skills/blob/main/evals/README.md#writing-eval-prompts) is the how; this skill is the running procedure.

3. Run the baseline, with the skill withheld

`npm run eval:graded -- --config evals/prompts/<skill>.yaml --var injectSkill=false`

• **An explicit, implicit, or anti-pattern test that passes at baseline has no discriminating power.** Redesign it. Never keep it as evidence, and never delete a test to make a run green. A negative