/eval-driven-development

name: eval-driven-development
description: How this repo does eval-driven development (EDD) for Smithers workflows — write the failing suite first, build until green, validate on a holdout, then optimize. Use when adding evals to a workflow, changing a prompt/model/graph that has a suite, setting up a dev/holdout split, or wiring eval gates into CI. For suite-authoring mechanics see skills/eval-writer/SKILL.md; this skill is the loop and the discipline.

Eval-Driven Development

One passing run proves nothing. EDD is the loop that makes a workflow's quality *repeatable*: encode acceptance criteria as a failing eval suite before building, go green, and never tune against the holdout. A change to a prompt, model, schema, or graph edge is "done" when the suite says so, not when a run looks good.

The loop

1. **Red first.** Before building or changing the workflow, write cases from the acceptance criteria and run the suite. It must fail (or lack coverage) — that failure is the spec. If it passes already, your cases are too weak: add the adversarial one that trips a weak run. 2. **Build until green.** Change the prompt, schema, agent, or graph. Re-run the dev suite. Iterate against the report, not against vibes. 3. **Validate on the holdout.** Once dev is green, run the `*-holdout` suite. Holdout cases are never used to tune prompts, pick models, or accept an optimization — they exist to catch overfitting to the dev split. 4. **Optimize (optional).** `smithers optimize` runs the dev suite twice (baseline + GEPA-patched) and writes an artifact. Accept it only if the holdout suite, run with `--optimization <artifact.json>`, does not regress. 5. **Gate in CI.** The `eval` command exits `0` when all cases pass, `1` on any failure, `4` for an invalid case file. Wire the non-zero exit into CI; a suite nobody runs is documentation.

Repo conventions

Suites live in `.smithers/evals/` as JSONL, one case per line:

• `<suite>.jsonl` — the **dev** split; tune against this.
• `<suite>-holdout.jsonl` — the **holdout** split; validate only.
• Every case carries `metadata`: `split` (`dev`/`holdout`), `category` (the

behavior class, e.g. `durable-routing`), and `criterion` (the plain-English acceptance criterion the case encodes).

• Reports land at `.smithers/evals/<suite>.json` by default; sweep runs go to

`.smithers/evals/reports/<label>/`.

• `.smithers/evals/run-sweep.sh <label> [concurrency]` runs every seeded

suite (dev + holdout) and prints a per-suite pass summary — the pattern for a full regression check before shipping a harness change.

Case shape:

{"id":"etl-silent-row-drop","input":{"prompt":"..."},"expected":{"status":"finished","outputContains":{"recommend":[{"recommendedWorkflow":"debug"}]}},"metadata":{"split":"dev","category":"durable-routing","criterion":"A root-cause defect lands on debug."}}

• `expected.status`: `finished`, `failed`, `waiting-approval`, etc.
• `expected.outputContains`: recursive partial match; arrays match by

containment. **The usual choice** — key it to the output schema's load-bearing typed fields, never to prose.

• `expected.output`: exact match (brittle; rare).
• `expected.errorContains`: for adversarial cases that *should* fail.
• `judge`: `{"instructions": "...", "threshold": 0.7}` for semantic checks

(tone, completeness) where JSON match can't work. Deterministic assertions AND the judge must both pass. Select the judge with `--judge-provider` / `--judge-model`.

Adding evals to a workflow

1. List the acceptance criteria. Turn each into ≥1 case: a happy path, the quality-gate criterion itself, and an adversarial/edge case. 2. Write `.smithers/evals/<suite>.jsonl` with `metadata.split: "dev"`, and a `<suite>-holdout.jsonl` of fresh cases drawn from the same criteria but not shared with dev. 3. Dry-run the shape before spending:

   bun apps/cli/src/index.js eval .smithers/workflows/<wf>.tsx \
     --cases .smithers/evals/<suite>.jsonl --suite <suite> --dry-run

4. Run it (in-repo use `bun apps/cli/src/index.js eval`; consumers use `bunx smthrs eval`):

   bun apps/cli/src/index.js eval .smithers/workflows/<wf>.tsx \
     --cases .smithers/evals/<suite>.jsonl --suite <suite> \
     --concurrency 4 --force

5. Read the report at `.smithers/evals/<suite>.json`; fix the workflow, not the test — unless the case encodes the criterion wrong, or the failure is environmental (see the classify rule below). 6. For graded, non-binary quality, attach scorers (`faithfulness`, `relevancy`, `schemaAdherence`, `llmJudge`) to the load-bearing `<Task>` and read them with `smithers scores <run-id>`. Assertions are the hard gate; scorers are the trend. 7. Add the suite to the sweep script / CI so it runs on every relevant change.

Discipline rules

• **Classify red before acting.** Every failed case is one of three things:

a product bug (fix the workflow), a wrong case (fix the case, with spec- change scrutiny), or a harness/environment fault (fix the harness). The report marks known environment faults INCONCLUSIVE and `smithers eval` exits `5` (not `1`) when they are the only reds: on that signal repair the harness and never touch the workflow. A red that could not have observed the workflow (connection refused, TLS failure, network denied, missing binary, OOM, rate limit) is not evidence against it.

• **Green ratchet.** If a case that passed in round N fails in round N+1 and

the only intervening change was to the harness or environment, that is a harness regression: revert the harness change. Never widen a suite's acceptance criteria while it is red; get back to the last green slice first.

• **Circuit breaker.** After 3 consecutive rounds with zero net new green

cases, stop iterating. Change strategy (gather evidence, widen scope) or escalate to a human via `smithers ask-human` with what you know. More