/output-dev-eval-testing

name: output-dev-eval-testing
description: Create offline evaluation tests for Output SDK workflows using @outputai/evals. Use when implementing test evaluators with verify(), creating dataset YAML files, building eval workflows, or running workflow tests via CLI.
allowed-tools: [Bash, Read, Write, Edit]

Offline Evaluation Testing

Overview

The `@outputai/evals` package provides an offline evaluation framework for testing workflow quality using datasets and evaluators. This is **complementary** to the runtime `evaluator()` from `@outputai/core`:

| Aspect | Runtime Evaluators (`@outputai/core`) | Offline Eval Tests (`@outputai/evals`) | |--------|----------------------------------------|------------------------------------------| | **When** | During workflow execution | After execution, at test time | | **Where** | `evaluators.ts` in workflow folder | `tests/evals/` in workflow folder | | **Purpose** | Live quality scoring with confidence | Dataset-driven pass/fail verification | | **Triggered by** | Workflow orchestration | `output workflow test` CLI command | | **Returns** | `EvaluationBooleanResult`, etc. | `Verdict` helpers (pass/partial/fail) |

Use offline eval testing when you want to validate workflow behavior against known datasets, build regression test suites, or assess subjective quality with LLM judges.

When to Use This Skill

• Creating files in `tests/evals/` or `tests/datasets/`
• Writing evaluators that use `verify()` from `@outputai/evals`
• Creating YAML dataset files for test cases
• Building eval workflows with `evalWorkflow()`
• Running `output workflow test` commands
• Setting up ground truth data for evaluators

Directory Structure

Add a `tests/` directory inside the workflow folder:

src/workflows/{workflow_name}/
├── workflow.ts
├── steps.ts
├── evaluators.ts          # Runtime evaluators (optional)
├── types.ts
└── tests/
    ├── datasets/
    │   ├── happy_path.yml
    │   └── edge_case.yml
    └── evals/
        ├── evaluators.ts  # Offline eval test evaluators
        ├── workflow.ts     # Eval workflow definition
        └── judge_topic@v1.prompt  # LLM judge prompts (optional)

Creating Evaluators with `verify()`

Import `verify` and `Verdict` from `@outputai/evals` (not `@outputai/core`):

// tests/evals/evaluators.ts
import { verify, Verdict } from '@outputai/evals';
import { z } from '@outputai/core';

`verify()` Signature

verify(options, checkFn)

**Options:**

• `name` — unique evaluator identifier (snake_case)
• `input` — Zod schema for the workflow input (optional, defaults to `z.any()`)
• `output` — Zod schema for the workflow output (optional, defaults to `z.any()`)

**Check function receives:**

{
  input,    // typed workflow input
  output,   // typed workflow output
  context: {
    ground_truth: Record<string, unknown>  // from dataset YAML
  }
}

**Returns:** any `Verdict` helper result.

Basic Example

import { verify, Verdict } from '@outputai/evals';
import { z } from '@outputai/core';

export const evaluateSum = verify(
  {
    name: 'evaluate_sum',
    input: z.object({ values: z.array(z.number()) }),
    output: z.object({ result: z.number() })
  },
  ({ input, output }) =>
    Verdict.equals(output.result, input.values.reduce((a, b) => a + b, 0))
);

Using Ground Truth

Ground truth values come from the dataset YAML and are available via `context.ground_truth`:

export const lengthCheck = verify(
  { name: 'length_check', input: blogInput, output: blogOutput },
  ({ output, context }) =>
    Verdict.gte(output.blog_post.length, Number(context.ground_truth.min_length ?? 100))
);

Verdict Helpers

All deterministic helpers return results with confidence `1.0`.

Equality & Comparison

| Method | Description | |--------|-------------| | `Verdict.equals(actual, expected)` | Strict equality (`===`) | | `Verdict.closeTo(actual, expected, tolerance)` | Within numeric tolerance | | `Verdict.gt(actual, threshold)` | Greater than | | `Verdict.gte(actual, threshold)` | Greater than or equal | | `Verdict.lt(actual, threshold)` | Less than | | `Verdict.lte(actual, threshold)` | Less than or equal | | `Verdict.inRange(actual, min, max)` | Within inclusive range |

String & Array

| Method | Description | |--------|-------------| | `Verdict.contains(haystack, needle)` | String includes substring | | `Verdict.matches(value, pattern)` | Regex match | | `Verdict.includesAll(actual, expected)` | Array contains all expected values | | `Verdict.includesAny(actual, expected)` | Array contains at least one expected value |

Boolean

| Method | Description | |--------|-------------| | `Verdict.isTrue(value)` | Value is `true` | | `Verdict.isFalse(value)` | Value is `false` |

Manual Verdicts

| Method | Description | |--------|-------------| | `Verdict.pass(reasoning?)` | Explicit pass | | `Verdict.partial(confidence, reasoning?, feedback?)` | Partial pass with confidence | | `Verdict.fail(reasoning, feedback?)` | Explicit fail |

LLM Judge Evaluators

Before writing a judge prompt, identify the specific failure mode via error analysis (`output-eval-error-analysis`). Design the judge following `output-eval-judge-prompt`. After writing it, validate against human labels using `output-eval-validate-judge`.

For subjective quality assessments, use judge functions with `.prompt` files:

import { verify, judgeVerdict, judgeScore, judgeLabel } from '@outputai/evals';

// Returns pass/partial/fail verdict from an LLM
export const evaluateTopic = verify(
  { name: 'evaluate_topic', input: blogInput, output: blogOutput },
  async ({ input, output, context }) =>
    judgeVerdict({
      prompt: 'judge_topic@v1',
      variables: {
        blog_title: output.title,
        blog_post: output.blog_post,
        required_topic: String(context.ground_truth.required_topic ?? input.topic)
      }
    })
);

/