/output-eval-dataset-design

name: output-eval-dataset-design
description: Design diverse eval datasets using dimension-based variation. Use when bootstrapping eval datasets, when real traces are sparse, or when existing datasets miss edge cases.
allowed-tools: [Bash, Read, Write, Edit]

Designing Eval Datasets

Overview

Diverse datasets catch more failures. This skill teaches dimension-based dataset design — systematically varying inputs along axes that target failure-prone regions of your workflow. The output is a set of YAML dataset files ready for `output workflow test`.

When to Use

• Bootstrapping an eval dataset for a new workflow
• Existing datasets only cover happy paths
• Real production traces are sparse (fewer than 50)
• Stress-testing specific failure hypotheses from error analysis

When NOT to Use

• You already have 100+ representative real traces — use stratified sampling from those instead of generating synthetic data
• You haven't done error analysis yet — do that first (`output-eval-error-analysis`) so your dimensions target real failure modes, not guesses

Step 1: Define Dimensions

Identify 3+ axes of input variation that target **anticipated failure modes**. Each dimension should vary one aspect of the input that you expect to influence output quality.

Deriving dimensions from error analysis

Map failure categories to input properties that trigger them:

| Failure Category | Triggering Input Property | Dimension | |-----------------|---------------------------|-----------| | Off-topic drift | Ambiguous or broad topics | Topic specificity: specific / broad / ambiguous | | Tone mismatch | Conflicting tone signals | Tone difficulty: simple / nuanced / contradictory | | Too short | Short or vague input | Input detail: minimal / moderate / comprehensive | | Missing sections | Many explicit requirements | Requirement count: 0 / 1-2 / 5+ | | Hallucinated URLs | Technical topics with real entities | Entity density: none / few / many |

Example dimensions for a blog generation workflow

| Dimension | Values | Why | |-----------|--------|-----| | Topic complexity | simple, technical, ambiguous | Technical and ambiguous topics trigger more hallucination and drift | | Tone request | none, formal, casual, contradictory | Explicit tone requests reveal tone-matching failures | | Length constraint | none, short (100w), long (1000w) | Extreme length constraints trigger truncation and padding | | Required sections | none, 1 section, 3+ sections | Multiple required sections stress structural compliance |

Aim for 3-5 dimensions. More than 5 creates an unmanageable combinatorial space.

Step 2: Draft Tuples

Create ~20 combinations of dimension values. Cover the extremes and the combinations most likely to cause failures.

Tuple selection strategy

1. **Cover every dimension value at least twice** — ensures no blind spots 2. **Pair difficult values together** — "ambiguous topic + contradictory tone + 3+ required sections" is where failures cluster 3. **Include a few easy combinations** — confirms the workflow works under normal conditions 4. **Avoid near-duplicates** — each tuple should test a distinct scenario

Example tuples

| # | Topic Complexity | Tone | Length | Required Sections | |---|-----------------|------|--------|-------------------| | 1 | simple | none | none | none | | 2 | simple | formal | short | 1 section | | 3 | technical | formal | long | 3+ sections | | 4 | technical | casual | short | none | | 5 | ambiguous | formal | long | 1 section | | 6 | ambiguous | contradictory | none | 3+ sections | | 7 | simple | contradictory | long | none | | 8 | technical | none | none | 3+ sections | | 9 | ambiguous | casual | short | 1 section | | 10 | simple | casual | long | 3+ sections | | 11 | technical | formal | short | 1 section | | 12 | ambiguous | none | long | none | | 13 | simple | formal | none | 3+ sections | | 14 | technical | contradictory | long | 1 section | | 15 | ambiguous | formal | short | 3+ sections | | 16 | simple | none | short | 1 section | | 17 | technical | casual | long | 3+ sections | | 18 | ambiguous | contradictory | short | none | | 19 | technical | formal | none | none | | 20 | ambiguous | casual | long | 3+ sections |

Review each tuple and ask: "Is this a realistic scenario a user might create?" Discard any that aren't.

Step 3: Convert Tuples to Workflow Inputs

Transform each tuple into a JSON object matching the workflow's `inputSchema` from `types.ts`.

Read the schema first

# Find the input schema
cat src/workflows/<workflow_name>/types.ts

Manual conversion (small datasets)

For each tuple, write the corresponding JSON input:

**Tuple 3:** technical + formal + long + 3+ sections

{
  "topic": "Quantum error correction techniques in superconducting qubit architectures",
  "tone": "formal",
  "min_length": 1000,
  "required_sections": ["Introduction", "Technical Background", "Current Approaches", "Challenges", "Conclusion"]
}

**Tuple 6:** ambiguous + contradictory + none + 3+ sections

{
  "topic": "Things that matter",
  "tone": "Write in a formal academic style but keep it super casual and fun",
  "required_sections": ["Overview", "Deep Dive", "Takeaways"]
}

Use realistic, natural-sounding inputs. Avoid test-looking data like "Test topic 1" or "Lorem ipsum."

LLM-assisted conversion (larger datasets)

For 20+ tuples, use an LLM to batch-convert tuples into realistic inputs. Create a prompt that takes the tuple values and the `inputSchema`, then generates a natural JSON input. Review every generated input for realism before using it.

Step 4: Generate Dataset Files

Run each input through the workflow to capture real output:

# Generate datasets one at a time
npx output workflow dataset generate blog_generator \
  --input '{"topic": "Quantum error correction", "tone": "formal", "min_length": 1000}' \
  --name technical_formal_long

npx output workflow dataset gener