Designing Eval Datasets

Designing Eval Datasets

Overview

Diverse datasets catch more failures. This skill teaches dimension-based dataset design — systematically varying inputs along axes that target failure-prone regions of your workflow. The output is a set of YAML dataset files ready for output workflow test.

When to Use

• Bootstrapping an eval dataset for a new workflow
• Existing datasets only cover happy paths
• Real production traces are sparse (fewer than 50)
• Stress-testing specific failure hypotheses from error analysis

When NOT to Use

• You already have 100+ representative real traces — use stratified sampling from those instead of generating synthetic data
• You haven't done error analysis yet — do that first (output-eval-error-analysis) so your dimensions target real failure modes, not guesses

Step 1: Define Dimensions

Identify 3+ axes of input variation that target anticipated failure modes. Each dimension should vary one aspect of the input that you expect to influence output quality.

Deriving dimensions from error analysis

Map failure categories to input properties that trigger them:

Example dimensions for a blog generation workflow