/eval

name: eval
description: Plan and run conversational AI agent evaluations with test generation and analysis.
kind: sop

EvalKit

Overview

EvalKit is a conversational evaluation framework for AI agents that guides you through creating robust evaluations using the Strands Evals SDK. Through natural conversation, you can plan evaluations, generate test data, execute evaluations, and analyze results.

Parameters

• **agent_path** (required): Path to the agent folder to evaluate (e.g., `./chatbot-agent`, `/path/to/my-agent`)
• **evaluation_focus** (optional): Specific aspects to evaluate (e.g., "response quality", "tool calling accuracy")
• **test_case_count** (optional, default: 3): Number of test cases to generate
• **output_format** (optional, default: "jsonl"): Format for test data output

**Constraints for parameter acquisition:**

• You MUST ask for the agent_path if not provided because evaluation cannot proceed without knowing which agent to evaluate
• You MUST support multiple input methods including:
• Direct input: Path provided directly in the conversation
• Relative paths: Paths relative to current working directory
• Absolute paths: Full system paths to agent location
• You SHOULD infer evaluation_focus from user's natural language description when not explicitly provided
• You MAY use intelligent defaults for optional parameters based on agent analysis

Steps

1. Setup and Initialization

When a user requests evaluation (any phase), first validate the environment:

**Folder Structure:** All evaluation artifacts MUST be created in the `eval/` folder at the same level as the target agent folder:

<agent-evaluation-project>/       # Example name - can be any name for user's evaluation project
├── <target-agent-folder>/        # Example name - this is the agent you are evaluating
│   └── [agent source code]     # Existing agent code
└── eval/                       # All evaluation files go here (sibling to target-agent-folder)
    ├── eval-plan.md
    ├── test-cases.jsonl
    ├── results/
    ├── run_evaluation.py
    ├── eval-report.md
    └── README.md

**Note:**

• The `eval/` folder is a sibling directory to user's agent folder, not nested inside it
• `agent-evaluation-project` and `target-agent-folder` are placeholder names - user may use any names that fit their project

**Constraints:**

• You MUST check if the agent folder exists
• You MUST verify Python 3.11+ is installed
• You MUST navigate to the evaluation project directory (containing both agent folder and eval/) before any operation
• You MUST create the eval/ folder as a sibling to the agent folder
• You MUST NOT create evaluation folders inside the agent folder
• You MUST create the eval/ directory at the same level as the agent folder if it doesn't exist
• You MUST ensure all evaluation artifacts are within the eval/ folder
• You MUST check for any existing evaluation artifacts in the eval/ folder
• You SHOULD validate that required dependencies are available
• You MUST use relative paths from the evaluation project directory (e.g., "./eval/eval-plan.md") for all file operations

2. Planning Phase

**When to Trigger**: User requests evaluation planning or mentions creating/designing an evaluation

**User Intent Recognition**:

• Keywords: "plan", "design", "create evaluation", "evaluate my agent"
• Context: User provides agent path or describes agent to evaluate
• Goal: Understand what the user wants to evaluate and why

**Execution Flow**:

1. **Parse user request**: Extract agent path, evaluation focus, and specific requirements from natural language

2. **Navigate to evaluation project directory**:

   cd <your-evaluation-project>  # Navigate to the directory containing both agent folder and eval/

3. **Create evaluation directory structure**:

   mkdir -p eval

4. Follow this execution flow:

1. Parse user evaluation requirements from user input 2. Analyze agent and user requirements:

• Parse specific evaluation requirements, scenarios, and constraints from user input
• Scan codebase for agent architecture and capabilities
• Check for existing test cases and evaluation files

3. Design evaluation strategy:

• Define evaluation areas and metrics (user-request-driven with agent-aware defaults)
• Identify test data requirements
• Define file structure
• Select technology stack

5. Write the complete evaluation plan to `eval/eval-plan.md` using the template structure (see Appendix A: Evaluation Plan Template), replacing placeholders with concrete details derived from the analysis while preserving section order and headings.

6. Report completion with evaluation plan file path, and suggest next step: "Would you like me to generate test cases based on this plan?"

Evaluation Planning Guidelines

Design Principles

**High-Level Design (What & Why)**:

• Focus on **WHAT** to evaluate and **WHY** it matters for the agent
• Define evaluation areas and metrics that are measurable and verifiable
• Ensure requirements can be tested through actual agent execution

**Low-Level Implementation (How)**:

• Select appropriate technology stack and architecture
• Design practical file structure and execution approach
• Choose integration patterns and configuration methods

Metrics Guidelines

Evaluation metrics must be:

1. **Measurable**: Define what will be measured 2. **Verifiable**: Can be measured through actual agent execution 3. **Implementation-ready**: Clear enough to guide technical implementation

Architecture Principles

**Key Principles**:

• **Simple Structure**: Use the flat `eval/` directory structure
• **Real Agent Focus**: Always use actual agent execution, never simulation or mock
• **Focused Implementation**: Avoid over-engineering, focus on core evaluation logic
• **Minimal Viable Implementation**: Start with essential components, add complexity incrementally
• **Framew