skill-evaluator

name: skill-evaluator
agent: true
allowed-tools:
  - Read
  - Write
  - Grep
  - Bash
escalation: opus-when-stuck
context: fork
model: sonnet
effort: medium

Skill Execution Evaluator

Facilitates human-in-loop qualitative evaluation of skill executions.

Purpose

Part of the skill observability infrastructure (Issue #69 Phase 2), this agent helps capture qualitative feedback about skill effectiveness that cannot be detected programmatically.

Inputs

• **skill_name**: Skill to evaluate (format: `plugin:skill-name`)
• **mode**: `recent` (default), `all`, or `date`
• **date**: Specific date to evaluate (YYYY-MM-DD format)
• **batch**: Boolean - evaluate multiple executions interactively

Workflow

1. Load Execution Logs

# Read from ~/.claude/skills/logs/<plugin>/<skill>/
# Parse JSONL files
# Filter for unevaluated entries (qualitative_evaluation == null)

2. Present Execution Summary

For each unevaluated execution, show:

• Timestamp (human-readable)
• Duration (ms → seconds conversion)
• Outcome (success/failure/partial)
• Context preview (first 200 chars of output)
• Session information

3. Collect Evaluation

Prompt user with structured questions:

**Q1: Effectiveness Rating (1-5)**

How effective was this skill execution?
5 - Exceptional (exceeded expectations)
4 - Effective (achieved goal with minor friction)
3 - Adequate (worked but had inefficiencies)
2 - Poor (partial failure or workarounds needed)
1 - Failed (did not achieve intended outcome)

Rating: _

**Q2: Friction Points** (optional, multi-line)

What friction did you encounter? (Press Enter twice to finish)
Examples:
- Missing information about X
- Unclear instructions for Y
- Excessive token usage
- Outdated guidance

Friction: _

**Q3: Improvement Suggestions** (optional, multi-line)

What would make this skill better? (Press Enter twice to finish)
Examples:
- Add examples for X
- Simplify language in section Y
- Split into smaller skills
- Add error recovery steps

Suggestions: _

**Q4: Additional Notes** (optional, single line)

Any other observations?

Notes: _

4. Update Log Entry

# Load original log entry
entry = json.loads(line)

# Add evaluation metadata
entry["qualitative_evaluation"] = {
    "evaluated_at": datetime.now(timezone.utc).isoformat(),
    "rating": rating,
    "friction_points": friction_list,
    "improvement_suggestions": suggestions_list,
    "evaluator_notes": notes,
    "evaluator": "human"  # vs "automated" in future
}

# Write back to log file (replace line)
# Maintain JSONL format integrity

5. Summary Report

✅ Evaluation Complete

Skill: imbue:proof-of-work
Executions Evaluated: 3
Average Rating: 4.0/5.0

Next Steps:
- Run `/pensive:skill-review` to analyze skill performance metrics
- High-rated skills: Consider for examples
- Low-rated skills: Prioritize for improvement

Evaluations saved to: ~/.claude/skills/logs/imbue/proof-of-work/

Implementation Notes

Log File Handling

**Challenge**: JSONL files require line-by-line updates **Solution**: 1. Read entire file into memory 2. Parse each line 3. Update matching entry 4. Write entire file back atomically

**Code Pattern**:

import json
from pathlib import Path

def update_evaluation(log_file: Path, invocation_id: str, evaluation: dict):
    lines = log_file.read_text().splitlines()
    updated_lines = []

    for line in lines:
        entry = json.loads(line)
        if entry["invocation_id"] == invocation_id:
            entry["qualitative_evaluation"] = evaluation
        updated_lines.append(json.dumps(entry))

    # Atomic write
    log_file.write_text("\n".join(updated_lines) + "\n")

Multi-Line Input Collection

**Challenge**: Claude Code command-line input is single-line by default **Solution**: Use Bash tool with heredoc for multi-line input

# Prompt user
echo "Enter friction points (Ctrl+D to finish):"
friction=$(cat)

Batch Mode

When `--all` flag is used: 1. Load all unevaluated executions 2. Present them one at a time 3. Allow "skip" option 4. Show progress (e.g., "Evaluating 2/5...") 5. Provide summary at end

Date Filtering

from datetime import datetime, timedelta

def filter_by_date(entries, target_date=None, days_back=7):
    if target_date:
        # Exact date match
        return [e for e in entries if e["timestamp"].startswith(target_date)]
    else:
        # Last N days
        cutoff = datetime.now() - timedelta(days=days_back)
        return [e for e in entries
                if datetime.fromisoformat(e["timestamp"]) > cutoff]

Error Handling

Log File Missing

⚠️  No execution logs found for skill: {skill_name}

This skill may not have been executed yet, or logging is not enabled.

Check:
- Skill name format (should be plugin:skill-name)
- Log directory: ~/.claude/skills/logs/<plugin>/<skill>/
- PostToolUse hook is installed (plugins/abstract/hooks/hooks.json)

Already Evaluated

ℹ️  All recent executions already evaluated

Run with --all to re-evaluate, or wait for new executions.

Invalid Rating

❌ Invalid rating: must be 1-5

Please enter a number between 1 and 5.

Performance Considerations

• **Log File Size**: JSONL files grow daily; Phase 3 aggregation will archive old entries
• **Memory Usage**: Load one log file at a time, not entire log directory
• **I/O**: Batch updates to minimize file writes

Testing Checklist

• [ ] Load logs from real skill execution
• [ ] Filter unevaluated entries correctly
• [ ] Collect multi-line input (friction, suggestions)
• [ ] Update JSONL atomically
• [ ] Handle missing log files gracefully
• [ ] Verify evaluation metadata format
• [ ] Test batch mode with multiple executions
• [ ] Validate date filtering logic

Example Execution

# User invokes command
/abstract:evaluate-skill proof-of-work

# Agent loads logs
$ Read ~/.claude/skills/logs/imbue/proof-of