/ct-grade

name: ct-grade
description: >-
  CLEO session grading and A/B behavioral analysis with token tracking. Evaluates agent
  session quality via a 5-dimension rubric (S1 session discipline, S2 discovery efficiency,
  S3 task hygiene, S4 error protocol, S5 progressive disclosure). Supports three modes:
  (1) scenario — run playbook scenarios S1-S5 via CLI; (2) ab — blind A/B
  comparison of different CLI configurations for same domain operations with token cost
  measurement; (3) blind — spawn two agents with different configurations, blind-comparator
  picks winner, analyzer produces recommendation. Use when grading agent sessions, running
  grade playbook scenarios, comparing behavioral differences, measuring token
  usage across configurations, or performing multi-run blind A/B evaluation with statistical
  analysis and comparative report. Triggers on: grade session, evaluate agent behavior,
  A/B test CLEO configurations, run grade scenario, token usage analysis, behavioral rubric,
  protocol compliance scoring.
version: 2.1.1
argument-hint: "[mode=scenario|ab|blind] [scenario=s1-s5|all] [runs=N] [session-id=<id>]"
allowed-tools: ["Bash(python *)", "Bash(cleo-dev *)", "Bash(cleo *)", "Bash(kill *)", "Bash(lsof *)", "Agent", "Read", "Write", "Glob"]
protocol: null
dependencies: []
sharedResources: []
compatibility:
  - claude-code
  - cursor
  - windsurf
  - gemini-cli
license: MIT
metadata:
  version: 2.1.1
  tier: internal
  install: internal
  lastReviewed: 2026-09-28
  stability: stable

Session Grading Guide

Session grading evaluates agent behavioral patterns against the CLEO protocol. It reads the audit log for a completed session and applies a 5-dimension rubric to produce a score (0-100), letter grade (A-F), and diagnostic flags.

When to Use Grade Mode

Use grading when you need to:

• Evaluate how well an agent followed CLEO protocol during a session
• Identify behavioral anti-patterns (skipped discovery, missing session.end, etc.)
• Track improvement over time across multiple sessions
• Validate that orchestrated subagents followed protocol

Grading requires audit data. Sessions must be started with the `--grade` flag to enable audit log capture.

Starting a Grade Session

CLI

# Start a session with grading enabled
ct session start --scope epic:T001 --name "Feature work" --grade

# The --grade flag enables detailed audit logging
# All CLI operations are recorded for later analysis

Running Scenarios

The grading rubric evaluates 5 behavioral scenarios that map to protocol compliance:

1. Fresh Discovery

Tests whether the agent checks existing sessions and tasks before starting work. Evaluates `session.list` and `tasks.find` calls at session start.

2. Task Hygiene

Tests whether task creation follows protocol: descriptions provided, parent existence verified before subtask creation, no duplicate tasks.

3. Error Recovery

Tests whether the agent handles errors correctly: follows up `E_NOT_FOUND` with recovery lookups (`tasks.find`), avoids duplicate creates after failures.

4. Full Lifecycle

Tests session discipline end-to-end: session listed before task ops, session properly ended, CLI usage patterns.

5. Multi-Domain Analysis

Tests progressive disclosure: use of `admin.help` or skill lookups, use of progressive disclosure for programmatic access.

Evaluating Results

CLI

# Grade a specific session
ct grade <sessionId>

# List all past grade results
ct grade --list

Understanding the 5 Dimensions

Each dimension scores 0-20 points, totaling 0-100.

S1: Session Discipline (20 pts)

| Points | Criteria | |--------|----------| | 10 | `session.list` called before first task operation | | 10 | `session.end` called when work is complete |

**What it measures**: Does the agent check existing sessions before starting, and properly close sessions when done?

S2: Discovery Efficiency (20 pts)

| Points | Criteria | |--------|----------| | 0-15 | `find:list` ratio >= 80% earns full 15; scales linearly below | | 5 | `tasks.show` used for detail retrieval |

**What it measures**: Does the agent prefer `tasks.find` (low context cost) over `tasks.list` (high context cost) for discovery?

S3: Task Hygiene (20 pts)

Starts at 20 and deducts for violations:

| Deduction | Violation | |-----------|-----------| | -5 each | `tasks.add` without a description | | -3 | Subtasks created without `tasks.find {exact:true}` parent check |

**What it measures**: Does the agent create well-formed tasks with descriptions and verify parents before creating subtasks?

S4: Error Protocol (20 pts)

Starts at 20 and deducts for violations:

| Deduction | Violation | |-----------|-----------| | -5 each | `E_NOT_FOUND` error not followed by recovery lookup within 5 ops | | -5 | Duplicate task creates detected (same title in session) |

**What it measures**: Does the agent recover gracefully from errors and avoid creating duplicate tasks?

S5: Progressive Disclosure Use (20 pts)

| Points | Criteria | |--------|----------| | 10 | `admin.help` or skill lookup calls made | | 10 | Progressive disclosure used for programmatic access |

**What it measures**: Does the agent use progressive disclosure (help/skills) for efficient protocol access?

Interpreting Scores

Letter Grades

| Grade | Score Range | Meaning | |-------|-----------|---------| | **A** | 90-100 | Excellent protocol adherence. Agent follows all best practices. | | **B** | 75-89 | Good. Minor gaps in one or two dimensions. | | **C** | 60-74 | Acceptable. Several protocol violations need attention. | | **D** | 45-59 | Below expectations. Significant anti-patterns present. | | **F** | 0-44 | Failing. Major protocol violations across multiple dimensions. |

Reading the Output

The grade result includes:

• **score/maxScore**: Raw numeric score (e.g., `85/100`)
• **percent**: Percentage score
• **grade**: Letter grade (A-F)
• **dimensions**: Per-dimension