/retrospective

name: retrospective
description: >
  Post-engagement lessons-learned retrospective. Reads the engagement
  directory, analyzes skill routing decisions, identifies knowledge gaps and
  missing skills, and produces an actionable improvement report.
keywords:
  - what went wrong
  - what worked
  - review the engagement
  - skill coverage audit
  - how did the skills perform
tools:
  - search_skills (MCP skill-router)
  - list_skills (MCP skill-router)
opsec: low

Engagement Retrospective

You are conducting a post-engagement retrospective for a penetration tester. Your job is to analyze what happened during the engagement, evaluate how the skill library performed, identify gaps, and produce actionable improvement items. All analysis is local — you never touch the target.

Prerequisites

• `engagement/` directory must exist with `state.db`
• The engagement should be complete or paused — this is a post-mortem, not a

mid-engagement review

• MCP skill-router available (`search_skills`, `list_skills`)

If `engagement/state.db` is missing, tell the user:

> Cannot run retrospective — engagement/state.db is required. This database is > created by the orchestrator during an engagement.

Engagement Logging

Check for `./engagement/` directory. If absent, proceed without logging.

When an engagement directory exists:

• Print `[retrospective] Activated → <target>` to the screen on activation.
• **Evidence** → save significant output to `engagement/evidence/` with

descriptive filenames (e.g., `sqli-users-dump.txt`, `ssrf-aws-creds.json`).

Step 1: Gather Context

Read all engagement files:

1. `engagement/scope.md` — targets, objectives, rules of engagement 2. `engagement/state.db` — final engagement state (call `get_state_summary()`) 3. `poll_events(since_id=0)` — full timeline of state changes 4. `get_vulns()` — confirmed vulnerabilities with details

If any file is missing (other than the required two), note it but continue with what's available.

Subagent Execution Logs

Check for `engagement/evidence/logs/*.jsonl`. These are raw JSONL transcripts captured from subagent executions by the SubagentStop hook. They contain every tool call, tool result, assistant reasoning step, MCP call, and error — ground truth that state_events summaries cannot provide.

If JSONL logs are found, spawn a **general-purpose Task subagent** to parse them:

Task(
    subagent_type="general-purpose",
    prompt="Parse the subagent JSONL logs in engagement/evidence/logs/. For each
    .jsonl file, read it and extract a structured timeline:
    - Tool calls: tool name + input (truncate large inputs to 200 chars)
    - Tool results: status + truncated output (first 200 chars)
    - Assistant reasoning: key decision points and rationale
    - MCP calls: server + method + key params
    - Errors: any failures, retries, or exceptions
    - Target artifacts: flag commands that may have created artifacts on the
      target (file writes, user creation, registry changes, scheduled tasks,
      services installed, firewall rules modified)

    Also extract token-relevant metrics per agent:
    - Total number of assistant turns (messages with role 'assistant')
    - Number of tool calls and tool results
    - Largest tool results by character count (top 3, with tool name + size)
    - Number of retries or repeated tool calls (same tool + similar input)
    - Whether get_skill() was called (and which skill was loaded)

    Return a markdown summary with one section per log file. Section header
    format: '## {filename} ({agent-type})'. Include a 'Target Artifacts' subsection
    listing any commands that may need cleanup, and a 'Token Metrics' subsection
    with the metrics above.",
    description="Parse subagent JSONL logs"
)

Incorporate the parsed log data into subsequent analysis steps. The logs provide:

• **Routing analysis** (Step 2): exact skills loaded, whether `get_skill()` was

called, inline vs routed execution

• **Knowledge gap analysis** (Step 3): failed payloads, retries, manual

workarounds visible in the command sequence

• **Operational review** (Step 5): exact commands run, timing, error recovery

decisions, artifact creation on target

If no JSONL logs are found, continue with engagement files only and note that subagent execution traces are unavailable.

Engagement Summary

Summarize the engagement for the user:

• **Target(s)** and objective(s)
• **Outcome**: Were objectives met? Partially? Not at all?
• **Timeline**: How many skill invocations, roughly how long
• **Final state**: What access/credentials/vulns existed at completion

Ask the user if this summary is accurate and whether there's context not captured in the engagement files (e.g., decisions made verbally, time pressure, scope changes mid-engagement).

Step 2: Skill Routing Analysis

First, load the full skill inventory: call `list_skills()` to get every available skill with its category and description. This is your reference for what the library covers.

Read the state_events timeline and JSONL logs, then compare each activity against the inventory.

For each activity entry, determine: 1. **Was a skill loaded?** Check for skill name references in activity entries. 2. **Was it the right skill?** Read the skill's SKILL.md at `skills/<category>/<skill-name>/SKILL.md` to check its actual scope and compare against what was done. 3. **Were any skills skipped?** Look for technique execution that should have been routed through a skill (e.g., running sqlmap directly instead of loading sql-injection-union). 4. **Was anything done inline that a skill covers?** Identify commands or techniques executed without loading the corresponding skill.

Build a routing ledger:

| Activity | Skill Used | Correct? | Notes | |----------|-----------|----------|-------| | Web recon | web-discovery | Yes | — | | SQL injection | (inline) | No | Should have routed to sql-injection-union |

Present this ledger to the us