/stata-skill-contributor

name: stata-skill-contributor
description: >
  Guide for contributing to the stata-skill project. Use when the user wants to
  run the eval pipeline, analyze test results, improve reference docs, add new
  package documentation, or work on roadmap items. Covers the testing
  infrastructure, multi-agent analysis workflow, cost estimates, and links to
  prior eval results and improvement history.

Stata Skill Contributor Guide

This skill helps you contribute to the [stata-skill](https://github.com/dylantmoore/stata-skill) project — the Stata reference skill for Claude Code. It covers how to run the evaluation pipeline, analyze results, and make data-driven improvements to the skill documentation.

Cost Warning

Running the eval pipeline uses Claude Agent SDK `query()` calls, which consume API credits.

| What | Approximate cost | |------|-----------------| | Single task, 1 run | $0.50 - $1.50 | | Single task, 5 runs | $2.50 - $7.50 | | All 24 tasks, 1 run each | $12 - $36 | | All 24 tasks, 5 runs each | $60 - $120 | | Full A/B comparison (2 arms × 24 tasks × 5 runs) | $120 - $240 | | Multi-agent post-test analysis (5 Sonnet subagents) | $3 - $8 |

Start small: run 1-3 tasks relevant to your change before running the full suite. Use `--runs 1` to validate the pipeline works before scaling up.

Quick Reference

| File | What it is | |------|-----------| | `tests/eval.py` | Evaluation harness — sends tasks to Claude, judges responses | | `tests/rubric.md` | 7-category scoring rubric (max 55 points) | | `tests/tasks/task_*.md` | 24 test task definitions | | `tests/runner-only-plugin/` | Minimal baseline plugin (execution instructions only) | | `tests/results/` | All run directories and aggregate JSON results | | `tests/EVAL_RESULTS_2026-03-22.md` | Prior eval results with full analysis and run index | | `ROADMAP.md` | Open problems and improvement directions | | `CONTRIBUTING.md` | General contribution guide (adding packages, filing issues) |

Running the Eval Pipeline

Basic commands

# Single task, single run — fast sanity check (~$1)
python tests/eval.py tests/tasks/task_01_data_cleaning.md

# Single task, 5 runs — measure variance (~$5)
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5

# Save results as a named baseline
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --save tests/results/my_baseline.json

# Compare against a baseline
python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --compare tests/results/my_baseline.json

A/B testing a doc change

# 1. Save baseline BEFORE your edit
python tests/eval.py tests/tasks/task_16_multiple_imputation.md --runs 5 \
    --save tests/results/before_mi_fix.json

# 2. Make your edit to the reference doc

# 3. Re-run and compare
python tests/eval.py tests/tasks/task_16_multiple_imputation.md --runs 5 \
    --compare tests/results/before_mi_fix.json

Look for: mean score going up, SD staying flat or going down, no new failure modes.

Testing modes

# Full skill (default) — loads the complete plugin
python tests/eval.py tests/tasks/task_*.md

# Runner-only — minimal plugin with just execution instructions, no reference docs
python tests/eval.py tests/tasks/task_*.md --runner-only

# No skill — no plugin at all
python tests/eval.py tests/tasks/task_*.md --no-skill

# Alternative skill version — point at a different plugin directory
python tests/eval.py tests/tasks/task_*.md --skill-path /path/to/modified/plugin

The runner-only baseline is the meaningful comparison. It isolates what the reference material adds beyond "here's how to run Stata." The no-skill baseline is weaker (agent can't even execute code).

Parallel execution

Run multiple tasks in parallel by launching separate processes:

python tests/eval.py tests/tasks/task_01_data_cleaning.md --runs 5 \
    --save tests/results/v3_task01.json &
python tests/eval.py tests/tasks/task_07_did.md --runs 5 \
    --save tests/results/v3_task07.json &
wait

Batch 3-4 tasks at a time to avoid resource exhaustion. Run directories are created atomically, so parallel execution is safe.

Output structure

Each run creates `tests/results/run_NNN/` containing:

| File | Contents | |------|----------| | `transcript.json` | The test agent's full response | | `judge_findings.md` | Per-category scores with justifications | | `metadata.json` | Score, model, skill_mode, cost, duration, timestamp | | `code_files/` | Any .do/.log files the agent wrote |

Aggregate results are saved to the `--save` path as a JSON list of per-run results.

Multi-Agent Post-Test Analysis

After running evals, dispatch Sonnet subagents to analyze the results in parallel. This is the workflow that surfaces actionable improvements.

Step 1: Generate the comparison table

# Quick script to build the comparison table from saved results
import json, statistics
for i in range(1, 25):
    tn = f"{i:02d}"
    with open(f"tests/results/v3_task{tn}.json") as f:
        full = [r["score"] for r in json.load(f) if r["score"] is not None]
    with open(f"tests/results/v3_runner_task{tn}.json") as f:
        runner = [r["score"] for r in json.load(f) if r["score"] is not None]
    fm, rm = statistics.mean(full), statistics.mean(runner)
    print(f"Task {tn}: full={fm:.1f} runner={rm:.1f} delta={fm-rm:+.1f}")

Step 2: Dispatch analysis subagents

Launch 4-5 parallel subagents, each investigating a specific pattern in the results. The prompts below are templates — adapt them to your actual results.

**Agent 1 — Regression analysis (skill hurts):** > Investigate tasks where the full skill scores LOWER than runner-only. Read the result JSONs, the relevant reference docs, and the judge reasoning for low-scoring runs. Identify specific examples in the reference docs that teach wrong patterns. Compare what the with-skill agent does vs the runner-only agent.

**Agent 2 — Redundancy analy