/scholar-experiment

name: scholar-experiment
disable-model-invocation: true
description: Systematic experimental results analysis workflow for ML/AI research papers. Connects experimental data to publication-ready Results sections with statistical validation, visualizations, and quality checks. Triggers on "analyze experimental results", "generate results section", "statistical analysis of experiments", "compare model performance", "create results visualization".
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion, TodoWrite
session-mode: none

Scholar Experiment: Results Analysis Workflow

A systematic workflow for analyzing ML/AI experimental results and generating publication-ready Results sections. Transforms raw experimental data into validated statistical analyses, publication-quality visualizations, and well-structured paper content.

Pre-load (before execution)

1. **Codebase docs**: If `.workflow/codebase/ARCHITECTURE.md` exists, read for project context 2. **Specs**: `maestro load --type spec --category coding` — load coding conventions 3. **Wiki knowledge**: `maestro search "academic writing research paper" --json` — top 5 entries as prior context 4. All optional — proceed without if unavailable

Architecture Overview

                     scholar-experiment
                           |
     ┌─────────────────────┼─────────────────────┐
     |                     |                       |
  [User Input]    [Experiment Context]    [Preferences]
     |                     |                       |
     └─────────┬───────────┘                       |
               v                                   |
┌──────────────────────────┐                       |
│  Phase 1: Data Loading   │ <─ preferences ───────┘
│  Load, validate, inspect │
└────────────┬─────────────┘
             │ cleanedData, dataProfile
             v
┌──────────────────────────┐
│  Phase 2: Statistical    │
│  Analysis & Testing      │
└────────────┬─────────────┘
             │ statisticalResults
             v
┌──────────────────────────┐
│  Phase 3: Visualization  │
│  Plots, charts, tables   │
└────────────┬─────────────┘
             │ figureSpecs, tableSpecs
             v
┌──────────────────────────┐
│  Phase 4: Results Writing│
│  Draft Results section   │
└────────────┬─────────────┘
             │ resultsDraft
             v
┌──────────────────────────┐
│  Phase 5: Quality Check  │
│  Validate & verify       │
└──────────────────────────┘
             │
             v
        [Output Files]
        - analysis-report.md
        - results-draft.md
        - visualization-specs.md

Key Design Principles

1. **Statistical rigor first**: Every claim must be backed by appropriate statistical tests with complete reporting (mean, SD/SE, p-value, effect size) 2. **Pre-test before test**: Always check assumptions (normality, variance homogeneity) before selecting parametric vs non-parametric tests 3. **Publication-quality output**: All visualizations must meet journal standards (vector format, colorblind-friendly, proper error representation) 4. **Complete reporting**: Never report p-values alone — always include effect sizes, confidence intervals, and descriptive statistics 5. **No cherry-picking**: Report all planned comparisons, not just significant results 6. **Reproducibility**: Document all analysis steps, parameters, and random seeds

Statistical Tools and Libraries

This workflow requires statistical computing capabilities. Recommended implementations:

**Python Stack** (recommended for ML/AI research):

import numpy as np              # Numerical computing
import pandas as pd             # Data manipulation
import scipy.stats as stats     # Statistical tests
import matplotlib.pyplot as plt # Visualization
import seaborn as sns           # Statistical visualization
from statsmodels.stats import multitest  # Multiple comparison corrections

**R Stack** (alternative for advanced statistics):

library(tidyverse)  # Data manipulation and visualization
library(stats)      # Statistical tests
library(effsize)    # Effect size calculations
library(multcomp)   # Multiple comparisons

**Minimum Requirements**:

• Statistical test functions (t-test, ANOVA, Mann-Whitney, Kruskal-Wallis, etc.)
• Effect size calculations (Cohen's d, eta-squared, r)
• Multiple comparison corrections (Bonferroni, Holm, FDR)
• Normality tests (Shapiro-Wilk, Kolmogorov-Smirnov)
• Variance homogeneity tests (Levene's test)

**Note**: If these libraries are not available, the workflow will guide you to use online statistical calculators or manual computation, but automated analysis is strongly recommended for reproducibility. 2. **Pre-test before test**: Always verify assumptions (normality, homogeneity of variance) before selecting parametric tests 3. **No cherry-picking**: Report all experimental runs, not just the best ones 4. **Publication-quality output**: All visualizations follow colorblind-friendly, vector-format, error-bar standards 5. **Reproducibility**: Track random seeds, hyperparameter ranges, compute resources, and experimental setup

Interactive Preference Collection

Before dispatching to phases, collect analysis preferences:

const prefResponse = AskUserQuestion({
  question: "How would you like to configure the analysis?",
  options: [
    {
      label: "Analysis Type",
      description: "Select the type of analysis",
      choices: [
        { value: "full", label: "Full Analysis", description: "Complete pipeline: stats + visualization + writing" },
        { value: "comparison", label: "Model Comparison", description: "Focus on comparing multiple models/methods" },
        { value: "ablation", label: "Ablation Study", description: "Focus on component contribution analysis" },
        { value: "visualization", label: "Visualization Only", description: "Generate visualization specs only" }
      ]
    },
    {
      label: "Statistical Reporting",
      desc