/design-experiment

name: design-experiment
description: "Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness designs are modes, not strategy nodes."

design-experiment

Purpose

Translate a falsifiable hypothesis into a statistically defensible experiment. Factorial, ablation, comparison, scaling, and robustness are modes.

When to use / not applicable

Use after a hypothesis has an operational construct and discriminating prediction. Not applicable when the hypothesis or measurable outcome is absent.

Input contract

mode_contracts:
  factorial: &experiment_input
    required: [falsifiable_hypothesis, outcome, factors, constraints]
    optional: [baseline, candidate_models, budget, robustness_axes]
    constraints: [analysis_plan_must_be_preregistered]
  ablation: *experiment_input
  comparison: *experiment_input
  scaling: *experiment_input
  robustness: *experiment_input

Execution protocol

Do not perform called SOP operations inline; each loaded SOP owns its contract and thresholds.

1. You MUST load skill `identify-variables` to operationalize the outcome, factors, controls, and their functional roles. You MUST load skill `enumerate-dimension-values` to define admissible levels or perturbation values. If the resource envelope or feasibility constraints cannot support an executable design, consider `analyze-constraints-readiness` before committing to a mode. 2. Choose `factorial`, `ablation`, `comparison`, `scaling`, or `robustness` mode. 3. You MUST load skill `specify-metrics` to preregister metrics, estimands, directionality, and decision thresholds. You MUST load skill `estimate-sample-size` to derive the sample or repetition requirement. You MUST load skill `select-statistical-method` to select the inference or estimation method before observing outcomes; specify power and the stopping rule. If the proposed metric or validator may share artifacts, labels, or assumptions with the system under test, consider `audit-validator-independence` before freezing the analysis plan. 4. You MUST load skill `construct-design-matrix` to construct the runnable matrix for the selected mode. You MUST load skill `design-randomness-protocol` to define seeds, repetitions, and propagation rules. You MUST load skill `specify-execution-environment` to capture interpretation-relevant hardware, software, data, configuration, and versions. You MUST load skill `specify-reproducibility-protocol` to define and test the intended reproduction level. You MUST load skill `optimize-design-under-budget` to select a feasible information-efficient design under the declared resource envelope; include resource and failure checks. Once the design has produced observations and the task changes from planning to inference, `analyze-experiment-results` may be the better next tactic.

Mode branches

For `comparison` or `robustness`, You MUST load skill `select-experimental-baseline` to choose a controlled baseline matched to the claim.

• `factorial`: vary multiple factors in a structured design so main effects and interactions are estimable within the declared resource envelope.
• `ablation`: remove or replace components systematically to attribute the outcome to individual parts and suspected interactions. You MUST load skill `map-ablation-components` to define ablatable units, dependencies, and legal removal or replacement operations.
• `comparison`: evaluate a target against controlled baselines with matched confounds, compute, tuning effort, and preregistered tests.
• `scaling`: instantiate geometric or otherwise justified scale points to test how the outcome changes across the declared regime.
• `robustness`: perturb relevant conditions or inputs and measure whether the claimed effect survives the defined stress space.

Output contract

mode_contracts:
  factorial:
    produces: [factor_level_matrix, factor_level_catalog, estimands_main_effects_interactions, metric_significance_plan, sample_power_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  ablation:
    produces: [ablation_matrix, baseline_anchors_full_minimal, attribution_contrasts, component_interaction_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  comparison:
    produces: [controlled_baseline_comparison, matched_confound_controls, seed_environment_protocol, statistical_comparison_plan, reproducibility_protocol]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  scaling:
    produces: [scaling_axes, geometric_scale_points, scaling_experiment_grid, curve_fit_plan, scale_budget_plan]
    delta_fields: [findings, decisions, uncertainties, open_questions]
  robustness:
    produces: [perturbation_stress_matrix, severity_axes, baseline_comparison, degradation_metrics, survival_criteria]
    delta_fields: [findings, decisions, uncertainties, open_questions]

Thresholds and quality gates

• `experiment-design` HARD-GATE and Budget Gate remain mandatory; no exit before declared minimum yield.
• Resource and run-scale gates are relative to the declared resource envelope and eligible factor/condition space; record numerator, denominator, batch increment, stopping reason, and source references.
• Scale, ablation, comparison, and robustness coverage must reach a justified relative floor over the declared design space; do not substitute an unreasoned fixed run count.
• Factor levels, comparison baseline, statistical test, significance threshold, sample-size rationale, and stopping rule must all be explicit.
• Significance threshold must be pre-registered, not chosen post-hoc.
• Budget-constrained design must report at least one feasible design under the stated resource envelope.
• Scaling mode retains the source criterion of a geometric progression, typically 4-8 points, while evaluating coverage relative to the declared scale domain. Relative audit: declared un