/preregistering-analysis

name: preregistering-analysis
description: Use before running any confirmatory analysis or looking at outcome data, when testing a hypothesis, computing a p-value, or about to claim an effect - locks predictions and decision rules before results are seen

Pre-Registering the Analysis

Overview

Write down the hypotheses, the exact analysis, the directional predictions, and the decision rules — and freeze them — *before* you look at the outcomes. Then run exactly that.

**Core principle:** If you decided the analysis after seeing the data, you cannot tell whether the data shaped the answer. A prediction is only evidence if it was made before the result.

**Violating the letter of this rule is violating the spirit of this rule.**

This is the pre-registration discipline — the science analog of test-driven development. In TDD you write the test before the code so you know the test tests something. Here you write the prediction before the result so you know the result confirms something.

The Iron Law

NO CONFIRMATORY CLAIM WITHOUT A PRE-REGISTERED PREDICTION FIRST

Looked at the outcomes before locking the analysis? That analysis is now **exploratory**. You cannot relabel it confirmatory. To make a confirmatory claim, pre-register and test on data you have not yet used.

**No exceptions:**

• Don't "lightly" reorder the analysis after a peek
• Don't add the covariate that "obviously" should be there once you saw the result
• Don't move the cutoff to where it "works"
• Don't keep an unregistered analysis and call it confirmatory because "it would have been my prediction anyway"

Looked? It's exploratory. Period.

When to Use

**Always, before:**

• Any hypothesis test you intend to report as confirmatory
• Any p-value, confidence interval, or effect estimate that will support a claim
• Loading or plotting the outcome variable in a way that reveals the relationship you'll test

**Exploratory work is allowed and valuable** — but it must be *labeled* exploratory and never dressed up as confirmation. Exploratory findings are hypotheses for the next pre-registration.

**Exceptions (decide with your human partner):**

• Purely descriptive reporting with no inferential claim
• Pure method development on simulated data
• Re-analysis of already-public results
• Feasibility mode, if your human partner opted into it (`science-superpowers:establishing-feasibility-first`) — registration is **deferred to the exit gate, not cancelled**, and nothing produced before it may be claimed as confirmatory

Don't Register Over Runs That Have Never Happened

A registration whose configurations have never executed is not protection — it is rework waiting to happen. Before freezing, confirm:

• Every configuration in the plan has run at least once, or is known to be runnable from a **measured** point (not an extrapolation)
• Every acceptance criterion has been observed on a real run of the pipeline it judges. Writing "error decreases monotonically" for a pipeline you have never executed is writing a pass/fail rule for behavior you have not seen
• The cheapest run that could falsify your prediction has been executed

That last one matters most. If a run costing seconds could disconfirm the prediction you are about to freeze, run it first. A validation benchmark on different data or a different geometry cannot contaminate the confirmatory test — it is a separate measurement — so there is no rigor argument for freezing before it. Freezing a prediction that a cheap run would have falsified costs a re-registration and buys nothing.

Where feasibility is genuinely unknown, that is the signal to offer feasibility mode rather than freeze early.

Thinking "I'll just check the result first, then pre-register"? Stop. That's the rationalization this skill exists to stop.

Lock — Execute — Separate

digraph prereg_cycle {
    rankdir=LR;
    lock [label="LOCK\nhypotheses, analysis,\npredictions, decision rules", shape=box, style=filled, fillcolor="#ffcccc"];
    falsifiable [label="Disconfirming\nresult specified?", shape=diamond];
    freeze [label="FREEZE\ncommit + timestamp", shape=box];
    execute [label="EXECUTE\nexactly as registered", shape=box, style=filled, fillcolor="#ccffcc"];
    separate [label="SEPARATE\nlabel anything else\nexploratory", shape=box, style=filled, fillcolor="#ccccff"];

    lock -> falsifiable;
    falsifiable -> freeze [label="yes"];
    falsifiable -> lock [label="no - unfalsifiable,\nreframe"];
    freeze -> execute;
    execute -> separate;
}

LOCK — Write the prediction before the result

Specify, in writing:

• **Hypotheses:** H0 and a directional H1 where justified
• **The primary analysis:** the one model/test, the exact variables, transformations, covariates, exclusions
• **The prediction:** what you expect, with direction (and magnitude if the survey gave you one)
• **The decision rule:** the exact result that confirms vs. disconfirms — e.g., "confirm H1 if the coefficient on `exposure` is positive with p < .005 in the pre-specified model"
• **Stopping rule:** the sample size is fixed in advance; you will not peek-and-extend
• **Multiplicity plan:** how many tests, and the correction (so you don't harvest one "significant" result from twenty)
• **Secondary/exploratory analyses:** listed and labeled as such

Verify it is falsifiable — the "watch it fail" step

A registration is only real if some possible result would disconfirm the prediction. State that result explicitly. If no result could disconfirm it, the prediction is unfalsifiable — reframe before freezing. This mirrors verifying a test fails for the right reason before trusting it.

FREEZE — Make the timestamp real

Write the pre-registration to `docs/science-superpowers/preregistrations/YYYY-MM-DD-<topic>.md`, then freeze it with `prereg.sh` (it ships alongside this skill's SKILL.md) — **before** touching outcomes:

# <skill-dir> = this skill's directory, e.g. $CLAUDE_PLUGI