/investigating-anomalous-results

name: investigating-anomalous-results
description: Use when a result is surprising, impossible, contradicts a sanity check, a pipeline fails, a model won't converge, or a replication fails - before adjusting anything

Investigating Anomalous Results

Overview

Random tweaks waste time and manufacture false findings. Quietly dropping the inconvenient data point, nudging the cutoff, or re-running until it "works" doesn't fix the problem — it fabricates a result.

**Core principle:** ALWAYS find the root cause before adjusting anything. An adjustment made before you understand the cause is, at best, noise and, at worst, fraud.

**Violating the letter of this process is violating the spirit of it.**

The first question for any anomaly: **is this a code bug, a data issue, or a real finding?** You cannot answer by guessing, and you must not "fix" it until you know — because a real finding is not something to fix.

The Iron Law

NO ADJUSTMENTS WITHOUT ROOT-CAUSE INVESTIGATION FIRST

No dropping data, changing a test, transforming a variable, re-running with new parameters, or "cleaning" until Phase 1 is complete. If you haven't found the cause, you cannot justify the adjustment.

When to Use

• A result is surprising, too good, or contradicts what's known
• An impossible value (negative count, probability > 1, age of 500)
• A pipeline errors or a model won't converge
• A result contradicts a sanity check or a simple cross-tab
• A replication or robustness check disagrees with the primary result
• Performance that's suspiciously high (often leakage)

**Use this ESPECIALLY when:**

• The anomaly is *convenient* (it supports your hypothesis) — that's when the urge to skip investigation is strongest
• Under time pressure
• You've already tried one quick adjustment

The Four Phases

Complete each before the next.

Phase 1: Characterize the Anomaly

**Before any adjustment:**

1. **Read the output/error carefully.** The exact value, the exact message, the stack trace. Note shapes, counts, and where in the pipeline it appears. 2. **Reproduce it deterministically.** Fixed seed, same data version. Does it happen every time? If not reproducible, gather more data — don't guess. 3. **Check recent changes.** What changed in the code, the data version, or the environment since it last looked right? `git diff`, data provenance, package versions. 4. **Gather evidence at each pipeline stage.** Instrument raw → cleaned → derived → result. Print shapes, ranges, null counts, and key summaries at each boundary. This reveals *where* the anomaly enters.

raw:     N=10342, outcome mean=0.31, nulls=0
cleaned: N=10298, outcome mean=0.31, nulls=0      <- 44 dropped, expected
derived: N=10298, feature mean=4e7                <- WRONG: unit blew up here
result:  coefficient enormous                     <- symptom; cause is upstream

5. **Classify the cause** with the evidence: code bug, data issue, or real finding. Do not proceed assuming one without evidence.

Phase 2: Pattern Analysis

1. **Find a working comparison.** A subset, variable, site, or time period that behaves as expected. What's different about the anomalous one? 2. **Compare against a reference.** A known correct result, a textbook expectation, a prior estimate from the survey. Read the reference method completely — don't skim. 3. **List every difference** between working and anomalous, however small. Don't assume "that can't matter."

Phase 3: Hypothesis and Test

1. **Form a single hypothesis:** "The anomaly is caused by X because Y." Write it down. Be specific. 2. **Test minimally** on the smallest slice that exhibits it. One variable at a time. 3. **Verify before continuing.** Confirmed → Phase 4. Not confirmed → new hypothesis. Don't stack adjustments. 4. **When you don't know, say so.** "I don't understand why the derived feature inflates" — then investigate or ask. Don't pretend.

Phase 4: Resolution

Resolve according to the cause you established:

• **Code bug:** Write a minimal failing check that reproduces it, fix the code at the source (not a downstream patch), re-run the affected steps, confirm the anomaly is gone and other results unchanged.
• **Data issue:** Handle it per a documented rule. If the rule was not pre-registered, the decision is a deviation — record it, and the affected analysis becomes exploratory. Dropping observations is legitimate ONLY when root cause shows they are invalid measurements (e.g., a sensor error), and that justification is documented. Never drop data because it's inconvenient.
• **Real finding:** It is a result, not a defect. Do NOT "fix" it. If it was not what you pre-registered, it is an exploratory finding — report it as such, and pre-register a fresh confirmatory test (ideally on new/held-out data) before claiming it.

**If 3+ adjustments fail:** STOP. The pipeline or the design may be wrong — repeated failures that each reveal a new problem indicate a structural issue, not a series of small bugs. Discuss the design with your human partner before another attempt. Re-opening the design or pre-registration must be documented as a deviation.

Red Flags - STOP and Investigate

• "Let me just drop that outlier and re-run"
• "I'll nudge the threshold until it's significant"
• "Probably a fluke, let me re-run with a different seed" (changing the seed to change the result)
• "The result I want appeared, no need to dig into why the first run didn't"
• "Quick adjustment now, understand it later"
• Proposing an adjustment before tracing where the anomaly enters the pipeline
• **An adjustment that happens to make the result support your hypothesis**

**All of these mean: STOP. Return to Phase 1.**

Your Human Partner's Signals You're Doing It Wrong

• "Is that even possible?" — you reported a value without sanity-checking it
• "Why did that point get dropped?" — you adjusted without documenting the cause
• "Did you check leakage?" — suspiciously good performance