/scientific-critical-thinking

name: "scientific-critical-thinking"
description: "Evaluating scientific evidence and claims. Covers study design hierarchy (RCT to expert opinion), effect sizes (OR, RR, NNT, Cohen's d), confounding, p-value vs clinical significance, GRADE quality assessment, reproducibility, and bias types (selection, information, confounding, reporting). Use when reading a paper or assessing claims."
license: "CC-BY-4.0"

Scientific Critical Thinking: Evaluating Evidence and Claims

Overview

Scientific critical thinking is the disciplined application of logical and methodological standards to evaluate whether a study's design, analysis, and interpretation support its conclusions. It is the skill that separates a researcher who synthesizes evidence from one who accumulates it. This guide covers the hierarchy of evidence, the mechanics of common biases, effect size interpretation, the p-value controversy, GRADE evidence grading, and common logical fallacies in the interpretation of scientific literature.

Key Concepts

1. Study Design Hierarchy

Study designs vary in their ability to support causal inference. The hierarchy below applies to questions about the effect of an intervention or exposure on an outcome:

Systematic reviews and meta-analyses of RCTs (highest causal certainty)
    ↓
Randomized Controlled Trials (RCTs)
    ↓
Non-randomized controlled trials / cluster-randomized trials
    ↓
Prospective cohort studies (follow exposure → outcome forward in time)
    ↓
Retrospective cohort studies
    ↓
Case-control studies (compare exposed vs. unexposed given outcome)
    ↓
Cross-sectional studies (measure exposure and outcome simultaneously)
    ↓
Case series and case reports
    ↓
Expert opinion, mechanistic reasoning, animal models (lowest causal certainty)

**Important exceptions**: For questions about rare outcomes, case-control designs are often more efficient than cohort studies. For questions about diagnostic accuracy, randomized designs are usually inappropriate — cross-sectional or cohort designs with verified reference standards are preferred. For harm questions, RCTs are often infeasible (ethical constraints), making large cohort studies the best available evidence.

2. Effect Measures and Their Interpretation

Effect measures quantify the relationship between an exposure/intervention and an outcome. Confusing them is a leading source of misinterpretation.

| Measure | Formula | Use case | Key interpretation | |---------|---------|----------|-------------------| | **Risk Ratio (RR)** | Risk in exposed / Risk in unexposed | Cohort studies, RCTs | RR = 2.0: exposed group has twice the risk | | **Odds Ratio (OR)** | Odds in exposed / Odds in unexposed | Case-control studies, logistic regression | Approximates RR when outcome is rare (<10%); overestimates RR for common outcomes | | **Hazard Ratio (HR)** | Instantaneous event rate ratio | Survival analysis (Cox regression) | HR = 0.7: 30% lower hazard of event per time unit in treated group | | **Number Needed to Treat (NNT)** | 1 / Absolute Risk Reduction | Clinical decision-making | NNT = 20: treat 20 patients to prevent 1 event | | **Absolute Risk Reduction (ARR)** | Risk_control − Risk_treated | Clinical impact | ARR = 2%: intervention reduces absolute event rate by 2 percentage points | | **Cohen's d** | (μ₁ − μ₂) / σ_pooled | Continuous outcomes, psychology | d = 0.2 small; 0.5 medium; 0.8 large | | **Pearson r** | Correlation coefficient | Association, not causal | r = 0.1 small; 0.3 medium; 0.5 large (Cohen 1988) |

**Common error**: Reporting only the relative risk reduction (e.g., "50% reduction in risk") without the absolute risk reduction. A treatment that reduces risk from 2% to 1% has a 50% relative reduction but only 1% absolute reduction (NNT = 100). The relative measure appears more impressive but the absolute measure is clinically relevant.

3. Bias Types

Bias is systematic deviation of results or inferences from the truth. Unlike random error (reduced by larger samples), bias is directional and not correctable by increasing sample size.

**Selection bias**: Systematic difference in characteristics between those selected and not selected for study.

• *Examples*: Healthy worker effect (workers healthier than general population), loss to follow-up bias (sicker patients drop out), volunteer bias
• *Detection*: Compare baseline characteristics of included vs. excluded participants; assess attrition patterns

**Information bias**: Systematic error in measuring exposure or outcome.

• *Recall bias*: Cases remember exposure better than controls (especially case-control studies for rare diseases)
• *Observer bias*: Assessors aware of exposure status rate outcomes differently
• *Detection*: Look for blinding of outcome assessors; validated measurement instruments; objective vs. self-reported outcomes

**Confounding**: A variable associated with both the exposure and the outcome, creating a spurious or masked association.

• *Positive confounding*: Confounder inflates the observed association
• *Negative confounding*: Confounder masks a true association
• *Detection*: Compare crude and adjusted effect estimates; large change (>10%) indicates confounding
• *Control methods*: Randomization (RCTs), multivariable adjustment, propensity score methods, restriction, matching

**Reporting bias**: Selective reporting of outcomes or results based on their statistical significance or direction.

• *Publication bias*: Positive results are published; null results are not
• *Outcome reporting bias*: Predefined outcomes not reported if non-significant; unplanned analyses reported if significant
• *Detection*: Compare registered protocol vs. published outcomes; funnel plot asymmetry for meta-analyses

4. The P-value and Statistical vs. Clinical Significance

A p-value is the probability of observing results at least as extreme as those obtained, under the null hypothesis. It is NOT the probability that the null h