/design-study

name: design-study
description: Use when checking a radiology or medical AI study design before drafting or submission. Reviews the analysis unit, cohort logic, leakage risks, comparator, validation strategy and reporting-guideline fit. AI-vs-expert benchmarks are /design-ai-benchmarking.
metadata:
  triggers: "study design, leakage check, cohort design, analysis plan, validation strategy, comparator design, bias check"

Design-Study Skill

Core Review Questions

Always inspect:

1. The exact research question. 2. The analysis unit: patient, lesion, exam, study, phase, report. 3. The index date or decision point. 4. How inclusion and exclusion criteria are applied. 5. Any information leakage. 6. The reference standard or endpoint definition. 7. A clinically meaningful comparator. 8. The validation strategy. 9. The uncertainty reporting required. 10. The best-fitting reporting guideline. 11. Whether exposure/outcome/covariate **definitions are literature-grounded** or invented ad hoc from the data dictionary. If ad hoc, defer to `/define-variables` before drafting Methods.

---

Standard Output

## Study Design Review
Question: ...
Study type: ...
Analysis unit: ...
Index date / prediction timepoint: ...

### Strengths
- ...

### Major validity risks
1. ...
2. ...

### Minimal fixes
- ...

### Reporting fit
- Recommended guideline: ...

### Decision
- Ready for analysis / Needs redesign / Drafting can proceed with limitations

Cite a reference only with a `/search-lit`-confirmed DOI or PMID; mark any other `[UNVERIFIED - NEEDS MANUAL CHECK]`. Never invent a clinical definition, diagnostic criterion, or guideline recommendation: flag an unconfirmed one `[VERIFY]` and ask the user.

---

Workflow

Phase 1: Reconstruct the study

Extract from the protocol, draft, slides, tables, or notes: clinical problem, intended use case, population, inputs, outputs, outcome definition, and timing of variable availability.

**Gate:** Present the reconstructed study summary (question, analysis unit, intended use) to the user and confirm it before proceeding — a wrong reconstruction misdirects the entire validity review.

Phase 2: Check structural validity

Read a reference only when its condition holds:

| File | Read it when | |---|---| | `references/dag_adjustment.md` | confounding control needs an explicit adjustment set | | `references/target_trial_emulation.md` | the design emulates a target trial | | `references/venue_accept_recipe.md` | a clinical DL / AI-validation study must decide **which venue tier the achievable design can be accepted at, and the one design move that reaches the tier above** (the bridge into `/find-journal`); skip when there is no publication-tier decision | | `references/combine_models_ablation_design.md` | the model is built by **combining / adapting / fine-tuning existing models** (nnU-Net, TotalSegmentator, SAM/MedSAM, a pretrained backbone) and the comparator must be designed as an ablation; skip for a model trained de novo | | `references/multi_model_comparison_design.md` | the contribution is **comparing several models / architectures head-to-head** and the comparison must be fair; skip for a single-model study (one model's ablation → the row above; AI-vs-human → `/design-ai-benchmarking`) | | `references/segmentation_failure_characterization_design.md` | the claim is that a segmentation model is **clinically usable**, not that it scores well; skip when the endpoint is benchmark accuracy (metric choice → `/model-assessment`; abstention / risk–coverage → `/model-assessment`) |

A. Analysis unit

Look for mismatches: a patient-level claim from lesion-level analysis; an exam-level split with patient overlap; phase-level samples treated as independent.

B. Leakage

Look for:

• postoperative features used for preoperative prediction
• normalization or thresholding performed before the data split
• repeated exams across train/test
• reader annotations derived from outcome information
• **input-text contamination for NLP/LLM extraction tasks**: if the input includes clinical history,

indication, impression, prior diagnosis, or referral text, confirm it does not name or strongly imply the target label. If it does, the task is information retrieval under label leakage, not phenotype inference: redesign the input mask, report a sensitivity analysis excluding those fields, or reframe.

• **construct dependence** (a predictor that is a definitional component of the outcome): (i) an input

that computes the outcome (fasting insulin and glucose when the outcome is HOMA-IR), or (ii) a near-tautological composite of the outcome's defining components (inflated, near-circular association). Test: "could this predictor be derived, in whole or part, from the outcome's definition or the same measurement?" If yes, exclude it, or keep it only as a labeled calibration probe.

F. Time origin & survivorship (incident / transition models)

For any time-to-event or incident/transition design, check before drafting:

• **Time origin per model.** Each incident model starts its at-risk clock at the correct origin.

Watch for **immortal-time bias** (a span in which the event cannot occur, misattributed to one group) and **left-truncation / delayed entry**.

• **Mediator-ascertainment-window survivorship.** A "progressor" / transition label conditional on

*surviving to* a later ascertainment (a second scan, a follow-up visit) is survivorship-biased; plan a landmark time or an explicit intermediate-state (multistate / illness-death) model.

• **Primary-analysis-set selection.** If the primary is not the full cohort, pre-specify why, from

the missingness mechanism *relative to the outcome*: complete-case regression is unbiased when missingness is independent of the outcome given the covariates (even with a covariate MNAR) and generally biased when it depends on the outcome, including under MAR, where multiple imputation is the us