/model-assessment

name: model-assessment
description: Use when validating or evaluating a trained medical-imaging model. Audits split leakage and validation design, computes task-correct held-out metrics (Dice + HD95, AUROC + AUPRC, FROC, calibration), and covers uncertainty/OOD and Grad-CAM explainability, each with a gate.
metadata:
  triggers: "model validation, validate AI model, imaging model validation, data leakage, split leakage, train test split, patient-level split, internal validation, external validation, validation design, leakage audit, segmentation model validation, classification model validation, detection model validation, nnU-Net validation, deep learning validation, CLAIM 2024, generalizability, held-out test set, model evaluation, held-out metrics, test set metrics, Dice, HD95, NSD, surface distance, Metrics Reloaded, AUROC, AUPRC, bootstrap CI, calibration, ECE, reliability diagram, subgroup analysis, slice metrics, mAP, FROC, segmentation metrics, detection metrics, evaluate predictions, interactive segmentation, promptable segmentation, SAM2, MedSAM2, nnInteractive, number of clicks, NoC, interactions-to-threshold, click budget, generative metrics, image synthesis, SSIM, PSNR, SNR, CNR, downstream task, multiclass classification, Obuchowski index, Harrell's C, time-dependent ROC, uncertainty, uncertainty quantification, UQ, epistemic, aleatoric, MC-dropout, monte carlo dropout, deep ensemble, conformal prediction, split conformal, prediction interval, coverage, calibration under shift, out-of-distribution, OOD detection, distribution shift, Mahalanobis, energy score, ODIN, selective prediction, abstention, reject option, deployment safety, DECIDE-AI, predictive uncertainty, explainability, interpretability, saliency, saliency map, grad-cam, gradcam, grad-cam++, attention map, attention rollout, integrated gradients, captum, pytorch-grad-cam, heatmap, class activation map, CAM, feature attribution, sanity check, Adebayo, model randomization, localization metric, pointing game, IoU with ground truth, XAI, explainable AI, model looks at, faithfulness"

Model-Assessment Skill

Assess a trained medical-imaging model (in-house, vendor or open-weights; segmentation, classification or detection) in the order the evidence is built: **Part A** designs and audits the validation study, **Part B** computes task-correct held-out metrics, **Part C** adds the uncertainty / OOD / abstention layer a deployment claim needs, and **Part D** makes an explainability analysis survive review. Run the parts the request needs (a Grad-CAM question starts at Part D), but no metric headline is reported before the Part A split gate is green. Each part ends in a stdlib gate whose verdict is reproduced from a file — never report a pass without running it.

Numbers come only from code executed on the supplied predictions, split table, or the researcher's executed UQ/XAI code; if predictions or ground truth are missing, say so and stop. Integrate MONAI / nnU-Net, MAPIE, captum, pytorch-grad-cam and pretrained OOD scorers by reference — never reimplement them, never build or train the model, never run a model on real patient data.

Elsewhere: building/training → `/model-scaffold`; choosing or vetting the model → `/model-selection`; data-stage preprocessing leakage → `/imaging-data`; paired model comparison / added value over a baseline / decision curves / MRMC / ICC / calibration tables → `/analyze-stats` (added value: `incremental_value.md`); AI-vs-expert reader rubric and IRR → `/design-ai-benchmarking`; LLM/MLLM → `/mllm-eval`; general validity → `/design-study`; classical-ML tabular calibration → `/radiomics-ml`; item-by-item reporting audit → `/check-reporting`; a finished manuscript → `/self-review` or `/peer-review` (the MD0–MD11 `model_development.md` probe).

Part A — Validation design

The rationale behind Phases 2–7 — the full leakage taxonomy, the internal-vs-external ladder, comparator design, run variance, test-set sizing, the reporting map — is in `${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). Verify citations via `/search-lit` (confirmed DOI/PMID), else mark `[UNVERIFIED - NEEDS MANUAL CHECK]`; flag an uncertain CLAIM 2024 / TRIPOD+AI / Metrics Reloaded item `[VERIFY]` and ask.

Phase 1 — Task, intended-use horizon, analysis unit

State the task, the **intended-use horizon** (screening, triage, pre-procedure, post-hoc), the **single headline metric** the conclusion leans on, and the **analysis unit** (per-patient / per-lesion / per-image). Everything downstream is read against this; a per-lesion metric is never reported as per-patient.

Phase 2 — Leakage audit (run first)

Produce the emitted split-assignment table (`patient_id,split`) and run:

python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
  --splits <split_assignment.csv> --out qc/split_leakage.json --strict

`PATIENT_OVERLAP` (a patient in ≥ 2 partitions) and `MISSING_SEED` (an unreproducible split) are Major, `SINGLE_PARTITION` Minor — proven by set arithmetic on the ID column the gate prints as `id_col` (pass `--id-col` with the patient identifier when it is not the one; an auto-picked column whose name is not patient-level, such as `image_id`, gets a Minor `ID_COL_NOT_PATIENT_LEVEL`). A design with patient overlap is never approved. Then walk the leakage the table cannot show (Kapoor & Narayanan, *Patterns* 2023): preprocessing fit before the split (normalisation, resampling, foundation-model embeddings, ComBat harmonisation over the whole cohort — `/imaging-data` gates the declared pipeline), site / scanner / burned-in-label shortcuts, and temporal leakage (a random split where future and past coexist). The decisive question: *could any value used in training have been computed only with knowledge of a test case?*

Phase 3 — Validation tier

Classify honestly: apparent → internal random split → cross-validation → temporal → geographic / external (different site, scanner,