/icml-reviewer

name: icml-reviewer
description: |
  Paper reviewer that evaluates machine learning research projects following official ICML reviewer guidelines. Provides comprehensive reviews with actionable feedback across all key dimensions: claims/evidence, relation to prior work, originality, significance, clarity, and reproducibility. Also provides formative feedback on incomplete drafts, proposals, and research code repositories.

  MANDATORY TRIGGERS: review paper, ICML review, paper review, evaluate paper, research paper feedback, ML paper review, conference review, academic review, paper critique, NeurIPS review, ICLR review, project proposal, research proposal, paper draft, early feedback, incomplete paper, work in progress, WIP review, review repo, review codebase, research project review

ICML Paper Reviewer

Enables rigorous review of ML research papers following official ICML guidelines.

Workflow

Step 1: Input Analysis & Mode Selection

**Determine input type:**

• **Complete paper**: PDF/text with abstract, methodology, experiments, results → Full Review Mode
• **Incomplete document**: Missing major sections, labeled draft/proposal, or user indicates early stage → Early-Stage Feedback Mode
• **Code repository**: User points to folder/repo path → Repository Review Mode

**For complete papers**, extract: title, abstract, main claims, methodology, experiments, results. Identify paper type: theoretical, methodological, algorithmic, empirical, bridge paper, or application-driven.

**For code repositories**, first explore: read README, scan code structure, find experiment scripts/results, identify the research question and what's implemented.

Step 2: Prior Work Grounding (Critical - All Modes)

This step applies to ALL input types. Grounding in reality is essential for any meaningful feedback.

1. Generate 3-5 search queries based on the research topic: benchmarks/baselines, same problem, related techniques 2. Use WebSearch to find recent arXiv papers and published work 3. Fetch abstracts of 5-10 most relevant papers 4. **Critically synthesize**:

• What specific claims in this paper are already addressed by prior work?
• What are the actual quantitative improvements over recent baselines?
• Are claimed "novelties" actually novel given the literature?
• What gaps truly exist vs. what the authors claim exists?

**Critical mindset**:

• Your job is to verify claims against reality, not accept them at face value
• Most papers overclaim—your review should ground their contributions in what the literature actually shows
• Default to skepticism: Assume claims are overstated until proven otherwise by evidence
• Authors have selection bias toward their own work; you represent the community's interests
• Be the critical voice that ensures published work actually advances the field

Then proceed to mode-specific evaluation.

---

Full Review Mode (Complete Papers)

Step 3: Systematic Evaluation

Evaluate across 7 dimensions (see `references/evaluation-criteria.md`). **Default to skepticism—require strong evidence to score highly.**

| Dimension | Key Questions (Answer with Literature Evidence) | |-----------|---------------| | Originality | Is this truly novel given recent work X, Y, Z? What specific aspects are incremental vs. novel? | | Importance | Why does this problem matter? What's the real-world impact? Who will care? | | Claims Support | Do experiments actually prove the claims? What alternative explanations exist? | | Experimental Soundness | Are baselines from 2023+? Are comparisons fair? What's missing? | | Clarity | Can I reproduce this from the paper? Are claims precisely stated? | | Community Value | Will this change how people work? Or just add noise? | | Prior Work Context | Are comparisons accurate? What recent work (last 2 years) is missing? |

**Evaluation mindset**:

• Start from neutral and require evidence to move up or down
• Compare every claim against what you found in the literature search
• Most papers are incremental—high originality scores are rare
• Weak baselines or missing comparisons are critical flaws, not minor issues

Step 4: Critical Cross-Check Against Literature

Before writing the review, explicitly verify:

1. **Baselines check**: List baselines used in paper. List baselines from your literature search of adjacent papers. What's missing? 2. **Methodology check**: How do 2-3 adjacent papers approach this problem? Does this paper follow similar methodology? If not, why not? 3. **Claims check**: List main claims. For each, cite specific evidence from experiments or proofs. If insufficient, note it. 4. **Citations check**: Which papers from your search are cited? Which are missing? Why? 5. **Novelty check**: List claimed novelties. For each, cite specific prior work that does or doesn't do this.

This step is not optional. Your review must reference specific findings from your literature search.

Step 5: Generate Review

Follow the ICML review form (see `references/review-template.md`):

1. **Summary** - Neutral, factual (should not be disputed by authors) 2. **Claims and Evidence** - Are claims supported? **Compare to what literature shows** 3. **Relation to Prior Work** - Proper context? Missing citations? **List specific missing papers** 4. **Strengths** - Specific and substantive, **compared to standards in adjacent work** 5. **Weaknesses** - Constructive, explain severity, **cite specific literature for comparison** 6. **Questions for Authors** - Numbered, explain impact on evaluation 7. **Minor Issues** - Typos, suggestions 8. **Overall Recommendation** - 1-5 scale with justification **grounded in literature comparison** 9. **Confidence Score** - 1-5 scale

Step 6: Quality Check

• Verify all claims in review are substantiated
• Ensure constructive tone
• Check specificity of strengths/weaknesses
• Confirm questions are actionable

Key Principles

Be Rigorous AND Constructive

Your primary duty is to the re