/evaluating-paper-relevance

name: Evaluating Paper Relevance
description: Two-stage paper screening - abstract scoring then deep dive for specific data extraction
when_to_use: After literature search returns results. When need to determine if paper contains specific data. When screening papers for relevance. When extracting methods, results, data from papers.
version: 1.0.0

<!-- ╔══════════════════════════════════════════════════════════════╗ ║ 本文件为开源 Skill 原始文档，收录仅供学习与研究参考 ║ ║ CoPaper.AI 收集整理 | https://copaper.ai ║ ╚══════════════════════════════════════════════════════════════╝

来源仓库: https://github.com/kthorn/research-superpower 项目名称: research-superpower 开源协议: MIT License 收录日期: 2026-04-02

声明: 本文件版权归原作者所有。此处收录旨在为社会科学实证研究者 提供 AI Agent Skills 的集中参考。如有侵权，请联系删除。 -->

Evaluating Paper Relevance

Overview

Two-stage screening process: quick abstract scoring followed by deep dive into promising papers.

**Core principle:** Precision over breadth. Find papers that actually contain the specific data/methods user needs, not just topically related papers.

When to Use

Use this skill when:

• Have list of papers from search
• Need to determine which papers have relevant data
• User asks for specific information (measurements, protocols, datasets, etc.)
• Screening papers one-by-one
• Any research domain (medicinal chemistry, genomics, ecology, computational methods, etc.)

Choosing Your Approach

**Small searches (<50 papers):**

• Manual screening with progress reporting
• Use papers-reviewed.json + SUMMARY.md only
• No helper scripts needed
• Report progress to user for every paper

**Large searches (50-150 papers):**

• Consider helper scripts (screen_papers.py + deep_dive_papers.py)
• Use Progressive Enhancement Pattern (see Helper Scripts section)
• Create README.md with methodology
• May want TOP_PRIORITY_PAPERS.md for quick reference
• Use richer JSON structure (evaluated-papers.json categorized by relevance)
• Consider using subagent-driven-review skill for parallel screening

**Very large searches (>150 papers):**

• Definitely use helper scripts with Progressive Enhancement Pattern
• Create full auxiliary documentation suite (README.md, TOP_PRIORITY_PAPERS.md)
• Consider citation network analysis
• Plan for multi-week timeline
• Strongly consider subagent-driven-review skill for parallelization
• May need multiple consolidation checkpoints

Two-Stage Process

Stage 1: Abstract Screening (Fast)

**Goal:** Quickly identify promising papers

**Score 0-10 based on:**

• **Keywords match (0-3 points)**: Does abstract mention key terms relevant to the query?
• **Data type match (0-4 points)**: Does it mention the specific information user needs?
• Examples: measurements (IC50, expression levels, population sizes), protocols, datasets, structures, sequences, code
• **Specificity (0-3 points)**: Is it specific to user's question or just general background/review?

**Decision rules:**

• Score < 5: Skip (not relevant)
• Score 5-6: Note in summary as "possibly relevant" but skip for now
• Score ≥ 7: Proceed to Stage 2 (deep dive)

**IMPORTANT: Report to user for EVERY paper:**

📄 [N/Total] Screening: "Paper Title"
   Abstract score: 8 → Fetching full text...

or

📄 [N/Total] Screening: "Paper Title"
   Abstract score: 4 → Skipping (insufficient relevance)

**Never screen silently** - user needs to see progress happening

Stage 2: Deep Dive (Thorough)

**Goal:** Extract specific data/methods from promising papers

1. Check ChEMBL (for medicinal chemistry papers)

**If paper describes medicinal chemistry / SAR data:**

Use `skills/research/checking-chembl` to check if paper is in ChEMBL database:

curl -s "https://www.ebi.ac.uk/chembl/api/data/document.json?doi=$doi"

**If found in ChEMBL:**

• Note ChEMBL ID and activity count in SUMMARY.md
• Report to user: "✓ ChEMBL: CHEMBL3870308 (45 data points)"
• Structured SAR data available without PDF parsing

**Continue to full text fetch for context, methods, discussion.**

2. Fetch Full Text

**Try in order:**

**A. PubMed Central (free full text):**

# Check if available in PMC
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pmc&term=PMID[PMID]&retmode=json"

# If found, fetch full text XML via API
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pmc&id=PMCID&rettype=full&retmode=xml"

# Or fetch HTML directly (note: use pmc.ncbi.nlm.nih.gov, not www.ncbi.nlm.nih.gov/pmc)
curl "https://pmc.ncbi.nlm.nih.gov/articles/PMCID/"

**B. DOI resolution:**

# Try publisher link
curl -L "https://doi.org/10.1234/example.2023"
# May hit paywall - check response

**C. Unpaywall (MANDATORY if paywalled):** **CRITICAL: If step B hits a paywall, you MUST immediately try Unpaywall before giving up.**

Use `skills/research/finding-open-access-papers` to find free OA version:

curl "https://api.unpaywall.org/v2/DOI?email=USER_EMAIL"
# Often finds versions in repositories, preprint servers, author copies
# IMPORTANT: Ask user for their email if not already provided - do NOT use claude@anthropic.com

Report to user:

⚠️  Paper behind paywall, checking Unpaywall...
✓ Found open access version at [repository/preprint server]

or

⚠️  Paper behind paywall, checking Unpaywall...
✗ No open access version available - continuing with abstract only

**D. Preprints (direct):**

• Check bioRxiv: `https://www.biorxiv.org/content/10.1101/{doi}`
• Check arXiv (for computational papers)

**If full text unavailable AFTER trying Unpaywall:**

• Note in SUMMARY.md: "⚠️ Full text behind paywall - no OA version found via Unpaywall"
• Continue with abstract-only evaluation (limited)

**CRITICAL: Do NOT skip Unpaywall check. Many paywalled papers have free versions in repositories.**

2. Scan for Relevant Content

**Focus on sections:**

• **Methods**: Experimental procedures, protocols
• **Results**: Data tables, figures, me