/building-screening-rubrics

name: Building Paper Screening Rubrics
description: Collaboratively build and refine paper screening rubrics through brainstorming, test-driven development, and iterative feedback
when_to_use: Starting new literature search. When automated screening misclassifies papers. When need to screen 50+ papers efficiently. Before creating screening scripts. When rescreening papers with updated criteria.
version: 1.0.0

<!-- ╔══════════════════════════════════════════════════════════════╗ ║ 本文件为开源 Skill 原始文档，收录仅供学习与研究参考 ║ ║ CoPaper.AI 收集整理 | https://copaper.ai ║ ╚══════════════════════════════════════════════════════════════╝

来源仓库: https://github.com/kthorn/research-superpower 项目名称: research-superpower 开源协议: MIT License 收录日期: 2026-04-02

声明: 本文件版权归原作者所有。此处收录旨在为社会科学实证研究者 提供 AI Agent Skills 的集中参考。如有侵权，请联系删除。 -->

Building Paper Screening Rubrics

Overview

**Core principle:** Build screening rubrics collaboratively through brainstorming → test → refine → automate → review → iterate.

Good rubrics come from understanding edge cases upfront and testing on real papers before bulk screening.

When to Use

Use this skill when:

• Starting a new literature search that will screen 50+ papers
• Current rubric misclassifies papers (false positives/negatives)
• Need to define "relevance" criteria before automated screening
• Want to update criteria and re-screen cached papers
• Building helper scripts for evaluating-paper-relevance

**When NOT to use:**

• Small searches (<20 papers) - manual screening is fine
• Rubric already works well - no need to rebuild
• One-off exploratory searches

Two-Phase Process

Phase 1: Collaborative Rubric Design

Step 1: Brainstorm Relevance Criteria

**Ask domain-agnostic questions to understand what makes papers relevant:**

**Core Concepts:**

• "What are the key terms/concepts for your research question?"
• Examples: specific genes, proteins, compounds, diseases, methods, organisms, theories
• "Are there synonyms or alternative names?"
• "Any terms that should EXCLUDE papers (false positives)?"

**Data Types & Artifacts:**

• "What type of information makes a paper valuable?"
• Quantitative measurements (IC50, expression levels, population sizes, etc.)
• Protocols or methods
• Datasets with accessions (GEO, SRA, PDB, etc.)
• Code or software
• Chemical structures
• Sequences or genomes
• Theoretical models
• "Do you need the actual data in the paper, or just that such data exists?"

**Paper Types:**

• "What types of papers are relevant?"
• Primary research only?
• Reviews or meta-analyses?
• Methods papers?
• Clinical trials?
• Preprints acceptable?

**Relationships & Context:**

• "Are papers about related/analogous concepts relevant?"
• Example: "If studying protein X, are papers about homologs/paralogs relevant?"
• Example: "If studying compound A, are papers about analogs/derivatives relevant?"
• Example: "If studying disease X, are papers about related diseases relevant?"
• "Does the paper need to be ABOUT your topic, or just MENTION it?"
• "Are synthesis/methods papers relevant even without activity data?"

**Edge Cases:**

• "Can you think of papers that would LOOK relevant but aren't?"
• "Papers that might NOT look relevant but actually are?"

**Document responses in screening-criteria.json**

Step 2: Build Initial Rubric

**Based on brainstorming, propose scoring logic:**

Scoring (0-10):

Keywords Match (0-3 pts):
  - Core term 1: +1 pt
  - Core term 2 OR synonym: +1 pt
  - Related term: +1 pt

Data Type Match (0-4 pts):
  - Measurement type (IC50, Ki, EC50, etc.): +2 pts
  - Dataset/code available: +1 pt
  - Methods described: +1 pt

Specificity (0-3 pts):
  - Primary research: +3 pts
  - Methods paper: +2 pts
  - Review: +1 pt

Special Rules:
  - If mentions exclusion term: score = 0

Threshold: ≥7 = relevant, 5-6 = possibly relevant, <5 = not relevant

**Present to user and ask:** "Does this logic match your expectations?"

**Save initial rubric to screening-criteria.json:**

{
  "version": "1.0.0",
  "created": "2025-10-11T15:30:00Z",
  "keywords": {
    "core_terms": ["term1", "term2"],
    "synonyms": {"term1": ["alt1", "alt2"]},
    "related_terms": ["related1", "related2"],
    "exclusion_terms": ["exclude1", "exclude2"]
  },
  "data_types": {
    "measurements": ["IC50", "Ki", "MIC"],
    "datasets": ["GEO:", "SRA:", "PDB:"],
    "methods": ["protocol", "synthesis", "assay"]
  },
  "scoring": {
    "keywords_max": 3,
    "data_type_max": 4,
    "specificity_max": 3,
    "relevance_threshold": 7
  },
  "special_rules": [
    {
      "name": "scaffold_analogs",
      "condition": "mentions target scaffold AND (analog OR derivative)",
      "action": "add 3 points"
    }
  ]
}

Phase 2: Test-Driven Refinement

Step 1: Create Test Set

**Do a quick PubMed search to get candidate papers:**

# Search for 20 papers using initial keywords
curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?db=pubmed&term=YOUR_QUERY&retmax=20&retmode=json"

**Fetch abstracts for first 10-15 papers:**

curl "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?db=pubmed&id=PMID1,PMID2,...&retmode=xml&rettype=abstract"

**Present abstracts to user one at a time:**

Paper 1/10:
Title: [Title]
PMID: [12345678]
DOI: [10.1234/example]

Abstract:
[Full abstract text]

Is this paper RELEVANT to your research question? (y/n/maybe)

**Record user judgments in test-set.json:**

{
  "test_papers": [
    {
      "pmid": "12345678",
      "doi": "10.1234/example",
      "title": "Paper title",
      "abstract": "Full abstract text...",
      "user_judgment": "relevant",
      "timestamp": "2025-10-11T15:45:00Z"
    }
  ]
}

**Continue until have 5-10 papers with clear judgments**

Step 2: Score Test Papers with Rubric

**Apply rubric to each test paper:**

for paper in test_papers:
    score