/collecting-sources

name: collecting-sources
description: Collects research sources with quality evaluation. Use when gathering or finding sources, building a source library, searching for academic papers, or performing RADAR assessment.

Collecting Sources

Phase 1 of two-phase research: discover, evaluate, and store sources with metadata. Synthesis (Phase 2) and fact-checking (Phase 3) are separate phases — this skill ends at handoff.

Everything lands in the task workspace:

work/<task-id>/
├── source_index.md      # human-readable catalogue
└── sources/
    ├── papers/          # academic papers (.txt from CORE/Unpaywall, .pdf from arXiv)
    ├── web/             # web pages (Markdown)
    ├── blogs/           # blog posts (Markdown)
    └── docs/            # documentation (Markdown/HTML)

Every source file has a `.meta.yaml` companion conforming to schema v1.0 — fields, missing-data conventions, and a complete example in `references/source-metadata-schema.md`.

1 — Setup

• `mkdir -p work/<task-id>/sources/{papers,web,blogs,docs}`
• Source IDs are sequential and zero-padded: `src_001`, `src_002`, …
• If `source_index.md` exists from a previous collection, load it for duplicate detection.

2 — Discovery

**Detect the domain** from the research question:

• **Academic**: "paper", "study", "peer-reviewed", "journal"; biomedical (clinical, drug, patient), CS/physics (algorithm, neural network, quantum), social sciences (policy, governance, economics); academic URLs given (arxiv.org, doi.org, pubmed.gov)
• **General web**: "tutorial", "guide", "how-to", documentation focus, no academic terminology

**Academic research** — discovery finds papers and extracts **identifiers** (DOI, arXiv ID, PubMed ID, exact title) for the retrieval chain; do not fetch full text during discovery. Run 3–5 targeted WebSearch queries mixing plain and site-specific forms:

• Biomedical: `site:pubmed.ncbi.nlm.nih.gov [topic]`
• CS/physics: `site:arxiv.org [topic]`
• Any domain: `site:scholar.google.com [topic]`, `[topic] research paper peer-reviewed`

Collect candidate metadata: title, authors, publication date, venue, abstract excerpt, identifiers.

**General web research** — run 3–5 targeted WebSearch queries with alternative phrasings; target authoritative sources (official docs, expert blogs, reputable sites); use site-specific searches when the site is known (`site:docs.python.org async`). Collect URLs, titles, authors, dates.

**Target: 15–25 candidates** from diverse source types, to be filtered to 8–15 on quality.

3 — Retrieval

Before fetching each candidate, check for duplicates: URL match against existing sources, DOI match against `doi` fields in existing `.meta.yaml` files. If already collected, skip and log.

Academic papers — fallback chain

Try in order; record which step succeeded as `retrieval_method`:

**1. CORE API** — always first; 46M full texts as plain text, all domains:

• WebFetch `https://api.core.ac.uk/v3/search/works?q=doi:[DOI]` — or `q=title:"[exact title]"` if no DOI
• If the JSON `fullText` field is non-empty, save it as `.txt` — done. Otherwise continue.
• Rate limit (free tier): 1 batch or 5 single requests per 10 seconds.

**2. Unpaywall MCP** — open-access PDF discovery and extraction:

• `unpaywall_get_fulltext_links(doi=...)` → `best_oa_location.url_for_pdf`; without a DOI, find one via `unpaywall_search_titles(query="exact title")`
• `unpaywall_fetch_pdf_text(doi=..., truncate_chars=50000)` → save as `.txt` (typical papers run 20,000–30,000 chars; surveys 90,000–100,000)
• Rate limit: 100,000 calls/day. If no OA version exists, continue.

**3. Domain repository:**

• **arXiv** (CS, physics, maths): prefer the arXiv MCP server if available; else query CORE by arXiv ID or title; last resort WebFetch `https://export.arxiv.org/pdf/[arxiv_id].pdf` — this stores the binary PDF only (WebFetch cannot extract PDF text). Max ~4 requests/second.
• **PubMed Central** (biomedical): WebFetch `https://pmc.ncbi.nlm.nih.gov/articles/PMC[PMCID]/` — full-article HTML, store as Markdown or HTML.

**4. WebFetch the landing page** — occasionally yields full text as HTML; at minimum, abstract and bibliographic metadata.

**5. Metadata-only** (paywalled, last resort):

• First search for open-access alternatives: `site:arxiv.org [title]`, `site:biorxiv.org [title]`, `[author] [title] pdf`.
• If unavailable: record all metadata from the landing page; set `file_path: "unavailable"` and `content_hash: "unavailable"`; list the attempted strategies in `provenance.notes`; flag "Metadata only (paywalled)" in the index.

**Social sciences caveat**: OA coverage is ~33% vs ~66% for STEM, and SSRN (1.74M preprints) has no API. Expect 50–60% metadata-only rates and say so in the collection summary.

Web pages, blogs, documentation

WebFetch (converts HTML to Markdown). Preserve HTML only when formatting is critical.

Storage

Classify `source_type` (`academic_paper`, `web_page`, `blog`, `documentation`) and store:

| Source | Path | Format | |--------|------|--------| | Paper via CORE/Unpaywall | `sources/papers/src_NNN.txt` | plaintext | | Paper via arXiv direct | `sources/papers/src_NNN.pdf` | pdf | | Paper, metadata-only | no file — `file_path: "unavailable"` | — | | Web page / blog / documentation | `sources/{web,blogs,docs}/src_NNN.md` | markdown (or html) |

Generate the content hash — `shasum -a 256 <file>` → `content_hash: "sha256:HEX"` — and compare against existing hashes to catch duplicate content fetched from different URLs.

4 — Metadata and RADAR

Write `src_NNN.meta.yaml` beside each source file, conforming to `references/source-metadata-schema.md`: core fields, source-type-specific fields, provenance (collected_by, collection_method, collected_at, task_id, notes), and `retrieval_method` for academic papers.

Assess every source on five RADAR dimensions, scored 1–5 with brief notes:

• **Relevance** — how directly it addresses the research question
• **Authority** — credibi