ref-downloader

ref-downloader

Stop losing an afternoon to chasing dozens of reference PDFs by hand. One DOI in, every reference PDF out — using your existing institutional access.

中文完整文档 / Full Chinese version

Status: beta (v0.4.1). Windows + Microsoft Edge verified path. macOS / Linux / Chromium untested. Expect rough edges around supplementary downloads and publisher-site changes. PR-worthy issues welcome.

Heads up — not a paywall bypass. ref-downloader uses your institutional access. If your university or organization subscribes to a journal, those refs work. If they don't, those refs become manual_pending for you to follow up on by hand.

Demo (30-second console preview)

$ python run_ref_downloader.py 10.1021/jacs.5c05017

=== Ref Downloader Wrapper ===
DOI:         10.1021/jacs.5c05017
PROJECT:     jacs.5c05017
Config:      config.example.toml + config.local.toml

>>> extract_refs.py
  Title: Designing Natural Cell-Inspired Heme-Spurred Membrane...
  References found: 38

>>> validate_refs.py
  Total: 38  Verified: 38  Failed: 0  No DOI: 0

>>> download_refs.py
  [ 1] downloaded (842 KB)        Lee2016_NatEnergy.pdf
  [ 2] downloaded (1.2 MB)        Wang2018_AdvMater.pdf
  [ 3] manual_pending (auth_redirect)
  [ 4] downloaded (655 KB)        Chen2019_JACS.pdf
  [ 5] failed (challenge_timeout)
  [ 6] ignored (ignored_institution_access)
  ... 31 more refs processed ...
  [38] downloaded (956 KB)        Park2024_JElectrochemSoc.pdf

========== Download report ==========
Total references:  38
Main PDFs:         33 downloaded · 3 manual_pending · 1 failed · 1 ignored
SI files:          12 captured
PDFs land in:      ./jacs.5c05017_refs/jacs.5c05017/
=====================================

Contents

• What you get
• Why not Zotero, scihub, or generic scrapers?
• Quick start
• Requirements
• Install
• Usage examples
• Configuration
• Architecture
• Supported publishers
• Known limitations
• Contributing
• Security
• License

What you get

• Paywalled refs work without setup. Drives your real Microsoft Edge profile, so any institutional login already in your browser carries through. No API keys, no proxies, no reverse engineering.
• One DOI in, every reference PDF out. Crossref-driven extraction + 17+ publisher-specific download paths (Wiley PDFDirect, Elsevier viewer, AIP loading-page wait — see per-publisher reliability tier), not generic scraping.
• Two routing modes from an agent (v0.4.1+). Mode A: hand the agent one paper, get all of its references. Mode B: hand the agent a custom list — DOIs / titles / arXiv-PMID IDs / BibTeX / abstract queries like "Wang 2024 Nature Energy papers" — and it resolves to DOIs and downloads them. See SKILL.md.
• You always know which refs failed and why. download_report.csv gives every ref a status + reason (manual_pending (auth_redirect), failed (challenge_timeout), ignored); events.jsonl keeps the per-ref event trace.
• Pick up where you left off after a VPN drop, browser crash, or Ctrl+C. State persists per project; rerunning skips already-downloaded refs and retries only the failures.

Why not Zotero, scihub, or generic scrapers?

• vs. Zotero's Find Available PDF — walks one paper at a time and silently gives up at SSO redirects. ref-downloader walks the whole reference list at once and treats SSO as a configurable step instead of a dead end.
• vs. scihub-style tools — don't carry your institutional license, so paywalled refs you legitimately have access to just fail. ref-downloader uses your authenticated browser session, so subscriptions you already pay for actually count.
• vs. generic web scrapers — don't know Wiley needs PDFDirect, Elsevier needs a viewer click, or AIP serves a Chinese loading page first. ref-downloader has 17+ publisher-specific paths plus Elsevier popup state machine + --auto mode retry queue (manual-pending refs get a second async attempt 60s later, hot-session preserved).
• vs. raw Playwright — gets blocked on Cloudflare / Radware / Turnstile-heavy sites. Set REF_DOWNLOADER_BROWSER=cloak to swap in cloakbrowser's stealth Chromium with humanized input — no code changes, same pipeline. See Configuration.

Quick start

The skill is self-contained under skills/ref-downloader/. Pick the install path for your agent framework:

git clone https://github.com/ltczding-gif/ref-downloader.git

# Pick ONE install destination for your agent framework:
#   Claude Code:        cp -r ref-downloader/skills/ref-downloader ~/.claude/skills/
#   Codex CLI:          cp -r ref-downloader/skills/ref-downloader ~/.codex/skills/
#   Copilot CLI / VSC:  cp -r ref-downloader/skills/ref-downloader .github/skills/
#   Project-local:      cp -r ref-downloader/skills/ref-downloader .agents/skills/

cd ~/.claude/skills/ref-downloader     # or wherever you copied it
pip install playwright pymupdf
playwright install msedge
cp config.example.toml config.local.toml      # then set [crossref].mailto

# In your agent: just describe the task; the skill triggers via its description.
# Direct CLI for testing: python scripts/run_ref_downloader.py 10.1021/jacs.5c05017

What you'll see: 30–80 refs discovered for a typical chemistry/physics paper, then a mix of downloaded (refs your institution covers), manual_pending (SSO bounce or paywall), and occasional failed (publisher quirk). Run on a DOI from a journal your institution actually subscribes to for the highest hit rate. Details below.

Requirements

• OS: Windows 10/11 (verified). macOS / Linux untested — PRs welcome.
• Browser: Microsoft Edge (Stable channel). The script claims your persistent Edge profile, so close all Edge windows before running.
• Python: 3.11 or newer (uses stdlib tomllib).
• Optional: A Zotero installation (auto-detects DOI from a PDF's filename via Zotero's SQLite database — much faster than text extraction).
• Optional: PyMuPDF (pip install pymupdf) for DOI extraction from PDF text when Zotero lookup is unavailable.

Install

As an agent skill (recommended)

Pick the install path for your agent framework:

Then install Python prereqs INSIDE the copied skill folder (the skill protocol doesn't manage Python deps):

cd ~/.claude/skills/ref-downloader            # or wherever you copied it
pip install playwright pymupdf                # or use the source's requirements.txt
playwright install msedge

cp config.example.toml config.local.toml
# Edit config.local.toml — at minimum set [crossref].mailto.
# Windows: notepad config.local.toml
# macOS / Linux: $EDITOR config.local.toml   (or vim / nano / code / ...)

As a Python tool (for developers)

If you want to hack on the code, the skill folder is a runnable Python project:

git clone https://github.com/ltczding-gif/ref-downloader.git
cd ref-downloader

pip install -r requirements.txt -r requirements-dev.txt
playwright install msedge

cp skills/ref-downloader/config.example.toml skills/ref-downloader/config.local.toml
# Edit config.local.toml — at minimum set [crossref].mailto.

# Run the offline test suite
python -m pytest tests/ -v

# Run the tool directly
python skills/ref-downloader/scripts/run_ref_downloader.py 10.1021/jacs.5c05017

Usage examples

(After install — paths assume the skill is at <SKILL_DIR>, e.g. ~/.claude/skills/ref-downloader/. In source, <SKILL_DIR> = skills/ref-downloader/.)

Input: a DOI

python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017

Default output: <cwd>/jacs.5c05017_refs/jacs.5c05017/

Input: a local PDF (with DOI in metadata or in PDF text)

python <SKILL_DIR>/scripts/run_ref_downloader.py "C:\path\to\your_paper.pdf"

Default output: <pdf_dir>/your_paper_refs/<doi-derived-name>/

Custom output directory

python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017 --output-dir refs/

Non-interactive (CI / batch)

python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017 --yes --auto

Alternate config file

python <SKILL_DIR>/scripts/run_ref_downloader.py 10.1021/jacs.5c05017 --config ./alt.toml

Configuration

All configuration lives in config.local.toml (gitignored). Copy config.example.toml to bootstrap.

Environment variables override file values:

See skills/ref-downloader/config.example.toml for full documentation.

Alternative backend: CloakBrowser (optional, for Cloudflare-heavy sites)

What it is. CloakBrowser is a third-party Python package by CloakHQ (MIT-licensed, available on PyPI as cloakbrowser). It ships a patched Chromium build with source-level anti-fingerprint changes designed to look like a normal browser to common bot-detection layers (Cloudflare Turnstile, Radware, DataDome, FingerprintJS, etc). Its launch_persistent_context_async() API is intentionally compatible with Playwright's — that's what lets ref-downloader swap backends with a single env var instead of rewriting the download flow.