pullmd

PullMD

Self-hosted URL-to-Markdown service for humans and AI agents.

PullMD takes any web URL and returns clean, readable Markdown — no navigation, no ads, no boilerplate. It auto-detects Reddit and Hacker News threads (with full comment trees), uses Cloudflare's native Markdown when available, runs Mozilla Readability + Trafilatura on static HTML, and as a last resort renders JavaScript-heavy pages via headless Chromium (Playwright sidecar) before extracting.

As of v3, PullMD goes beyond web pages: it also converts documents (PDF, Office, EPUB), images, audio, and YouTube videos to Markdown, and emits a leaner, token-efficient body by default. See What's new in v3 below.

It ships as:

• a PWA frontend with raw/rendered and live-frontmatter view toggles, one-tap sharing of the output to other apps (Web Share API), a download button that saves the result as a .md file under the server-suggested name, dark/paper themes, history, archive, share links, and conversion of local HTML files (drag-and-drop on desktop, file picker on desktop and mobile)
• a REST API at GET /api?url=…
• an MCP server at POST /mcp (Streamable-HTTP transport, stateless)
• a Claude Code skill as a downloadable zip

Every conversion gets an 8-hex share id that works as a stable live-endpoint: GET /s/:id returns the cached markdown and re-fetches from the source if older than one hour. Use the share id as a fixed URL that always returns fresh content — useful for subreddit feeds and similar.

What's new in v3

PullMD v3 grows from a web-page reader into a general anything-to-Markdown service for agents, with a leaner default output. Everything beyond plain web extraction is opt-in and degrades gracefully - left unconfigured, v3 handles web pages exactly like v2, just with a cleaner body by default.

• Clean body by default - the Markdown body is now just # Title + content. The source URL, fetch date, and all metadata moved into the YAML frontmatter, so nothing is duplicated and you spend fewer tokens. Reddit posts follow the same rule: subreddit, author, upvotes, and publish date live in the frontmatter (subreddit, author, upvotes, published), not the body. This is the one breaking change: set PULLMD_SOURCE_HEADER=true to restore the old inline header, and use PULLMD_FRONTMATTER_FIELDS to trim which fields are emitted. See MIGRATION.md.
• Documents → Markdown - PDF, Word, PowerPoint, Excel, EPUB and more, by URL or upload (POST /api/file, drag-and-drop in the PWA).
• High-quality PDF tables (OCR) - an opt-in, vendor-neutral OCR tier (?pdf=ocr) for table-grade PDF conversion, with automatic fallback to the free path.
• Images & audio → Markdown - opt-in captioning and transcription via any OpenAI-compatible or local model; runs inside pullmd, no extra container required.
• YouTube transcripts - title, description and transcript with clickable timecodes, no API key required.
• Richer frontmatter - extraction source, quality, and (for media/OCR) model + token/page usage for cost tracking, plus a configurable field allowlist.

Self-hosters upgrading from v2.x: the clean-body change is the only breaking one - MIGRATION.md has the one-line opt-out. Everything else is additive.

Added in the 3.x line since then:

• Hacker News pipeline (3.1) - items, comment permalinks and listings through a purpose-built converter, plus Web Share and an instant frontmatter toggle in the PWA.
• X-Transcript-Status (3.2) - tells a transient YouTube rate-limit apart from a genuinely missing transcript.
• SSRF protection (3.3) - private, loopback, link-local, CGNAT and cloud-metadata targets are rejected by default, on every fetch path and every redirect hop.
• Query-scoped extraction (3.4) - ?query= returns only the sections relevant to a question, with a max_tokens budget.
• Site recipes opened up (3.5/3.6) - JSON-LD-to-frontmatter, a contributor guide, and select.content so a recipe can name the article body outright.
• Coverage guard (3.7) - recovers pages where extraction kept only a sliver of the body; see PULLMD_COVERAGE_GUARD.
• Account controls (3.8) - a non-admin can clear entries from their own history, self-registration can be closed with PULLMD_ALLOW_SIGNUP, and scripts/admin.js create-user creates accounts from the shell.
• Download button (3.9) - the PWA saves a result as a .md file, named by the server via X-Suggested-Filename and optionally date-prefixed.
• Sidecar health endpoint (3.10) - GET /api/status answers 503 when a configured sidecar stops responding, so a dead renderer shows up as an alert instead of quietly degrading extraction.
• Configurable cache retention (3.11) - PULLMD_CACHE_RETENTION_DAYS sets how long cache rows and share links live; default 90 days, 0 keeps them forever.

Quick start

Pre-built multi-arch images (linux/amd64, linux/arm64) live on Docker Hub. Drop the compose file somewhere and run:

mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# → http://localhost:3000

That's it. No .env needed: every variable has a sensible default and PullMD listens on port 3000. Add a .env next to the compose file to override anything (see Configuration).

docker-compose.yml (zero-config, abridged)

services:
  pullmd:
    image: aeternalabshq/pullmd:latest
    container_name: pullmd
    restart: unless-stopped
    ports:
      - "${PORT:-3000}:3000"
    environment:
      - PUBLIC_URL=${PUBLIC_URL:-http://localhost:${PORT:-3000}}
      - TRAFILATURA_URL=http://trafilatura:8001/extract
      - PLAYWRIGHT_URL=http://playwright:8002/render
      - MARKITDOWN_URL=http://markitdown:8003/convert
      - CACHE_DB=/data/cache.db
    volumes:
      - ./data:/data
    networks:
      - pullmd-internal
    depends_on:
      - trafilatura
      - playwright
      - markitdown

  trafilatura:
    image: aeternalabshq/pullmd-trafilatura:latest
    container_name: pullmd-trafilatura
    restart: unless-stopped
    networks:
      - pullmd-internal

  playwright:
    image: aeternalabshq/pullmd-playwright:latest
    container_name: pullmd-playwright
    restart: unless-stopped
    networks:
      - pullmd-internal

  markitdown:
    image: aeternalabshq/pullmd-markitdown:latest
    container_name: pullmd-markitdown
    restart: unless-stopped
    mem_limit: ${MARKITDOWN_MEM_LIMIT:-1g}
    networks:
      - pullmd-internal

networks:
  pullmd-internal:
    driver: bridge

Abridged for readability — the docker-compose.yml in the repo additionally passes every optional .env variable through to the containers (Reddit credentials, auth, media/OCR keys, YouTube options, output shaping). Use the curl -O command above rather than copying this block, or .env overrides beyond the basics won't reach the containers.

Note: the Playwright sidecar adds ~3.7 GB to your image cache (Chromium + Firefox + WebKit binaries from the official Playwright base image). It's optional — leave PLAYWRIGHT_URL unset and the playwright service block off, and PullMD silently degrades to static extraction with a fallback note in the metadata.

Note: the MarkItDown sidecar is optional. Leave MARKITDOWN_URL unset and remove the markitdown service block to disable document conversion. Web-page URLs always work without it.

Mirror on GHCR: ghcr.io/aeternalabshq/{pullmd,pullmd-trafilatura,pullmd-playwright,pullmd-markitdown}. Replace the image: lines if you prefer GitHub's registry.

Behind Traefik

For deployments behind Traefik with TLS, use docker-compose.traefik.yml instead. Same images, but with Traefik labels and the proxy external network. Set HOST_DOMAIN in .env:

curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.traefik.yml
echo "HOST_DOMAIN=pullmd.example.com" > .env
docker compose -f docker-compose.traefik.yml up -d

Local development (no Docker)

git clone https://github.com/AeternaLabsHQ/pullmd.git
cd pullmd
npm install
npm start             # http://localhost:3000
npm test              # node --test

Configuration

All variables go in .env (copy from .env.example):

v3.0.0 output format change: the markdown body is clean by default - just # Title followed by content. The source URL, fetch date, and all extraction metadata remain in the YAML frontmatter unchanged - the body no longer duplicates them. Set PULLMD_SOURCE_HEADER=true to restore the old inline header. Use PULLMD_FRONTMATTER_FIELDS to pick which frontmatter fields are emitted (handy for trimming tokens in agent pipelines).