/job-scraper

name: scrape
description: >
  Finds new job postings matching your profile via installed portal-search CLIs
  (LinkedIn, local job boards, and any skills added with /add-portal). Deduplicates
  across runs. Triggers on: job scrape, find jobs, search jobs, new jobs, job search,
  scrape jobs, /scrape
allowed-tools: Read, Write, Edit, Glob, Grep, Bash(bun --version), Bash(bun run .agents/skills/*/cli/src/cli.ts *), WebFetch, WebSearch, Agent, AskUserQuestion

Job Scraper

---

How It Works

This skill searches job portals using the **installed portal-search CLIs** in `.agents/skills/` (plus WebSearch as a fallback), using queries from your profile. It deduplicates against previously seen jobs and the application tracker, and presents new matches with a quick fit assessment.

Invocation

The user triggers this skill by saying things like:

• "Find new jobs"
• "Scrape for jobs"
• "Any new positions?"
• "/scrape"

Optional arguments:

• A focus area, e.g. "/scrape data science" or "/scrape geophysics"
• "broad" to run all search categories, e.g. "/scrape broad"
• "health" to run the portal health check only (Step 4.75), without searching, deduplicating, or presenting jobs - e.g. "/scrape health", or "/scrape health jobnet" to probe one portal even if disabled

---

Execution Steps

Step 0: Load State

1. Read `job_scraper/seen_jobs.json` (create if missing - start with `{"seen": {}}`) 2. Read `job_search_tracker.csv` to extract already-applied companies+roles 3. Read `search-queries.md` (this directory) for the search strategy

Step 1: Search

Read `search-queries.md` (this directory) for the search strategy. By default, run the top 3 priority query categories. If the user said "broad", run all categories. If the user specified a focus area (e.g. "data science"), prioritize queries from that category.

**Use the installed CLI tools as the primary search mechanism.** Fall back to `WebSearch` only for portals that do not have a CLI skill, or if `bun` is unavailable on the system.

1a. Check bun availability

bun --version

If this fails (bun not installed), skip to **1c (WebSearch fallback)** for all portals and note the fallback in the Step 5 output.

1b. Run CLI tools (primary — run these in parallel where possible)

Discover all installed portal CLI skills by reading every `SKILL.md` found under `.agents/skills/*/SKILL.md`. Each file documents that portal's exact CLI flags and usage examples. **Use each portal's own documented interface — do not guess flags.** This approach automatically includes any new portals added via `/add-portal` without requiring changes to this file.

**Honor the `enabled` toggle.** A portal is enabled unless its `SKILL.md` frontmatter sets `enabled: false` (a missing key means enabled — the default). Skip each disabled portal and record it for the Step 5 summary. A fork can thus keep a portal installed but sit out a run without deleting its directory.

For each **enabled** portal skill:

1. Read its `SKILL.md` to find the correct `bun run …` invocation and supported flags. 2. Translate the query terms from `search-queries.md` into that portal's flag format (e.g. `--key`, `--search-string`, `--query`, filter codes — whatever the portal's SKILL.md specifies). 3. Scope to the last 14 days using the portal's supported recency flag (`--jobage`, `--since <YYYY-MM-DD>`, `--order PublicationDate`, etc. — as documented per portal). 4. Cap results to ~20 per call using the portal's limit flag. 5. Use `--format json` for machine-readable output.

Run all portal CLI calls in parallel where possible using the Agent tool. Collect all `results` arrays into a single pool for Step 2, keeping each result tagged with its source portal skill (for Step 2 `detail` lookups).

If a CLI tool exits with a non-zero code, log the error message and continue — do not abort the whole search.

1c. WebSearch fallback

Use `WebSearch` for:

• Portals listed in `search-queries.md` that do **not** have a corresponding directory under `.agents/skills/`
• Any portal whose CLI fails at runtime
• When bun is unavailable (Step 1a failed)

Use the site-specific query strings from `search-queries.md` directly as WebSearch queries for these portals.

Step 2: Fetch & Parse

For each promising result from Step 1:

**From CLI results:** Search output already includes title, company, location, date, and URL. For jobs worth a deeper look, fetch full detail with that portal's `detail` command (see its SKILL.md — do not guess flags) to extract **key requirements**, **application deadline**, and a brief description snippet.

**From WebSearch results:** Use `WebFetch` on the posting URL and extract the same fields manually. If it returns HTTP 403, retry with browser headers via curl per `.claude/skills/job-application-assistant/09-web-research.md` before giving up — most bank and corporate sites reject WebFetch's user agent while serving browsers normally.

**Store a URL that actually resolves to the posting.** A listing-page URL with a `#fragment` appended (`.../jobs/ciso/#ikerian`) is not a posting: it fetches fine and returns unrelated job titles, which makes every later `/rank` and `/apply` run fail on that entry. When WebSearch only yields a listing page, search the employer's own careers site for the role and store that URL instead, or drop the candidate rather than saving a fragment link.

For every candidate:

• Skip if the URL or company+title combo already exists in `seen_jobs.json`
• Skip if the company+role already appears in `job_search_tracker.csv`

Step 2.5: Mass-Posting Detection (within this run)

A distribution pattern worth flagging to the user as a caution signal, not as an accusation against the employer - it describes how a listing is being distributed, not a verdict on whether the company is legitimate. It alone proves nothing is wrong (companies do legitimately hire the same role across several cities); flag it so the user can factor it