gh-scraper

name: gh-scraper
description: 'Fetches all GitHub API data for a repo (REST + GraphQL) in two sequential groups with parallel calls inside each; writes raw JSONL for oss:repo-warden axis scorers. TRIGGER when: spawned by /oss:analyse (vitality mode) to fetch raw GitHub data. NOT for axis scoring or report generation. NOT for direct user invocation.'
tools: Bash
model: haiku
effort: medium
color: cyan

<role>

Data collection agent, /oss:analyse (vitality mode). Fetches required GitHub data (REST + GraphQL) in two sequential groups with parallel calls inside each, writes raw JSONL, returns path. Scoring: 3 parallel oss:repo-warden instances.

NOT for axis scoring — oss:repo-warden owns all axis scoring. NOT for report formatting, terminal summary, or adversarial review — /oss:analyse (vitality mode) Steps 4–7 own those.

</role>

<inputs>

The caller supplies session sentinels under `${TMPDIR:-/tmp}`; Step 1 reads these values only, never shell-variable placeholders from the prompt:

• `analyse-gh-owner-${CSID}` — `GH_OWNER`, GitHub owner or org
• `analyse-gh-repo-${CSID}` — `GH_REPO`, GitHub repository name
• `vitality-data-file-${CSID}` — `DATA_FILE`, output path for raw JSONL (one JSON object per line)

Prompt asks for anything but a GitHub fetch (code review, bug hunt, Q&A) → run nothing, return only `{"status":"error","file":"","error":"out of scope: gh-scraper only fetches GitHub data"}`. Missing, empty or unsubstituted sentinel values → Step 1 returns an error envelope before auth, API calls or output writes. Never review, probe or rewrite supplied code.

</inputs>

<workflow>

Step 1 — Setup

Load `GH_OWNER`, `GH_REPO`, `DATA_FILE` from the caller's session sentinels, then compute time anchors:

set -e
export CSID="${CLAUDE_CODE_SESSION_ID:-$PPID}"
export GH_HOST=github.com
DATA_FILE=""
SETUP_ERROR="setup failed — see stderr"
trap 'python -c '\''import json,sys; print(json.dumps({"status":"error","file":sys.argv[1],"error":sys.argv[2]}))'\'' "$DATA_FILE" "$SETUP_ERROR"' ERR
IFS= read -r GH_OWNER < "${TMPDIR:-/tmp}/analyse-gh-owner-${CSID}" 2>/dev/null || GH_OWNER=""
IFS= read -r GH_REPO < "${TMPDIR:-/tmp}/analyse-gh-repo-${CSID}" 2>/dev/null || GH_REPO=""
IFS= read -r DATA_FILE < "${TMPDIR:-/tmp}/vitality-data-file-${CSID}" 2>/dev/null || DATA_FILE=""
SETUP_ERROR="missing, empty or unsubstituted scrape sentinel — rerun /oss:analyse vitality"
[ -n "$GH_OWNER" ] && [ -n "$GH_REPO" ] && [ -n "$DATA_FILE" ] || { echo "[gh-scraper] ERROR: $SETUP_ERROR" >&2; false; }
case "$GH_OWNER/$GH_REPO/$DATA_FILE" in
    *'$'*|*'<'*|*'>'*) echo "[gh-scraper] ERROR: $SETUP_ERROR" >&2; false ;;
esac
SETUP_ERROR="setup failed — see stderr"
# cross-platform: macOS BSD vs GNU/Linux
if date -v-1d +%Y-%m-%d 2>/dev/null | grep -q '^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]$'; then
    # BSD date (-v offset) — verify output shape, not just exit code
    CUTOFF_30D=$(date -u -v-30d +%Y-%m-%dT%H:%M:%SZ)    # timeout: 5000
    CUTOFF_90D=$(date -u -v-90d +%Y-%m-%dT%H:%M:%SZ)    # timeout: 5000
    CUTOFF_180D=$(date -u -v-180d +%Y-%m-%dT%H:%M:%SZ)  # timeout: 5000
    CUTOFF_3Y=$(date -u -v-1095d +%Y-%m-%d)              # timeout: 5000
else
    CUTOFF_30D=$(date -u -d '30 days ago' +%Y-%m-%dT%H:%M:%SZ)    # timeout: 5000
    CUTOFF_90D=$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%SZ)    # timeout: 5000
    CUTOFF_180D=$(date -u -d '180 days ago' +%Y-%m-%dT%H:%M:%SZ)  # timeout: 5000
    CUTOFF_3Y=$(date -u -d '1095 days ago' +%Y-%m-%d)             # timeout: 5000
fi

# auth preflight — fail fast before any API calls
gh auth status --hostname github.com 2>/dev/null || { SETUP_ERROR="not authenticated — run gh auth login --hostname github.com"; echo "[gh-scraper] ERROR: $SETUP_ERROR" >&2; false; }  # timeout: 6000

# rate-limit preflight — warn if <80 calls remain (~80 needed for full scrape)
for BUCKET in core search graphql; do
    case "$BUCKET" in
        core) RATE_REMAINING=$(gh api rate_limit --jq '.resources.core.remaining' 2>/dev/null || echo unknown) ;;  # timeout: 6000
        search) RATE_REMAINING=$(gh api rate_limit --jq '.resources.search.remaining' 2>/dev/null || echo unknown) ;;  # timeout: 6000
        graphql) RATE_REMAINING=$(gh api rate_limit --jq '.resources.graphql.remaining' 2>/dev/null || echo unknown) ;;  # timeout: 6000
    esac
    RATE_MIN=80
    [ "$BUCKET" = search ] && RATE_MIN=10
    case "$RATE_REMAINING" in
        ''|*[!0-9]*) ;;
        *)
            if [ "$RATE_REMAINING" -lt "$RATE_MIN" ]; then
                RATE_RESET=$(gh api rate_limit --jq ".resources.$BUCKET.reset | todate" 2>/dev/null || echo 'unknown time')
                echo "[gh-scraper] WARN: only $RATE_REMAINING $BUCKET API budget remaining — results may be incomplete; reset at $RATE_RESET"  # timeout: 6000
            fi
            ;;
    esac
done

# DATA_FILE set by caller — do NOT inject PID suffix; breaks handoff (vitality.md reads original path)
echo "[gh-scraper] analysing $GH_OWNER/$GH_REPO"  # timeout: 5000
mkdir -p "$(dirname "$DATA_FILE")"  # timeout: 5000
# fresh snapshot per run: Group 2 appends, Step 4 merges Group 1 in — no stale same-day record survives
printf '' > "$DATA_FILE"
# persist across Bash calls — Check 41: fresh shell per call; trailing \n required, `read` fails without it
printf '%s\n' "$GH_OWNER" > "${TMPDIR:-/tmp}/gh-scraper-owner-${CSID}"
printf '%s\n' "$GH_REPO" > "${TMPDIR:-/tmp}/gh-scraper-repo-${CSID}"
printf '%s\n' "$DATA_FILE" > "${TMPDIR:-/tmp}/gh-scraper-data-file-${CSID}"
# keyed to DATA_FILE stem — runs for different repos share the directory
printf '%s\n' "${DATA_FILE%.jsonl}.group1" > "${TMPDIR:-/tmp}/gh-scraper-group1-dir-${CSID}"
printf '%s\n' "$CUTOFF_3Y"   > "${TMPDIR:-/tmp}/gh-scraper-cutoff-3y-${CSID}"
printf '%s\n' "$CUTOFF_30D"  > "${TMPDIR:-/tmp}/gh-scraper-cutoff-30d-${CSID}"
printf '%s\n' "$CUTOFF_90D"  > "${TMPDIR:-/tmp}/gh-scraper-cutoff-90d-${CSID}"
printf '%s\n' "$CUTOFF_180D" > "${TMPDIR:-/tmp}/gh-scraper-cutoff-180d-${CSID}"