/find-data

name: find-data
description: "Find a dataset for a Data2Story blog. Accepts a topic, a URL, or a DIP-style category. Downloads + validates against 4 completeness gates before handing off to /data2story-pro. Local-first: searches Economist/Pudding/TidyTuesday clones before going online. Supports --validate-only to audit a folder you already have."
argument-hint: "<topic | URL | category | folder-path> [--mode single|theme] [--source Economist|Pudding|tidytuesday] [--out ./datasets/<name>] [--validate-only]"
allowed-tools: Bash(python:*), Read, Write, Glob, Grep, WebSearch, WebFetch, AskUserQuestion

find-data

Turn an idea, URL, or category into a `phase2/datasets/<name>/` folder that the `/data2story-pro` pipeline can run on without crashing.

You are the **gatekeeper before the 7-agent newsroom**. Detective, Analyst, Editor, Designer, Programmer, Auditor, Inspector all assume the data is already there, parseable, and provenanced. Your job is to make sure that's true before they start.

Refuse to mark a folder ready until it passes 4 gates. See `references/completeness_gates.md` for the criteria. The gates are codified in `tools/audit.py`, not in this prose.

Prerequisites

Python deps for the `tools/`:

• **`pandas`** — required (`audit.py` reads/inspects CSV/JSON).
• **`openpyxl`** — required **only** when a source is `.xlsx` (`audit.py`'s

`pd.ExcelFile`); a mid-run missing-`openpyxl` is the usual cause of an XLSX audit error.

• **`urllib`** — stdlib, no install (`fetch.py` downloads, `audit.py` HEAD-checks).

One install line:

pip install pandas openpyxl

After editing anything under `tools/`, run the no-network regression suite: `py tools/selftest.py` (exit 0 = pass).

Resolve paths first

• `SKILL_DIR` = directory containing this `SKILL.md`
• `WORKSPACE` = ancestor that contains `phase2/datasets/` (the parent repo root), if one exists
• `DATASETS_ROOT` = `WORKSPACE/phase2/datasets` when `WORKSPACE` exists; otherwise `./datasets`

(clone-relative, under the current working dir) — the open-source clone has no `phase2/datasets`

• `BLOGS_ROOT` = `WORKSPACE/phase2/blogs` when `WORKSPACE` exists
• `OUT_DIR` default = `DATASETS_ROOT/<name>` (so `./datasets/<name>` on an OSS clone). Callers may

override with an explicit `--out`, which always wins.

• `INPUT` = first positional argument from `$ARGUMENTS`
• Parse flags from `$ARGUMENTS`: `--mode`, `--source`, `--out`, `--validate-only`

Never hard-code machine paths. Resolve at runtime by walking up from `SKILL_DIR`.

Step 0 — Classify the input

Decision tree (the FIRST condition that matches wins):

1. **`--validate-only` flag present** → MODE = `validate-only`. `INPUT` must be an existing folder path. 2. **`INPUT` is an existing folder path** (and `--validate-only` absent) → still ask the user before re-fetching; default to `validate-only` behaviour with a confirmation. 3. **`INPUT` starts with `http://` or `https://`** → MODE = `url`. 4. **`INPUT` (case-insensitive) exactly matches a DIP category** from `phase2/datasets/data_is_plural/dip_category_summary.md` → MODE = `category`. 5. **Otherwise** → MODE = `topic`.

State your classification out loud in one line before doing anything else.

**No local corpora (open-source clone):** on a machine without the `Economist/`, `Pudding/`, `tidytuesday/` clones and without the Data-is-Plural CSV under `phase2/datasets/`, `browse_local.py` and `dip_query.py` both return empty (no crash). In that case there is nothing to match locally, so `topic`/`category` discovery degrades straight to WebSearch (Step 1). No flag is needed — detect it from the empty `browse_local.py` + `dip_query.py` results. Keep the local-first path intact for machines that DO have the corpora.

Step 1 — Branch on mode

validate-only mode

Just audit. Skip discovery, skip fetch.

python "SKILL_DIR/tools/audit.py" "<folder>"

This writes `<folder>/validate.json` and prints the gate summary. Read it back with `Read`, surface the verdict, and stop.

If `overall.ready_for_data2story` is true → print the `/data2story-pro <folder>` command for the user to copy.

If it's false → list the specific gate failures and recommend remediations.

url mode

Determine OUT_DIR (use `--out` if given, else derive a slug from the URL path's last meaningful segment, store under `DATASETS_ROOT/<slug>/` — i.e. `./datasets/<slug>/` on an open-source clone with no `phase2/datasets`).

Branch on URL shape:

| URL pattern | Tool | |---|---| | `github.com/{owner}/{repo}/tree/{branch}/{path}` | `python fetch.py github-folder <url> <out_dir>` | | `github.com/{owner}/{repo}/blob/{branch}/{path}` | Same (fetch.py auto-detects blob) | | `github.com/{owner}/{repo}` with no path | Refuse — tell user to narrow to a folder | | Anything else | `python fetch.py url <url> <out_dir>` |

`fetch.py` is source-aware: if the URL points to `TheEconomist/graphic-detail-data`, `the-pudding/data`, or `rfordatascience/tidytuesday` AND the local clone already contains that path, it copies from the local clone (no network). Otherwise it fetches via the GitHub API. This happens automatically — do not handle it yourself.

URL normalization (share-page vs direct-download)

Several hosts serve a human-facing **page** at the obvious URL, not the raw bytes, so a naive download returns HTML instead of a `.csv`/`.json`/`.xlsx`. `fetch.py` auto-rewrites GitHub `/blob/` URLs to `raw.githubusercontent.com` and, as a backstop, **warns on stderr** if a data-extension download comes back as an HTML page (`<!doctype html>` / `<html`). The other common share-hosts are **not** auto-rewritten — pass the **direct-download** form yourself:

| Host | Page URL (returns HTML) | Direct-download form to use | |---|---|---| | GitHub | `github.com/<o>/<r>/blob/<branch>/<path>` | auto-handled → `raw.githubusercontent.com/<o>/<r>/<branch>/<path>` | | Google Drive | `drive.google.com/file/d/<id>/view` | `drive.google.com/uc?export=download&id