/data-download

name: data-download
description: 'Discover, evaluate, and download publicly available datasets from the internet. Infers data needs from a research question or task, selects authoritative sources, downloads reproducibly, validates file integrity, and documents provenance. Pauses for user input when authentication, API keys, or major tradeoffs require a decision. Use when user says "download data", "get data", "find a dataset", "I need boundary files", "download census data", or needs any external dataset for analysis.'
argument-hint: [data-need-or-research-question]
allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob, WebSearch, WebFetch, Agent

Data Download: Discover, Evaluate, and Acquire Research Data

Find and download data for: **$ARGUMENTS**

Overview

This skill turns a data need into a validated, documented, locally available dataset. It does NOT blindly download from the first URL found. It reasons from the research question to the right data, evaluates candidate sources, downloads reproducibly, validates the result, and records provenance.

The workflow is always: **data need → source discovery → source evaluation → download → validation → provenance documentation**.

Constants

• **DATA_DIR = `data/`** — Top-level data directory. Raw downloads go to `data/raw/`, processed outputs to `data/processed/`.
• **MANIFEST_FILE = `data/DATA_MANIFEST.md`** — Provenance log for all downloaded datasets.
• **MAX_UNATTENDED_SIZE_MB = 500** — If a single file exceeds this size, pause and confirm with the user before downloading.
• **VERIFY_AFTER_DOWNLOAD = true** — Always validate downloaded files (existence, size, format, openability).
• **PREFER_PUBLIC = true** — When both public and authenticated sources exist, prefer the public source unless the authenticated source is materially better.

> Override via argument, e.g., `/data-download "US census tracts" — data dir: datasets/, max size: 2000`.

---

Phase 0: Understand the Data Need

Before searching for data, classify what is actually needed.

Step 0.1: Infer Data Requirements

Extract from the user's request, the research question, or existing project files (`research_contract.md`, `output/EXPERIMENT_PLAN.md`, `RESEARCH_PLAN.md`):

| Dimension | Question to answer | |---|---| | **Geography** | What study area? Country, state, city, bounding box, global? | | **Time range** | What period? Single year, multi-year, historical, real-time? | | **Spatial resolution** | Point, tract, county, grid cell, pixel? What resolution? | | **Temporal resolution** | Daily, monthly, annual, static snapshot? | | **Variables / bands** | What specific attributes, indicators, or spectral bands? | | **Data type** | Vector (points, lines, polygons), raster, tabular, network? | | **File format** | GeoJSON, Shapefile, GeoPackage, GeoParquet, GeoTIFF, COG, NetCDF, CSV? | | **Licensing** | Must it be open? CC-BY? Public domain? Commercial use? | | **Quality** | Research-grade? Official statistics? Authoritative boundaries? | | **Size** | How many features/pixels/rows? Will it fit in memory? |

If the request is too vague (e.g., "get me some data"), STOP and ask the user to specify at least:

• What geographic area?
• What variables or themes?
• What time period?

Step 0.2: Categorize the Data Need

Map the need to one or more data categories:

| Category | Examples | Typical sources | |---|---|---| | **Administrative boundaries** | Country, state, county, tract, district | Census/TIGER, GADM, Natural Earth, OSM | | **Transportation / infrastructure** | Roads, railways, airports, utilities | OSM, DOT, OpenStreetMap Overpass | | **Elevation / terrain** | DEM, slope, aspect, hillshade | USGS 3DEP, SRTM, Copernicus DEM, ALOS | | **Land cover / land use** | NLCD, ESA WorldCover, MODIS LC | MRLC, ESA, USGS, Google Dynamic World | | **Environmental / climate** | Temperature, precipitation, drought, soil | PRISM, ERA5, NOAA, WorldClim, SoilGrids | | **Weather** | Observations, forecasts, station data | NOAA ISD/GHCN, Open-Meteo, ERA5 | | **Air quality** | PM2.5, ozone, AQI | EPA AQS, OpenAQ, Copernicus CAMS | | **Hydrology** | Rivers, watersheds, streamflow, flood zones | NHD, USGS NWIS, HydroSHEDS, FEMA NFHL | | **Disaster / hazard** | Hurricane tracks, wildfire, earthquake, flood | NOAA IBTrACS, NIFC, USGS Earthquake, FEMA | | **Census / socioeconomic** | Population, income, housing, education | US Census/ACS, WorldPop, IPUMS, Eurostat | | **Public health** | Disease rates, hospital locations, SVI | CDC, HHS, EPA EJScreen, County Health Rankings | | **Remote sensing imagery** | Landsat, Sentinel, MODIS, VIIRS | USGS EarthExplorer, Copernicus Open Access, NASA LAADS | | **Points of interest** | Schools, hospitals, parks, businesses | OSM, HIFLD, local open data portals | | **Vegetation / agriculture** | NDVI, crop type, cropland extent | MODIS, Sentinel-2, USDA CDL | | **Ocean / coastal** | Bathymetry, sea surface temp, shorelines | NOAA NCEI, GEBCO, Copernicus Marine |

Write the data requirement summary to `data/DOWNLOAD_PLAN.md`:

# Data Download Plan

**Task**: [user's request or research question]
**Date**: [today]

## Required Datasets

### Dataset 1: [descriptive name]
- Category: [from table above]
- Geography: [study area]
- Time range: [period]
- Spatial resolution: [resolution]
- Variables: [list]
- Format preference: [format]
- Priority: REQUIRED / NICE-TO-HAVE

### Dataset 2: [descriptive name]
...

## Constraints
- Licensing: [requirement]
- Size budget: [limit]
- Authentication: [any known requirements]

---

Phase 1: Discover Data Sources

Step 1.1: Source Search Strategy

For each required dataset, search for sources in this priority order:

1. **Official government portals** — Census.gov, data.gov, USGS, NOAA, EPA, Eurostat, national mapping agencies 2. **Intergovernmental organizations** — UN, World Bank, WHO, FAO, OECD 3. **Well-known research repositories** — NASA Earthdata, Copernicus, PANGAEA, Zenod