/data-finder

name: data-finder
description: >-
  Find and assess datasets for a research question. Dispatches Explorer agents to search across data source categories, then Explorer-Critic to stress-test each candidate. Produces a ranked list with feasibility grades. Make sure to use this skill whenever the user wants to identify or evaluate data sources — not to search for papers or run analysis. Triggers include: "find data", "what data should I use", "find a dataset for this", "where can I get data on X", "assess datasets", "what datasets exist for", "help me find data", "is there data on this", "what are my data options", "I need data for this project", or any request to locate empirical data sources for a research question.
argument-hint: "[research topic or 'from spec']"
allowed-tools: ["Read", "Grep", "Glob", "Write", "WebSearch", "WebFetch", "Task"]

Data Finder

Find and assess datasets for your research question. Two Explorer agents search in parallel across data source categories; an Explorer-Critic then stress-tests each candidate against the research design.

**Input:** `$ARGUMENTS` — a topic, or `from spec` to read the research question from `quality_reports/`.

---

Step 1: Read Research Context

1. Find the most recent `quality_reports/project_spec_*.md` or `quality_reports/specs/*.md` — extract:

• Research question
• Empirical strategy (DiD, RDD, IV, etc.)
• Treatment variable (what varies)
• Outcome variable (what we measure)
• Controls needed
• Time period of interest
• Geography (national, state, county, individual)
• Unit of observation (individual, household, firm, establishment)

2. Read `references/domain-profile.md` if it exists — extract the Common Datasets section (domain-specific datasets to check first).

3. If no research spec exists, extract the variables and strategy from `$ARGUMENTS` directly. If the request is vague, ask: *"What are the treatment and outcome variables, and what empirical strategy did you have in mind?"*

---

Step 2: Dispatch Two Explorer Agents in Parallel

Split the source categories between two Explorer agents to parallelize the search.

**Explorer A — Institutional Data:**

Task prompt: "You are an Explorer agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].
Domain datasets (check first): [list from domain-profile if available].

Your source categories to search:
1. Public microdata (CPS, ACS, NHIS, MEPS, SIPP, QWI)
2. Administrative data (Medicare/Medicaid, IRS, SSA, vital statistics, court records)
3. Survey panels (PSID, HRS, Add Health, NLSY97/79, BHPS/UKHLS)

For each dataset found, produce the full Explorer report format.
Follow the Explorer agent instructions."

**Explorer B — Broader and Alternative Sources:**

Task prompt: "You are an Explorer agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].
Domain datasets (check first): [list from domain-profile if available].

Your source categories to search:
1. International data (World Bank, OECD, Eurostat, IMF, IPUMS International)
2. Novel/alternative (satellite, web scraping, proprietary, RCT registries)
3. Any field-specific datasets not covered by Explorer A

For each dataset found, produce the full Explorer report format.
Follow the Explorer agent instructions."

---

Step 3: Dispatch Explorer-Critic

After both Explorer agents complete, dispatch the Explorer-Critic with the full combined dataset list.

Task prompt: "You are an Explorer-Critic agent. Research question: [question].
Empirical strategy: [strategy].
Variables needed — Treatment: [X], Outcome: [Y], Controls: [list],
Time period: [period], Geography: [geo], Unit: [unit].

Here is the combined dataset list from the Explorer agents:
[paste all Explorer findings]

Apply the 5-point critique to each dataset:
1. Measurement validity
2. Sample selection
3. External validity
4. Identification compatibility
5. Known issues

Produce adjusted feasibility grades and deal-breaker flags.
Follow the Explorer-Critic agent instructions."

---

Step 4: Produce Ranked Output

After the Explorer-Critic completes, compile the final ranked report:

1. Sort datasets by adjusted feasibility grade (A first, then B, then C, then D). 2. Within each grade, sort by identification compatibility score (highest first). 3. Separate out deal-breaker datasets into the rejection table.

---

Step 5: Save Report

Save to `quality_reports/data_exploration_[sanitized_topic].md`:

# Data Exploration: [Topic]

**Date:** [YYYY-MM-DD]
**Research question:** [one sentence]
**Empirical strategy:** [method]
**Variables sought:** Treatment = [X], Outcome = [Y], Controls = [list]

---

## Top Candidates (Grade A–B)

### 1. [Dataset Name] — Grade: A/B

**Provider:** [Name] | **Access:** [Public/Restricted/etc.] | **URL:** [link]

**Coverage:** [time period] | [geography] | [unit of observation] | N ≈ [size]

**Key Variables:**
- Treatment proxy: [variable]
- Outcome: [variable]
- Controls available: [list]

**Explorer-Critic Assessment:**
- Measurement validity: [1-2 sentences]
- Sample selection: [1-2 sentences]
- External validity: [1-2 sentences]
- Identification compatibility: [focused on the proposed strategy]
- Known issues: [specific documented problems]

**Bottom line:** [1-2 sentences — viable and under what conditions]

---

[Repeat for all A and B grade datasets]

---

## Accessible With Effort (Grade C)

[Brief summaries — name, access path, main limitation, why C not B]

---

## Rejection Table

| Dataset | Reason for Rejection | Deal-breaker? |
|---------|---------------------|---------------|
| [Name] | [Explorer-Critic finding] | YES/NO |

---

## Recommended Path Forward

1. **Best dataset:** [Name] — [o