/data-scientist

name: data-scientist
description: >-
  Data science methodology for Python research: EDA, validation, causal inference (IV, DiD, RD, synthetic control), clustering/PCA/UMAP, supervised ML, geospatial, visualization. Method selection guidance. For syntax, load tool-specific skills.
metadata:
  audience: any-agent
  domain: research-methodology

Data Scientist Skill

Rigorous data science methodology and mindset for Python research. Covers EDA, data validation, transformation verification, documentation standards, visualization design, descriptive analysis, statistical modeling, causal inference method selection (IV, DiD, RD, synthetic control), unsupervised analysis (clustering, PCA, UMAP), supervised ML methodology (prediction vs. inference, cross-validation, model interpretation, fairness), and geospatial analysis. Provides methodology decisions and analytical approach guidance. For implementation syntax, load the tool-specific skill (polars, statsmodels, plotnine, pyfixest, scikit-learn, geopandas, etc.). Use for any data analysis, exploration, transformation, or modeling task — especially when choosing methods, checking assumptions, or structuring an analysis.

Establishes a rigorous, methodical approach to data science work. This skill is about *how* to think and work, not specific tools. Load specialized skills (polars, plotnine, plotly, marimo, etc.) for tool-specific guidance.

Core Principles - NON-NEGOTIABLE

These five principles must guide ALL data science work. They are not optional.

Principle 1: Data Robustness First

**ALWAYS check data before operating on it.**

Before ANY analysis or transformation:

• Check shape, types, and memory usage
• Examine value distributions and ranges
• Identify and characterize missing values (count, percentage, pattern)
• Understand what uniquely identifies each row (granularity)
• Look for outliers and anomalies

Be VERBOSE about what you're checking and what you find. Never assume data is clean.

# ALWAYS start with this pattern
print(f"Shape: {df.shape}")
print(f"Columns: {df.columns.to_list()}")
print(f"Types:\n{df.dtypes}")
print(f"Null counts:\n{df.null_count()}")
print(f"Sample:\n{df.sample(5)}")

This principle applies only when you are conducting actual data work. Do NOT conduct net new analyses or data inspections when tasked with compiling past work (e.g., analytic notebook creation), or synthesizing prior analyses into a report (e.g., final report writing).

Principle 2: Documentation First

**ALWAYS understand or create data documentation.**

Before analysis:

• Seek data dictionaries, schemas, or documentation
• Understand where data comes from (provenance)
• Learn collection methods and their implications
• Identify known quality issues or caveats
• Clarify what each column means in business context

If documentation doesn't exist, CREATE IT as you learn about the data.

Principle 3: Verify Every Operation

**NEVER assume a transformation worked correctly.**

For EVERY data operation:

• Check row counts before and after
• Examine random samples of affected rows
• Validate that expected changes occurred
• Confirm no unintended side effects
• Document what you checked and what you found

# Before transformation
print(f"Before: {len(df)} rows, columns: {df.columns.to_list()}")
sample_before = df.filter(pl.col("id").is_in([1, 42, 100]))

# After transformation
print(f"After: {len(result)} rows, columns: {result.columns.to_list()}")
sample_after = result.filter(pl.col("id").is_in([1, 42, 100]))
print(f"Sample comparison:\nBefore:\n{sample_before}\nAfter:\n{sample_after}")

Principle 4: Thorough Code Documentation (ENFORCED)

**Write extensive comments explaining your reasoning. This is MANDATORY, not optional.**

In research workflows, follow the **Inline Audit Trail (IAT)** standard (see `agent_reference/INLINE_AUDIT_TRAIL.md`). The IAT standard is enforced during QA review — scripts with sparse documentation receive WARNING findings.

Every code block should explain:

• WHAT you're trying to accomplish (the goal) → IAT Type 2: Intent Comment
• WHY you chose this approach (the reasoning) → IAT Type 3: Reasoning Comment
• WHAT assumptions you're making (the dependencies) → IAT Type 4: Assumption Comment

For tests and validations, explain:

• What behavior you're checking
• What would indicate success vs. failure
• Why this check matters

Principle 5: Focus on Research Questions

**Balance rigor with usefulness.**

Always consider:

• What question are we actually answering?
• What level of rigor does this decision require?
• Are there multiple valid approaches with different tradeoffs?
• Should I check with the user before proceeding?

CHECK IN with users when:

• Multiple valid methodologies exist
• Tradeoffs between precision and practicality arise
• Findings are surprising or counterintuitive
• Scope might need adjustment

Related Skills - When to Load

**Core Workflow Skills (Load Together):**

• `polars` - Required for DataFrame operations; data-scientist provides methodology, polars provides syntax
• `marimo` - Required for creating validated notebooks; data-scientist defines validation patterns, marimo provides implementation

**For Data Analysis Workflows:**

In the research pipeline, data-scientist methodology is applied within the **file-first execution pattern**:

• Write script files FIRST (to `scripts/stage{N}_{type}/`)
• Execute via Bash with automatic output capture wrapper script
• Validation results get automatically embedded in scripts as comments
• Marimo notebook assembles validated scripts for interactive review

Closely read `agent_reference/SCRIPT_EXECUTION_REFERENCE.md` for the mandatory file-first execution protocol covering complete code file writing, output capture, and file versioning rules.

**Load for Specific Needs:**

What task are you performing?
├─ Data visualization (any kind)
│   └─ Stage 8.2 — FIRST read visualization reference files be