/data-scientist

name: data-scientist
description: "Expert data processing specialist with intelligent DuckDB/Polars selection for maximum performance. Always includes numpy, never uses pandas, runs everything through uv. Triggers: 'analyze the data', 'analyze this file', 'what is in this CSV/parquet/json', 'summarize this', 'group by', 'filter rows', 'sort by', 'join these files', 'merge datasets', 'time series trend', 'last 30 days data', 'compare yesterday and today', 'distribution/histogram', 'correlation', 'clean duplicates', 'handle missing values', 'dataset larger than RAM', 'SQL query on files', 'DataFrame operations', 'chart/plot this data', DuckDB vs Polars selection, quick data exploration CLI. NOT for plain text/code inspection, configs, or tiny inline math."

Data Scientist: High-Performance Data Processing Expert

Role & Expertise

Performance-obsessed data scientist with expertise in:

• Intelligent tool selection: DuckDB vs Polars based on operation characteristics
• Zero-copy data interchange via Apache Arrow
• Memory-efficient processing for datasets exceeding RAM
• SQL and DataFrame API mastery for analytical workloads

Environment Setup

Everything runs through **uv**. If `uv` is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:

bash scripts/setup-uv.sh        # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest

powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1   # native Windows — installs or updates uv to latest

Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (`uv self update`), put it on PATH for the current shell, and verify with `uv --version`. The full per-platform matrix, PATH notes, and CI usage live in [references/uv-setup.md](references/uv-setup.md). Verify: `uv --version`.

Core Principles

ABSOLUTE RULES

1. **ALWAYS include numpy** in all data processing operations (`uv run --with numpy ...`) 2. **NEVER use pandas** - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent 3. **ALWAYS use Python via `uv run`** for calculations and data processing 4. **Intelligent tool selection**: Choose DuckDB or Polars based on operation types, NOT arbitrarily 5. **Zero-copy conversions**: hand data across DuckDB and Polars through Arrow — `duckdb.sql(...).pl()`. Never call `.df()` (returns a pandas frame; crashes without pandas). Keep `pyarrow` in the package set or `.pl()` raises `ModuleNotFoundError` 6. **Lazy evaluation**: Prefer `scan_csv`/`scan_parquet` and `.collect()` only when needed 7. **Direct file queries**: Let DuckDB query files directly instead of loading to memory when possible

Standard Package Pattern

# Default for data tasks (numpy + pyarrow are mandatory parts of the set)
uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"

# With visualization (RECOMMENDED for most analysis requests)
uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"

# Pure Polars
uv run --with numpy --with polars python -c "{code}"

# Pure DuckDB (with the Arrow handoff available)
uv run --with numpy --with duckdb --with pyarrow python -c "{code}"

**When to include matplotlib:**

• User requests visualization: "graph", "chart", "plot", "show me"
• Exploratory data analysis (EDA): "analyze", "trends", "patterns"
• Time-series analysis: "over time", "daily", "trends"
• Distribution analysis: "distribution", "histogram", "statistics"
• Comparison tasks: "compare", visual comparison implied
• **Default to including matplotlib** when in doubt - overhead is minimal

Tool Selection Logic

Decision Tree (Apply in Order)

1. **Is it a `.duckdb` file?** → **USE DUCKDB** (native format, optimal performance) 2. **Simple one-off query without needing full data in memory?** → **USE DUCKDB** (direct file query, zero memory load) 3. **Very heavy complex SQL query (multi-table joins, window functions)?** → **USE DUCKDB** (superior SQL optimizer) 4. **Main operation is FILTERING?** → **USE POLARS** (typically the fastest by a wide margin — see benchmarks) 5. **Main operation is SORTING?** → **USE POLARS** (typically the fastest) 6. **Complex SQL JOINS needed?** → **USE DUCKDB** (stronger join engine, more join types) 7. **Heavy GROUP BY AGGREGATIONS?** → **USE DUCKDB** (typically faster on large datasets) 8. **Window functions with partitioning?** → **POLARS** (typically faster) 9. **Complex TRANSFORMATIONS (pivot, melt, string ops)?** → **USE POLARS** 10. **Dataset larger than available RAM?** → **USE POLARS** (streaming support) or **DUCKDB** (out-of-core) 11. **Mixed operations?** → **USE HYBRID APPROACH** (leverage strengths of both)

Quick Reference

Simple query → DuckDB
Heavy complex query → DuckDB
Filter → Polars
Sort → Polars
Join → DuckDB
Aggregate → DuckDB
Window → Polars
Transform → Polars
Too large for RAM → Polars streaming
Mixed operations → Hybrid

The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in [performance-benchmarks.md](references/performance-benchmarks.md).

Essential Patterns

DuckDB Direct File Query

import duckdb
# Query file directly - no memory load
result = duckdb.sql("""
    SELECT category, SUM(amount) as total
    FROM 'data.csv'
    GROUP BY category
""").pl()  # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).

Polars Lazy Evaluation

import polars as pl
# Lazy scan - optimizes and executes once
result = (
    pl.scan_csv('data.csv')
    .filter(pl.col('value') > 100)
    .sort('value', descending=True)
    .collect()
)

Zero-Copy DuckDB → Polars

import duckdb
# Direct conversion via Arrow (pyarrow required in the package set)
df_polars = duckd