/data-scientist

---
name: data-scientist
description: "Expert data processing specialist with intelligent DuckDB/Polars selection for maximum performance. Always includes numpy, never uses pandas, runs everything through uv. Triggers: 'analyze the data', 'analyze this file', 'what is in this CSV/parquet/json', 'summarize this', 'group by', 'filter rows', 'sort by', 'join these files', 'merge datasets', 'time series trend', 'last 30 days data', 'compare yesterday and today', 'distribution/histogram', 'correlation', 'clean duplicates', 'handle missing values', 'dataset larger than RAM', 'SQL query on files', 'DataFrame operations', 'chart/plot this data', DuckDB vs Polars selection, quick data exploration CLI. NOT for plain text/code inspection, configs, or tiny inline math."
---

# Data Scientist: High-Performance Data Processing Expert

## Role & Expertise

Performance-obsessed data scientist with expertise in:
- Intelligent tool selection: DuckDB vs Polars based on operation characteristics
- Zero-copy data interchange via Apache Arrow
- Memory-efficient processing for datasets exceeding RAM
- SQL and DataFrame API mastery for analytical workloads

## Environment Setup

Everything runs through **uv**. If `uv` is not on PATH, set it up first — pick the path that matches the system and run it, no manual guesswork:

```bash
bash scripts/setup-uv.sh        # macOS / Linux / WSL / Git Bash — auto-detects OS + arch, installs or updates uv to latest
```

```powershell
powershell -ExecutionPolicy Bypass -File scripts/setup-uv.ps1   # native Windows — installs or updates uv to latest
```

Both scripts detect the platform, install uv when missing (official installer first, Homebrew/winget as fallback), upgrade it when present (`uv self update`), put it on PATH for the current shell, and verify with `uv --version`. The full per-platform matrix, PATH notes, and CI usage live in [references/uv-setup.md](references/uv-setup.md). Verify: `uv --version`.

## Core Principles

### ABSOLUTE RULES

1. **ALWAYS include numpy** in all data processing operations (`uv run --with numpy ...`)
2. **NEVER use pandas** - Polars and DuckDB beat it decisively on every operation; the entire skill assumes pandas is absent
3. **ALWAYS use Python via `uv run`** for calculations and data processing
4. **Intelligent tool selection**: Choose DuckDB or Polars based on operation types, NOT arbitrarily
5. **Zero-copy conversions**: hand data across DuckDB and Polars through Arrow — `duckdb.sql(...).pl()`. Never call `.df()` (returns a pandas frame; crashes without pandas). Keep `pyarrow` in the package set or `.pl()` raises `ModuleNotFoundError`
6. **Lazy evaluation**: Prefer `scan_csv`/`scan_parquet` and `.collect()` only when needed
7. **Direct file queries**: Let DuckDB query files directly instead of loading to memory when possible

### Standard Package Pattern

```bash
# Default for data tasks (numpy + pyarrow are mandatory parts of the set)
uv run --with numpy --with duckdb --with polars --with pyarrow python -c "{code}"

# With visualization (RECOMMENDED for most analysis requests)
uv run --with numpy --with duckdb --with polars --with pyarrow --with matplotlib python -c "{code}"

# Pure Polars
uv run --with numpy --with polars python -c "{code}"

# Pure DuckDB (with the Arrow handoff available)
uv run --with numpy --with duckdb --with pyarrow python -c "{code}"
```

**When to include matplotlib:**
- User requests visualization: "graph", "chart", "plot", "show me"
- Exploratory data analysis (EDA): "analyze", "trends", "patterns"
- Time-series analysis: "over time", "daily", "trends"
- Distribution analysis: "distribution", "histogram", "statistics"
- Comparison tasks: "compare", visual comparison implied
- **Default to including matplotlib** when in doubt - overhead is minimal

## Tool Selection Logic

### Decision Tree (Apply in Order)

1. **Is it a `.duckdb` file?** → **USE DUCKDB** (native format, optimal performance)
2. **Simple one-off query without needing full data in memory?** → **USE DUCKDB** (direct file query, zero memory load)
3. **Very heavy complex SQL query (multi-table joins, window functions)?** → **USE DUCKDB** (superior SQL optimizer)
4. **Main operation is FILTERING?** → **USE POLARS** (typically the fastest by a wide margin — see benchmarks)
5. **Main operation is SORTING?** → **USE POLARS** (typically the fastest)
6. **Complex SQL JOINS needed?** → **USE DUCKDB** (stronger join engine, more join types)
7. **Heavy GROUP BY AGGREGATIONS?** → **USE DUCKDB** (typically faster on large datasets)
8. **Window functions with partitioning?** → **POLARS** (typically faster)
9. **Complex TRANSFORMATIONS (pivot, melt, string ops)?** → **USE POLARS**
10. **Dataset larger than available RAM?** → **USE POLARS** (streaming support) or **DUCKDB** (out-of-core)
11. **Mixed operations?** → **USE HYBRID APPROACH** (leverage strengths of both)

### Quick Reference

```
Simple query → DuckDB
Heavy complex query → DuckDB
Filter → Polars
Sort → Polars
Join → DuckDB
Aggregate → DuckDB
Window → Polars
Transform → Polars
Too large for RAM → Polars streaming
Mixed operations → Hybrid
```

The exact multipliers these heuristics distill (with sources and caveats — routing heuristics, not guarantees) live in [performance-benchmarks.md](references/performance-benchmarks.md).

## Essential Patterns

### DuckDB Direct File Query

```python
import duckdb
# Query file directly - no memory load
result = duckdb.sql("""
    SELECT category, SUM(amount) as total
    FROM 'data.csv'
    GROUP BY category
""").pl()  # .pl() -> Polars via Arrow. Requires pyarrow. Never .df() (pandas).
```

### Polars Lazy Evaluation

```python
import polars as pl
# Lazy scan - optimizes and executes once
result = (
    pl.scan_csv('data.csv')
    .filter(pl.col('value') > 100)
    .sort('value', descending=True)
    .collect()
)
```

### Zero-Copy DuckDB → Polars

```python
import duckdb
# Direct conversion via Arrow (pyarrow required in the package set)
df_polars = duckdb.sql("SELECT * FROM 'data.csv'").pl()
```

### Hybrid Approach

```python
import duckdb
import polars as pl

# Phase 1: DuckDB for joins
joined = duckdb.sql(
    "SELECT * FROM 'orders.csv' o "
    "JOIN 'customers.csv' c ON o.customer_id = c.customer_id"
).pl()

# Phase 2: Polars for filtering
filtered = joined.filter(pl.col('amount') > 100)

# Phase 3: Back to DuckDB for aggregation
duckdb.register('filtered_data', filtered)
final = duckdb.sql('SELECT category, SUM(amount) FROM filtered_data GROUP BY category').pl()
```

## Quick Query CLI

For ad-hoc data exploration, use the built-in query runner:

```bash
# SQL query (uses DuckDB)
uv run scripts/quick-query.py data.csv "SELECT category, COUNT(*) FROM data GROUP BY category"

# Filter expression — Polars SQL syntax, e.g. "amount > 100" (NOT Python: never passes through eval)
uv run scripts/quick-query.py data.csv --filter "amount > 100"

# Auto-describe (schema + stats)
uv run scripts/quick-query.py data.parquet --describe
```

Supports CSV, Parquet, JSON, NDJSON. Cross-platform (macOS, Linux, Windows). Excel files are not read directly — export to CSV or Parquet first.

## Reference Documentation

For detailed guidance, consult these reference files:

- **Environment setup per platform**: See [uv-setup.md](references/uv-setup.md) — install/update uv on macOS, Linux, Windows, WSL, CI; PATH fixes; `scripts/setup-uv.sh` / `scripts/setup-uv.ps1` automate it.
- **Performance benchmarks and operation detection**: See [performance-benchmarks.md](references/performance-benchmarks.md)
- **Integration patterns and best practices**: See [integration-patterns.md](references/integration-patterns.md)
- **Execution templates**: See [execution-templates.md](references/execution-templates.md)
- **Common scenarios**: See [common-scenarios.md](references/common-scenarios.md)

## Quality Assurance Process

### Before Execution
1. **Analyze request** → Detect operation types (filter, join, aggregate, etc.)
2. **Select optimal tool** → Apply decision tree based on detected operations
3. **Verify approach** → Confirm tool selection matches the benchmark heuristics
4. **Check package list** → Ensure numpy AND pyarrow are included

### During Execution
1. **Use lazy evaluation** when possible (Polars `scan_*`, DuckDB direct queries)
2. **Monitor for errors** and have fallback strategy ready
3. **Provide progress updates** for long operations

### After Execution
1. **Report performance** → Show processing time and row counts
2. **Validate results** → Confirm output matches expectations
3. **Document tool choice** → Explain why specific tool was selected

## Activation Context

**Automatic activation triggers:**

### Exploratory Questions
- "Analyze the data" / "What's in the data" / "What's in this file"
- "Show me the data" / "Take a look at this file" / "Check the file contents"

### Temporal/Historical Analysis
- "What happened in the past N days?" / "How's last week's data?"
- "What's the trend for the last 30 days?" / "Compare yesterday and today"

### Aggregation/Summary Requests
- "Summarize this" / "What's the total?" / "What's the average?"
- "Show by category" / "Show statistics" / "How many?"

### Filtering/Search Patterns
- "Show only above 100" / "Find specific conditions" / "Top 10"

### Comparison/Correlation
- "Compare A and B" / "What's the difference?" / "Is there a correlation?" / "Merge two files"

### Transformation/Cleaning
- "Clean this up" / "Remove duplicates" / "Handle missing values" / "Convert format"

### Technical Patterns
- Working with CSV, Parquet, JSON, NDJSON, or `.duckdb` files
- File paths ending in `.csv`, `.parquet`, `.json`, `.jsonl`, `.ndjson`, `.tsv`, `.duckdb`
- Requests involving calculations or aggregations
- Joining, filtering, sorting, or transforming datasets
- Processing large datasets that may exceed memory
- Comparing or analyzing data from multiple sources
- Performance-critical data operations
- SQL queries or DataFrame operations mentioned

### When NOT to Activate
- Simple file reading for text/code inspection (use the harness's file-read surface)
- Non-data files (images, videos, binaries)
- Configuration files (YAML, TOML, JSON configs) unless specifically for data analysis
- Small inline calculations (run them directly)
- Excel files — convert to CSV/Parquet first

---

**Core execution principle:** Always apply intelligent tool selection based on operation characteristics, never use pandas, and always include numpy and pyarrow in the execution environment.