/scientific-data-preprocessing

name: scientific-data-preprocessing
description: "⚠️ CRITICAL USER EXPERIENCE-BASED SKILL - ALWAYS CONSULT BEFORE DATA PREPROCESSING ⚠️ Prevents catastrophic errors (88.9% error rate in V1.0 case study) through multi-level feature analysis, data leakage detection, and semantic validation. MANDATORY for: data preprocessing, feature engineering, standardization, normalization, interpolation, missing value handling, feature selection, or ANY data transformation task. Covers grouped time-series, cross-sectional, panel data. Detects: time travel leakage, causal inversion, ID misuse, semantic-numeric fallacies, distribution blindness. User's hard-won lessons from real project failures."

Scientific Data Preprocessing Skill

⚠️ **CRITICAL: USER'S HARD-WON EXPERIENCE - MANDATORY CONSULTATION** ⚠️

This skill encapsulates painful lessons learned from real preprocessing disasters (88.9% error rate documented). **ALWAYS use this skill for planning, reflection, and validation when ANY data preprocessing is involved.**

**Why this skill is mandatory:**

• Based on actual project failures (V1.0, V2.0 case studies)
• Prevents data leakage that causes production disasters
• Catches semantic errors AI agents commonly make
• Saves weeks of debugging and model retraining

**When to invoke (DO NOT SKIP):**

• ✅ Before starting ANY data preprocessing task
• ✅ During preprocessing for reflection and validation
• ✅ After preprocessing for comprehensive audit
• ✅ When reviewing AI-generated preprocessing code

---

Core Mission

Prevent catastrophic preprocessing errors in grouped time-series data by applying multi-level feature analysis and respecting data structure boundaries.

When to Use This Skill

**MANDATORY consultation - trigger immediately when:**

Data Preprocessing Tasks (ALWAYS)

• Any data cleaning, transformation, or preparation work
• Loading and preparing data for modeling
• Creating training/test splits
• Handling missing values (imputation, deletion)
• Feature scaling/normalization/standardization
• Encoding categorical variables
• Feature engineering or construction
• Feature selection or dimensionality reduction

Data Structure Types (ALWAYS)

• Preprocesssing time-series data with natural groupings (matches, sessions, patients, experiments)
• Sports analytics (tennis, basketball, etc.)
• Medical/clinical data with patient groupings
• Panel data or longitudinal studies
• Any grouped/hierarchical data structure

Quality Assurance (ALWAYS)

• Auditing existing preprocessing for data leakage or semantic errors
• Reviewing AI-generated preprocessing code for common pitfalls
• Validating preprocessing before model training
• Debugging unexpected model performance

Critical Checkpoints (NEVER SKIP)

• ✅ **BEFORE**: Planning preprocessing strategy
• ✅ **DURING**: Reflecting on decisions and checking for errors
• ✅ **AFTER**: Comprehensive validation and audit

**Trigger keywords that MUST invoke this skill:**

• "preprocess", "preprocessing", "data cleaning", "data preparation"
• "standardize", "normalize", "scale", "transform"
• "impute", "fill missing", "handle NaN"
• "encode", "one-hot", "categorical"
• "feature engineering", "feature selection", "feature construction"
• "train test split", "cross validation split"
• "interpolate", "smooth", "aggregate"

Not For / Boundaries

This skill does NOT:

• Handle purely cross-sectional data (ungrouped, single timepoint)
• Make domain-specific feature engineering decisions (you decide business logic)
• Choose ML models (focuses on preprocessing only)
• Handle distributed/big data infrastructure (assumes data fits in memory)

Required inputs before proceeding: 1. Confirmation that data has groups (e.g., match_id, patient_id, session_id) 2. Understanding of whether goal is within-group (relative) or cross-group (absolute) comparison 3. Domain constraints on data ranges/units

Quick Reference

Multi-Level Feature Analysis Framework

**Level 1: Data Type**

# Check data types
df.dtypes  # int64, float64, object, etc.

**Level 2: Feature Type Classification**

# Binary (0/1)
binary_features = [col for col in df.columns if df[col].nunique() == 2]

# Categorical (finite discrete values)
categorical_features = [col for col in df.select_dtypes(include='object').columns]

# Continuous (infinite possible values)
continuous_features = [col for col in df.select_dtypes(include=['float64', 'int64']).columns
                       if df[col].nunique() > 10]

**Level 3: Data Structure**

# Check for grouping
print(f"Number of groups: {df['group_id'].nunique()}")
print(f"Avg points per group: {df.groupby('group_id').size().mean():.1f}")

# Check for time-series
df_sorted = df.sort_values(['group_id', 'timestamp'])

**Level 4: Physical Meaning**

# Validate physical ranges
assert df['speed_mph'].max() < 200, "Speed exceeds physical limit"
assert df['distance_meters'].min() >= 0, "Negative distance impossible"

Critical Processing Decision Tree

# Decision: Within-group or global processing?
def choose_processing_scope(data, feature, goal):
    """
    goal = 'relative' → within-group (e.g., "this point was intense FOR THIS MATCH")
    goal = 'absolute' → global (e.g., "this was an intense point OVERALL")
    """
    if goal == 'relative':
        return 'within_group'
    elif goal == 'absolute':
        return 'global'
    else:
        raise ValueError("Goal must be 'relative' or 'absolute'")

Pattern 1: Within-Group Interpolation (CORRECT)

from scipy.interpolate import CubicSpline
import numpy as np

# ✅ CORRECT: Interpolate within each group
for group_id in df['match_id'].unique():
    mask = df['match_id'] == group_id
    group_data = df.loc[mask, 'speed_mph'].copy()

    # Get valid (non-NaN) indices
    valid_idx = group_data.notna()
    valid_positions = np.where(valid_idx)[0]
    valid_values = group_data[valid_idx].values

    if len(valid_posi