/data-analyst

name: data-analyst
description: This skill should be used when analyzing CSV datasets, handling missing values through intelligent imputation, and creating interactive dashboards to visualize data trends. Use this skill for tasks involving data quality assessment, automated missing value detection and filling, statistical analysis, and generating Plotly Dash dashboards for exploratory data analysis.

Data Analyst

Overview

This skill provides comprehensive capabilities for data analysis workflows on CSV datasets. It automatically analyzes missing value patterns, intelligently imputes missing data using appropriate statistical methods, and creates interactive Plotly Dash dashboards for visualizing trends and patterns. The skill combines automated missing value handling with rich interactive visualizations to support end-to-end exploratory data analysis.

Core Capabilities

The data-analyst skill provides three main capabilities that can be used independently or as a complete workflow:

1. Missing Value Analysis

Automatically detect and analyze missing values in datasets, identifying patterns and suggesting optimal imputation strategies.

2. Intelligent Imputation

Apply sophisticated imputation methods tailored to each column's data type and distribution characteristics.

3. Interactive Dashboard Creation

Generate comprehensive Plotly Dash dashboards with multiple visualization types for trend analysis and exploration.

Complete Workflow

When a user requests complete data analysis with missing value handling and visualization, follow this workflow:

Step 1: Analyze Missing Values

Run the missing value analysis script to understand the data quality:

python3 scripts/analyze_missing_values.py <input_file.csv> <output_analysis.json>

**What this does**:

• Detects missing values in each column
• Identifies data types (numeric, categorical, temporal, etc.)
• Calculates missing value statistics
• Suggests appropriate imputation strategies per column
• Generates detailed JSON report and console output

**Review the output** to understand:

• Which columns have missing data
• The percentage of missing values
• The recommended imputation method for each column
• Why each method was recommended

Step 2: Impute Missing Values

Apply automatic imputation based on the analysis:

python3 scripts/impute_missing_values.py <input_file.csv> <analysis.json> <output_imputed.csv>

**What this does**:

• Loads the analysis results (or performs analysis if not provided)
• Applies the optimal imputation method to each column:
• **Mean**: For normally distributed numeric data
• **Median**: For skewed numeric data
• **Mode**: For categorical variables
• **KNN**: For multivariate numeric data with correlations
• **Forward fill**: For time series data
• **Constant**: For high-cardinality text fields
• Handles edge cases (drops rows/columns when appropriate)
• Generates imputation report with before/after statistics
• Saves cleaned dataset

**The script automatically**:

• Drops columns with >70% missing values
• Drops rows where critical ID columns are missing
• Performs batch KNN imputation for correlated variables
• Creates detailed imputation log

Step 3: Create Interactive Dashboard

Generate an interactive Plotly Dash dashboard:

python3 scripts/create_dashboard.py <imputed_file.csv> <output_dir> <port>

**Example**:

python3 scripts/create_dashboard.py data_imputed.csv ./visualizations 8050

**What this does**:

• Automatically detects column types (numeric, categorical, temporal)
• Creates comprehensive visualizations:
• **Summary statistics table**: Descriptive stats for all numeric columns
• **Time series plots**: Trend analysis if date/time columns exist
• **Distribution plots**: Histograms for understanding data distributions
• **Correlation heatmap**: Relationships between numeric variables
• **Categorical analysis**: Bar charts for categorical variables
• **Scatter plot matrix**: Pairwise relationships between variables
• Launches interactive Dash web server
• Optionally saves static HTML visualizations

**Access the dashboard** at `http://127.0.0.1:8050` (or specified port)

Individual Use Cases

Use Case A: Quick Missing Value Assessment

When the user wants to understand data quality without imputation:

python3 scripts/analyze_missing_values.py data.csv

Review the console output to understand missing value patterns and get recommendations.

Use Case B: Imputation Only

When the user has a dataset with missing values and wants cleaned data:

python3 scripts/impute_missing_values.py data.csv

This performs analysis and imputation in one step, producing `data_imputed.csv`.

Use Case C: Visualization Only

When the user has a clean dataset and wants interactive visualizations:

python3 scripts/create_dashboard.py clean_data.csv ./visualizations 8050

This creates a full dashboard without any preprocessing.

Use Case D: Custom Imputation Strategy

When the user wants to review and adjust imputation strategies:

1. Run analysis first:

   python3 scripts/analyze_missing_values.py data.csv analysis.json

2. Review `analysis.json` and discuss strategies with the user

3. If needed, modify the imputation logic or parameters in the script

4. Run imputation:

   python3 scripts/impute_missing_values.py data.csv analysis.json data_imputed.csv

Understanding Imputation Methods

The skill uses intelligent imputation strategies based on data characteristics. Key methods include:

• **Mean/Median**: For numeric data (mean for normal distributions, median for skewed)
• **Mode**: For categorical variables (most frequent value)
• **KNN (K-Nearest Neighbors)**: For multivariate numeric data where variables are correlated
• **Forward Fill**: For time series data (carry last observation forward)
• **Interpolation**: Fo