/exploratory-data-analysis

name: exploratory-data-analysis
description: "Perform comprehensive exploratory data analysis on scientific data files across 200+ file formats. This skill should be used when analyzing any scientific data file to understand its structure, content, quality, and characteristics. Automatically detects file type and generates detailed markdown reports with format-specific analysis, quality metrics, and downstream analysis recommendations. Covers chemistry, bioinformatics, microscopy, spectroscopy, proteomics, metabolomics, and general scientific data formats."
allowed-tools: Bash, Read, Write, Edit, Glob, Grep

Exploratory Data Analysis

Overview

Perform comprehensive exploratory data analysis (EDA) on scientific data files across multiple domains. This skill provides automated file type detection, format-specific analysis, data quality assessment, and generates detailed markdown reports suitable for documentation and downstream analysis planning.

**Key Capabilities:**

• Automatic detection and analysis of 200+ scientific file formats
• Comprehensive format-specific metadata extraction
• Data quality and integrity assessment
• Statistical summaries and distributions
• Visualization recommendations
• Downstream analysis suggestions
• Markdown report generation

When to Use This Skill

Use this skill when:

• User provides a path to a scientific data file for analysis
• User asks to "explore", "analyze", or "summarize" a data file
• User wants to understand the structure and content of scientific data
• User needs a comprehensive report of a dataset before analysis
• User wants to assess data quality or completeness
• User asks what type of analysis is appropriate for a file

Supported File Categories

The skill has comprehensive coverage of scientific file formats organized into six major categories:

1. Chemistry and Molecular Formats (60+ extensions)

Structure files, computational chemistry outputs, molecular dynamics trajectories, and chemical databases.

**File types include:** `.pdb`, `.cif`, `.mol`, `.mol2`, `.sdf`, `.xyz`, `.smi`, `.gro`, `.log`, `.fchk`, `.cube`, `.dcd`, `.xtc`, `.trr`, `.prmtop`, `.psf`, and more.

**Reference file:** `references/chemistry_molecular_formats.md`

2. Bioinformatics and Genomics Formats (50+ extensions)

Sequence data, alignments, annotations, variants, and expression data.

**File types include:** `.fasta`, `.fastq`, `.sam`, `.bam`, `.vcf`, `.bed`, `.gff`, `.gtf`, `.bigwig`, `.h5ad`, `.loom`, `.counts`, `.mtx`, and more.

**Reference file:** `references/bioinformatics_genomics_formats.md`

3. Microscopy and Imaging Formats (45+ extensions)

Microscopy images, medical imaging, whole slide imaging, and electron microscopy.

**File types include:** `.tif`, `.nd2`, `.lif`, `.czi`, `.ims`, `.dcm`, `.nii`, `.mrc`, `.dm3`, `.vsi`, `.svs`, `.ome.tiff`, and more.

**Reference file:** `references/microscopy_imaging_formats.md`

4. Spectroscopy and Analytical Chemistry Formats (35+ extensions)

NMR, mass spectrometry, IR/Raman, UV-Vis, X-ray, chromatography, and other analytical techniques.

**File types include:** `.fid`, `.mzML`, `.mzXML`, `.raw`, `.mgf`, `.spc`, `.jdx`, `.xy`, `.cif` (crystallography), `.wdf`, and more.

**Reference file:** `references/spectroscopy_analytical_formats.md`

5. Proteomics and Metabolomics Formats (30+ extensions)

Mass spec proteomics, metabolomics, lipidomics, and multi-omics data.

**File types include:** `.mzML`, `.pepXML`, `.protXML`, `.mzid`, `.mzTab`, `.sky`, `.mgf`, `.msp`, `.h5ad`, and more.

**Reference file:** `references/proteomics_metabolomics_formats.md`

6. General Scientific Data Formats (30+ extensions)

Arrays, tables, hierarchical data, compressed archives, and common scientific formats.

**File types include:** `.npy`, `.npz`, `.csv`, `.xlsx`, `.json`, `.hdf5`, `.zarr`, `.parquet`, `.mat`, `.fits`, `.nc`, `.xml`, and more.

**Reference file:** `references/general_scientific_formats.md`

Workflow

Step 1: File Type Detection

When a user provides a file path, first identify the file type:

1. Extract the file extension 2. Look up the extension in the appropriate reference file 3. Identify the file category and format description 4. Load format-specific information

**Example:**

User: "Analyze data.fastq"
→ Extension: .fastq
→ Category: bioinformatics_genomics
→ Format: FASTQ Format (sequence data with quality scores)
→ Reference: references/bioinformatics_genomics_formats.md

Step 2: Load Format-Specific Information

Based on the file type, read the corresponding reference file to understand:

• **Typical Data:** What kind of data this format contains
• **Use Cases:** Common applications for this format
• **Python Libraries:** How to read the file in Python
• **EDA Approach:** What analyses are appropriate for this data type

Search the reference file for the specific extension (e.g., search for "### .fastq" in `bioinformatics_genomics_formats.md`).

Step 3: Perform Data Analysis

Use the `scripts/eda_analyzer.py` script OR implement custom analysis:

**Option A: Use the analyzer script**

# The script automatically:
# 1. Detects file type
# 2. Loads reference information
# 3. Performs format-specific analysis
# 4. Generates markdown report

python scripts/eda_analyzer.py <filepath> [output.md]

**Option B: Custom analysis in the conversation** Based on the format information from the reference file, perform appropriate analysis:

For tabular data (CSV, TSV, Excel):

• Load with pandas
• Check dimensions, data types
• Analyze missing values
• Calculate summary statistics
• Identify outliers
• Check for duplicates

For sequence data (FASTA, FASTQ):

• Count sequences
• Analyze length distributions
• Calculate GC content
• Assess quality scores (FASTQ)

For images (TIFF, ND2, CZI):

• Check dimensions (X, Y, Z, C, T)
• Analyze bit depth and value range
• Extract metadata (channels, timestamps, spatial calibration)
• Calculate intensity statist