/exploratory-data-analysis

name: exploratory-data-analysis
description: >-
  Methodology for exploratory data analysis on scientific files. Decision
  frameworks by data type (tabular, sequence, image, spectral, structural,
  omics), quality assessment, report generation, format detection across 200+
  formats. Use when given a data file for initial exploration or to pick an
  analysis before a pipeline.
license: CC-BY-4.0

Exploratory Data Analysis for Scientific Data

Overview

Exploratory data analysis (EDA) is the systematic examination of scientific data files to understand their structure, content, quality, and characteristics before formal analysis. This knowhow covers methodology for detecting file types, selecting appropriate analysis approaches, assessing data quality, and generating comprehensive reports across all major scientific data domains.

Key Concepts

Scientific Data Type Categories

| Category | Common Formats | Typical Analysis | Key Libraries | |----------|---------------|-----------------|---------------| | **Tabular** | CSV, TSV, XLSX, Parquet | Summary statistics, distributions, correlations, missing values | pandas, polars | | **Sequence** | FASTA, FASTQ, SAM/BAM | Length distribution, quality scores, GC content, alignment stats | BioPython, pysam | | **Image/Microscopy** | TIFF, ND2, CZI, DICOM | Dimensions (XYZCT), intensity stats, metadata, calibration | tifffile, aicsimageio, nd2reader | | **Spectral** | mzML, SPC, JCAMP, FID | Peak detection, baseline, S/N ratio, resolution | pymzml, nmrglue, pyteomics | | **Structural** | PDB, CIF, MOL, SDF | Atom counts, bond validation, B-factors, completeness | BioPython, RDKit, MDAnalysis | | **Array/Tensor** | NPY, HDF5, Zarr, NetCDF | Shape, dtype, value range, NaN/Inf check, chunk structure | numpy, h5py, zarr, xarray | | **Omics** | H5AD, MTX, VCF, BED | Feature/sample counts, sparsity, annotation completeness | scanpy, pyranges, cyvcf2 |

Format Detection Strategy

1. **Extension-based**: Map file extension to category (primary method) 2. **Magic bytes**: Check file header for binary format identification (HDF5: `\x89HDF`, GZIP: `\x1f\x8b`) 3. **Content sniffing**: For ambiguous extensions (.txt, .dat, .csv), inspect first lines for delimiters, headers, or format markers 4. **Compound extensions**: Handle `.ome.tiff`, `.nii.gz`, `.tar.gz` by checking from the rightmost extension inward

Data Quality Dimensions

• **Completeness**: Missing values, empty fields, truncated records
• **Consistency**: Matching dimensions, compatible dtypes, cross-field validation
• **Validity**: Values within expected ranges, proper encoding, format compliance
• **Provenance**: Instrument metadata, software versions, processing history
• **Uniqueness**: Duplicate records, redundant features

Decision Framework

Data file received
├── What is the file type?
│   ├── Known extension → Look up in format reference
│   ├── Unknown extension → Magic bytes / content sniffing
│   └── Directory (e.g., .d, .zarr) → Check internal structure
│
├── What category does it belong to?
│   ├── Tabular → Summary stats, distributions, correlations
│   ├── Sequence → Length/quality distributions, composition
│   ├── Image → Dimensions, channels, intensity, metadata
│   ├── Spectral → Peaks, baseline, resolution, S/N
│   ├── Structural → Atom/bond validation, geometry checks
│   ├── Array → Shape, dtype, value range, sparsity
│   └── Omics → Feature counts, sample QC, annotation check
│
├── How large is the file?
│   ├── Small (<100 MB) → Load fully, comprehensive analysis
│   ├── Medium (100 MB–1 GB) → Sample or lazy evaluation
│   └── Large (>1 GB) → Stream/chunk, representative sampling
│
└── What is the analysis goal?
    ├── Pre-pipeline QC → Focus on completeness, format compliance
    ├── Data understanding → Statistics, distributions, patterns
    ├── Troubleshooting → Compare against expected format/values
    └── Documentation → Full report with recommendations

Quick Reference: Analysis Approach by Data Type

| Data Type | First Check | Core Analysis | Visualization | |-----------|------------|---------------|---------------| | Tabular | dtypes, shape, nulls | describe(), correlations, outliers | histograms, scatter, heatmap | | Sequence | record count, format | length dist., quality, composition | quality plots, length histogram | | Image | dimensions, bit depth | intensity stats, channel info | thumbnail, histogram | | Spectral | scan count, m/z range | peak detection, TIC, baseline | spectrum plot, TIC chromatogram | | Structural | atom/residue count | B-factors, missing residues | Ramachandran, contact map | | Array | shape, dtype | statistics, NaN check | slice visualization | | Omics | genes × cells matrix | sparsity, QC metrics | violin plots, PCA |

Best Practices

1. **Always check file integrity first** — verify file is complete (not truncated) and readable before deep analysis. Check file size against expectations 2. **Sample large files before full analysis** — for files with millions of records, analyze a representative sample (first N records, random sample, or stratified sample) to get quick feedback 3. **Use lazy/streaming readers when available** — `pl.scan_parquet()`, `h5py` dataset slicing, `pysam` indexed access prevent memory overflows 4. **Validate metadata against data** — cross-check stated dimensions vs actual data, verify column count matches header, confirm timestamps are monotonic 5. **Report data quality quantitatively** — "5.2% missing values in column X" is more useful than "some missing values". Include completeness percentages, outlier counts, and format compliance scores 6. **Consider data provenance** — note instrument type, software version, processing steps, and any preprocessing already applied. This context affects downstream analysis choices 7. **Generate actionable recommendations** — don't just describe the data; suggest specific preprocessing steps (normalization method, imputation str