/degenerate-input-filtering

name: degenerate-input-filtering
description: "Filter degenerate, uninformative inputs before statistical tests: single-sequence alignments, empty files, constant features, zero-variance inputs, all-NaN columns. See nan-safe-correlation for NaN-aware correlation; statistical-analysis for test guidance."
license: CC-BY-4.0

Degenerate Input Filtering Guide

Overview

Degenerate inputs are data points that carry no statistical information: constant-value features, all-NaN columns, single-sequence alignments, empty files, and similar edge cases. When these reach a statistical test or model, the result is meaningless -- a correlation of NaN, a p-value of 1.0, a score of 0.0, or an outright crash. This guide establishes the mandatory practice of detecting and removing such inputs before any analysis, and of reporting every removal so that downstream consumers know the effective sample size.

Key Concepts

What Counts as Degenerate

A data point is degenerate when it cannot contribute to the statistic being computed. The root cause is always the same: the input lacks the variation or completeness that the method requires.

| Type | Example | Why It Fails | |---|---|---| | Constant-value feature | Gene with identical expression across all samples | Variance = 0; correlation, t-test, fold-change are all undefined | | All-NaN feature | Column with no valid observations | Every aggregation returns NaN | | Single-sequence alignment | BLAST result with one sequence | Score = 0.0; no pairwise comparison is possible | | Empty file | 0-byte FASTA or CSV | Parser crashes or returns an empty frame | | Single value after grouping | One sample in a treatment group | Within-group variance is undefined; group comparison is meaningless | | Zero-length sequence | Empty FASTA entry | Alignment and k-mer tools fail or produce nonsense | | Near-constant feature | Gene with one outlier and N-1 identical values | Technically non-zero variance but correlation is dominated by a single point |

Why Silent Failures Are Dangerous

Many numerical libraries do not raise errors on degenerate input. Instead they return sentinel values:

• `numpy.corrcoef` returns `nan` for constant columns without warning.
• `scipy.stats.spearmanr` returns `(nan, nan)` when one array is constant.
• `scipy.stats.ttest_ind` returns `(nan, nan)` for zero-variance groups.

These NaN values propagate silently through pipelines, contaminating aggregated statistics, heatmaps, volcano plots, and ranked gene lists. By the time a researcher notices the problem, it may be unclear which upstream step introduced the NaN.

The Reporting Obligation

Filtering without reporting is nearly as harmful as not filtering. When items are silently dropped, the effective sample or feature count differs from what the user expects, leading to incorrect power calculations and misleading summary statistics. Every filtering step must print:

1. The count of items removed. 2. The reason for removal. 3. The count of items remaining.

Decision Framework

Use this tree to determine which filtering checks apply to your data before running a statistical analysis:

Is the input tabular (DataFrame)?
├── Yes
│   ├── Are there columns with zero unique values (all NaN)? → Remove, report count
│   ├── Are there columns with exactly one unique value (constant)? → Remove, report count
│   ├── Are there columns with fewer than N valid observations? → Remove, report count
│   └── Are there near-constant columns (1 outlier, rest identical)? → Flag or remove
└── No (file list, sequence set, etc.)
    ├── Are there empty files (0 bytes)? → Skip, report count
    ├── Are there single-entry collections (e.g., 1-sequence alignment)? → Skip, report count
    └── Are there zero-length entries within a file? → Filter entries, report count

| Scenario | Recommended Check | Threshold | |---|---|---| | Gene expression matrix before DE analysis | Remove zero-variance genes, remove genes detected in fewer than N samples | N = max(3, 10% of samples) | | Correlation analysis between two feature sets | Remove features constant in either set; require >= 10 shared non-NaN pairs | nunique >= 2 in both vectors; shared observations >= 10 | | Multiple sequence alignment scoring | Skip alignments with < 2 sequences | Sequence count >= 2 | | Survival analysis with grouped covariates | Remove groups with < 2 events | Events per group >= 2 | | PCA or clustering on a feature matrix | Remove zero-variance and all-NaN features | Variance > 0 and at least 1 non-NaN value | | Batch correction (e.g., ComBat) | Remove features absent in any batch | Feature detected in all batches |

Best Practices

1. **Filter before any statistical computation, not after.** Degenerate inputs can cause division-by-zero, NaN propagation, and inflated multiple-testing corrections. Filtering after the test means the damage is already done.

2. **Use a single reusable filtering function.** Centralizing the logic prevents inconsistencies between scripts and ensures the reporting format is uniform. See the reference implementation in the Workflow section below.

3. **Print counts at every filtering step.** Even when the count is zero, printing "0 items removed" confirms that the check ran. This makes logs self-documenting and auditable.

4. **Set domain-appropriate thresholds rather than relying on generic defaults.** A gene expression analysis might require detection in at least 10% of samples, while a proteomics dataset with more missingness might use 5%. Document the threshold and the rationale.

5. **Treat near-constant features with caution.** A feature with one non-identical value technically has non-zero variance, but its correlation with anything is driven entirely by that single point. Consider flagging these separately rather than including them blindly.

6. **Never filter silently.** Dropping data without reporting is a form of hidden data manipulation. Every removal must be logged with th