/algo-nlp-lda

name: "\"algo-nlp-lda\""
description: "\"Implement LDA topic modeling to discover latent topics in document collections. Use this skill when the user needs to extract topics from a text corpus, categorize documents by theme, or explore thematic structure — even if they say 'what are the main topics', 'topic extraction', or 'document clustering by theme'.\"."
allowed-tools: Read, Glob, Grep

LDA Topic Modeling

Overview

Latent Dirichlet Allocation models each document as a mixture of topics and each topic as a distribution over words. Discovers K latent topics from a corpus without supervision. Uses Gibbs sampling or variational inference. Complexity: O(N × K × iterations) where N = total word tokens.

When to Use

**Trigger conditions:**

• Discovering latent themes in a large document collection
• Organizing/categorizing documents by automatically discovered topics
• Exploratory text analysis when categories are unknown

**When NOT to use:**

• When categories are known (use supervised classification)
• For short texts (tweets, titles) — too few words per document for reliable topic assignment
• When you need semantic understanding (use embeddings)

Algorithm

IRON LAW: The Number of Topics K Must Be Chosen, Not Discovered
LDA does NOT tell you how many topics exist. K is a hyperparameter.
Too few topics: overly broad, mixed themes. Too many: fragmented,
redundant topics. Use coherence score (C_v) to compare K values,
but the final choice requires human judgment on topic interpretability.

Phase 1: Input Validation

Preprocess: tokenize, remove stop words, apply lemmatization. Build document-term matrix. Filter: remove terms appearing in <5 or >50% of documents. **Gate:** Clean DTM, vocabulary size reasonable (1K-50K terms).

Phase 2: Core Algorithm

1. Choose K (start with √(N/2), try range K=5,10,15,20,...) 2. Set hyperparameters: α = 50/K (document-topic density), β = 0.01 (topic-word density) 3. Run LDA (Gibbs sampling: 1000+ iterations, or variational inference) 4. Extract: topic-word distributions (top 10-20 words per topic) and document-topic distributions

Phase 3: Verification

Evaluate: topic coherence (C_v score, higher is better), manual inspection of top words per topic, check for "junk" topics (mixed incoherent words). **Gate:** Coherence score acceptable, topics are humanly interpretable.

Phase 4: Output

Return topics with top words and document assignments.

Output Format

{
  "topics": [{"id": 0, "label": "finance", "top_words": ["revenue", "profit", "quarter", "growth"], "coherence": 0.55}],
  "doc_topics": [{"doc_id": "d1", "dominant_topic": 0, "topic_distribution": [0.7, 0.1, 0.2]}],
  "metadata": {"K": 10, "coherence_avg": 0.48, "documents": 5000, "vocabulary": 8000}
}

Examples

Sample I/O

**Input:** 1000 news articles, K=5 **Expected:** Topics like: {politics, sports, technology, business, entertainment} with coherent top words per topic.

Edge Cases

| Input | Expected | Why | |-------|----------|-----| | Very short documents | Poor topic assignment | Too few words for reliable mixture estimation | | Homogeneous corpus | 1-2 topics dominate | All documents are similar, limited topic diversity | | K=1 | Single topic = corpus vocabulary | Degenerate case, no discrimination |

Gotchas

• **Stop words MUST be removed**: LDA will create "junk" topics dominated by common words ("the", "is", "and") if stop words remain.
• **Topic labeling is manual**: LDA gives word distributions, NOT topic names. You must interpret and label topics based on top words.
• **Reproducibility**: Gibbs sampling is stochastic. Different random seeds give different topics. Run multiple times and check stability.
• **Dynamic topics**: Standard LDA assumes topics are static. For evolving corpora (news over years), use Dynamic Topic Models.
• **Hyperparameter sensitivity**: Low α produces documents with fewer, more distinct topics. Low β produces topics with fewer, more specific words. Tune or use automatic methods.

References

• For coherence metrics and K selection, see `references/topic-evaluation.md`
• For dynamic and correlated topic models, see `references/advanced-lda.md`