/analytical-pm

name: analytical-pm
description: "Structured analytical and metrics framework for AI product roles. Covers: metrics, goal-setting, root-cause analysis, trade-offs, A/B tests."
argument-hint: "[interview question]"

Analytical PM Skill

Apply a structured framework to PM analytical, metrics, root-cause, and trade-off questions targeting AI product roles.

When to Use

• User asks "What metrics would you use for X"
• User asks "How would you measure success for X"
• User asks "Metric X dropped 20%, diagnose it"
• User asks about trade-offs between two product decisions
• User asks "Define a North Star metric for X"
• User says `/analytical-pm` followed by a question
• Any question about metrics, goals, root-cause analysis, A/B tests, or trade-offs

Context

• **Tuned for**: AI product roles at frontier AI companies
• **What matters**: Translating product intuition into measurable outcomes and debugging complex systems with data
• **Common pitfall**: Picking vanity metrics or being too qualitative. Be rigorous and quantitative.

Three Question Types

TYPE A: Metrics / Goal-Setting Questions

"Define success metrics for X" / "What would you measure for X" / "Set goals for X"

**Framework: Analytical (6 Steps)**

Step 1: Clarify the Product

• What is the product? Who uses it? What value does it deliver?
• What stage is it in? (launch, growth, mature, declining)
• What's the business model? (subscription, API usage, freemium, enterprise)

Step 2: Define the North Star Metric (NSM)

The NSM must capture the **core value exchange** between product and user.

• Formula: NSM = [engagement unit] per [user segment] per [time period]
• Example (ChatGPT): # of successful conversations per weekly active user
• Example (LLM API platform): # of API calls generating production value per monthly active developer
• Example (Claude): # of tasks completed per weekly active user

**Decompose the NSM** into a metric tree:

NSM = Factor A x Factor B x Factor C

Step 3: Supporting Metrics (3-5)

Leading indicators that the NSM will grow. Organized by AARRR:

• **Acquisition**: New users/developers, sign-up conversion
• **Activation**: First successful use, time-to-value
• **Retention**: D7/D30 retention, usage frequency
• **Revenue**: ARPU, conversion to paid, API spend
• **Referral**: Organic invites, word-of-mouth, virality coefficient

Step 4: Counter / Guardrail Metrics (2-3)

What we must NOT break while optimizing the NSM:

• **Quality**: Response accuracy, hallucination rate, harmful content rate
• **Safety**: Content policy violations, user reports, model refusals (false positive rate)
• **Trust**: User satisfaction (CSAT/NPS), enterprise churn, data privacy incidents
• **System**: Latency (TTFT, TPS), error rate, uptime

Step 5: Ecosystem Metrics

For platform companies, measure ecosystem health:

• Developer ecosystem: # of apps built, API integrations, plugin adoption
• Partner ecosystem: Revenue through partners, integration depth
• Content ecosystem: User-generated content, model fine-tunes, custom GPTs

Step 6: Trade-offs Between Metrics

Identify 2-3 key tensions:

• Growth vs. Safety (more users vs. more moderation needed)
• Speed vs. Quality (faster responses vs. more accurate responses)
• Revenue vs. Access (monetization vs. mission of broad access)

State how you'd resolve each (e.g., set guardrail thresholds, A/B test, phased rollout).

---

TYPE B: Root-Cause / Diagnostic Questions

"Metric X dropped 20% this week. Diagnose it."

**Framework: MECE (Mutually Exclusive, Collectively Exhaustive)**

Step 1: Clarify

• Which metric exactly? Over what timeframe? What's the baseline?
• Is this relative or absolute? Sudden or gradual?
• Any known events (launches, incidents, seasonality)?

Step 2: Segment to Isolate

Break the metric down systematically:

• **By user segment**: New vs. existing, free vs. paid, geography, platform (web/mobile/API)
• **By product surface**: Which feature/page/endpoint is affected?
• **By time**: When exactly did the drop start? Correlated with any deploy/event?
• **By funnel stage**: Where in the funnel is the drop?

Step 3: Hypothesize (MECE)

Generate hypotheses that are mutually exclusive and collectively exhaustive:

**Internal factors:**

• Product change (new deploy, A/B test, feature removal)
• Technical issue (latency increase, outage, bug, model regression)
• Data/instrumentation issue (logging break, tracking change, attribution error)

**External factors:**

• Seasonality (holiday, weekend, school schedule)
• Competitor action (new feature launch, pricing change)
• Market event (news cycle, regulatory change, viral moment)
• Platform change (app store policy, browser update, API deprecation)

Step 4: Validate

For each hypothesis, state:

• What data would confirm/deny it
• What team/tool you'd use to investigate
• Priority order for investigation

Step 5: Recommend Action

• Short-term: Immediate mitigation
• Medium-term: Root cause fix
• Long-term: Monitoring/alerting to catch this earlier

---

TYPE C: Trade-off Questions

"Feature A would increase engagement but decrease revenue. Ship or not?"

**Framework: 3 Trade-off Types**

Type 1: Similar Product Cannibalization

Product A vs. Product B serving overlapping users.

• Quantify cannibalization risk (user overlap, usage substitution)
• Measure incremental value (does total pie grow?)
• Run holdout experiment

Type 2: Same Product, Different Variations

Version A vs. Version B of the same feature.

• Define ship/no-ship criteria upfront
• A/B test with clear primary metric and guardrails
• Set duration and statistical significance threshold
• Consider long-term effects (novelty bias, learning curves)

Type 3: Different Products, Same Surface

Feature X vs. Feature Y competing for the same real estate.

• Score each on: impact to NSM, strategic value, user demand, effort
• Consider: Can they coexist? Is this a false dichotomy?

-