/mkt-experiment

name: mkt-experiment
description: Autonomous growth experimentation framework. Creates A/B/multivariate experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Use when creating or managing marketing experiments, logging data points, scoring experiments, or generating weekly scorecards.

Growth Engine

Autonomous growth experimentation framework based on Karpathy's autoresearch pattern applied to marketing. Creates experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Supports batch mode (up to 10 variants simultaneously).

Usage

Use this skill when:

• Creating or managing A/B or multivariate experiments for any marketing channel
• Logging experiment data points after content is published or campaigns run
• Scoring experiments to determine statistical winners
• Checking the playbook for proven best practices before creating new content
• Generating weekly scorecards across all channels
• Monitoring campaign pacing and health

Do NOT use for:

• One-off content creation (use the playbook output as input, but don't run the engine)
• Non-experiment analytics or reporting
• Campaign setup in external platforms (this tracks experiments, not campaign config)

Commands

Create an experiment

python3 experiment-engine.py create \
  --agent <agent_name> \
  --hypothesis "What you expect to happen" \
  --variable "<variable_name>" \
  --variants '["variant_a", "variant_b"]' \
  --metric "<primary_metric>" \
  --cycle-hours 24

Add `--batch-mode` for 3-10 variant tests. Add `--min-samples N` to override auto-detection.

Log a data point

python3 experiment-engine.py log \
  --agent <agent_name> \
  --experiment-id <EXP-ID> \
  --variant "<variant_name>" \
  --metrics '{"metric_name": value}'

Score an experiment

python3 experiment-engine.py score --agent <agent_name> --experiment-id <EXP-ID>

Statuses: `running` → `trending` → `keep` (winner) or `discard` (loser)

Winners auto-promote to the playbook. Requires p < 0.05 AND ≥ 15% lift.

List experiments

python3 experiment-engine.py list --agent <agent_name> [--status running|trending|keep|discard]

Check the playbook

python3 experiment-engine.py playbook --agent <agent_name>

Always check the playbook before creating new content to apply proven best practices.

Suggest next experiments

python3 experiment-engine.py suggest --agent <agent_name>

Generate weekly scorecard

python3 autogrowth-weekly-scorecard.py [--weeks N] [--output file.md]

Check campaign pacing

python3 pacing-alert.py [--json]

Exit code 0 = on pace, 1 = alerts present.

Workflow

1. Before creating content: `playbook` → apply proven rules 2. When publishing: `log` → record which variant was used and its metrics 3. Periodically: `score` → check if experiments have reached statistical significance 4. Weekly: `autogrowth-weekly-scorecard.py` → review all channels 5. After completing experiments: `suggest` → pick the next variable to test

Configuration

Required Environment Variables

| Variable | Description | |----------|-------------| | `GROWTH_ENGINE_DATA_DIR` | Data directory (default: `./data/experiments`) | | `GROWTH_ENGINE_AGENTS` | Comma-separated agent names (default: `content,email,linkedin,seo,blog`) |

Optional Tuning

| Variable | Default | Description | |----------|---------|-------------| | `HIGH_VOLUME_AGENTS` | `content,email` | Agents needing only 10 samples/variant | | `LOW_VOLUME_AGENTS` | `seo,linkedin,blog` | Agents needing 30 samples/variant | | `P_WINNER` | `0.05` | p-value threshold for winner | | `P_TREND` | `0.10` | p-value threshold for trending | | `LIFT_WIN` | `15.0` | Minimum % lift for keep decision | | `BOOTSTRAP_ITERATIONS` | `1000` | Bootstrap resamples for CI | | `BATCH_MODE_MAX_VARIANTS` | `10` | Max variants in batch mode |

Pacing Alert Variables

| Variable | Description | |----------|-------------| | `PIPELINE_API_URL` | Pipeline/CRM API endpoint | | `PIPELINE_AUTH_TOKEN` | Bearer token for pipeline API | | `RECRUITING_API_URL` | Recruiting API endpoint | | `RECRUITING_AUTH_TOKEN` | Bearer token for recruiting API | | `EMAIL_API_URL` | Email platform API base URL | | `EMAIL_AUTH_TOKEN` | Bearer token for email platform | | `OUTBOUND_CAMPAIGNS` | JSON: `{"name": "campaign-id"}` | | `RECRUITING_CAMPAIGNS` | JSON: `{"name": "campaign-id"}` | | `DAILY_LEAD_TARGET` | Leads/day target (default: 10) | | `WEEKLY_CANDIDATE_TARGET` | Candidates/week target (default: 400) |

Dependencies

pip install numpy scipy