datadog-labs-agent-skills

Datadog Skills for AI Agents

Datadog skills for Claude Code, Codex CLI, Gemini CLI, Cursor, Windsurf, OpenCode, and other AI agents.

Skills

Install

Setup Pup

# Homebrew (macOS/Linux) — recommended
brew tap datadog-labs/pack
brew install datadog-labs/pack/pup

# Or build from source
git clone https://github.com/datadog-labs/pup.git && cd pup
cargo build --release
cp target/release/pup ~/.local/bin

Pre-built binaries are also available from the latest release.

# Authenticate
pup auth login

Add Skill(s)

For JUST dd-pup:

npx skills add datadog-labs/agent-skills \
  --skill dd-pup \
  --full-depth -y

For ALL skills:

npx skills add datadog-labs/agent-skills \
  --skill dd-pup \
  --skill dd-monitors \
  --skill dd-logs \
  --skill dd-apm \
  --skill dd-docs \
  --skill dd-browser-sdk \
  --skill dd-audit \
  --skill service-remapping \
  --skill enable-dsm \
  --skill agent-install \
  --skill enable-ssi \
  --skill verify-ssi \
  --skill troubleshoot-ssi \
  --skill onboarding-summary \
  --skill upgrade-browser-sdk-v7 \
  --skill setup-browser-sdk-sourcemaps \
  --skill dd-audit-security-investigation \
  --skill dd-audit-key-compromise \
  --skill dd-audit-cost-spike-investigation \
  --skill dd-audit-compliance-report \
  --skill dd-audit-ai-activity \
  --skill agent-observability-experiment-analyzer \
  --skill agent-observability-experiment-bootstrap \
  --skill agent-observability-trace-rca \
  --skill agent-observability-eval-bootstrap \
  --skill agent-observability-eval-pipeline \
  --skill agent-observability-session-classify \
  --skill agent-observability-auto-experiment \
  --skill agent-observability-build-eval-from-annotations \
  --skill agent-observability-replay-trace \
  --skill agent-observability-online-experiment \
  --skill k9-ownership-byod-setup \
  --full-depth -y

Agent Observability (LLMO)

The agent-observability directory contains ten skills for working with Agent Observability data:

Eval pipeline flow:

agent-observability-session-classify    agent-observability-trace-rca → agent-observability-eval-bootstrap
 (classify sessions)          (diagnose why)      (build evals)

Run agent-observability-trace-rca to understand why an app is failing by analyzing eval judge verdicts or runtime errors across production traces. Then run agent-observability-eval-bootstrap to generate evaluator code that captures those failure patterns. Pass the RCA output directly to agent-observability-eval-bootstrap to seed it with the discovered failure taxonomy.

Use agent-observability-eval-pipeline to run all three steps in sequence with checkpoints between each phase.

Use agent-observability-session-classify independently to evaluate whether individual assistant sessions satisfied user intent, combining Agent Observability trace data with RUM behavioral signals.

Use agent-observability-build-eval-from-annotations when an annotation queue already holds human labels: it fits an LLM judge to those labels, scores it against them, iterates on its errors, and publishes the winner as a disabled evaluator draft. agent-observability-eval-bootstrap is the no-labels counterpart — it proposes evaluators from trace inspection alone, with nothing to measure them against.

Use agent-observability-experiment-bootstrap to bootstrap a reproducible experiment through the Python ddtrace.llmobs SDK or the Node dd-trace SDK. Python remains the default adapter; generated artifacts can use inline records, local files, or named Datadog datasets.

The bootstrap skill keeps adapter-specific contracts in its references/ directory and loads only the selected Python or Node SDK reference. Python provider and evaluator-style references live under references/python/ and are loaded separately when needed.

Install

# Claude Code — copy any or all skills
cp -r agent-observability/agent-observability-experiment-analyzer ~/.claude/skills
cp -r agent-observability/agent-observability-experiment-bootstrap ~/.claude/skills
cp -r agent-observability/agent-observability-trace-rca ~/.claude/skills
cp -r agent-observability/agent-observability-eval-bootstrap ~/.claude/skills
cp -r agent-observability/agent-observability-eval-pipeline ~/.claude/skills
cp -r agent-observability/agent-observability-session-classify ~/.claude/skills
cp -r agent-observability/agent-observability-online-experiment ~/.claude/skills

MCP Requirements

To use these skills, add the following MCP toolsets to your agent:

claude mcp add --scope user --transport http "datadog-llmo-mcp" 'https://mcp.datadoghq.com/v1/mcp?toolsets=llmobs,experiments,feature-flags,core'

Usage

# Analyze experiments
experiment-analyzer <experiment_id>                         # single experiment
experiment-analyzer <baseline_id> <candidate_id>            # compare two experiments
experiment-analyzer <id(s)> <question>                      # ask a specific question
experiment-analyzer <id(s)> [question] --output notebook    # export to Datadog notebook

# Root-cause why an app is failing
What's wrong with <ml_app> based on its evals over the last 24h
Analyze eval failures for <eval_name> over the last week
Look at the errors on <ml_app> over the last 24h

# Generate evaluator code from production traces
/eval-bootstrap <ml_app>                                    # cold start
/eval-bootstrap <ml_app> [paste eval-trace-rca output here] # seeded from RCA
/eval-bootstrap <ml_app> --data-only                        # emit JSON spec instead of Python SDK code

# Bootstrap an experiment (Python SDK remains the default)
/agent-observability-experiment-bootstrap                                                  # 3-record inline Python sample
/agent-observability-experiment-bootstrap --dataset ./data/qa.json --format ipynb          # local JSON dataset, Python notebook
/agent-observability-experiment-bootstrap --dataset-name qa_v3 --project-name customer-qa  # existing Datadog dataset
/agent-observability-experiment-bootstrap --evaluator-style remote                         # server-side RemoteEvaluator stubs
/agent-observability-experiment-bootstrap --adapter node --format mjs --task-source app:answer # Node SDK artifact

# Create a live-traffic Agent Observability experiment
/agent-observability-online-experiment

# Classify a session
/eval-session-classify <session_id>

# Guided end-to-end pipeline (6 narrated phases — classify → RCA → eval bootstrap → dataset → experiment → analyze)
/agent-observability-eval-pipeline <ml_app>
/agent-observability-eval-pipeline <ml_app> --timeframe now-30d --trace-limit 25 --format ipynb

Software Delivery (dd-software-delivery)

The dd-software-delivery directory contains workflow skills for CI/CD visibility and test reliability:

Workflow:

unblock-pr → (if flaky failure) → triage-flaky-test → quarantine or fix

Backend

Both skills auto-detect the available backend at runtime:

• MCP mode (preferred): uses the Datadog software-delivery MCP tools (search_datadog_ci_pipeline_events, get_datadog_flaky_tests, retry_datadog_ci_job, etc.). Enables PR quality/security insights and native GitHub Actions retry.
• pup mode (fallback): uses the pup CLI. PR quality/security data is not available; GitHub Actions retry falls back to gh run rerun.

Pass --backend pup to force pup mode regardless of MCP availability.

MCP Requirements

Connect the Datadog MCP server with the software-delivery toolset:

claude mcp add --scope user --transport http "datadog-mcp" \
  'https://mcp.datadoghq.com/v1/mcp?toolsets=core,software-delivery'

Prerequisites

Requires pup CLI for pup mode (and as a fallback). See Setup Pup.

Install

# Claude Code — copy any or all skills
cp -r dd-software-delivery/unblock-pr ~/.claude/skills
cp -r dd-software-delivery/triage-flaky-test ~/.claude/skills

Or via npx:

npx skills add datadog-labs/agent-skills \
  --skill dd-software-delivery/unblock-pr \
  --skill dd-software-delivery/triage-flaky-test \
  --full-depth -y

Usage