/agentsop-observability-setup

name: agentsop-observability-setup
version: 0.1.0
description: |
  Enhancement-overlay skill — the DECISION + WIRING layer for LM observability that the
  single-backend skills [[langsmith]], [[phoenix]], [[mlflow]] do NOT cover. Each of those
  installs one backend; none of them help you DECIDE which backend fits your stack/scale/budget,
  nor give you a one-line autolog that turns it on fast. Use when starting any LM project,
  before the first deploy, or the moment someone asks "why did it do that?" and there are no traces
  to answer with. The skill picks a backend by stack (LangSmith for LangChain/LangGraph; Phoenix
  for OSS/local OpenTelemetry; MLflow for ML-shops already on MLflow; Langfuse for self-host),
  wires one-line autolog, verifies traces land, and adds eval hooks — instrumenting BEFORE you
  need it. Cross-links the first-debug-move skill [[agentsop-prompt-history-inspect]]. Do NOT activate to
  re-teach a backend you already chose (defer to its own skill), or for non-LM ML experiment
  tracking with no LLM calls (that is plain MLflow).

Observability Setup — Which Backend + One-Line Autolog

> *"Instrument before you need it. The cheapest debugging session is a trace you already have."*

This is an **ENHANCE overlay**. The local skills [[langsmith]], [[phoenix]] and [[mlflow]] each teach **one** backend deeply. This skill sits one level up: it answers the question those skills cannot — *which one, and how do I turn it on in a single line right now* — then hands off to the chosen backend's own skill for depth.

---

1. 何时激活 (When to activate)

Activate at one of three moments — earlier is always cheaper:

| Trigger | Signal | |---|---| | **Starting any LM project** | First `dspy.LM` / `ChatOpenAI` / `LlamaIndex` / agent graph in the repo, and no tracing wired yet | | **Before first deploy** | About to ship an LM feature to real users with no trace UI — the AP-15 trap (§6) | | **"Why did it do that?" with no traces** | A bug surfaced, you reach for history, and there is nothing recorded — you are debugging blind | | **Multi-component pipeline** | Retriever + reranker + LLM + tools; per-call printing ([[agentsop-prompt-history-inspect]]) is no longer enough | | **Cost / latency regression** | Need aggregate token & latency dashboards across runs, not a single printout |

**Do NOT activate** when:

• You have already chosen a backend and just need its API — defer to [[langsmith]] / [[phoenix]] / [[mlflow]] directly.
• The task is classic ML experiment tracking with **no LLM calls** — that is plain MLflow, no decision needed.
• You only need to read **one** rendered prompt right now — that is [[agentsop-prompt-history-inspect]] (the cheaper first move).

The relationship to [[agentsop-prompt-history-inspect]]: that skill is the *reactive* first move (dump one prompt, no setup). This skill is the *proactive* layer — wire persistent tracing so the next "why" is answered by a trace that already exists, not a frantic re-run.

---

2. 核心心智模型 (Core mental model)

   Instrument BEFORE you need it
   ───────────────────────────────────────────────
   pick backend      one-line          verify         add eval
   by constraints →  autolog turns  →  a trace    →   hooks (judge,
   (stack/scale/      it on            actually        datasets,
    budget)           (~1 line)        landed          alerts)
   ───────────────────────────────────────────────
   cost of skipping each stage compounds: a missing trace at deploy
   becomes a multi-hour blind-debug later (AP-15, §6).

Three load-bearing ideas:

1. **Choosing is the hard part, not wiring.** Every modern backend offers ~one-line autolog (`langsmith` env vars, `px.launch_app()`, `mlflow.<framework>.autolog()`). The real cost is picking the one that won't lock you in or under-serve you at scale (§4 decision table, §5 cases). 2. **The cheapest debugging is a trace you already have.** A trace recorded proactively costs near-zero; reconstructing one after a production bug costs hours and may be impossible (no repro). 3. **One backend, owned end-to-end.** Don't wire two trace backends "to be safe" — pick one, standardize on it, and let OpenTelemetry GenAI semantics keep you portable if you outgrow it (§7).

---

3. SOP (Standard operating procedure)

A five-step path. Each step gates the next.

Step 1 — Pick the backend by constraints

Run the decision table in §4 (OP-1). Inputs: **stack** (is it LangChain/LangGraph? OSS-only? already on MLflow?), **scale** (dev-only vs high-volume production), **budget/hosting** (managed-OK vs must-self-host). Output: exactly one backend.

Step 2 — One-line autolog

Wire the single call/env-var for the chosen backend (OP-2). Resist building a custom tracing layer first — autolog gets you a trace today; you can refine later.

Step 3 — Verify traces actually land

Run one real LM call, then open the trace UI and confirm the call appears with inputs, outputs, latency, and token counts (OP-3). A backend that is "configured" but shows no traces is the #1 silent failure (wrong project name, env var not exported, sampling at 0).

Step 4 — Add eval hooks

Once raw traces flow, attach what makes them actionable: an LLM-as-judge or rule evaluator, a dataset built from real traces for regression testing, and basic cost/latency alerts (OP-4). This is what turns "we have logs" into "we catch regressions before users do."

Step 5 — Hand off to the backend skill

Hand off to the chosen backend's own skill ([[langsmith]] / [[phoenix]] / [[mlflow]]) for the deep API once tracing and evals are flowing (OP-6).

**Step gating, explicitly.** Do not start Step *n+1* until Step *n* is observably true. The most common failure is jumping from Step 2 (wired) straight to Step 4 (evals) without Step 3 (verify) — you build evaluators on top of a trace stream that was silently empty the whole time. Each step has a one-line proof: Step 1 → a backend name written down; Step 2 → a