/agentsop-dspy

name: agentsop-dspy
version: 0.1.0
description: |
  Operating SOP for DSPy (Stanford NLP) — the declarative framework for "programming, not prompting" language models.
  Activate when the user says any of: "use DSPy", "compile a prompt", "optimize prompts/programs", "MIPRO/MIPROv2",
  "BootstrapFewShot", "GEPA", "Signatures + Modules", "teleprompter", "auto-tune prompts for a different LM",
  or whenever a brittle hand-crafted prompt pipeline needs to be turned into a *compiled*, measurable, swappable program.
  Do NOT activate for one-shot prompt tweaks, no-metric exploratory work, or pipelines where prompts must remain
  human-authored verbatim — use raw prompting or LangChain templates instead.

DSPy SOP — Programming, Not Prompting

> *"DSPy isn't a prompt-optimization agent framework. It's the LLM compiler for the shortest, cleanest code."* > — Eito Miyamura [eito.substack.com/p/dspy-the-most-misunderstood-agent] > > *"Prompts are effectively the weights of an LLM application."* > — Core philosophy [arxiv.org/abs/2310.03714]

---

1. 何时激活 (When to activate)

Activate this skill when **any** of the following triggers are present in the user's intent or codebase:

| Trigger | Signal | |---|---| | Imports / mentions | `import dspy`, `dspy.Signature`, `dspy.ChainOfThought`, `dspy.ReAct`, `Predict`, `MIPROv2`, `BootstrapFewShot`, `GEPA`, `teleprompter`, `compile(` on an LM program | | Tasks | "auto-tune this prompt", "I want to swap GPT-4 for a smaller model without re-engineering prompts", "I have 50/200/1000 labeled examples — optimize this", "compile a pipeline for our metric", "distill GPT-4 into Llama-3-8B" | | Symptoms | Hand-written prompts grow past ~50 lines; brittleness on model swap; the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design | | Cross-skill bridges | LangGraph node calls an LLM and needs better prompts → wrap the node body in a DSPy module. LlamaIndex retriever feeds a reranker → DSPy-compile the reranker against a labeled set |

**Do NOT activate** when:

• The task is one-shot ("just answer this question once") — use raw `client.messages.create`.
• No evaluation metric is possible and none is willing to be built — DSPy without a metric is just verbose prompting.
• Prompts must remain human-authored verbatim for compliance, audit, or stylistic reasons.
• The team is in *rapid exploration* mode where the task signature itself is changing daily — compile only after the signature stabilizes [dspy.ai/learn/optimization/overview/].

---

2. 核心心智模型 (Core mental model)

DSPy's full name is **D**eclarative **S**elf-improving **Py**thon. The three primitives form a PyTorch-like compile chain [arxiv.org/abs/2310.03714]:

┌─────────────┐    ┌──────────┐    ┌──────────────┐    ┌─────────┐
│  Signature  │ →  │  Module  │ →  │ Teleprompter │ →  │ Compile │
│ (what)      │    │ (how)    │    │ (optimizer)  │    │ (tune)  │
└─────────────┘    └──────────┘    └──────────────┘    └─────────┘
   I/O spec       Predict/CoT/      MIPROv2/GEPA/      Bake demos
   field names     ReAct/PoT        BootstrapFewShot   + instructions
   = semantic     = strategy        = search algorithm  into JSON

**Three mental shifts** the agent must internalize:

1. **Prompts are weights.** The prompt string is not the artifact you ship — the *compiled program* (a JSON of demonstrations + instructions + structural choices) is. You ship `program.json`, not a `.txt` prompt [dspy.ai/tutorials/saving/].

2. **Signatures carry semantic load.** `question -> answer` is not the same as `query -> response`. DSPy uses the *field names* as the only natural-language hint the optimizer has about intent before it sees data. Name them like you'd name function parameters in well-written code [dspy.ai/learn/programming/signatures/].

3. **Compile is a hyperparameter search, not a one-shot call.** Compilation runs hundreds-to-thousands of LM calls (typically $2–$3 USD, 6–20 minutes, 3.2k API calls in the reference run). Costs scale with `num_trials × |trainset| × |program LM calls|` [dspy.ai/faqs/].

**The PyTorch analogy** is load-bearing. Signatures ≈ `nn.Module.forward()` shape contract. Modules ≈ `nn.Linear` / `nn.Transformer`. Teleprompters ≈ `torch.optim.Adam`. `compile()` ≈ training loop. `save()/load()` ≈ checkpoint.

---

3. SOP 工作流 (SOP workflow)

The DSPy team is explicit about a **three-stage gate** [dspy.ai/learn/]:

> "It's unproductive to launch optimization runs using a poorly designed program or a bad metric."

Do not skip stages. Each stage has an exit criterion.

Stage 1 — Programming (no optimizer yet)

1. **Pin the task as a Signature.** Start inline (`"question -> answer"`); upgrade to a class-based `dspy.Signature` with `InputField(desc=...)` / `OutputField(desc=...)` when types matter or fields need disambiguation. 2. **Pick the lowest-power Module that works.** Default to `dspy.ChainOfThought`. Use `dspy.Predict` for trivial classification, `dspy.ReAct` only when tools are needed, `dspy.ProgramOfThought` for arithmetic-heavy tasks [dspy.ai/learn/programming/modules/]. 3. **Compose with plain Python control flow.** Subclass `dspy.Module`, instantiate sub-modules in `__init__`, call them in `forward()`. No special DSL. 4. **Run zero-shot on 5–10 hand-picked examples.** Look at outputs with `dspy.inspect_history(n=3)`.

**Exit criterion:** the un-optimized program produces *plausible* outputs on 5+ examples. Not great — plausible.

Stage 2 — Evaluation (no optimizer yet)

5. **Build a dev set.** Documented sweet spot: **30 examples = minimum useful, 300 = recommended, 200+ required for MIPROv2** to avoid overfitting [dspy.ai/learn/optimization/overview/]. 6. **Write a metric**: `def metric(example, pred, trace=None) -> float|bool`. Start with exact-match; only escalate to LLM-as-judge when the task demands it (open-ended generation, multi-criteria). 7. **Run `dspy.Evaluate(devset=dev, metric=metric, num_threads=16)`*