/agentsop-prompt-compilation

name: agentsop-prompt-compilation
version: 0.1.0
phase: D
tier: core
frequency: medium
status: opinionated
layer: enhance-overlay
decision_layer_only: true
defers_implementation_to: ["dspy", "dspy-sop", "metric-design"]
description: The compile-readiness gate for prompt auto-optimization. Decide whether you have earned the right to run an optimizer (DSPy MIPROv2 / GEPA / BootstrapFewShot) before spending compute. Two preconditions only — a real metric, and enough examples for the optimizer you picked. Garbage metric in, garbage prompt out. Pick the optimizer by data scale; GEPA inverts the scale assumption (~10 examples + textual feedback).

prompt-compilation — The Compile-Readiness Gate

> "It's unproductive to launch optimization runs using a poorly designed program or a bad metric." > — DSPy core team [dspy.ai/learn/optimization/overview/]

> "Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out." > — this skill's operating principle (synthesized from the line above + DSPy Case C)

This is an **enhancement-overlay decision skill**. It answers exactly one question the broad `[[dspy]]` library skill buries under API surface: **have you earned the right to run an optimizer yet, and which one?** It produces a go / no-go gate plus an optimizer pick. It defers *every* implementation detail — Signature syntax, module choice, `compile()` calls, save/deploy — to `[[dspy]]` and the full workflow in `[[agentsop-dspy]]`. It defers metric *construction* to `[[agentsop-metric-design]]`; this skill only checks the metric *exists and is validated*, then uses it as the gate.

The trap it removes: people reach for `MIPROv2(auto="heavy")` because the API is right there, before they have a metric worth maximizing or enough data to avoid memorization. Compilation is a hyperparameter search costing hundreds-to-thousands of LM calls ($2–$40+, minutes-to-hours) [dspy.ai/faqs/]. Spending that on an un-validated metric or 8 examples is pure waste.

---

1. 何时激活 (When to Activate)

Activate when **all three** of these are plausibly true (the gate then *confirms* them):

• **A hand-tuned prompt has plateaued.** The team has manually iterated few-shot examples / wording past the point

of obvious returns. Symptom from `[[agentsop-dspy]]` §1: "the team manually tunes few-shot examples; a metric exists but isn't being used to drive prompt design."

• **A metric exists (or can be built).** There is a `metric(example, pred) -> bool|float` — or one can be written

and human-validated. Without this, do not activate; the optimizer has nothing to maximize.

• **Labeled examples exist.** There is a dev set. The *count* determines which optimizer is even legal (§3, §4.2).

Concrete triggers in intent or codebase:

| Trigger | Signal | |---|---| | Spend intent | "auto-tune this prompt", "should I run MIPRO?", "is it worth compiling?", "GEPA vs MIPROv2", "optimize prompts for our metric" | | API reach | `MIPROv2(`, `BootstrapFewShot(`, `dspy.GEPA(`, `teleprompter`, `optimizer.compile(` about to be called | | Symptom | hand-tuned prompt stuck; few-shot examples curated by hand; metric written but only used for reporting, not optimization |

**Do NOT activate when:**

• The task is one-shot or the Signature/I-O contract is still churning daily — compile only after it stabilizes

[dspy.ai/learn/optimization/overview/]; otherwise you pay compile cost for prompts you'll throw away.

• No metric is possible and none will be built — then this is verbose prompting, not compilation. Route to

`[[agentsop-metric-design]]` first; if the user refuses any success criterion, the gate stays closed.

• Compliance requires verbatim human-authored prompts — optimized prompts are machine-generated artifacts.
• You only need *parse safety* (a typed Signature), not *quality optimization* — that is `[[agentsop-dspy]]` Stage 1 and

the `signature-design` overlay. Promoting prose to a typed Signature and *compiling* it are two different gates.

---

2. 核心心智模型 (Core Mental Model)

**"Compile when you can measure. The optimizer maximizes your metric — garbage metric in, garbage prompt out."**

An optimizer (MIPROv2, GEPA, BootstrapFewShot) is a black-box search over prompt instructions + few-shot demos that maximizes `metric(pred, example)`. It has no taste. It will faithfully chase *whatever the metric rewards*, biases and all. From `[[agentsop-dspy]]` Case C: "DSPy will optimize toward whatever the metric rewards. A bad metric becomes a bad program at scale." Two corollaries make this a *gate*, not a step:

1. **The metric is the precondition, not a tunable.** Before any compute is spent, the metric must (a) exist and (b) agree with human judgment on ≥20 spot-checks. An un-validated metric means the expensive search optimizes the metric's blind spot. This is non-negotiable [dspy.ai/learn/evaluation/metrics/; Case C]. Construction is `[[agentsop-metric-design]]`'s job; this skill only *checks the receipt*.

2. **Data scale is a hard floor, not a preference.** Below the optimizer's example floor you are not training, you are memorizing. The DSPy 20/80 train/val split exists *because* "prompt-based optimizers often overfit to small training sets" [dspy.ai/learn/optimization/overview/]. The floor differs per optimizer (§4.2).

The two-gate picture

            ┌──────────────────────────────────────────────┐
 Gate 1     │  METRIC: exists?  AND  human-validated ≥20?   │  ── No ─► STOP. Build/validate metric ([[agentsop-metric-design]]).
 (measure)  └──────────────────────────────────────────────┘
                              │ Yes
                              ▼
            ┌──────────────────────────────────────────────┐
 Gate 2     │  EXAMPLES: ≥ floor for the optimizer I want?  │  ── No ─► Pick a lower-floor optimizer, collect data,
 (data)     └──────────────────────────────────────────────┘           or STOP (use LabeledFewShot as a floor).