/agentsop-per-model-artifacts

name: agentsop-per-model-artifacts
version: 0.1.0
description: >-
  Lifecycle SOP for **per-model prompt artifacts** — the compiled prompts, instructions,
  few-shot demos, edit-format pins, and embedding-bound indices that change behavior when
  the underlying LM, dataset, or framework version changes. Activate when adopting compiled
  prompts (DSPy, GEPA, BootstrapFewShot output), when supporting multiple LMs in production,
  when a provider deprecates a model snapshot, or when a framework deprecates a config
  surface (LlamaIndex `ServiceContext` → `Settings`, Aider edit-format defaults). Do NOT
  activate for one-off raw prompt edits or for truly model-agnostic system prompts that have
  been swap-tested. Search keywords: prompt portability, model swap, recompile prompt, model
  deprecation, prompt per model, prompt breaks on new model, version compiled prompts.

Per-Model Prompt Artifacts — SOP

> *"Prompts are effectively the weights of an LLM application."* > — DSPy core philosophy [arxiv.org/abs/2310.03714] > > *"Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the artifact — recompile."* > — dspy-sop SKILL, Dilemma Case B

---

1. 何时激活 (When to activate)

Activate this skill when **any** of the following appears in the user's intent, codebase, or workflow:

| Trigger | Signal | |---|---| | Adopting compiled prompts | `compiled.save("v1.json")`, `dspy.load_program()`, `BootstrapFewShot`, `MIPROv2`, `GEPA`, LangChain Hub `hub.push/pull`, prompt files checked into `prompts/` or `artifacts/` | | Multi-LM production | The same program runs against ≥2 of: `gpt-4o-*`, `gpt-4o-mini-*`, `gpt-4.1-*`, `claude-3-5-sonnet-*`, `claude-3-7-sonnet-*`, `claude-3-opus-*`, `Llama-3-*`, `Llama-3.1-*`, `DeepSeek-V3`, `gemini-2.5-pro` | | Provider deprecation | OpenAI/Anthropic deprecation notice mentioning a pinned snapshot; alias rollover (`gpt-4o` → new dated snapshot); silent model behavior drift reports | | Framework deprecation | LlamaIndex `ServiceContext` → `Settings`; LangChain `LLMChain` → LCEL; DSPy major version bump; aider edit-format default change | | Symptoms | `prompts/system_prompt.txt` (no model in path), generic alias pins (`model="gpt-4o"`), missing `parent_artifact` lineage, hand-edited compiled JSON, no held-out test set re-runs | | Cross-skill bridges | DSPy compile produced output → ship via this SOP. LlamaIndex index baked with embed model → tag artifact per this SOP. Aider edit-format pin → per-model config artifact per Recipe 6 (R2) |

**Do NOT activate** when:

• Raw, one-off prompt edits with no compile step and no production deploy.
• A genuinely model-agnostic system prompt that has been **swap-tested** across ≥3 LMs with <2-point dev metric drift.
• Prompts that must remain verbatim human-authored for compliance — versioning still matters, but the optimizer/recompile loop does not apply.
• The whole pipeline lives behind a vendor's managed prompt (e.g. OpenAI's Prompt Library) where the vendor owns the artifact.

---

2. 核心心智模型 (Core mental model)

The artifact is a triple, not a string

                ┌────────── ARTIFACT ──────────┐
                │                              │
   program  ×   LM (snapshot)  ×   dataset (hash)   →   compiled.json + metadata
   (code)       (provider+date)    (canonical hash)
                │                              │
                └─ change any axis → new artifact ─┘

Three load-bearing claims:

1. **A compiled prompt is not portable.** The DSPy doctrine (sibling skill, Case B): swap the LM → recompile. Reusing GPT-4o-compiled `program.json` on Llama-3-8B typically loses 15–30 points (R1 §5). The asymmetry is real and documented.

2. **The snapshot, not the alias, is the identity.** `gpt-4o` is a moving alias; `gpt-4o-2024-08-06` is an immutable identifier. Pinning to the alias means your behavior changes silently when OpenAI rolls the alias forward. Anthropic does not even roll aliases — Sonnet 3.5 (`claude-3-5-sonnet-20240620`) and Sonnet 3.5 v2 (`claude-3-5-sonnet-20241022`) are different models with different behavior; the org that pinned the wrong one in 2024-10 ate a regression in 2025-10 when the older one was retired (R1 §4).

3. **The dataset is part of the identity.** Compiled prompts overfit their training distribution. Two artifacts compiled from the "same dataset" that turn out to differ by 50 examples produce silently different programs. A `sha256` over canonicalized dataset bytes, embedded in the artifact path, makes this impossible to confuse (OP-9).

The PyTorch checkpoint analogy

| PyTorch checkpoint | Per-model prompt artifact | |---|---| | `.pt` weights file | `program.json` (compiled prompt) | | Architecture (forward pass) | DSPy Signature + Module graph (source-of-truth Python) | | Dataset version | `dataset_sha256_8` in path | | Hyperparameters | optimizer config (`MIPROv2 auto="light"`, seed, demos) | | Framework version | `dspy_version`, `python_version` in metadata | | Eval score on val | `dev_score` in REGISTRY | | Final test score | `test_score` in REGISTRY | | Model registry (MLflow) | `artifacts/REGISTRY.jsonl` or MLflow (Recipe 5) | | Promotion to prod | git tag `prompt/<program>/<snapshot>/v<n>` |

The same engineering discipline applies. Anyone who would not check a `.pt` into prod without a checkpoint registry should not check a compiled prompt into prod without an artifact registry.

Why "alone is not enough"

A bare `compiled.save("v1.json")` produces a JSON file that, viewed in isolation, looks model-agnostic. The instructions and demos are text — they could be for any model. But they were *selected by an optimizer running calls against a specific LM*. The model's distributional response to the bootstrapping queries shaped which demos got kept. The artifact is implicitly LM-conditioned without saying so. The SOP exists to make this conditioning explicit and auditable.

---

3. SOP 工作流 (SOP workflow)

A five-stage