/agentsop-llm-artifact-versioning

name: agentsop-llm-artifact-versioning
version: 0.1.0
description: >-
  Enhancement overlay — version the WHOLE deployable LLM-app artifact as one bundle: prompts
  + compiled programs + model snapshot pins + retrieval config + eval-set version, versioned
  together so a deploy is reproducible and rollback is atomic. Activate when preparing to
  deploy an LLM app, when asking "what exactly is running in prod right now?", when a deploy
  must be reproducible months later, or when an incident needs a clean rollback. The core
  reframe: an LLM app artifact is NOT an ML model — it is a manifest over many
  independently-mutable parts, not one weights file. Do NOT activate for one-off prompt
  edits with no deploy, for a single-component demo, or where a vendor owns the whole prompt
  lifecycle. For versioning ONE compiled prompt use [[agentsop-per-model-artifacts]]; for
  the CI comparison mechanism use [[agentsop-regression-gate]]. Search keywords: prompt
  versioning, reproducible deploy, what is running in prod, rollback LLM app, model pinning,
  prompt registry, version prompts and config.

Versioned, Reproducible LLM Artifact — Overlay SOP

> *"Prompts are effectively the weights of an LLM application."* > — DSPy core philosophy [arxiv.org/abs/2310.03714] (R1 §1) > > *"Treat the compiled program as a (program × LM) pair. Changing the LM invalidates the > artifact — recompile."* > — dspy-sop SKILL, Dilemma Case B (R1 §2)

This is an **enhancement overlay**, not a framework SOP. It sits on top of whatever stack you use (DSPy, LangChain, raw API) and adds one discipline: **define, pin, and version the entire deployable bundle as a unit.** It is the broad sibling of [[agentsop-per-model-artifacts]] — that skill versions one compiled prompt; this one versions everything that ships together.

---

1. 何时激活 (When to activate)

Activate when **any** of these appears in the user's intent, codebase, or workflow:

| Trigger | Signal | |---|---| | Preparing to deploy | "ship this to prod", a `Dockerfile`/`deploy.yaml`/serving entrypoint wrapping an LLM app, a release checklist | | "What is running in prod?" | Nobody can name the exact prompt text + model snapshot + retriever config currently serving traffic | | Reproducibility need | "reproduce the deploy from last quarter", an audit, a regulator asking what produced an output | | Rollback need | Incident: prod behavior changed and the team needs the last known-good *combination* of components back | | Drift symptoms | Score moved, no code change merged; or "we updated the prompt but forgot which model it was tuned for" | | Multi-component apps | RAG + reranker + synthesizer + judge, each naming its own model/config, none bundled | | Cross-skill bridges | DSPy `save_program` produced a compiled program → it is *one component* of the bundle; pin the rest. Per-prompt lifecycle handled by [[agentsop-per-model-artifacts]] → wrap as a bundle component here. |

**Do NOT activate** when:

• One-off prompt edit with no deploy and no reproduction requirement — just edit and run.
• A single-component proof-of-concept where there is exactly one prompt, one model, no

retrieval, and it will never be reproduced. Flat `v1.json` is fine.

• A vendor owns the whole lifecycle (OpenAI Prompt Library, Anthropic Workbench managed

prompts, fully vendor-managed RAG) — let them version it.

• The team rewrites the app daily during exploration — bundle versioning has no shelf life

until the app shape stabilizes (same boundary as dspy-sop "signature still changing").

---

2. 核心心智模型 (Core mental model)

An LLM app artifact ≠ an ML model

The single most common mistake is reasoning about an LLM app the way you reason about a trained model. They are not the same shape.

| ML model | LLM app artifact | |---|---| | One weights file (`.pt`, `.safetensors`) | A **manifest** over many parts | | Identity = file hash | Identity = hash of the *whole bundle* | | Mutates only on retrain | Each part mutates independently and silently | | Versioned by a model registry | Versioned by a bundle manifest + tag |

The deployable artifact is:

            ┌──────────────── DEPLOYABLE BUNDLE (one tag) ────────────────┐
            │                                                             │
  prompts       compiled        model pins        retrieval       eval-set
  (text +       programs        (snapshot id      config          version
   hashes)      (program.json)   per call site)   (index ptr,     (sha256)
            │                    embed model,                             │
            │                    top_k, reranker)                        │
            │                                                             │
            └── version them TOGETHER, or you can't reproduce a deploy ──┘

**The load-bearing claim:** the deployable artifact = prompts + compiled programs + model pins

• retrieval config + eval-set version. **Version them together or you cannot reproduce a

deploy** — and you cannot roll back without producing a never-tested combination.

Why "together" is non-negotiable

Each part can change without touching the others (R1 §3, §4, §6):

• A model **alias** rolls forward server-side (`gpt-4o` → new snapshot) — prompt unchanged,

behavior changed.

• A model **snapshot** gets a silent server-side patch (R1 §4) — pin unchanged, behavior

changed.

• A retriever knob moves in a dashboard (top_k 5→10) — no commit, behavior changed.
• The eval set gains 50 examples — the "same" score now means something different (R1 §5).

If these are versioned separately, "the deploy" is not a thing you can name. If they are versioned as one bundle with one id, the deploy is reproducible and rollback is atomic (R1 §8).

The registry analogy and its limit (R1 §7)

MLflow's Model Registry gives the right *primitives*: versioning, stage transitions (Staging/Production/Archived), reproduce-from-config, compare-versions [`~/.claude/skills/mlflow/SK