/agentsop-code-execution-decision

name: agentsop-code-execution-decision
version: 0.1.0
description: >-
  Decision rubric for when an LM agent should write-and-run code (Program-of-Thought / code
  interpreter) versus reason in natural language: classify each step as deterministic-
  computable (emit + execute code, feed the result back) vs judgment (stay in prose). Use
  when designing or debugging an agent step that does arithmetic/parsing/data transforms,
  when prose reasoning hallucinates a computation (under-coding), or when a sandbox round-
  trip is wasted on a judgment task (over-coding). Search keywords: code interpreter, agent
  does math wrong, calculator hallucination, when to run code vs reason, program of thought,
  PoT, tool vs reasoning.
domain: |
  Deciding when an LM agent should write-and-run code (Program-of-Thought / code interpreter)
  versus reason in natural language. An enhancement overlay on top of DSPy's ProgramOfThought
  module, which ships the mechanism but not the decision rubric.
type: enhancement-overlay
source: |
  DSPy ProgramOfThought module + docs [dspy.ai/learn/programming/modules/];
  "Program of Thoughts" (Chen et al., arXiv 2211.12588);
  PAL: Program-aided Language Models (Gao et al., arXiv 2211.10435);
  OpenAI Code Interpreter / Assistants code-tool docs;
  Anthropic code-execution tool docs;
  LangChain PythonREPLTool;
  sibling skill output-format-by-model (PoT for math/parse content shape).
audience: |
  Coder-agent authors, tool-use harness designers, RAG/eval pipeline builders deciding
  whether a step should emit executable code or stay in prose.
status: tool-skill

Code-Execution Decision — emit-code-vs-reason

> **One-liner**: LMs are unreliable calculators but reliable coders. When the answer needs > determinism and precision — arithmetic, exact data manipulation, deterministic transforms — > emit code and run it. When the answer needs judgment, taste, or open-ended synthesis, reason > in natural language. The cost of getting this gate wrong is silent: prose arithmetic > *hallucinates a plausible-looking wrong number*, and over-coding a judgment task burns a > sandbox round-trip for nothing.

This is an **enhancement overlay**. DSPy already gives you `dspy.ProgramOfThought` (PoT) — the *mechanism* for write-then-execute. What it does not give you is the *decision rubric* for when to reach for it. That rubric is this skill. Cross-link the sibling [[agentsop-output-format-by-model]] (which decides *how* code-shaped content should be serialized) and [[agentsop-test-fix-loop]] (which closes the execute → error → retry loop).

---

1. 何时激活 (When to activate)

Activate this skill **before** committing a step to a reasoning strategy whenever the task has a **verifiable, deterministic core** — or whenever you catch an agent doing arithmetic in prose.

| Trigger | Signal | |---|---| | Arithmetic / math | "compute the compound interest", "what's 17.5% of $4,392.18", multi-step word problems, unit conversions, date deltas | | Precise data manipulation | "sort these 240 rows by the third column", "dedupe and count", "join these two lists on id", "parse this CSV and sum column B" | | Deterministic transforms | regex extraction, string reformatting, base conversion, hashing, sorting, set operations | | Symbolic / combinatorial | "how many distinct permutations", "solve this system of equations", calendar/scheduling math | | You see a model doing math in prose | "Let me add: 1,204 + 8,991 + ... = 10,195" — almost always worth a code check | | Choosing a DSPy module | deciding between `ChainOfThought` and `ProgramOfThought` for a signature [dspy.ai/learn/programming/modules/] |

**Anti-triggers** (do NOT reach for code execution):

• The task is **judgment**: tone, summarization, ranking by quality, "is this reply empathetic?", design tradeoffs, open-ended explanation.
• The "computation" is trivial and within the model's reliable range (single-digit arithmetic, a 3-item count) — the sandbox round-trip costs more than it saves.
• No sandbox is available and the determinism requirement is soft.
• The output's *consumer* is a human reading prose, and an approximate answer is acceptable.

---

2. 核心心智模型 (Core mental model)

> **LMs are unreliable calculators but reliable coders.**

A language model predicts the *next token*, not the *correct value*. When you ask it to add `48,217 + 9,884` in prose, it emits the most *plausible-looking* digit sequence — which is frequently wrong, and wrong in a way that looks right. The same model can write `48217 + 9884` as a Python expression flawlessly, because emitting the *program* is a pattern-matching task it is genuinely good at, and the *Python interpreter* is a deterministic oracle. This decoupling — model writes the recipe, interpreter computes the result — is the entire thesis of Program-of-Thought (PoT) [arXiv 2211.12588] and PAL [arXiv 2211.10435].

            ┌────────────────────────────────────────────────────────┐
            │  THE GATE: does this answer need determinism/precision?  │
            └────────────────────────────────────────────────────────┘
                       │                                  │
              YES (computable)                     NO (judgment)
                       │                                  │
                       ▼                                  ▼
            ┌─────────────────────┐            ┌─────────────────────┐
            │ EMIT CODE           │            │ REASON IN PROSE     │
            │ model writes recipe │            │ model is the engine │
            │ interpreter = oracle│            │ no oracle exists    │
            └─────────────────────┘            └─────────────────────┘
                       │
                       ▼
            sandbox → run → feed result back into LM → LM narrates/uses it

**Two failure modes the gate prevents:**

| Failure | Mechanism | Symptom | |---|---|---| | **Under-coding** (reason when you should compute) | LM h