/agentsop-output-format-by-model

name: agentsop-output-format-by-model
version: 0.1.0
description: >-
  Pick an LM output format per (task x consumer x model) rather than by reflex: different formats carry different cognitive load (e.g. code-in-JSON makes the same model write worse code than plain-text+diff, while asking for prose when you need a typed object fails the other way). Use when designing or debugging an LM's output schema, choosing between plain text / diff / JSON / tool-call / grammar-constrained output, or when a model's quality drops after wrapping its output in a structured format.
domain: LM output schema design for coder-agents and any LM workflow whose output is consumed by code
source: |
  Aider's "code-in-JSON" benchmark + unified-diff post + edit-format docs;
  DSPy adaptor system; "Let Me Speak Freely?" (arXiv 2408.02442);
  StructEval (arXiv 2505.20139); Anthropic text-editor tool / writing-tools-for-agents;
  OpenAI structured outputs + tool_use docs; Outlines/Guidance grammar libs.
audience: |
  Anyone designing an LM output format — coder-agent authors, RAG pipeline builders,
  tool-use harness designers, prompt engineers shipping typed objects.
status: tool-skill

Output-Format-by-Model — 让格式服务于任务，而不是反过来

> **One-liner**: Different output formats carry different cognitive load for the model. Code-in-JSON is the canonical proof: the same model writes worse code when wrapped in a JSON tool-call than when emitted as plain text + diff. The reverse failure (asking for prose when you need a typed object) is just as common. Pick format per (task × consumer), not by reflex.

---

1. 何时激活 (When to activate)

Activate this skill **before** committing to an output schema in any of these situations:

| Trigger | Signal | |---|---| | Designing a coder-agent | "should the model return a `apply_patch` tool call or plain-text diff?" | | Adding a tool to an existing agent | "tool input has a `code` / `query` / `sql` / `regex` field — should I nest it in JSON or leave it as a string?" | | Building extraction / classification | "should I use `dspy.Predict` typed fields, Pydantic + `response_format=json_schema`, or just markdown?" | | Wiring an evaluator | "the metric needs a number — but the model also has to *reason* to produce it" | | Migrating a working prompt to "structured outputs" | someone said "let's make it safer with JSON schema" | | Tool-call harness adds latency / errors | repeated `json.JSONDecodeError`, escaping bugs, truncated outputs |

**Anti-triggers** (skip this skill):

• The format is fixed by an external API (e.g. you must return OpenAI function-call JSON — no choice).
• One-shot exploratory prompting where no consumer parses the output yet.
• The task is *itself* about JSON (e.g. "fix this malformed JSON") — see §6.

---

2. 核心心智模型 (Core mental model)

2.1 Three-layer claim

            ┌──────────────────────────────────────────────────┐
            │ FORMAT FOLLOWS FUNCTION                          │
            │                                                  │
            │  Some formats add cognitive load to the model    │
            │  and measurably degrade quality on the           │
            │  *content* the format is supposed to wrap.       │
            └──────────────────────────────────────────────────┘
                         ▲                          ▲
                         │                          │
         What's being consumed?       Who consumes it?
         (code? prose? entities?      (human reader? parser?
          number? action selection?)   downstream LM? compiler?)
                         │                          │
                         └──────────┬───────────────┘
                                    ▼
                       FORMAT SELECTION
                  (text+diff | markdown | JSON | tool_use
                   | grammar-constrained | typed field)

Aider's `code-in-json` benchmark is the load-bearing empirical anchor:

• GPT-4 Turbo refactor benchmark with **unified diff**: **61%** pass.
• Same model with **JSON-wrapped code**: **20%** pass.
• That is a **3×** swing from format alone, not from model choice or prompt rewording. [aider.chat/2023/12/21/unified-diffs.html, aider.chat/2024/08/14/code-in-json.html]
• The pattern holds across every major model tested in 2024 (Claude Sonnet, DeepSeek Coder, GPT-4o). Even with OpenAI strict-mode JSON validity enforced, **the code *inside* the JSON degraded** — more `SyntaxError` / `IndentationError`. Sonnet kept syntax clean but still scored lower overall. [aider.chat/2024/08/14/code-in-json.html]

Academic generalization, same year:

• "Let Me Speak Freely?" (arXiv 2408.02442) — **format-restricted reasoning drops 10–15%** on math and complex analysis vs free-form-then-convert. [arxiv.org/abs/2408.02442]
• StructEval (arXiv 2505.20139) — even o1-mini only scores 75.58 average on structural-output generation. [arxiv.org/html/2505.20139v1]

2.2 The two reflexes to unlearn

| Reflex | When it's wrong | |---|---| | "Structured output is always safer." | False for code, multi-step reasoning, free-form prose. Strictness ≠ quality of contents. | | "Markdown is only for humans." | False — markdown is *also* the highest-fidelity wire format for many LM-to-LM hand-offs (Aider uses it; DSPy's default chat adaptor uses field-marked markdown over JSON for many signatures). |

2.3 Mental model: format as a *tax* on the generation surface

Every formatting requirement consumes some of the model's attention budget. The tax is:

• **Low** for formats the model has seen millions of times in pretraining (markdown, unified diff, Python).
• **Medium** for JSON containing simple scalar fields (entity extraction, dates, booleans).
• **High** for JSON containing code, multi-line strings with escapes, or deeply nested reasoning. Each `\n` becomes `\\n`; each quote becomes `\"`; the model has to track this *while also* solving the actual problem.

⇒ Heuristic: **the more semantically dense