/agentsop-test-fix-loop

name: agentsop-test-fix-loop
version: 0.1.0
description: |
  Decision protocol for wiring a verify-then-fix loop around a code-editing LLM
  agent. The agent edits → runs lint/test → reads the output → fixes → re-runs,
  bounded by an iteration cap and an escalation rule. Activates whenever a coder
  agent has a verifiable success criterion (exit code, type-checker output,
  failing assertion) and the user wants the agent to converge to "green" on its
  own. Framework-agnostic — wraps Aider's `--auto-lint`/`--auto-test`, an
  OpenHands SWE-Bench loop, a manual LangGraph cycle, or Claude Code's bash
  tool just the same.
domain: coder-agent / tool-result-feedback
audience: engineers wiring LLM agents that must converge on a verifiable spec
trigger_keywords:
  - "auto-lint"
  - "auto-test"
  - "test-fix loop"
  - "fix until tests pass"
  - "verify-then-fix"
  - "iterate until green"
  - "agent feedback loop"
  - "iteration cap"
when_to_use:
  - "any code-edit flow with a verifiable success command (pytest, ruff, mypy, eslint, tsc, go test, cargo check)"
  - "wrapping a coding agent so it doesn't return until lint+tests are clean"
  - "SWE-Bench-style runs (one issue → patch → tests → fix → submit)"
  - "CI guardrail where a PR must be green before the agent declares done"
when_not_to_use:
  - "the success criterion is subjective ('looks good') — there's no signal to feed back"
  - "the verifier takes >5 min and you need the agent interactive — async the loop"
  - "human review is the gate (use HITL skill instead)"
  - "edits are exploratory / WIP — the loop will fight the user's incomplete code"

Test-Fix Loop · SOP

> One-liner: **The test result IS the next prompt.** Wiring the verifier is > 20% of the work; framing its output as a useful feedback message is 80%.

---

1. 何时激活 (Activation Rules)

Activate this skill when **any** of the following triggers fire:

• The user says "have the agent fix until tests pass", "run lint and tests

automatically", "iterate until green", or invokes `aider --auto-test`, `cline --yes`, or an OpenHands-style headless agent.

• The task has a **verifiable success command**: a non-zero exit code on

failure (pytest, ruff, mypy, eslint, tsc, go test, cargo check, npm run build, make check, …).

• You're wrapping a code-editing LLM in a script/CI step and need to decide:

*when does the agent return?*

• The agent just made an edit and the next message in the loop would be

"here's what the verifier said".

**Do not activate** when:

• Success is **subjective** (writing prose, designing UX). The loop has no

feedback signal worth replaying.

• The verifier is **slow + interactive** (full E2E suite, multi-min builds).

Either async-ify the loop, or run a fast subset (`pytest -x -k changed`) in the loop and gate the slow suite at PR review.

• The gate is **human approval**, not a machine check — use the HITL skill.

---

2. 核心心智模型 (Core Mental Model)

2.1 The test result IS the next prompt

The agent's *next turn* is conditioned almost entirely on the message you inject between edit-N and edit-N+1. That message — formatted from `stdout`, `stderr`, `exit_code` — **is the prompt**. The framework labels it "tool result" or "verifier output" but mechanically it is a user-role message the LM consumes verbatim.

⇒ **Framing the feedback dominates the model choice.** A 4000-line raw pytest dump prompts a worse fix than a 30-line "first failing test, traceback, the diff you just applied" digest, *regardless of the model behind it*.

2.2 Four primitives

+-----------------+   +-----------------+   +-----------------+   +-----------------+
| 1. Verifier     |   | 2. Capture      |   | 3. Format       |   | 4. Iteration    |
|    command      |   |    (stdout +    |   |    feedback     |   |    bound        |
|                 |   |     stderr +    |   |    message      |   |                 |
| - pytest -x     |   |     exit_code)  |   | - first error   |   | - max N tries   |
| - ruff check    |   | - timeout cap   |   | - last K lines  |   | - escalate /    |
| - mypy --strict |   | - byte cap      |   | - drop noise    |   |   commit / skip |
| - eslint .      |   | - kill on hang  |   | - keep colors=0 |   |                 |
+-----------------+   +-----------------+   +-----------------+   +-----------------+

Drop any one of these and the loop fails:

• No verifier → no signal; the agent guesses "done".
• No capture → the model can't read stderr; tracebacks live in stderr.
• No formatting → 25k-token output distracts the model

(see Aider's 25k context-drift threshold).

• No iteration bound → infinite loop; the OpenHands SWE-Bench infinite-loop

bug `[oh/6357]` is the canonical failure case.

2.3 Why a separate skill (vs "just give the agent a bash tool")

Naively: "let the agent run `pytest` and read the output". This breaks because:

1. The agent doesn't know **which** command to run (project-specific). 2. The agent dumps the **full output** into context every iteration, blowing the 25k threshold by iter 3. 3. The agent has **no termination contract** — it'll keep trying after the test passes "to be safe", or keep trying after 30 failures "to be helpful". 4. The agent makes **edits with no audit trail** — if iter 4 was the right fix, you can't bisect because nothing is committed.

The loop is a contract: *verifier wiring + output capture + feedback framing

• iteration cap + per-fix git commit*. Treat it as one operation, not five.

2.4 What "green" means

| Verifier returns | Interpretation | Next action | |---|---|---| | `exit 0`, no diagnostics | True success | Commit + exit loop | | `exit 0`, warnings | Soft success | Commit + log; optionally surface to user | | `exit != 0`, parseable error | Actionable failure | Format → feed back → next iter | | `exit != 0`, unparseable (e.g. segfault, OOM) | Environment / infra failure | Escalate; do not re-prompt the LM | | Timeout / hang | Likely infinite loop in