/content-refinement-agent

name: content-refinement-agent
description: Step 5 of the PaperOrchestra pipeline (arXiv:2604.05018). Iteratively refine drafts/paper.tex by simulating peer review and applying targeted revisions, with strict accept/revert halt rules, deterministic 0-100 decision bands (Accept/Minor/Major/Reject) that drive a target-met early stop, and a Devil's Advocate concession-threshold guard that blocks acceptance on unresolved critical findings. Maintains a worklog and snapshots each iteration so revert is real, not symbolic. TRIGGER when the orchestrator delegates Step 5 or when the user asks to "refine the draft", "iterate on the paper", or "run peer review on this paper".
data_access_level: verified_only

Content Refinement Agent (Step 5)

Faithful implementation of the Content Refinement Agent from PaperOrchestra (Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49–51).

**Cost: ~5–7 LLM calls** (App. B), typically ~3 refinement iterations, each consisting of one reviewer call and one revision call.

The paper highlights this step as one of the largest contributors to overall quality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute acceptance-rate improvement (Fig. 4). Get this step right.

Inputs

• `workspace/drafts/paper.tex` — output of Step 4
• `workspace/inputs/conference_guidelines.md`
• `workspace/inputs/experimental_log.md` — used as ground truth for the

hallucination check

• `workspace/citation_pool.json` / `workspace/refs.bib` — the allowed

bibliography

Outputs

• `workspace/refinement/iter1/`, `iter2/`, `iter3/` — per-iteration snapshots

containing `paper.tex`, `paper.pdf`, `review.json`, `score.json`

• `workspace/refinement/worklog.json` — append-only history of decisions
• `workspace/final/paper.tex` and `workspace/final/paper.pdf` — copy of the

best accepted snapshot

The refinement loop

prev_score = score(paper.tex)                  # baseline from initial draft
snapshot iter0/

for iter in 1..ITER_CAP (default 3):
    1. simulate_review(paper.tex) → review.json
       (uses `references/reviewer-rubric.md` rubric)

    2. apply_revision(paper.tex, review.json) → new_paper.tex
       (uses verbatim Refinement Agent prompt at `references/prompt.md`)

    3. snapshot iter<N>/ with new_paper.tex, review.json
       latexmk -pdf new_paper.tex → iter<N>/paper.pdf

    4. score(new_paper.tex) → curr_score

    5. decide via score_delta.py:
       - if curr.overall > prev.overall:                       ACCEPT
       - elif curr.overall == prev.overall and net_subaxis ≥0: ACCEPT
       - else:                                                 REVERT

    6. apply_worklog.py to append the decision

    7. if REVERT or no actionable weaknesses or iter == ITER_CAP: HALT

    paper.tex ← new_paper.tex   (only on ACCEPT)
    prev_score ← curr_score

cp <best iter>/paper.tex → workspace/final/paper.tex

The "best" snapshot at HALT is the one with the highest accepted overall score. On a REVERT halt, the best is the iteration immediately before the revert.

Step-by-step

0. Pre-refinement integrity gate

Before snapshotting or scoring the initial draft, run two gates in order:

**Gate A — AI failure modes** (load `references/ai-failure-modes.md`, runs once):

Load `references/ai-failure-modes.md` (which points to `skills/shared/ai_failure_modes.md`). Run all 7 checks against the draft and the inputs. This gate runs **once only**, at the start of iteration 1.

• CONFIRMED failure → write HALT entry to worklog.json, report to user, stop.
• SUSPECTED failure → add WARNING comment to paper.tex, log in worklog.json, continue.
• No failures → proceed.

**Gate B — Claim-evidence provenance** (runs once, WARN gate):

python skills/paper-orchestra/scripts/claim_evidence_gate.py \
    --paper  workspace/drafts/paper.tex \
    --log    workspace/inputs/experimental_log.md \
    --out    workspace/claim_evidence_report.json \
    --out-md workspace/claim_evidence_map.md

The gate sorts every number in the draft into `supported` (the value is in `experimental_log.md`), `attributed` (the sentence carries a citation or a prior-work cue), or `needs evidence` (neither). Only the third category is a finding. See `references/claim-evidence-map.md`.

Exit 0 → PASS, proceed normally. Exit 1 → WARN: unsupported numeric claims found. Log in worklog.json as: `{gate: "claim_evidence", status: "WARN", unsupported_count: N, report: "workspace/claim_evidence_report.json"}` Pass the `needs evidence` rows of `workspace/claim_evidence_map.md` to the revision agent in Step 3 as an additional instruction: "The following values appear in the paper but cannot be corroborated in experimental_log.md and carry no citation — restate them from logged values, attribute them, or remove the claim. Do not weaken the sentence into vagueness to make the number defensible." A row that survives two iterations should be deleted rather than reworded again. Do NOT halt on Gate B warnings; the revision agent will address them.

**Gate C — Reverse outline** (runs once, advisory):

python skills/content-refinement-agent/scripts/reverse_outline.py \
    --paper workspace/drafts/paper.tex \
    --out   workspace/reverse_outline.md \
    --json  workspace/reverse_outline.json

Strips the draft to one line per paragraph — its topic sentence — and flags paragraphs with no topic sentence, two messages, or a citation dump. Read `workspace/reverse_outline.md` before the first reviewer call and pass it into the reviewer call as the input for the **Logical Flow** axis. Structural findings enter the revision agenda as *reorder / merge / split / cut* instructions; sentence-level rewriting cannot fix a sequencing problem, and iterations spent polishing a misordered section still count against the budget. See `references/reverse-outline.md`.

**Gate D — Read research brief** (every run, no exit code):

If `workspace/research_brief.md` exists, read it before all rev