/ace-rsi-toolkit

name: ace-rsi-toolkit
description: Run ACE (Agentic Context Engineering, Zhang et al. 2025) on an agent the sequential offline way - a Reflector reads each training Trace, a Curator adds the lessons to a playbook of rules the agent reads before every task, and the final playbook is measured on a published Benchmark.
version: 2026.10.12.1

ACE

[Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models](https://arxiv.org/abs/2510.04618), Zhang et al., 2025 ([arXiv:2510.04618v1](https://arxiv.org/abs/2510.04618v1), §3–4 and Appendix B). GitHub: [ace-agent/ace](https://github.com/ace-agent/ace). Verified implementation revision `82709de050e1db6e6ef2f07bcb0393560b94992a`, chiefly `ace/ace.py` (`_train_single_sample`, `_offline_train`), `ace/core/reflector.py`, `ace/core/curator.py`, `ace/prompts/` and `playbook_utils.py`. This Skill runs the **sequential offline** algorithm: the Target (the paper's Generator) solves one training case at a time with the current playbook, a Reflector diagnoses that execution, and a Curator adds the lessons the playbook lacks as itemized rules; the final playbook is then measured on the frozen Benchmark. Run the method the way the paper does; every departure is a declared adaptation reported at the end.

Before you start

This Skill needs `run_subagent`, so it runs in a top-level Session. If the message only names it, ask which agent to train and on which published Benchmark. Otherwise resolve the inputs below and ask only for a missing Target or Benchmark; everything else has a default.

• `test_agent_id` — the Target (the Evaluation Center's Test Agent): an existing agent of this Project, or `create: <id>` when the user wants a fresh one, created with no plugins by `penguin agent create --agent-id <id> --project-id "$PROJECT_ID" --json` (an id matches `^[a-z][a-z0-9_-]{1,63}$`). If that id exists already, stop and ask; never reinitialize an agent. The Target is never the agent you run as: training rewrites its State while you work.
• `benchmark_id` — a Benchmark of this Project whose `benchmark_config.toml` status is published (not a literal `draft` or `failed`). Its cases (every directory of it holding a `statement/README.md`) are both the training samples and the measured case set. Without one, list the published Benchmarks and ask; this Skill never builds one.
• `test_benchmark_id` (optional) — a second published Benchmark held out from training: H1 (the initialized State) and the final playbook are measured on it too, and nothing of it reaches the Reflector or the Curator.
• `runs` — runs per case in every measurement; default 1.
• The runtime — the `provider` / `model_id` pair the user names, else your Environment's, fixed for the whole experiment (`references/evaluation.md`, controller step 2). The Reflector and the Curator run on your own model; record it.
• The settings below. Use the smoke profile unless the user asks for other values. A target score or a round limit is not an ACE setting: say so and leave it out.

| Setting | Smoke profile (default) | Source | | --- | --- | --- | | `num_epochs` | 1 | release default 1; the paper's experiments use up to 5 | | `max_num_rounds`: reflections per incorrect sample | 3 | release default 3; the paper's experiments use up to 5 | | `curator_frequency`: curate after every k-th sample | 1 | batch size 1, as in the paper's experiments | | `playbook_token_budget` | 80000 | release default | | `use_bulletpoint_analyzer`: embedding deduplication | false | release default | | Training order | the case ids, sorted | fixed before the first execution |

Budget, for C training cases, E epochs, M rounds, k = `curator_frequency` and R runs: at most `E × C × (2 + M)` training executions of the Target, `2 × C × R` executions for the H1 and final measurements (plus `2 × C' × R` on a test Benchmark of C' cases), at most `E × C × M` Reflector calls and `E × ⌊C / k⌋` Curator calls. Before the first execution, state the profile and these numbers. Never launch the paper's full budget (5 epochs, 5 rounds) implicitly: a profile above the smoke one runs only when the user asks for it, after you have restated its arithmetic.

Paths and records

Resolve every path from your Environment's `App Data Dir`, as `references/evaluation.md` does; never search for the Project:

PROJECT_DIR  = <app_data_dir>
PROJECT_ID   = basename of PROJECT_DIR
PENGUIN_HOME = dirname of PROJECT_DIR
TARGET       = PROJECT_DIR/agents/<test_agent_id>
STATE        = TARGET/agent_state
BENCHMARK    = PROJECT_DIR/benchmarks/<benchmark_id>
CASE         = BENCHMARK/<case_id>       # statement/README.md is public; rubric/README.md is private
SCOREBOARD   = BENCHMARK/scoreboard.yaml
SKILL_DIR    = PROJECT_DIR/agents/<your Agent ID>/agent_state/skills/ace-rsi-toolkit
PLAYBOOK     = STATE/skills/ace-playbook # SKILL.md + rules.yaml, the only learned slot
OUT          = <this Session's scratchpad>/ace-rsi-toolkit/<experiment_id>

`<experiment_id>` is `<test_agent_id>-<UTC time as YYYYMMDDTHHMMSSZ>`. Before the first execution, write `OUT/experiment.yaml`; its `dimensions` place the run in the Awesome-RSI taxonomy:

method: ACE
paper: "Agentic Context Engineering: Evolving Contexts for Self-Improving Language Models"
arxiv_url: https://arxiv.org/abs/2510.04618
github_url: https://github.com/ace-agent/ace
source_revision: 82709de050e1db6e6ef2f07bcb0393560b94992a
settings: { num_epochs: 1, max_num_rounds: 3, curator_frequency: 1, playbook_token_budget: 80000, use_bulletpoint_analyzer: false, correct: "score = 100" }
test_agent_id: <id>
benchmark_id: <id>
test_benchmark_id: <id or null>
training_order: [<case ids>]
runs: 1
runtime: { provider: <provider>, model_id: <model_id>, thinking_level: <level> }
roles_model: { provider: <your Provider>, model_id: <your Model ID> }
h1: { version: <n>, sha256: { <every file of STATE but .vault.toml>: <hash> } }
dimensions:
  artifact: playbook of itemized rules
  source: the Tar