/awm-rsi-toolkit

name: awm-rsi-toolkit
description: Run AWM (Agent Workflow Memory, Wang et al. 2024) on an agent - induce reusable workflows from its successful trajectories, online over a training stream or offline from supplied experiences, then freeze the workflow memory and measure it on a published Benchmark.
version: 2026.10.12.1

AWM

[Agent Workflow Memory](https://arxiv.org/abs/2409.07429), Wang et al., 2024 ([arXiv:2409.07429v1](https://arxiv.org/abs/2409.07429v1), §2.1–2.3 and Appendix A). GitHub: [zorazrw/agent-workflow-memory](https://github.com/zorazrw/agent-workflow-memory). Verified implementation revision `8c0ff8cd11d648c8fceb99e4e42f37e3b75381b1`, chiefly `mind2web/offline_induction.py`, `mind2web/memory.py`, `mind2web/prompt/`, `webarena/pipeline.py`, `webarena/induce_prompt.py` and `webarena/autoeval/`. AWM induces **workflows**, reusable sub-routines that each pair a goal with a sequence of steps (an observation, the reasoning and the action, with the instance values abstracted away), from successful trajectories, and gives the agent all of them as memory: online, adding the workflows of every execution a success evaluator admits before the next task, or offline, inducing once from a pool of canonical experiences. Run the method the way the paper does; every departure is a declared adaptation reported at the end.

Before you start

This Skill needs `run_subagent`, so it runs in a top-level Session. If the message only names it, ask which agent to train and on which published Benchmark. Otherwise resolve the inputs below and ask only for a missing Target or Benchmark (or, in `offline` mode, a missing experience pool); everything else has a default.

• `test_agent_id` — the Target (the Evaluation Center's Test Agent): an existing agent of this Project, or `create: <id>` when the user wants a fresh one, created with no plugins by `penguin agent create --agent-id <id> --project-id "$PROJECT_ID" --json` (an id matches `^[a-z][a-z0-9_-]{1,63}$`). If that id exists already, stop and ask; never reinitialize an agent. The Target is never the agent you run as: training rewrites its State while you work.
• `benchmark_id` — a Benchmark of this Project whose `benchmark_config.toml` status is published (not a literal `draft` or `failed`). Its cases (every directory of it holding a `statement/README.md`) are the training stream of `online_train` and the measured case set. Without one, list the published Benchmarks and ask; this Skill never builds one.
• `test_benchmark_id` (optional) — a second published Benchmark held out from training: H1 (the initialized State) and the final memory are measured on it too, and nothing of it reaches the Judge or the Inducer.
• `runs` — runs per case in every measurement; default 1.
• The runtime — the `provider` / `model_id` pair the user names, else your Environment's, fixed for the whole experiment (`references/evaluation.md`, controller step 2). The Judge and the Inducer run on your own model; record it.
• `mode`, fixed before any task evidence is read:
• `online_train` (default) — the paper's online loop (run, judge, induce, integrate) over the Benchmark's cases as a training stream; the memory is then frozen and measured.
• `offline` — one induction from a supplied pool of canonical experiences, each a task with its observed trajectory: a folder the user names (annotated or model-synthesized), or the Test Sessions of scoreboard runs that scored 100. Record the pool's provenance. Never make a success out of a failed run, and never switch to an online judge without changing the mode.
• `criteria`, for `online_train` — `autoeval` (default), standing in for the source's neural success evaluator: a Judge that sees only the public task, the trajectory and the outputs; or `gt`, only when the user chooses it: an execution succeeds when it scores 100, never by a tuned partial threshold.

Smoke profile (the default; the paper also makes one pass): `online_train`, one pass over the cases in sorted id order, one execution per case, `criteria: autoeval`, the incremental induction profile (below). Another pass is an extension the user must ask for, never an answer to disappointing scores. A target score or a round limit is not an AWM setting: say so and leave it out.

Budget, for C cases and R runs: `online_train` takes C stream executions of the Target, at most C Judge calls (none with `gt`) and at most C Inducer calls; `offline` takes one Inducer call. Both add `2 × C × R` executions for the H1 and final measurements (plus `2 × C' × R` on a test Benchmark of C' cases). Before the first execution, state the mode and these numbers. Never launch a larger budget implicitly: restate its arithmetic and run it only when the user asks for it.

Paths and records

Resolve every path from your Environment's `App Data Dir`, as `references/evaluation.md` does; never search for the Project:

PROJECT_DIR  = <app_data_dir>
PROJECT_ID   = basename of PROJECT_DIR
PENGUIN_HOME = dirname of PROJECT_DIR
TARGET       = PROJECT_DIR/agents/<test_agent_id>
STATE        = TARGET/agent_state
BENCHMARK    = PROJECT_DIR/benchmarks/<benchmark_id>
CASE         = BENCHMARK/<case_id>       # statement/README.md is public; rubric/README.md is private
SCOREBOARD   = BENCHMARK/scoreboard.yaml
SKILL_DIR    = PROJECT_DIR/agents/<your Agent ID>/agent_state/skills/awm-rsi-toolkit
MEMORY       = STATE/skills/awm-workflows # SKILL.md + workflows.md, the only learned slot
OUT          = <this Session's scratchpad>/awm-rsi-toolkit/<experiment_id>

`<experiment_id>` is `<test_agent_id>-<UTC time as YYYYMMDDTHHMMSSZ>`. Before the first execution, write `OUT/experiment.yaml`; its `dimensions` place the run in the Awesome-RSI taxonomy:

method: AWM
paper: "Agent Workflow Memory"
arxiv_url: https://arxiv.org/abs/2409.07429
github_url: https://github.com/zorazrw/agent-workflow-memory
source_revision: 8c0ff8cd11d648c8fceb99e4e42f37e3b75381b1
settings: { mode: online_train, criteri