/ape-rsi-toolkit

name: ape-rsi-toolkit
description: Reproduce APE (Large Language Models are Human-Level Prompt Engineers, Zhou et al. 2022) on an agent and a frozen Benchmark, or on the house-style brief demo task shipped with this Skill: induce instructions from input/output examples, measure each one, keep the best.
version: 2026.10.12.1

APE

[Large Language Models are Human-Level Prompt Engineers](https://arxiv.org/abs/2211.01910) (Zhou et al., University of Toronto, Vector Institute and University of Waterloo, 2022; [paper v2](https://arxiv.org/abs/2211.01910v2), §3 and §5.3), code at [keirp/automatic_prompt_engineer](https://github.com/keirp/automatic_prompt_engineer). Verified implementation revision `eac521c79a78965245ce7745dcc9f6b0792c7ec7`: `automatic_prompt_engineer/ape.py`, `generate.py`, `evaluate.py` and `template.py`, `experiments/run_instruction_induction.py` and `experiments/evaluation/instruction_induction/exec_accuracy.py`. This Skill runs **non-iterative forward-generation APE with execution scoring**, the paper's default: a proposer infers candidate instructions from input/output demonstrations in one pass, every candidate is measured on the selection cases, and the best is kept. It implements neither the optional iterative Monte Carlo resampling of §5.3 nor bandit (UCB) allocation nor log-likelihood scoring; the public `simple_ape` helper uses UCB, so never claim that helper's profile. Run the method the way the paper does; every departure is a declared adaptation reported at the end.

Before you start

This Skill is the experiment's controller and needs `run_subagent`; if that tool is missing you are a subagent: stop and say the Skill needs a top-level Session. If the request only names this Skill, ask for the Target, the Benchmark and the demonstrations, and offer the demo task. Otherwise resolve the inputs below, asking only for one the method needs and the request does not give:

• `test_agent_id`: the Target, an existing Agent of this Project, or `create: <id>` for a fresh one. Never the Agent running this Skill: its workers are this Agent.
• `benchmark_id`: a published Benchmark of this Project; its cases are APE's selection set (the source's `eval_data`). When the user asks for the demo task, build it from `references/demo-task.md` first: it fixes the Target, the Benchmark, the demonstrations and the budget, so ask nothing.
• `case_ids`: every case unless the user names a subset. `runs`: Runs per Case, 1 unless given.
• Demonstrations: the input/output pairs the proposer induces from (the source's `prompt_gen_data`).
• The demo task ships five.
• Otherwise ask for a file of pairs, or for explicit permission naming the Cases whose gold answers may serve as outputs; those Cases then leave the selection set. A private answer is not authorized because it exists.
• Never stand failed Target outputs or scores in for demonstrations: that is not APE. Until you have them, wait: open no Rubric and generate nothing.
• Keep demonstrations and selection cases apart, declare any overlap, and never use test cases for either.
• The Target's runtime, fixed as `references/evaluation.md` says. The proposer is this Session's model.
• The method's parameters, with the values of `run_instruction_induction.py`:

| Parameter | Source | Smoke profile (the default; also the demo task) | | --- | --- | --- | | `num_subsamples` | 3 | 1 | | `num_demos` | 5 | 5, at most the number of demonstrations | | `num_prompts_per_subsample` | 30 | 4 | | Selection examples per candidate | `num_samples` = min(20, the eval set), sampled per prompt | every selection case × `runs` | | Proposer sampling | `text-davinci-002`, temperature 0.9, top_p 0.9, at most 50 tokens | the provider's defaults (Penguin sets none); record it |

The smoke profile applies unless the user sets parameters. The source's 90 candidates (3 × 30) are never a default: run them only when the user asks, after you state the arithmetic (see "Budget") and they confirm. Rounds, a target score or a round limit are not parameters of non-iterative APE: report against a target if given, but never stop or skip on it.

Paths and records

The paths of `references/evaluation.md`, from the Environment's `App Data Dir`, plus the snapshots and this experiment's output directory in this Session's scratchpad (`<agent_id>` and `<session_id>` are your own):

PROJECT_DIR  = <app_data_dir>
PROJECT_ID   = basename of PROJECT_DIR
PENGUIN_HOME = dirname of PROJECT_DIR
TARGET       = PROJECT_DIR/agents/<test_agent_id>
STATE        = TARGET/agent_state
BENCHMARK    = PROJECT_DIR/benchmarks/<benchmark_id>
SCOREBOARD   = BENCHMARK/scoreboard.yaml
SNAPSHOTS    = TARGET/snapshots
OUT          = <app_data_dir>/agents/<agent_id>/scratchpad/<session_id>/ape-rsi-toolkit/<experiment_id>/

`experiment_id` is `<UTC timestamp>-<test_agent_id>`. Before the first measurement, write `OUT/experiment.yaml`:

method: APE
paper: Large Language Models are Human-Level Prompt Engineers (Zhou et al., 2022)
arxiv_url: https://arxiv.org/abs/2211.01910
github_url: https://github.com/keirp/automatic_prompt_engineer
source_revision: eac521c79a78965245ce7745dcc9f6b0792c7ec7
skill: { name: ape-rsi-toolkit, version: <installed version>, sha256: <SKILL.md digest> }
test_agent_id: <id>
benchmark_id: <id>
case_ids: [<selection case ids>]
runs: <runs>
demonstrations: { source: <where the pairs come from>, ids: [<ids>], permitted_fields: [input, output], overlap: <none, or which> }
runtime: { target: { provider, model_id, thinking_level }, proposer: { provider, model_id } }
parameters: { <every parameter above as used, with the sampling seed> }
state: { initial_version, instruction_sha256, skill_sha256, reader_sha256 }
artifact: skills/ape-instruction/instruction.md in the Target's Agent State
source: the demonstrations (generation) and the Benchmark's cases (selection)
feedback: the proposer sees demonstrations only; scores reach this Session, never the proposer
upd