/email-evals

name: email-evals
description: Author and safely run deterministic email-agent evaluation suites with dedicated e2a test agents. Use when a user wants to define synthetic email cases, validate a suite, inspect its dry-run plan, run it after approval, or regrade changed assertions.

Email evals

Gather the suite one answer at a time

Ask one logical question at a time; do not dump a questionnaire. Ask exactly one prompt, wait for the answer, then advance. Conditionally omit a later prompt only when an earlier answer proves its field irrelevant; never bundle prompts.

<!-- email-evals:authoring-prompts:start --> 1. <!-- email-evals:field=use-case --> **use case** — Ask: “What is the synthetic use case?” 2. <!-- email-evals:field=existing-target-runtime --> **existing target runtime** — Ask: “Does the existing target runtime already run?” 3. <!-- email-evals:field=dedicated-actor-environment-name --> **dedicated actor environment name** — Ask: “What is the dedicated actor environment name?” 4. <!-- email-evals:field=dedicated-target-environment-name --> **dedicated target environment name** — Ask: “What is the dedicated target environment name?” 5. <!-- email-evals:field=expected-action --> **expected action** — Ask: “What is the expected action?” 6. <!-- email-evals:field=exact-allowed-recipients --> **exact allowed recipients** — Ask: “Which exact allowed recipients are required?” 7. <!-- email-evals:field=sender --> **sender** — Ask: “What is the sender?” 8. <!-- email-evals:field=reply-to --> **Reply-To** — Ask: “What is the Reply-To expectation?” 9. <!-- email-evals:field=thread --> **thread** — Ask: “What is the thread expectation?” 10. <!-- email-evals:field=subject --> **subject** — Ask: “What is the subject expectation?” 11. <!-- email-evals:field=required-facts --> **required facts** — Ask: “What required facts apply?” 12. <!-- email-evals:field=forbidden-patterns --> **forbidden patterns** — Ask: “What forbidden patterns apply?” 13. <!-- email-evals:field=attachments --> **attachments** — Ask: “What attachments are expected?” 14. <!-- email-evals:field=timeout --> **timeout** — Ask: “What timeout applies?” 15. <!-- email-evals:field=lifecycle --> **lifecycle** — Ask: “What lifecycle outcome is required?” <!-- email-evals:authoring-prompts:end -->

This skill does not build or start the target agent runtime.

Refuse real customer messages, identifiers, customer domains, or production-derived fixtures. Immediately propose a synthetic replacement, such as a fictional order identifier and a `.test` mailbox. Keep every case and attachment synthetic.

Do not scaffold until these answers are sufficient to make deterministic assertions.

Contain the evaluation before setup

Never change e2a protection. Tell the user to configure containment separately, in the same dedicated account for e2a, with an account-scoped API key:

• actor: `allowlist/block [target]`
• target: `allowlist/block [actor, probes...]`

Use dedicated actor and target agents only. Do not infer, broaden, or repair protection settings; surface protection failures during validation instead.

Scaffold, edit, and validate

After gathering sufficient answers, scaffold the suite:

Resolve the absolute directory containing this loaded `SKILL.md` resource. For the tool call, set `EMAIL_EVALS_LAUNCHER` to the absolute path formed by joining that resource directory with `email-evals.sh`. This is an agent-local value, not a persisted environment variable. Do not derive it from the user's current directory or a client-specific plugin-root variable.

"$EMAIL_EVALS_LAUNCHER" scaffold --root <suite-root> --name <suite-name> --target-env <target-env> --actor-env <actor-env>

Edit the generated `suite.yaml` and case YAML files to reflect the answers. The credential name is fixed outside suite authority: export only `E2A_EVAL_API_KEY`. YAML interpolation is limited to the documented actor, target, and probe mailbox fields; never interpolate names, actions, timing, subjects, bodies, patterns, or attachment metadata.

Then verify the checked-in, single-file plugin runtime bundle and validate the suite. Setup performs no package installation and never copies or executes JavaScript or dependencies beneath the suite root:

Resolve the absolute directory containing this loaded `SKILL.md` resource. For the tool call, set `EMAIL_EVALS_LAUNCHER` to the absolute path formed by joining that resource directory with `email-evals.sh`. This is an agent-local value, not a persisted environment variable. Do not derive it from the user's current directory or a client-specific plugin-root variable.

"$EMAIL_EVALS_LAUNCHER" setup --root <suite-root>
"$EMAIL_EVALS_LAUNCHER" validate --suite <suite-root>/suite.yaml

Show the complete alias-only dry-run plan, its `approvalDigest`, and any protection failures. Validation is the preflight gate; do not run a suite while it reports a protection or capability failure. Treat the digest as an opaque value; it binds the resolved identities, origin, containment posture, stimuli, assertions, recipients, and execution limits without revealing them.

The public `https://api.e2a.dev` origin is the default. If the suite names a custom/local origin, the operator must independently authorize the exact origin on every command with `--trusted-origin <origin>`; cleartext origins are restricted to loopback. Never infer this flag from suite content.

Request approval immediately before sending

Ask for explicit user approval immediately before `run`, because it sends real email between the dedicated agents. Do not treat earlier authoring answers as approval.

Only after that approval, run:

Resolve the absolute directory containing this loaded `SKILL.md` resource. For the tool call, set `EMAIL_EVALS_LAUNCHER` to the absolute path formed by joining that resource directory with `email-evals.sh`. This is an agent-local value, not a persisted environment variable. Do not derive it from the user