/browser-automation

name: browser-automation
description: Drive the PenguinHarness desktop app's built-in browser from the shell with `penguin browser` — open pages, read them as simplified HTML or text, act with JavaScript and trusted clicks and typing, and pull structured data out of them (orders, search results, tables), signed in with the user's own accounts. Use it for any task on a website that needs a real browser or the user's sign-in, such as finding an Amazon order.

Browser automation

The desktop app has a built-in browser in its side panel: the **Browser** tab of the dock. `penguin browser` drives it from your shell. Its tabs are shared by every conversation and keep the sign-ins made in them, so a page you open is one the user can watch and take over. Every command acts on the active tab unless `--tab <id>` names another.

The browser exists only inside the desktop app. A server started any other way (`penguin web`, Docker, a remote machine) has none.

Before you start

If the user's message only names this skill without a task, ask what they want done on the web. Then check the browser:

penguin browser status

`status: available` is followed by the open tabs. `status: unavailable (…)` (exit code 1) means the desktop app is not what runs this conversation, or it has no window open: tell the user that the built-in browser needs the PenguinHarness desktop app, open, and stop there — never answer from guessed page content.

A `warning:` line under `memory:` means the browser is using too much memory or holds too many tabs. Close the tabs you opened and no longer need (`penguin browser close <tab-id>`) before opening more, and reuse a tab (`open` without `--new-tab`) where you can.

The loop

1. **Go**: `penguin browser open <url>` navigates the active tab (and opens one when there is none); `--new-tab` opens another. It waits for the load and prints `tab <id> · <title> · <url>`. 2. **Look**: `penguin browser scan --text` first — plain text, cheap. `penguin browser scan` returns simplified HTML when you need selectors. Both print the tab header, the tab list (`tabs: *12 Your Orders | 15 Google`, the active tab starred), a `---` rule, then the page. 3. **Act**: `penguin browser exec` runs JavaScript in the page. Put the script in a quoted heredoc and `return` exactly what you need, as compact JSON for anything structured. 4. **Check**: exec prints labelled lines — `status:` and `tab:`, `return:`, `diff:` (how many elements changed, the largest change indented beneath), `transients:` (messages that appeared and may be gone again, such as "Added to cart"), `new tabs:`, and a `note:`. When they do not settle whether it worked, scan again.

Prefer one precise exec over repeated scans: a scan costs thousands of tokens, a targeted exec a few dozen.

exec

penguin browser exec <<'EOF'
const cards = [...document.querySelectorAll('.result')].slice(0, 10);
return JSON.stringify(cards.map((c) => ({
  title: c.querySelector('h2')?.innerText.trim(),
  href: c.querySelector('a')?.href,
})));
EOF

• The value is the script's explicit `return`, or else its last expression: `penguin browser exec 'document.title'` prints the title. Top-level `await` works. **Put an explicit `return` on its own last line**: it is the one form that means the same in every script. A last line that is a statement (`el.click();`) gives `return: undefined`.
• Quote the heredoc delimiter (`<<'EOF'`) so the shell leaves `$`, backticks and quotes alone. `--file script.js` works too, and so does a one-liner argument.
• `return:` is cut at 8000 characters. For more, add `--save out.json`: the whole value goes to the file (a string as-is, anything else as JSON) and only its start is printed. Then read the file.
• `--no-monitor` skips change tracking: faster, for scripts that only read.
• `--timeout 60s` for a slow script (the default is 15 s). Poll inside the script for content that loads late (see [reference/page-recipes.md](reference/page-recipes.md)).
• `status: failed` with an `error:` line means the script threw; the exit code is 1.
• `page: reloaded` in the status line means the page navigated while the script ran, so its JavaScript context is gone. Scan or exec again on the new page.

Rules that save retries

• **Navigating and acting on the new page are two calls.** A script that sets `location.href` or clicks a link and then reads the page fails, because the page it was running in is gone. Navigate first (`open`, or an exec that only navigates), then act in the next exec.
• **Never guess selectors.** Scan first and take them from the HTML. Big sites generate their class names; prefer ids, `name`, `aria-label`, `data-*` attributes, roles and visible text.
• Scan shortens long lists to three items plus `[FAKE ELEMENT] N more items hidden, selector: "…"`. Query that selector in an exec for the rest.
• Scan leaves out hidden, floating and covered elements (sidebars, overlays, closed menus). If something you expect is missing, look for it with exec (`document.body.innerText`, `querySelectorAll`).
• A scan that is empty or incomplete may be a page still rendering: wait a moment and scan again before concluding anything.
• **Trusted input.** An `el.click()` from JavaScript is an untrusted event that some sites ignore: buttons that open popups, custom dropdowns, file pickers. Use `penguin browser click '<selector>'` (`--index n` for the n-th match) or `click --at x,y` — a real mouse move, press and release. `penguin browser type '<text>' --selector '<css>' --submit` types for real and presses Enter.
• Setting a field from JavaScript needs the native value setter and an `input` event, or React and Vue will not notice (recipe in [reference/page-recipes.md](reference/page-recipes.md)).
• Check `disabled` before clicking a button; a disabled button's click does nothing.
• Popups and `target=_blank` links open as new tabs: exec and click list them under `new tabs:`. Continue there with `--tab <id>` or