/testbench-test-handling

name: testbench-test-handling
description: Use this skill when running or writing automated tests against the testbench — the three-phase execution protocol every test case follows, live progress on the Pi's web UI, blocking prompts for physical operator actions (button press, cable swap, power cycle), the TestbenchDriver Python API, and activity log queries. Use it for authoring a pytest suite as well as for tracking a manual run. For driving one instrument, use that instrument's skill instead. Triggers on "test progress", "test session", "test spec", "test case", "test harness", "run the tests", "write a test", "TestbenchDriver", "human interaction", "operator", "activity log", "test panel".

ESP32 Test Automation

Base URL: `$TESTBENCH_URL` — see Step 0

**The portal and the MQTT broker are always-on infrastructure. A test never starts, stops or restarts them** — doing so breaks whatever else is using the bench, and a test that needs a restart to pass is testing the restart.

The execution protocol

Every test case runs in three phases, and the panel shows which one is current so the operator can follow along without a terminal.

| Phase | Panel shows | What happens | |-------|-------------|--------------| | **Preconditions** | `[TC-100] Preconditions: checking DUT reachable...` | Verify **and establish** each precondition from the spec. If the AP is not running, start it. Fail only when a precondition is genuinely unrecoverable. | | **Execute** | `[TC-100] Step 2: <the step from the spec>` | Run the spec's steps one at a time, checking the expected result after each. | | **Result** | `TC-100: PASS` / `TC-100: FAIL — <what was seen>` | Record PASS/FAIL/SKIP with the observed value, not just the verdict. |

Rules that decide whether a run is worth anything:

1. **The spec is the script.** Execute its preconditions, steps and pass criteria as written — not an improved version of them. 2. **Update the panel before each action**, never after. The panel is how the operator knows what the bench is doing to the hardware right now. 3. **Record baselines.** Where the spec says "record X as `X_before`", capture it and compare in the result phase. 4. **Generate test credentials per run.** A random SSID and password for the test AP proves the DUT used what it was provisioned with, rather than a network it had already cached. 5. **Write the results out.** A markdown file with test ID, name, result, details and timestamps — the panel is live state, not a record.

Workflows that span several instruments — provision, reboot, re-provision, soak — are in [`references/common-workflows.md`](references/common-workflows.md).

Step 0: Point at a bench

There are several benches and their addresses move, so nothing here writes one down. `$BENCH` is not usable either — a container cannot resolve mDNS. Discover the bench and export its URL:

export TESTBENCH_URL=$(sudo python3 .claude/skills/esp-idf-handling/discover-testbench.py \
                         --url --name <bench-hostname>)
curl -s "$TESTBENCH_URL/api/info"        # confirm before anything else

`--url` refuses to guess when more than one bench answers, so `--name` is required whenever a second bench is powered on. `TESTBENCH_URL` is the same variable `pytest --wt-url` falls back to.

Step 0.5: Reach the DUT through the API, never through raw RFC2217

The slots expose RFC2217 ports, and it is tempting to open one with pyserial and read the device directly. **Do not do this to a DUT.** Use `/api/flash`, `/api/serial/monitor`, `/api/serial/reset` and `/api/chip/info`, which implement each chip's sequences correctly.

The reason is that **serial control lines are not inert on modern parts**. On an ESP32-C3, -S3 or any native-USB device, the USB-Serial/JTAG controller reads them as boot-mode signals: **DTR asserted selects download mode, RTS asserted holds the part in reset**. pyserial asserts both on open by default, and the Linux CDC-ACM driver asserts them too. So merely *connecting to look at* a DUT can stop it — and what you then observe is your own connection, not the firmware.

That failure is vicious because it is silent and it is self-confirming: the device prints nothing, which reads as a crash or a hang; a reset appears to fix it; and repeating the observation reproduces the silence, which feels like evidence. On a UART-bridge part (CP2102, CH340) the same lines usually drive an auto-reset circuit, so the effect is a restart rather than a halt — quieter still, because the device looks alive.

**To write to a device, use `POST /api/serial/write`** (FR-030). It takes `text` or `hex`, opens nothing, and disturbs no control line.

**The bench's own DUT answers.** `test-firmware/` gives the bench a device that talks back, so a write can be *proved* to have arrived rather than assumed. One line in, one line out, no echo and no prompt:

| Command | Reply | Use | |---|---|---| | `ping` | `OK pong` | the write reached the device | | `status` | `OK status wifi=… ap_mode=… ip=… mac=…` | what the DUT thinks its network is | | `scan` | `OK scan <n>`, then `<rssi> <ch> <auth> <ssid>` per line | **what the DUT's own radio can hear** | | `info` | `OK info project=… version=… idf=…` | which image is *actually* running | | `mark <text>` | `OK mark <text>` | an observable on demand, without waiting for the heartbeat | | `wifi <ssid> [pass]` | `OK wifi stored …` then reboots | provision over the wire, no radio needed | | `forget` / `reboot` | `OK …` then reboots | back to the portal / restart |

`scan` is the one worth reaching for first when a DUT will not join. The bench can scan and the DUT could not, so a failure to meet had only one witness — and `NO_AP_FOUND` from the DUT reads as the DUT's fault when it is just as often the AP that is not on the air. Two radios reporting is a measurement; one is an assertion.

`info` settles the other recurring waste of time: after a flash, ask the device what it is runni