/playwright-camera-mask-testing

name: playwright-camera-mask-testing
description: >-
  Specialization of playwright-harness for CAMERA / segmentation testing in the
  browser: put a real PERSON in front of getUserMedia so MediaPipe / selfie
  segmentation produces an actual mask, then verify background-replacement /
  blur / shader effects by vision. Use when the task is "test the mask on web",
  "verify segmentation in the browser", "feed the camera a person fixture",
  "check a camera/background effect in Chrome", or any time a web app must SEE
  a human through the webcam without physical hardware. Read playwright-harness
  FIRST for script conventions and execution; this skill only adds the
  camera-feed pieces. The web analog of android-emulator-mask-testing.
  Validated on Linux/WSL, headless Chromium, against MediaPipe Selfie
  Segmentation.

Camera / Mask Testing: specialization of playwright-harness

**Read `playwright-harness` first.** The base owns prerequisites, the run pattern (write to `/tmp/pw-*.mjs`, run with `playwright` resolvable), and the drive/assert

• WebGL-GPU patterns. This skill changes only how the camera is fed. MediaPipe

selfie segmentation runs in WASM and works headless.

Prerequisites (in addition to the base)

`ffmpeg`, only to turn the still fixture into the looping `.y4m` Chrome plays as a fake camera; no system install needed, `npm i ffmpeg-static` in a /tmp dir gives a binary path.

This skill ships two fixtures (derivations and provenance in `fixtures/README.md`):

• **`fixtures/person-web.png`**: a 640x480 webcam-shaped frame, full-body

subject occupying ~82% of frame height, vertically centered, on a clean light-gray ground (`#CDD0D4`). Derived from `android-emulator-mask-testing/fixtures/person-framed.png`. Web presents the capture file **1:1** (no sensor crop), so do NOT reuse the Android fixture directly; its left-quarter framing compensates an emulator sensor shift that does not exist on web.

• **`fixtures/office-empty.png`**: a 1280x720 empty office, the **no-person

negative case**: a realistic scene (furniture, plants, windows, hard shadows) with nothing segmentable in it. Use it to mock the scene itself when the feed should contain no one.

The three camera modes (pick by what you are asserting)

1. **Green-ball mode** (Chromium's built-in synthetic feed): proves the pipeline RUNS end to end (getUserMedia resolves, effects apply, no page errors). Useless for mask quality; the feed has no person.

   args: [
     '--use-fake-device-for-media-stream',
     '--use-fake-ui-for-media-stream',          // auto-grants the permission prompt
     '--autoplay-policy=no-user-gesture-required',
   ]

2. **Person mode** (file-backed feed): proves mask SHAPE and compositing (person kept, background replaced/blurred). Add one flag; it **requires** `--use-fake-device-for-media-stream` to be present too:

   args: [
     '--use-fake-device-for-media-stream',
     '--use-file-for-fake-video-capture=/tmp/person-web.y4m',
     '--use-fake-ui-for-media-stream',
     '--autoplay-policy=no-user-gesture-required',
   ]

The file must be **`.y4m`** (uncompressed YUV4MPEG2) or **`.mjpeg`**; Chrome loops it forever. Generate the y4m from the fixture at test time; it is too big to keep around (size = W x H x 1.5 bytes x frames; 640x480 x 2 s @ 15 fps ≈ 14 MB). No system ffmpeg needed: `npm i ffmpeg-static` in a /tmp dir.

   ffmpeg -loop 1 -i fixtures/person-web.png -t 2 -r 15 -pix_fmt yuv420p /tmp/person-web.y4m

Keep dimensions even (yuv420p requirement) and resolution ≤ 720p. A malformed y4m (odd dimensions, wrong `pix_fmt`) makes Chrome play nothing with no error: a dead-looking feed that mimics segmentation falling through, so regenerate the y4m before any deeper diagnosis.

3. **Empty-scene mode** (same flags, `office-empty.png` as the file): the guaranteed-clean no-person feed. The fixture is chosen because it does NOT false-positive (validated clean at a 0.75 segmentation-confidence cutoff; that is the fixture's property, not an app's tuned threshold), so whole-frame assertions stay unconfounded: background replacement should cover the WHOLE frame, blur should blur the whole frame uniformly, and any sharp person-shaped region that appears is the app's bug (an empty mask not honored), not the feed's. Use it for effects like blur where a person, real or spurious, would muddy the read.

Asserting the mask

Gate on page errors (collect `pageerror` + console `error`, filter favicon/DevTools noise) the same as the base, then drive the effect and **Read the screenshot** (vision):

• **Working mask:** the person is kept (sharp, full silhouette) and the

background is replaced (image effect) or blurred (blur effect).

• **Segmentation fell through:** the frame is unchanged, or the WHOLE frame got

the effect (no stencil). Check the network panel/console: web MediaPipe typically loads from a CDN, so an offline run or a strict CSP kills it.

• **Empty mask:** the person vanishes into the background layer.
• **Halo / ragged edges:** real signal, not a harness bug; this rig is how you

tune the app's threshold/hardness controls. Iterate the controls and re-Read.

A static fixture **cannot test temporal stability** (flicker, edge crawl); for that, encode a moving subject: `ffmpeg -i subject.mp4 -t 5 -r 15 -pix_fmt yuv420p -vf scale=640:480 /tmp/subject.y4m` (same loop behavior).

Serving an exported static site

Use the base's serving recipe (playwright-harness, "Serving an exported static site"). The camera-specific trap: a pre-rendered page can look alive while its entry JS 404s and the camera is never requested, so a camera that never starts on the deployed site IS that bug until proven otherwise.

GPU note (delta)

WASM segmentation survives SwiftShader, so green-ball and person modes work headless with no GPU flags. Only HEAVY generative-shader laye