/testing

name: testing
version: 0.3.0
description: >
  Writes and documents the test suite for a generated cli-web-* CLI (Phase 3): unit
  tests with mocked HTTP, live E2E tests, subprocess tests via _resolve_cli, and the
  TEST.md plan/results record. Use after the methodology skill completes
  implementation.
when_to_use: >
  Trigger phrases: "write tests for cli-web-*", "start Phase 3", "create TEST.md",
  "add E2E tests", "add subprocess tests", "test the CLI". Not for traffic capture,
  implementation, or quality validation (standards).

CLI-Anything-Web Testing

Write and document tests for cli-web-* CLIs. This skill owns the full testing lifecycle: test implementation and test documentation (plan + results).

Copy this checklist and check off items as you complete them:

Phase 3 Progress:
- [ ] Prerequisites: implementation complete, CLI installed, <APP>.md exists
- [ ] Auth verified working (auth login + status) — auth CLIs only
- [ ] Unit tests written (mocked HTTP, typed-exception + helper coverage)
- [ ] E2E tests written (live round-trips, FAIL not skip on missing auth)
- [ ] Subprocess tests written (_resolve_cli pattern)
- [ ] TEST.md Part 1 generated (generate-test-docs.py plan)
- [ ] Full suite green incl. CLI_WEB_FORCE_INSTALLED=1 subprocess run
- [ ] TEST.md Part 2 appended (generate-test-docs.py results)
- [ ] phase-state marked complete

---

Prerequisites (Hard Gate)

Do NOT start unless:

• [ ] Implementation is complete (all core modules + commands exist)
• [ ] `pip install -e .` succeeds and `cli-web-<app>` is on PATH
• [ ] `<APP>.md` exists with API map and auth scheme

If implementation is incomplete, invoke the `methodology` skill first. If the methodology phase is marked `failed` in phase-state, follow `skills/shared/RECOVERY.md` §phase-state Check Failures.

---

Auth Must Be Working Before E2E Tests

For auth-required sites: run `cli-web-<app> auth login` then `auth status` (must show valid). Tests that skip or catch auth errors are broken — use `pytest.fail()` if auth is missing (CONVENTIONS.md §Auth Rules "Tests"). No-auth sites skip auth setup entirely.

---

Write Tests

**Goal:** Comprehensive test suite. Document what you're testing as you write it — TEST.md Part 1 (the plan) is written alongside the test code, not as a separate gate before it.

Testing Layer Strategy

The standard three-layer suite is: **unit tests (mocked HTTP)** + **live E2E tests** + **subprocess tests**. This covers fast correctness, real integration, and installed CLI.

| Layer | File | Purpose | |-------|------|---------| | Unit | `test_core.py` | Core functions with mocked HTTP. No network. Fast. | | E2E live | `test_e2e.py` | Real API calls. Require auth — FAIL (not skip) without it. | | CLI subprocess | `test_e2e.py` | Installed `cli-web-<app>` via `_resolve_cli()`. Full end-to-end. | | Integration (VCR) | `test_integration.py` | Recorded HTTP cassettes via VCR.py. Reproducible, no network. Recommended for RPC protocols. |

**Optional — fixture replay layer:** Only add this if the site has complex HTML parsing or non-trivial response transformations worth preserving. For straightforward JSON APIs, fixture replay adds maintenance cost without much benefit.

| Layer (optional) | File | Purpose | |-----------------|------|---------| | E2E fixture | `test_e2e.py` | Replay captured responses from `tests/fixtures/`. Verifies parsing logic. |

Parallel Test Writing

Dispatch `test_core.py` and `test_e2e.py` writing as parallel subagents — they're independent. Start unit tests during Phase 2 if possible (they don't depend on commands).

Testing Rules

• Unit tests: `unittest.mock.patch` for HTTP, real CSS class names in HTML fixtures
• E2E: require auth (pytest.fail if missing), verify response body fields not just status
• Subprocess: `_resolve_cli("cli-web-<app>")` — see `references/resolve-cli-pattern.md`
• HTML scraper assertions: check actual fields (name, id, price), not just `isinstance(results, list)`
• See `references/test-code-examples.md` for patterns

VCR.py Integration Tests (Recommended for RPC/GraphQL)

For complex protocols, add a recorded-cassette layer between unit and live E2E — real responses, replayed offline. Setup, recording workflow, and the marker convention: `references/vcr-testing.md`.

Fixture Realism (for HTML scrapers)

If the CLI uses HTML scraping (BeautifulSoup, lxml), unit test fixtures must mirror the real page's CSS class structure — not a generic simplified table.

A fixture like `<table><tr><td>GK</td><td>95</td></tr></table>` will pass even if the real parser is completely broken against the live site's actual markup. The parser was written to match specific CSS classes (`table-player-name`, `platform-ps-only`, `table-pos-main`) — the fixture must have those same classes.

When to apply this: any CLI module that calls `.find(class_=...)` or `.find_all(...)` on response HTML. If the module only parses JSON (`resp.json()`), skip this — JSON fixtures are naturally structural.

Practical check: look at your parser's `.find(class_="...")` calls. If your fixture HTML doesn't contain those exact class names, the fixture is not testing the parser.

CLI Output Sanity Checks (Critical)

Every `--json` output must be checked for raw protocol leakage. These are bugs the agent MUST catch before declaring tests pass:

# In E2E tests, assert the output is real data, not raw RPC fragments:
def test_chat_returns_text_not_rpc(client, notebook_id):
    """Chat answer must be human-readable text, not raw batchexecute chunks."""
    result = client.chat_query(notebook_id, "What is this about?")
    # RED FLAGS — fail if any of these appear in the answer:
    assert "wrb.fr" not in result, "Raw RPC data leaked into chat output"
    assert "af.httprm" not in result, "Raw RPC data leaked into chat output"
    assert '"di"' not in result, "Raw RPC data leaked into chat output"
    assert len(result) > 50, "Answer too sh