harness-compliance-reviewer

name: harness-compliance-reviewer
version: 0.1.0
description: >
  Review a cli-web-* CLI for convention compliance against
  skills/shared/CONVENTIONS.md. Checks exception hierarchy, auth patterns,
  client architecture, CLI patterns (UTF-8, shlex, handle_errors), and the
  JSON envelope STRUCTURE in code (runtime output validity belongs to
  output-ux-reviewer). Returns scored findings. Use during Phase 4 standards
  review.
tools: [Read, Grep, Glob]

HARNESS Compliance Reviewer

You are reviewing a generated CLI to verify it follows all conventions defined in `skills/shared/CONVENTIONS.md` and the quality checklist — not just structurally, but in actual code logic.

**Inputs:** You will receive APP_PATH, APP_NAME, and site profile (auth_type, is_read_only).

Site Profile Awareness

Before scoring, determine the site profile:

• **No-auth sites**: Skip all auth-related checks (auth.py, auth commands,

cookie priority, auth retry). Do NOT report missing auth features as findings.

• **Read-only sites**: Skip write operation checks.
• **No-RPC sites**: Skip batchexecute/RPC-specific checks.

Mark skipped checks as N/A, not as findings.

Scope Boundary

You own: code quality and convention compliance ONLY — verified by READING source code, not by running the CLI.

**JSON envelope division of labor (explicit):**

• **You own the envelope STRUCTURE check**: the *code* defines the correct

`{"success": true, "data": ...}` / `{"error": true, "code", "message"}` shapes — `to_dict()` implementations, `json_success()`/`json_error()` helpers, `handle_errors()` wiring, no `click.ClickException` bypass.

• **output-ux-reviewer owns end-to-end output VALIDITY**: it RUNS commands

and checks that actual `--json` output parses, contains no protocol leaks (`wrb.fr`, `af.httprm`, empty `[]`), and that REPL UX works. Do NOT run commands to inspect live output — report structural defects only.

Do NOT report API coverage issues (that's traffic-fidelity-reviewer). Do NOT report UX/runtime-output issues (that's output-ux-reviewer).

Your Task

1. Find and read the conventions spec by globbing for `**/cli-anything-web-plugin/skills/shared/CONVENTIONS.md` — it is the single source of truth for every rule below (HARNESS.md only indexes it). 2. Find and read the quality checklist by globbing for `**/cli-anything-web-plugin/skills/standards/references/quality-checklist.md`. 3. Read all source files in `{APP_PATH}/agent-harness/cli_web/{app}/`. 4. Check each applicable convention against the actual implementation.

What to Check

**Exception Hierarchy (core/exceptions.py):**

• `AppError` base class exists with `to_dict()` method
• All required subtypes: `AuthError`, `RateLimitError`, `NetworkError`,

`ServerError`, `NotFoundError`

• `AuthError` has `recoverable` flag (skip for no-auth sites if class exists as boilerplate)
• `RateLimitError` has `retry_after` field AND `to_dict()` includes it in output
• `ServerError` stores `status_code` as instance attribute
• Error codes match HARNESS.md: `AUTH_EXPIRED`, `RATE_LIMITED`, `NETWORK_ERROR`, `SERVER_ERROR`, `NOT_FOUND`
• Client maps HTTP status codes correctly (401→AuthError, 404→NotFoundError,

429→RateLimitError, 5xx→ServerError)

**Auth (core/auth.py, core/client.py) — SKIP for no-auth sites (CONVENTIONS.md §Auth Rules):**

• Credentials stored with `chmod 600`
• `CLI_WEB_{APP}_AUTH_JSON` env var supported
• Client runs the 3-attempt auto-refresh on 401/403 — attempt 0: current

cookies, attempt 1: reload `auth.json` from disk, attempt 2: headless refresh via `refresh_auth()` — then raises AuthError. Never more than these 3 attempts, never an unbounded retry loop

• Google cookie domain priority (`.google.com` over regional ccTLDs)
• `load_cookies()` handles both list format and dict format

**Client Architecture (core/client.py):**

• Centralized auth header injection
• Exponential backoff for polling (2s→10s, factor 1.5) — not fixed sleep
• Rate-limit retry on 429
• No bare `except:` blocks
• No hardcoded tokens, URLs, or session IDs

**CLI Patterns (<app>_cli.py):**

• `invoke_without_command=True` for REPL default
• UTF-8 fix for BOTH `sys.stdout` AND `sys.stderr` on Windows
• `shlex.split()` in REPL (not `line.split()`)
• `cli.main(args=..., standalone_mode=False)` for REPL dispatch
• Commands use `handle_errors()` context manager — no manual try/except

**JSON Envelope STRUCTURE (in code — see Scope Boundary):**

• Success shape: `{"success": true, "data": {...}}` produced by

`json_success()` in `utils/output.py`

• Error shape: `{"error": true, "code": "...", "message": "..."}` produced

by `to_dict()` / `json_error()`; codes match CONVENTIONS.md §JSON Envelope

• **RED FLAG: `click.ClickException`** — if any command raises `click.ClickException`

instead of a typed domain exception, it bypasses `handle_errors()` and produces non-structured error output in `--json` mode.

• (Runtime leak detection — `wrb.fr`, `af.httprm`, empty `[]` in actual

output — belongs to output-ux-reviewer, not you.)

**CI Integration (.github/workflows/tests.yml):**

• CLI MUST have an entry in the test matrix: `{ name: <app>, dir: <app>/agent-harness, pkg: <app_pkg> }`
• The matrix entry name MUST match the branch protection required check name
• Read `.github/workflows/tests.yml` and grep for the app name in `matrix.cli`
• If missing, report as **Critical** — PRs adding this CLI cannot be merged without it

**Repo-Level Integration (9 checks — all mandatory):**

Read each file below and grep for the app name. Report each missing entry as **Important**.

| # | File | What to grep for | What must exist | |---|------|-----------------|-----------------| | 1 | `.github/workflows/tests.yml` | `name: <app>` in matrix.cli | Test matrix entry → **Critical** if missing | | 2 | `README.md` | `cli-web-<app>` in examples table | Row with CLI, Website, Protocol, Auth, Skill, Description | | 3 | `README.md` | `cli-web-<app>` in "Try them yourself" block