/evaluating-as-user

name: evaluating-as-user
description: Evaluates software quality from an end user's perspective. Use when emulating a real user to assess a running web app, CLI tool, or API.

Evaluating as a User

Methodology and judgement for evaluating a running product as a real user would: derive what to test from the PRD, structure the evaluation, assess quality dimensions, format findings. Covers **what to evaluate and how to think about quality** — browser mechanics are in playwright-cli (preloaded); agent role, inputs, status codes, and handoff are in the agent definition.

**Applies to:** web applications, CLI tools, APIs. **Not covered:** source code review (reviewing-code), automated test generation, accessibility audits (WCAG), performance benchmarking.

Workflow

Step 1: Derive user journeys from the PRD

Read the PRD. For each P0 requirement, derive one user journey using this structure:

| Field | Description | |-------|-------------| | **Starting point** | Where the user begins (URL, command, state) | | **Actions** | Step-by-step sequence the user would take | | **Observable outcome** | What confirms the requirement is satisfied |

**How to derive actions:** Treat the requirement as a goal, not a task. Ask: what would a person with this goal actually do? Start from the entry point a new user would reach, not the most direct technical path.

**Priority:** P0 requirements first. Add P1 journeys only if time permits after all P0 journeys and universal checks are complete.

**Non-web projects:** For CLI tools, the starting point is the terminal and the actions are command sequences. For APIs, the starting point is an endpoint and the actions are the sequence of requests a user of the API would make. The journey structure is the same.

Step 2: Run PRD-derived journeys

Attempt each derived journey in sequence. As you go:

• Execute each action in the journey
• Record what you observe at each step in `walkthrough.md` (what you did, what you expected, what you saw)
• Take a screenshot at each meaningful state transition (web projects)
• Note any friction, confusion, missing feedback, or failure

When a journey succeeds, note which quality dimensions (Step 4) were satisfied or strained along the way. When a journey fails, record at what step and what the failure was.

**If a severity-4 issue is encountered mid-journey:** record it, then attempt to continue or skip to the next journey. Do not abandon the evaluation — continue with remaining journeys and universal checks, and highlight the catastrophic finding in the final report.

Example walkthrough entry:

Journey 1, Step 3 — Submit registration form
Did: Filled in name, email, and password fields; clicked "Create account"
Expected: Loading state on button, then redirect to dashboard or confirmation screen
Saw: Button remained active with no visual change for ~4 seconds, then page refreshed to the same form with no error message
Quality lens: Communicativeness failure (no feedback); Reliability feel strained (unpredictable outcome)

Step 3: Apply universal checks

After PRD journeys, run these checks on every product regardless of what the PRD specifies:

| Check | What to do | What to look for | |-------|-----------|-----------------| | **First landing** | Load the primary screen fresh | Does the product communicate its purpose within 5 seconds? Is there a clear visual hierarchy? Does the user know what to do next? | | **First-time empty state** | Access the product as a new user with no data | Are empty views explained and actionable, or blank and confusing? | | **Error paths** | Submit invalid input, trigger validation, attempt impossible actions | Are errors in plain language with a clear fix suggestion? Do they appear where the user is looking? | | **Zero-results** | Search or filter for something that will not exist | Is there a clear message that the search worked but found nothing? Is there an actionable next step? | | **Visual consistency** | Navigate to three or more different views | Do colour, spacing, typography, and interaction patterns hold consistent throughout? | | **Feedback absence** | Trigger any async action (form submission, data save, deletion) | Does the product visibly confirm that the action happened? |

**Non-web CLI tools:** Replace first landing with "first run of the tool with no arguments or `--help`". Replace visual consistency with "consistency of output format, flag naming, and error message style". Skip screenshot-dependent checks.

**APIs:** Replace first landing with "calling the root endpoint or documentation endpoint". Replace visual checks with "consistency of response envelope structure, error format, and status code usage".

Step 4: Assess quality dimensions

**Note:** These dimensions are a lens you apply throughout Steps 2, 3, and 5 — not a separate sequential activity. Return here as a reference when assessing findings, not as a distinct pass after universal checks.

For each journey and each universal check, assess the product across these five dimensions. You do not need a score per check — use the dimensions as a lens to ensure findings cover the full experience, not just functional failures.

| Dimension | The question | What failure looks like | |-----------|-------------|------------------------| | **Communicativeness** | Does the product tell me what's happening? | Silent submissions, missing loading states, absent feedback, cryptic error codes | | **Learnability** | Can I figure out how to use it without help? | Jargon, non-standard navigation patterns, no contextual guidance, forcing users to memorise | | **Reliability feel** | Does it feel trustworthy and consistent? | Visual inconsistencies, unpredictable behaviour, poor error recovery, performance anxiety | | **Efficiency** | Does it respect my time? | Unnecessary steps, buried primary actions, no defaults, repetitive data entry | | **Emotional tone** | Does it feel crafted or careless? | Unstyled edge