/ui-automation-workflows

name: ui-automation-workflows
description: Accessibility-first UI automation using IDB. Query accessibility tree (fast, 50 tokens) before screenshots (slow, 170 tokens). Use when automating simulator interactions, tapping UI elements, finding buttons, or testing user flows. Covers idb-ui-describe, idb-ui-tap, idb-ui-find-element patterns.

UI Automation Workflows

**Use the `execute_idb_command` MCP tool for all UI automation**

The xclaude-plugin provides the `execute_idb_command` MCP tool which consolidates all IDB UI automation operations into a single, token-efficient dispatcher.

⚠️ CRITICAL: Always Use MCP Tools First

**This is the most important rule:** When automating UI interactions, you MUST use the `execute_idb_command` MCP tool.

• ✅ **DO**: Invoke `execute_idb_command` for all UI automation, element finding, and accessibility queries
• ✅ **DO**: If the MCP tool fails, adjust parameters and retry
• ✅ **DO**: Read error messages and debug the parameters
• ❌ **NEVER**: Fall back to bash `idb` commands
• ❌ **NEVER**: Use `idb` directly in bash
• ❌ **NEVER**: Run `idb` commands in a terminal

**Why?** The MCP tool provides:

• Structured error handling
• Token efficiency (consolidated into 1 tool vs. verbose bash output)
• Proper integration with the xclaude-plugin architecture
• Accessibility-first patterns built-in

If `execute_idb_command` fails, the issue is with parameters or app state - not that you should use bash.

Core Principle: Accessibility Before Screenshots

**Always query the accessibility tree first.** Only use screenshots as a fallback.

Use the `execute_idb_command` MCP tool with operation `describe` to access the accessibility tree.

Why Accessibility-First?

| Approach | Time | Tokens | Reliability | |----------|------|--------|-------------| | Accessibility tree | ~120ms | ~50 | Survives theme changes | | Screenshot | ~2000ms | ~170 | Breaks on visual changes |

**Result: 3-4x faster, 80% cheaper, more reliable**

Standard Workflow

1. Check Accessibility Quality (Optional) - Use `execute_idb_command`

Before starting automation, check if the app has good accessibility support:

Invoke the `execute_idb_command` MCP tool:

{
  "operation": "check-accessibility",
  "target": "booted"
}

**Interprets:**

• **"excellent"** or **"good"**: Proceed with accessibility-first workflow
• **"poor"** or **"insufficient"**: May need to rely more on screenshots

**Note:** Most modern iOS apps have good accessibility support. Skip this check if you're confident.

2. Query Accessibility Tree - Use `execute_idb_command` with operation: "describe"

**This is your starting point for all UI automation:**

Invoke the `execute_idb_command` MCP tool:

{
  "operation": "describe",
  "target": "booted",
  "parameters": {
    "operation": "all"
  }
}

**Returns:**

{
  "elements": [
    {
      "label": "Login",
      "type": "Button",
      "frame": { "x": 100, "y": 400, "width": 175, "height": 50 },
      "centerX": 187,
      "centerY": 425,
      "enabled": true,
      "visible": true
    },
    {
      "label": "Email",
      "type": "TextField",
      "value": "",
      "frame": { "x": 50, "y": 300, "width": 275, "height": 44 },
      "centerX": 187,
      "centerY": 322
    }
  ]
}

**Use `centerX` and `centerY` for tap coordinates.**

3. Find Your Element

**Option A: Search by Label/Text (Preferred)**

{
  "operation": "find-element",
  "target": "booted",
  "parameters": {
    "query": "Login"
  }
}

**Option B: Manual Search**

From the accessibility tree response, find the element you want by:

• `label`: Button text, field labels
• `type`: Button, TextField, Cell, etc.
• `value`: Current input value
• `visible`: Only interact with visible elements

4. Interact with Element

**Tap:**

{
  "operation": "tap",
  "target": "booted",
  "parameters": {
    "x": 187,
    "y": 425
  }
}

**Input Text:**

{
  "operation": "input",
  "target": "booted",
  "parameters": {
    "text": "user@example.com"
  }
}

**Keyboard Actions:**

{
  "operation": "input",
  "target": "booted",
  "parameters": {
    "key": "return"
  }
}

Available keys: `return`, `home`, `delete`, `space`, `escape`, `tab`, `up`, `down`, `left`, `right`

5. Verify State

After interaction, query accessibility tree again to verify:

{
  "operation": "describe",
  "target": "booted"
}

Common Patterns

Pattern: Login Flow

1. describe → Find "Email" text field
2. tap → Focus email field
3. input → Type email
4. describe → Find "Password" text field
5. tap → Focus password field
6. input → Type password
7. describe → Find "Login" button
8. tap → Submit form
9. describe → Verify next screen

Pattern: Navigate and Tap

1. describe → Get all buttons
2. find-element → Search for specific button
3. tap → Execute tap
4. describe → Verify navigation

Pattern: Fill Form

1. describe → Get all text fields
2. For each field:
   - tap → Focus field
   - input → Enter text
   - input key:return → Next field
3. describe → Find submit button
4. tap → Submit

Pattern: Scroll and Find

1. describe → Check if element visible
2. If not visible:
   - gesture (swipe up) → Scroll
   - describe → Check again
3. find-element → Locate target
4. tap → Interact

Gestures

Swipe

{
  "operation": "gesture",
  "target": "booted",
  "parameters": {
    "gesture_type": "swipe",
    "direction": "up",
    "duration": 200
  }
}

Directions: `up`, `down`, `left`, `right`

Button Presses

{
  "operation": "gesture",
  "target": "booted",
  "parameters": {
    "gesture_type": "button",
    "button": "HOME"
  }
}

Buttons: `HOME`, `LOCK`, `SIRI`, `SIDE_BUTTON`, `APPLE_PAY`, `SCREENSHOT`, `APP_SWITCH`

When to Use Screenshots (Fallback Only)

**Only use screenshots if:**

1. **Accessibility quality is "poor"**

   { "operati