/testdriver-locating-elements

name: testdriver:locating-elements
description: Locate UI elements using AI

<!-- Generated from locating-elements.mdx. DO NOT EDIT. -->

Overview

The TestDriver element finding system uses AI. It finds elements on the screen with natural language descriptions. The `find()` method returns an `Element` object. You can [interact with the object](/interacting-with-your-app).

Finding Elements

find()

Find an element on the screen with a natural language description.

const element = await testdriver.find(description)

**Parameters:**

• `description` (string) - A natural language description of the element to find

**Returns:** `Promise<Element>` - The Element instance that TestDriver found

**Example:**

// Find a button
const submitButton = await testdriver.find('the submit button');

// Find an input field with context
const emailField = await testdriver.find('email input field in the login form');

// Find an element by visual characteristics
const redButton = await testdriver.find('red button in the top right corner');

<Tip> Be specific in your descriptions. Include visual details, location context, or nearby text to make the accuracy better. </Tip>

Element Class

The `Element` class represents a located (or to-be-located) UI element. It provides methods for interaction and properties for element information. For interaction methods like `click()` and `hover()`, see [Interacting With Your App](/interacting-with-your-app).

Methods

found()

Check if the element was successfully located.

element.found()

**Returns:** `boolean` - True if element coordinates were found

**Example:**

const element = await testdriver.find('login button');
if (element.found()) {
  await element.click();
} else {
  console.log('Element not found');
}

find()

Re-locate the element, optionally with a new description.

await element.find(newDescription)

**Parameters:**

• `newDescription` (string, optional) - New description to search for

**Returns:** `Promise<Element>` - This element instance

**Example:**

// Re-locate if the UI changed
const element = await testdriver.find('submit button');
// ... page updates ...
await element.find(); // Re-locate with same description

// Or update the description
await element.find('blue submit button'); // Now looking for blue button

Properties

Element properties provide additional information about located elements. Properties are available after a successful `find()` call.

coordinates

Get the element's coordinates object containing all position information.

const coords = element.getCoordinates()
// or access directly
element.coordinates

**Returns:** `Object | null` - Coordinate object with `{ x, y, centerX, centerY }`

**Example:**

const button = await testdriver.find('submit button');
const coords = button.coordinates;

if (coords) {
  console.log(`Top-left: (${coords.x}, ${coords.y})`);
  console.log(`Center: (${coords.centerX}, ${coords.centerY})`);
}

x, y, centerX, centerY

Direct access to coordinate values. Always available after successful `find()`.

element.x        // Top-left X coordinate (number)
element.y        // Top-left Y coordinate (number)
element.centerX  // Center X coordinate (number)
element.centerY  // Center Y coordinate (number)

**Example:**

const button = await testdriver.find('submit button');
console.log(`Button at: (${button.x}, ${button.y})`);
console.log(`Button center: (${button.centerX}, ${button.centerY})`);

// Use for custom mouse operations
await testdriver.click(button.centerX, button.centerY);

width, height

Element dimensions in pixels. Available when AI detects element bounds.

element.width   // Width in pixels (number | null)
element.height  // Height in pixels (number | null)

**Example:**

const button = await testdriver.find('submit button');

if (button.width && button.height) {
  console.log(`Button size: ${button.width}x${button.height}px`);
  
  // Check if button is large enough
  if (button.width < 50) {
    console.warn('Button might be too small');
  }
}

boundingBox

Complete bounding box information including position and dimensions.

element.boundingBox

**Returns:** `Object | null` - Bounding box with all dimension data

{
  x: number,        // Top-left X
  y: number,        // Top-left Y
  width: number,    // Width in pixels
  height: number    // Height in pixels
}

**Example:**

const element = await testdriver.find('dialog box');

if (element.boundingBox) {
  const { x, y, width, height } = element.boundingBox;
  console.log(`Dialog: ${width}x${height} at (${x}, ${y})`);
  
  // Calculate if element is in viewport
  const rightEdge = x + width;
  const bottomEdge = y + height;
  console.log(`Element extends to (${rightEdge}, ${bottomEdge})`);
}

screenshot

Base64-encoded PNG screenshot of the screen when element was found. Only available in DEBUG mode or when an error occurs.

element.screenshot

**Returns:** `string | null` - Base64-encoded PNG image

**Example:**

const element = await testdriver.find('error message');

if (element.screenshot) {
  // Save screenshot to file
  const fs = require('fs');
  const base64Data = element.screenshot.replace(/^data:image\/\w+;base64,/, '');
  fs.writeFileSync('element-screenshot.png', Buffer.from(base64Data, 'base64'));
  console.log('Screenshot saved');
}

<Warning> Screenshots can be large. They're automatically excluded from error messages to prevent memory issues. </Warning>

text

Text content extracted from the element by AI (if available).

element.text

**Returns:** `string | null` - Element's text content

**Example:**

const message = await t