/visual-intelligence

name: visual-intelligence
description: Integrate your app with iOS Visual Intelligence for camera-based search and object recognition. Use when adding visual search capabilities.
allowed-tools: [Read, Write, Edit, Glob, Grep, Bash, AskUserQuestion]
last_verified: 2026-07-16
review_by: 2027-06-22

Visual Intelligence

Integrate your app with iOS Visual Intelligence to let users find app content by pointing their camera at objects.

When This Skill Activates

• User wants camera-based search in their app
• User asks about visual search integration
• User wants to surface app content in system searches
• User needs to handle visual intelligence queries

Overview

Visual Intelligence lets users: 1. Point camera at objects or use screenshots 2. System identifies what they're looking at 3. Your app provides matching content 4. Results appear in system UI

Your app implements:

• `IntentValueQuery` to receive search requests
• `AppEntity` types for searchable content
• Display representations for results

Platform Availability (WWDC26 297)

• Visual Intelligence runs on iOS, iPadOS, and macOS — the same entities, query, and OpenIntent code works unchanged on all three. Handle both **camera captures of physical objects** (iOS) and **screenshots of digital media** (iPad/Mac) as input.
• On Mac, the input pixel buffer can be **much larger** than what you'd encounter on iPhone — consider whether resizing is necessary before matching.

Quick Start

1. Import Frameworks

import VisualIntelligence
import AppIntents

2. Create App Entity

struct ProductEntity: AppEntity {
    var id: String
    var name: String
    var price: String
    var imageName: String

    static var typeDisplayRepresentation: TypeDisplayRepresentation {
        TypeDisplayRepresentation(
            name: LocalizedStringResource("Product"),
            numericFormat: "\(placeholder: .int) products"
        )
    }

    var displayRepresentation: DisplayRepresentation {
        DisplayRepresentation(
            title: "\(name)",
            subtitle: "\(price)",
            image: .init(named: imageName)
        )
    }

    // Deep link URL
    var appLinkURL: URL? {
        URL(string: "myapp://product/\(id)")
    }
}

3. Create Intent Value Query

struct ProductIntentValueQuery: IntentValueQuery {
    func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
        // Search using labels
        if !input.labels.isEmpty {
            return await searchProducts(matching: input.labels)
        }

        // Search using image
        if let pixelBuffer = input.pixelBuffer {
            return await searchProducts(from: pixelBuffer)
        }

        return []
    }

    private func searchProducts(matching labels: [String]) async -> [ProductEntity] {
        // Search your database using provided labels
        // Return matching products
    }

    private func searchProducts(from pixelBuffer: CVReadOnlyPixelBuffer) async -> [ProductEntity] {
        // Use image recognition on the pixel buffer
        // Return matching products
    }
}

SemanticContentDescriptor

The system provides this object with information about what the user is looking at.

Properties

| Property | Type | Description | |----------|------|-------------| | `labels` | `[String]` | Classification labels from Visual Intelligence | | `pixelBuffer` | `CVReadOnlyPixelBuffer?` | Raw image data |

Usage Patterns

**Label-based Search:**

func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
    // Labels like "shoe", "sneaker", "Nike" etc.
    let labels = input.labels

    // Search your content using these labels
    return products.filter { product in
        labels.contains { label in
            product.tags.contains(label.lowercased())
        }
    }
}

**Image-based Search:**

func values(for input: SemanticContentDescriptor) async throws -> [ProductEntity] {
    guard let pixelBuffer = input.pixelBuffer else {
        return []
    }

    // Convert to CGImage for processing
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer)
    let context = CIContext()

    guard let cgImage = context.createCGImage(ciImage, from: ciImage.extent) else {
        return []
    }

    // Use your ML model or image matching logic
    return await imageSearch.findMatches(for: cgImage)
}

Multiple Result Types

Use `@UnionValue` when your app has different content types.

Rules (WWDC26 297):

• **An app can have only ONE `IntentValueQuery` that accepts a `SemanticContentDescriptor`.** All result types must flow through that single query — a `@UnionValue` enum with one case per entity type.
• **Every entity type in the union needs its own `OpenIntent`** — without one, results of that type can't appear in image search.
• Think beyond pixel matching: an album matched by image similarity can also surface the artist's **nearby concerts** — be creative about the type of content you return based on the context.

@UnionValue
enum SearchResult {
    case product(ProductEntity)
    case category(CategoryEntity)
    case store(StoreEntity)
}

struct VisualSearchQuery: IntentValueQuery {
    func values(for input: SemanticContentDescriptor) async throws -> [SearchResult] {
        var results: [SearchResult] = []

        // Search products
        let products = await productSearch(input.labels)
        results.append(contentsOf: products.map { .product($0) })

        // Search categories
        let categories = await categorySearch(input.labels)
        results.append(contentsOf: categories.map { .category($0) })

        return results
    }
}

Display Representations

Create compelling visual representations for search results.

Result Card Real Estate (WWDC26 297)

• The search-result card gives about **three lines of text** for a title and subtitle, plus a thumbnail image — put the most im