/gemini-live-api

name: gemini-live-api
metadata:
  category: AiAndMachineLearning
description: >-
  Generates a Gemini LiveAPI client service class in the user's chosen programming
  language. Use when the user wants to build, scaffold, or integrate a client
  that connects to the Gemini Enterprise LiveAPI websocket endpoint, handles
  session setup/resumption, bearer token refresh, and sending/receiving
  `ClientMessage`/`ServerMessage` protos. Don't use for general (non-live,
  non-bidirectional) Gemini API usage such as one-shot `generateContent`,
  embeddings, image/video generation, or fine-tuning — use the `gemini-api`
  skill for those.

LiveAPI Service Skill

This skill provides instructions for generating a **LiveAPI client service class** that connects to the Gemini Enterprise Live API over WebSockets. The generated client handles bidirectional streaming, bearer-token authentication via Application Default Credentials (ADC), transparent session resumption, and `ClientMessage` / `ServerMessage` proto exchange.

The skill also produces a demo frontend + backend service so the user can interactively validate the generated client (text, audio, video, transcription, and interrupt handling).

Prerequisites

Before running the generation flow, ensure the following are available on the host:

• A Google Cloud project with the Vertex AI / Gemini Enterprise Agent Platform

APIs enabled.

• Application Default Credentials configured on the host running the generated

client:

    gcloud auth application-default login

• A destination output folder supplied by the user (e.g. `/tmp/liveapi_out`)

where the generated code, environment, and demo will be written. **Never** mutate the host's system Python environment.

• The user's chosen implementation language (Python is the default and

reference language for this skill).

Reference Files

Provided files in `references/` (do **not** treat these as standalone skills — they are loaded on demand):

• `client_server_messages.md`: Public reference for the `ClientMessage` /

`ServerMessage` schemas used by the Live API.

• `client_server_messages.proto`: The proto definition generated from

`client_server_messages.md`.

• `session_manager.md`: Describes how to correctly handle sessions, buffering,

and resumption on disconnection.

Steps

Step 1: Copy the reference files

Copy `client_server_messages.md`, `client_server_messages.proto`, and `session_manager.md` from this skill's `references/` folder into the user's destination output folder. These files become the source of truth for the generated client.

Step 2: Reconcile with the public documentation

Examine the public documents linked from `client_server_messages.md`. If there are any discrepancies between the public documents and the copied `client_server_messages.md` / `client_server_messages.proto`, update the copies in the destination folder so the generated client compiles and runs against the current server contract.

Step 3: Implement the client class

Implement a class in the user's chosen language that:

• Imports the local `client_server_messages.proto` types (`ClientMessage`,

`ServerMessage`).

• Opens a WebSocket connection to the Live API endpoint.
• Exposes async methods so the user can send and receive data to/from the

model.

For languages that require an isolated runtime (e.g. Python), create an isolated environment (e.g. `venv`) **inside the destination folder** and generate a bash script (e.g. `setup.sh`) that recreates the environment and installs dependencies. **Never** install into the system interpreter or the user's global site-packages, and never instruct the user to run `sudo pip install`.

Initialization parameters

The user provides the following at construction time:

• `project_id`
• `location`
• `model_id`
• `config`: a `ClientMessage` with the `setup` field populated.

Authentication

Obtain a bearer token via Application Default Credentials, attach it to the WebSocket connect request as `Authorization: Bearer <token>`, refresh the token before or upon expiry, and reuse the refreshed token on every reconnection (including `go_away` and unexpected disconnects). **Do not** hard-code a long-lived API key as the only auth mechanism.

Public async API

The class MUST expose the following async methods, gated on receipt of a `setup_complete` `ServerMessage` before sending:

• `send_realtime_data(data)`: send realtime input. `data` is a `ClientMessage`

carrying a `realtime_input` field.

• `send_client_content(data)`: send non-realtime, turn-based content that

contributes to history. `data` is a `ClientMessage` carrying a `client_content` field.

• `receive()`: yield `ServerMessage` instances parsed from the WebSocket

stream.

Do not expose synchronous blocking variants as the primary API surface.

Step 4: Write a test file

Once the client is implemented, generate a test file that initializes the connection and exercises sending `text`, `audio`, and `video` data and receiving the responses. Ask the user for any information required to run the test (project, model, media samples).

Step 5: Generate `how_to_run.md`

Provide a `how_to_run.md` in the destination folder that documents the generated class. Include full examples showing how to build `ClientMessage` payloads for every supported modality, how to send them, and how to receive data from the model.

Step 6: Generate a demo frontend + backend service

Create scripts that deploy the implementation as a service with both a frontend UI and a backend service (any language). The service MUST reuse the `ClientMessage` / `ServerMessage` protos from Step 1 for wire traffic. Through the UI the user should be able to:

• Start a new connection / close the current connection.
• Select the model to use.
• Select input sources (audio and/or video from camera or screenshot) and

str