/rudder-ai-events-verify

name: rudder-ai-events-verify
description: "Verifies RudderStack AI events locally: captures what an instrumented app or MCP server sends, validates it against the AI events v0.3 schema, and checks joins, counts, identity and content. Use when testing AI events, or when AI or MCP events look wrong or missing."
allowed-tools: "Bash(python3 *), Bash(python *), Bash(npm *), Bash(npx *), Bash(node *), Bash(uv *), Bash(env *), Bash(curl *), Bash(tar *), Bash(lsof *), Bash(kill *), Bash(go build *), Bash(go run *), Bash(date *), Read, Write, Edit, Glob, Grep"

RudderStack AI Events Verification

Proves the instrumentation sends what `ai-events.yaml` promises, before any real traffic. Runs offline: a stub or mock RudderStack client, a mock model, no provider keys. Scripts and the checklist: `references/verification.md`.

Workflow

1. **Read the plan.** `ai-events.yaml` gives the surfaces, identity sources, content capture setting, legacy events and event rules to check. No plan: check against the spec only, and say so in the report. 2. **Scratch directory.** Create one outside the repo (for example `$TMPDIR/ai-events-verify/`). Captures, schemas, scripts and the virtualenv go there, never into the user's source tree. 3. **Get the schemas.** `ai-events-v0.3.schema.json` from the `@rudderstack/ai` npm release that ships spec v0.3, or the Go module's `spec/` directory (it is not in the Python wheel), and the client-event schema from `rudder-ai-events-instrument`. 4. **Capture** what each planned surface sends (skip surfaces the app does not have):

• **A. Stub client**, when the app has a test runner that can call its routes or job functions: `configure()` with an in-memory client (Go: an `analytics.Client` stub passed to `Configure` / the MCP wrapper), then drive the surfaces.
• **B. Mock data plane and mock model**, when it has no such tests, or is a script or cron job: run the app unchanged with its RudderStack data plane URL and provider base URL pointed at a local mock server, then send sample requests or run the job.
• **C. Live Events**, after A or B passes: a development source in RudderStack, inspected in Live Events or through `rudder-mcp-workflow`.
• **D. MCP servers**: the stub or the mock data plane, driven by an MCP client script (list tools, valid and invalid calls, `_meta` with an `interaction_id`).
• Browser events: a recorder in place of `rudderanalytics`, appended to the same capture.

5. **Run the app's own tests** too: instrumentation must not break them. 6. **Validate** with `check_events.py` and the flags that match the plan: schema, `specVersion: 0.3.0` and a known SDK (`context.wrapperLibrary.name` `@rudderstack/ai`, `rudderstack-ai` or `ai-sdk-go`) per event, one server `ai_response_received` per `interaction_id` (and at most one client copy), client copies of the prompt, answer and feedback unless the plan opts out, joins on `interaction_id` and `conversation_id`, feature, surface, user ID and session, no content unless the plan turns capture on, system runs without a user ID, MCP events without argument values and with planned server names and intents, legacy events still sent. Compare the printed background runs and MCP calls with the plan. 7. **Fix and repeat.** A failure in app wiring goes back to `rudder-ai-events-instrument`; a wrong plan assumption goes back to `rudder-ai-events-setup` (plan back to `draft`). A gap that matches a known limit (setup skill, "Known limits") is reported, not fixed. 8. **Record the time.** On the first PASS, set `metadata.time_to_integrate.verified_at` in `ai-events.yaml` (`date -u +%Y-%m-%dT%H:%MZ`), and ask the user for `hours_spent` and `notes`; leave them `null` if they do not say. 9. **Report.**

Report

Keep it short and factual:

AI events verification: PASS | FAIL
Capture: A stub client | B mock data plane | C Live Events
App tests: <command> -> <result>
Events: ai_user_prompt_created 1, ai_response_received 2, ai_user_action 1, chatTurnCompleted 1
Checks: schema PASS, one answer per turn PASS, joins PASS, identity PASS, content PASS, background n/a, mcp PASS, legacy PASS   # n/a for surfaces the app lacks
Known limits hit: <e.g. Flask route: no client_disconnected>
Time to integrate: started <started_at>, verified <verified_at>, hours <hours_spent or "not recorded">
Not verified: <what could not run, e.g. streaming route, real data plane delivery, warehouse load>
Next: <fixes, or: point the app at a dev source and watch Live Events>

Common failures

| Finding | Usual cause | |---|---| | Several `ai_response_received` for one turn | model calls outside `aiInteraction` / `ai_interaction` / `RunInteraction` (Go: a call made with a context that is not the interaction's), a stream not returned or bound, a Python thread pool without `propagate`, a LangChain handler on the model constructor | | `ai_user_action` without a matching answer | the client sent a different ID than the server used; check the `x-interaction-id` hand-off | | `prompt_text` / `response_text` present | `captureContent` on, globally or per interaction | | `userId` on a background run | `aiTask` / `ai_task` not used, or a plain interaction opened inside the job | | No events from a script or job | missing `flush()` before exit (Go: `client.Close()`); a `false` from Node.js/Python `flush()` means a client did not finish within 10 s | | No `total_cost_usd` | expected unless the plan registers prices or the provider reports cost | | Legacy event missing | flag defaulting to off, or the old call was removed | | Server answer with no client copy | the answer copy is not sent on every final state (error, stop), or a sampling wrapper or ad blocker dropped it; in production it can also mean the user left: the server answer then has `client_disconnected: true` (not detectable on Django or Flask) | | `specVersion` is `0.2.0` | SDK 0.4.0 installed instead of the release that ships spec v0.3 | | `trace_id` missing | expected without