/live-mastracode-instrumentation

name: live-mastracode-instrumentation
description: Reproduce and measure a Mastra Code runtime bug against a real model by driving the built TUI headlessly in tmux while every layer (network, provider stream, run engine, TUI) appends timestamped JSONL. Use when a bug depends on real provider timing, streaming order, or event flow that mocked tests and fixtures may not reproduce — e.g. wrong token rates, flicker, stuck status, ordering races, or "only happens sometimes with a real model".

Live Mastra Code instrumentation

Run the real TUI against a real model, have an agent type the prompts, and log timestamps at each layer. Comparing the layers step by step shows where timing or ordering goes wrong: at the provider, in the network wrapper, in stream conversion, in the run engine, or in the UI reducer.

Pair this with `debugging-difficult-bugs`: instrument, reproduce, read the log before fixing, and clean up afterwards. Afterwards, turn the finding into deterministic checked-in coverage (unit tests plus a TUI E2E scenario under `mastracode/tui/e2e/tui/`). A live run is evidence, not a regression test.

Cost and consent

The TUI uses the user's configured model and stored credentials (for example a Claude Max OAuth login), so every prompt spends their quota. Use short prompts. Tell the user which model you ran and roughly how many sessions.

1. Add temporary diagnostics

Append one JSON line per event to `<cwd>/debug-token-rate.jsonl` (or a similarly named file) with `appendFileSync`. Rules:

• **Metadata only**: timestamps, event types, ids, lengths, and token counts. Never log prompt, response, or tool content.
• Tag every record with a unique `build` marker (for example `'my-bug-diagnostics-v1'`). That lets you confirm the build you ran contains the instrumentation and find every leftover when cleaning up.
• Wrap each append in `try {} catch {}` so logging can never break the run.
• Take the timestamp before writing. A synchronous append adds a little latency to the path you are measuring, which is fine for millisecond-scale timing. If you are measuring anything finer, push records onto an in-memory array and write them out once when the run ends.
• Log at the earliest and latest point of each layer you care about, so the timestamps can be lined up per step.

Useful hook points, from outermost to innermost:

| Layer | Where | What to log | | --------------- | ------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Network | The provider's `fetch` wrapper, e.g. `buildAnthropicOAuthFetch` in `mastracode/sdk/src/providers/claude-max.ts` | request start; headers arrival (status, `content-encoding`); each body chunk via a pass-through `TransformStream` (bytes, SSE `event:` names). Re-wrap the body with `new Response(body, { status, statusText, headers })`. | | Provider stream | `convertFullStreamChunkToMastra` in `packages/core/src/stream/aisdk/v5/transform.ts` | raw chunk type and delta length as the AI SDK hands it over | | Run engine | `processStreamChunk` in `packages/core/src/agent-controller/session-run-engine.ts` | chunk type, `getChunkProducedAt(chunk)`, current message id, step-start `startedAt`, usage. Skip per-delta chunks here if the provider layer already logs them. | | TUI | `mastracode/tui/src/tui/event-dispatch.ts` (web mirror: `mastracode/factory-ui/src/ui/domains/chat/services/runtime.ts`) | event type, ids, relevant state before and after, and any computed value (e.g. `usage.computed` with tokens, window, instantaneous and displayed rate) |

For other providers, find the equivalent `fetch` option or wrapper in `mastracode/sdk/src/providers/`.

2. Build and verify the instrumentation shipped

pnpm install --frozen-lockfile   # fresh worktrees only
pnpm build:mastracode            # ~50s once deps are built
rg -l "my-bug-diagnostics-v1" mastracode/tui/dist packages/core/dist mastracode/sdk/dist

If `rg` finds nothing in a `dist`, the run will not produce that layer's records.

3. Drive the TUI in tmux

Start the TUI in a throwaway git repo so the agent's tool calls default to scratch files. Setting the working directory is not a sandbox: do not grant the session extra allowed paths, and keep prompts pointed at the scratch repo.

mkdir -p /tmp/mc-repro && git -C /tmp/mc-repro init -q
tmux new-session -d -s mcrepro -x 200 -y 50 -c /tmp/mc-repro \
  "node $PWD/mastracode/tui/dist/cli.js"
sleep 14   # startup

Send each prompt, then send `Enter` as a separate key press, then wait and read the screen:

tmux send-keys -t mcrepro "/new"; sleep 1; tmux send-keys -t mcrepro Enter; sleep 3
tmux send-keys -t mcrepro "Please do these one at a time as separate tool calls (not in parallel): write 3 files named a1.md through a3.md, each with a ~100 word paragraph about a different tree, writing a sentence of explanation before each tool call. Then run ls -la with execute_command and summarize in three sentences."
sleep 1; tmux send-ke