/run-load-test

name: run-load-test
description: Run the Omnigent load test and produce a results file explaining the latencies. Load when the user wants to load-test / stress-test / benchmark Omnigent under concurrency ("load test omnigent", "stress test the server", "how many hosts/sessions/turns can it handle", "load test real agent turns / conversations", "run a load test"). The test makes each simulated user a real omnigent host that creates host-bound sessions and drives real multi-turn conversations with a mocked LLM; it boots its own local stack (dev/loadtest/run.py). Gather inputs, run it, then read the generated summary.md and explain the latency distribution (avg/median/p95/p99, throughput, failures). NOT for single-request latency micro-benchmarks (that is dev/benchmarks/).

Run the Omnigent load test

Drives `dev/loadtest/` end to end: collect inputs → run → read `summary.md` → explain the latencies. **Each Locust user is a real `omnigent host`** that registers over the host tunnel, creates host-bound sessions, and drives **real multi-turn conversations** — every turn is a genuine post→idle loop through the host's runner, with the **LLM mocked** (zero latency) so the numbers are Omnigent's own overhead. `-u N` scales the number of hosts.

It **boots its own local stack** (server + mock LLM), so there is no server to point at, and it runs **from a repo checkout** only. For single-request latency micro-benchmarks (not concurrency), that is a different tool: `dev/benchmarks/`.

1. Ensure deps (repo checkout)

pip install -e '.[loadtest,dev,agents-sdk]'   # or: uv sync --extra loadtest --extra dev --extra agents-sdk

Run with that same interpreter (e.g. `.venv/bin/python`), from the repo root.

2. Gather inputs

Ask the user (AskUserQuestion when several are unknown); all have defaults.

| Input | Flag | Default | Notes | |---|---|---|---| | Hosts | `--users` | 4 | Concurrent hosts (N) — the main scale knob. | | Spawn rate | `--spawn-rate` | 1 | Hosts started per second. | | Run time | `--run-time` | 120s | `40s` / `5m` / `1h`. | | Sessions/host | `--sessions-per-user` | 2 | Host-bound sessions each host drives. | | Turns/session | `--turns-per-session` | 4 | Turns per session — history grows across them. | | Reply length | `--reply-words` | 60 | Words in the mocked (streamed) reply per turn. |

**Capacity caveat — say this to the user if they ask for large N:** turns run on real host + runner subprocesses, so N hosts × M sessions = N×M runner processes on *this* box. It is capacity-limited by design (real turns, not faked). Start at `--users 2 --sessions-per-user 1 --turns-per-session 2 --run-time 40s` to confirm the stack boots (~10-30s), then ramp to a few dozen hosts at most. At high N the load box saturates before the server (Locust warns about CPU).

3. Run

python dev/loadtest/run.py \
    --users <N> --spawn-rate <R> --run-time <T> \
    --sessions-per-user <S> --turns-per-session <TU>

It boots the stack, prints the server URL + registered agent, runs Locust, and writes `dev/loadtest/results/omnigent_load_test-<timestamp>/`.

4. Read and explain

`Read` the `summary.md` and relay it. Focus on:

• **Outcome / failures** first. Exit 0 + 0 failures = PASS. Non-zero failures are

the headline — check `console.log` and, for a host that failed to register, the per-host `results/.../host-workspaces/<name>/host.log`. At high N, failures usually mean the *load box* saturated, not the server.

• **turn** — the headline latency: one full post→idle agent turn on a host's

runner (mocked LLM), so it is Omnigent's per-turn overhead. It **grows across a conversation** as history accumulates, so a rising p95/p99 with larger `--turns-per-session` is expected and is the interesting signal.

• **host online** — host tunnel registration cost; **session create** — the

host-bound create; **Ops/s** — aggregate throughput at this concurrency.

If failures appeared or the tail looks high, suggest a concrete next step (lower N if the load box is saturated, raise `--turns-per-session` to study history growth, lengthen `--run-time` for steady state, or check server logs/metrics).

Notes

• Scenario file: `dev/loadtest/omnigent_load_test.py`; driver + report:

`dev/loadtest/run.py`. Full reference: `dev/loadtest/README.md`.