/doca-gpunetio-ib-write-lat

license: Apache-2.0
name: doca-gpunetio-ib-write-lat
description: >
  Use this skill when the user is measuring GPU-kernel-initiated RDMA
  WRITE latency through doca-gpunetio — building and running the
  `gpunetio_ib_write_lat` client + server pair under
  `doca/tools/gpunetio_ib_write_lat/`, checking GPU-NIC pairing,
  reading the half-iter / full-iter / CUDA-side usec columns,
  characterizing median / p99 / jitter for a real-time control loop,
  picking GPUNetIO vs GPI vs CPU-initiated `perftest`, or weighing the
  latency-vs-batching trade-off. Trigger even without 'GPUNetIO' or
  'ib_write_lat': 'GPU kernel RDMA latency benchmark', 'how fast can a
  CUDA kernel post a WRITE', 'p99 RDMA latency on H100 + ConnectX',
  'kernel-launched WR tail latency', or 'compare GPU-init vs CPU-init
  perftest'. Route elsewhere for bandwidth runs
  (doca-gpunetio-ib-write-bw), the GPI surface (doca-gpi), library
  debugging (doca-gpunetio), or DOCA install.
metadata:
  kind: tool
compatibility: >
  Requires DOCA SDK installed at /opt/mellanox/doca on Linux
  (Ubuntu 22.04/24.04 or RHEL/SLES) with an InfiniBand-capable
  ConnectX or BlueField RNIC. Requires NVIDIA GPU with CUDA
  Toolkit and `nvidia_peermem` loaded; client and server hosts
  each need a GPU-NIC pair on a common PCIe / NVLink fabric.
  Reads pkg-config doca-gpunetio / doca-rdma / doca-common and
  builds from the source tree at
  /opt/mellanox/doca/tools/gpunetio_ib_write_lat against the installed DOCA.

DOCA GPUNetIO ib_write_lat

**Where to start:** This is a tool skill for the GPUNetIO- flavored `ib_write_lat` benchmark shipped under `doca/tools/gpunetio_ib_write_lat/` (a client + server pair, built from source against the installed DOCA via `meson`). It measures the latency of an RDMA WRITE work request when the WR is posted **from a CUDA kernel through the doca-gpunetio device-side surface**, in a ping-pong cadence. Open [`TASKS.md`](TASKS.md) and start at [`## configure`](TASKS.md#configure) for the GPU-NIC pairing precondition and the build pattern; jump to [`## run`](TASKS.md#run) for the single-iteration smoke flow. Open [`CAPABILITIES.md`](CAPABILITIES.md) when the question is *what this tool actually measures*, *how it differs from the GPI sister tool on the same physical operation*, or *how to interpret the half-iter / full-iter / CUDA-side usec output and the median / p99 / jitter characterization*. If DOCA is not installed yet, route to [`doca-setup`](../../doca-setup/SKILL.md) first; if the user is still deciding between GPUNetIO and GPI as a programming surface, the picture in [`../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpunetio/CAPABILITIES.md#capabilities-and-modes) and [`../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes`](../../libs/doca-gpi/CAPABILITIES.md#capabilities-and-modes) is the first stop.

Example questions this skill answers well

The CLASSES of `doca-gpunetio-ib-write-lat` questions this skill is built to answer, each with one worked example. The class is the load-bearing piece; the worked example is one instance.

• **"What GPU-init RDMA-WRITE latency / jitter can the

GPUNetIO path deliver for a real-time / control-loop workload?"** — worked example: *"measure per-iteration WRITE latency between two hosts with an H100 + ConnectX-7 on each side, target the median and the p99 separately"*. Answered by the GPU-NIC pairing precondition in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)

• the bring-up flow in

[`TASKS.md ## configure`](TASKS.md#configure) + [`TASKS.md ## run`](TASKS.md#run).

• **"This is the GPUNetIO tool — how does the latency

number differ from the GPI programming surface?"** — worked example: *"the team is using GPI; should I expect GPUNetIO to beat / tie / lose vs GPI?"*. Answered by the *"same physical operation, different runtime framework"* rule in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)

• the cross-link to the GPI library skill

[`../../libs/doca-gpi/CAPABILITIES.md`](../../libs/doca-gpi/CAPABILITIES.md) (note: `doca/tools/` ships no GPI `ib_write_lat` benchmark binary — GPI is a programming surface, not a shipped benchmark tool).

• **"Median vs p99 vs jitter — which one is the actual

answer for a real-time control loop?"** — worked example: *"my control loop has a deadline; the median is well under the budget but p99 spikes; do I quote the median or the p99?"*. Answered by the median-vs-p99-vs-jitter rule in [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability)

• the eval-loop overlay in

[`TASKS.md ## test`](TASKS.md#test).

• **"What is the latency-vs-batching trade-off specific

to GPU-init RDMA?"** — worked example: *"my CUDA kernel could batch multiple WRs to amortize the GPU-side overhead; what does that buy me on latency vs what does it cost me?"*. Answered by the latency-vs-batching trade-off in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes).

• **"What version of DOCA + CUDA Toolkit do I need for

this binary to build and run?"** — worked example: *"my install has DOCA at one semver and CUDA at another; will the ToT-shipped `gpunetio_ib_write_lat` even link?"*. Answered by the version overlay in [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility).

• **"How do I read the half-iter / full-iter / CUDA-side

usec columns?"** — worked example: *"the binary printed half-iter, full-iter, and a CUDA-side number — what is the right column to quote for one-way latency vs round-trip vs cross-check?"*. Answered by the column- semantics rule in [`CAPABILITIES.md ## Observability`](CAPABILITIES.md#observability).

Audience

This skill serves **external developers and performance engineers who need a reproducible measurement of the latency of an RDMA WRITE WR when the WR is