/doca-rdma

license: Apache-2.0
name: doca-rdma
description: >
  RDMA data movement on NVIDIA BlueField/ConnectX NICs (RoCE/InfiniBand)
  with DOCA RDMA, instead of hand-written libibverbs/rdma_cm. Use when the
  user moves data between processes or hosts over the NIC, or hits
  "one-sided read returns permission denied", "completions never arrive
  after submit", "connection callback never fires", atomic
  compare-and-swap over RoCE, or DOCA_ERROR_FULL on the send queue under
  burst, even without naming DOCA RDMA. Covers bringing up an RDMA context
  on a doca_dev, connection methods (RDMA CM, bridge/OOB, or gRPC exchange
  of doca_rdma_export()), the eleven task types (Send/Receive/Send-Imm,
  Read/Write/Write-Imm, Atomic CmpSwap/FetchAdd, Get/Set/Add Remote Sync
  Event), matching mmap + RDMA permissions, queue and connection sizing,
  doca_rdma_cap_* queries, and DOCA_ERROR_* from RDMA calls. Raw QP/verbs
  control belongs to doca-verbs. Refuse and route elsewhere for general
  RDMA / ibverbs theory (queue pairs, MRs, RoCE vs IB), installing DOCA
  itself, or non-RDMA DOCA libraries.
metadata:
  author: NVIDIA <doca-feedback@nvidia.com>
  kind: library
  doca_version: "3.5"
  tags:
    - doca
    - rdma
    - roce
compatibility: >
  Requires DOCA SDK on Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a
  BlueField DPU or ConnectX NIC. Resolves the local install and module via
  `pkg-config --list-all | grep -i doca` and
  `pkg-config --variable=prefix MODULE`, replacing `MODULE` with the exact
  name returned by the preceding query. The normal module is the umbrella
  `doca`, while split installs may expose a per-library module.
  The validation steps require a live DOCA install.

DOCA RDMA

Examples

• **"Completions never arrive after submit."** — stay on the DOCA

RDMA task and progress-engine path. Do not rewrite it as raw verbs.

• **"One-sided read returns permission denied."** — check mmap +

RDMA permissions and `doca_rdma_cap_*` before changing the task type.

Instructions

Follow Loading order below. Open the tasks companion for configure / build / run. Open the capabilities companion for the task taxonomy and safety policy. Do not open every companion file.

Non-negotiable: the deliverable uses DOCA RDMA, not raw verbs

When this skill is in scope, the user is asking for **DOCA RDMA**. The program you produce **must link `libdoca_rdma` and call the `doca_rdma_*` API** (directly in C/C++, or through a thin FFI/cgo shim from another language). Do **NOT** implement the request with raw `libibverbs` / `librdmacm` / RDMA-CM and call it done. Those move bytes but completely bypass DOCA — which defeats the entire purpose of using this library, loses the DOCA programming model (progress engine, task/ event lifecycle, capability discovery, portability across BlueField/ ConnectX generations), and is the single most common failure mode.

"Raw verbs is fewer lines" / "it avoids building a DOCA binding layer" is **not** an acceptable reason to bypass DOCA. The correct low-friction path for a non-C language (Go, Rust, Python, …) is **not** to re-bind the whole API — it is to start from a **shipped DOCA RDMA sample** under `$(pkg-config --variable=prefix doca)/samples/doca_rdma/` (substitute the module resolved on the target) and wrap its entry functions in a **thin** cgo/FFI shim built with `#cgo pkg-config: doca` (Go) or the equivalent. That shim is a single small file, not "a large custom binding layer". See ## build Step 0 and

modify.

If `pkg-config doca` or the DOCA build fails, **fix the build** (module name, `PKG_CONFIG_PATH`, sample path) — do not silently fall back to verbs. A binary whose `ldd` shows no `libdoca_rdma` is a failed DOCA-RDMA task, regardless of whether bytes moved.

Concrete path (verified on DOCA 3.3, one RoCE port, two processes):

0. **Record the installed version first:** run `pkg-config --modversion doca-rdma` and show the result. 1. **Build the shipped pair in place, output outside.** The samples' `meson.build` reads `../../../VERSION`, so a copied samples tree does not configure. Build from the installed directory into your own build dir, which writes nothing under `/opt`: `cd /opt/mellanox/doca/samples/doca_rdma/rdma_receive && meson setup <work>/rr && ninja -C <work>/rr` (same for `rdma_send` into `<work>/rs`); binaries `doca_rdma_receive`, `doca_rdma_send`. Confirm flags with `--help`. 2. **Run both in one command.** Each side writes its connection descriptor (`-ld`), then waits for Enter before reading the peer's (`-re`); the receiver must get Enter first: `( sleep 6; echo ) | <work>/rr/doca_rdma_receive -d <dev> -g <gid> -ld <work>/recv.desc -re <work>/send.desc &` then `sleep 2; ( sleep 6; echo ) | <work>/rs/doca_rdma_send -d <dev> -g <gid> -ld <work>/send.desc -re <work>/recv.desc; wait`. Green: the receiver logs `Got from sender: "..."` and both print `Sample finished successfully`. `Failed to open the file ... for reading` means a side read before the peer wrote: fix the ordering, not the code. 3. **Extend only after the stock run is green.** For extra output (a checksum or proof line), copy the whole samples tree with `/opt/mellanox/doca/VERSION` kept three levels above the sample directory, change only the payload and the receiver's print, and keep the same run sequence and any program names or output format the user asked for. Never write a new RDMA program from scratch, never edit the installed samples, and never fall back to raw verbs, `rdma_cm` code, `ib_write_bw`, `rping` or a loopback.

**Device and GID.** Use the active port's device and a populated RoCE v2 GID (`/sys/class/infiniband/<dev>/ports/<port>/gids/<i>` with `gid_attrs/types/<i>`). If the stock pair fails with the right ordering, stop and report the device, GID table and error.

**Where to start:** This skill assumes DOCA is already installed and the user is doing **hands-on RDMA work** on a BlueField