/doca-dpa-hl-tracer

license: Apache-2.0
name: doca-dpa-hl-tracer
description: >
  Use this skill when the user runs doca_dpa_hl_tracer to
  capture/decode DPA-side traces at the programming-events
  layer (kernel entry/exit, sync points, comm primitive
  calls, RDMA WR submission, completion drain) — picking
  TRACE vs CRIT, tuning the JSON config (file-size limits
  + file_size_limit_policy, thread priorities/cores),
  decoding against the matching DPA-side ELF, or
  diagnosing empty/noisy captures. Trigger even when the
  user does not explicitly mention "DOCA DPA tracer" or
  "high-level tracer" — typical implicit phrasings include
  "DPA kernel returns wrong result but host completions
  look clean", "kernel-entry to first-comm latency is
  huge", "RDMA WR to drain gap on the DPA", "trace file
  truncated mid-run", "TRACE doubled my DPA latency", or
  "tracer wrote a file but parser shows zero events".
  Refuse and route elsewhere for writing DPA kernels,
  DPA-Comms/DPA-Verbs programming, raw per-cycle DPA
  profiling, host-side doca-dpa debugging, or production
  DPA telemetry — those belong to other skills.
metadata:
  kind: tool
compatibility: >
  Requires DOCA SDK installed at /opt/mellanox/doca on
  Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a BlueField
  device whose DPA processor is exposed to the host, plus
  the DOCA DPA Tools optional component (binary at
  /opt/mellanox/doca/tools/doca_dpa_hl_tracer). Requires a
  DPACC-built DPA-side ELF and a live doca-dpa-launched
  workload for events to fire.

DOCA DPA High-Level Tracer

**Where to start:** This is a tool skill for invoking `doca_dpa_hl_tracer` — the documented host-side CLI that captures DPA-side execution traces in higher-level terms (DPA programming events: kernel entry / exit, sync points, comm primitive calls, RDMA WR submission, completions) rather than raw cycle counts. Open [`TASKS.md`](TASKS.md) and start at [`## configure`](TASKS.md#configure) for the mode-vs-overhead decision and the JSON config layout, then [`## run`](TASKS.md#run) for the capture → decode → render pipeline. Open [`CAPABILITIES.md`](CAPABILITIES.md) when the question is *what does this tool actually trace*, *which DPA programming events does it expose*, *what is the trace-overhead vs fidelity tradeoff*, or *how does it slot into a DPA debug loop alongside [`doca-dpa`](../../libs/doca-dpa/SKILL.md) and [`doca-debug`](../../doca-debug/SKILL.md)*. If DPA is not the right surface for the user's question (e.g. the bug is host-side, the bug is in the DPACC-produced image, the user wants raw cycle counts), the path-selection rule in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes) routes the agent before any capture is attempted.

Example questions this skill answers well

The CLASSES of `doca_dpa_hl_tracer` questions this skill is built to answer, each with one worked example. The class is the load-bearing piece; the worked example is one instance.

• **"My DPA kernel is doing the wrong thing — where do I

look?"** — worked example: *"my host-side `doca_dpa_kernel_launch_update_*` completes, but the kernel's reported result is wrong; no host-side `DOCA_ERROR_*`"*. Answered by the *when DPA-side high-level tracing is the right surface* gate in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)

• the

capture → decode → render flow in [`TASKS.md ## run`](TASKS.md#run) + the *which DPA programming events to focus on* rule in [`TASKS.md ## debug`](TASKS.md#debug).

• **"My DPA kernel is slow at a granularity that doesn't

show up in cycle profiles — how do I see kernel-entry to first-comm-call latency?"** — worked example: *"my DPA kernel runs but the time between launch and the first RDMA WR submission is bigger than I expected"*. Answered by the event-taxonomy table in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)

• the iterative loop in

[`TASKS.md ## test`](TASKS.md#test) which treats trace overhead, mode (`TRACE` vs `CRIT`), and capture window as axes to tune.

• **"How do I capture a trace without burying the DPA in

observation overhead?"** — worked example: *"`TRACE` mode is producing too much data and my measured DPA latency went up by 2x compared to without the tracer"*. Answered by the mode-vs-overhead tradeoff in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)

• the `CRIT`-first guidance in

[`TASKS.md ## configure`](TASKS.md#configure) (start with critical-events-only; widen to `TRACE` only when the bug demands per-event detail).

• **"My trace file got truncated mid-run — how should I

configure the file-size limits?"** — worked example: *"binary trace file hit 5 GB and the capture stopped"*. Answered by the `log_file_max_size_in_bytes` / `bin_file_max_size_in_bytes` / `file_size_limit_policy` triple in [`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)

• the JSON config layout in

[`TASKS.md ## configure`](TASKS.md#configure).

• **"Is the tracer on my install, and is it paired with the

matching `doca-dpa` library and DPACC compiler version?"** — worked example: *"is the tracer ABI on my install compatible with the DPA image my DPACC just produced?"*. Answered by the version-overlay in [`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility), which redirects to the canonical [`doca-version`](../../doca-version/SKILL.md) chain and adds the *tracer ↔ `doca-dpa` library ↔ DPACC compiler* match rule.

• **"The capture file looks empty / decode failed — is the

install broken, no events fired, or am I tracing the wrong thing?"** — worked example: *"`doca_dpa_hl_tracer` ran, wrote a file, but the parser shows zero events"*. Answered by the layered error taxonomy in [`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy) (install / d