/deepstream-dev

name: deepstream-dev
description: NVIDIA DeepStream SDK development with Python pyservicemaker API. Use when building video analytics pipelines, GStreamer-based video processing, TensorRT inference integration, object detection/tracking, or Kafka/message broker integration.
owner: NVIDIA CORPORATION
metadata:
  author: "NVIDIA CORPORATION <info@nvidia.com>"
service: deepstream
version: 1.1.1
reviewed: 2026-04-24
license: CC-BY-4.0 AND Apache-2.0

DeepStream Development Skill

This skill requires access to all of the reference documents listed in the `references/` directory below. Ensure they are available before executing the workflow.

When this skill is active, **ALWAYS read the relevant reference documents** before generating code. Do NOT rely on memory - the reference documents contain critical details about exact property names, correct API usage, and common pitfalls.

SDK and Architecture Quick Reference

DeepStream SDK Version Requirements

• **GStreamer**: 1.24.2
• **NVIDIA Driver**: 590+
• **CUDA**: 13.1
• **TensorRT**: 10.14.1.48
• **Platforms**: Ubuntu 24.04 (x86_64 and ARM64/Jetson)

Typical Pipeline Flow

Source → Stream Muxer → Inference → [Tracker] → OSD → Renderer

Components in `[brackets]` are **optional** -- only add them when the user explicitly requests them.

| Stage | Role | Key Element(s) | Required? | |-------|------|-----------------|-----------| | Source | Input from files, RTSP, cameras | `nvurisrcbin` (preferred), `nvmultiurisrcbin`, `filesrc` | Yes | | Stream Muxer | Batches streams for inference | `nvstreammux` | Yes | | Inference | TensorRT model execution | `nvinfer`, `nvinferserver` | Yes | | Tracker | Multi-object tracking across frames | `nvtracker` | **Only if requested** | | OSD | Draws bounding boxes, labels, overlays | `nvosdbin` | Yes (for visualization) | | Renderer | Display or save output | `nveglglessink`, `nv3dsink`, `filesink` | Yes |

Memory Model

DeepStream uses NVIDIA Video Memory Manager (NVMM) for zero-copy GPU buffer transfers. Caps strings use `memory:NVMM` to indicate GPU memory (e.g., `video/x-raw(memory:NVMM), format=NV12`).

Critical Rules

1. **Only Add Requested Components**: Do NOT add pipeline elements the user did not ask for.

• **Tracker (`nvtracker`)**: Only add when the user explicitly requests tracking or object IDs across frames
• **Secondary GIEs**: Only add when the user requests classification or attribute extraction
• **Analytics (`nvdsanalytics`)**: Only add when the user requests line crossing, ROI counting, etc.
• **Message broker (`nvmsgbroker`/`nvmsgconv`)**: Only add when the user requests Kafka/cloud messaging
• When in doubt, build the **minimal working pipeline** and let the user ask for additions

2. **Default to `nvurisrcbin` for Sources**: When the user says "camera", "stream", "video", or provides a file path:

• Always use `nvurisrcbin` -- it handles RTSP, HTTP, and local files (`file://`) transparently
• Only use `filesrc` + `qtdemux` + parser when the user explicitly needs raw file source control
• For RTSP/live sources, also set `live-source=1` on `nvstreammux` and `sync=0` on the sink
• Convert local paths to URI: `"file://" + os.path.abspath(path)`

3. **Metadata Iteration**: Use `.frame_items` and `.object_items` (returns iterators, NOT lists)

• NEVER use `len()` on these - iterate to count
• Iterator can only be consumed once

4. **Request Pad Syntax**: Use `"sink_%u"` template, NEVER literal pad names

   pipeline.link(("decoder", "mux"), ("", "sink_%u"))  # CORRECT
   # pipeline.link(("decoder", "mux"), ("", "sink_0"))  # WRONG - will fail

5. **Platform Detection for Sinks**:

   import platform
   sink_type = "nv3dsink" if platform.processor() == "aarch64" else "nveglglessink"

• For WSL2 Ubuntu 24 Docker, this default selection must be overridden.
• **WSL2 + Ubuntu 24 Docker**: If `/proc/version` contains `microsoft` or `wsl`

and `/etc/os-release` has `VERSION_ID="24.04"`, the generated app must never create a display branch or display sink (`nveglglessink`, `nv3dsink`, etc.), even if the prompt asks for display. Do not rely on a `--no-display` flag for this case. Generate encoded MP4 output only (`nvv4l2h264enc` -> `h264parse` -> `mp4mux`/`qtmux` -> `filesink`) and make the default run path write the annotated video file. In the generated `README.md`, explicitly explain that WSL2 Ubuntu 24 Docker is MP4-output-only because display sinks are disabled by a known issue. If the user explicitly requested display, add an inline code comment and README note explaining: `Display requested but disabled due to WSL2 Ubuntu 24 Docker limitation — MP4 output generated instead.`

• **Non-WSL targets**: Do not add WSL-specific behavior or WSL limitation text to

generated apps or READMEs. Use the normal platform display sink selection above.

6. **Buffer Cloning**: Always clone buffers for async processing

   tensor = buffer.extract(0).clone()  # CRITICAL

7. **Queue Types**:

• `queue.Queue` → Use with `threading.Thread`
• `multiprocessing.Queue` → Use with `multiprocessing.Process`
• Using wrong type causes silent data loss!

8. **nvinfer Config Format**:

• YAML: Use `property:` section (NOT `model:`), `key: value` with space after colon
• INI: Use `[property]` section, `key=value` with equals sign
• Section MUST be named `property`

9. **nvmsgbroker is a SINK**: Cannot have downstream elements - use `tee` to split pipeline

10. **ALL Sinks Need async=0 for Tee Splits or Dynamic Sources**: CRITICAL for state transitions

    # When using tee splits OR dynamic sources, ALL sinks MUST have async=0
    pipeline.add("nveglglessink", "sink", {
        "sync": 0, "qos": 0,
        "async": 0  # CRITICAL - prevents state transition deadlock
    })

**Symptom if missing**: Pipeline stays