/deepstream-sop

name: "deepstream-sop"
description: >
  Use this skill when building, deploying, evaluating, debugging, or measuring
  latency for the DeepStream SOP Inference Microservice — a GPU-accelerated
  FastAPI service that detects whether operators perform assembly-line steps in
  order via event boundary detection (GEBD) plus VLM classification. Trigger
  even if the user does not name it: verify operator step sequence, detect
  missing or out-of-order SOP steps, score factory/work-cell video for
  procedure compliance, run VLM-based SOP checking on industrial cameras, or
  call /v1/chat/completions with a file, RTSP, or Basler camera. Also trigger
  for its internals: SOPVideoProcessor, DeepStream GEBD model (e.g. DDM) via
  Triton CAPI, nvds_custom_postprocess, Cosmos Reason 1/2 vLLM, SSE streaming,
  Kafka NvProto/JSON output, Basler/Pylon camera + emulation, Docker compose,
  chunk-level latency. Do NOT trigger for generic DeepStream pipelines, object
  detection/tracking, NIM imports, or video summarization.
owner: "windy@nvidia.com"
service: "deepstream-sop"
version: "1.0.0"
license: "CC-BY-4.0 AND Apache-2.0"
reviewed: "2026-04-08"
metadata:
  author: "Wind Yuan <windy@nvidia.com>"
  tags:
    - deepstream
    - sop
    - vlm
    - triton
    - gpu
  languages:
    - python
  frameworks:
    - deepstream
    - triton
    - fastapi
  domain: video-analytics

DeepStream SOP Inference Microservice Skill

This skill guides AI coding assistants in building, extending, and debugging the **NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice** — a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance monitoring on industrial video feeds.

**Reference repository**: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp **Local reference code**: `sop-inference-bp/` directory (from a local clone of the repository)

---

Models

Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).

| Stage | Role | Model class | Default | Swap via | |------|------|-------------|---------|----------| | Stage 1 (CV) | Per-frame boundary scoring → chunk segmentation | **Generic Event Boundary Detection (GEBD)** | **DDM** ([MCG-NJU/DDM](https://github.com/MCG-NJU/DDM)) via Triton Python backend | Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5) | | Stage 3 (VLM) | Per-chunk action classification | Vision-language model via vLLM | **Cosmos Reason 1 7B** (Reason 2 also supported) | Set `VLLM_MODEL_PATH` to a different HF ID or local path |

"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).

**Chunking is selectable per request** (§ 2): default `ddm-net` uses GEBD; `uniform` produces fixed-length chunks and **bypasses Stage-1 GEBD** (§ 3, § 6). DDM temporal window is configurable via `FRAMES_PER_SIDE` / `SEQUENCE_BATCH` (§ 4, § 5), with optional **TensorRT** (§ 5).

---

Architecture Overview

Runs in a Docker container (`nvds-action-sop`) alongside a Kafka container. Full diagram: [`references/sop_architecture.svg`](references/sop_architecture.svg).

**Data flow through the 4-stage `SOPVideoProcessor` pipeline (per-request):**

Input Sources                    Docker Container: nvds-action-sop
─────────────                    ──────────────────────────────────────────────────
Video Files ──┐                  FastAPI Server (port 8300)
RTSP Streams ─┤── base64/       ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘   file/rtsp/       │
                  camera           │ ModelInitializer: VLM first, then DDM dummy pipeline
                                   │ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
                                   │
                                   ▼ SOPVideoProcessor (per-request)
                                   ┌────────────────────────────────────────────────┐
                                   │ Stage 1: DeepStream Pipeline (GPU)             │
                                   │   Source → nvstreammux → tee1                  │
                                   │    ├─[inference] queue1 → nvdspreprocess       │
                                   │    │  → nvinferserver (Triton CAPI + DDM)      │
                                   │    │  → InferOutputTensorParser → score_queue  │
                                   │    ├─[frames]  queue3 → nvvideoconvert         │
                                   │    │  → capsfilter → appsink                   │
                                   │    │  → DecodedFrameRetriever → frame_queue    │
                                   │    └─[RTSP out] queue → convert → H.264 enc    │  (optional, § 18)
                                   │       → rtppay → udpsink → RTSPServer (§ 18)   │  opt-in only
                                   │              │ boundary scores                 │
                                   │              ▼                                 │
                                   │ Stage 2: Clip Post-Process                     │
                                   │   Boundary detection → chunk segmentation      │
                                   │              │ video frames + timestamps        │
                                   │              ▼                                 │
                                   │ Stage 3: VLM Inference                         │
                                   │   Embedded vLLM (Cosmos Reason 1/2)            │
                                   │   Frame sampling at VLM_FPS → classification   │
                                   │              │ action labels                    │
                                   │              ▼                                 │
                                   │ Stage 4: SOP Checker                           │
                                   │   Sequence validation → missing/misordered     │
                                   │