/alibabacloud-maxframe-video-frame-pipeline

name: alibabacloud-maxframe-video-frame-pipeline
description: This skill should be used when the user asks to "build a frame extraction job" / "视频抽帧 / 抽关键帧", "label driving images with a VLM" / "图像打标 / image labeling with Qwen-VL", "compute image embeddings" / "图像向量化 / multi-modal embedding", "build a video_table / image_table / clip_dir_table for AI FUNC", "扫 OSS 建 video meta 表", or mentions driving-scene / ADAS / 智驾 / 智能驾驶 / 自动驾驶 / 路测 / 行车记录仪 / 座舱 video or image pipelines on MaxFrame + OSS + ODPS. Not for audio (use driving-audio-maxframe-job).

Driving Video MaxFrame Job

Overview

Generate customer-neutral MaxFrame job scaffolds for driving-video workflows. Frame extraction uses an ffmpeg-based `mf.apply_chunk` UDF (with `with_fs_mount` + `with_running_options`); image labeling and embedding use AI FUNC multi-modal models loaded via `read_odps_model`. Keep all stages in one lazy DAG by default and externalize all runtime settings.

When to Use

• Building a video manifest table by scanning OSS, when the customer has

videos sitting in OSS but no inventory table yet (Stage 0)

• Frame extraction from in-car / cabin / dashcam / road-test video tables
• Clip-to-keyframe labeling pipelines (`clip_dir_table` → keyframe rows → labels)
• Image labeling jobs, including direct-image requests that ask for labeling

plus embedding outputs

• Image / multi-modal embedding jobs derived from image workflows

Minimum Input

• `scenario_name`
• `input_shape`
• `targets`
• `output_table` or `output_tables`

Decision Flow

1. **Check for an upstream `video_table`** when the request starts from videos. If the customer does not already have an ODPS inventory table with a `video_path` column, prepend Stage 0 (manifest build): a plain PyODPS + `alibabacloud_oss_v2` script that lists the OSS prefix with `list_objects_v2_paginator` and writes a `video_path / size_bytes / last_modified` table. See [references/build_video_meta.md](references/build_video_meta.md). Skip Stage 0 only when the customer points at an existing inventory table or hands over a small in-Python path list for PoC. 2. Map the request into one of four abstract scenario types: video frame extraction, clip-to-keyframe labeling, image labeling, or embedding. 3. Treat `targets` as a separate output dimension inside the chosen scenario type. 4. Default to **single-job, single-output** for downstream stages even when the request starts from `video_table` and asks for downstream labeling/embedding. Chain `frame_extraction → image_labeling` in one lazy DAG. Only emit a split-video pair (`video_frame_extraction.py` + `image_labeling.py`) when the customer explicitly says they need an intermediate frame table for reuse, audit, or independent retry. (Stage 0 manifest build is always a separate script — it is not a MaxFrame stage.) 5. Ask for any missing minimum inputs; if ambiguity remains, use a generic, customer-neutral scaffold with explicit user-fill fields. 6. Generate code, schema guidance, and a short walkthrough.

Migration Rule

When modernizing existing online jobs, replace UDF + direct DashScope calls with AI FUNC. Do not generate MaxFrame UDF wrappers for labeling or embedding, do not call DashScope HTTP / OpenAI-compatible clients directly, and do not require `DASHSCOPE_API_KEY`.

Pipeline Selection

| input_shape | targets | pipeline | |---|---|---| | `oss_prefix` (no inventory table yet) | `manifest` | video-manifest (Stage 0; PyODPS + alibabacloud_oss_v2) | | `video_table` | `frame-extraction` only | video-frame-extraction | | `video_table` | `frame-extraction` + labeling/embedding | single-job: frames lazy → image-labeling lazy (default); split only on explicit ask | | `clip_dir_table` | `labeling` | clip-to-keyframe labeling | | `image_table` | `labeling` and/or `embedding` | image-labeling | | `labeled_image_table` | `embedding` | image-labeling |

For `image_table` with both `labeling` and `embedding`, route to a single image-labeling pipeline that emits both outputs in the same DAG.

If `video_table` is requested but the customer doesn't have one yet, run the `video-manifest` Stage 0 first; the manifest table then satisfies `video_table` for the downstream stages.

Input Contract

**Required:**

• `scenario_name`
• `input_shape`: `oss_prefix` | `video_table` | `clip_dir_table` | `image_table` | `labeled_image_table`
• `targets`: one or more of `manifest`, `frame-extraction`, `labeling`, `image-labeling`, `embedding`
• For `oss_prefix`: `oss_bucket` + `oss_region` + `oss_endpoint` + `video_input_table` (the meta output). `oss_endpoint` is required because Stage 0 embeds it into every `video_path` URI (`oss://<endpoint>/<bucket>/<key>`) — downstream frame_extraction's `with_fs_mount(OSS_ROOT, ...)` matches on the full prefix, so a bare `oss://<bucket>/<key>` URI would silently break the path substitution. Stage 0 uses OSS SDK v2 with a RAM user AK/SK (`OSS_ACCESS_KEY_ID` / `OSS_ACCESS_KEY_SECRET`); `role_arn` is only used inside the MaxFrame DAG, not Stage 0.
• For all other shapes: `output_table` (single-stage, default) or

`output_tables` (`{"frames": "...", "final": "..."}`) only when split-video is explicitly requested

**Optional:** `source_table`, `partition_col`, `partition_value`, `oss_root`, `oss_mount_path`, `oss_role_arn` (MaxFrame-side), `oss_prefix`, `video_extensions`, `meta_lifecycle`, `frame_fps`, `frame_sample_count`, `ffmpeg_timeout_sec`, `label_prompt_style`, `vlm_model`, `embedding_model`, `embedding_dim`, `model_project`, `gpu_quota`, `gpu_count`, `need_intermediate_table`, `need_oss_writeback`.

Reference Map

| Topic | Reference | |---|---| | Stage 0 — build video meta table from OSS prefix (PyODPS + alibabacloud_oss_v2) | [references/build_video_meta.md](references/build_video_meta.md) | | AI FUNC multi-modal call shapes (generate / embed / messages= / dimensions=) | [references/ai_func_calls.md](references/ai_func_calls.