/hf-cloud-serving-image-selection

name: hf-cloud-serving-image-selection
description: 'Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says "deploy this LLM", "host this HuggingFace model", "serve this fine-tuned model", "deploy this embedding model", "host a reranker", "serve a sentence-transformers model", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings/cross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.'

Serving Image Selection

The serving container is the single thing most likely to break a SageMaker deployment that "looked correct on paper". Wrong container, stale tag, or the wrong AMI — all produce the same opaque `Failed to pass health check` error.

Rule zero: HuggingFace images always win

When both a HuggingFace-curated family (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-sglang`, `tei`, `huggingface-pytorch-inference`) and a generic family (`vllm`, `vllm-omni`, `sglang`, `djl-inference`) can serve the model, **the HuggingFace one is mandatory, not preferred**. The only valid reasons to use a generic image:

1. **Verified incompatibility** — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed). 2. **No HuggingFace tag exists in the target region** and mirroring is not an option. 3. **The HuggingFace image is in "Known-broken images"** below.

A **newer version number on the generic repo is not a reason**. The AWS `vllm` repo often publishes a higher vLLM version than `huggingface-vllm`; an older-but-compatible `huggingface-vllm` tag still wins. "Latest vLLM" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied.

Where image URIs come from

**Primary source: AWS's official Deep Learning Containers catalog.**

URL: https://aws.github.io/deep-learning-containers/reference/available_images/

This page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2/ECS/EKS). When picking a URI for a deployment, **read it from this page directly** — copy the example URL, substitute `<region>` with the user's region, and pass it to `deploy.py --image-uri`.

The example URLs use `763104351884` as the account ID for most regions. A few regions use different accounts (e.g. `eu-south-1` uses `692866216735`). Check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) when in doubt.

**Exception: none currently.** Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via `mirror_image.py` and pass the resulting URI directly.

Quick decision

| Model | Container family | How to get the URI | |---|---|---| | HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.) | **HuggingFace vLLM** | AWS catalog → "HuggingFace vLLM Inference" (ECR repo `huggingface-vllm`) | | Same as above, multimodal | **HuggingFace vLLM-Omni** | AWS catalog → "HuggingFace vLLM-Omni Inference" (ECR repo `huggingface-vllm-omni`) | | HuggingFace embeddings | TEI | AWS catalog → "HuggingFace Text Embeddings Inference" | | Encoder / cross-encoder rerankers (BERT-family `*ForSequenceClassification`) | TEI | Same as embeddings | | **Generative rerankers** (causal-LM, e.g. Qwen3-Reranker) | HuggingFace vLLM | Same as text-generation LLMs — **not TEI**, see "Rerankers: TEI or vLLM?" | | Text-to-image / diffusion (Stable Diffusion, FLUX) | DJL Inference | AWS catalog → "DJL Inference" — **not** HF Inference Toolkit, see "Known-broken images" | | HuggingFace classifiers, NER, QA, summarization | HF Inference Toolkit (CPU) | AWS catalog → "HuggingFace PyTorch Inference"; GPU tags currently broken — see "Known-broken images" | | User specifically wants SGLang | HuggingFace SGLang | AWS catalog → "HuggingFace SGLang Inference" | | No compatible `huggingface-vllm` tag (verified incompatibility or region gap — see "Rule zero") | vLLM (AWS) | AWS catalog → "vLLM" section — fallback only, never for version freshness | | User specifically wants DJL-LMI | DJL Inference | AWS catalog → "DJL Inference" | | Amazon Nova | SageMaker JumpStart | Use JumpStart, not raw endpoint creation | | Custom inference code | BYOC | User provides URI |

**HuggingFace-curated DLCs are mandatory when one is compatible (see "Rule zero").** `huggingface-vllm` is layered directly on the AWS vLLM DLC — **identical `SM_VLLM_*` env contract and the same cu130 AMI rule** — and adds current `transformers`, current `huggingface_hub` + `hf_xet` (avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS `vllm` image is a compatibility escape hatch only; it usually shows a higher vLLM version than `huggingface-vllm`, and that is not a reason to pick it.

**Do not use TGI.** Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since [PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960), June 2026, its `ModelBuilder` auto-routes `text-generation` to the HuggingFace vLLM DLC and multimodal tasks to Huggin