/hf-cloud-serving-image-selection

name: hf-cloud-serving-image-selection
description: "Select and verify the current region-specific serving container URI for a SageMaker model deployment. Use after the deployment pathway is chosen and before endpoint code; never infer an image URI from memory."
license: Apache-2.0 (modified; see UPSTREAMS.json)

Serving Image Selection

The serving container is the single thing most likely to break a SageMaker deployment that "looked correct on paper". Wrong container, stale tag, or the wrong AMI — all produce the same opaque `Failed to pass health check` error.

Rule zero: HuggingFace images always win

When both a HuggingFace-curated family (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-sglang`, `tei`, `huggingface-pytorch-inference`) and a generic family (`vllm`, `vllm-omni`, `sglang`, `djl-inference`) can serve the model, **the HuggingFace one is mandatory, not preferred**. The only valid reasons to use a generic image:

1. **Verified incompatibility** — the model needs an architecture/modality/feature no available HuggingFace tag supports, confirmed against the catalog (not assumed). 2. **No HuggingFace tag exists in the target region** and mirroring is not an option. 3. **The HuggingFace image is in "Known-broken images"** below.

A **newer version number on the generic repo is not a reason**. The AWS `vllm` repo often publishes a higher vLLM version than `huggingface-vllm`; an older-but-compatible `huggingface-vllm` tag still wins. "Latest vLLM" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied.

Where image URIs come from

**Primary source: AWS's official Deep Learning Containers catalog.**

URL: https://aws.github.io/deep-learning-containers/reference/available_images/

This page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2/ECS/EKS). When picking a URI for a deployment, **read it from this page directly** — copy the example URL, substitute `<region>` with the user's region, and pass it to `deploy.py --image-uri`.

The example URLs use `763104351884` as the account ID for most regions. A few regions use different accounts (e.g. `eu-south-1` uses `692866216735`). Check the [Region Availability page](https://aws.github.io/deep-learning-containers/reference/region_availability/) when in doubt.

**Exception: none currently.** Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via `mirror_image.py` and pass the resulting URI directly.

Quick decision

| Model | Container family | How to get the URI | |---|---|---| | HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.) | **HuggingFace vLLM** | AWS catalog → "HuggingFace vLLM Inference" (ECR repo `huggingface-vllm`) | | Same as above, multimodal | **HuggingFace vLLM-Omni** | AWS catalog → "HuggingFace vLLM-Omni Inference" (ECR repo `huggingface-vllm-omni`) | | HuggingFace embeddings | TEI | AWS catalog → "HuggingFace Text Embeddings Inference" | | Encoder / cross-encoder rerankers (BERT-family `*ForSequenceClassification`) | TEI | Same as embeddings | | **Generative rerankers** (causal-LM, e.g. Qwen3-Reranker) | HuggingFace vLLM | Same as text-generation LLMs — **not TEI**, see "Rerankers: TEI or vLLM?" | | Text-to-image / diffusion (Stable Diffusion, FLUX) | DJL Inference | AWS catalog → "DJL Inference" — **not** HF Inference Toolkit, see "Known-broken images" | | HuggingFace classifiers, NER, QA, summarization | HF Inference Toolkit (CPU) | AWS catalog → "HuggingFace PyTorch Inference"; GPU tags currently broken — see "Known-broken images" | | User specifically wants SGLang | HuggingFace SGLang | AWS catalog → "HuggingFace SGLang Inference" | | No compatible `huggingface-vllm` tag (verified incompatibility or region gap — see "Rule zero") | vLLM (AWS) | AWS catalog → "vLLM" section — fallback only, never for version freshness | | User specifically wants DJL-LMI | DJL Inference | AWS catalog → "DJL Inference" | | Amazon Nova | SageMaker JumpStart | Use JumpStart, not raw endpoint creation | | Custom inference code | BYOC | User provides URI |

**HuggingFace-curated DLCs are mandatory when one is compatible (see "Rule zero").** `huggingface-vllm` is layered directly on the AWS vLLM DLC — **identical `SM_VLLM_*` env contract and the same cu130 AMI rule** — and adds current `transformers`, current `huggingface_hub` + `hf_xet` (avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS `vllm` image is a compatibility escape hatch only; it usually shows a higher vLLM version than `huggingface-vllm`, and that is not a reason to pick it.

**Do not use TGI.** Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since [PR #5960](https://github.com/aws/sagemaker-python-sdk/pull/5960), June 2026, its `ModelBuilder` auto-routes `text-generation` to the HuggingFace vLLM DLC and multimodal tasks to HuggingFace vLLM-Omni.)

Full reasoning for each family in `references/model-to-image.md`.

Rerankers: TEI or vLLM?

"Reranker" covers two very different architectures, and picking wrong wastes a full endpoint-creation cycle (~20 min) before TEI rejects the model:

• **Encoder cross-encoders** (BAAI/bge-reranker-*, mixedbread, most `sentence-transformers` rerankers) — BERT-family models with a classification head. `config.json` has `architectures: [..ForSequenceClassification]` on a TEI-supported encoder type. → **TEI**.
• **Generative rerankers** (Qwen/Qwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes/no token. `config.json` has `architectures: [..ForCau