/cloud-inference-routing-workflow

name: cloud-inference-routing-workflow
description: Choose the right cloud AI inference, model training, model conversion, or GPU infrastructure path before implementation. Use when Codex works with Runpod, Hugging Face, AWS, AWS Lambda inference, Vast.ai, CoreWeave, GPU Pods, Serverless GPU endpoints, model weights, vLLM, TGI, SGLang, custom containers, LoRA/adapters, dataset upload, model conversion, cheap GPU rentals, or managed inference endpoints.

Cloud Inference Routing Workflow

Purpose

Choose the smallest correct cloud AI inference path before changing provider resources, spending GPU budget, moving model weights, or deploying endpoints.

The practical decision is whether the agent should use an official provider plugin, provider MCP server, provider CLI, SDK, managed inference endpoint, rented GPU instance, cluster platform, or a stack-owned implementation workflow.

Source Check

Use repo-local model code, Dockerfiles, provider configs, checked-out scripts, installed provider plugins, installed provider CLIs, bundled MCP servers, and official provider documentation before making claims about current behavior.

Preferred official sources:

• Runpod docs and bundled MCP servers for Runpod Pods, Serverless, Flash, templates, volumes, registries, and docs lookup.
• Hugging Face plugin and `huggingface_hub` CLI for Hub repos, models, datasets, Spaces, Inference Endpoints, jobs, and papers.
• AWS Agent Toolkit, AWS CLI, AWS SDKs, and AWS docs for Lambda, SageMaker, Bedrock, ECS, EKS, Batch, IAM, billing, and observability.
• Vast.ai docs and CLI for low-cost rented GPU instances.
• CoreWeave docs for Kubernetes-backed GPU infrastructure, clusters, storage, networking, and observability.

Translate documentation into the concrete project decision it changes.

Routing Workflow

1. Inspect the workload:

• task: inference endpoint, batch inference, fine-tuning, full training, eval, model conversion, quantization, benchmark, notebook, or one-off GPU shell
• model family, parameter count, precision, context length, adapters, tokenizer, and required runtime
• expected VRAM, CPU RAM, disk, network, container image, driver, CUDA, and startup time
• latency, throughput, concurrency, cold-start tolerance, uptime, privacy, and cleanup needs
• artifact flow: model weights, private datasets, outputs, logs, checkpoints, and caches

2. Choose the provider lane:

• quick managed inference
• quick custom endpoint
• cheap flexible GPU box
• training or conversion workspace
• Kubernetes or production cluster
• existing provider familiarity

3. Prefer familiar first-party paths when they fit:

• Runpod for fast GPU Pods, Serverless endpoints, Flash, cheap/flexible experiments, templates, and resource management through MCP.
• Hugging Face for model and dataset repos, conversion/publishing workflows, Inference Endpoints, Spaces, jobs, and Hub-native collaboration.
• AWS when the project already uses AWS accounts, IAM, S3, CloudWatch, Lambda, SageMaker, Bedrock, ECS, EKS, or Batch.

4. Route to official tools:

• Use `runpod` MCP for Runpod resource mutations and `runpod-docs` MCP for docs lookup when available.
• Use the upstream Runpod skills under the exported `skills/` tree for Runpod `flash`, `runpodctl`, and companion CLI details. Refresh those skills with `npx skills update` or `npx skills add runpod/skills` instead of hand-editing them.
• Use the Hugging Face Codex plugin and Hugging Face CLI for Hub, model, dataset, Space, endpoint, or job work when available.
• Use the AWS Agent Toolkit and AWS CLI or SDKs for AWS inference work when available.
• Use provider CLIs or docs for Vast.ai and CoreWeave until a provider-owned agent surface is available or a concrete repeated Socket workflow justifies a new slice.

5. Confirm boundaries before mutation:

• provider account, project, region, org, namespace, and budget
• API key or profile source
• model and dataset license or private-use boundary
• exact resources to create, update, stop, delete, or scale
• expected hourly cost, storage cost, bandwidth cost, idle behavior, and teardown path

6. Choose validation:

• read-only docs, list, or pricing query for planning
• local smoke test or CPU fallback before GPU spend
• minimal GPU probe before full training or endpoint deployment
• endpoint health, latency, throughput, logs, and cost check after deployment
• cleanup verification for stopped Pods, terminated instances, deleted endpoints, detached volumes, and lingering storage

Provider Fit

Runpod

Use Runpod first when Gale wants quick and easy GPU access, cheap/flexible infrastructure, Serverless endpoints, GPU Pods, Flash, templates, network volumes, or direct resource management from an agent.

Use `runpod-docs` MCP for docs questions. Use `runpod` MCP for resource operations after confirming `RUNPOD_API_KEY`, account, resource name, GPU type, cost, and cleanup.

Use the installed upstream Runpod skills for details:

• `skills/flash` for `runpod-flash` SDK and CLI workflows.
• `skills/runpodctl` for Runpod CLI workflows.
• `skills/companion-clis` for Hugging Face, GitHub, Docker, and AWS CLI support commonly needed around Runpod deployments.

Hugging Face

Use Hugging Face first when the work is model or dataset centered: Hub repos, checkpoints, safetensors, GGUF or Core ML conversion artifacts, datasets, evals, Spaces, Inference Endpoints, or sharing/publishing.

Prefer the installed Hugging Face Codex plugin and official CLI. Do not reimplement Hub auth, repo management, or endpoint setup in Socket unless a small routing note removes real ambiguity.

AWS And Lambda

Use AWS first when the project already depends on AWS IAM, S3, CloudWatch, Lambda, SageMaker, Bedrock, ECS, EKS, or Batch.

Treat Lambda inference as a narrow fit: small models, lightweight CPU inference, short-lived request handlers, or calls out to Bedrock/SageMaker/hosted