/serverless-modal

name: serverless-modal
description: "Run GPU workloads on Modal — training, fine-tuning, inference, batch processing. Zero-config serverless: no SSH, no Docker, auto scale-to-zero. Use when user says \"modal run\", \"modal training\", \"modal inference\", \"deploy to modal\", \"need a GPU\", \"run on modal\", \"serverless GPU\", or needs remote GPU compute."
argument-hint: "[task-description]"
allowed-tools: Bash(*), Read, Grep, Glob, Edit, Write

Modal Cloud GPU — Training & Inference

Task: $ARGUMENTS

Overview

**Modal** is a serverless GPU cloud. Key advantages over SSH-based platforms (vast.ai, remote servers):

• **Zero config**: no SSH, no Docker, no port forwarding. Write Python → `modal run` → done.
• **Auto scale-to-zero**: billing stops the instant your code finishes. No idle instances.
• **Local-first**: run `modal run` from your laptop. Code, data, and results stay local; only the GPU function runs remotely.
• **Reproducible environments**: dependencies declared in code via `modal.Image`, not system-level packages.

Treat the `modal.Image` chain as the RENDERED form of the declarative env spec in `../shared-references/compute-env-contract.md` — same spec fields (base, ordered pip phases, env vars, smoke probes), same `env:<name>@<specHash>` ledger entry in `.aris/compute/modal.md`, same three-tier validation before a long run.

**Best for**: Users without a local GPU who need to debug CUDA code, run small-scale tests, or iterate quickly on experiments. The $5 free tier (no card) is enough for code debugging; $30 (with card) covers most small-scale experiment runs.

**Trade-off**: Modal costs more per GPU-hour than vast.ai or Lightning for some GPU tiers, but eliminates setup time and idle billing, often making it cheaper for short/medium workloads. For long training runs (>4 hours), consider vast.ai for lower $/hr.

Authentication

pip install modal
modal setup          # Opens browser login, writes token to ~/.modal.toml
# Verify:
modal run -q 'print("ok")'

• Sign up: https://modal.com (GitHub/Google login)
• Free (no card): **$5/month** — enough for quick tests
• Free (with card): **$30/month** — bind a payment method at https://modal.com/settings for the full free tier. Set a **workspace spending limit** to prevent accidental overcharge (Settings → Usage → Spending Limit)
• Academic: apply for $10k credits | Startups: apply for $25k credits
• Secrets: `modal secret create huggingface-secret HF_TOKEN=hf_xxxxx`

> **Recommended setup**: Bind a card to unlock $30/month, then immediately set a spending limit (e.g., $30) so you never exceed the free tier. Modal will pause your workloads when the limit is hit. > > **SECURITY WARNING**: Always bind your card and set spending limits directly on https://modal.com/settings in your browser. NEVER enter payment information, card numbers, or billing details through Claude Code or any CLI tool. Only the official Modal website is safe for payment operations.

Pricing (source: modal.com/pricing, per-second billing)

| GPU | $/sec | ≈$/hr | VRAM | Bandwidth GB/s | Free budget → hours | |---|---|---|---|---|---| | T4 | $0.000164 | $0.59 | 16GB | 300 | ~8.5 hr ($5) / 50.8 hr ($30) | | L4 | $0.000222 | $0.80 | 24GB | 300 | ~6.3 hr / 37.5 hr | | A10 | $0.000306 | $1.10 | 24GB | 600 | ~4.5 hr / 27.3 hr | | L40S | $0.000542 | $1.95 | 48GB | 864 | ~2.6 hr / 15.4 hr | | A100-40GB | $0.000583 | $2.10 | 40GB | 1555 | ~2.4 hr / 14.3 hr | | A100-80GB | $0.000694 | $2.50 | 80GB | 2039 | ~2.0 hr / 12.0 hr | | H100 | $0.001097 | $3.95 | 80GB | 3352 | ~1.3 hr / 7.6 hr | | H200 | $0.001261 | $4.54 | 141GB | 4800 | ~1.1 hr / 6.6 hr | | B200 | $0.001736 | $6.25 | 192GB | 8000 | ~0.8 hr / 4.8 hr |

CPU: $0.047/core/hr | RAM: $0.008/GiB/hr (GPU typically 90%+ of total cost)

!! Cost Estimation Required !!

Before EVERY run, estimate cost and show to user for confirmation.

Key insights:

• Inference bottleneck is **memory bandwidth**, not compute → high-bandwidth GPUs are often cheaper overall
• 7-8B BF16 inference needs **~22GB VRAM** (weights 15G + KV cache 1G + overhead), T4 (16GB) insufficient
• H100 is often **cheaper than L4** for benchmarks (11x faster but only 5x more expensive)

Cost Estimation Template (required before every run)

Cost estimate (Modal):
  Model: [name] ([params], [precision])
  VRAM: ~[X]GB (weights + KV cache + overhead)
  GPU: [type] ([VRAM]GB, $[X]/sec = $[X]/hr, bandwidth [X] GB/s)
  Estimate: ~[N] min, ~$[X]

7-8B BF16 Benchmark Cost Comparison

| GPU | Speed tok/s | $/hr | 1000 samples x 200tok cost | Duration | |---|---|---|---|---| | **H100** | **224** | $3.95 | **$0.98** | **15 min** | | A100-40GB | 104 | $2.10 | $1.12 | 32 min | | L4 | 20 | $0.80 | $2.22 | 167 min |

Workflow

Step 1: Analyze Task → Estimate Cost → Choose GPU

Same analysis as any GPU skill — determine VRAM needs from model size, pick GPU, estimate hours, calculate cost. See pricing table above.

**VRAM Rules of Thumb:** | Model Size | FP16 VRAM | Recommended GPU | |---|---|---| | ≤3B | ~8GB | T4, L4 | | 7-8B | ~22GB | L4, A10, A100-40GB | | 13B | ~30GB | L40S, A100-40GB | | 30B | ~65GB | A100-80GB, H100 | | 70B | ~140GB | H100:2, H200 |

Step 2: Generate Modal Launcher

Based on the task type, generate the appropriate launcher script.

Pattern A: One-Shot GPU Function (training, evaluation, benchmark)

The most common pattern for `run-experiment` integration. Wraps an existing training script:

import modal

app = modal.App("experiment-name")
# One .pip_install() call per SPEC PHASE (chained calls install in order, so a
# pinned torch in the first call can't be dragged by packages in the second —
# the rendered form of compute-env-contract.md's ordered pip_phases):
image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("torch")                                        # phase 1: pins
    .pip_install("transformers", "accelerate", "datasets", "wandb")  #