/run-experiment

name: run-experiment
description: Deploy and run ML experiments on local, remote, Vast.ai, or Modal serverless GPU. Use when user says "run experiment", "deploy to server", "跑实验", or needs to launch training jobs.
argument-hint: "[experiment-description]"
allowed-tools: Bash(*), Read, Grep, Glob, Edit, Write, Skill(serverless-modal)

Run Experiment

Deploy and run ML experiment: $ARGUMENTS

Workflow

Step 1: Detect Environment

Read the project's `CLAUDE.md` to determine the experiment environment:

• **Local GPU** (`gpu: local`): Look for local CUDA/MPS setup info
• **Remote server** (`gpu: remote`): Look for SSH alias, conda env, code directory
• **Vast.ai** (`gpu: vast`): Check for `vast-instances.json` at project root — if a running instance exists, use it. Also check `CLAUDE.md` for a `## Vast.ai` section.
• **Modal** (`gpu: modal`): Serverless GPU via Modal. No SSH, no Docker, auto scale-to-zero. Delegate to `/serverless-modal`.

**Modal detection:** If `CLAUDE.md` has `gpu: modal` or a `## Modal` section, the entire deployment is handled by `/serverless-modal`. Jump to **Step 4: Deploy (Modal)** — Steps 2-3 are not needed (Modal handles code sync and GPU allocation automatically).

**Environment contract** (`../shared-references/compute-env-contract.md`): read existing server notes or the provider's ledger before preparing the environment. Reuse the user's confirmed conda/venv and working command; a missing ARIS ledger does not require rebuilding. If no relevant validation exists, run one small, task-relevant smoke in that environment and record the command and actual output. Reuse prior validation when the environment and execution path have not changed. New or rebuilt environments need the spec, import/kernel checks and a fresh agent-follows-doc pass once setup is stable; substantive installation or invocation changes need the affected checks. Plain prose/log edits do not trigger a rebuild or fresh recheck. This execution evidence does not replace research acceptance.

**Preparation failures:** carry the same failure, attempted fixes and remaining budget across child skills in the existing run notes/tracker. After two targeted repairs to the same environment problem make no new progress, stop repeating preparation and return the exact command/error to the main owner for takeover or an authorized alternative. Missing permissions, authentication, network access or a compatible driver are environment prerequisites, not reasons to rewrite the training script. Keep the original code-sanity retry budget and reviewer gates.

**Vast.ai detection priority:** 1. If `CLAUDE.md` has `gpu: vast` or a `## Vast.ai` section:

• If `vast-instances.json` exists and has a running instance → use that instance
• If no running instance → call `/vast-gpu provision` which analyzes the task, presents cost-optimized GPU options, and rents the user's choice

2. If no server info is found in `CLAUDE.md`, ask the user.

Step 2: Pre-flight Check

Check GPU availability on the target machine:

**Remote (SSH):**

ssh <server> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader

**Remote (Vast.ai):**

ssh -p <PORT> root@<HOST> nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader

(Read `ssh_host` and `ssh_port` from `vast-instances.json`, or run `vastai ssh-url <INSTANCE_ID>` which returns `ssh://root@HOST:PORT`)

**Local:**

nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv,noheader
# or for Mac MPS:
python -c "import torch; print('MPS available:', torch.backends.mps.is_available())"

Free GPU = memory.used < 500 MiB.

Step 3: Sync Code (Remote Only)

Check the project's `CLAUDE.md` for a `code_sync` setting. If not specified, default to `rsync`.

Option A: rsync (default)

Prefer the project's existing deployment command or explicit file list. Otherwise, allow directory traversal and include the configuration/small resources read by the actual experiment command, alongside Python sources. For example:

rsync -avz --prune-empty-dirs \
  --exclude='.*' --exclude='__pycache__/' --exclude='/venv/' --exclude='/env/' \
  --exclude='data/' --exclude='datasets/' --exclude='weights/' \
  --exclude='checkpoints/' --exclude='outputs/' --exclude='wandb/' \
  --include='*/' --include='*.py' \
  --include='/configs/sanity.json' --include='/assets/vocab.json' \
  --exclude='*' "<local_src>/" "<server>:<remote_dst>/"

Replace the two explicit config/resource paths with those your command needs; add other required small files explicitly. Keep exclusions before directory/file includes, and add project-specific private paths there. Hidden agent/config files, data, weights and outputs stay out of this example. Use the project source root, not the user's home directory; do not replace these rules with an unfiltered copy. A successful sync exit is not a launch check: run the smallest actual experiment from the configured remote code directory and inspect its log and result file.

Option B: git (when `code_sync: git` is set in CLAUDE.md)

Push local changes to remote repo, then pull on the server:

# 1. Push from local
git add -A && git commit -m "sync: experiment deployment" && git push

# 2. Pull on server
ssh <server> "cd <remote_dst> && git pull"

Benefits: version-tracked, multi-server sync with one push, no rsync include/exclude rules needed.

Option C: Vast.ai instance

Sync code to the vast.ai instance (always rsync, code dir is `/workspace/project/`):

rsync -avz -e "ssh -p <PORT>" \
  --include='*.py' --include='*.yaml' --include='*.yml' --include='*.json' \
  --include='*.txt' --include='*.sh' --include='*/' \
  --exclude='*.pt' --exclude='*.pth' --exclude='*.ckpt' \
  --exclude='__pycache__' --exclude='.git' --exclude='data/' \
  --exclude='wandb/' --exclude='outputs/' \
  ./ root@<HOST>:/workspace/project/

Install dependencies per the env contract