/setup

name: setup
description: Install sparkrun and configure DGX Spark clusters

<Purpose> Provides complete reference for installing sparkrun, creating and managing cluster configurations, setting up SSH mesh for multi-node inference, configuring CX7 networking, Docker group membership, file permissions, page cache, earlyoom OOM protection, and diagnostics on NVIDIA DGX Spark systems. </Purpose>

<Use_When>

• User wants to install or update sparkrun
• User wants to create, modify, or manage cluster configurations
• User wants to set up SSH for multi-node inference
• User wants to configure CX7 network interfaces
• User wants to fix file permissions on cluster hosts
• User wants to clear page cache on cluster hosts
• User wants to set up Docker group or earlyoom
• User asks about sparkrun configuration or setup
• User is getting started with DGX Spark inference for the first time

</Use_When>

<Do_Not_Use_When>

• User wants to run, stop, or monitor workloads -- use the run skill instead
• User wants to manage recipe registries or create recipes -- use the registry skill instead

</Do_Not_Use_When>

<Steps>

Installation

# Ensure that uv is installed
uv --version

# uv can be installed with (IF NEEDED)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install sparkrun as a CLI tool
uvx sparkrun setup install

# Update sparkrun + registries (top-level shortcut)
sparkrun update

# Update to latest version (and registries)
sparkrun setup update

# Update sparkrun only (skip registry sync)
sparkrun setup update --no-update-registries

# Update channels (advanced): stable (PyPI, default), beta (develop), alpha (develop-next)
sparkrun update --beta      # switch to and update the beta channel
sparkrun update --alpha     # or --yolo; bleeding-edge develop-next
sparkrun update --stable    # switch back to the stable PyPI release

Setup Wizard (Recommended for First-Time Setup)

The wizard handles all setup steps in a single guided flow:

# Interactive wizard (auto-launches when no default cluster exists)
sparkrun setup wizard

# Pre-populate hosts and cluster name
sparkrun setup wizard --hosts <ip1>,<ip2> --cluster mylab

# Non-interactive (accept all defaults)
sparkrun setup wizard --yes --hosts <ip1>,<ip2>

# Dry-run preview
sparkrun setup wizard --dry-run --hosts <ip1>,<ip2>

The wizard performs these phases: 1. **Cluster Setup** — detects CX7 peers, creates cluster, sets default 2. **SSH Mesh** — passwordless SSH across all hosts + control machine 3. **CX7 Configuration** — high-speed networking (if CX7 detected) 4. **Docker Group** — ensures user can run Docker without sudo 5. **Sudoers Entries** — scoped sudoers for fix-permissions and clear-cache 6. **earlyoom** — OOM protection to prevent system hangs

Running `sparkrun setup` with no subcommand auto-launches the wizard when no default cluster is configured.

Cluster Management

Clusters are named host groups saved in `~/.config/sparkrun/clusters/`.

# Create a cluster (first host = head node)
sparkrun cluster create <name> --hosts <ip1>,<ip2>,... [-d "description"] [--user <ssh_user>]
sparkrun cluster create <name> --hosts <ips> --transfer-mode push --transfer-interface cx7

# Set as default (used when --hosts/--cluster not specified)
sparkrun cluster set-default <name>

# View clusters
sparkrun cluster list
sparkrun cluster show <name>
sparkrun cluster default

# Modify
sparkrun cluster update <name> --hosts <new_hosts> [--user <user>] [-d "desc"]
sparkrun cluster update <name> --add-host 10.0.0.5
sparkrun cluster update <name> --add-host 10.0.0.5,10.0.0.6
sparkrun cluster update <name> --remove-host 10.0.0.2
sparkrun cluster update <name> --transfer-mode push --transfer-interface cx7
sparkrun cluster delete <name>
sparkrun cluster unset-default

Cluster Options

| Option | Description | |--------|-------------| | `--hosts, -H` | Comma-separated host list | | `--hosts-file` | File with hosts (one per line) | | `--user, -u` | SSH username for this cluster | | `--cache-dir` | HuggingFace cache directory for this cluster | | `--transfer-mode` | Resource transfer mode (auto, local, push, delegated) | | `--transfer-interface` | Network interface for transfers (auto, cx7, mgmt) | | `--add-host` | Add host(s) to the cluster (repeatable, comma-ok) | | `--remove-host` | Remove host(s) from the cluster (repeatable, comma-ok) |

SSH Setup

Multi-node inference requires passwordless SSH between all hosts. sparkrun bundles a mesh setup script.

# Set up SSH mesh across cluster hosts (interactive -- prompts for passwords)
sparkrun setup ssh --cluster <name>
sparkrun setup ssh --hosts <ip1>,<ip2> [--user <username>]

# Include extra hosts (e.g. control machine) in the mesh
sparkrun setup ssh --cluster <name> --extra-hosts <control_ip>

# Exclude the local machine from the mesh
sparkrun setup ssh --cluster <name> --no-include-self

# Dry-run to see what would happen
sparkrun setup ssh --cluster <name> --dry-run

**IMPORTANT:** The SSH setup script runs interactively (prompts for passwords on first connection). Do NOT capture its output -- let it pass through to the terminal.

CX7 Networking

Configure ConnectX-7 network interfaces on cluster hosts for high-speed transfers.

# Auto-detect CX7 interfaces and configure with defaults
sparkrun setup cx7 --cluster <name>
sparkrun setup cx7 --hosts <ip1>,<ip2>

# Override subnets
sparkrun setup cx7 --cluster <name> --subnet1 192.168.11.0/24 --subnet2 192.168.12.0/24

# Force reconfiguration and set MTU
sparkrun setup cx7 --cluster <name> --force --mtu 9000

# Dry-run
sparkrun setup cx7 --cluster <name> --dry-run

Requires passwordless sudo on target hosts. Will prompt for sudo password if needed.

Docker Group Membership

Ensure the SSH user can run Docker commands without sudo.

sparkrun setup docker-group --cluster <name>
sparkrun setup docker-group --hosts <ip1>,<ip2> [--user <username>]
sparkrun setup docker-group