/huggingface-llm-trainer

name: huggingface-llm-trainer
description: "Train or fine-tune language models with TRL or Unsloth on Hugging Face Jobs, including SFT, DPO, GRPO, reward models, and GGUF conversion. Use for cloud LLM training; use huggingface-vision-trainer for vision tasks."
license: Apache-2.0 (modified; see UPSTREAMS.json)

TRL Training on Hugging Face Jobs

Overview

Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required — models train on cloud GPUs and results are automatically saved to the Hugging Face Hub.

**TRL provides multiple training methods:**

• **SFT** (Supervised Fine-Tuning) — standard instruction tuning
• **DPO** (Direct Preference Optimization) — alignment from preference data
• **GRPO** (Group Relative Policy Optimization) — online RL training
• **Reward Modeling** — train reward models for RLHF

See `references/training_methods.md` for method overviews and selection guidance.

When to Use Unsloth

Use **Unsloth** (`references/unsloth.md`) instead of standard TRL when GPU memory is limited (~60% less VRAM), speed matters (~2x faster), training large models (>13B), or training Vision-Language Models (Unsloth has `FastVisionModel` support). See `scripts/unsloth_sft_example.py` for a production-ready training script.

Key Directives

1. **Submit jobs via `hf jobs uv run` (CLI) or the `hf_jobs()` MCP tool if the Hugging Face MCP server is configured** — pass the training script inline, don't save to a local file unless the user explicitly requests it. If the user asks to "train a model" or "fine-tune", create the training script AND submit the job immediately. 2. **Always include Trackio** for real-time monitoring — use `scripts/` templates. 3. **Provide job details after submission**: job ID, monitoring URL, estimated time; note the user can request status checks later. 4. **Use example scripts as templates**: `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, etc.

Local Script Execution

Repository scripts use PEP 723 inline dependencies. Run them with `uv run`:

uv run scripts/estimate_cost.py --help
uv run scripts/dataset_inspector.py --help

Prerequisites Checklist

**Account & Authentication:**

• Hugging Face account with Pro/Team/Enterprise plan (Jobs require a paid plan); authenticated login.
• **HF_TOKEN for Hub push is CRITICAL** — the training environment is ephemeral, so results are lost unless pushed to the Hub. Token must have write permissions. Pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in the job config.

**Dataset Requirements:**

• Must exist on the Hub or be loadable via `datasets.load_dataset()`.
• Format must match the training method (SFT: messages/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only). **Always validate unknown datasets first** (see Dataset Validation below).
• Size appropriate for hardware (demo: 50-100 examples on t4-small; production: 1K-10K+ on a10g-large/a100-large).

**Critical Settings:**

• Timeout must exceed expected training time — default 30min is too short; minimum recommended 1-2 hours. The job fails and loses all progress if the timeout is exceeded.
• Hub push must be enabled: `push_to_hub=True`, `hub_model_id="username/model-name"`, `secrets={"HF_TOKEN": "$HF_TOKEN"}`.

Asynchronous Jobs

Training jobs run in the background and can take hours. After submitting: report the job ID, monitoring URL, and estimated time; wait for the user to request status checks rather than polling. Initial logs can take 30-60 seconds to appear.

Quick Start

**Sequence length:** TRL config classes use `max_length` (not `max_seq_length`). Default is `max_length=1024` (truncates from right) — override higher for longer context, lower under memory constraints, or `None` for vision models (to avoid cutting image tokens).

Approach 1: UV Scripts (default choice)

UV scripts use PEP 723 inline dependencies for clean, self-contained training:

hf_jobs("uv", {
    "script": """
# /// script
# dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"]
# ///
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import trackio

dataset = load_dataset("trl-lib/Capybara", split="train")
dataset_split = dataset.train_test_split(test_size=0.1, seed=42)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=dataset_split["train"],
    eval_dataset=dataset_split["test"],
    peft_config=LoraConfig(r=16, lora_alpha=32),
    args=SFTConfig(
        output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model",
        num_train_epochs=3, eval_strategy="steps", eval_steps=50,
        report_to="trackio", project="my_project", run_name="my_run",
    ),
)
trainer.train()
trainer.push_to_hub()
""",
    "flavor": "a10g-large",
    "timeout": "2h",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})

The `script` parameter accepts inline code or a publicly-accessible/Hub/GitHub/Gist URL — **local file paths do not work** (jobs run in isolated containers with no access to the local filesystem). To use a local script, upload it to the Hub first (`hf upload ...`) and reference its resolved URL.

Approach 2: TRL Maintained Scripts

Run TRL's battle-tested example scripts directly from a URL, passing CLI-style `script_args` (`--model_name_or_path`, `--dataset_name`, `--output_dir`, `--push_to_hub`, `--hub_model_id`). Available at https://github.com/huggingface/trl/tree/main/examples/scripts.

Approach 3: HF Jobs CLI

When no `hf_jobs`-style tool is available, use the `hf jobs` CLI directly. **Flags must come before the script URL**, the subcommand order is `hf jobs uv run` (not `run uv`), and use `--secrets` (plural):

hf jobs uv run \
  --flavor a10g-large --timeout 2h --secrets HF_TOKEN \
  "https://huggingface.co/user/repo/resolve/main/train.py"

Check status: `hf jobs ps`, `hf jobs logs <job-id>`, `hf jobs inspect <job-id>`, `hf