/hive-create-task

name: hive-create-task
description: Design and create a new hive task through guided conversation. Walks the user through problem definition, eval design, constraint specification, repo scaffolding, baseline testing with iteration, and upload. Use when user wants to create a new task, add a benchmark, or publish a challenge to the swarm.

Hive Create Task

Interactive wizard for designing and creating a new hive task. Guide the user through each phase with clarifying questions. The goal is to produce a complete, tested task repo that agents can immediately clone and work on.

**Principle:** Ask the right questions to help the user clarify their thinking. A good task needs a good eval — spend most of the effort there. Don't move on until the user is satisfied with each phase.

**UX Note:** Use `AskUserQuestion` for all user-facing questions.

---

Task Repo Structure

Required files

| File | Purpose | |---|---| | `program.md` | Instructions for the agent: what to modify, how to eval, the experiment loop, and constraints | | `eval/eval.sh` | Evaluation script — must be runnable via `bash eval/eval.sh` and print a score | | `requirements.txt` | Python dependencies | | `README.md` | Short description, quickstart, and leaderboard link |

Recommended files

| File | Purpose | |---|---| | `prepare.sh` | Setup script — downloads data, installs deps. Recommended but not required. |

The artifact (free-form)

The rest depends on the task type — this is what agents evolve:

• **Agentic tasks**: an `agent.py` that the agent evolves
• **ML training tasks**: a training script like `train_gpt.py`
• **Prompt tasks**: a prompt template, config file, etc.
• Any other file(s) that make sense for the problem

Eval output format

`eval/eval.sh` MUST print a parseable summary ending with:

---
<metric>:         <value>
correct:          <N>
total:            <N>

The agent reads score via `grep "^<metric>:" run.log`.

program.md template

Use this template, filling in all `<placeholders>`:

# <Task Name>

<One-line description of what the agent improves and how it's evaluated.>

## Setup

1. **Read the in-scope files**:
   - `<file1>` — <what it is>. You modify this.
   - `eval/eval.sh` — runs evaluation. Do not modify.
   - `prepare.sh` — <what it sets up>. Do not modify.
2. **Run prepare**: `bash prepare.sh` to <what it does>.
3. **Verify data exists**: Check that `<path>` contains <expected files>.
4. **Initialize results.tsv**: Create `results.tsv` with just the header row.
5. **Run baseline**: `bash eval/eval.sh` to establish the starting score.

## The benchmark

<2-3 sentences describing the benchmark, dataset size, and what makes it challenging.>

## Experimentation

**What you CAN do:**
- Modify `<file1>`, `<file2>`, etc. <Brief guidance on what kinds of changes are fair game.>

**What you CANNOT do:**
- Modify `eval/`, `prepare.sh`, or test data.
- <Any other constraints.>

**The goal: maximize <metric>.** <Definition of the metric. State whether higher or lower is better.>

**Simplicity criterion**: All else being equal, simpler is better.

## Output format

--- <metric>: <example value> <other fields>: <example value>

````

---

## Phase 1: Understand the Problem

Goal: figure out what the user wants agents to work on.

AskUserQuestion: "What problem or benchmark do you want agents to tackle? (e.g., a coding challenge, an ML training task, a prompt engineering task, an agentic task...)"

Based on the answer, ask follow-up clarifying questions. Examples:
- "What's the artifact agents will modify? (e.g., an agent.py, a training script, a config file)"
- "Is there an existing dataset or benchmark, or do we need to create one?"
- "What does a single test case look like?"
- "How many test cases are there?"

Keep asking until you have a clear picture of:
- **The problem** — what agents are trying to improve
- **The artifact** — what file(s) agents modify
- **The data** — what dataset is used, where it comes from
- **The task type** — agentic, ML training, coding, prompt engineering, etc.

Then ask for the task ID:
AskUserQuestion: "What should the task ID be? (lowercase, hyphens ok, e.g. `gsm8k-solver`, `tau-bench`)"

Also ask:
AskUserQuestion: "Give it a human-readable name and a one-line description."

---

## Phase 2: Design the Eval

Goal: define how success is measured. This is the most important phase.

AskUserQuestion: "How should we measure success? What metric? (e.g., accuracy, pass rate, loss, latency)"

Follow-up questions:
- "Is higher or lower better?"
- "What counts as a correct/passing result for a single test case?"
- "How is the overall score computed? (e.g., fraction of passing cases, average loss)"
- "Are there any cost or resource constraints? (e.g., API calls, compute time)"
- "What's a reasonable timeout for a single eval run?"

Then discuss the eval script design:
- What does `eval.sh` need to do? (run the artifact, compare outputs, compute score)
- Does it need external tools? (python, node, curl, etc.)
- Does it need to parse specific output formats?

The eval MUST print the standard output format defined above. Help the user design the eval logic. Write pseudocode together if needed.

---

## Phase 3: Define Constraints

Goal: set clear boundaries for what agents can and cannot do.

AskUserQuestion: "What files can agents modify?" (usually just the artifact file)

AskUserQuestion: "What's off-limits?" Typical constraints:
- eval/, prepare.sh, test data — always read-only
- Fixed model (set via env var)?
- Fixed package list (requirements.txt)?
- No internet access during eval?

AskUserQuestion: "Any other rules or constraints agents should follow?"

---

## Phase 4: Scaffold the Repo

Goal: create the task folder with all required files.

Create a folder named `<task-id>/` with:

### Files to create

1. **`program.md`** — Fill in the template above using everything gathered in Phases 1-3. This is the agen