/dataset

description: "Manage dataset manifests for data that is too large or unsuitable to store directly in the wiki. The wiki becomes the index/interface; the data stays external."
argument-hint: "list [--status <status>] [--storage <mode>] [--view summary|manifests|schema|locations] [--limit N] [--format table|list] | add \"<title>\" [--location <path-or-url>] [--format <fmt>] | show <slug-or-path> | scan-outputs [--dry-run] | migrate-output <output-path> [--dry-run|--apply] | profile <slug> [--dry-run|--apply] | sample <slug> [--limit N] [--dry-run|--apply] [--include-archived] [--wiki <name>] [--local]"
allowed-tools: Read, Write, Edit, Glob, Grep, Bash(ls:*), Bash(wc:*), Bash(date:*), Bash(mkdir:*), Bash(du:*), Bash(stat:*), Bash(head:*), Bash(file:*)

Your task

**Resolve the wiki.** Do NOT broadly search the filesystem — follow these steps: 1. Read `$HOME/.config/llm-wiki/config.json`. If it has `hub_path`, expand leading `~` only (not tildes in `com~apple~CloudDocs`) and prefer that path; use `resolved_path` only as a fallback cache when the expanded `hub_path` is unavailable and `resolved_path` is initialized. If config has only `resolved_path`, use it. If the configured path can be statted but reading `wikis.json` or listing `topics/` fails with `Operation not permitted`, stop and ask the user to grant Full Disk Access/iCloud Drive access to the launcher; do not fall back to `~/wiki` or `resolved_path`. Do not write machine-specific `resolved_path` into shared configs. 2. If no config → read `$HOME/wiki/_index.md`. If it exists → HUB = `$HOME/wiki`. If nothing found, ask the user where to create the wiki. 3. **Wiki location** (first match): `--local` → `.wiki/` in CWD; `--wiki <name>` → `HUB/wikis.json` lookup with portable path resolution (`<HUB>`, `~`, absolute, or HUB-relative); if the registry path is stale, fall back to `HUB/topics/<name>`; CWD has `.wiki/` → use it; else → HUB. 4. Read `<wiki>/_index.md` to verify. If missing → stop with "No wiki found. Run `/wiki init` first."

Archive rule: dataset commands operate on active topic wikis by default. If a named target is archived, stop and ask the user to restore it or rerun with `--include-archived`. When explicitly included, list/update manifests only inside that archived topic path and label the result as archived. This is separate from dataset manifest `status: archived`.

After resolving the wiki, read the dataset reference at `skills/wiki-manager/references/datasets.md`, then run the requested subcommand.

Dataset manifests are for large or external data that should not be copied into `raw/` or `wiki/`. The wiki stores pointers, schema notes, small samples, profiles, query recipes, and provenance. Actual datasets remain at their original filesystem path, object store, URL, database, or archive.

Be opinionated about whether a dataset manifest is warranted:

• Good fit: large, mutable, remote, compressed, binary, database-backed, or

query-oriented data that should stay outside markdown.

• Too small: a short CSV/JSON/text file that can be ingested as one immutable

`raw/data/` source.

• Too operational: a next-action queue about a corpus belongs in inventory,

with an optional link to a dataset manifest.

• Too broad: many independent sources/pages should usually be

`ingest-collection`, not a dataset manifest.

Before larger pivots, preview the shape: one dataset manifest plus any linked inventory record, sample/profile/query notes, and what data will remain external. Do not ask the user to approve a big migration without showing this sample.

Parse $ARGUMENTS

Subcommands:

• **list**: List dataset manifests, optionally filtered by status/storage and view mode.
• **add "<title>"**: Create a dataset manifest under `datasets/<slug>/`.
• **show <slug-or-path>**: Read one dataset manifest and related profile/sample indexes.
• **scan-outputs [--dry-run]**: Find output artifacts that describe datasets but still live in `output/`.
• **migrate-output <output-path> [--dry-run|--apply]**: Convert one legacy output artifact into one or more dataset manifests. Default is **dry-run**. `--apply` is required to write.
• **profile <slug> [--dry-run|--apply]**: Record lightweight dataset size/format/schema observations. Never load the full dataset into the wiki.
• **sample <slug> [--limit N] [--dry-run|--apply]**: Save a tiny representative sample or sample recipe under the dataset folder.

Statuses: `proposed`, `active`, `external`, `archived`, `unavailable`.

Storage modes: `local`, `remote`, `external`, `hybrid`.

Schema statuses: `unknown`, `inferred`, `declared`, `validated`.

Archive flag: `--include-archived` explicitly allows the selected wiki itself to be archived.

Ensure Structure

Dataset structure is lazy. Do not create it during read-only operations, and do not create empty sample/profile/query folders until the relevant command needs them.

Before a write:

1. Ensure `datasets/` exists with `_index.md`. 2. For a manifest write, ensure `datasets/<slug>/` exists with `_index.md`. 3. For `profile`, ensure `datasets/<slug>/profiles/_index.md`. 4. For `sample`, ensure `datasets/<slug>/samples/_index.md`. 5. For query recipes, ensure `datasets/<slug>/queries/_index.md`. 6. Missing structure may be created, but never move or copy user datasets into the wiki.

Add

1. Slugify the title using normal wiki filename rules. 2. Create `datasets/<slug>/` with `_index.md`. Do not create `samples/`, `profiles/`, or `queries/` yet unless the user is also writing one of those notes now. 3. Write `MANIFEST.md` with frontmatter per `references/datasets.md`. 4. Record any `--location`, `--format`, size, access, license, or source hints provided by the user. 5. Append to `log.md`: `## [YYYY-MM-DD] dataset | added <slug>` 6. Rebuild `datasets/_index.md`.

List

1. If `datasets/` or `datasets/_index.md` is missing, report that there are no dataset manifests yet and do not create files. 2. Read `datasets/_in