/arcgis-to-portaljs

description: Migrate a whole ArcGIS Hub site (opendata.arcgis.com or a Hub Premium custom domain) into a PortalJS Arc portal end-to-end. Harvests the Hub /data.json (DCAT-US) inventory, exports every FeatureService layer through the ArcGIS REST query API (resultOffset paging), converts each to the serverless dual tier (PMTiles render + GeoParquet query) with tabular items to Parquet, pushes everything to Cloudflare R2 via Git LFS, appends dual-tier datasets.json entries, and writes a source-vs-derived parity report. The reusable arcgis-to-portaljs migrator.
allowed-tools: Read, Write, Edit, Bash(ogr2ogr:*), Bash(ogrinfo:*), Bash(tippecanoe:*), Bash(duckdb:*), Bash(git:*), Bash(curl:*), Bash(jq:*), Bash(npx:*), Bash(node:*), Bash(mkdir:*), Bash(cp:*), Bash(wc:*), Bash(zip:*), Bash(command:*), WebFetch

/arcgis-to-portaljs

Migrate an entire **ArcGIS Hub** open-data site into a **PortalJS Arc** portal in one pass. Every Hub site is machine-readable — a DCAT-US catalog at `/data.json`, and every dataset behind it is an ArcGIS REST FeatureService with a documented query API — so migration is a **harvest → export → convert → publish → verify** pipeline that is almost fully automated. The tooling is the reusable **`arcgis-to-portaljs` migrator**: input = a Hub URL, output = a ready-to-deploy PortalJS catalog + a parity report. This is the repeatable asset, not a one-off.

Input is one Hub URL (e.g. `https://hub-lewisville.opendata.arcgis.com` or `https://streamwaterdata.co.uk`). Output: `datasets.json` entries, data on R2, and `arcgis-parity-report.md`.

What this composes — do NOT reinvent

This skill is the **ArcGIS-Hub-specialized orchestrator**. It reuses three existing pieces and adds two ArcGIS-specific steps (the REST export loop and the parity report):

| Step | Owned by | This skill's role | |------|----------|-------------------| | Harvest `/data.json` (DCAT-US → canonical) | `/portaljs-migrate` (`dcat` source) | Reuse its DCAT-US field map; add Hub item **classification** (FeatureService vs table vs non-data). | | **Export FeatureService → GeoJSON** (REST paging) | **new here** | `/portaljs-migrate`'s `arcgis` mode only *links* the query URL; this **downloads** the features so they can be converted and served serverless. | | Convert → PMTiles + GeoParquet (dual tier) | `/portaljs-add-geo` | Reuse its exact `ogr2ogr`/`tippecanoe`/`duckdb` recipes per layer. | | Publish: bulk Git-LFS → R2 + `datasets.json` | `/portaljs-migrate` (`download` mode) + `/portaljs-add-geo` (§10 dual-tier entry) | Reuse the bulk Giftless push; emit dual-tier entries. | | **Parity report** (source vs derived) | **new here** | Record counts, bbox, attribute schema, geometry validity per layer. |

> **Naming note (Phase 0, for review):** the bead names the deliverable `arcgis-to-portaljs` > (a directional, pitch-friendly asset name). The rest of the suite is prefixed `portaljs-`. > Renaming to `portaljs-migrate-arcgis` for suite consistency is a trivial follow-up if the > reviewer prefers it — the skeleton is name-agnostic.

Scope — Phase 0 skeleton

Per the migration epic (po-0qe §6), Phase 0 delivers the **automatable core** — the pipeline above, proven runnable. Explicitly **out of scope here** (tracked as child beads under po-0qe, do not build in this pass):

• **Content pages / branding** (Hub layout pages → MDX) — Phase 2 (Stream).
• **Multi-publisher org structure** (one namespace/repo per publisher) — Phase 2 (Stream).
• **Scheduled sync + cutover playbook** (re-harvest on cadence, redirect map, DNS flip) — Phase 3.
• **Koop FeatureServer-compat adapter** (keep old REST consumers working) — Phase 4 (optional).
• **3D scenes / StoryMaps / drone imagery** — no serverless open equivalent; link-out only.

Required input — ask, don't error

• **Hub URL** (required) — the site root, e.g. `https://hub-lewisville.opendata.arcgis.com`

or a Hub Premium custom domain (`https://streamwaterdata.co.uk`). The skill appends `/data.json` itself; accept either the root or a full `/data.json` URL.

• **Portal directory** — path to the target PortalJS portal (default: current directory). If

none exists, tell the user to run `/portaljs-new-portal` first.

• **Project slug** — the Arc deploy slug for the R2 push (same slug `/portaljs-deploy` uses).

Optional flags: `--limit N` (cap datasets, for a trial run), `--only <slug,slug>` (migrate a subset by Hub slug), `--dry-run` (harvest + plan + print, write nothing), `--namespace-mode theme|owner` (default `owner` — one namespace per Hub publisher; see `/portaljs-migrate`).

**If the Hub URL is missing, ask for it — never dead-end.** Example:

I need an ArcGIS Hub site URL to migrate. Provide the site root, e.g.
  https://hub-lewisville.opendata.arcgis.com
(a Hub Premium custom domain like https://streamwaterdata.co.uk also works.)

Runtime — native shell-out

Same native stack as `/portaljs-add-geo` (GDAL, tippecanoe, duckdb+spatial) plus `jq` for JSON slicing and `node`/`curl` for the REST paging. No server, no container — the whole migration runs on the operator's machine, keeping PortalJS's zero-backend bet.

Steps

1. Gather input (interview if thin)

From `$ARGUMENTS` extract `HUB_URL`, `PORTAL_DIR` (default `.`), `PROJECT_SLUG`, and the optional flags. Normalize `HUB_URL`: strip a trailing `/data.json` and any trailing slash to get `SITE_ROOT`; `DATA_JSON="$SITE_ROOT/data.json"`. If `HUB_URL` is absent, emit the prompt above and stop.

2. Check native tools (hard gate)

Delegate to `/portaljs-add-geo` §2 — detect `ogr2ogr`, `tippecanoe`, `duckdb` (+ `spatial`), and additionally `jq`. Any missing → print the per-OS install and stop with `ERROR: [arcgis-to-portaljs] MISSING_TOOLS <tools>`:

for t in ogr2ogr tippecanoe duckdb jq node; do
  command -v "$t" >/dev/null 2>&1 || MISSING="$MISSING $t"
done
# macOS:  brew install gdal tippecanoe duckdb jq node
# Debian: sudo apt-get install -y gdal-bin duckdb jq nodejs   # tippecan