/portaljs-migrate

description: Migrate (harvest) datasets between open-data platforms. Reads CKAN, a DCAT-US /data.json catalog (DKAN, ArcGIS Hub, data.gov), a DCAT / DCAT-AP RDF feed (JSON-LD, Turtle, or RDF/XML — data.europa.eu, national DCAT-AP portals, GeoDCAT-AP), Socrata, OpenDataSoft, or an ArcGIS FeatureServer, and writes them to a static PortalJS catalog (datasets.json, link-by-URL or download data files into Cloudflare R2 via Git LFS / Giftless) or pushes them into a CKAN instance over its API.
allowed-tools: Read, Write, Edit, Bash, WebFetch

/portaljs-migrate

Harvest datasets from an external open-data platform into an existing `portaljs-catalog` portal. The source's datasets are read over its API, mapped to the portal's canonical dataset shape, and written into `datasets.json` (the static catalog's single source of truth) — so the `/search` catalog and the `/@<namespace>/<slug>` showcases render them like any hand-added dataset.

This is the **copy-into-the-portal** path. It is the inverse of [`/portaljs-connect-ckan`](connect-ckan.md): connect-ckan keeps the source authoritative and reads it live at build time; `/portaljs-migrate` takes a one-time (re-runnable) snapshot into the static catalog, so the portal stands alone and needs no backend.

Hub-and-spoke model

Every source is read into one **canonical** shape (the template's `Dataset`/`Resource` type — a Frictionless-aligned `{ slug, namespace, name, description, resources[] }`), then written to the target from that canonical form. Add a source once and it migrates to every target.

**Sources:**

| Source | `--source` | How it's read | Covers | | ------ | ---------- | ------------- | ------ | | **CKAN** | `ckan` | REST API (`package_search` / `package_show`) | any CKAN instance | | **DCAT-US `/data.json`** | `dcat` | one catalog document (plain JSON) | **DKAN, ArcGIS Hub, data.gov**, other DCAT-US publishers | | **DCAT / DCAT-AP RDF feed** | `dcat-rdf` | RDF catalog in JSON-LD, Turtle, or RDF/XML | **data.europa.eu**, national **DCAT-AP** portals (SE/CH/DE), GeoDCAT-AP, any DCAT 2/3 RDF feed | | **Socrata** | `socrata` | Discovery API + per-dataset resource exports | Socrata-powered open-data sites | | **OpenDataSoft** | `ods` | Explore API v2 catalog + exports | ODS-powered portals | | **ArcGIS FeatureServer / MapServer** | `arcgis` | layer metadata + GeoJSON query | individual ArcGIS map/feature services |

> DKAN, ArcGIS Hub, and data.gov publish a DCAT-US `/data.json` — use the **dcat** source for > those whole catalogs. Use **arcgis** for an individual FeatureServer/MapServer (each layer > becomes a GeoJSON dataset, which `/data.json` doesn't expose). > > **`dcat` vs `dcat-rdf`.** `dcat` reads the flat DCAT-US **`/data.json`** (Project Open Data > JSON — `dataset[]` with `distribution[]`). `dcat-rdf` reads a **DCAT / DCAT-AP RDF graph** > serialized as JSON-LD (`.jsonld`), Turtle (`.ttl`), or RDF/XML (`.rdf`) — the form > data.europa.eu and national DCAT-AP portals publish. This is the **inbound** counterpart of > [`/portaljs-add-dcat`](portaljs-add-dcat.md) (which EXPOSES the portal as DCAT-AP RDF): the two > make a portal a full **two-way DCAT interop node** — expose to, and harvest from, national > DCAT-AP portals. Both read through the SAME profile registry (`lib/metadata/dcat-profiles.ts`), > so expose and consume stay in sync.

**Targets:**

| Target | `--target` | Writes | | ------ | ---------- | ------ | | **Static PortalJS catalog** (default) | `static` | `datasets.json` in a `portaljs-catalog` portal (+ in `download` mode, data files pushed to Cloudflare R2 via Git LFS / Giftless) | | **CKAN instance** | `ckan` | datasets/resources into a CKAN backend via `package_create` / `resource_create` (needs a write API key) |

The CKAN target enables platform-to-platform moves — **CKAN→CKAN** and **DKAN→CKAN** — since any reader can feed any writer through the canonical shape.

Required input — ask, don't error

**Source:**

• **Source type** — `ckan`, `dcat`, `dcat-rdf`, `socrata`, `ods`, or `arcgis` (auto-detected

from the URL if omitted; see step 3).

• **Source URL** (required) — e.g. a CKAN base URL, a DCAT-US `/data.json` URL, a DCAT-AP RDF

feed URL (`…/catalog.jsonld` / `.ttl` / `.rdf`, or a portal page that autodiscovers one via `<link rel="alternate">`), a Socrata or OpenDataSoft site root, or an ArcGIS `…/FeatureServer` (or `…/MapServer`) URL.

• **Filters** (optional) — CKAN: org / group names. Socrata/ODS: pass a search term or

category to scope large catalogs.

**Target** — `--target static` (default) or `--target ckan`:

• **static**: **Portal directory** (optional, default current dir); **copy mode** `link`

(default) or `download` (step 5b).

• **ckan**: **target CKAN URL** (required) and a **write API key** read from the

`CKAN_API_KEY` env var (required — never pass it on the command line or hardcode it); an optional **owner org** to file every dataset under (step 7b).

**Common:**

• **`--dry-run`** (optional) — preview what would be written, change nothing.
• **`--replace`** (optional, static target) — clear existing `datasets.json` entries first

(default: upsert alongside what's already there, e.g. the sample datasets).

**If the source URL is missing, ask for it (and the source type if unclear) — never dead-end with a missing-input error.** For `--target ckan`, if the target URL or `CKAN_API_KEY` is missing, ask rather than failing.

Steps

1. Gather input from `$ARGUMENTS` (interview if thin)

Extract:

• `SOURCE_TYPE` — `ckan` | `dcat` | `dcat-rdf` | `socrata` | `ods` | `arcgis` (default: auto-detect in step 3).
• `SOURCE_URL` — required; strip any trailing slash.
• `ORG_FILTER` / `GROUP_FILTER` — lists (CKAN source only; default empty).
• `TARGET` — `static` | `ckan` (default `static`).
• `PORTAL_DIR` — default `.` (static target).
• `COPY_MODE` — `link` | `download` (default `link`; static target).
• `TARGET_CKAN_URL` — required for `ckan` target; strip any trailing slash.