/dkg-importer

name: dkg-importer
description: Bulk-import a large RDF graph (code graph, corpus, GitHub history, etc.) into a DKG node's working memory. Use this skill when you need to push more than a few thousand triples in a single import — it codifies the chunking budgets, the assertion-loop shape, the resumability manifest, and the canonical URI rules so your importer converges with every other importer in the workspace.

DKG Importer Skill

This skill is the **agent-readable manual for bulk imports** against a DKG V10 node. If you are about to write more than a few thousand triples in one logical operation — a code graph, a Markdown corpus, a GitHub issue archive, a domain-specific dataset — read this first. It documents the contract every existing in-tree importer follows, so the graphs you produce join naturally with graphs other agents and the scanners produce.

For the general node API surface (auth, contextGraphs, SWM/VM publish, SPARQL) see [`packages/cli/skills/dkg-node/SKILL.md`](../dkg-node/SKILL.md). This skill sits one layer above: it assumes you already know how to call `dkg_knowledge_asset_*` and focuses on **how to call them at scale, repeatedly, without losing data on restart and without fragmenting the graph against parallel producers**.

1. The chunking contract (read first)

The daemon's `/api/knowledge-assets` create + `/api/knowledge-assets/<name>/{wm/write,swm/share}` loop **is** the chunked-write API. There is no `/api/import/bulk` and there will not be one (see [ADR 0002](../../../../docs/adr/0002-importer-chunking-contract.md) for the rejected-alternative analysis). To push a large graph you call the loop many times, with each call staying under fixed budgets.

| Constant | Value | Where it lands | |---|---|---| | `CHUNK` | **5,000 quads** | Per `POST /api/knowledge-assets/<name>/wm/write` call | | `ROOT_CHUNK` | **not used** | `POST /api/knowledge-assets/<name>/swm/share` takes no root list: a share always covers the whole Knowledge Asset. Size each Knowledge Asset under the 4 MiB share cap instead (§1.1) | | Max concurrent writes within one assertion | **1** (sequential) | The daemon does not parallelise intra-assertion writes; the manifest in §3 tracks per-assertion state anyway | | Max concurrent assertions | **4** | Safe across assertions; keeps memory bounded for laptop-class nodes |

These constants are conservative: a 5,000-quad N-Quads payload serialises at roughly 1.0-1.5 MB, well under the daemon's 10 MB `MAX_BODY_BYTES` cap. Going larger gives no throughput win and risks a 413 on URIs that serialise on the heavy end.

1.1 Known daemon caps and the exact error strings they produce

These are the three hard caps on the loop, with the verbatim error text the daemon emits. The HTTP body caps live in [`packages/cli/src/daemon/http-utils.ts`](../../../../packages/cli/src/daemon/http-utils.ts); the gossip cap lives in [`packages/core/src/constants.ts`](../../../../packages/core/src/constants.ts).

| Endpoint | Cap | Constant | Trigger | Error response | |---|---|---|---|---| | `POST /api/knowledge-assets/<name>/wm/write` | **10 MB** request body | `MAX_BODY_BYTES` | N-Quads payload too large | `HTTP 413` "Request body too large (>10485760 bytes)" | | `POST /api/knowledge-assets/<name>/swm/share` | **256 KB** request body | `SMALL_BODY_BYTES` | Not reached in practice: the body is only `{ contextGraphId, subGraphName? }` | `HTTP 413` "Request body too large (>262144 bytes)" | | `POST /api/knowledge-assets/<name>/swm/share` | **4 MiB** gossip message | `DKG_GOSSIP_MAX_MESSAGE_BYTES` | The Knowledge Asset's encoded payload exceeds 4 MiB | `HTTP 413`, `code: "SWM_GOSSIP_PAYLOAD_TOO_LARGE"`: "Promoted assertion too large for gossip (XXXX KB, limit 4 MB). Reduce the complete assertion payload size." |

Shares are **atomic**: a share always sends the whole Knowledge Asset, and an `entities` array is refused with `400 KA_ATOMIC_SHARE_REQUIRED` (`"entities": "all"` is still accepted and changes nothing). So the 4 MiB gossip cap is a cap on the size of one Knowledge Asset (triples × N-Quads length). For a Knowledge Asset above about 12,000 typical-sized triples, expect to hit it. Size your partitions so each Knowledge Asset stays under it, and split a partition into more Knowledge Assets when it does not. See [§5 Error handling](#5-error-handling) for the recovery.

**Self-tune from `/api/status`.** Future versions of the daemon advertise their current per-call limits at `/api/status` under an `importLimits` block. If present, use those values — they reflect any operator-side tuning. If absent (older daemon), use the constants above verbatim.

2. The write loop

For each logical slice of triples (one slice ≈ one source artefact: one file, one PR, one document, one record group):

POST /api/knowledge-assets   { name, subGraphName, contextGraphId }
POST /api/knowledge-assets/<name>/wm/write   { quads: [...] }   ── one or more times
POST /api/knowledge-assets/<name>/swm/share   { contextGraphId, subGraphName? }   ── once, for the whole KA

Reference implementation — see [`scripts/lib/dkg-daemon.mjs`](../../../../scripts/lib/dkg-daemon.mjs) for `DkgClient`. `writeAssertion` auto-chunks at a conservative 500-triple default (override via the second-argument `batchSize`); `promote` shares the whole Knowledge Asset, so call it once per Knowledge Asset and pass no `entities` array.

TypeScript sketch

import { DkgClient } from './scripts/lib/dkg-daemon.mjs';

const client = new DkgClient({ token: process.env.DKG_TOKEN });
await client.ensureProject({ id: 'my-corpus', name: 'My Corpus' });
await client.ensureSubGraph(client.cgId, 'code');

async function ensureAssertion(client, body) {
  try {
    await client.request('POST', '/api/knowledge-assets', body);
  } catch (err) {
    if (err.status === 400 && /already exists/i.test(JSON.stringify(err.body ?? err.message))) {
      return;
    }
    throw err;
  }
}

for (const partition of partitions) {