/databricks-migration

name: databricks-migration
description: "Ports existing Databricks notebooks and jobs to Fabric, covering dbutils to notebookutils, secret scopes to Key Vault, DBFS to OneLake, Unity Catalog mapping to schema-enabled Lakehouses, Jobs and Delta Live Tables to Spark Job Definitions and Pipelines, and Photon to the Native Execution Engine. Use whenever Databricks code has to be converted. For Fabric notebook code that is not being migrated, use spark-cli."

> **Telemetry — MANDATORY.** Every `api.fabric.microsoft.com` call must carry > `x-ms-fabric-skill: databricks-migration` (`az rest`: `--headers "x-ms-fabric-skill=databricks-migration"`), > including every LRO poll, `fabric_lro` and retry. Snippets omit it — add it anyway.

> **CRITICAL NOTES** > 1. To find workspace details (including its ID) from a workspace name: list all workspaces, then use JMESPath filtering > 2. To find item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace, then use JMESPath filtering > 3. `dbutils.widgets` has **no direct equivalent** in Fabric — use notebook parameters (cell tag `parameters`); `notebookutils.runtime.context` is execution metadata, not parameter storage. If showing context fields, use documented names such as `currentWorkspaceId`, `currentWorkspaceName`, `currentNotebookId`, `currentNotebookName`, `isForPipeline`, and `isForInteractive`; `activityId` is the Livy job ID > 4. `dbutils.library` (runtime library install) has **no equivalent** — use Fabric Environments for reproducible library management > 5. Map each Unity Catalog catalog to a schema-enabled Lakehouse by default. This preserves the source `schema.table` hierarchy, with the Lakehouse representing the catalog; collisions arise only if multiple catalogs are intentionally consolidated into one Lakehouse > 6. For an under-specified workspace-wide migration, ask focused questions about inventory, workload topology, security, data locations, and runtime constraints before recommending a Fabric topology > 7. A completed Fabric migration must not retain executable `dbutils.*` calls in dual-runtime branches or `try/except` guards — replace the calls and Databricks paths outright

Databricks → Microsoft Fabric Migration

Prerequisite Knowledge

Read these companion documents before executing migration tasks:

• [COMMON-CORE.md](../../common/COMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery
• [COMMON-CLI.md](../../common/COMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI
• [SPARK-AUTHORING-CORE.md](../../common/SPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, Spark job execution

For notebook and Lakehouse creation, see [spark-cli](../spark-cli/SKILL.md). For Fabric Warehouse DDL/DML authoring, see [sqldw-cli](../sqldw-cli/SKILL.md).

---

Table of Contents

| Topic | Reference | |---|---| | **Migration Orchestrator** | [migration-orchestrator.md](resources/migration-orchestrator.md) | | Migration Workload Map | [§ Migration Workload Map](#migration-workload-map) | | Complete `dbutils` → `notebookutils` Mapping | [dbutils-to-notebookutils.md](resources/dbutils-to-notebookutils.md) | | Unity Catalog → Fabric Lakehouse Schemas | [catalog-migration.md](resources/catalog-migration.md) | | Before/After Code Patterns | [code-patterns.md](resources/code-patterns.md) | | Cluster Config → Fabric Spark Pools | [§ Cluster Config → Fabric Spark Pools](#cluster-config--fabric-spark-pools) | | Databricks Jobs → Spark Job Definitions | [§ Databricks Jobs → Spark Job Definitions](#databricks-jobs--spark-job-definitions) | | Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts | [§ Delta Sharing → Fabric External Data Sharing and OneLake Shortcuts](#delta-sharing--fabric-external-data-sharing-and-onelake-shortcuts) | | MLflow → Fabric ML Experiments | [§ MLflow → Fabric ML Experiments](#mlflow--fabric-ml-experiments) | | Post-Migration Validation & Testing | [validation-testing.md](resources/validation-testing.md) | | Migration Gotchas & Troubleshooting | [migration-gotchas.md](resources/migration-gotchas.md) | | Multi-Notebook Migration Protocol | [§ Multi-Notebook Migration Protocol](#multi-notebook-migration-protocol) | | Failure Reporting | [§ Failure Reporting](#failure-reporting) | | Must / Prefer / Avoid | [§ Must / Prefer / Avoid](#must--prefer--avoid) | | Authentication & Token Acquisition | [COMMON-CORE.md § Authentication](../../common/COMMON-CORE.md#authentication--token-acquisition) | | Lakehouse Management | [SPARK-AUTHORING-CORE.md § Lakehouse Management](../../common/SPARK-AUTHORING-CORE.md#lakehouse-management) | | Notebook Management | [SPARK-AUTHORING-CORE.md § Notebook Management](../../common/SPARK-AUTHORING-CORE.md#notebook-management) |

Context Loading Guide

> **IMPORTANT — Load only what you need.** Do NOT read all resource files upfront. Load the specific file for the phase you are executing:

| When | Read This File | |---|---| | User asks to migrate a workspace (full orchestration) | [migration-orchestrator.md](resources/migration-orchestrator.md) | | Applying code transforms (dbutils, namespaces, paths) | [dbutils-to-notebookutils.md](resources/dbutils-to-notebookutils.md) + [code-patterns.md](resources/code-patterns.md) | | Resolving Unity Catalog namespace collisions | [catalog-migration.md](resources/catalog-migration.md) | | Post-migration verification | [validation-testing.md](resources/validation-testing.md) | | Troubleshooting failures or known issues | [migration-gotchas.md](resources/migration-gotchas.md) |

---

Migration Workload Map

| Databricks Component | Fabric Target | Severity | Notes | |---|---|---|---| | **All-purpose cluster** (notebooks, REPL) | Fabric Notebook (Starter Pool or Custom Pool) | Info | No persistent cluster — Fabric provisions compute on session start | | **Job cluster** (automated jobs) | **Spark Job Def