/databricks-lakeflow-connect

name: databricks-lakeflow-connect
description: "Build managed ingestion pipelines into Databricks using Lakeflow Connect. Use when ingesting from SaaS apps (Salesforce, Workday Reports, ServiceNow, Google Analytics 4, HubSpot, Confluence) or databases (SQL Server cloud and on-prem; PostgreSQL/MySQL CDC in PuPr) into Unity Catalog with serverless pipelines."
compatibility: Requires databricks CLI (>= v0.294.0)
metadata:
  version: "0.1.0"
parent: databricks-core

Lakeflow Connect

Build managed ingestion pipelines that pull from SaaS apps and databases into Unity Catalog Delta tables, governed end-to-end and powered by serverless Lakeflow Spark Declarative Pipelines (formerly Delta Live Tables / DLT).

**Status:** mixed catalog — GA connectors for production use, plus Public Preview, Beta, and Private Preview connectors that expand over time. See the connector catalog below.

---

What Is Lakeflow Connect?

Managed connectors for ingesting data from SaaS applications and databases. The resulting ingestion pipeline is governed by Unity Catalog and powered by serverless compute and Lakeflow Spark Declarative Pipelines.

Three frames to keep in mind:

• **Simple and low-maintenance** — no client code to write, no message bus to operate; connector + UC Connection + a serverless pipeline.
• **Unified with the lakehouse** — credentials stored in UC, output is governed Delta, runs on Jobs and SDP like any other workload.
• **Efficient incremental processing** — change tracking / CDC / schema evolution / retries are built in.

There are four architecture patterns:

1. **SaaS pull** — connector reads from an external SaaS via OAuth or API key, lands in a streaming Delta table. 2. **Database CDC via gateway** — an ingestion gateway runs in the customer's network, stages change events to a UC Volume, a serverless ingestion pipeline applies them as CDC into Delta. 3. **Query-based** — for sources without native CDC (Oracle / Teradata / SQL Server / PG / MySQL query-based, Snowflake / Redshift / Synapse / BigQuery via Foreign Catalog), the connector issues periodic queries instead of subscribing to a change feed. 4. **Community connectors** — template-based, out of scope for this skill.

---

Is Lakeflow Connect the right tool?

Decide this before you build. Lakeflow Connect is the managed **pull** path for SaaS apps and databases — it is not the answer for every ingestion intent.

| If the source is... | Use | Skill | |---|---|---| | A SaaS app or database with a managed connector (Salesforce, Workday, ServiceNow, GA4, HubSpot, Confluence, SQL Server, ...) | **Lakeflow Connect** | this skill | | Files on cloud object storage (S3 / ADLS / GCS) | Auto Loader | **databricks-pipelines** | | A source you want to query in place, no copy | Lakehouse Federation | — | | An app or device that **pushes** events at you | Zerobus | **databricks-zerobus-ingest** | | A partner offering a Delta share | Delta Sharing | — |

Full reasoning, including the Federation-vs-Connect and Auto-Loader-vs-Connect trade-offs, is in [4-ingestion-decision-tree.md](references/4-ingestion-decision-tree.md).

---

Connector catalog

Lakeflow Connect ships connectors at multiple release stages. **GA** and **Public Preview** connectors are production-supported; **Beta** and **Private Preview** are early-access and not production-supported.

GA connectors

Full coverage in this skill.

| Source | Type | Auth | Reference | |--------|------|------|-----------| | Salesforce (Sales / Service / etc.) | SaaS pull | OAuth U2M | [1-saas-connectors.md](references/1-saas-connectors.md) | | Workday Reports (RaaS) | SaaS pull | OAuth refresh token / basic | [1-saas-connectors.md](references/1-saas-connectors.md) | | ServiceNow | SaaS pull | OAuth U2M / basic | [1-saas-connectors.md](references/1-saas-connectors.md) | | Google Analytics 4 | SaaS pull (via BigQuery) | Service-account JSON | [1-saas-connectors.md](references/1-saas-connectors.md) | | HubSpot | SaaS pull | OAuth | [1-saas-connectors.md](references/1-saas-connectors.md) | | Confluence | SaaS pull | OAuth | [1-saas-connectors.md](references/1-saas-connectors.md) | | SQL Server (cloud) | Database CDC | DB user + change tracking / CDC | [2-database-connectors.md](references/2-database-connectors.md) | | SQL Server (on-prem) | Database CDC | DB user + ExpressRoute / Direct Connect | [2-database-connectors.md](references/2-database-connectors.md) |

Public Preview connectors

Production-supported. Configuration may evolve before GA. Deep coverage is being added incrementally; until then, see the [public connector reference](https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/connectors) for current setup steps.

| Source | Type | Auth | |--------|------|------| | NetSuite | SaaS pull | OAuth | | Dynamics 365 | SaaS pull | OAuth | | PostgreSQL CDC | Database CDC | DB user + gateway | | MySQL CDC | Database CDC | DB user + gateway | | Oracle / Teradata / SQL Server / PG / MySQL (query-based) | Database query | DB user | | Snowflake / Redshift / Synapse / BigQuery (Foreign Catalog) | Database query | Foreign Catalog | | SFTP | File pull | Key / password |

Beta and Private Preview

Early-access connectors are not production-supported. The list changes month to month; check the [public connector reference](https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/connectors) for current availability.

For the Lakeflow-Connect-vs-Auto-Loader-vs-Federation-vs-Delta-Sharing decision, see [4-ingestion-decision-tree.md](references/4-ingestion-decision-tree.md).

---

Required Tools

• **Databricks CLI v0.294.0+** for `databricks pipelines create` and `databricks connections create`. Verify with `databricks --version`.
• **Databricks SDK for Python** (`databricks-sdk>=0.85.0`) if you prefer SDK over CLI.
• **Declarative Automation Bundles** if authoring as IaC (recommended for any pipeline that ships to a customer environment).

No extra connector-specifi