/databricks-zerobus-ingest

name: databricks-zerobus-ingest
description: "Build Zerobus Ingest clients for near real-time data ingestion into Databricks Delta tables via gRPC. Use when creating producers that write directly to Unity Catalog tables without a message bus, working with the Zerobus Ingest SDK in Python/Java/Go/TypeScript/Rust, generating Protobuf schemas from UC tables, or implementing stream-based ingestion with ACK handling and retry logic."
compatibility: Requires databricks CLI (>= v1.0.0)
metadata:
  version: "0.1.0"

Zerobus Ingest

Build clients that ingest data directly into Databricks Delta tables via the Zerobus gRPC API.

**Status:** Generally Available. Charges are billed against the Jobs Serverless SKU. Check the [Zerobus overview](https://docs.databricks.com/ingestion/zerobus-overview) for the current status of specific features (some, such as Streaming-table targets and Arrow Flight, may be in Beta).

**Documentation:**

• [Zerobus Overview](https://docs.databricks.com/ingestion/zerobus-overview)
• [Zerobus Ingest SDK](https://docs.databricks.com/ingestion/zerobus-ingest)
• [Zerobus Limits](https://docs.databricks.com/ingestion/zerobus-limits)

---

What Is Zerobus Ingest?

Zerobus Ingest is a serverless connector that enables direct, record-by-record data ingestion into Delta tables via gRPC. It eliminates the need for message bus infrastructure (Kafka, Kinesis, Event Hub) for lakehouse-bound data. The service validates schemas, materializes data to target tables, and sends durability acknowledgments back to the client.

**Core pattern:** SDK init -> create stream -> ingest records -> handle ACKs -> flush -> close

---

Quick Decision: What Are You Building?

| Scenario | Language | Serialization | Reference | |----------|----------|---------------|-----------| | Quick prototype / test harness | Python | JSON | [references/2-python-client.md](references/2-python-client.md) | | Production Python producer | Python | Protobuf | [references/2-python-client.md](references/2-python-client.md) + [references/4-protobuf-schema.md](references/4-protobuf-schema.md) | | JVM microservice | Java | Protobuf | [references/3-multilanguage-clients.md](references/3-multilanguage-clients.md) | | Go service | Go | JSON or Protobuf | [references/3-multilanguage-clients.md](references/3-multilanguage-clients.md) | | Node.js / TypeScript app | TypeScript | JSON | [references/3-multilanguage-clients.md](references/3-multilanguage-clients.md) | | High-performance system service | Rust | JSON or Protobuf | [references/3-multilanguage-clients.md](references/3-multilanguage-clients.md) | | Schema generation from UC table | Any | Protobuf | [references/4-protobuf-schema.md](references/4-protobuf-schema.md) | | Retry / reconnection logic | Any | Any | [references/5-operations-and-limits.md](references/5-operations-and-limits.md) |

If not specified, default to python.

---

Common Libraries

These libraries are essential for Zerobus data ingestion and are typically NOT pre-installed on Databricks:

• **databricks-zerobus-ingest-sdk**: Zerobus SDK for high-performance streaming ingestion
• **databricks-sdk**: Databricks workspace client for authentication and metadata
• **grpcio-tools**: only needed to compile a `.proto` for Protobuf serialization

Install them through the **job/cluster library configuration** (see [Installing Libraries](#installing-libraries) below) rather than pip-installing at runtime — the SDK cannot pip-install on serverless compute.

grpcio-tools and protobuf compatibility

`grpcio-tools` must match the runtime's `protobuf` version. If proto compilation fails with a version error, pin a compatible build (for older `protobuf` 5.26/5.29 runtimes, `grpcio-tools==1.62.0` works); otherwise use the latest release.

---

Prerequisites

You must never execute the skill without confirming the below objects are valid:

1. **A Unity Catalog managed Delta table** to ingest into 2. **A service principal id and secret** with `MODIFY` and `SELECT` on the target table 3. **The Zerobus server endpoint** for your workspace region 4. **The Zerobus Ingest SDK** installed for your target language

See [references/1-setup-and-authentication.md](references/1-setup-and-authentication.md) for complete setup instructions.

---

Minimal Python Example (JSON)

from zerobus.sdk.sync import ZerobusSdk
from zerobus.sdk.shared import RecordType, StreamConfigurationOptions, TableProperties

sdk = ZerobusSdk(server_endpoint, workspace_url)
options = StreamConfigurationOptions(record_type=RecordType.JSON)
table_props = TableProperties(table_name)

stream = sdk.create_stream(client_id, client_secret, table_props, options)
try:
    # Pass a dict for JSON streams; the SDK serializes it.
    record = {"device_name": "sensor-1", "temp": 22, "humidity": 55}
    offset = stream.ingest_record_offset(record)
    stream.wait_for_offset(offset)  # Block until durably written
finally:
    stream.close()

---

Detailed guides

| Topic | File | When to Read | |-------|------|--------------| | Setup & Auth | [references/1-setup-and-authentication.md](references/1-setup-and-authentication.md) | Endpoint formats, service principals, SDK install | | Python Client | [references/2-python-client.md](references/2-python-client.md) | Sync/async Python, JSON and Protobuf flows, reusable client class | | Multi-Language | [references/3-multilanguage-clients.md](references/3-multilanguage-clients.md) | Java, Go, TypeScript, Rust SDK examples | | Protobuf Schema | [references/4-protobuf-schema.md](references/4-protobuf-schema.md) | Generate .proto from UC table, compile, type mappings | | Operations & Limits | [references/5-operations-and-limits.md](references/5-operations-and-limits.md) | ACK handling, retries, reconnection, throughput limits, constraints |

---

You must always follow all the steps in the Workflow

Workflow

0. **Display the plan of your execution** 1. **Determine the type of client** and se