/azure-databricks

name: azure-databricks
description: Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Spark clusters, Unity Catalog, Delta Lake, Lakeflow/Lakebase pipelines, or Model Serving, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory).
compatibility: Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata:
  generated_at: "2026-11-08"
  generator: "docs2skills/1.0.0"

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

> **IMPORTANT for Agent**: Use the **Category Index** below to locate relevant sections. For categories with line ranges (e.g., `L35-L120`), use `read_file` with the specified lines. For categories with file links (e.g., `[security.md](security.md)`), use `read_file` on the linked reference file

> **IMPORTANT for Agent**: If `metadata.generated_at` is more than 3 months old, suggest the user pull the latest version from the repository. If `mcp_microsoftdocs` tools are not available, suggest the user install it: [Installation Guide](https://github.com/MicrosoftDocs/mcp/blob/main/README.md)

This skill requires **network access** to fetch documentation content:

• **Preferred**: Use `mcp_microsoftdocs:microsoft_docs_fetch` with query string `from=learn-agent-skill`. Returns Markdown.
• **Fallback**: Use `fetch_webpage` with query string `from=learn-agent-skill&accept=text/markdown`. Returns Markdown.

Category Index

| Category | Location | Description | |----------|----------|-------------| | Troubleshooting | L37-L198 | Diagnosing and fixing Azure Databricks issues: cluster/jobs, Spark/SQL errors, agents, Auto Loader/Lakeflow pipelines, connectors (SaaS/DB), audit logs, Git/Power BI, Feature Store, Model Serving. | | Best Practices | L199-L427 | Best-practice patterns for Databricks: cost, performance, reliability, governance, security, networking, streaming, Lakeflow, RAG/AI Search, ML/LLM, Model Serving, Unity Catalog, and BI. | | Decision Making | [decision-making.md](decision-making.md) | Guides for making architecture and migration decisions in Azure Databricks: choosing runtimes, compute, AI/ML options, connectors, Lakeflow/Lakehouse patterns, costs, and upgrade paths. | | Architecture & Design Patterns | [architecture-patterns.md](architecture-patterns.md) | Architectural blueprints and design patterns for Databricks/Lakebase: data modeling, CDC/ETL, streaming, governance, HA/DR, networking, security, feature stores, and multi-agent/AI apps. | | Limits & Quotas | [limits-quotas.md](limits-quotas.md) | Limits, quotas, and constraints for Databricks compute, AI/agents, Lakeflow pipelines/connectors, model serving, Lakebase, Unity Catalog, and workspace/SQL features. | | Security | [security.md](security.md) | Identity, access control, encryption, networking, compliance, and secrets for securing Azure Databricks, Unity Catalog, AI/agents, Lakeflow, Lakebase, and external connectors. | | Configuration | [configuration.md](configuration.md) | Configuring Azure Databricks and Lakebase: account/workspace/network settings, security/compliance, cost controls, AI/ML/agents, Lakeflow pipelines, connectors, CLI/bundles, and SQL/storage options. | | Integrations & Coding Patterns | [integrations.md](integrations.md) | Patterns and how‑tos for connecting Databricks to external systems (SQL/NoSQL, cloud storage, BI, AI/LLM tools), building agents and Lakeflow pipelines, and using SDKs/CLIs for programmatic integration. | | Deployment | [deployment.md](deployment.md) | Deploying and automating Azure Databricks workspaces, apps, agents, ML models, Lakeflow/Lakebase pipelines, and GPU workloads using IaC (ARM, Bicep, Terraform), Bundles, CI/CD, and various CLIs/UIs. |

Troubleshooting

| Topic | URL | |-------|-----| | Reference Azure Databricks diagnostic audit log events | https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/audit-logs | | Handle long-running tasks with Supervisor Agent continuation | https://learn.microsoft.com/en-us/azure/databricks/agents/agent-bricks/multi-agent-supervisor-long-running-tasks | | Agent Bricks FAQ for building and running agents | https://learn.microsoft.com/en-us/azure/databricks/agents/concepts/faq | | Debug custom code agents on Azure Databricks | https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/debug-agent | | Monitor Genie Agents activity with audit logs | https://learn.microsoft.com/en-us/azure/databricks/ai-bi/admin/audit | | Troubleshoot Azure Databricks Artifact Registry issues | https://learn.microsoft.com/en-us/azure/databricks/artifact-registry/troubleshooting | | Troubleshoot Azure Databricks classic compute termination errors | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes | | Debug Spark applications using Databricks Spark UI | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui | | Monitor Databricks dashboard usage with audit logs | https://learn.microsoft.com/en-us/azure/databricks/dashboards/monitor-usage | | Troubleshoot common Databricks CLI issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting | | Use Databricks app details for monitoring and troubleshooting | https://learn