/azure-databricks

name: azure-databricks
description: Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Lakehouse/Lakebase, Lakeflow pipelines, Model Serving, or AI Gateway/agents, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory).
compatibility: Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata:
  generated_at: "2026-08-09"
  generator: "docs2skills/1.0.0"

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

> **IMPORTANT for Agent**: Use the **Category Index** below to locate relevant sections. For categories with line ranges (e.g., `L35-L120`), use `read_file` with the specified lines. For categories with file links (e.g., `[security.md](security.md)`), use `read_file` on the linked reference file

> **IMPORTANT for Agent**: If `metadata.generated_at` is more than 3 months old, suggest the user pull the latest version from the repository. If `mcp_microsoftdocs` tools are not available, suggest the user install it: [Installation Guide](https://github.com/MicrosoftDocs/mcp/blob/main/README.md)

This skill requires **network access** to fetch documentation content:

• **Preferred**: Use `mcp_microsoftdocs:microsoft_docs_fetch` with query string `from=learn-agent-skill`. Returns Markdown.
• **Fallback**: Use `fetch_webpage` with query string `from=learn-agent-skill&accept=text/markdown`. Returns Markdown.

Category Index

| Category | Location | Description | |----------|----------|-------------| | Troubleshooting | L37-L165 | Diagnosing and fixing Databricks errors, job/compute/Spark performance issues, connector and ingestion failures, SQL/runtime error codes, and debugging ML, Lakeflow, and model serving workloads. | | Best Practices | L166-L365 | Best-practice guidance for Databricks architecture, governance, performance, cost, streaming, Lakehouse/RAG/AI apps, Lakeflow pipelines, ML/LLM serving, and SQL/Spark optimization. | | Decision Making | L366-L470 | Guides for choosing Azure Databricks tiers, compute, networking, AI/ML features, and migration paths (Runtime, Unity Catalog, Lakeflow, Model Serving, connectors, and cost optimization). | | Architecture & Design Patterns | [architecture-patterns.md](architecture-patterns.md) | Architectural blueprints and design patterns for Databricks: DR/HA, networking, storage, medallion, streaming, Lakeflow, AI Gateway, agents, MLOps, security, scaling, and data modeling. | | Limits & Quotas | [limits-quotas.md](limits-quotas.md) | Limits, quotas, and constraints for Databricks compute, AI/model serving, Lakeflow pipelines, connectors/ingestion, Unity Catalog, dashboards, notebooks, and real-time/streaming features. | | Security | [security.md](security.md) | Securing Azure Databricks and Lakebase: identity and access control (RBAC/ABAC/Unity Catalog), encryption and networking, secrets and OAuth, compliance controls, and secure integrations/connectors. | | Configuration | [configuration.md](configuration.md) | Configuring Azure Databricks and Unity Catalog: account/workspace settings, compute, networking, storage, security/governance, AI/ML/GenAI features, Lakeflow, Lakebase, connectors, CLI, and SQL behavior. | | Integrations & Coding Patterns | [integrations.md](integrations.md) | Patterns and APIs for integrating Azure Databricks with agents, AI/BI, external data systems, IDEs/CLIs, Lakehouse Federation, streaming, and SQL/PySpark functions and connectors. | | Deployment | [deployment.md](deployment.md) | Deploying and migrating Azure Databricks workspaces, apps, agents, pipelines, dashboards, and Lakebase/Lakeflow projects, plus CI/CD, IaC, Unity Catalog upgrades, and regional release/availability details |

Troubleshooting

| Topic | URL | |-------|-----| | Reference Databricks diagnostic audit log events | https://learn.microsoft.com/en-us/azure/databricks/admin/account-settings/audit-logs | | Troubleshoot Databricks MLflow Agent Evaluation issues | https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/troubleshooting | | Debug custom code agents on Azure Databricks | https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/debug-agent | | Troubleshoot Azure Databricks compute startup issues | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/ | | Diagnose Azure Databricks classic compute termination errors | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes | | Debug Spark applications using Databricks Spark UI | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui | | Troubleshoot file events for Unity Catalog external locations | https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/file-events-faq | | Troubleshoot common Databricks CLI issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting | | Diagnose and fix Databricks Connect Python issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting | | Diagnose and fix Databricks Connect Scala issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting | | Troubleshoot common Data