/azure-databricks

name: azure-databricks
description: Expert knowledge for Azure Databricks development including troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. Use when using Unity Catalog, Delta Lake/Lakehouse, Lakeflow pipelines, Lakebase, or model/feature serving, and other Azure Databricks related development tasks. Not for Azure Synapse Analytics (use azure-synapse-analytics), Azure HDInsight (use azure-hdinsight), Azure Machine Learning (use azure-machine-learning), Azure Data Factory (use azure-data-factory).
compatibility: Requires network access. Uses mcp_microsoftdocs:microsoft_docs_fetch or fetch_webpage to retrieve documentation.
metadata:
  generated_at: "2026-09-13"
  generator: "docs2skills/1.0.0"

Azure Databricks Skill

This skill provides expert guidance for Azure Databricks. Covers troubleshooting, best practices, decision making, architecture & design patterns, limits & quotas, security, configuration, integrations & coding patterns, and deployment. It combines local quick-reference content with remote documentation fetching capabilities.

How to Use This Skill

> **IMPORTANT for Agent**: Use the **Category Index** below to locate relevant sections. For categories with line ranges (e.g., `L35-L120`), use `read_file` with the specified lines. For categories with file links (e.g., `[security.md](security.md)`), use `read_file` on the linked reference file

> **IMPORTANT for Agent**: If `metadata.generated_at` is more than 3 months old, suggest the user pull the latest version from the repository. If `mcp_microsoftdocs` tools are not available, suggest the user install it: [Installation Guide](https://github.com/MicrosoftDocs/mcp/blob/main/README.md)

This skill requires **network access** to fetch documentation content:

• **Preferred**: Use `mcp_microsoftdocs:microsoft_docs_fetch` with query string `from=learn-agent-skill`. Returns Markdown.
• **Fallback**: Use `fetch_webpage` with query string `from=learn-agent-skill&accept=text/markdown`. Returns Markdown.

Category Index

| Category | Location | Description | |----------|----------|-------------| | Troubleshooting | L37-L179 | Diagnosing and fixing Databricks issues: Spark and SQL errors, Lakeflow/Auto Loader and pipelines, connectors/ingestion, model serving, feature store, CLI/IDE, and audit/log-based troubleshooting. | | Best Practices | L180-L390 | End-to-end Databricks best practices: cost, governance, security, compute, BI, streaming, Lakehouse storage, RAG/GenAI, ML/LLMOps, Lakeflow, connectors, and performance tuning. | | Decision Making | [decision-making.md](decision-making.md) | Guides for architectural and cost decisions in Azure Databricks: choosing compute, runtimes, pipelines, AI/ML options, connectors, governance, and planning/migrating workloads and tools. | | Architecture & Design Patterns | [architecture-patterns.md](architecture-patterns.md) | Architectural blueprints and patterns for Databricks: DR/HA, networking, storage, Lakehouse/medallion, Lakeflow ETL/CDC, Lakebase, AI agents, Feature Store, MLOps, and AI governance. | | Limits & Quotas | [limits-quotas.md](limits-quotas.md) | Limits, quotas, and constraints for Databricks compute, AI/BI, connectors, Lakeflow, Lakehouse Real-Time, Unity Catalog, Lakebase, model serving, and SQL/notebook behavior. | | Security | [security.md](security.md) | Securing Azure Databricks: identity and access control (RBAC/ABAC), Unity Catalog governance, encryption and keys, network/IP controls, OAuth and token auth, compliance profiles, and secure external connections. | | Configuration | [configuration.md](configuration.md) | Configuring and managing Azure Databricks and Unity Catalog: accounts, networking, security, AI/ML, Lakeflow, SQL, storage, connectors, bundles, CLI, observability, and cost/governance settings. | | Integrations & Coding Patterns | [integrations.md](integrations.md) | Patterns and how-tos for integrating Databricks with apps, agents, external data systems, BI tools, and ML/AI workflows, plus detailed APIs, CLIs, connectors, and SQL/PySpark function references. | | Deployment | [deployment.md](deployment.md) | End-to-end deployment of Azure Databricks: workspaces, Unity Catalog, Lakeflow, Lakebase, apps/agents, model & feature serving, CI/CD, IaC (ARM/Bicep/Terraform/Bundles), and networking setup. |

Troubleshooting

| Topic | URL | |-------|-----| | Troubleshoot Databricks Agent Evaluation issues | https://learn.microsoft.com/en-us/azure/databricks/agents/agent-evaluation/troubleshooting | | Debug custom code agents on Databricks | https://learn.microsoft.com/en-us/azure/databricks/agents/custom-agents/debug-agent | | Detect and clean up unused AI Search endpoints | https://learn.microsoft.com/en-us/azure/databricks/ai-search/unused-endpoints | | Resolve Databricks classic compute termination error codes | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/cluster-error-codes | | Debug Spark applications using Databricks Spark UI | https://learn.microsoft.com/en-us/azure/databricks/compute/troubleshooting/debugging-spark-ui | | Monitor Databricks dashboard usage with audit logs | https://learn.microsoft.com/en-us/azure/databricks/dashboards/monitor-usage | | Troubleshoot common Databricks CLI issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/troubleshooting | | Use Databricks app details for monitoring and troubleshooting | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-apps/view-app-details | | Troubleshoot Databricks Connect for Python issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/python/troubleshooting | | Troubleshoot Databricks Connect for Scala issues | https://learn.microsoft.com/en-us/azure/databricks/dev-tools/databricks-connect/scala/troubleshooting | | Troubleshoot Databricks Terraform provi