/databricks-data-discovery

name: databricks-data-discovery
description: "Discover, explore, and query Databricks data via Genie — the CLI equivalent of the Genie One MCP. MUST be invoked whenever the user asks to find or locate data ('what tables are in X', 'where does X live', 'which catalog/schema has Y'), answer a natural-language question about the data, or write a SQL query."
compatibility: Requires databricks CLI >= v1.9.0 for `databricks genie ask` (older CLIs fall back to the deprecated `databricks experimental genie ask` alias)
metadata:
  version: "0.1.0"
parent: databricks-core

Databricks Data Discovery

This skill **routes data work** — decide first:

• the request is about *the data* — finding it, asking questions of it, or

generating SQL → delegate to **Genie One**: `databricks genie ask -s <session-label> "..."` (see Routing below).

• writing files or anything else → use your own coding-agent tools.

Genie One just needs an authenticated CLI profile (the parent `databricks-core` skill covers auth/profiles if you need it) — but route the data request to Genie first; don't detour into manual catalog browsing.

Routing — Genie vs your coding agent

**Route to Genie when the request is about the data:**

• **Finding or locating data** — "what tables are in wanderbricks?", "where does X

live?", which catalog/schema holds something. Let Genie resolve a fuzzy or partial name across all your data instead of guessing or grepping catalogs yourself.

• **Questions about the data** — "how many…", "what's the average…", "which … the

most…", trends, breakdowns.

• **Generating SQL from a question** — a first-cut query you can run, adapt, or drop

into a file.

• **Understanding or profiling data** to inform a query, dashboard, or app.

**Route to your own coding agent (do NOT use Genie) for everything else:**

• **Writing or editing files** — a `.sql` file, notebook, dashboard, app, config.

Genie finds the data and produces the SQL; *you* write the file.

• **Authoring jobs, pipelines, apps, or DABs** — use those product skills.
• **DDL or writes** — `CREATE` / `INSERT` / `UPDATE` / `DELETE`.
• **General programming, git, infra, and any non-data task.**

**Key principle:** data discovery, data questions, and query generation → Genie One. Everything else → your coding agent.

Why prefer Genie for data questions

Genie runs *inside the Databricks data plane* with governed, first-hand access to the org's Unity Catalog metadata, metric views, and curated semantic context — context you do **not** have when reverse-engineering schemas with ad-hoc SQL. For data questions it is often higher-quality and more performant than doing the discovery yourself, and it keeps improving as a managed Databricks capability. Don't default to writing your own discovery SQL just because you can.

How to ask Genie

Always pass a session label with `-s`, and prefer reusing the **same** one: a follow-up can only continue a conversation if the first ask set the session label, and reusing it lets later questions build on everything asked so far ("summarize all of the above"). Use a fresh session label only to start a deliberately separate session, or distinct session labels to run several in parallel.

The command is `databricks genie ask` (CLI >= v1.9.0). On an older CLI it lives under `databricks experimental genie ask` — same flags and behavior; use that exact fallback if `databricks genie ask` is not found.

# Always pass a session label, and reuse the SAME one so follow-ups build on each other
databricks genie ask -s trips "How many bookings were there last week?"
databricks genie ask -s trips "Break that down by destination"
databricks genie ask -s trips "Summarize all of the above"

# --include-sql also prints the SQL Genie ran (use it to generate a query, too)
databricks genie ask -s trips "Write SQL for the top 5 destinations by revenue" --include-sql

# --output json gives a parseable result
databricks genie ask -s trips "Top 5 destinations by revenue" --output json
# → {"status":"completed","conversation_id":"…","text":"…","tool_calls":[{"name":"execute_sql","sql":"…","title":"…"}]}

# Older CLI (< v1.9.0) — same command under the deprecated experimental alias:
#   databricks experimental genie ask -s trips "How many bookings were there last week?"

Genie searches across all the data you can see, runs SQL, and streams a grounded answer — rendered with the executed SQL and, where it helps, a terminal chart. It auto-resolves a SQL warehouse (override with `--warehouse-id`); nothing to pick or set up.

• **Streams live**: the answer, the agent's steps, and any SQL/results appear as

they arrive. Answers usually take ~5–30s; a stalled stream (no data for ~10 min) fails with a clear message, and Ctrl-C or `kill` (SIGTERM) cancels cleanly.

• **Picking a session label**: any string works — a topic like `trips`, or `$$` for a

per-shell session. Default to reusing one session label so follow-ups keep full context; use a fresh one only for a deliberately separate session. An expired session label just starts fresh on the next ask. No id to copy around.

• **Parallelism**: to run sessions at the same time, give each its own session label

(`-s q1`, `-s q2`, …) — independent session labels don't interfere. Within a single session label keep calls sequential: send a follow-up after the previous turn returns, and never fire two asks at once on the *same* session label (they'd split into two conversations and only one mapping would survive).

• **Structured output**: `--output json` gives `{status, conversation_id, text,

tool_calls[]}`, where `tool_calls` includes the SQL Genie executed; `--raw` dumps the raw event stream. Note `--output json` buffers and prints once at the end (no live streaming) — use it for parsing, the default text output for interactive use.

• **Generating a query**: ask Genie to "write SQL for …" and read the SQL from the

response (it's in the answer text, and `--i