/add-tracing

name: add-tracing
description: Add OpenTelemetry tracing spans to Clojure code following Metabase tracing conventions. Use when instrumenting backend code with trace coverage.

Add Tracing Spans to Clojure Code

This skill helps you add OpenTelemetry (OTel) tracing spans to the Metabase backend codebase using the custom `tracing/with-span` macro.

Reference Files

• `src/metabase/tracing/core.clj` - `with-span` macro, group registry, SDK lifecycle, `best-effort-sanitize-sql`, Pyroscope integration
• `src/metabase/task/impl.clj` - `defjob` macro that wraps Quartz jobs with root spans
• `.clj-kondo/config/modules/config.edn` - Module boundary configuration

Module Architecture

The tracing module has a deliberately minimal API surface. **Only 2 namespaces are public** (listed in `:api` in the module config):

| Namespace | Role | Status | |---|---|---| | `tracing.core` | Primary API: `with-span`, groups, SDK lifecycle, Pyroscope, MDC, `best-effort-sanitize-sql` | **Public API** | | `tracing.init` | Side-effect loader for `quartz` and `settings` | **Public API** (init convention) | | `tracing.attributes` | `best-effort-sanitize-sql` implementation (re-exported via `tracing.core`) | Internal | | `tracing.settings` | Setting definitions (`MB_TRACING_*` env vars) | Internal | | `tracing.quartz` | Quartz JDBC proxy + JobListener | Internal |

**Rules:**

• Only require `[metabase.tracing.core :as tracing]` from outside the module. `tracing/best-effort-sanitize-sql` and all other public functions are available from this single namespace.
• Do not add new API namespaces. Add new public functions to `tracing.core` instead.
• Do not require internal namespaces (`tracing.attributes`, `tracing.settings`, `tracing.quartz`) from outside the module.
• `:uses :any` on the `core` module does NOT bypass the target module's `:api` check — internal namespaces are still enforced.

Cyclic Dependency Avoidance

`tracing/core.clj` is required by many modules across the codebase. It **must NOT** compile-time require `tracing.settings`, as this creates transitive cyclic load dependencies (e.g., `settings/core -> tracing/settings -> tracing/core -> events/impl -> events/core`).

Instead, `tracing/core.clj` uses `requiring-resolve` for settings access:

;; CORRECT — lazy runtime resolution, no compile-time dependency
((requiring-resolve 'metabase.tracing.settings/tracing-enabled))

;; WRONG — creates cyclic load dependency
(require '[metabase.tracing.settings :as settings])
(settings/tracing-enabled)

External library namespaces (clj-otel API, SDK, exporters) are safe to require normally — they don't participate in Metabase namespace cycles.

**Important:** `requiring-resolve` must use **literal quoted symbols**. Kondo hooks validate that `required-namespaces` are all simple symbols, so dynamic construction fails:

;; CORRECT — literal quoted symbol
(requiring-resolve 'metabase.tracing.settings/tracing-endpoint)

;; WRONG — kondo hook rejects this: "Assert failed: (every? simple-symbol? required-namespaces)"
(requiring-resolve (symbol "metabase.tracing.settings" "tracing-endpoint"))

Quick Checklist

When adding tracing spans:

• [ ] Module has `tracing` in its `:uses` set in `.clj-kondo/config/modules/config.edn`
• [ ] Added `[metabase.tracing.core :as tracing]` to ns requires (alphabetically sorted)
• [ ] Span wraps a meaningful I/O boundary (not pure computation)
• [ ] Group matches the domain (check `src/metabase/tracing/core.clj` for registered groups; add a new one if none fit)
• [ ] Span name follows dot-notation convention (`"domain.subsystem.operation"`)
• [ ] Attributes use namespaced keywords (`:search/query-length`, `:db/id`)
• [ ] No sensitive data in attributes (use `best-effort-sanitize-sql` for HoneySQL, never raw SQL)
• [ ] No new tracing namespaces created (add to `tracing.core` instead)
• [ ] No `DO_NOT_ADD_NEW_FILES_HERE.txt` violations in the target directory
• [ ] Run `clj-kondo --lint <files>` to verify 0 errors, 0 warnings
• [ ] Add or update tests in the corresponding `test/` path (see Testing section below)
• [ ] Run tests: `clojure -X:dev:test :only <test-ns>`

The `with-span` Macro

(tracing/with-span group span-name attrs & body)

• **group** - A keyword selecting which trace group this span belongs to (e.g., `:tasks`, `:sync`)
• **span-name** - A string identifying the span in traces (e.g., `"search.execute"`)
• **attrs** - A map of span attributes (e.g., `{:db/id 42}`)
• **body** - The code to execute inside the span

**When disabled:** zero overhead -- single atom deref + boolean check, body runs directly. **When enabled:** creates OTel span AND injects `trace_id`/`span_id` into Log4j2 MDC for log-to-trace correlation.

Trace Groups

Groups are registered in `src/metabase/tracing/core.clj`. Check that file for the current list. The general rule: **match the group to the domain, not the call site.** If code runs inside a Quartz job but is logically search work, use `:search`, not `:tasks`.

To add a new group:

;; In src/metabase/tracing/core.clj
(register-group! :my-domain "Description of what this covers")

Users enable groups via `MB_TRACING_GROUPS=tasks,search,sync` (comma-separated, or `"all"`).

Naming Conventions

Span Names

Use dot-separated hierarchical names: `"domain.subsystem.operation"`. The domain prefix should match the group name:

search.execute              -- `:search` group
sync.fingerprint.table      -- `:sync` group
task.session-cleanup.delete -- `:tasks` group
db-app.collection-items     -- `:db-app` group

Attributes

Use namespaced keywords. The namespace groups related attributes:

:db/id              -- Database ID (integer)
:db/engine          -- Database engine name (string)
:db/statement       -- Sanitized SQL (string, via best-effort-sanitize-sql)
:search/engine      -- Search engine name (string)
:search/query-length -- Query string leng