/databricks-mlflow-evaluation

name: databricks-mlflow-evaluation
description: "MLflow 3 GenAI agent evaluation. Use when writing mlflow.genai.evaluate() code, creating @scorer functions, using built-in scorers (Guidelines, Correctness, Safety, RetrievalGroundedness), building eval datasets from traces, setting up trace ingestion and production monitoring, aligning judges with MemAlign from domain expert feedback, or running optimize_prompts() with GEPA for automated prompt improvement."
compatibility: Requires databricks CLI (>= v1.0.0)
metadata:
  version: "0.1.0"
parent: databricks-core

MLflow 3 GenAI Evaluation

Scope vs upstream `mlflow/skills`

The OSS `mlflow/skills` repo ships [`agent-evaluation`](https://github.com/mlflow/skills/tree/main/agent-evaluation) and related skills (`instrumenting-with-mlflow-tracing`, `analyze-mlflow-trace`, `retrieving-mlflow-traces`, `querying-mlflow-metrics`) that cover the generic MLflow GenAI evaluation workflow — `mlflow.genai.evaluate()`, scorers/judges, datasets, tracing setup, and the 5-step evaluation loop.

This skill **layers Databricks-specific patterns on top of that workflow** rather than restating it. Use this skill when you need any of:

• **Unity Catalog trace ingestion** — production traces written into UC tables, log-based monitoring (`patterns-trace-ingestion.md`).
• **MemAlign judge alignment via UC SME labeling sessions** — aligning custom judges against domain-expert feedback collected in Databricks (`patterns-judge-alignment.md`).
• **`optimize_prompts()` GEPA loop** — Databricks' automated prompt-optimization driver running on a UC dataset (`patterns-prompt-optimization.md`).
• **Databricks-flavored scorer/dataset patterns** — UC-table-backed datasets, tagging traces in the Databricks UI for inclusion (`patterns-datasets.md`, `patterns-scorers.md`).

For everything else — generic `mlflow.genai.evaluate()` calls, scorer authoring patterns, dataset creation outside Databricks, MLflow tracing setup that isn't UC-table-bound — the upstream `mlflow/skills/agent-evaluation` skill is the canonical source and is kept current by the MLflow team.

Before Writing Any Code

1. **Read GOTCHAS.md** - 15+ common mistakes that cause failures 2. **Read CRITICAL-interfaces.md** - Exact API signatures and data schemas

End-to-End Workflows

Follow these workflows based on your goal. Each step indicates which reference files to read.

Workflow 1: First-Time Evaluation Setup

For users new to MLflow GenAI evaluation or setting up evaluation for a new agent.

| Step | Action | Reference Files | |------|--------|-----------------| | 1 | Understand what to evaluate | `user-journeys.md` (Journey 0: Strategy) | | 2 | Learn API patterns | `GOTCHAS.md` + `CRITICAL-interfaces.md` | | 3 | Build initial dataset | `patterns-datasets.md` (Patterns 1-4) | | 4 | Choose/create scorers | `patterns-scorers.md` + `CRITICAL-interfaces.md` (built-in list) | | 5 | Run evaluation | `patterns-evaluation.md` (Patterns 1-3) |

Workflow 2: Production Trace -> Evaluation Dataset

For building evaluation datasets from production traces.

| Step | Action | Reference Files | |------|--------|-----------------| | 1 | Search and filter traces | `patterns-trace-analysis.md` (MCP tools section) | | 2 | Analyze trace quality | `patterns-trace-analysis.md` (Patterns 1-7) | | 3 | Tag traces for inclusion | `patterns-datasets.md` (Patterns 16-17) | | 4 | Build dataset from traces | `patterns-datasets.md` (Patterns 6-7) | | 5 | Add expectations/ground truth | `patterns-datasets.md` (Pattern 2) |

Workflow 3: Performance Optimization

For debugging slow or expensive agent execution.

| Step | Action | Reference Files | |------|--------|-----------------| | 1 | Profile latency by span | `patterns-trace-analysis.md` (Patterns 4-6) | | 2 | Analyze token usage | `patterns-trace-analysis.md` (Pattern 9) | | 3 | Detect context issues | `patterns-context-optimization.md` (Section 5) | | 4 | Apply optimizations | `patterns-context-optimization.md` (Sections 1-4, 6) | | 5 | Re-evaluate to measure impact | `patterns-evaluation.md` (Pattern 6-7) |

Workflow 4: Regression Detection

For comparing agent versions and finding regressions.

| Step | Action | Reference Files | |------|--------|-----------------| | 1 | Establish baseline | `patterns-evaluation.md` (Pattern 4: named runs) | | 2 | Run current version | `patterns-evaluation.md` (Pattern 1) | | 3 | Compare metrics | `patterns-evaluation.md` (Patterns 6-7) | | 4 | Analyze failing traces | `patterns-trace-analysis.md` (Pattern 7) | | 5 | Debug specific failures | `patterns-trace-analysis.md` (Patterns 8-9) |

Workflow 5: Custom Scorer Development

For creating project-specific evaluation metrics.

| Step | Action | Reference Files | |------|--------|-----------------| | 1 | Understand scorer interface | `CRITICAL-interfaces.md` (Scorer section) | | 2 | Choose scorer pattern | `patterns-scorers.md` (Patterns 4-11) | | 3 | For multi-agent scorers | `patterns-scorers.md` (Patterns 13-16) | | 4 | Test with evaluation | `patterns-evaluation.md` (Pattern 1) |

Workflow 6: Unity Catalog Trace Ingestion & Production Monitoring

For storing traces in Unity Catalog, instrumenting applications, and enabling continuous production monitoring.

| Step | Action | Reference Files | |------|--------|-----------------| | 1 | Link UC schema to experiment | `patterns-trace-ingestion.md` (Patterns 1-2) | | 2 | Set trace destination | `patterns-trace-ingestion.md` (Patterns 3-4) | | 3 | Instrument your application | `patterns-trace-ingestion.md` (Patterns 5-8) | | 4 | Configure trace sources (Apps/Serving/OTEL) | `patterns-trace-ingestion.md` (Patterns 9-11) | | 5 | Enable production monitoring | `patterns-trace-ingestion.md` (Patterns 12-13) | | 6 | Query and analyze UC traces | `patterns-trace-ingestion.md` (Pattern 14) |

Workflow 7: Judge Alignment with MemAlign

For aligning an LLM judge to match domain expert preferences. A well-aligned judg