/e2e-medallion-architecture

name: e2e-medallion-architecture
description: "Plans and builds end-to-end Fabric data platforms on the Bronze, Silver, and Gold medallion pattern: ingestion to analytics, PySpark and Delta layers, pipeline orchestration, per-layer Spark tuning, Materialized Lake View versus notebook choices for Silver and Gold, and Direct Lake handoff. Use for layered architecture design, planning and recommendations. For questions over an existing Power BI report use fabriciq."

> **Telemetry — MANDATORY.** Every `api.fabric.microsoft.com` call must carry > `x-ms-fabric-skill: e2e-medallion-architecture` (`az rest`: `--headers "x-ms-fabric-skill=e2e-medallion-architecture"`), > including every LRO poll, `fabric_lro` and retry. Snippets omit it — add it anyway.

> **CRITICAL NOTES** > 1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering > 2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering

End-to-End Medallion Architecture

Prerequisite Knowledge

Read these companion documents — they contain the foundational context this skill depends on:

• [COMMON-CORE.md](../../common/COMMON-CORE.md) — Fabric REST API patterns, authentication, token audiences, item discovery
• [COMMON-CLI.md](../../common/COMMON-CLI.md) — `az rest`, `az login`, token acquisition, Fabric REST via CLI
• [SPARK-AUTHORING-CORE.md](../../common/SPARK-AUTHORING-CORE.md) — Notebook deployment, lakehouse creation, job execution
• [notebook-api-operations.md](../spark-cli/references/authoring/resources/notebook-api-operations.md) — **Required for notebook creation** — `.ipynb` structure requirements, cell format, `getDefinition`/`updateDefinition` workflow

For Spark-specific optimization details, see [data-engineering-patterns.md](../spark-cli/references/authoring/resources/data-engineering-patterns.md).

---

Architecture Overview

**Medallion Architecture** is a data lakehouse pattern with three progressive layers:

| Layer | Purpose | Optimization Profile | Use Case | |-------|---------|---------------------|----------| | **Bronze** (Raw) | Land raw data exactly as received | Write-optimized, append-only, partitioned by ingestion date | Audit trail, reprocessing, lineage | | **Silver** (Cleaned) | Deduplicated, validated, conformed data | Balanced read/write, partitioned by business date | Feature engineering, operational reporting | | **Gold** (Aggregated) | Pre-calculated metrics for analytics | Read-optimized (ZORDER, compaction), partitioned by month/year | Power BI reports, dashboards, ad-hoc analytics via SQL endpoint |

• **Bronze**: Schema-on-read — flexible schema, Delta time travel supports audit and rollback
• **Silver**: Schema enforcement — reject non-conforming writes; handle schema evolution with `mergeSchema` when sources change
• **Gold**: Strict schema governance — curated, business-approved datasets only

---

Must/Prefer/Avoid

MUST DO

• **Choose lakehouse architecture** based on schema-enabled availability (see [infrastructure-orchestration.md](../spark-cli/references/authoring/resources/infrastructure-orchestration.md)):
• **Preferred:** Schema-enabled lakehouse → create ONE workspace + ONE lakehouse with `bronze`, `silver`, `gold` schemas
• **Legacy:** Non-schema-enabled → create separate workspaces per layer (Bronze, Silver, Gold) for governance and access control
• **Use Livy API for schema and table creation** — to create schemas and tables in a schema-enabled lakehouse, submit Spark SQL statements via Livy sessions (`POST /livyApi/versions/2023-12-01/sessions` → `POST .../statements`). This is the only programmatic REST path for DDL operations (CREATE SCHEMA, CREATE TABLE) in Fabric lakehouses.
• Add **metadata columns** in Bronze: ingestion timestamp, source file, batch ID
• Apply **data quality rules** in the Bronze-to-Silver transformation (deduplication, null handling, range validation)
• Use **Delta Lake format** for all medallion layer tables
• Use **partition-aware overwrite** in Silver/Gold writes to avoid reprocessing unchanged data
• Include **validation steps** after each layer (row counts, schema checks, anomaly detection)
• Follow the **`.ipynb` validation + Fabric nuances** in [notebook-api-operations.md](../spark-cli/references/authoring/resources/notebook-api-operations.md#ipynb-validation--fabric-nuances) when creating notebooks via REST API — every code cell must include `"outputs": []` and `"execution_count": null`
• **Complete the full end-to-end flow** — do not stop after creating notebooks; always bind lakehouses, execute notebooks sequentially (Bronze → Silver → Gold), verify results, and connect Power BI to the Gold layer unless the user explicitly requests a partial setup
• In every MLV-versus-notebook recommendation, state that MLVs require a **schema-enabled lakehouse**. Hand off MLV definition and incremental-refresh review to `spark-cli` **authoring** mode; hand off scheduling, refresh, monitoring, and failure diagnosis to `spark-cli` **mlv** mode.
• For recurring MLV refresh, give the exact interactive path **Lakehouse → Materialized lake views → Manage → Schedules**. For automation, use `POST /workspaces/{workspaceId}/lakehouses/{lakehouseId}/jobs/refreshMaterializedLakeViews/schedules`; never invent `/mlvRefreshSchedules` or route recurring refresh through notebook scheduling.

PREFER

• Incremental processing (watermark pattern) over full refresh
• Separate notebooks per layer for independent testing and debugging
• ZORDER on frequently filtered columns in Gold tables
• Running OPTIMIZE after writes in Silver and Gold layers
• Environment-specific Spark configs (write-heavy for Bronze, balanced for Silver, read-heavy for Gold)
• OneLake shortcuts to expose Gold data to consumer workspaces without duplication
• Clear layer ownership: engineers own Bronze/Silv