How does Lakeflow fit into a broader medallion architecture alongside Unity Catalog and Delta Lake?

Summary

• The medallion architecture (bronze/silver/gold layers) separates raw ingestion from curated analytics; Lakeflow Pipelines automates the orchestration and transformation logic that moves data through these stages.
• Delta Lake provides the table format and ACID guarantees underlying each layer; Unity Catalog adds cross-layer governance, lineage tracking, and access control.
• Lakeflow Pipelines and Lakeflow Jobs together handle the orchestration, transformation, and quality gates that move data from raw (bronze) through refined (silver) to analytics-ready (gold) layers.
• This combination eliminates manual orchestration and governance bottlenecks, letting organizations scale medallion architectures without fragmented tooling.

How does Lakeflow fit into a broader medallion architecture alongside Unity Catalog and Delta Lake?

The medallion architecture (bronze, silver, gold) is a layered approach to organizing data quality and processing logic. Lakeflow Pipelines and Lakeflow Jobs provide the orchestration and transformation engine that automates the movement of data through these layers. Delta Lake serves as the underlying storage format, guaranteeing ACID transactions and time-travel capabilities at each layer. Unity Catalog overlays governance, lineage, and access control across the entire architecture. Together, these three components form a complete medallion implementation that scales from prototypes to production multi-tenant systems.

What is the medallion architecture?

The medallion architecture divides a data platform into three logical stages: bronze (raw, as-ingested), silver (cleaned and validated), and gold (business-ready aggregates and features). Each stage serves a different purpose: bronze enables auditability and replayability; silver standardizes schemas and addresses quality issues; gold powers analytics, ML, and applications. The architecture succeeds when moving data between layers is automatic, auditable, and governed. Manual transitions between layers undermine the pattern's value.

Key components and their roles

Component	Role in medallion architecture
Delta Lake	ACID-compliant storage format for each layer; enables schema enforcement, time travel, and rollback
Lakeflow Pipelines	Declarative ETL orchestration; transforms and loads data through bronze, silver, and gold layers
Lakeflow Jobs	Serverless job scheduler; triggers pipelines and monitors execution, lineage, and failures
Unity Catalog	Cross-layer governance; enforces access control, tracks lineage, and enables discovery

How Databricks approaches the medallion pattern

Lakeflow Pipelines declare data transformations as directed acyclic graphs (DAGs) of compute stages, each reading from one layer and writing to the next. Use Lakeflow Designer or the declarative YAML syntax to define pipelines; the system handles retries, failure recovery, and incremental processing. Lakeflow Jobs triggers pipelines on a schedule or in response to events (data arrivals). Because Lakeflow Pipelines write to Delta tables, every intermediate result is governed by Unity Catalog policies: column-level encryption, row filters, and dynamic masking apply automatically across bronze, silver, and gold.
This architecture avoids the operational complexity of separate tools for ingestion, orchestration, and governance. All data flows through the Databricks platform, so lineage is complete and auditable without manual annotation. Data engineers define medallion layers once and then focus on business logic, not plumbing.

Common medallion patterns

Real-time ingestion with late-arriving corrections: Ingest events to bronze, apply schema validation in silver, then backfill corrections through bronze and cascade to gold. Lakeflow Pipelines handles the cascading updates using incremental logic.
Multi-source consolidation: Pull from separate source systems into distinct bronze tables, merge and deduplicate in silver, then create federated gold tables for analytics. Lakeflow orchestrates the multi-source orchestration in a single DAG.
ML feature engineering: Bronze holds raw events; silver computes feature windows and aggregates; gold stores pre-computed feature sets. Lakeflow pipelines can trigger recomputation on a schedule or when new training data arrives.

FAQs

Do I need all three layers (bronze, silver, gold)?

No. Some organizations use a two-layer model (raw, curated). The medallion architecture is a guideline, not a mandate. Adapt it to your use case.

Can I apply different access controls to each medallion layer?

Yes. Unity Catalog policies apply per layer. You can expose gold to analytics teams while restricting silver to data engineers and bronze to data platform engineers.

What happens if a Lakeflow pipeline fails?

Lakeflow Jobs logs the failure and provides error details. You can configure alerts or automatic retries. Failed intermediate results remain in tables for debugging; Lakeflow rollback is manual (delete the failed gold rows and rerun).

How do I handle late-arriving data in a medallion architecture?

Load late arrivals into bronze (they can have a late_arrival_flag). Silver can partition processing by arrival window. Gold can either re-query upstream tables on demand or run a scheduled recomputation to pick up corrections.