What is the medallion (bronze, silver, gold) architecture?

Summary

• The medallion architecture is a data design pattern that organizes lakehouse tables into bronze, silver and gold layers, improving the structure and quality of data step by step as it flows from one layer to the next.
• Bronze keeps source data in its raw state with minimal validation; silver holds cleansed, deduplicated and validated records; gold holds refined, business-level models that feed dashboards, ML and applications.
• Because bronze retains the raw input, any silver or gold table can be rebuilt from it without going back to the source system.
• It's a pattern, not a product or a Spark feature. In open-source Apache Spark, Spark Declarative Pipelines (SDP) can express each layer as a streaming table or materialized view.

What is the medallion (bronze, silver, gold) architecture?

The medallion architecture is a way to organize lakehouse data into three layers of increasing quality: bronze for raw data as it arrived, silver for validated and cleansed records, and gold for business-ready models. Each layer is computed from the one before it, so when a transformation has a bug you fix the code and reprocess from bronze instead of re-reading the source system. The pattern is also called a multi-hop architecture. It describes how tables relate to each other, not something you install, so it applies equally to jobs on open-source Spark or a managed platform.

What each layer holds

Layer	What it holds	Why it matters
Bronze	Data landed from source systems in its original format, with minimal validation. Databricks recommends storing most fields as string, VARIANT or binary so unexpected schema changes don't drop data.	It's the replay point. Later layers can be rebuilt without returning to the source.
Silver	Cleansed, deduplicated and normalized records, after schema enforcement, null handling and joins. It should always include at least one validated, non-aggregated representation of each record.	Gives every consumer one trustworthy version of each entity to build on.
Gold	Highly refined views of the data, typically modeled as a dimensional model with defined relationships and measures.	Serves dashboards, ML and applications without each consumer repeating the cleanup.

How data moves through the layers

Data flows one way, from bronze to silver to gold, and each hop is a transformation you can rerun. A few practices follow from the definitions:

• Land everything in bronze first. Bronze can combine batch and streaming ingestion from object storage, message buses such as Apache Kafka, and other systems. Keep records as received; cleanup belongs downstream.
• Read bronze incrementally. When sources are append-only, Databricks' guidance is to configure most reads from bronze as streaming reads, so silver processes only new records.
• Validate in silver, model in gold. Type casting, deduplication and quality rules belong in silver. Aggregates and business logic belong in gold.
• Design gold for its readers. Gold is usually organized as consumption-ready, project-specific databases.

A medallion pipeline in Spark Declarative Pipelines

Spark Declarative Pipelines arrived in Apache Spark 4.1. You define datasets and the queries that produce them, and Spark handles the execution graph, dependency ordering, parallelism, checkpoints and retries. The medallion layers map directly onto SDP datasets: streaming tables for bronze and silver, a materialized view for gold. One pipeline project can mix Python and SQL source files, so bronze can ingest from Kafka in Python:

from pyspark import pipelines as dp

@dp.table
def orders_bronze():
    return (
        spark.readStream.format("kafka")
        .option("kafka.bootstrap.servers", "localhost:9092")
        .option("subscribe", "orders")
        .load()
    )

while silver and gold stay in SQL:

CREATE STREAMING TABLE orders_silver
AS SELECT CAST(key AS STRING) AS order_id,
          CAST(value AS STRING) AS payload,
          `timestamp` AS event_time
FROM STREAM(orders_bronze)
WHERE value IS NOT NULL;

CREATE MATERIALIZED VIEW orders_gold
AS SELECT to_date(event_time) AS order_date, COUNT(*) AS orders
FROM orders_silver
GROUP BY to_date(event_time);

spark-pipelines dry-run catches syntax, analysis and graph errors without reading or writing data, and spark-pipelines run executes the pipeline. When a silver rule changes, spark-pipelines run --full-refresh orders_silver clears that dataset's data and checkpoints and recomputes it. That's the medallion promise in practice: bronze stays intact while the layers built on it are rebuilt.

How Databricks approaches the medallion architecture

On Databricks, Lakeflow Pipelines extend and interoperate with Spark Declarative Pipelines and run on Databricks Runtime, so the streaming tables and materialized views above carry over, in triggered or continuous mode. Three managed pieces line up with the layers:

• Bronze: Databricks recommends Auto Loader, the cloudFiles source, in Lakeflow pipelines for incremental ingestion of new files from cloud storage.
• Silver: expectations apply data quality checks to each record, for example @dp.expect_or_drop("valid_order", "order_id IS NOT NULL") in Python or CONSTRAINT valid_order EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW in SQL. Expectations are a Lakeflow Pipelines feature; the open-source SDP guide doesn't include them.
• All layers: Unity Catalog captures lineage automatically for queries run on Databricks, down to the column level, so you can trace a gold metric back to the bronze tables behind it.

FAQs

Do I need exactly three layers?

No. Databricks describes the medallion architecture as a recommended best practice, not a requirement, and the layers denote data quality rather than a fixed number of tables. What matters is keeping the raw data and making every step toward business-ready tables a transformation you can rerun.

Is the medallion architecture specific to Delta Lake or Databricks?

No. It's a data design pattern for organizing lakehouse tables, so it isn't a product feature. You can build it on open-source Apache Spark, including Spark Declarative Pipelines, as the example above does.

Should bronze be loaded by batch or streaming?

Either, or both. Bronze can mix streaming and batch ingestion. Reading bronze into silver as a stream, when sources are append-only, keeps silver incremental.

Does the medallion architecture work with a data mesh?

Yes. Databricks' glossary entry states that the medallion architecture is compatible with the concept of a data mesh.