How do I build my first data pipeline on a lakehouse?

Summary

• Use Lakeflow Declarative Pipelines (SQL or Python) to build your first pipeline: you declare the tables you want, and Databricks handles orchestration, incremental processing, checkpoints, and retries.
• Follow the medallion architecture: land raw data in a bronze layer, clean and validate it in silver, and aggregate business-ready metrics in gold.
• Ingest with Lakeflow Connect and Auto Loader, which incrementally discover and process new files and source records as they arrive.
• Store everything as Delta tables governed by Unity Catalog, so every table gets governance, lineage, and access control automatically.
• Add data quality with expectations, then schedule and automate the pipeline with Lakeflow Jobs.

How do I build my first data pipeline on a lakehouse?

A lakehouse data pipeline moves data from raw sources into clean, query-ready tables. On Databricks, the fastest path for a first pipeline is Lakeflow Declarative Pipelines (formerly Lakeflow Spark Declarative Pipelines): you declare the tables you want in SQL or Python, and the platform builds the dependency graph, processes only new data, manages checkpoints, and recovers from errors for you. Data lands as Delta tables governed by Unity Catalog, so governance and lineage come built in. See the ETL pipeline tutorial to follow along end to end.

Build your first pipeline, step by step

1. Ingest raw data. Use Lakeflow Connect to bring in data from databases, SaaS applications, files, and event streams. For files landing in cloud storage, Auto Loader automatically detects and processes new files as they arrive, with exactly-once processing. See the Auto Loader documentation.
2. Create the pipeline. In your workspace, choose New → ETL Pipeline, name it, select your Unity Catalog catalog and target schema, and pick SQL or Python for your first source file. The Lakeflow Pipelines editor shows your code, a live pipeline graph, and table previews side by side.
3. Build the bronze layer. Define a streaming table that reads raw source files or records incrementally with Auto Loader. Streaming tables are Delta tables that support incremental, exactly-once processing.
4. Build the silver layer. Add a streaming table that reads from bronze, then filter, deduplicate, and reshape the data. Attach expectations to validate records inline; failing rows can be dropped, flagged, or made to halt the pipeline.
5. Build the gold layer. Use a materialized view to compute business-ready aggregates and KPIs. Materialized views cache results and refresh automatically when their inputs change, which makes them well suited to dashboards.
6. Publish and automate. Set the pipeline's destination to Unity Catalog, then schedule it with Lakeflow Jobs and configure failure notifications so it runs on its own.

Why build data pipelines on the Databricks lakehouse

• Declarative, less code to maintain. With Lakeflow Declarative Pipelines you describe the result you want; the platform derives the execution plan, processes only new or changed data, and manages checkpoints for exactly-once guarantees. See Lakeflow Pipelines: efficient declarative data pipelines.
• SQL or Python, mixed freely. Analysts can write declarative SQL and engineers can use the Apache Spark DataFrame API in Python; a single pipeline can combine both languages across files.
• Built-in data quality and observability. Expectations validate data inline, and the pipeline tracks lineage, performance, and health metrics automatically.
• Governed by Unity Catalog. Every table the pipeline creates is a Delta table managed by Unity Catalog, with unified governance, lineage, and fine-grained access control.
• A proven design pattern. The medallion architecture (bronze, silver, gold) organizes data by progressive refinement so each layer has a clear purpose. See Data pipeline architecture on Databricks and the medallion architecture guide.

Getting started

• Follow the Build an ETL pipeline tutorial to create your first pipeline end to end.
• Set up Unity Catalog so your pipeline lands governed Delta tables.
• Configure ingestion with Lakeflow Connect and Auto Loader.
• Review the Lakeflow Declarative Pipelines documentation to learn streaming tables, materialized views, and expectations.

FAQs

What is the easiest way to build a first pipeline on Databricks?

Use Lakeflow Declarative Pipelines. You declare bronze, silver, and gold tables in SQL or Python, and Databricks handles orchestration, incremental processing, and data quality automatically.

What is the medallion architecture?

It is a design pattern that organizes data into three layers: bronze for raw ingested data, silver for cleaned and validated data, and gold for business-ready aggregates. Each layer refines the data further.

Should I use SQL or Python for my pipeline?

Both are fully supported. SQL is declarative and familiar to analysts; Python offers flexibility for complex logic through the Spark DataFrame API. A single pipeline can use both.

How do I load new data as it arrives?

Use Auto Loader with a streaming table. It incrementally detects and processes new files in cloud storage with exactly-once semantics, so you do not reprocess old data.