What should I use to build production data pipelines?

Summary

• On Databricks, build production pipelines with Lakeflow Pipelines for the transformations and Lakeflow Jobs for orchestration. Pipelines are a declarative framework for batch and streaming data in SQL and Python.
• Production concerns have built-in features: pipelines order steps by dependency and retry transient failures, expectations validate records, and the pipeline event log records progress and data quality checks.
• Unity Catalog governs the data with access control, column-level lineage and audit logging.
• Declarative Automation Bundles describe jobs and pipelines as source files, so they go through source control, code review and CI/CD.

What should I use to build production data pipelines?

Use Lakeflow Pipelines to define the transformations and Lakeflow Jobs to orchestrate them. A Lakeflow pipeline is a declarative framework for batch and streaming data pipelines in SQL and Python: you define the datasets, and the pipeline works out execution order, parallelism and incremental updates. Lakeflow Jobs coordinates pipelines together with notebooks, Python scripts and other tasks. If your team already orchestrates with Apache Airflow, Airflow can trigger Lakeflow Jobs through the open source Databricks provider.

What makes a data pipeline production-ready?

Requirement	What to look for	How Databricks covers it
Dependency handling	Steps run in the right order, in parallel where possible	Pipelines analyze the dependencies between datasets and orchestrate execution automatically
Incremental processing	Only new or changed data is processed	Streaming tables process each record once; materialized views refresh incrementally when supported
Retries and repair	Recovery without rerunning everything	Transient failures retry at the task, flow and pipeline level; Jobs can re-run only unsuccessful tasks
Data quality	Bad records are caught early	Expectations warn, drop the row or fail the update
Lineage and audit	Trace data back to its sources	Unity Catalog captures column-level lineage and audit logs
Scheduling and alerts	Triggers and failure notifications	Jobs support scheduled, table-update, file-arrival and continuous triggers, plus notifications
Deployment	Version control and repeatable releases	Declarative Automation Bundles

How does Databricks approach production pipelines?

Ingest. Use Auto Loader for files in cloud object storage, a Lakeflow Connect managed connector for databases and SaaS applications where one exists, and streaming tables for message buses such as Apache Kafka.
Transform. A medallion layout in one pipeline looks like this:

CREATE OR REFRESH STREAMING TABLE orders_bronze
AS SELECT * FROM STREAM read_files(
  's3://mybucket/orders/',
  format => "json"
);

CREATE OR REFRESH STREAMING TABLE orders_silver(
  CONSTRAINT valid_amount EXPECT (amount >= 0) ON VIOLATION DROP ROW
)
AS SELECT * FROM STREAM(orders_bronze);

CREATE OR REFRESH MATERIALIZED VIEW orders_daily
AS SELECT
  date_trunc('day', order_time) AS order_date,
  count(*) AS order_count,
  sum(amount) AS total_revenue
FROM orders_silver
GROUP BY order_date;

An expectation takes one of three actions: warn (the default), drop the row, or fail the update. A materialized view that defines expectations is fully refreshed on each update, so the example keeps the expectation on the streaming table.
Orchestrate. A Lakeflow Job can run a pipeline task alongside notebook and Python script tasks. Add a trigger and notifications for failures or long durations. If a multi-task run fails, a repair run re-runs only the unsuccessful tasks and their dependents. Jobs doesn't make tasks idempotent, so check that a re-run can't duplicate data.
Monitor and govern. The pipeline event log holds audit logs, data quality checks, pipeline progress and data lineage. Unity Catalog enforces access control and logs activity for auditing.
Deploy. Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs and pipelines as source files for source control, testing and CI/CD.

What practices help in production?

• Layer data as bronze, silver and gold, which Databricks recommends as a best practice for improving data quality step by step.
• Run pipelines as a service principal instead of a personal account.
• Land raw files in a Unity Catalog volume so lineage and access control reach back to the landing zone.
• Check pipeline code with a dry run before creating or updating tables.
• Develop new pipelines on serverless compute, which Databricks recommends.

FAQs

Should I use Airflow or Lakeflow Pipelines?

They work at different layers. Lakeflow Pipelines defines and runs transformations. Airflow is an orchestrator that represents pipelines as directed acyclic graphs of operations. Databricks documents triggering a Lakeflow Job from Airflow with the DatabricksRunNowOperator in the open source Databricks provider, which Databricks doesn't directly support.

How do I handle slowly changing dimensions?

Use AUTO CDC in Lakeflow Pipelines. It handles change data capture feeds, including out-of-order events, and supports SCD Type 1 and Type 2. For custom code, Delta Lake supports MERGE for upserts.

What happens when a pipeline fails?

The pipeline retries transient failures progressively. An expectation set to fail the update stops processing until you fix the pipeline code. In a job, a repair run re-runs only the unsuccessful tasks.

How do I run a pipeline on a schedule?

Add the pipeline to a Lakeflow Job as a pipeline task, then give the job a scheduled, table-update, file-arrival or continuous trigger.

How do I scale to large datasets?

Serverless pipelines run on Databricks-managed compute and remove most infrastructure configuration. On classic compute, enhanced autoscaling adds and removes compute as workload volume changes, and it is the default for new pipelines.