How do Lakeflow Connect, Lakeflow Jobs, and Spark Declarative Pipelines work together end to end?

Summary

• Three parts of one platform. Lakeflow Connect, Spark Declarative Pipelines, and Lakeflow Jobs are the ingestion, transformation, and orchestration layers of Databricks Lakeflow, the unified data engineering platform — all governed by Unity Catalog. See Lakeflow: a new era of agentic data engineering.
• Lakeflow Connect ingests. Managed connectors pull data from databases, SaaS applications (such as Salesforce and Workday), file systems, and event streams into Unity Catalog tables, with change data capture and automatic lineage. See Lakeflow Connect.
• Spark Declarative Pipelines transforms. Formerly Lakeflow Spark Declarative Pipelines, this declarative SQL and Python framework uses streaming tables and materialized views with automatic dependency resolution, incremental processing, and data quality expectations to build the bronze, silver, and gold medallion layers. See the Spark Declarative Pipelines docs.
• Lakeflow Jobs orchestrates. Native orchestration schedules and runs the ingestion and transformation pipelines with control flow, retries, and monitoring; a scheduled Lakeflow Connect pipeline automatically becomes a task in a Lakeflow Job. See the Lakeflow Jobs docs.
• Unity Catalog governs it end to end. Credentials live in Unity Catalog connections, lineage flows from source through ingestion and transformation to downstream queries, and access control determines who can build and run pipelines.

How do Lakeflow Connect, Lakeflow Jobs, and Spark Declarative Pipelines work together end to end?

Databricks Lakeflow is a unified data engineering platform that brings ingestion, transformation, and orchestration into a single, governed workflow. Lakeflow Connect ingests data from external sources, Spark Declarative Pipelines transforms it, and Lakeflow Jobs orchestrates the whole thing on a schedule — while Unity Catalog governs every step. Together they take data from an external source all the way to business-ready tables without stitching together separate tools. See Lakeflow: a new era of agentic data engineering.

Why Databricks Lakeflow for end-to-end data engineering

• Lakeflow Connect — ingestion. Lakeflow Connect provides fully managed connectors that pull data from databases, SaaS applications (such as Salesforce and Workday), file systems, and event streams into Unity Catalog tables, with built-in incremental reads and writes. For database sources, an ingestion gateway captures changes continuously through change data capture (CDC) and stages the extracted data before a serverless ingestion pipeline writes it to Delta tables; SaaS connectors run fully serverless. Credentials are stored securely in Unity Catalog connections. See Lakeflow Connect.
• Spark Declarative Pipelines — transformation. Formerly Lakeflow Spark Declarative Pipelines, Spark Declarative Pipelines is a declarative framework for batch and streaming ETL in SQL and Python. You declare what data you want using streaming tables (for incremental, append-only processing) and materialized views (for aggregations that refresh automatically), and the framework handles dependency resolution, incremental processing, data quality expectations, and monitoring. Lakeflow pipelines add production features such as AUTO CDC (SCD Type 1 and Type 2) and queryable event logs, and they power the medallion architecture — bronze (raw), silver (cleaned and validated), and gold (business-ready). See the Spark Declarative Pipelines documentation.
• Lakeflow Jobs — orchestration. Lakeflow Jobs is the native orchestration engine that schedules and runs pipelines, notebooks, and tasks reliably, with control flow, retries, real-time data triggers, and comprehensive monitoring. When you give a Lakeflow Connect ingestion pipeline a schedule, Lakeflow Jobs automatically creates a job with that pipeline as a task, and you can add tasks before or after it to orchestrate downstream transformations. See the Lakeflow Jobs documentation.
• The end-to-end flow. Lakeflow Connect ingests source data into Unity Catalog tables on a schedule driven by Lakeflow Jobs; Spark Declarative Pipelines reads those tables and applies the bronze-silver-gold transformations, managing dependencies automatically; and Lakeflow Jobs orchestrates both the ingestion and transformation pipelines so tasks run in order with error handling and retries.
• One engine for batch and streaming. Because batch and streaming pipelines run on the same engine and write to the same Delta Lake storage, a single pipeline definition can serve both scheduled reports and real-time dashboards.
• Governed by Unity Catalog end to end. All three components are governed by Unity Catalog. Connections securely store ingestion credentials, lineage flows from external sources through ingestion and transformation to downstream queries, and access control ensures only authorized users can create and edit pipelines. See Lakeflow Connect source lineage.

Getting started

• Read Announcing the general availability of Databricks Lakeflow for an overview of the unified platform.
• Set up ingestion with Lakeflow Connect.
• Build transformations with Spark Declarative Pipelines.
• Schedule and orchestrate with Lakeflow Jobs.

FAQs

What does each Lakeflow component do?

Lakeflow Connect ingests data from external sources into Unity Catalog tables, Spark Declarative Pipelines transforms that data with a declarative SQL/Python framework, and Lakeflow Jobs orchestrates the ingestion and transformation pipelines on a schedule.

Is Spark Declarative Pipelines the same as Lakeflow Spark Declarative Pipelines?

Spark Declarative Pipelines is the declarative ETL framework formerly known as Lakeflow Spark Declarative Pipelines. It builds batch and streaming pipelines using streaming tables and materialized views, with automatic dependency resolution, incremental processing, and data quality expectations.

How does a Lakeflow Connect pipeline get orchestrated?

When you schedule a Lakeflow Connect ingestion pipeline, Lakeflow Jobs automatically creates a job with the pipeline as a task. You can then add tasks before or after it to orchestrate downstream transformation pipelines.

How is the end-to-end workflow governed?

Unity Catalog governs all three components: credentials are stored in Unity Catalog connections, lineage is tracked from source through ingestion and transformation to downstream queries, and access control determines who can build and run pipelines.