What is Lakeflow Connect?

Summary

• Lakeflow Connect is Databricks' unified data ingestion solution that moves data from diverse sources into the lakehouse with managed connectors and built-in governance.
• It organizes connectors by source type: databases (with CDC support), SaaS applications, file sources, streaming platforms, and direct API ingestion.
• Incremental read and write logic ensures efficient data movement; subsequent runs ingest only changed data rather than full refreshes.
• All pipelines integrate with Unity Catalog for credential management, access control, and governance across your data stack.
• Lakeflow Connect fits within the broader Lakeflow platform alongside Pipelines (transformation) and Jobs (orchestration) for end-to-end data engineering.

What is Lakeflow Connect?

Lakeflow Connect is Databricks' unified data ingestion solution that simplifies moving data from diverse sources into the lakehouse. It provides a consolidated platform for ingesting from databases, SaaS applications, cloud storage, streaming platforms, and more. It combines managed connectors, incremental processing, and built-in governance to reduce the complexity and operational burden of data ingestion.

Data ingestion fundamentals

Data ingestion is the process of extracting data from source systems and loading it into a data platform. Teams typically face three challenges: managing credentials across many sources, selecting efficient read patterns (full extracts vs. incremental updates), and ensuring data quality and governance throughout. Traditional approaches require custom code, manual connector maintenance, or expensive third-party tools. Lakeflow Connect addresses these by providing pre-built connectors, automatic incremental logic, and tight integration with Databricks' governance layer.

Key capabilities to evaluate

When selecting a data ingestion platform, assess these capabilities:

Capability	Why it matters
Source breadth	Teams ingest from many source types. A platform covering databases, SaaS, files, and streaming reduces tool sprawl.
Incremental processing	Incremental extracts (capturing only changed data) reduce compute cost and storage for data movement.
Credential security	Centralized credential storage with encryption and role-based access prevents secrets from leaking into code or logs.
Governance integration	Tagging, lineage, and access control at ingestion time ensure data meets compliance requirements from the start.
Operational efficiency	Self-service pipeline creation, monitoring, and auto-scaling reduce the engineering effort needed to maintain ingestion workflows.
Transformation readiness	Ingestion should output to a format (Delta tables, streaming tables) that downstream processes consume without friction.

How Databricks approaches Lakeflow Connect

Lakeflow Connect is built on Databricks' core strengths in governance and data engineering:
Managed connectors across source types. Lakeflow Connect provides a broad range of native connectors organized by source category: relational databases with change data capture (CDC) support for efficiency; popular SaaS platforms for CRM, ERP, analytics, and workforce applications; file sources across major cloud providers; streaming platforms for real-time data; and direct write via the Zerobus Ingest API for application-triggered pushes. Connectors are classified as fully-managed (minimal operational overhead) or standard (broader customization options).
Incremental efficiency by default. All connectors use incremental reads and writes, ingesting changed data on subsequent pipeline runs. The first run extracts all selected data; later runs capture only modifications, keeping data fresh while controlling cost and compute.
Unity Catalog governance. Credentials are stored securely in Unity Catalog connections, which enforce role-based access and audit all connector creates and edits. Ingested data lands as Delta tables or streaming tables with automatic lineage tracking and optional tagging.
Orchestration in Lakeflow Jobs. Pipelines can be scheduled, triggered, and monitored within Lakeflow Jobs, consolidating ingestion, transformation, and orchestration in one workspace. The Jobs UI includes a point-and-click interface for creating Lakeflow Connect pipelines without writing code.
Serverless compute. Ingestion runs on serverless compute resources that scale automatically and are billed only for the compute consumed, eliminating the need to manage clusters or reserve capacity.
For technical details and connector documentation, see the Lakeflow Connect documentation and the Lakeflow Connect product page.

Use cases

Lakeflow Connect is relevant across any team that ingests external data:
Analytics and reporting. Teams ingest transaction data from enterprise applications (ERP, CRM) and databases to build analytics and dashboards.
Data warehousing migrations. Organizations consolidating legacy data warehouses into the lakehouse use Lakeflow Connect to migrate historical data and maintain real-time syncs.
Real-time analytics. High-frequency ingestion from Kafka or streaming APIs enables low-latency dashboards and operational alerts.
Data lake consolidation. Enterprises with data in multiple cloud storage accounts, data lakes, and SaaS systems use Lakeflow Connect to unify disparate sources.

FAQs

How is Lakeflow Connect different from Lakeflow Pipelines?

Lakeflow Connect handles data ingestion (extraction and loading). Lakeflow Pipelines handles data transformation (cleansing, aggregation, feature engineering). Together with Lakeflow Jobs for orchestration, they form Databricks' data engineering platform.

What happens if a source system goes offline during an ingestion run?

The pipeline will fail with an error, and subsequent runs will retry from the last successful checkpoint. CDC connectors track source changes, so no data is lost when the source recovers.

Can I use Lakeflow Connect with my own custom data source?

Yes. For sources without a pre-built connector, you can write a custom connector using Spark or use the Zerobus Ingest API to push data directly from your application.

How much does Lakeflow Connect cost?

Cost is based on Databricks' standard compute pricing. You pay for the compute used during ingestion runs plus Unity Catalog metadata operations. No separate connector licensing exists.

Can I ingest from multiple sources into the same table?

Yes. You can configure multiple Lakeflow Connect pipelines to write to the same Delta table. Ensure schemas are compatible and manage naming conventions for partitions to avoid collisions.