What's the best ETL tool for Databricks

Summary

• Lakeflow is the data engineering solution built into Databricks. Lakeflow Connect ingests, Lakeflow Pipelines transforms, Lakeflow Jobs orchestrates, and Lakeflow Designer gives analysts a visual canvas.
• Pipelines are declarative SQL or Python. Lakeflow Pipelines works out dependencies, orders and parallelizes execution, retries failures, and keeps materialized views current with incremental processing.
• Serverless jobs take compute management off your plate. Autoscaling and Photon are enabled automatically, Databricks upgrades the runtime for you, and standard performance mode uses fewer DBUs for jobs that can start more slowly.
• Event-based triggers keep data fresh. Table update and file arrival triggers start a job when its inputs change, not only on a fixed clock.
• Governance comes from Unity Catalog. Managed ingestion pipelines are governed by Unity Catalog, and lineage across jobs and pipelines is captured automatically down to the column level.

What's the best ETL tool for Databricks

For ETL that runs on Databricks, the native choice is Lakeflow. The docs describe it as an end-to-end data engineering solution for data engineers, software developers, SQL developers, analysts, and data scientists. Four parts cover the whole ETL path: Lakeflow Connect for ingestion, Lakeflow Pipelines for transformation, Lakeflow Jobs for orchestration, and Lakeflow Designer for visual data preparation. Because each part runs inside Databricks, the tables it produces can be governed by Unity Catalog alongside the rest of your data.

What does each part of Lakeflow do?

Component	ETL stage	What it handles
Lakeflow Connect	Extract and load	Database (change data capture), SaaS, file, query-based, and streaming connectors, plus community and custom connectors for sources without a built-in option.
Lakeflow Pipelines	Transform	Declarative batch and streaming pipelines in SQL and Python, built on Apache Spark Declarative Pipelines, using flows, streaming tables, materialized views, and sinks.
Lakeflow Jobs	Orchestrate	Tasks such as notebooks, Python scripts, SQL, pipelines, ingestion pipelines, and dbt, with schedules, triggers, branching, and loops.
Lakeflow Designer	Visual prep	A drag-and-drop canvas for analysts. All transformations are backed by code, and Genie Code can generate or refine steps from natural language.

How does Lakeflow Connect handle ingestion?

Managed connectors create ingestion pipelines that write to streaming tables. Those pipelines are governed by Unity Catalog and powered by serverless compute and Lakeflow pipelines. On the first run, a pipeline ingests all of the selected data from the source and tracks changes in parallel. Each later run uses that change tracking to ingest only what changed since the prior run, when possible. Database connectors add an ingestion gateway and staging storage to support continuous change capture.

How does Lakeflow Pipelines handle transformation?

You declare the tables you want, and the engine handles execution:

• Dependencies. It analyzes the dependencies of your defined objects and orchestrates their order of execution and parallelization automatically.
• Retries. Retry logic escalates from the Spark task, to the flow, to the entire pipeline.
• Incremental processing. Streaming tables process each record once, assuming an append-only source. An incremental processing engine keeps materialized views current by reprocessing only new or changed source data when feasible.
• Modes. A triggered pipeline refreshes available data and then stops. A continuous pipeline keeps tables fresh as new data arrives.
• Data quality. Expectations check each record. On a violation you can warn (the default, which keeps the record), drop the record, or fail the update. Metrics appear in the pipeline UI and the pipeline event log, and advanced patterns can quarantine invalid records.

How does Lakeflow Jobs orchestrate the work?

Lakeflow Jobs coordinates multiple tasks as part of a larger workflow.

• Triggers. Scheduled, table update, file arrival, continuous, or none, where runs start manually with Run now or programmatically from other orchestration tools.
• Control flow. Run if conditions on upstream outcomes, If/else tasks, For each loops, retries, and disabled tasks.
• Serverless compute. Autoscaling and Photon are enabled automatically, and Databricks upgrades the Databricks Runtime version for you. Performance optimized mode starts and runs faster for time-sensitive work. Standard performance mode uses the same SKU but consumes fewer DBUs, in exchange for higher startup latency.
• Monitoring. The Runs tab, the matrix view, and a timeline view for multi-task jobs show durations and dependencies. For serverless jobs, queries and query profiles are integrated into the timeline. Run history is kept for 60 days, and the system.lakeflow schema can be joined with billing tables to monitor job cost.

What do you gain by running ETL inside Databricks?

Unity Catalog captures lineage automatically, down to the column level. It shows which queries and files populate a table, which jobs and notebooks transform it, and which dashboards consume the results. One documented gap: jobs that use the Jobs API runs submit request or the spark submit task type are unavailable in lineage views.
For engineering practice, Declarative Automation Bundles (formerly Databricks Asset Bundles) define jobs and pipelines as source files. They support source control, code review, testing, and CI/CD, including bundle deployments started from a GitHub workflow.

FAQs

Can I keep using dbt?

Yes. The dbt task runs dbt projects from workspace folders or a remote Git repository. The SQL that dbt generates runs on a serverless or pro SQL warehouse, or you can choose dbt CLI compute to run dbt Core.

Can an external scheduler start Lakeflow Jobs?

Yes. Jobs can be started through the Jobs REST API, the Databricks CLI, the SDKs, or other orchestration tools, and the docs list Apache Airflow among the supported external options.

Should I pick performance optimized or standard mode?

Use performance optimized for time-sensitive workloads. Use standard performance mode for scheduled work that can tolerate slower startup, since it consumes fewer DBUs.

Do I need Lakeflow Designer?

Only if analysts want a visual canvas. Engineers can author the same transformations in SQL or Python with Lakeflow Pipelines, and Lakeflow Jobs can run Designer work as a Visual data prep task.