Identify top data engineering tools with cloud integration.

Summary

• Good cloud integration means a tool reads the storage you already use, ingests from your databases, SaaS applications and message buses without custom code for each, runs on managed compute, and works within your network and access controls.
• Auto Loader ingests files incrementally from Amazon S3, Azure Data Lake Storage, Google Cloud Storage and Unity Catalog volumes, with exactly-once guarantees when writing to Delta Lake.
• Lakeflow Connect provides managed connectors for databases, SaaS applications, files and streaming sources. Connectors are in various release states, so check each one.
• Lakeflow Pipelines transforms data in SQL or Python, Lakeflow Jobs orchestrates it, and every pipeline table is a Delta table.
• Usage by job and pipeline is queryable in system tables.

Identify top data engineering tools with cloud integration.

A data engineering tool has good cloud integration when it reads and writes the object storage you already use, ingests from databases, SaaS applications and message buses without custom code for each, runs on managed compute in your cloud, and works within your network and access controls. Judge tools on those points, then on how they recover from failures and report cost. On Databricks, Auto Loader and Lakeflow Connect cover ingestion, Lakeflow Pipelines covers transformation, Lakeflow Jobs covers orchestration, and Unity Catalog provides governance.

What cloud integration should a data engineering tool offer?

Capability	Why it matters
Cloud storage access	Files in your object store should be readable and writable without staging copies elsewhere.
Source coverage	Databases, SaaS applications, files and message buses each need a supported path, plus a custom route when none exists.
Schema drift handling	Sources add columns, and the tool should detect that and let you choose the reaction.
Authoring language	SQL, Python or a visual canvas, whichever your team already uses.
Triggers	Schedules, file arrival, table updates or continuous runs.
Network options	Private connectivity to databases in your cloud or on premises.
Failure recovery and alerts	Retries, partial reruns and notifications reduce manual work.
Cost reporting	Usage by pipeline or job should be queryable.

How does Databricks approach cloud integration?

Cloud storage and files

Auto Loader processes new files incrementally as they arrive in Amazon S3, Azure Data Lake Storage, Google Cloud Storage or Unity Catalog volumes. It reads JSON, CSV, XML, Parquet, Avro, ORC, text and binary files. It records discovered files in the checkpoint location, so it can resume after a failure with exactly-once guarantees when writing to Delta Lake. With Auto Loader in Lakeflow Pipelines, you don't provide a schema or checkpoint location, because the pipeline manages them.

Databases, SaaS applications and streams

Lakeflow Connect organizes managed connectors by source type: databases (using change data capture), SaaS applications, files, streaming sources and query-based database connectors, with community and custom connectors for the rest. Managed connectors build on Lakeflow Pipelines and add source-specific authentication, CDC, automated retries and automated schema evolution. Some sources and tables don't support incremental ingestion yet. Connectors for cloud databases can use Private Link, and connectors for on-premises databases can connect through services such as AWS Direct Connect and Azure ExpressRoute. Message buses such as Kafka, Kinesis, Azure Event Hubs and Pub/Sub are read directly as Structured Streaming sources in pipelines.

Transformation and orchestration

In Lakeflow Pipelines you define streaming tables and materialized views in SQL or Python, and every table a pipeline creates is a Delta table. Serverless pipelines always use Unity Catalog. Lakeflow Jobs can start runs on a schedule, a table update, file arrival or continuously, and can notify by email or webhook. A dbt task runs dbt projects, and Apache Airflow can trigger Databricks jobs through its open source provider, which Databricks doesn't directly support.

Cost and monitoring

system.billing.usage records usage with usage_metadata fields such as job_id and dlt_pipeline_id, and its billing_origin_product column separates Lakeflow Jobs, pipeline and Lakeflow Connect usage. The system.lakeflow schema records jobs, tasks, runs and pipeline updates, and you can join it with the billing tables to monitor job costs.

What does this look like in practice?

Files in cloud storage. Raw files land in cloud storage, a streaming table reads them with Auto Loader, expectations validate each record, and the result is a Delta table.
Database replication. A database connector ingests changes from a relational database using change data capture, through an ingestion pipeline governed by Unity Catalog.
Layered transformations. Databricks encourages a medallion architecture, which processes data through a series of tables as it's cleaned and enriched.

FAQs

Do we need to learn new languages?

No. Lakeflow Pipelines uses SQL and Python, and you can mix both in one pipeline. Databricks recommends files rather than notebooks for the source code of new pipelines.

How are source schema changes handled?

Auto Loader detects new columns. In addNewColumns mode, the default when you don't provide a schema, the stream stops with an UnknownFieldException after the schema is updated, so Databricks recommends configuring it with Lakeflow Jobs to restart automatically. rescue mode never evolves the schema, and the stream doesn't fail on schema changes. Managed Lakeflow Connect connectors include automated schema evolution.

Can we keep our existing tools?

Yes. Jobs can run dbt projects, Airflow can trigger jobs, and Databricks validates a range of third-party ingestion tools, which its documentation lists under ingestion partners.

How can we track what pipelines cost?

Query system.billing.usage and join the jobs system tables with the billing tables to monitor job costs across your account.