When does it make sense to replace my ETL stack with a no-pipeline architecture?

Summary

• A no-pipeline architecture removes the steps whose only job is copying data. It does not remove transformation, validation, or history tracking.
• On Databricks, three documented patterns skip a copy step: Lake Transactional/Analytical Processing (LTAP) for operational data in Lakebase, Lakehouse Federation for read-only queries against external databases, and Zerobus Ingest for writing events straight into Delta tables.
• Databricks still recommends ingestion for most use cases, because it scales to high data volumes, low-latency querying, and third-party API limits.
• Keep a declarative pipeline layer, such as Lakeflow Pipelines, when you need transformations, data quality rules, or change history.

When does it make sense to replace my ETL stack with a no-pipeline architecture?

It makes sense for the parts of your stack whose only job is to copy data from one system to another, when a Databricks capability can read the data where it already lives. It stops making sense when the pipeline also transforms, validates, or historizes the data, or when data volume and latency call for ingestion. Databricks' own guidance is that ingestion, which copies data, is recommended for most use cases.

What does no-pipeline mean on Databricks?

No-pipeline is not a Databricks product term. The closest documented patterns each remove a copy step:

• LTAP on Lakebase. The docs say LTAP "removes the change data capture (CDC), replication, and transformation pipelines that teams traditionally maintain to copy operational data into a separate analytical system." It applies to data your applications write to Lakebase.
• Lakehouse Federation. It allows you to query external data sources without moving your data, with read-only access through Unity Catalog foreign catalogs.
• Zerobus Ingest. A push-based streaming API that writes data straight into Unity Catalog Delta tables "with no message bus to run," so producers write directly to the table.

How do the options compare?

Pattern	What it removes	Documented limits
LTAP on Lakebase	The CDC and replication copy between a transactional and an analytical system	Applies to data in Lakebase; Lakehouse//RT querying Lakebase is Beta; Unity Catalog governs analytical access while Postgres privileges govern application connections
Lakehouse Federation	The ingestion job, for ad hoc access	Read-only; queries are pushed down and run on both Databricks and remote compute
Zerobus Ingest	The message bus and ingestion job between producers and the lakehouse	Producers write records to Delta tables; the gRPC, REST, and OpenTelemetry interfaces are GA
Lakeflow Connect managed connectors	Hand-written extraction code, though a pipeline remains	Connectors are in various release states
Lakeflow Pipelines	Manual orchestration of Spark and Structured Streaming code	You still define the transformations

When should I keep a pipeline?

• High volume or strict latency needs. Databricks recommends ingestion because it scales to accommodate high data volumes, low-latency querying, and third-party API limits. The trade-off the docs name is duplicate data that might become stale.
• Transformations. Lakeflow Pipelines is a declarative framework for batch and streaming pipelines in SQL and Python. It orchestrates flows automatically and retries transient failures progressively, from the Spark task to the flow to the entire pipeline.
• Data quality rules. Expectations validate data as it flows through a pipeline, and each can warn, drop the record, or fail the update.
• Change history. AUTO CDC handles out-of-order change events and supports SCD Type 1 and Type 2.

How should I start?

For ingestion, the docs say: "Databricks recommends starting with the most managed layer. If it doesn't satisfy your requirements (for example, if it doesn't support your data source), drop down to the next layer." The Lakeflow Connect overview describes fully-managed connectors as building on Lakeflow pipelines, which in turn build on Structured Streaming.

FAQs

Is Lakehouse Federation a replacement for ingestion?

Not for most use cases. Databricks recommends ingestion for most use cases and positions federation for ad hoc reporting, BI, and proofs of concept. When a source supports both federation and Lakeflow Connect, it recommends Lakeflow Connect if higher data volumes and lower latency are priorities.

Do I lose governance without a pipeline?

The documented patterns keep Unity Catalog in the path. Federated tables are governed through Unity Catalog foreign catalogs with table-level access controls. For registered Lakebase data, Unity Catalog applies permissions, lineage, and audit to the external compute that reads it, while application connections use Postgres privileges.

What about files and message buses?

Pipelines support all data sources available in Databricks. Auto Loader provides incremental, idempotent loading from cloud object storage, and pipelines can ingest directly from message buses such as Apache Kafka. When the destination is the lakehouse, the Zerobus Ingest docs call it the most direct path.

Which of these capabilities are generally available?

Release stages differ. Registering Lakebase in Unity Catalog and the Zerobus Ingest gRPC, REST, and OpenTelemetry interfaces are GA, while Lakehouse//RT querying Lakebase is Beta and Lakeflow Connect connectors are in various release states. Check each page before you plan around one.