Open-source data engineering

Summary

• Open-source data engineering means assembling pipelines from open-source projects instead of one vendor's platform. A common set is Apache Spark for processing, Delta Lake or Apache Iceberg for tables, Apache Kafka for event streams and Apache Airflow for orchestration.
• Structured Streaming lets you express a streaming computation the same way as a batch computation, so one Spark engine covers batch and incremental pipelines.
• Delta Lake and Apache Iceberg both document serializable isolation, time travel, row-level changes and Spark Structured Streaming support, so check engine, catalog and tooling support when you choose.
• Spark runs on its standalone cluster manager, Hadoop YARN or Kubernetes. On Databricks, Delta Lake is the default table format and Lakeflow Pipelines extend the open-source Spark Declarative Pipelines framework.

Open-source data engineering
Open-source data engineering means building pipelines from open-source projects rather than a single vendor's platform. A typical stack uses Apache Spark for processing, Delta Lake or Apache Iceberg for tables, Apache Kafka for event streams, Apache Airflow for orchestration and dbt for SQL transformation. You choose each layer and can self-manage it. Kafka's docs, for example, say you can choose between self-managing Kafka and using managed services from a variety of vendors.

What does an open-source data engineering stack include?

Layer	Projects	What the project documentation says
Processing	Apache Spark, Apache Flink	Spark is a unified analytics engine for large-scale data processing, with Spark SQL, pandas API on Spark, MLlib, GraphX and Structured Streaming. Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
Files and tables	Apache Parquet, Delta Lake, Apache Iceberg, Apache Hudi	Parquet is a column-oriented data file format. Delta Lake extends Parquet files with a file-based transaction log. Iceberg and Hudi are open table formats.
Event streaming	Apache Kafka, Kafka Streams	Kafka publishes, stores and processes streams of events. Kafka Streams is a client library for applications whose input and output data live in Kafka clusters.
Orchestration	Apache Airflow	An open-source platform for developing, scheduling and monitoring workflows, defined in Python.
Transformation	dbt	You write SQL select statements and dbt builds modular data models. Its docs list a free, fully open-source (Apache 2.0) distribution called dbt OSS.
Catalogs	Hive metastore, Unity Catalog OSS, Apache Atlas	The Hive metastore is a central repository of metadata. Unity Catalog OSS covers tables, files, functions and AI models. Atlas provides governance services for Hadoop.

How do batch and streaming work together in Spark?

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. You write the query as you would for static data, and Spark runs it incrementally as new data arrives. Delta Lake applies the same idea to storage: a Delta table is a batch table as well as a streaming source and sink. Iceberg also supports Spark Structured Streaming reads and writes.
This sketch reads a Kafka topic and appends it to a Delta table. It needs the Kafka connector and Delta Lake packages on the Spark classpath, plus Delta Lake's SparkSession configuration:

events = (spark.readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "host1:port1,host2:port2")
    .option("subscribe", "topic1")
    .load()
    .selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)"))

(events.writeStream
    .format("delta")
    .outputMode("append")
    .option("checkpointLocation", "/tmp/delta/events/_checkpoints/")
    .start("/tmp/delta/events"))

The checkpointLocation option tells Spark where to save the query's progress, such as the offsets processed in each trigger, so a restarted query continues where it left off.

How do Delta Lake and Iceberg compare?

Both are open projects. The table lists what each project's own documentation says, not performance, so choose by the engines and catalog you need to support.

Capability	Delta Lake	Apache Iceberg
Concurrency	Serializable isolation levels	Serializable isolation, with optimistic concurrency for multiple writers
History	Time travel for rollbacks and audit trails	Time travel and version rollback
Schema	Schema enforcement, plus automatic schema updates on write	Schema evolution: add, drop, update or rename
Row-level changes	Merge, update and delete	MERGE INTO, row updates and targeted deletes
Engines	Connectors for Apache Spark, Apache Flink and Apache Hive	Spark, Flink and Hive connectors maintained in the main repository

How does Databricks approach open-source data engineering?

Apache Spark is the technology powering compute clusters and SQL warehouses in Databricks. Delta Lake is the default table format, and Databricks originally developed the Delta Lake protocol and continues to contribute to the open-source project. Unity Catalog also supports Apache Iceberg tables.

• Lakeflow Pipelines extend and are interoperable with Spark Declarative Pipelines (SDP), the Apache Spark framework for batch and streaming pipelines in SQL and Python, and run on Databricks Runtime.
• Lakeflow Jobs orchestrate tasks, including pipelines, and integrate with external systems such as dbt. You can also trigger Databricks jobs from Airflow through the open-source Databricks provider, which Databricks does not directly support.
• Auto Loader provides a Structured Streaming source called cloudFiles that incrementally processes new files in cloud storage.
• Photon is a Databricks-native vectorized query engine that is compatible with Apache Spark APIs.
• Databricks Connect is built on open-source Spark Connect and is available for Python, R and Scala.

FAQs

Can I run these tools without Databricks?

Yes. Spark runs on its own standalone cluster manager, on Hadoop YARN or on Kubernetes, and Kafka can be self-managed.

What is the difference between Spark Declarative Pipelines and Lakeflow Pipelines?

SDP is the open-source Apache Spark framework. Lakeflow Pipelines extend it and add capabilities that SDP lacks. For example, the AUTO CDC APIs are not supported by Apache Spark Declarative Pipelines.

How do I choose between Delta Lake and Iceberg?

Check which engines and catalogs you need to support. Databricks supports both in Unity Catalog, and its Iceberg reads (UniForm) feature generates Iceberg metadata for Delta Lake tables so Iceberg clients can read them.

Can I use open-source clients with Databricks?

Yes. Databricks Connect, built on Spark Connect, lets you run Spark code from an IDE or application against Databricks compute.