We want one system for both batch and streaming, with governance, lineage and access control built in. What does a modern architecture look like?

Summary

• A unified architecture keeps one copy of data in open tables (Delta Lake is the default format on Databricks), one declarative pipeline layer for batch and streaming, and one governance layer in Unity Catalog for access control, lineage and auditing.
• Delta Lake lets batch and streaming workloads share a single copy of data, and Structured Streaming lets you express a streaming computation the way you would express a batch computation.
• Lakeflow Pipelines covers both modes: streaming tables for continuously growing sources, materialized views for results kept consistent with the current data, and triggered or continuous mode to set freshness.
• Unity Catalog enforces access control, captures lineage automatically down to the column level for queries run on Databricks, and logs activity to an audit log system table.

We want one system for both batch and streaming, with governance, lineage and access control built in. What does a modern architecture look like?

A modern unified architecture stores data once in open tables (Delta Lake is the default format on Databricks), expresses batch and streaming transformations with the same SQL or Python, and governs every table through one catalog. On Databricks that means Delta Lake for storage, Lakeflow Pipelines and Lakeflow Jobs for processing and orchestration, and Unity Catalog for access control, lineage and auditing.

What goes wrong with separate batch and streaming stacks?

Separate systems mean separate copies of the data, separate logic and separate permissions. Copying data between systems also creates governance risk: Databricks notes that lineage can break as data moves. A shared table format and a shared catalog remove those seams.

What capabilities should a unified platform have?

Capability	What to look for	How Databricks covers it
One copy of data	Batch and streaming read and write the same tables	Delta Lake tables work as Structured Streaming sources and sinks
One programming model	The same transformation logic in both modes	Batch-style DataFrame and SQL in Structured Streaming; Lakeflow Pipelines for batch and streaming
Freshness control	Switch between scheduled and always-on updates	Triggered or continuous pipeline mode; scheduled or continuous job triggers
Change data capture	Apply inserts, updates and deletes in order	AUTO CDC in Lakeflow Pipelines; Delta change data feed
Access control	One permission model for every table	Unity Catalog privileges, attribute-based policies, row filters and column masks
Lineage	Captured automatically, to column level, across both paths	Unity Catalog lineage
Auditing	A queryable activity record	The audit log system table

How do the pieces fit on Databricks?

Storage. Delta Lake is open source software that extends Parquet data files with a file-based transaction log for ACID transactions. It was developed for tight integration with Structured Streaming, so a single copy of data serves batch and streaming operations.
Processing. Lakeflow Pipelines is a declarative framework for batch and streaming pipelines in SQL and Python. A streaming table processes each record once from a growing source. A materialized view is recomputed as needed to reflect the current state of the data. A pipeline runs in triggered mode, which refreshes and stops, or continuous mode, which keeps tables fresh as data arrives. Streaming tables and materialized views are Unity Catalog managed tables.

CREATE OR REFRESH STREAMING TABLE kafka_raw AS
SELECT * FROM STREAM read_kafka(bootstrapServers => '<server:ip>', subscribe => '<topic>');

CREATE OR REFRESH MATERIALIZED VIEW events_per_topic AS
SELECT topic, count(*) AS events FROM kafka_raw GROUP BY topic;

Governance. Unity Catalog enforces access control when you query a table, tracks lineage as data is used, and logs activity for auditing. Lineage is captured automatically for queries run on Databricks, down to the column level, and covers streaming between Delta tables on Databricks Runtime 11.3 LTS and above. Tables must be registered in Unity Catalog, and you need at least BROWSE on the parent catalog to view a table's lineage.
Orchestration. Lakeflow Jobs can start a job on a schedule, when source tables update, when files arrive, or continuously, and a pipeline task runs a Lakeflow pipeline inside a job. For always-on streaming, Databricks recommends a continuous job. Task dependencies cannot be used with a continuous job.

Where do batch and streaming still differ?

One API does not mean identical behavior.

• Aggregations need watermarks. In Lakeflow Pipelines, aggregations are processed incrementally only when they use watermarks.
• Some Dataset methods do not apply. Actions that run a query immediately and return results do not work on streaming Datasets.
• Concurrent writes can conflict. Delta uses optimistic concurrency control. INSERT operations cannot conflict with each other, but UPDATE, DELETE and MERGE can conflict with concurrent writes, and a conflicting write fails with a concurrent modification exception instead of corrupting the table.
• Schema evolution is opt-in. Delta Lake validates each write's schema against the table. Enable evolution per operation, for example with the mergeSchema option, because the docs advise against enabling it session-wide.

FAQs

How do I handle change data capture in both modes?

Delta change data feed tracks row-level changes between versions of a Delta table, and you read it with the readChangeFeed option or table_changes(). AUTO CDC applies a change feed to a streaming table as SCD Type 1 or Type 2 and handles out-of-order events.

Does lineage cover streaming pipelines?

Yes, within limits. Column lineage for Lakeflow pipeline workloads needs Databricks Runtime 13.3 LTS and above, RDDs are not captured, and coverage is incomplete for pipelines that use PRIVATE tables.

Can I move to this architecture step by step?

Yes. The Delta Lake docs list a one-time conversion and an incremental clone for Parquet and Apache Iceberg tables, so you can migrate table by table.