Simplifying streaming pipelines with Lakeflow - Data Engineering with Databricks

Summary

• Simplified Streaming: Lakeflow makes it easy to build real-time pipelines using the same tools and APIs you already use for batch processing.
• Sub-100ms Latency: With Real-Time Mode, Photon, and serverless execution, Lakeflow Spark Structured Streaming delivers sub-100ms latencies at scale.
• Unified Governance: All streaming data stays in one unified system with built-in governance, orchestration, and monitoring through the Databricks Data and AI platform.

Simplifying streaming pipelines with Lakeflow - Data Engineering with Databricks

Databricks Lakeflow makes it easy to build real-time pipelines using the same tools and APIs you already use for batch. No need to learn a new system or manage separate stacks. With robust capabilities such as Real-Time Mode, Photon, and serverless execution, you can now achieve sub-100ms latency at scale — unlocking a whole new class of streaming applications.
Watch this video to learn how to simplify and scale your streaming data pipelines, and operate with confidence with built-in governance, orchestration, and monitoring.
Additional resources:
Watch 2025 Data + AI Summit session, A Deep Dive on Structured Streaming’s New TransformWithState API - https://youtu.be/ipWNpSVEZ7o?feature=shared
Learn more about Lakeflow Declarative Pipelines - https://www.databricks.com/product/data-engineering/lakeflow-declarative-pipelines
Blog: What’s New in Lakeflow Declarative Pipelines: July 2025 - https://www.databricks.com/blog/whats-new-lakeflow-declarative-pipelines-july-2025

Chapters

• 0:00 The explosion of real-time data
• 1:03 Challenges in data streaming
• 1:54 Spark structured streaming
• 2:36 Databricks lakeflow platform
• 3:26 Industry validation and usage
• 4:11 Technical demo of lakeflow
• 7:54 Key takeaways and next steps

FAQs

What challenges does Lakeflow solve for streaming data pipelines?

Lakeflow addresses three main challenges: it eliminates the need to learn new languages and APIs, it reduces complex operational tooling and monitoring setup, and it unifies real-time and historical data in one system for consistent governance.

What latencies can you achieve with Lakeflow Structured Streaming?

With Real-Time Mode, Photon, and serverless execution, Lakeflow can deliver sub-100ms latencies. This enables operational use cases where milliseconds matter, such as fraud detection or dynamic pricing.

How can you combine historical and streaming data for real-time predictions?

Lakeflow integrates batch and streaming seamlessly within Structured Streaming. Teams can join real-time event streams with historical context from Delta Lake, enabling real-time predictions with full historical context built into each decision.

What is the best way to run real-time data analytics?

Lakeflow on Databricks provides SQL-based real-time analytics that handles streaming data natively. It maintains consistency between batch and streaming results, ensuring analytics dashboards and reports always reflect true current state.

How does Structured Streaming compare to Apache Flink?

Structured Streaming provides SQL-based streaming on Spark with deep integration into Delta Lake and Databricks governance. While Flink is a specialized streaming engine, Structured Streaming offers simpler SQL interfaces and tighter Databricks integration.

Full transcript

0:00 The explosion of real-time data

Explosion of real-time data. This data is constantly being generated by every individual machine and organization. There are business transactions in operational systems, customer and partner interactions on the web, thirdparty data services in the cloud, and IoT data generated by sensors and devices. Even while sitting and watching this video, you're generating data. Real time data is everywhere. All this real-time data creates new opportunities to build innovative applications that do things like detect fraud, provide personalized offers to customers, dynamically adjust pricing in real time, predict when a machine or part is going to fail, and much more.
But implementing streaming has historically been really hard for most organizations. There have mainly been three challenges. It involves learning new languages and new APIs associated with specialized technologies and data platforms. This makes it extremely difficult for existing data practitioners to work with streaming data using the tools and languages they already know. It involves building complex operational tooling and stitching together different services to perform tasks such as monitoring and data quality testing, recovery and restart from failure, infrastructure scaling, and more.

1:03 Challenges in data streaming

This makes it incredibly difficult to deploy and maintain trusted, reliable, and efficient data streaming pipelines in your production environment. Finally, real-time data is typically in a separate system from historical data, which makes it really hard to govern and share data consistently in one place. That's where Spark structured streaming comes in. It lets you build streaming pipelines using the same APIs and tools you already use for batch.

1:54 Spark structured streaming

So, there's no need to learn a new system or manage two separate stacks. It's production ready with built-in fault tolerance and exactly once guarantees. And now with real-time mode, Spark structured streaming can deliver sub 100 millisecond latencies, making it possible to power operational use cases where milliseconds matter. It also supports rich stateful operations like joins, aggregations, and even per key custom logic, all with watermarking to manage state efficiently.
Spark structured streaming gives you the right foundation and datab bricks is the best place to run it in production. With LakeFlow, databicks provides an optimized Spark runtime with Photon and real-time mode for sub-second latency. All as part of a fully managed serverless platform that handles autoscaling, fault recovery, and infrastructure for you. It's deeply integrated with Unity catalog. So your streaming data is governed the same way as your batch data with consistent security, lineage, and audit controls across all workloads.

2:36 Databricks lakeflow platform

And with LakeFlow declarative pipelines, you get built-in orchestration, testing, and monitoring. No need to stitch together tools for reliability or observability. It's everything you need to build, deploy, and operate real-time pipelines with the power of Spark and the simplicity of a fully managed platform. Today, thousands of organizations are building streaming data applications on the databicks data intelligence platform from financial services and retail to manufacturing, healthcare, and more.

3:26 Industry validation and usage

And it's not just validated by customers. Independent analysts agree. Datab Bricks was named the leader of leaders in both the 2024 IDC Marketscape for worldwide analytic stream processing as well as the 2025 ISG provider lens for streaming data platforms. Whether you're detecting fraud in real time, powering personalized customer experiences or enabling AI on fresh data, databicks is the proven platform for streaming at scale.
Now, let me show you why databicks is the best place to run streaming pipelines. Here you can see a running pipeline in the IDE for data bricks engineering. We already pulled in data from multiple sources using late flow connect. Now this pipeline is taking that data, cleansing it and transforming it for downstream consumption. You actually don't need to know anything about streaming in order to get started. Are you more familiar with SQL?

4:11 Technical demo of lakeflow

Will want to do incremental data processing that automatically keeps track of where it left off. This code here declares a new table, incrementally pulls in raw data, checks for valid values, and then drops invalid rows. All you had to do was declare the logic and not worry about how it was going to be executed. Are you already familiar with Spark structured streaming? Then you'll recognize this code here. We're pulling in two tables and doing a stream static join.
Notice all we had to do was declare the logic and return a streaming data frame. That's it. No checkpoint, no write stream call. This is the beauty of declarative programming. Even your most complex use cases can be implemented. Yes, this is an arbitrary stateful operations call running inside lakeflow declarative pipelines. This pipeline is powered by structured streaming. So you automatically get the fault tolerance and exactly once guaranteed.
Notice the DAG running here on the right. LakeFlow Declarative Pipelines automatically detects the dependencies between your tables and shows you the lineage of your data. You can click on any of the tables to see sample data and custom metrics for that table. All of your tables will automatically be created inside the UC catalog and schema that you declare once and they will be managed just like all the other assets inside datab bricks.
You can also, if you choose, put each table in a different catalog and schema when you declare them. It is up to you. You also don't need to manage your compute. Just pick serverless and it is all done for you. It even includes an optimized autoscaling algorithm that automatically detects spiky workloads and behaves accordingly. Scaling down during periods of low activity and scaling up to handle bursts of data. The event log captures all the information you need to monitor your pipeline, including backlog metrics, processing time, and even quality metrics if you use expectations.
When you publish your event log to Unity catalog, you can create custom AIBI dashboards to centralize the monitoring of all your pipelines. No need to spin up and maintain an external tool. Now remember, Spark structured streaming has unified streaming and batch. Say that you no longer want to run this pipeline as a streaming pipeline continuously. You want a batch pipeline instead. Just click one button and set up any schedule you want in LakeFlow jobs.
That's it. Now you have a batch pipeline. No coding changes required. Do you already have Spark structured streaming jobs and don't want to change any code? Datab bricks is the best place to run Spark structured streaming. Just deploy them as LegFlow jobs. You get all the benefits, the governance, security and lineage of UC, the performance of Photon and the orchestration of LakeFlow jobs. In addition, you get the expertise of some of the top contributors to Spark structured streaming when you run your streaming pipelines on data bricks.
Here are the three points I want you to take away from this video. First, databicks makes it easy to build real-time pipelines using the same tools and APIs you already use for batch. No need to learn a new system or manage separate stacks. Second, with real time mode, photon and serverless execution, you can now achieve sub 100 millisecond latency at scale, unlocking a whole new class of streaming applications. And third, you can operate with confidence thanks to built-in governance, orchestration, and monitoring with LakeFlow.

7:54 Key takeaways and next steps

Are you ready to simplify and scale your streaming pipelines? Head over to databicks. com/legflow to get started. Thank you.

Learn more about the Databricks Data and AI platform.