Skip to main content

Introduction to Unified Data Engineering with Databricks Lakeflow

Summary

  • Databricks Lakeflow unifies ingestion, transformation, and orchestration on a single platform, spanning Lakeflow Connect, Spark Declarative Pipelines, Lakeflow Jobs, and Lakeflow Designer into one cohesive data engineering experience.
  • Declarative Pipelines within Lakeflow provide automatic CDC, fan-out, and materialized views, while Lakeflow Jobs delivers centralized orchestration observability across all pipeline types.
  • An end-to-end demo shows ingesting Salesforce data via Lakeflow Connect, authoring a pipeline with Genie Code, orchestrating workflows through Lakeflow Jobs, and enabling no-code data prep for business users via Lakeflow Designer.

Introduction to Unified Data Engineering with Databricks Lakeflow

Watch: Introduction to Unified Data Engineering with Databricks Lakeflow
Teams today struggle with pipeline fragility, brittle orchestration, and tool sprawl that slows delivery and increases maintenance costs. Lakeflow eliminates that friction by unifying ingestion, transformation, and orchestration on the Data Intelligence Platform, with reliability and governance built into the foundation. this video shows how Databricks Lakeflow turns Spark into the most advanced execution engine for data engineering on the planet— with agentic data pipelines underpinned by managed ingestion connectors, declarative transformations, native orchestration, and intelligent incremental processing built into execution, not layered on top.
Talk By: Ray Zhu, Director, Product Management, Databricks ;

Chapters

FAQs

What is Databricks Lakeflow?

Databricks Lakeflow is the unified data engineering product on the Databricks Data and AI platform that covers the full pipeline lifecycle: managed ingestion via Lakeflow Connect, declarative transformations via Spark Declarative Pipelines, native orchestration via Lakeflow Jobs, and no-code ETL via Lakeflow Designer. It replaces fragmented tool stacks with a single, governed platform.

What components make up Lakeflow?

Lakeflow consists of four main components: Lakeflow Connect for managed ingestion, Spark Declarative Pipelines for transformation with automatic CDC and fan-out, Lakeflow Jobs for orchestration with centralized observability, and Lakeflow Designer for no-code data prep by business users. Together they cover the end-to-end data engineering lifecycle without requiring separate tooling.

How does Lakeflow support AI and analytics workloads?

Lakeflow delivers the clean, fresh, governed data that AI models, Genie spaces, BI dashboards, and analytics applications depend on, and Genie Code enables AI-assisted pipeline authoring through natural language. The platform's declarative execution model also provides the reliable, automatically maintained foundation needed for agentic data pipeline development.

How do Lakeflow Declarative Pipelines handle incremental processing?

Lakeflow Declarative Pipelines apply proven execution semantics for incrementalization, change data capture, and fan-out automatically, so engineers focus on declaring the desired output rather than hand-coding stateful ETL patterns. This makes both batch and streaming pipelines easier to build, debug, and evolve as business requirements change over time.

Full transcript

[00:08] Uh what a crowd. Uh it's great that you guys choose me over a World Cup game that is going on. So that takes a lot. Thank you for doing this. Um that being said, before I get started, this is a intro session for Leglow. Before I I get
[00:20] started, uh can I have a quick show of hands? How many of you guys have already been using Leglow? Some of you guys. So that's great. most of you guys are in the in the right session. Um so this talk will have two
[00:32] two parts. Uh at the first part I'm just going to give you guys an overview of flake flow. Uh the original motivation why we decide to do that and uh the different components the different aspects of flake flow. Um some of that
[00:44] was already uh talked a bit about by um Lean Bilau at the keynote session this morning if you were there and if you paid attention. Uh so you might already know some of that. Uh but the second half um I recorded a demo uh to sort of
[00:56] walk you through all of these different capabilities within LakeFlow so that hopefully you know within with this time uh it'll give you a fairly good sense about like concretely how these product work how do they look like and how much
[01:08] of that you can go back and apply to your um uh applications and use cases. Um that being said let's kick that off. As mentioned I'm Ray I'm from the product team uh been with LakeFlow for the past four years. Uh and there's a
[01:22] survey in the end. I would appreciate if you guys uh fill it up so that we can keep improving. Um this is a slide that you guys already saw from below's talk this morning and because he stole my slide uh which is whether you are doing
[01:34] analytics whether you are doing AI whether you're doing applications everything starts with good data and lakeflow is the product for data bricks where covers the end to end data engineering capabilities so that your
[01:46] organizations your teams can have the good data to work on whether you are doing BI reporting whether you are doing genie whether you are doing AI machine learning training everything starts with data Now if you think of like the end to end
[01:59] data journey in the end you have the consumption ready data whether it's been pre-agregated cleaned up transformed with all the telemetry informations that you have but at the beginning it's actually fairly messy because typically
[02:11] you have a lot of different systems at an organization right like some of your data is living in the you know transactional databases that's powering your applications some of the data might live in some form of message buses
[02:23] you've been streaming from IoT devices Some of these lives in your SAS applications like Salesforce, like Mcketto. So in reality, you have a lot of these different data sources uh with you know um inconsistent data structure
[02:36] and format. You have to do a lot of work in between to get the data into a stage that is ready for consumption and typically that involves three steps. One is ingest which is sort of step number one, right? You're going to have to
[02:48] integrate all of these different data sources and bring these data into a unified open storage uh delta iceberg uh format. Then the next you're going to have to clean it up. You have to transform it. They have different data
[03:00] structure. Some of these are messy text strings. You're going to have to extract them out. And after that you have to aggregate them because for some of the the queries that you use very very often, chances are you want a better performance. You're going to have to aggregate them. Then finally you need
[03:13] something to tie all that stuff together, right? You have to pull the data in. Then you have to transform the data. Then you have to trigger the downstream workloads and that's where orchestration comes in. So orchestration the capability where you run schedule,
[03:26] you run triggers to tie all of these different workloads together and it provides a centralized observability and productionization feature so that all of your data pipelines can continue to run as is.
[03:38] And you know this might reflect to many of you guys sort of current data and AI application data stack which is it's it's actually doesn't look as clean as the previous slide that I showed. It's kind of very very messy. I'm sure like
[03:51] you have you know dozens of different toolings and you know you have legacy data integration tools like Informatica like data stage then later on you know if you have people who are familiar with a spark likely they built some spark
[04:04] based data processing layers. If you're doing streaming or real time, maybe you've been using Flink uh and you probably have some form of message bus, whether it's with Kafka, Kinesis, popsup um so and you know and lately like if
[04:18] you have SQL users, they probably have their own tool preference. DBT is very very popular um in that regard. And you likely also have now you know business users, analysts who wants to do data analytics. they probably have some form
[04:31] of data preparation tools uh like uh outrex like data IU. So you have all that messy architecture and structure and that is causing quite a few different problems. Uh number one is you lack a centralized governance layer
[04:44] because all of these different tools they they they were done by different vendors. They have different governance layer. They have different permission models. They have different access controls. So you don't really have a unified governance. And problem number two is that it takes a lot of time to
[04:57] integrate for these different data source, different tools to talk to each other. And what that means is that often times you are doing a lot of the data movement and cleaning and processing. Likely there's a lot of duplicate data across different parts of your organizations so that different users
[05:10] can have different consumption patterns. And finally, it's it's very fragile in a sense that if you have something that's not showing up in one tool, likely it's happening maybe three steps upstream. And because they all live in different
[05:22] systems, it takes a lot of time to debug and figure out what has gone wrong. And what that means is that now these I'm sure everybody you know many of you guys now you have developers, business users, data engineers now they're using AI or
[05:36] maybe the AI itself is create creating pipelines. What that means is that you're going to have 10x more pipelines putting on this system and what's going to happen is things going to fall apart. It'll put a lot of you know pressure on this sort of brittle infrastructure that
[05:49] all of you guys are running and maintaining today. And because all of these problems it's about three years ago we set out to do this lakeflow vision which is we think we should do one product to do end to-end data engineering fund ingestion
[06:01] transformation orration well integrated running on the exact same engine operate with the same governance model with unity catalog and operate on the exact same uh open data lake with delana
[06:13] iceberg. So we want to solve this problem to make sure you have a unified, scalable, stable, reliable infrastructure that can power the growth of your organizations. So that's what we did uh with LakeFlow. It's got a a few
[06:26] different components. Uh Lakeflow connect is what we use for simple data ingestion. Uh declarative pipelines. It's a abstraction we put on top of the Spark data frame. Uh it runs on the Spark engine to do large scale
[06:38] transformation aggregation. Then we have LakeFlow jobs that is our orchestration observability and productionization product and sit on top which we just announced uh GA this morning lakeflow designer that's a you know AI first UI
[06:53] based tool for business users analysts to do data preparation data pipelining without writing any code and uh all of this as I mentioned they run on the same exact engine governed by unity catalog
[07:05] and operate on the same uh open data formats um Let's dive in a little bit for each of these components. Uh LakeFlow connect uh it's got a porn and and click UI
[07:17] which I'm going to show you in the second half of the talk where I will do the demo. Uh it actually runs on declarative pipelines the the middle transformation layer but it has an abstraction layer so that it simplifies. All you need to do is configure a
[07:29] source, configure a destination and make some settings on schedule how you want the data to look like type one or type two. Then you're ready to go. Um and finally all of the data that's been ingested into uh delta iceberg table
[07:42] format is centrally governed by you see you can actually configure things like PII tagging on the way in so that once the data lands even before the data lands you have all the data
[07:54] classification and the ideal masking going on um in the in the delta lake. So you know layflow connect we announced uh two years ago and over the past year we focused on one thing which is you have a
[08:08] lot of different data sources. We focus on increase the number of data source coverage. Uh I believe last year when we were here we talked about 30 40 connectors. Now we have more than 100 connectors covering the broader spectrum of enterprise SAS applications,
[08:20] databases um event streams uh and cloud storage uh systems. declarative pipeline uh what it is I kind of briefly mentioned it's an
[08:33] abstraction layer sitting on top of spark initially we did this product was spark is very very powerful it's highly scalable for large a lot of data processing but actually not everybody knows spark especially you know as more
[08:45] and more users coming in to be a data practitioner the learning curve is still quite so high so we figured actually you know what we can do something more than the data frame API we can put all of these data in common patterns, things
[08:58] like uh DDUP, things like fan out, things like uh auto CDC, things like materialized views. All of these things are like actually very brittle, very hard to build, right? Build right in a sense that the data is always accurate and it's performant. So then we say
[09:11] like, you know, it's too hard for everybody to reinvent that wheel. Why don't we just do that? So we're going to offer a high level abstraction API sitting on top of data frame. So for all of these common data in patterns, we already covered that for you. So a very
[09:24] popular one is called um auto CDC. It's very simple. It sucks in uh change feeds and gives you the type one type two table. The challenge with uh CDC is always like for out of order. So because
[09:37] your change fees can be out of order, it's very hard to implement. So we basically turn a few hundred lines of code into like five lines of declarative API. So that's what the clear pipeline is about. And once you put all of these
[09:49] different processing logic into a data pipeline, it figures out uh the dependency graph for you and runs everything in the right execution order. So that is the cloud pipelines and the big news uh is last year we open sourced
[10:01] it. Uh we actually always plan to open source it but because we're so busy iterating on the product we couldn't do it. uh we heard it from a lot of customers like it's very very important uh to for organizations to operate on
[10:13] open format so that you know you don't have interlocking and you can do local development against open source project so we did exactly that and we continue to add more and more functionalities to the open source spark declarator pipeline project
[10:26] and uh we also build a purpose-built ID I don't know for those of you guys if you are familiar with the dbt looks very very similar um it's primarily designed as an end toend interface so you stay there. It's got everything you need to
[10:39] build a pipeline. I'm going to show you in the second half. You can see a DAG. You can see data preview. You can see your uh file tree structure on the right. You have Genie code that can help you author uh data pipelines. It's like
[10:51] a co-author as you develop the pipelines with you. And uh jobs it's our actually job is the the first product we did um at LakeFlow at the very beginning. people use that
[11:03] to orchestrate the spark based workloads and um and today not only jobs will orchestrate databicks workloads we've expanded it outside of data bricks so that even if you have workloads for
[11:15] example reverse etto at the tail end of data bricks maybe you need to send things back to your sales forceful systems or database systems we wanted jobs to have that end toend visibility and and and to see like the entire data flow end to end and manage that
[11:27] productionization capability uh for you and uh as I mentioned it's sort of our oldest product in the family and it's been well trusted and at running at a huge scale across different industry
[11:39] segments for different customers they all run their mission critical workloads using like flow jobs uh as productionization then designer uh it's super slick it's super cool actually the fun fact is that when
[11:52] we did designer uh the intention was that we think there are a lot of business users uh analysts they might not want to write code so we actually designed it as interface to to get people to do data preparation without writing code. But actually after we put
[12:05] it out there during public preview, there's a lot of data engineers actually end up using it. Uh, personally I'm a big fan of it. I've used it. It's very slick. You guys should give it a try. It's super easy. Um, there's no upfront configuration or anything like that. Um,
[12:18] and the team is actually adding capabilities so that you can flip back back and forth between the UI and the SQL code. So what what that means is that there are two things. One is that even for data engineers if you write SQL
[12:30] code you can actually use this as a productivity tool. It's not that you don't know how to write SQL but you can express the logic in a much easier way. Let it spit out the SQL code. The second things for you guys if if you are data engineers you have your business team
[12:42] writing these applications and data pipelines. Uh you probably want to review it to make sure that it's accurate it's performant and likely you don't want to review a graph of things. So you probably want to look at the code and and make changes to make it more
[12:54] performant and then you can flip back into the UI view. So pretty excited uh about this product. Um as I mentioned it's self-service you know there's no upfront configuration. If you're a platform team if you're a data team
[13:07] there's no like extra tax on your team to set it up for your downstream business users. They can just come into databicks workspace uh click a button and get going and uh production because again it runs
[13:19] on declarative pipeline. So a lot of the things that I just mentioned like autoc CDC fan out these different patterns it's got that uh accuracy and performance built in. So when you use that UI to build the pipelines it's
[13:31] going to translate into these declarative pipeline primitives under the hood. Um and finally as I mentioned uh it's AI native but we're working on a code view of it so that you can flip
[13:43] back and forth enable data engineers as as well as uh people to review the the data pipeline for performance and accuracy. Okay, so this is lake flow. It's unified everything. What that means is that now
[13:55] you have a entire end toend data flow. Everything governing unity catalog and against the same uh data. You no longer need to move data, do a bunch of copy and back and forth. Um everything's integrated out of
[14:08] the box which I'm going to show you uh in the demo. You don't need to do these peace meal like put this together, do this, then do that. We already done that work for you. And finally, it's reliable. Everything runs on top of the Spark engine. Even for the UI based uh
[14:21] tool, it runs on Spark engine. Everything translate into a performant, reliable and scalable uh engine under the hood. And uh you know now with this not only it enabled uh agentic data engine what
[14:34] that means is actually it means two things. One is that moving forward as you use AI to build data pipelines whether you are data engineers writing code or whether you are business users doing visual data preparations you can put all that extra load on top of this
[14:48] unified architecture it'll be reliable it'll be scalable and the second thing which personally I'm uh very excited about is that you guys heard about zero ops um this morning which is a forever dream for for data teams data user you
[15:00] don't want to wake up middle of the night what we're going to do because of this unified architecture we have end to end lineage and a visibility into what's going on with the system. Uh zero ops, there will be background agent that's running 24 by7 and keep watching all the
[15:12] metrics, errors, data quality, freshness, classification signals to figure out if anything is wrong. Um and it'll try to test the potential fix in a sandbox safe environment. So you as an
[15:24] operator when you when you wake up in the morning, you just pipe go in and see zerops already figured out you have these three potential issues. I have already tried to test the fix for you. It works. you should review and go fix that. Personally, I think it's
[15:36] gamechanging and uh none of that would have been possible if we couldn't do this unified stack with end to-handed visibility and everything centrally governed in UC. Okay, with that I'm going to pop back
[15:49] into a demo. I recorded a video. I'm going to walk you guys through so you know you get a concrete feel for everything I just talked about which is what are they uh really and um
[16:01] and by the upfront apologies I don't have very good video editing skills so I think the resolution is a little bit off um can you guys see fine folks at the back or maybe no
[16:13] okay I'm gonna oh no folks at front a little bit I'm going to do a verbal walk through so that hopefully folks at the back you'll get a you'll get a sense uh for what I'm talking about and showing
[16:25] on the screen. Um so this is the homepage. We're going to start with uh LakeFlow Connect. So this is a wall of like these different connectors that I I mentioned and we continue to add more and more support
[16:38] for it. And down here below does this thing has a pointer? Yeah, maybe we can do that. Oh, maybe it doesn't work.
[16:50] Uh never mind. I'm just going to use my uh mouse pointer down here below you see something called a community connect connector. So what we did is that we realize there like hundreds of different systems out there. People wanted to bring the data into data bricks. We
[17:03] going to do hundreds we can do 100 200 but we cannot get into hundreds or thousands. So we actually put a framework out there where users like you yourself and everybody in the community can build your own connectors on top of
[17:16] the lakeflow connect infrastructure and framework so that we can dramatically expand the connector coverage. Uh something might be specific for your organization. Um in this case uh I'm
[17:28] going to try to do Salesforce just for demonstration purpose. And here is the user interface. You start with something called a connection which is the connectivity to your Salesforce system. And you have you can do one of the two setups. One is I can create a connection
[17:42] on the go. Second is a lot of the organizations maybe you have an adaming that is managing the connection to the upstream data sources. So the adaming in this case for example has created these different connections uh for me. Then I
[17:54] can just pick and choose one of them to use. So in this case I'm going to pick the first one. It's called enterprise salesforce. So what is a connection? Um so this is a connection. It's a securable object governing in the Unity
[18:07] catalog. It includes the connectivity information and also OOTH uh token information so that it can connect to the system. Everything is securely governed in the Unity catalog. None of that is stored as a plain text. Um then
[18:20] let's go back. Um I'm going to pick this one. Click on next. Uh give it a name. And I can I give it a name. And in in this case also I want to pick where I want
[18:33] the data to get ingested into. So I'm going to pick a um catalog.
[18:46] so I had a duplicate pipeline name. So I'm I'm going to add a uh prefix to it. Uh click on next. So what's happening right now is behind the
[18:58] scenes it's uh spinning up a computer resource using the UC connection you just configured to fetch data from Salesforce. Nothing has been ingested yet. So you get a sense of the preview before you decide what you want to ingest. So in this case you can see on
[19:11] the screen there's a list of the schemas it's has it has fetched from Salesforce. You can choose to ingest every every single one of them. You can choose to ingest a part of them. And in this case for de demonstration purpose I'm just going to pick a specific one. I'm going
[19:24] to pick that account object um to do the ingestion. And here you can see a few different settings. You can change the name of the table. So maybe you don't want to call it account in the in the lakehouse. The second one is called
[19:36] historical tracking which is you can choose to have a type one table which mirrors whatever is in the source. You can also choose a type two table which keeps tracks the record of everything that has changed in the past.
[19:49] So I'm going to do that. Um pick my catalog, pick my schema and okay so here is another aspect in the in the last step you can configure a
[20:02] schedule. Um and this schedule actually behind the scenes is using lakeflow jobs. That's the unified aspect of it. Leg flow jobs is a scheduling triggering observability system. So when you create a connector, if you put on a schedule,
[20:15] it's managed by LakeFlow job behind the scenes and you can go to Leglow job to do the end toend observability and you can add other things uh into the job as part of the the configuration. In this case, I'm going to remove this schedule
[20:27] because at the end after I create a pipeline, I'm going to go create a job tying these two things together. So you get a sense for like how uh a job can or multiple different tasks. Um click save the pipeline. Get that
[20:39] going. Okay, so this is another aspect of what I call it unification. So this graph, you know, if you pay attention to what I when I talked about the cloud pipelines, it looks very very similar because it's exactly like flow pipelines running
[20:52] under the hood on the same engine. So whether you use connect, whether you use the cloud pipelines, actually everything runs at the same uh underlying engine including liquflow designer also runs on the same engine. So in this case I I picked to ingest one table and um it is
[21:06] get going uh declared pipeline is being created under the hood. Now it's trying to fetch uh the records and you can see that there are 13 rows that has been moved into my catalog as it account
[21:19] table. I can go to Unity catalog to make sure that um the table is there.
[21:34] So here's my catalog. Uh I pop in here's the schema that I created for this demo. There's a account table in here. you can look at sample data. So, and you can see like these are basically the 13 rows we just ingest from from
[21:46] Salesforce. So, that's uh connect and by the way I'm looking at the customer type because when when I do the pipeline part of it, I'm going to do some filtering with it. So next uh we're going to create a pipeline right so at this point
[21:59] we have Salesforce we bring some data as a raw account table into my catalog and my my schema and before the demo actually I created a separate table which is like a fact table that with all the sales records for each of the
[22:12] accounts so we actually have two tables right now one is the account table we just ingested the other one is a pre-created fact table about sales information for different accounts so now we're going to use uh pipelines to build some transformations
[22:24] Okay, so this one will be really hard to see I think especially for for folks at the back but I can verbally tell you the the panel information on the left hand side you have a file nav tree. So when
[22:36] you develop pipelines you have like a DAG of different tables different transformations. So it's not required but generally best practice is we put each of the transformation as a individual file. So when you go back you
[22:48] can look at each pipeline maps to your folder structure and underneath there there could be 10 20 30 different files each represent a particular um transformation in the middle that is your coding panel right so before the
[23:00] age of uh AI people basically you just type code you can validate the code and run the code but now with genie code my demo is going to show you I'm I'm purely going to use genie code and that's what's on the right hand panel um in
[23:13] this case uh let me fast forward a bit I So in this case um oops what I'm doing is that I'm actually
[23:25] giving it a very specific instruction because in this case I'm a data engineer. So I know exactly what I want to do. I know I have an account table. I know I have account sales table. I want uh declare a pipeline to perform a join
[23:38] first perform some filtering on the account table to filter out a specific account type. Then the second step I want to uh declare pipeline genie code to say that I want to join these two tables to perform that operation. So I
[23:51] I'll show you a different construct later on when I demo designer which you give a very different type of instruction. In this case I'm giving very specific instruction because I know what I want and here the code has been generated and you can accept and review.
[24:04] So one interesting part which is uh probably none of you can see uh the interesting line is actually not about the filtering because that's very easy to get right the interesting line is the skip change commit so in this case it's
[24:16] actually using a streaming engine under the hood so when you ingest from Salesforce you want to constantly pulling new data as they show up right and when I do the filtering it's also streaming from the account table that's being ingested and for streaming engine
[24:29] it requires a source to be append only so if you have any deletes or updates happening in the source table it's going to fail. It's always a best practice to to annotate and to say that if there's any change commits in my source table I
[24:42] want you to skip that only give me the new data and that is a very common mistake actually you know a lot of new users to spark streaming actually miss it so in this case because we've done all the skills data engineering uh
[24:54] skills with genie code behind the scenes it knows it knows that I should put this line of code in so all all of that best practice sort of built in uh on the get-go and you can also see on the right hand side. Um,
[25:07] note that I didn't tell it which key I want to join on, which one I want to filter on. I think it'll figure out the schema of these different tables when I perform the join and find out that actually account ID is the common key you want to perform the join on. So
[25:20] that's what um I'm going to do that and when we do the join I'm going to use a primitive called a materialized views. uh because for anytime if you have any transformation that involving relatively complex aggregations or joins
[25:33] materialized view is a much easier primitive to reason about uh than streaming. So that's exactly what I'm going to use here. Um in this case Genie Code uh is thinking about it. It's looking at the schema
[25:46] figure out the join key and it should spit out a code uh very soon. And as I mentioned, because it knows to follow best practice, it's not going to clob the code on this existing
[25:58] file, it's going to create a new file in the left nav tree with this new uh transformation.
[26:10] And that's that's one thing that I wish um you know, which will be uh constantly improving, which is when you call an LLM bot, it's kind of slow uh these days. It takes like 10 seconds, 5 seconds for it to think about. That's another reason
[26:22] why like we spend a lot of energy optimizing the the the skills for these agents behind the scenes so that it doesn't do a bunch of sort of discovery to iterate through what's the right thing to do. Uh but still it's work in
[26:34] progress. Um okay so in this case you know it spit out a new uh file with a new table to perform the join but in this case it simply does a join. There's no aggregation. It just look up the counting you know the count ID and do an
[26:47] inner join between the two tables. The next I'm going to say well I want to do a join but at the same time please do some do some aggregations in it. And finally there's one thing to call out on the right which uh is very very
[27:00] important is after genie coder spit out the code it actually doesn't just present the code to for you it's doing a feature called a validation behind the scenes. So what a validation does is it actually runs the code on the
[27:12] declarative uh pipeline engine but it it doesn't materialize the data. What it does is that it runs the code, analyze the data graph, fetch the schema to make sure that the code will execute but without really executing or materialize
[27:24] the data. What that means is that when Genie code spit out these code, it has already performed a lot of the necessary step validation. So your chance of like these code will just work after the authoring experience is very very high.
[27:38] So that's pretty unique. uh and in this case uh I hit run and you see at the bottom panel there's a data graph right so you have a streaming table which filters out a certain account type from
[27:50] the from the table we just ingested then we have two different materialized views one is performing simple join the other one performs a join with the aggregation um here now I'm going to um execute it
[28:09] and note that uh I ever told the system that run this table first once this table has been refreshed update these two tables because the fact that it's a declarative data frame so the system knows you're reading this data from that data so that this whole data graph is
[28:22] being built this is relatively simple one I've seen people being building data graph with like hundreds of tables so as you author these tables you just think about the data dependencies you don't need to specify them manually it'll done the data graph analysis for you and when
[28:34] we run that during execution it'll execute things in the right order and do small things like if you have a very complex stag of data graph if one of them fail you you want to stop everything downstream but you want to
[28:46] keep all the other independent DAGs nodes continue to run all of these are like you know good innovations that's not possible until we declare pipelines because now the system is aware of the data flow graph okay and by the in this
[28:59] case you can see that actually you can't see but uh I can see from here is it actually tells you another piece of visual validation it tells you like how many rows have been up updated uh by the by the engine. So in this case I'm
[29:13] getting seven rows for the first streaming table because remember that we ingest 13 rows. This one put a filter on it. I got seven rows for the plain join uh where we just do uh you know without any aggregation it's returning um 350
[29:29] rows because of the data from the the fact sales table. for the ones with the aggregation it's returning seven rows because it's joined it does a group I on the account ID which is why you can have this visual clue to say like well that looks right then you can um get going
[29:53] okay next what I'm going to do is that I'm going to do jobs right so we already created a lakeflow connect I have injection pipeline now I have a declarative pipeline that's done doing some filtering some aggregations. Now what I want to do is I want to put these two things together. I want to say that
[30:06] every time you first do the ingestion once you finish the ingestion I want you to run the pipeline for the aggregation and transformation and filtering. So that's where I come to jobs. This is our auctionation product. As I mentioned
[30:18] it's not only about data brick stuff and it's not only about like flow stuff. There's actually a variety of things you can orchestrate. You can even put like I want to refresh a BI dashboard at the tail end once the pipeline is run. You
[30:30] canrate everything in one uh place. So in this case I'm first going to select uh the ingestion Salesforce pipeline we just uh created.
[30:46] So we have that. Then I'm going to add the second declarative pipeline we just created. And note that uh down below you have quite a few different configurations uh which is very interesting for productionization. One is parameter. So
[30:58] you can parameterize your pipelines. For example um you can p parameterize the date like every time you run it which data from which date you want to ingest and you can actually use jobs to trigger
[31:10] the workload with the parameter passing down into the end to end of the flow. Uh that's very powerful feature. Second thing is that this is a simple one. I just have two tasks depend upon each other. But as your D gets complicated, you can specify logics like if one of
[31:24] your pipelines depend upon 10 upstream ingestion uh pipelines, you can say that only run me if all of them have finished or succeeded or only run me if two of them have been succeeded. So you can define all all of these different you
[31:37] know complex and flexible orchestration logics for what you need. Okay. So, I'm going to do that and then I'm going to start a task. Then I'm
[31:50] going to show you like why you want to do that um in in a in a job. Okay. Once I've connect these two nodes, then you need to give it the final thing which is when should I run and you have a few different ways to run it. One is that
[32:03] run it on a schedule. The most common one I run it at, you know, every day, every hour. The other one which is very very powerful, you can you can run it based on a trigger. And today we have table trigger. We have file arrival trigger which is very very powerful.
[32:16] Just imagine let's say you have a pipeline that's moving data from your cloud storage let's say S3 bucket. You can say I want to run it every day. But what if the the file like arrive just right after your trigger schedule? Then you're going to miss that. You're going
[32:29] to wait for another 24 hours. File arrival trigger basically means that the system is constantly looking at your uh at your cloud storage bucket. As long as soon as you have a new file that's being
[32:41] written into it, it'll trigger it. it'll trigger the the job to run. The other aspect the table arrival trigger for the same reason like if if your pipeline is reading and processing certain data from certain table instead of saying like run
[32:53] it every day you can say as soon as there are new tables sorry new data uh or changes to the data in the source table trigger this particular job. So you can do that very very powerful. Then finally for those of you who have uh low
[33:07] latency requirements you can configure a job to run continuously. Essentially, it's running 24 by7. As soon as there's new data, it'll pick it up and keep processing. So, that's the that's the low latency processing mode. But, of
[33:19] course, because you have a compute that's always up and running, um the cost will be higher than if you were to do trigger or uh scheduled uh fashion. So, in this case, I'm just going to do
[33:31] schedule. I'm going to run it uh once a day, which is now connecting back. Why I remove that uh schedule from the lakeflow connect? Because I'm going to do it here. this schedule is going to be put on both of these um two different task types. So this is uh we kind of
[33:44] showed orration. This is the other aspect of why you want to do that in jobs is you it gives you the the centralized observability. So let me go back a little bit. Um I think I missed the
[33:59] so it gives you the centralized observability. So in this case it used to be without an or auxation capability now you have two things you need to go to connect to look at the injection metrics has it been run how many rows have been ingested you need to go to the
[34:12] pipeline do the same thing but with jobs all of the tasks that you put in jobs it's got a centralized observability so in this case it shows me that the top bar is the job health status and the bottom two there are two like green
[34:25] boxes one of them is the ingestion task and the other one is a pipeline task. So you can do all of the observability in one place and if you click on uh the specific task type it takes you to the
[34:37] specific uh pipeline uh to show you the more details. So in this case you can pop in and see the tables that has been ingested. You can look at performance metrics and if I if I go back I can do
[34:49] the exact same thing to pop into the declar pipeline. In this case it's still running. Uh ingestion has finished. The pipeline is still running and you can monitor the status here.
[35:01] And there you go. It's it's executing now. It's finishing up with the rows that we expect to show up. And uh final thing to show at job is we shipped the new capability on top called job health.
[35:14] So whenever you have a bunch of tasks within a job often times when you do observability and troubleshooting if it fails it's actually very hard to pinpoint which one is is wrong or the
[35:26] other aspect which is all of your tasks are like perfectly running but your data is not fresh or the data volume showing up in your table has anomaly. So job health is actually an aggregated information for everything that's been
[35:38] orchestrated by this particular job. So when you go to a production page seeing a list of jobs, you can look at a health score to get a quick overview of like which one which job requires your attention without going through a lot of
[35:50] click and back and forth to figure that out. And uh finally I'm going to show you uh LakeFlow Designer. Um it's very exciting. Uh every time I I use and demo it, I think it's it's super cool. I
[36:03] highly encourage you guys go uh play with it. Okay, so the UI might look similar, but I'll tell you the subtle difference, which is on the left panel. If you remember when I was doing declarative pipelines,
[36:15] it's it's a it's a folder, a file tree with different files because you're authoring code. Each file is a code. In this case, since we're targeting analysts and business users, they're not thinking about code or a file. They're thinking about operators, which is why
[36:28] designer is centered around operators. It says that I want to join, I want to aggregate, I want to filter. So you can drag and drag these dra drag and drop these operators onto the middle canvas which is a DAG ccentric. If you think of
[36:40] the declarative pipeline it's co-cententric right the big part of the panel is the code in the designer case the big part of the panel is the DAG the visual DAG you can interact with and on the right hand side it's the same genie
[36:52] code. Um so this time I'm going to give a very different prompt. Remember like when I did the pipeline, I give a very specific instructions on I want to filter, I want a materialized view. Here I'm just going to say I have these three
[37:05] tables in my catalog and in my schema. Just go look at what's in it and create a random data flow for me. Um it's it's a very high level prompt with a different intent. But in real use cases,
[37:17] the business user will probably see that say something like I want to look at uh the trend of my sales information over the past 20 weeks and I want to look at these account by account. So you you
[37:29] express that in a very very different intent compared to the data engineers and designer is going to figure that out and create a data uh data prep visual flow uh for you. So I just kicked off that prompt. I just said, you know, I
[37:41] have these tables in my catalog. Just create a random data flow for me and see how that looks like. Um, all right. It's thinking it's doing it. Very soon, you should see a DAG showing
[37:54] up um in the middle panel. And at the bottom, you're going to see a preview capabilities, which I think is very, very handy for you to figure out whether it's doing the right thing as you intended. So, there you go. Uh, I didn't
[38:07] do anything. I just give a simple prompt. It's generating a data flow graph reading data from the two tables that I have in my catalog. It's doing some filtering, some joining, some aggregations. What I showed is to use
[38:19] the Genie code interface. You can also do drag and drop. And you can see there's a configuration panel. Uh you can do that as well if you want. And at the bottom, you can see for each of the operators, it shows you data preview
[38:31] what's before the operator, what's after the operator. So it's very very easy and handy for you to verify that it it is doing intended for what you wanted it to do. And uh then I'm going to show you one
[38:43] more thing which is you know instead of using Genie code you can also do the drag and drop and the cool thing is that uh it'll figure out based on where you place that operator node and connect the nodes for you. So in this case I already
[38:57] have some aggregation. I'm going to drag and drop a particular filtering node onto the canvas. So you see like there's a billing country. I think there's US, France. I'm
[39:09] going to do some filtering. So I'm going to drag and drop the filter node next to the canvas. Now because of proximity of where I'm putting that node, designer figured that out. I probably want to do the filtering on top of the aggregation node. So it's already connected for me.
[39:22] I don't need to manually connect the two dots. And in this case, I'm just going to type in I want to filter everything down to USA. And there you go. Apply it. Everything's good. And at this point, by
[39:34] the way, none of the middle nodes has been materialized in your catalog. Everything is sort of in the cache. You're iterating with it and you're verifying it. And in the end, you can materialize the final output as a
[39:46] materialized view uh in your catalog. And for that materialized view again because of the unification it's using the declarative pipeline engine under the hood. So every time when you need to schedule and run it again uh it'll do incremental computation in materialized
[39:58] views. And as a user you don't even need to know about it. And here the similarly you can uh that's what I'm doing. I'm going to configure a table name. Uh I want to materialize the data in my catalog in my
[40:12] demo schema. And I'm going to run it. And let's go to the lefth hand side. I refresh it. Now you have a materialized view which has just been created from LakeFlow designer. And
[40:25] similarly you can put a schedule on it and in this case uh of course it's actually running on the same uh legflow job for the orration capability and I can also go to the job we just created
[40:37] and to add this data flow at the end of the tail uh at the tail end of the note. Um this is it. Uh I want to pop back so thanks for coming. This is uh lakeflow overview. It unified the entire data
[40:50] engineering stack from ingestion transformation to orchestration observability and now even with a visual data prep tool sitting on top everything governing UC. Uh that's what I have. Uh thanks for coming. Um I'll hand around a
[41:04] little bit. I think I have a few minutes in case folks have questions. U thanks again for coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.