Get Started Ingesting Data with Lakeflow Connect

Summary

Lakeflow Connect has expanded to over 100 native connectors spanning databases, SaaS applications, file sources such as SharePoint and Google Drive, and streaming systems including Kafka and RabbitMQ.
The ZeroBus Ingest feature enables sub-5-second change data capture latency, eliminating architecture hops and reducing the cost and complexity of near-real-time data pipelines.
An open-source framework allows teams to build, scale, and contribute their own connectors using the same robust foundation that powers Databricks-native connectors, with AI embedded skills for intelligent ingestion.

Get Started Ingesting Data with Lakeflow Connect

Thousands of organizations use Lakeflow Connect to turn raw data into actionable intelligence. In this video, we’ll show you how to streamline your ingestion strategy and accelerate your path to value. We’ll cover: Expanding the 80+ connector ecosystem: Explore what’s new across databases (e.g., Oracle, Postgres, SQL Server), applications (e.g., Salesforce, Workday, ServiceNow, Google Analytics), file sources (e.g., SharePoint, Google Drive), and message buses (e.g., Kafka, RabbitMQ). Introducing open-source tools: Learn how to build, scale, and even open source your own connectors using the same robust framework. Optimizing cost and performance: Discover how to slash costs and complexity by eliminating architecture hops with Zerobus Ingest. Take advantage of new cost and performance levers. And learn what we’re building behind the scenes to help. Join us and learn how to stop battling pipelines and start building value.
Talk By: Giselle Goicochea, Sr. Product Marketing Manager, Databricks, Inc. ; Peter Pogorski, Staff Product Manager, Databricks ;

Chapters

FAQs

What is Lakeflow Connect?

Lakeflow Connect is the managed ingestion solution on the Databricks Data and AI platform that connects enterprise data sources to the lakehouse through a growing ecosystem of native connectors. It covers databases like Oracle and Postgres, SaaS apps like Salesforce and Workday, file sources like SharePoint and Google Drive, and streaming systems.

What is ZeroBus Ingest and how does it reduce latency?

ZeroBus Ingest is a low-latency bridging capability within Lakeflow Connect that delivers sub-5-second change data capture by eliminating intermediary architecture hops. This reduces both the complexity and the cost of streaming data pipelines without requiring separate infrastructure.

How many connectors does Lakeflow Connect support?

Lakeflow Connect has expanded to more than 100 native connectors, covering databases, SaaS applications, managed file sources, and message buses. The ecosystem also supports open-source community connectors built with the same underlying framework as the native connectors.

Can I build my own connectors for Lakeflow Connect?

Yes — Databricks provides an open-source framework that allows teams to build and scale their own connectors using the same robust foundation as native Lakeflow Connect connectors. These community connectors can also incorporate AI embedded skills for more intelligent ingestion workflows.

Full transcript

[00:09] Hello everyone. I am so excited um because it is the last session of the day. Woohoo. >> Yes. No, that is appropriate. Thank you for that. Um we made it. So, let's get
[00:21] started. Um I I hope you're having a great day one. Um, you know, obviously this has been a very exciting summit. Um, so we are the last session of the day and you could have been in lots of
[00:33] other places, but I appreciate you making it here. Uh, if you wanted to learn more about what LakeFlow Connect is, what's new with Lake Flow Connect, and how to get started, you are in the
[00:45] right place. Uh, a quick formality uh, before we dig in. This is our forward-looking statement. Um basically we put this before every presentation uh but it is forformational purposes only
[01:01] but let's do some introductions. So my name is Jazelle Guacachea. I'm here with my colleague Peter Pagorski. Hey Peter just wave. Uh this is actually my third
[01:13] uh data and AI summit and I'm the lakeflow connect product marketing lead uh previously from AWS and Peter is my partner in crime as the staff product manager or leading databases.
[01:27] Okay, so here's a quick road map on today's session so you kind of know where we're headed. Um, I won't read all of this to you, but we'll start off with kind of the data landscape, what is
[01:41] LakeFlow Connect, the ecosystem. Uh, Peter's going to dig into some awesome demos, which we're excited to share with you, and some product news. Uh, we'll talk about our Zerobus Injust offering,
[01:55] which is very exciting for our streaming uh, friends, and then some customer success stories and wrap it up with a Q&A. We'll leave as much time as we can uh for Q&A at the end and we'll share
[02:07] some resources as well. Okay. So, let's look at today's data landscape. Now, when you probably saw Ali's keynote this morning um when he
[02:20] was talking about ingest, one of the key things was enterprise data is trapped across SAS apps, legacy systems, cloud storage. And what's happening is the data volume is completely outpacing
[02:33] traditional governance. So the stakes here are really high because reliable AI demands complete internal data. So if your AI is working from an incomplete foundation, it's going to hallucinate.
[02:46] You're not going to have like valuable insights and then you can't share that information downstream to like AI agents. So this reality creates three major roadblocks and data bricks has
[02:59] built a comprehensive uh approach to solve them. But let's talk about the problems. So the first thing is brittle ingestion um you know obviously manual ETL third party connections terrible
[03:11] right siloed data starves AI from of business logic and reasoning. So when agents are forced to guess, they produce untrustworthy outputs, which is very
[03:24] very terrible for all of um all of the teams that depend on this data. And then finally, engineering heavy ingestion processes create bottlenecks that delay time to insights. So let's talk about
[03:36] some good news. How do we solve this? Well, as you may have heard today, uh, we announced 100 plus native connectors, um, spanning martekch, adtech, um, enterprise knowledge. We'll talk about
[03:49] those a ton today. Um, having that unified data foundation for AI provides that enterprise context which is so crucial. Uh and then democratizing ingestion with a UI driven noode
[04:03] experience really helps unlock this for all of your uh organization. All right. So we say connect any source to your data and AI platform nearly
[04:17] every source. And here apart from being a cool animation uh it's truly amazing to see the growth of the connect ecosystem uh driven by incredible
[04:29] customer demand uh and engagement and this has just all been in the last year. All the logos you see here actually make up this announcement of 100 plus connectors on one platform.
[04:42] Now remember I said customer demand 3,000 plus organizations are already running on connect converting their raw data into decisions dashboards and AI.
[04:55] So what is connect? So for those of you not familiar how many of you have never used connect? Just to raise no worries, no judgment. Okay, you're in the right
[05:07] place. I'm so glad to hear that. Uh it is a built-in way to bring your data into the lakehouse. Now you can connect to nearly every popular data source all in one platform. You can choose from 100
[05:20] plus connectors and deploy via the flexible easy UI. So you see that little screenshot there. It's kind of representative of what you're going to be dealing with, but it's very similar. Um and you can choose from 100
[05:33] connectors. And by deploying it um you're also giving or providing highfidelity enterprise context and reliable pipelines to power that trusted AI agent reasoning.
[05:48] Okay. So now let's double click into what I mean when I say connectors. For many of you, you're just looking at, oh well, I've just got some Salesforce data or I've got, you know, data from SQL
[06:00] Server. Uh the truth is you've got data in all these different silos and so we like to think about these groupings. Um th this is what lakeflow connect supports. So SAS applications as you can
[06:12] imagine is ingesting data from popular enterprise sources like Salesforce, Workday, Google Analytics, Service Now. This is the power of connect. You're able to bring in all those uh data
[06:26] sources into your one lakehouse. Now um there is something we'll talk about later. Peter will actually talk about it. You see it says including community connectors. So in the cases where we
[06:39] don't offer a connector, there are community connectors. You'll learn more about that. Then there's database connectors. Obviously databases are so important and we actually cover both CDC
[06:51] and querybased. So in cases where you know CDC like SQL server Oracle you know in cases where um you don't have CDC access or you don't need it you just want to query um we also offer that so
[07:04] terror data um my SQL uh and Oracle as well. Now manage file sources this is such a cool um area of connect for us.
[07:16] It's actually one of my favorite um stories from customers that are using SharePoint and they're trying to ingest data and they're actually using uh intelligent document processing. So, we're working with different parts of
[07:29] the platform and they're able to scan. You know, one of our customers I'll talk about later is Panasonic and one of their business units had 7,000 contracts they needed to process and so they used
[07:41] AI parse um and all of the tools from intelligent document processing to ingest that data. We'll talk about that, but that is a huge area for us. um you know we hear from customers that they
[07:55] really really want to ingest all of these multiple unstructured data types and then finally there are zerobus ingest. So for those of you who haven't heard of zerobus um there are several sessions at summit about zerobus from
[08:09] our colleague Vicki um who really showcases this is a gamecher pushing high event data directly into the lakehouse without a message bus. So
[08:21] things like IoT, telemetry, clickstream data, like massive amounts of data. Um and they've had some several announcements that have been key um at Summit this time.
[08:34] Okay. So as we're thinking about this, I did want to share two examples. Um we're powering these critical industry use cases across these different connectors. So in cases where you've got like
[08:46] Martekch for example, you've got connectors that help solve the challenge of highly fragmented customer or campaign data. So if they centralize data from platforms like Meta or Google like so Meta Ads, Google Ads or Tik Tok
[09:00] or Dynamics 365 or HubSpot, teams can now build a unified view of the customer journey. So it's not just thinking about oh I'm just pulling in data like there is this really amazing use case around
[09:14] it. Um so for example in ad buying marketers frequently change budgets and bids. So with the Google ads connector they can capture this historical state which allows teams to accurately analyze
[09:26] the results against the exact platform state at that time rather than just the latest overwritten data. And then the other use case around enterprise knowledge. So I did talk a little bit
[09:38] about this already uh with share your point but frankly these connectors solve the challenge of unlocking some of the organizational knowledge um whether it's you know in Google sheets or excel um
[09:50] these customers can now seamlessly sync their data directly to delta tables which has just been a gamecher.
[10:03] Okay. Now finally I did want to close with this very important note. So for many organizations data silos in SAS apps and operational databases prevent agents from reasoning. We talked about this already. So when you say I want to complete like enterprise memory for AI
[10:17] agents with Genie and LakeFlow connect you can get production grade ingestion from Salesforce, Netswuite, Dynamics 365, Workday and more. And you can really ask Genie anything um once you
[10:31] have all of the data in and feel confident that it's all trusted data um and you have full um context or enterprise memory.
[10:45] And then the last thing um one more thing we're also making it easier for organizations to permanently lower the TCO of high volume ingestion with the lakeflow connect free tier. So all the wonderful connectors we talked about, you know, all of that massive scale that
[10:58] we've expanded in the last year, we've heard from customers this is so important to connect their data. So just keep in mind um you get 100 DBUs per
[11:10] workspace per day and that kind of is equivalent to 100 million rows. Uh that's going to depend on your data characteristics and the source system, things like row size, schema width, connector type, but that's really
[11:24] exciting for people who just want to test it out and see how it um helps them bring enterprise data into their lakehouse. Now, I'm going to switch gears here a little bit uh just briefly to talk about
[11:37] zerobusting inest. Um this is direct streaming to your unified data platform. Now we're offering via gRPC which was the core uh product release rest hotel
[11:51] and now Kofco compatible APIs. So let's talk about what zero zerobus and just uh key benefits are. I'll just briefly go through this but for those of you that are not familiar the diagram I think is
[12:04] going to help. So the first thing is performance at scale. Uh near realtime writes in under 5 seconds and high throughput up to 100 megabits per second. Um but performance only matters if your producers can connect without
[12:17] friction. So a migration should be as simple as a config change. So since reaching general availability earlier this year, they actually have expanded to meet data producers where they
[12:29] already operate. So, if you follow along here on the diagram, you've got things like IoT devices um or backend services or mobile apps. These are all connecting
[12:41] through different interfaces and generating massive amounts of data. And so, the performance of bringing all the data in is huge because your existing Kafka producers now through the Kafka compatible um APIs can push data
[12:56] straight into data bricks. Before they used to have oh you know the message bus and everything else when you had hotel, gRPC and rest but all of that this multi-interface flexibility provides
[13:09] that direct low latency bridge uh to the cloud for global enterprises. One of the customers using um zerobus is meta and we'll talk about them at the end.
[13:21] And then for Zerobus, one case study we'll share with you, uh, Toyota Motor Corporation, they're the world's largest automaker. They're streaming energy efficiency data from devices on their shop floor into data bricks using zero
[13:34] busing and so they're actually powering predictive energy optimization to hit their sustainability targets. So they needed to cool their factories uh in a
[13:48] way that was you know reactive uh responsive not reactive and they were able to use zero bus ingest to support their carbon neutrality strategy. So it's not really a data story, it's actually an operations story and a
[14:01] sustainability story. And so the impact of using zerobus ingest uh has been tremendous for Toyota. And this is across all their factories uh in Japan. And then this was the example I
[14:14] mentioned for Meta. Uh they're actually using Zerobus and just to bridge its on premises data centers to the cloud uh enabling rapid development of datadriven solutions at scale. So enough slides for now. We're going to take a break and go
[14:28] into a live demo and actually see how LakeFlow Connect works in action. Peter, you have it ready to go, right?
[14:40] >> Yep. >> Awesome. >> Well, thanks Jazelle. >> You're welcome. >> So, now that Jazelle walked through what LakeFlow Connect is, why did we build it, and what we offer, I'm going to actually show LakeFlow Connect in
[14:52] action. And for this demo, I'm going to be presenting from the scenario of a soccer club called Spark City FC. And they're an underdog team. And against
[15:04] all odds, they've made it to the finals against the Rovers on Saturday. And there's two critical people that are under a lot of pressure right now. We have Coach Larott. She's the head coach
[15:16] and she needs to pick a lineup that's going to give them the best shot at winning. And then we have Coach Mustache, who's her assistant, and he's the one who actually has to go pull data
[15:28] out of all of their different silos and put the information together in order to give Coach Lariat a good answer. And the problem is their data is spread out amongst a bunch of different systems
[15:41] that don't actually talk to each other. So what does Coach Larott actually want to know? She wants to know three things. Who should start on Saturday? Is there anyone that's at risk to play? So, has
[15:55] there been anyone running hot in training or is there anyone recovering from an injury? And then, what have we actually done against the rovers in the past that's been successful? And these
[16:07] are really hard questions to answer when the data needed to answer them lives in the three different silos. So, here's where that data actually lives. We have Salesforce which has the
[16:19] player administration and scouting reports. We have Oracle which has the operational data. This is going to include every match, every training session and all of the medical history
[16:31] of the players. And then finally, our analysts keep the match history in Excel which they upload every day to SharePoint. And this is a very common s scenario amongst most organizations
[16:44] which have a CRM, they have a database and then a bunch of spreadsheets that their analysts work with every day. And answering one question often means thinking about how you're going to join
[16:57] all of that data or asking one of your co-workers to go do that for you. And now if you want to build an ingestion solution yourself, here's what that would actually look like. You'd
[17:09] probably be writing a custom pipeline for every single source and this has a lot of challenges associated with it. You'd need to understand the API of the different clients, how change capture
[17:22] works, and then how to actually configure your streaming jobs. And then once you've actually built it, you're stuck maintaining it. So it takes months of work to build, and it never really stops being work.
[17:35] But here's that same picture with LakeFlow Connect. All of that plumbing in the middle is just gone. You simply pick a source, set up a connection, choose a schedule, and data bricks is
[17:47] going to handle the rest. Like Jazelle mentioned, we now have over a 100 connectors available with new ones launching every single month. And the data is going to stay fresh. Everything lands in Unity catalog governed right
[18:01] from day one. So, let me show you just how easy it is to actually bring in data. I'm going to start off by ingesting from one of the most common SAS applications,
[18:13] Salesforce. And I'm starting on the add data page. This is where I can see all of the sources which are available for me to ingest. We have a ton of connectors available in the UI and then
[18:25] even more of them available through the API. So, I'm going to go ahead and pick a connection to Salesforce. If I didn't already have one set up, I could quickly and easily create a new one through
[18:37] OOTH. Then I'll give the pipeline a name. I'll call it Spark City FC Salesforce pipeline. And then I'm going to need to tell it where to keep the event logs. And this is where all of the run history and logs
[18:51] from the pipeline live. So I'm going to drop those in the Spark City FC catalog. Once I do that, I'm going to have to pick the data that I actually want to pull in from Salesforce.
[19:03] And in this case, I'm going to search the catalog for Salesforce, which is a prefix on the name of the custom objects we have. So for our lineup question, there's really three things I care
[19:15] about, which is going to be player availability, player contracts, and rival player profiles. So I'm going to grab those three objects. There's also additional features I could enable like
[19:27] column selection. So if I had sensitive data on one of those columns, I could choose to exclude it. Then I'm going to pick where the data lands within my workspace. So I'd already set up a catalog and schema for Salesforce data
[19:40] ahead of time. So I'll just point it there. And then finally, I'll move on to choosing a schedule for my pipeline. For Salesforce, by default, we set a
[19:52] six-hour schedule, but you can modify this to be basically any interval you would like. And with that, I can run the pipeline. And now we can start to see the data successfully ingested from
[20:04] Salesforce into our catalog. So now we have Salesforce ingested. I didn't have to write any code. I was just able to go right through the UI.
[20:16] Next up, I'm going to ingest from the Oracle database. But before I do that, there's one decision I need to make with in relation to databases, and that's how I should pick up the changes from the
[20:28] source. And LakeFlow Connect has two different options. Native CDC, which is going to read the database's change log. It's going to catch every single historical event on the database, and it's really efficient. The other option
[20:42] is querybased capture. And querybased capture is simple because it just watches a cursor column on the source. And it's going to pull in whatever is new since the last pipeline run without you needing to actually modify any
[20:55] setting or do anything on the database whatsoever. So with LakeFlow Connect, you have the flexibility to choose the best data extraction option that actually fits your needs.
[21:12] So now I'll navigate into the Oracle connector setup. First, I'll select an existing connection to an Oracle database. Though, just like with Salesforce, I could easily set up a new one right in the wizard. And then I'm going to be presented with
[21:25] two options for how I want to ingest the data. The first is the native C native CDC, which is reading the logs from the source database to capture capture inserts and updates as they happen. And
[21:38] the second is going to be query based capture which is using the cursor column on the table to incrementally pull in new and updated records. So a few things you want to consider when making this choice are how much load can you put on
[21:52] your source database, how fresh do you actually need your data to be and whether you have the database permissions to read the CDC logs. So for this one I went with query based capture and I'm giving the pipeline a name uh
[22:05] sparkfc oracle pipeline. Once again I pick an event log location just like on Salesforce where the logs and the events from the pipeline get stored. And then with querybased capture I have the
[22:18] option to choose between using classic or serverless compute. And you're usually going to want to think about your networking connection to your source database when you're making a choice here. But in general, we would
[22:31] always recommend going with serless since the since it's going to give you the fastest and most seamless experience. Next, I'll need to enter a service name for an Oracle database, which is going
[22:43] to let it fetch the available schemas and tables for me to ingest. And then I just need to select the tables that I want to ingest from the database. And then like I mentioned, if I want to
[22:55] ingest a table incrementally, I just select a cursor column and then I can select a primary key or have it automatically inferred for me. And I can also choose the batch in justest
[23:07] instead, which would do a full load of the table on each pipeline run. And that's especially useful if you have a table without any cursor column and you want to get started very quickly. Once
[23:19] again, I'll pick a target destination schema where the data is replicated into. And here I can also choose a schedule for the pipeline, but once an hour is going to be plenty for this Oracle
[23:31] database use case. And now that I'm running the pipeline, we can once again see the data flowing in from Oracle. And I was able to do this without having to modify anything
[23:43] on my database because I was using querybased capture. So there's a bunch of other notable lakeflow connect features which I didn't
[23:55] highlight in the demo but can actually be enabled on your lakeflow connect pipelines. So you can do things like um capture std type 2 history if you want the full history and audit log of all of
[24:08] your changes. You can use column selection to filter out data or you can use row filtering on uh SAS and querybased capture pipelines to pull only the specific data you need. And
[24:21] then we also have automatic schema evolution and failure recovery built in on every pipeline. So far we've covered the SAS applications and databases. But just
[24:34] like most organizations, some of SparkFC FC's most important data lives in files. They use spreadsheets, PDFs, and documents. And this is where most of the day-to-day operations are actually
[24:47] updated by the analysts. So, LakeFlow Connect can pull in these documents straight from SharePoint or Google Drive. And spreadsheets and CSVs will get parsed into tables automatically.
[24:59] Well, for PDFs and images, there's an AI function that can pull out the text, um, tables. It can also even pull out diagrams. But in the case of SparkFC, we're going to focus on the Excel
[25:12] workbooks. So, here I'm going to ingest an Excel file from SharePoint. And I'm starting off by showing you the SharePoint site itself. So, this is where analysts are
[25:24] storing and updating their Excel file after each game. And then inside that Excel file, they're keeping results against their rival teams along with notes about how each game played out.
[25:37] And just like most organizations, SparkFC is managing and updating their most critical data in Excel. So I can bring that in and I'm going to click on the SharePoint button to get
[25:49] started. I'll pick a connection and on the next step once again give a pipeline name and choose an event log location.
[26:03] Then I can choose the type of data I want to ingest. Either structured data or unstructured data like PDFs and images. We support a very wide variety of file formats here. So you can bring in almost all of your SharePoint data.
[26:15] Then I'm going to paste in a link to the SharePoint site itself. And this is going to fetch all of the different folders and files that I could ingest from the source. And I can choose to ingest multiple files at once or I can
[26:29] ingest one at a time. And I'll select the file format in this case Excel. Finally, I'll pick the destination catalog where the tables will be created.
[26:43] And you can also choose another schedule for this pipeline. So once I run the pipeline, I can see the data actually coming in from SharePoint. And we can navigate into the catalog and we can see our delta table is formatted exactly
[26:57] like our Excel file was. So we have all of our data now in the LOS. And just like that, we removed all three
[27:09] silos. Everything's in one catalog now. It's governed so we can make sure only the right people can see the right data. It's fresh because every pipeline is syncing on its own schedule and it's all
[27:22] in one place with the business context attached which means now we can finally ask the challenging questions that only our data can answer.
[27:37] So let let's now open Genie which is going to have all of the context from the data we just ingested and we play the Rovers on Saturday. So, I want to find out what the best lineup would be, as well as formation and plays that have worked best against them in the past.
[27:50] So, I'm gonna ask Genie this question. And while Genie is working, let me explain what's happening behind the scenes to answer the questions about Saturday's game. We needed the data from
[28:02] three completely different systems. And we just brought that all together from Salesforce, Oracle, and SharePoint into the govern lakehouse with LakeFlow Connect. And this creates a governed
[28:14] semantic semantic layer which is what makes Genie super powerful. Genie is drawing on the full context of that semantic layer, the business definitions, the relationships and the
[28:26] metadata across all three sources. This enables it to interpret my question the way the soccer club actually thinks about its data. And because new data is continuously ingested, each time I go to
[28:39] Genie, I'm going to get answers that are relevant to the soccer club right now and not at some point in time in the past. It's always going to be fresh. And then with that, we should see Genie
[28:53] coming up with an answer. So we can see that Genie says we should play 433 formation. And it's going to give me a list of the best players which are available to play. This previously would have taken hours to pull the data from
[29:06] all three sources and join it together. But now we can quickly get our questions answered. So just to recap, LakeFlow Connect keeps the data flowing in and Genie turns it
[29:19] into answers anyone can ask in plain language. The questions that used to be a major challenge are now answered with a quick conversation. And because Genie is using our organization's latest data as context,
[29:32] we're always going to get answers with what's true right now and what's actually specific to our own organization.
[29:44] And now that I went through the demo, I'm just going to go over a few cool new features that we're offering in Linkflow Connect. The first is community connectors which Shazelle briefly mentioned earlier and we talked a lot about the breadth of our
[29:57] connectors. But we know our customers still have different proprietary systems or they have an urgent need to ingest from a niche system that we don't currently support. And for this we built
[30:09] community connectors. They're open- source connectors with templates and AI embedded skills that enable you to build connectors to almost any source. and they have the same unified management
[30:21] experience of LakeFlow Connect. And there's really four key things to know about them. The first is how these connectors are built. They run on top of Spark's Python data source API. And
[30:33] there's a simple LakeFlow connect interface with AI provided skills that can easily be implemented. The skills are optimized for REST API sources, but
[30:45] you can build a connector to basically any source. Second is going to be governance. So your credentials with the community connector are going to live in secrets under Unity catalog. You're
[30:57] going to get the you see permissions audit and full lineage. Third is observability. And the nice part here is that there's nothing special to learn. Your custom connector
[31:10] is going to get the exact same experience as any other lakeflow pipeline. It's going to get the same UI, the same event logs, the same metrics, and the same automatic retries. And then finally, it's deployment. Once you build
[31:23] one of these community connectors, you can deploy them directly from the AD data UI in your data bricks workspace. So this makes it really easy for anyone in your organization to get started.
[31:37] Next, for our database connectors, we're introducing a new simplified architecture, which we're calling the integrated CDC connector. And this is currently in beta. If you've used our CDC connectors in the past, you'll know
[31:51] that you previously had to run two different pipeline types. One which was a gateway to extract the data and another which was an ingestion pipeline which would apply the data into target
[32:03] tables. And the integrated con uh integrated CDC connector f folds all of that logic into a single pipeline. It's easier to deploy and manage with all of
[32:15] the great functionality of existing database connectors. So with it, we're introducing the ability to fully schedule the pipeline. So you don't always need to have a gateway component running continuously. And you can choose
[32:28] to run it fully on classic or serverless compute. And then for customers that have specific needs that only allow for inbound connections to their on-remise database, we wanted to provide a
[32:41] solution where the connections actually initiated within the onremise data center. So we introduced this reverse SSH tunnel which is a small host inside the data center that dials outbound on
[32:55] port 22 and lands in a proxy VM within the datab bricks workspace VPC. So, this lets LakeFlow Connect ingest from the database without you ever needing to open an inbound firewall rule on your
[33:08] onremise networking environment. And with that, I'll hand it back to Jazelle to go through some of our awesome customer case studies.
[33:29] >> Hear me? Okay. Yes, that was awesome. Thank you, Peter. Um, love the demo and love that you're able to see uh all of the wonderful uh ways we can connect uh to these systems. So, I'm looking at time and I want to make sure we have enough time for questions. So, I want to
[33:43] very quickly just showcase one specific client and ironically enough they were here um two sessions ago in this very room. Uh Panasonic, there is a QR code
[33:55] because I know we're short on time. So please, please, please take a look at that case study that was recently published. Now, Panasonic is one of these customers where, you know, obviously they're a global brand. Um, but their specific central data
[34:09] infrastructure team serves as the data backbone for multiple business units spanning sales, supply chain, and HR. And so they had fragmented legacy pipelines and they'll be the first to
[34:22] tell you which they did. Um complex data silos and they had frequent outages and severely delayed critical daily reporting. Um specifically some massive SAP tables. They had some other things
[34:36] with SFTP SharePoint. They standardized on the datab bricks platform using LakeFlow connect to automate ingestion from all of those sources replacing brittle legacy ETL with a unified
[34:49] governed data foundation and some of the impact is there. So literally from hours to minutes that largest SAP table um and it was actually using one of the recommended approaches from SAP using
[35:02] autoloader um to ingest uh from hours literally to minutes. Um, so this is definitely a great one to look at and this is a quote uh 50% faster reports
[35:14] and then some resources which I think is really important for those of you who are not familiar. Obviously we have our technical documentation, we have case studies, we have blogs, but for many of
[35:26] you who want to dig deep into the actual you know um you know development rather data bicks community if you are not already familiar please just snap that QR code. Um we have amazing resources
[35:41] and a thriving community that has grown even in just the last year where we have contributions from partners from people in the community with blogs. Um there's a lot of um uh commenting and
[35:53] coordination with uh data bricks. So definitely check that out. And then two other um sections I'm going to share here. Since we're Tuesday, there's a lot of other sessions. Uh we've been talking
[36:05] about unstructured data ingestion. Please check out um Jason's session, but Peter uh and Sonia have another session for those of you who want to dig in. Um so it's more of an L200 um session for
[36:19] going under the hood, not just for databases, applications um but also file sources. Uh and then Vicki is doing a very very awesome deep dive. I've seen the demos uh for zerobus injust on
[36:32] Thursday. And then those customer-led sessions, they will be available on YouTube as well, the Panasonic one included. Um, but we also have one from a NASCAR team called TrackHouse Racing,
[36:44] uh, using Zerobus, uh, and Cortiva Agrocience, uh, which is a very cool one as well. And then, yes, reminder to fill out your surveys. Um, for questions, these are
[36:58] some of the frequently asked questions we have. Um, I'll just leave those on the screen, but if you would like to have questions answered, we will be here and happy to take them here at um at the
[37:12] front. So, I'll just leave that there. I think that's it. Yeah, that's it. Thank you all for coming. I know it's late. I appreciate your time and hopefully this helped.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.