Under the Hood: Scalable Ingestion Strategies for Databases, Applications, and File Sources

Summary

Lakeflow Connect is Databricks' built-in managed ingestion solution with over 100 connector sources, supporting CDC-based ingestion for databases like Oracle, MySQL, and Postgres and query-based ingestion for sources like Teradata, Redshift, and Synapse.
This video covers new features including Workday incremental ingestion, Salesforce formula fields with mutual TLS, community connectors for custom APIs, and networking patterns like reverse SSH tunnels for on-premises Oracle databases.
A decision framework for choosing between CDC and query-based capture is presented alongside cost and performance best practices, with a live Genie demo answering operational questions from all five ingested data sources simultaneously.

Under the Hood: Scalable Ingestion Strategies for Databases, Applications, and File Sources

Data engineers spend far too much time maintaining fragile scripts for a fragmented landscape of databases, applications, and file sources. In this technical session, we’ll explore how Lakeflow Connect allows you to build robust pipelines without the operational burden of DIY infrastructure. We’ll dive into: Database ingestion: A look under the hood at efficient CDC for sources like Oracle, MySQL, and Postgres, alongside query-based ingestion for sources like Teradata, Redshift, and Synapse. Application and file ingestion: What’s new—and happening behind the scenes—across our newest connectors for customer 360, employee 360, financial analytics, and more. Cost and performance optimization: Technical best practices for networking, security, and minimizing impact on source systems—plus how to use new features to slash TCO and boost performance. We’ll wrap up with how to create your own application and database connectors. Stop managing infrastructure and start delivering data.
Talk By: Peter Pogorski, Staff Product Manager, Databricks ; Sonia Bendre, Associate Product Manager, Databricks ;

Chapters

FAQs

What is Lakeflow Connect and what sources does it support?

Lakeflow Connect is Databricks' built-in managed ingestion solution with over 100 connector sources that delivers data incrementally into Unity Catalog as governed tables. It supports database sources like Oracle, MySQL, Postgres, Teradata, Redshift, and Synapse, as well as SaaS applications including Salesforce and Workday, plus community connectors for custom APIs.

What is the difference between CDC and query-based ingestion in Lakeflow Connect?

Change Data Capture (CDC) reads the transaction log of a source database to capture every insert, update, and delete as it happens, providing low-latency incremental ingestion with minimal load on the source system. Query-based ingestion periodically polls the source with SQL queries, which works for systems that do not expose a transaction log but may place more load on the source.

How does Lakeflow Connect handle on-premises database sources?

For on-premises sources like Oracle that cannot be reached directly from cloud compute, Lakeflow Connect supports a reverse SSH tunnel networking pattern that establishes a secure outbound connection from the source environment to Databricks. This avoids the need to open inbound firewall rules on the source system.

How do AI agents use ingested data from Lakeflow Connect?

Once data from SaaS apps, databases, and file sources is ingested into Unity Catalog as governed tables, it forms a centralized semantic layer that AI agents can query to answer operational questions. This video demonstrates a Genie agent answering live lineup and contract questions by querying all five ingested data sources simultaneously.

Full transcript

[00:10] Hi everyone. Thank you all for joining us today. Uh in this session we're going to walk you through how to build a robust end-to-end ingestion engine entirely from scratch using Lakehouse Connect. Quick housekeeping, we will be talking
[00:22] about future or forward-looking features and products. Um so just a reminder, these features are subject to change. As a brief overview, Lakehouse Connect is the built-in way to get data into
[00:35] Databricks. It enables you to pick from over 100 managed connector sources to deploy ingestion pipelines via the UI or API so that you can incrementally bring your data into Unity Catalog as governed
[00:47] tables. This keeps your Lakehouse up-to-date and consolidates your data into a centralized governed foundation, which is especially important for building a semantic layer that AI agents can use to answer critical questions for your
[00:59] organization. I'm Sonia Benje, the PM for SaaS connectors on Lakehouse Connect, and I'll be co-presenting with Peter Pogorski, who is the PM for database connectors on Lakehouse Connect. During the session, we'll cover new features,
[01:12] best practices, and common patterns that we found to work really well, whether you're dealing with SaaS apps, databases, or custom APIs. So let's dive in. We're going to ground everything we talk
[01:24] about today by following the case of a fictional soccer team called Spark City FC. They're an underdog club that's made it all the way to the global finals, and they need every data advantage that they can get to beat their rivals and take
[01:37] home the trophy. The two key characters we'll follow are Coach Lariat. She is the head coach who needs fast, accurate data to make lineup decisions, and Coach Mustache. He's the assistant coach responsible for
[01:49] gathering and consolidating all of that data before the upcoming match. Coach Lariat wants to be able to answer several questions, like who's cleared to get on the pitch according to their contract, but also medically, who might
[02:02] still be recovering. How does the rival team play? What are their signature strategies? Which past competitors are they maybe similar to? Um what lineup gives us the best shot at winning, and several more questions.
[02:15] These questions are simple, but answering them requires pulling together data from many different systems, which is far from easy using the traditional approach of manually building a custom pipeline for every single data source.
[02:29] Coach Mustache needs to pull things like player contracts from Salesforce, medical records from Workday, competitive intel from a niche provider called Pitch IQ, historical match data
[02:41] from an on-prem Oracle instance, and performance statistics from SQL Server. Each pipeline requires custom code, its own streaming job, and its own infrastructure. The result is a fragile
[02:53] and overwhelming scrip- web of scripts that are painful to build and even more painful to maintain. Lakeflow Connect eliminates all of this complexity. Instead of building and
[03:05] maintaining a unique pipeline per source, you get over 100 managed connectors in a single platform. You just tell us what data you need, and we'll take care of the rest. You can set up these ingestion pipelines
[03:18] through your preferred interface, whether that's the UI, a notebook, or a DAB, a declarative automation bundle, which is what we will walk through for most of this presentation. A DAB allows you to organize your entire ingestion
[03:30] setup into a single version controlled repository. The folder structure is simple, and you can follow along as we build it up on the left pane here. To start, you need a root Databricks configuration file, a resources folder
[03:43] for pipeline definitions, and a source folder for any notebooks. We'll begin with the pipeline definition for Workday reports. In our soccer team scenario, Workday is where the club
[03:56] stores medical records for players um and it tells us who might be available for a match um what versus who might still be recovering. The pipeline definition is straightforward. At the top, you specify a name, a destination
[04:08] catalog, a schema, as well as the Unity catalog connection containing your authentication details. We'll also specify the source object to ingest, which in this case is a Workday report that we reference with its URL.
[04:21] This is the standard format for any Lake Lo Connect pipeline definition. But we'll go a bit further to specify a few more details that enable incremental ingestion on this report. A new feature
[04:33] that we're excited to launch into beta right after Summit. Historically, the Workday connector only supported snapshot ingestion, where we read all rows in the table during each pipeline update, and then apply any updates to the destination table.
[04:47] We're now introducing support for incremental ingestion, where we only read the data that's changed since the last pipeline update. The goal is to improve connector performance, cost efficiency, and data source load.
[04:59] What this means practically for Spark City FC is that the coaches get updated clear player clearance data the moment that it changes in Workday. So, they can make real-time strategic decisions based on who's available.
[05:13] There is some one-time setup on the Workday side. Your admin needs to add a date field to your report that monotonically increases with each new row. This is the equivalent of the last updated column we saw in the example.
[05:26] It will be used as your cursor column, so the connector can track which rows have changed and which ones need to be updated. Next, your admin has to define two inclusive prompts on the column, one for
[05:38] the start of the ingestion window and one for the end. Finally, they will just share those the aliases of those two prompts with you. Once you have Workday set up, we'll come back to your Workday pipeline file and specify these details. The sequence by
[05:52] field, which is the cursor column on your report, um and the XML aliases of the inclusive prompts, which are passed here. Note that we set the start filter such that it falls to your chosen start date on the first run, but picks up from the
[06:05] last pipeline updated on every subsequent run. And we set the end filter with a days log so that we're only ingesting complete days of data. For this feature, your report must be append only. If this fits your use case,
[06:19] we recommend that you use this incremental ingestion feature whenever possible to avoid full refreshes that put unnecessary load on your Workday instance and consume compute you just don't need.
[06:31] Next, we'll set up the Salesforce pipeline. For Spark City FC, Salesforce is where things like player contracts live. The pipeline definition follows the same format as for Workday reports, except we have a couple feature
[06:44] callouts. Incremental ingestion support for formula fields, mutual TLS authentication through our UC connection, and row-level filtering. You can see each one is just an additional line. So, let's talk through
[06:56] each one. We'll start with support for incremental ingestion of formula fields. Formula fields are computed values or derived metrics in Salesforce. For example, Spark City FC has one
[07:09] called days until expiry or days until contract expiry, which simply calculates the difference in days between the current date and the expiration date in each player's contract. These fields can be tricky because
[07:21] Salesforce doesn't update caller cursor columns like system mod stamp or last modified date when formula fields change. So, by default, Lake Flow Connect does not ingest sales Salesforce
[07:33] formula fields incrementally. Instead, it takes a snapshot of formula fields on each pipeline run, then joins them with the rest of the table. However, we now have a beta feature for incremental formula field ingestion.
[07:46] Instead of a full snapshot, the connector ingest non-formula fields incrementally and then recomputes the formulas as a materialized view. So again, you get the freshest computed values without requiring a full snapshot
[07:59] run. This drastically improves performance and reduces cost, so we recommend enabling this feature whenever possible. Next, let's talk about mutual TLS authentication. Before ingesting from
[08:13] any source, you need to create a Unity Catalog connection so that Databricks can authenticate with that source. Historically, we only offered OAuth to connect to Salesforce, which relies on standard TLS, where the
[08:26] server presents certificate to prove its identity, but the client does not. However, some enterprise use cases require stronger guarantees. For example, Spark City FC also stores compensation information in Salesforce.
[08:39] In such cases, we recommend mutual TLS because while standard TLS only verifies the server, mutual TLS means both parties verify Both party verify each other using
[08:51] certificates. To set it up, you generate a CA-signed certificate in Salesforce, then store the certificate and credentials in a Unity Catalog connection in Databricks. From there, you just reference the connection by name in your pipeline
[09:03] definition um and you're set to go. This feature is currently in private preview, so if you're interested in trying it out, please reach out to your account teams. And if you have any feedback, feel free to come up to a to us afterwards.
[09:17] Finally, our new row filtering feature allows you to ingest only the records that you actually need. In our example, we're only pulling players with their status set to active. There's no point ingesting historical records for players
[09:30] who left the club two seasons ago. This cuts API quota usage significantly, reduces compute, and shrinks your storage footprint. For teams working with large Salesforce instances, we
[09:42] recommend trying this beta feature to scope your ingestion. It's also available for other SaaS connectors like ServiceNow and Google Analytics.
[09:56] Next, Coach Mustache needs competitive data from a niche provider called Pitch IQ. They don't have any managed connector yet, but this is not a problem because he can use community connectors to extend Lake Flow Connect. Using Databricks provided guidelines and
[10:09] AI to fill in the gaps, you can cover your long-tail sources while keeping the governance and observability advantages of our managed connectors. The Databricks provided workflow also includes testing the new connector.
[10:23] The YAML definition file follows roughly the same structure as managed connectors, so it slots right into your existing DAB.
[10:35] We'll now move on to SQL Server, where Spark City FC stores data about historical matches. The pipeline follows roughly the same general structure again that you've already seen with one important callout, connector type field. Lake Flow Connect gives you two ways to
[10:48] capture changes from databases, native CDC, change data capture, or query-based connectors. For SQL Server, we're using CDC which reads directly from the database transaction log to pick up data changes
[11:00] as they happen rather than polling. This lowers latency and lessens the load on the source system. In a few minutes, Peter will walk you through the full decision framework you need to help you choose the right approach for your use case.
[11:16] Finally, Coach Mustache needs to ingest from Oracle, which holds player performance stats. The structure is the same, but here we're using query-based ingestion instead of CDC. And since this is an on-prem database, we connect through a reverse SSH tunnel,
[11:29] which Peter will also cover comprehensively in his database deep deep dive. All right. So, now here's what the complete DAB looks like when it all comes together. We've got just one repository, one
[11:41] Databricks YAML file, and under the resources folder, all the pipelines are defined. Now, Coach Mustache can deploy the entire thing with a single command, Databricks bundle deploy. That's it. And
[11:53] the entire ingestion engine, every source, every pipeline is deployed atomically, consistently, and repeatedly across environments.
[12:06] Now, if you're sitting there thinking, "This is great, but we went through so much code." I've got some more good news. With Genie code, you don't even have to write any of it yourself. Simply ask Genie code, and it will spin up all the pipeline definition files and
[12:19] corresponding job definitions to actually schedule those pipelines in a matter of minutes. As a technical user, Genie code is what Coach Mustache relies on for quick building.
[12:33] Now that we've gone through this process, I'll hand it over to Peter for a deep dive into the database connectors.
[12:50] >> Cool. Thanks, Sonya. So, now that we saw how to create a great automation bundle, I'm going to dive into our database connectors a little bit deeper. And one thing to remember with databases is they can come in
[13:02] basically any configuration imaginable. Like, you can change anything about an Oracle database, a SQL Server database. You really need to consider how your source is configured when you think
[13:15] about setting up ingestion to these different databases. So, I'm going to walk through a few things here which help you get the most efficient experience when you're using Lake Flow Connect with your database connectors.
[13:29] And first thing I'm going to need to do is I'm going to need to decide how should my data be extracted? And like Sonia was mentioning, we have two different options for a database connectors. You can either use native
[13:42] CDC or query based capture. And now if I choose to use native CDC, there might be different options available on the database. Such as on SQL Server, you can choose between CDC
[13:55] or change tracking. And then I'm going to need to think about how am I actually connecting my source database to my Databricks workspace, so we can pull the data from that database into your workspace on the
[14:09] cloud. And then finally, how fresh do I actually need that data? And what's the log retention on my source database?
[14:22] So this first decision, how to pick up changes from the database. As I mentioned, you have two different options here. Native CDC is going to read the database's change log, and it's going to change every it's going to
[14:34] capture every change happening, and it's the most efficient way of extracting data. CDC is especially great when you have a lot of changes coming through the database, or you want to have lower
[14:46] latency ingestion, or when you actually care about the full history of changes coming through, or let's say you don't have any cursor column on your table, or you don't have a primary key, then CDC
[14:59] is a great option as well. And then our other option is going to be query based capture. And query based capture is very simple. It's just going to watch a cursor column on your table
[15:11] and pull whatever is new since the last pipeline run. It doesn't require any setup on your database whatsoever, so you can quickly get started right away. And this is especially great if you have
[15:24] a database where maybe you only have access to the read replica or you're not able to enable CDC for some reason. Maybe your DBA won't let you or you just want to get started as quickly as
[15:36] possible, then query based capture is a great choice. And now next up, if you did choose native CDC, you're going to have some additional options in the change
[15:48] mechanisms. In SQL Server is where this is going to matter the most because it gives you the option between change tracking and change data capture. And the choice is really going to come down to two things here. One, does your
[16:01] table have a primary key? And two, do you need to capture all of the historical changes on the source database for an SCD type two ingestion? So, if a table has a primary key, we're
[16:14] typically going to recommend using change tracking. All it's doing is looking at the records that have had a row updated or changed, which makes it very lightweight on the source.
[16:27] And then if your table doesn't have a primary key or you do want to capture that SCD type two history, you're going to want to opt for change data capture.
[16:43] Now, we need to figure out how the Databricks workspace is actually going to connect to the source database. And we have three good paths to choose from here. If the database is in your own cloud and your Databricks workspace has been
[16:56] configured with a VPC or VNet that's peered to the subnet with your databases, the simplest way is just going to be deploying a classic compute cluster within your
[17:08] Databricks VPC or VNet. And we fully support that with Lake Flow Connect where you can run either on classic or serverless compute. Then we have a private link option,
[17:20] which is more of a targeted option when you have a private endpoint connecting the database to the Databricks workspace. And this is going to be like a one-to-one mapping between each
[17:32] database and private link that you set up. And private links are really good because they very easily let you select between using classic compute or serverless compute.
[17:44] And then finally for on-prem sources, which are sitting behind a firewall, and maybe your organization only allows outbound traffic from that from that on-premise environment, we have a reverse SSH tunnel solution, where that
[17:58] connection will be initiated from within your on-prem data center. And then question four is mostly related to data freshness. So, the shorter the
[18:10] interval that you schedule your pipeline, the sooner the changes are going to be replicated into your target tables. But if your organization doesn't need data as frequently, you might want to reduce your compute
[18:23] costs. And then you can schedule your pipeline to run at longer intervals. But the key thing you want to keep in mind if you are doing that is you need to schedule your pipeline frequently
[18:35] enough that the log retention isn't going to cause a truncation on the source. If the logs do disappear, then we'll need to do a full refresh on the table.
[18:47] And then another feature uh we also offer is actually called automatic full refresh. And this can be used when things happen on your source table like some sort of
[18:59] destructive sche uh destructive schema evolution. So, it could be like a column going from a double to an int. And with this feature you can have your table automatically
[19:11] refresh within the window that you've scheduled so that you don't need to manually go in, interact with the pipeline, reduce the error, and try and full refresh your table.
[19:23] And then finally, if you are using the query base capture method, we're only able to capture the state of the source table at the point in time when the pipeline's running. So, if you do want those intermediate states, you might
[19:36] want to consider using native CDC. So, how did I decide what we would do at Spark City FC? Starting with the match database running on SQL Server, it has a lot of changes
[19:49] flowing through it. I actually have full access to the primary, so I can easily go in and enable change tracking for my tables. And the other thing to consider with this match database is
[20:01] a lot of the match data gets hard deleted and corrected all the time. And the change log is going to capture all of that for me. So, for SQL Server, I went with the native change data capture.
[20:13] And then on our scouting warehouse, which is on Oracle, we're going to run it in a different way. With the Oracle database, I only have access to the secondary read replica, and my DBA won't allow me to enable
[20:28] supplemental logging. So, for this one, I'm going to have to go with query base capture and use the updated add timestamp to incrementally bring in my data.
[20:41] So, now I'm going to demo deploying the SQL Server connector, but before I do that, I wanted to introduce our new integrated CDC connector architecture, which is currently in beta. And if you've used our CDC connectors in
[20:54] the past, you'll know that we previously had two different pipelines. One which was a gateway pipeline, and one which was an ingestion pipeline. The gateway would extract the data, and the ingestion pipeline would actually apply
[21:08] that data into your target tables. But with the integrated CDC connector, we're folding all of that logic into a single pipeline. So, let's it's going to make it easier for you to deploy and manage, but it's going to give you all
[21:21] of the same great functionality of our existing database connectors. And with this integrated CDC connector architecture, we're also introducing the ability to fully schedule the pipeline.
[21:34] So, previously the gateway needed to run in continuous mode. Now you can choose to fully schedule and run this pipeline on whatever interval you like. Um you also have the option to fully run
[21:46] it on classic or serverless compute. So, much more flexibility in the compute options as well. And with that, I'll jump into a demo.
[22:01] So, since I'm using native CDC on SQL Server, before I get started, there's going to be some initial setup that I need to do on the source database to make sure that the change data is available for me to ingest. And to do that, I'll connect to my
[22:13] source database, and I'm going to first run the utility scripts. And these utility scripts contain stored procedures that help enable things like CDC and change tracking. They set up DDL
[22:25] objects so that I can track when things like schema evolution happen, and they provide simple options for debugging. So, you can find these utility scripts in our documentation along with
[22:37] step-by-step guidance. And then after I run the initial utility script, I'm going to run a stored procedure to enable change tracking on the database. And we chose change tracking here over CDC to place less
[22:50] load on the database. We had a primary key on every single table. Now that I've run all of those scripts, my database is ready for ingestion. So, I'm on the add data page. Here you can see all of our managed connectors and
[23:04] community connectors. We have a ton of them through the UI, even more through the API. And I can start ingesting my SQL Server data by clicking on the add SQL Server button.
[23:17] Here, I'm going to pick a connection to my SQL Server setup. So, I'll select one that already exists, but if I didn't have one, I could easily create a new one. Um and then once I'm here, I have the
[23:31] option to choose between change data capture or query-based capture. And since I want to capture these changes in the most efficient way possible, I'm going to go with change data capture and
[23:43] give the pipeline a name. So, we'll call it Spark FC SQL Server pipeline. Then I need to tell the pipeline where to keep the event logs, and this is going to be all of the
[23:55] run history and logs from the pipeline. And then finally, I'll need to choose between either fully classic or fully serverless compute on the pipeline. And depending on your networking
[24:08] considerations, you might want to go one way or the other, but we would always recommend serverless compute for the fastest and most seamless experience. I'll then also enable auto full refresh
[24:21] for all of my tables. So, that way if something does happen, like a destructive schema change, the table can be refreshed automatically without me manually needing to go in and interrupt
[24:33] the pipeline. So, once the pipeline loads, the schemas and tables will show up that I can ingest from the source. And I can pick what I pick which tables or schemas I want to
[24:45] ingest, and in this case, I'm going to pick the entire schema. And once I do this, I'll select a target catalog where the data is absolute
[24:57] actually replicated into. Then our pipeline's going to move on to validation. And this is where we're going to check with the source database was configured correctly in order to use native CDC. So, we'll check things like
[25:11] has change tracking been enabled, have the DDL capture objects been set up, and have they been created so we can detect any schema evolution. And if there was any error detected, it would provide you
[25:23] with a fix-it script that you could run directly on your database in order to remove the error. Finally, I pick a schedule for the pipeline. That schedule is a Lake Flow job um attached to the
[25:35] pipeline. And then finally, we can see the data flowing in. And now we'll have our SQL Server data successfully ingested.
[25:47] Now, I'm also going to demo deploying the Oracle connector, but before I do that, I want to explain the networking that we use to connect to it. Since the Oracle database is on premise and Spark FC only allows outbound
[26:00] connections from the data center, we set up a reverse SSH tunnel. And this reverse SSH tunnel is a small host inside the Databricks workspace which is dialing outbound on port 22 and landing
[26:13] in a proxy VM within the within the Databricks workspace. So, this enables Lake Flow Connect to ingest from the database without you ever needing to open an inbound rule on
[26:26] your on-premise firewall. So now, I'll navigate into the Oracle connector setup. And first, I'm going to select an existing connection to an Oracle database just like I did with SQL
[26:39] Server, though I could also easily set up a new connection here. And then I'll go back to the page where I have the two options to select how I want to extract my data. And based on
[26:51] the criteria that I went through, I selected query-based capture. So, this is going to use the cursor column on the table to incrementally pull in new and updated records on each pipeline run.
[27:04] From here, I'll give the pipeline a name. We'll call this one the Spark FC Oracle pipeline. And then, after we do that, we'll have to pick an event log location.
[27:24] So, I'm just filling in my name here, and then we'll pick an event log location. And then, once again, with the query-based connectors, I have the option of choosing between serverless or classic compute. So, you can choose any compute policy
[27:39] which leads pipelines to be deployed on. There's no set policy that you need to use. And then, once again, you'll want to think about what your networking is for your database when you're selecting between classic or serverless compute.
[27:52] Um, with serverless, I don't need to worry about VMs being available or any quotas or limits. So, we'll go with serverless again. And then, on Oracle, I need to enter a service name for my Oracle database in
[28:06] order to fetch the tables and schemas that are available for me to ingest. Once I do that, I can select the tables I want to ingest. And with query-based capture, I'm going to select which column maps to the
[28:19] cursor column for incremental loads. Um, I'll also select any primary key, or I could choose to have that primary key automatically inferred for me. And with the query-based connector, if
[28:32] my table didn't have a primary key or a cursor column, I could choose to just batch load the entire table instead, so that I can quickly get started. I don't need
[28:44] to worry about having that cursor column in place, but it will be more load. And then we have the schedule on the pipeline, so I can choose to run this at whatever frequency I would like, once
[28:56] again. And then finally, we can see the data actually being ingested and landing in the workspace. And now it's able to do all of this without actually having to configure anything on my database.
[29:10] And now that we've gone a bit in Now that we've gone more in-depth on the database setup, I'm going to pass it back to Sonia, where we can see how we can actually get insights from this data.
[29:36] >> Thanks, Peter. So now, just like that, all five data silos that we began with are gone. Everything is in one catalog now. It's governed so that the right people see the right data. It's fresh because each pipeline keeps syncing on
[29:48] its own schedule, and it's all in one place with the business context attached, which means we can finally do something fun with it. Coach Leary can now ask Genie questions based on all the context we just
[30:01] ingested. We play our rivals on Saturday, so she wants to find out what the best lineup would be, as well as formations and plays that we that have worked best against them in the past.
[30:13] Now that all the data's been ingested through Lake Flow Connect, she can just ask Genie one this question. While Genie's working, let me explain what's happening behind the scenes. To answer the question about the upcoming game, we need data from five completely
[30:27] different systems. We just brought all of that together from Salesforce, Workday, Pitch IQ, SQL Server, and Oracle into the governed lakehouse with Lake Flow Connect. This creates a governed semantic layer or an ontology,
[30:40] which is exactly what makes Genie so powerful. Genie is drawing on the full context of Spark City FC's ontology, the business definitions, relationships, and metadata across all five sources. This
[30:53] enables it to interpret Coach Lerat's question the way our soccer club actually thinks about its data. And because new data is continuously ingested each time I go to Genie, I'm going to get answers based on the most
[31:05] up-to-date information. And as we go through, it'll take even more context from previous sources that we might have joined and ingested. So,
[31:17] we'll just wait a little longer. And now, here's our answer. Genie says we should play the 4-3-3 formation and even provides a list of the best players that are available. Previously, it would have taken hours to
[31:30] pull the data out of all the sources and join it together just for us to manually have to figure out our optimal strategy. But now, we can just ask Genie.
[31:43] So, that's the story. Technical users can leverage Genie code to help create and deploy Lake Lo Connect pipelines that keep data flowing in. And then, business users can leverage Genie 1 to ask questions in plain English. What used to be a major challenge is now just
[31:56] a quick conversation. And while we followed a fictional case, Lake Lo Connect is already delivering significant value to over 3,000 customers, including some of the world's
[32:09] largest enterprises who are using it to replace brittle pipelines with a single governed ingestion layer.
[32:21] Specifically, Porsche Holding Salzburg, Europe's largest automotive retail company, had fragmented data systems with siloed customer engagement data. So, they turned to Lake Lo Lake Lo Connect to ingest data from Salesforce, transform it in both batch
[32:33] and streaming modes, and deploy a more modern personalized buying experience. Ultimately, developing with Lexicon Connect was 85% faster than building a custom homegrown solution. But even beyond R&D cost, Porsche was also able
[32:46] to improve their end customer's experience with that customer 360 view. I'll end on this slide with frequently asked questions, um but that's all we have to present for you today. Feel free
[32:59] to come to us after if you have more questions, and do remember to complete your survey and provide feedback about this presentation so that we can make it even better. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.