Sponsored by: Starburst | Unlocking Enterprise AI With Databricks and Starburst

Summary

Enterprise AI projects most often fail not because of model capability but because of three data foundation problems: data still needs to be moved before models can use it, data lacks the business-specific context models need for accurate answers, and AI-generated outputs are not trusted enough to act on.
Starburst extends the Databricks Data and AI platform with federated query access across distributed data sources, allowing teams to analyze data where it lives while integrating with Unity Catalog for consistent governance, security, and lineage.
This video presents a framework moving organizations from data informed to data activated—where AI drives intelligent decisions with traceability and auditability for every AI-generated answer rather than just producing dashboards for human review.

Chapters

FAQs

Why do enterprise AI projects fail at the data layer rather than the model layer?

Enterprise AI leaders consistently report that their models are ready but their data is not. The three structural blockers described in this video are: data still needs to be moved and pipelined into a form models can use; data lacks business context such as company-specific metric definitions; and AI outputs are not trusted because there is no traceability back to the source data.

How does Starburst work with Databricks to address the data movement problem?

Starburst provides federated query access that allows models and analytics to query data where it already lives—across distributed databases, data lakes, and warehouses—without requiring ETL copies into a central location. This eliminates the data movement bottleneck and reduces the pipeline maintenance burden that delays AI projects on the Databricks Data and AI platform.

How does Starburst integrate with Unity Catalog for governance?

Starburst works with Unity Catalog to provide consistent governance, security, and data lineage across both Databricks and external data sources. Access controls and audit trails apply uniformly whether data is stored in the Databricks lakehouse or queried remotely through Starburst's federated layer.

What does 'data activated' mean compared to 'data informed'?

Data informed describes the traditional state where dashboards provide insights that humans then act on manually. Data activated means AI systems are making and executing decisions autonomously, with every action backed by traceable and auditable data. For this to work, AI outputs must be trusted—which requires the data foundation of connectivity, business context, and governance to be solid.

Full transcript

[00:09] So so welcome. I today I'm going to be talking about how Starburst and Databricks can work together in unlocking enterprise AI. And so I'm I'm Matt Fuller, co-founder at
[00:22] Starburst and VP of our our products for AI initiatives. And so what I'd like to start is I want to start with the problem that we're all familiar with. I talked to a lot of different leaders at enterprise
[00:36] companies and one thing they will always tell me is they're not failing at AI. What they're failing at is making their data ready for AI. And so there really these three things that are holding them
[00:49] back. One is that AI is waiting for their data. Their models are ready, but the data still has to be moved, copied, transformed, and pipeline into something that the models can actually use.
[01:03] The second is that the data lacks the business context. The model might know what revenue means because it was trained on the corpus of information that that has some definition of revenue, but it doesn't know what your
[01:16] revenue means. It doesn't know what the company's revenue means. In fact, you might have multiple definitions of revenue and it just simply is not going to know that because it lacks that business context. And the third part is the AI outputs are
[01:28] not trusted. And so when there's no trust, nobody acts on on them. And so this large AI investment really just kind of turns into either a a product or dashboard that no one is really using.
[01:41] And so the the aspirations for AI are real, but the but the roadblocks are structured. And let's take a look at this here. So if we look up here on the the top top right there, what
[01:55] leaders what leaders want. What do leaders want? They want to meet making intelligent decisions on on AI enhanced productivity and they want to be able to kind of automate these actions. And and so this is what
[02:08] gets funded, this what shows up in the board decks and talked about in the leadership meetings and so on. But oftentimes what's holding them back or what's actually causing these AI projects to fail is actually here on the
[02:21] left, right? This we'll call the data the data foundation. So that's connectivity and access to your data. That's the ability to scale on large amounts of data. It's actually the the
[02:33] trust and clarity and we'll talk more about that in terms of kind of the business context associated with that and like really this is what's holding them back and every one of these is the foundational problem that the
[02:45] data foundational problem that we talk about. In fact, this foundational problem has existed for a long while. But but what's exacerbating it is is
[02:57] trying to bring AI into production. I like to think of this is like a cracked foundation and AI is almost like shining light through these cracks making it more and more apparent. And so So Starburst approach is we want to
[03:10] focus on this foundation first because once you have the right data foundation, that's when you can unlock what you want with with AI. And so
[03:22] one you know one thing I want to kind of focus on and talk about is you know we talked about the foundational gap but also AI costs are not under control. And so the study by Maverick we you know we
[03:36] see that 67% of companies are actually actively planning to repatriate some of their AI workflows. And other 19% are actually considering the same thing. And so that's nearly nine in in every 10
[03:50] 10 companies are actually looking at pulling their AI workloads back from the the public cloud. And so, these are real challenges that um customers and enterprises are facing. And what we're going to talk about is how Databricks
[04:02] and Starburst can kind of help change this story that these numbers tell. And so, uh together um we'll talk about how Databricks and Starburst work together. And let's first
[04:17] uh There we go. Um we'll kind of first talk about the the two different products and how these work side by side. Um and so, you know, we're all we're all, you know, familiar with Databricks uh inside the data lakehouse. Um you might be um using
[04:31] using it for ETL or machine learning on on model development on one platform. Uh there's Unity Catalog, of course, for the governance and lineage and discoverability of the data. Um Mosaic AI for building and serving, you know,
[04:43] production agents. And then, of course, uh Delta Lake and Iceberg on object storage. And you know, there's um you know, plenty plenty more in the platform as well. And then, if we look at Starburst, how it complements Databricks
[04:55] is this allows access across your entire data estate, including lakes. Either lakes that are external to Databricks or um the data lakehouses that reside within Databricks themselves. But this uh with Starburst, we allow you to get
[05:08] access to your entire data estate. Being able to federate queries across um lots of different data sources, whether it's data lakes or operational stores or uh you know, NoSQL systems or a real-time
[05:21] systems. You may be able to connect into that and and and get access to that that data. And so, as part of that, we've integrated with Unity Catalog to help extend Unity Catalog governance beyond the the lakehouse. And as part of that,
[05:35] we'll talk about this more is being able to create these AI data products. These are highly curated data assets that can sit um on top of data that might come from a lakehouse in Databricks, but also be combining it with data that might
[05:48] exist outside the Databricks uh ecosystem. Having trouble with this clicker here. There we go. Um and so, just kind of a high-level diagram of how, you know, how these two
[06:00] products can work together. So, at Starburst, we provide this federated query engine, right? So, we have um a variety of of ways to access data through through Databricks, but, you know, we'll focus on Delta Lake and
[06:12] Iceberg. And so, when you issue a query through Starburst, we can combine the data from, say, Delta Lake or Iceberg and combine it whether it's data that's on on premise, like a traditional on-prem uh data center, for example, or
[06:25] connected into SaaS applications, or operational systems. Your your Postgres, your Postgreses, your Oracles, or even external data lakehouses, whether it's um uh a lakehouse outside of Databricks, or
[06:37] uh a legacy uh Hadoop uh dupe lake that it resides either on the cloud or or with um uh you know, within a traditional data center. But, Starburst, when you issue the query, it we've integrated with
[06:49] Unity Unity Catalog so that all the access controls are are observed, and being able to combine that with access controls um to to outside the the data state itself. And so, through this, we
[07:02] can create uh data products which are which are leveraged um you know, through Starburst to to be able to kind of retrieve that that data. And so, I kind of talked about high-level about um how, you know, the
[07:16] two products work together. I want to give a a deeper dive on Starburst itself. And the way I like to think about this is um uh you know, I kind of have these these four steps along the way, and we'll kind of talk through through each of these.
[07:29] So, the the first part is um because you don't move your data to a Starburst, when you go to deploy Starburst, you can deploy it in in connect into data wherever that lives. And that's our
[07:41] flexible data foundation. So, being able to deploy Starburst on prem, on the cloud, or or a hybrid of that. And that allows you to get value for AI in
[07:53] in weeks rather than months because you can connect into that data right away. You don't have to wait for the data to be moved and and made ready for for AI because you're connecting into that. Um the next is you really need a
[08:05] performance analytics engine. And that's where our federated query engine comes into play. Being able to query the data, combining the data say through Databricks, through other other data sources as I mentioned earlier.
[08:17] This next piece, we'll spend some time on this, but you know, one ground truth for every decision, or really the enterprise context layer. With Starburst, we can provide context of of the data to to the models.
[08:30] But if you don't have the right business context, it's not going to be able to make make sense of of that data. And we'll kind of talk through that. And then last is we want AI that can act, not just advise. Being able to take
[08:42] action. And we'll talk about the trusted agentic interface. Hold on. I'm having trouble with the clicker. All
[08:54] right, there we go. Um so, I'll just cover this this briefly, but the Starburst product is actually based on a open source project called Trino. Some of you may have heard of it. We've built our data platform around
[09:06] that. It allows fast access to data. And we've we've enhanced the engine so it's it's faster than open source open source project and also provides a high level of concurrency.
[09:19] Um but really what I want to focus on today is the bottleneck is is no longer the model, it's the the actual meaning itself.
[09:32] There we go. And so, I mentioned this earlier, but the challenge is data has no shared meaning. And so, I like to use the example of of revenue. Uh the finance team might see revenue as recognized GAAP revenue. The sales team
[09:45] might look at it as booked ARR. And the product team might look at it as as gross uh gross billings. And so um uh you know, it's one word, but there's three teams and there's three answers.
[09:57] And so um you know, before AI, you know, an analyst would be able to ask questions and clarify what is meant by revenue. But the agents are going to assume, and when they assume they're going to hallucinate, they're going to produce um
[10:10] you know, incorrect results potentially, and also ultimately uh your users will end up kind of losing trust in the system. And so being able to provide that shared meaning is absolutely critical um in order to um you know,
[10:23] provide that trust of the users. And so as I mentioned, you know, the model knows what revenue means, but it does not know what your revenue means. And so I mentioned uh data products a
[10:36] bit earlier. Just to spend a little bit more um time on data products themselves is these are purpose-built, well-documented, governed data products. They're You can think of it as like your your your gold or even platinum uh data
[10:49] assets. And they can they can span data across multiple data sources, whether it's data that's in your data lakehouse combined with say um you know, user behavior data in in an operational
[11:01] store, for example. And um consumers, whether it's for uh dashboards or uh ad hoc users or agents, can consume data through these data products, allowing
[11:13] you to access data to your entire uh data estate through through these highly curated uh data products.
[11:25] There we go. Um but but so these foundations are right, but what I want to spend, you know, some time on is the data products that a lot of people talk about today. They're often uh human discoverable or designed for humans, right? So, the metric logic might
[11:38] live in a BI tool, definitions live in a wiki, but the in the and in the business rules live in an analyst's head. But, what agents ultimately need is they need these to be machine-readable as well. So,
[11:50] you need that metric logic embedded, you need the definitions to travel with the data, and business rules encoded and not and not remembered in an analyst's head.
[12:02] And so, the real unlock are, you know, thousands of agents navigating cleanly across every data product without being told which one to use and getting smarter and smarter with every query. So, kind of one, you know, being able to
[12:15] ask a question with with any agent any and any data product. And so, we call this the enterprise context layer, where there's the structure. This is where the
[12:27] definitions, the metrics, the rules that define how the business actually works. And this also includes the meaning, not just what these metrics are, but you know, how it all connects together. This
[12:39] allows the agents to reason and not just simply look up these these different rules. And then finally, you need governance, right? These access controls applied at the moment the metadata and data is accessed and that so that every,
[12:53] you know, answer is audible and you can kind of trace trace everything. And so, when we when we kind of look at data products themselves, we'll talk about,
[13:06] you know, our trusted agentic interface we call we call Ada. I'll talk about that in the next slide. But, imagine, you know, you're you're interacting with this with this data through through an agent, for example.
[13:18] You want to be able to query through the data products that have all these different metrics and business rules and synonyms and all the the broad, you know, kind of business meaning behind the data. And this is all
[13:30] collected and harvested through a variety of of different things. So, whether it's from traditional business catalogs, BI tools, query histories, um from the analyst analyst's head, and and so on. And so, the same way that
[13:43] Starburst can help federate access to the data itself to for the purposes of of AI, we're also be able to access the business context wherever it lives. So, not having to say to create a single
[13:56] source of truth, but actually going out to those different sources of truth and and harvesting it in from a single location. So, you have both your your access to your data from a single point of access and access to your your metadata from that single point. And so,
[14:09] that's really the key in order um to kind of create this this trusted AI agentic interface. And so, the you know, the the challenge is we want to kind of challenge the status quo of of the business. So, um we have uh
[14:23] an an agent where you can have a natural language interaction with it. And so, you'll be able to ask questions of your data, but not only that, be able to ask it to take actions based on what it what it found. So, maybe, for example, um if
[14:37] you're doing like a threat detection use case, once it identifies that threat from a security log, actually be able to file a ticket or notify someone or actually um look on how how to resolve it. And so, that's that's what Ada does.
[14:50] But, the really the core part here is um it's the ability to connect to all of your data. So, it's not just for data that might exist in a single platform, but being able to interact with the data
[15:03] through Ada across your entire data estate. But, also what what gives the the value here is because you have that business meaning, when users are asking business questions, it's able to kind of bind that business meaning um
[15:16] to the question, so it can actually formulate the right, you know, queries underneath um you know, kind of underneath the within the platform in order to know what data sets to reach out to, what data products to reach out to to ultimately answer answer that
[15:29] question and give a a response back. And so, you know, it's really become this this enterprise team. And as I mentioned earlier, it's not, um, necessarily just a chat agent. Sure, you can do that, but
[15:41] really what the unlock is is to bring you kind of from that data informed to data activated, which means like being able to actually take action on the insights themselves.
[15:54] And so, um, you know, in closing, you know, we talked about the the foundation for your AI needs. So, the ability, um, the the critical ability to be able to take data that's from your your lakehouse, say in Databricks, and combine it with data across your entire
[16:06] data estate, be able to kind of create those data products through Starburst. And those data products are really critical because it contains, um, not only the data that you need access for for your your downstream AI use cases,
[16:19] but the actual business metadata itself that kind of helps provide that that full picture. And so, um, with with that, um, thank you for, um, attending, uh, the session today. And I'll I'll be, you know,
[16:32] hanging out here for any questions, uh, at the end.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.