Beyond the Lakehouse: Supercharging Databricks with Contextual Intelligence

Summary

• Context intelligence layer: DataHub extends Databricks Unity Catalog with comprehensive metadata intelligence across the entire ecosystem, providing critical context for data governance on Databricks Data and AI platform.
• Real-time integration: DataHub's event-driven architecture instantly propagates changes across systems, decreases data incidents by 60% and enables rapid discovery from days to minutes on Databricks Data and AI platform.
• AI-assisted discovery: Using real-time platform APIs, DataHub enhances LLMs with cross-platform metadata and AI-powered search, accelerating model time-to-market with reliable, compliant data.

Beyond the Lakehouse: Supercharging Databricks with Contextual Intelligence

Talk by Gabriel Lyons, Founding Engineer, Datahub (Sponsor Speaker)
While Databricks powers your data lakehouse, DataHub delivers the critical context layer connecting your entire ecosystem. We'll demonstrate how DataHub extends Unity Catalog to provide comprehensive metadata intelligence across platforms. DataHub's real-time platform: *Cut AI model time-to-market with our unified REST and GraphQL APIs that ensure models train on reliable and compliant data from across platforms, with complete lineage tracking *Decrease data incidents by 60% using our event-driven architecture that instantly propagates changes across systems *Transform data discovery from days to minutes with AI-powered search and natural language interfaces. Leaders use DataHub to transform Databricks data into integrated insights that drive business value. See our demo of syncback technology—detecting sensitive data and enforcing Databricks access controls automatically—plus our AI assistant that enhances' LLMs with cross-platform metadata.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is DataHub?

DataHub is an open-source data context and metadata management platform that extends Databricks Data and AI platform by providing comprehensive metadata intelligence across the entire data ecosystem.

How does DataHub reduce data incidents?

According to this video, DataHub's event-driven architecture instantly propagates changes across systems, enabling teams to detect and respond to data issues faster, decreasing incidents by 60%.

How can business users ask natural language questions to explore company data?

Natural language interfaces like AI/BI Genie enable business users to explore data without SQL by translating conversational questions into database queries, making data accessible to non-technical stakeholders. These systems learn organizational context, data relationships, and business terminology to interpret questions accurately. Integration with governance systems ensures data access respects organizational policies while enabling rapid, self-service analytics across teams.

How should data and context be governed for AI agents?

Effective AI agent governance requires defining what data agents can access, establishing clear policies for data use, and maintaining audit trails of agent decisions for compliance and improvement. Context governance ensures agents receive accurate, relevant information needed for decisions without unnecessary data exposure or privacy violations. This approach combines technical controls through Unity Catalog with business rules that specify appropriate agent behavior and limitations.

Full transcript

Okay. Well, thank you all so much for joining. Um, today I'm going to talk about uh something fundamental to what you've seen throughout this whole conference. So through this conference, you've been talking about all the exciting things that you can do with data, things like business intelligence, getting training machine learning models. But something that underlies all of that is this question of context. And what I'm going to show is where we've come with context, how we thought about that, and also what we see the future of this being.
I'm going to give a sneak preview, a demonstration of where we're headed uh with data hub and context and how that relates uh to the new world that we're entering. So before I get started, I wanted to share a little bit about myself. I'm the founding engineer of the data hub project and I started thinking about how to help people use data, leverage context about data and take advantage of data while I was at Airbnb. So there I helped Airbnb scale to thousands of data practitioners.
Um as the company was rapidly growing, we were growing from hundreds to thousands of data practitioners. There were multiple business verticals that were expanding at Airbnb and we had to think about how can we adjust our approach to collecting data to governing data and using data so that we can support um this massively growing number of data users and data subjects. So as on the core logging team I was the tech lead there and we figured out how to scale to that number of users.
From there I thought supporting a thousands of data practitioners is great but how can I take this and bring it to the next hundred thousand or hundreds of thousands of data practitioners. So at this point I joined data hub as a founding engineer to help not just bring data discovery to everyone but also uh help with things like data governance, data compliance and data quality. So, DataHub is the number one open-source project for data and context management.
We've grown to over 15,000 members of our open source Slack community. We have over a thousand production deployments and we're really proud of partnering with incredible technical partners. We have people like Visa, Netflix, Apple, Slack. These are just a few of the thousand plus production deployments who are part of our open source community contributing to the data hub platform and helping gather context so that everyone can take advantage of all the amazing data that they have.
So why are we talking about data and context management now? This is something I've been working on for a long time. So what's the point of talking about this at this exact point in time? For us, we believe that this is a really unique changing point in how we think about managing data context and how we think about managing um things like data discovery and data governance. And that's because we're transitioning from the types of audience that needs this data context.
It used to be that just humans would come in and need to discover data, need to learn how to use data responsibly and trust it. And we're moving into a world where soon we believe that agents and LLM and AIS are going to eclipse the amount of need that humans have for this type of data trust and data governance. So for example, I'm sure something that you've seen over the course of this conference is data bricks genie.
That's something in particular that we're really really excited about just like Asian bricks which I think was just just released. With both of these tools, they're really powerful ways where you can ask these amazing natural language questions. And I've had so much fun uh playing around with both of these tools. Uh you can ask these natural language questions and get answers. It's almost magical using these things and it really feels like this is the future.
However, without context, these tools can also be limited. So for example, I loaded in some sample data about um this mobile app for my coffee shop and I wanted to know well what's our app conversion rate and in this case because of some missing context Genie also is able to make um miscalculations. So here without the context of how to calculate conversion rate it produced the wrong query made the wrong metric and came to an incorrect you know slightly out of out of accuracy conclusion.
So of course loading it with the meaning and the definition of uh conversion rate that's absolutely something I'm sure that data bricks is going to do and these are only going to continue to improve. However, we also know that every organization, every business is going to have its own unique context and intuition and knowledge that is not going to be universal. And you need a way to manage that, especially when you're working with this new type of data user.
So, the visual metaphor that I think of is like a genie in a bottle. We have data bricks genie. It's this absolutely magical amazing tool, but it's limited. it's limited by the context that it has. So, as we're thinking about how to use this and where we're going in the future, we want to think how can we by providing context let the genie out of the bottle and give it that amazing magical power that we are all so excited about using it for.
And the way that we answer this question is by thinking about how have we solved this problem for humans. So for humans for a long time we've been thinking about how can humans find data how can they discover it how can they understand it how can they know how to use it tr in a in a correct way in a safe way in a compliant way and how can we make sure that they're using the authoritative data these are all problems that we've already thought about solving and this is not a problem that's exclusive to humans.
What we're seeing is that agents have this exact same need where they also need to make sure that they can find and use properly the authoritative data and not just pick any data off the shelf, give it a shot and see what happens. So, we feel like we're at really an incredible inflection point where there's all this opportunity to improve, but without context, we're going to be limited. So what we want to do is take and what we've done is we've taken a few lessons that we learned with helping humans gain context about data and then we applied them to how we can help agents and LLMs do the same thing.
So I'm going to jump into a demo soon of how we're helping LLMs. But first let me go through some of the lessons that we learned and so you'll see how we've applied them. So lesson number one, relevancy matters. And this is something that we really had to learn the hard way. Um I worked with another number number of our customers and people in the open source community and we learned that relevancy and ranking is just as important if not more important than being able to gather all the results together.
So if I search for something like customer or conversion rate or app usage, if I'm comparing it with any of the data warehouses or data platforms that I've seen, you'll see hundreds if not thousands of results. So if I just have this long stream of results, I really have to dig in and make sure that the first couple results that I'm presenting are the trustworthy and authoritative ones. I spent a lot of time with one of our customers, Chime, and also one of the top design technology firms in the world.
Both of them had this intense need that they and they demanded that the relevancy of their search results were correct and that the top results were always the most authoritative. So we spent a lot of time looking at all different signals, things like freshness, correctness, completeness, ownership that goes into trustworthiness and putting that into our relevancy algorithm so that we can make sure that we're not just giving you a hand basket of all sorts of jumbled things. some that you can trust, some that you can't, but that we're providing you authoritative results right in the top of the set.
The next lesson that we learned was that not just relevancy matters, but context matters, too. And in all of our different data systems, what we see is that there's a lot of heterogeneity. So, we don't see organizations just using exclusively one system and they're done with it, but you have lots of different systems. You have upstream production databases. You have pipeline and transformation tools. Maybe multiple warehouses, storage layer, compute layer.
Then there's downstream tools, machine learning models, business intelligence tools. So you have all this context separated out and it kind of becomes a little jumbled and just having the context is one thing, but being able to bring it together is the absolute key. So another customer that I worked with of ours, Forsquare, when they were working on their data catalog and their data governance, it was really important to them that they had a central catalog where it wasn't just context from their data lake or context from their transformation tool, but they had Airflow pipelines they were pulling in their storage layer.
They also had defined data products and data domains in code and they sunk that in too. So here they were able to gather together all the different discrete context from the business from the operational side from the data side and pull it together into a single place where they could actually make sense of it. So having context is one thing but being able to bring it together is what really come makes it come alive and what's really important.
Yep. So that one I think I got a spoiler there. So lesson three is that um so we had relevancy matters, context matters. The final lesson is that ubiquity matters too. So we can come gather all this important context together but if it's just stored in one place that really limits the power that it has to unlock uh you know data usage for your users. So something that was really important to us early on was gathering that context but also syncing it back.
So if we have some insight from an upstream production database, we want to provide that to the BI users. We want to provide that to the warehouse too. So we've built things like uh data brick syncback so that if we have some insight that we're able to generate, we can provide that to all the different places that data work is done. So those are the three lessons that we learned. Relevancy matters, context matters, ubiquity matters.
And again, this is what we learned when helping humans discover and use data correctly. Now, with this new world that we're transitioning into, we started thinking, well, what does this mean for things like AI agents and LLM to use this context? Do these lessons still apply? And what we realized is the answer is absolutely. So next up, I'm going to attempt to do a live demo showcasing how providing this context to agents rapidly improves their ability to work and we'll about will really change how we all interact with data in the future.
So let me transition to that now. So let's take a scenario where I am the owner of a coffee shop chain again and I want to understand well what sort of products correlate with a repeat customer usage. What are the products that I want to really sell to my customers so that they'll come back for more and more coffee. So first I want to take the classic the old school world where it's humans discovering data and walk through how that might look.
So I might come in to uh my catalog. So here I'm in data hub and I might type in something like customer visits. So I can hit enter. Then I'll be presented with some searches and again or search results. And again this is where having that relevancy is really important. I don't want to have to look through all the different results that might be relevant here. But I want to be able to trust what comes up on top. You can see there might be multiple platforms, multiple systems.
And again, usually what we'll see here is hundreds to thousands of results. This ranking is super important. From there, as a human, I'll have to come in and try to understand how is the schema used. That's when I might pop into lineage and see context about how is this data derived here. I can see it coming from uh production databases. I can see that being synced by events um into the warehouse. Then I'm also able to see on the flip side how that get gets consumed by downstream uh business intelligence tools.
Again, this is how I'm starting to learn how to properly use this data, how to take advantage of it. Now, there's a lot of other insights that I might get. There's all these tabs to open up. I can see common queries to see, well, this is how the data is used. This is how it's queried by the platform. I can also look at statistics to see, well, what are column, what columns have null values, what are sample values, and what's the distribution of some of these numbers so I can start to understand the shape of it.
There's all these different boxes to start opening up, things like quality, etc. And it puts a lot of load on humans. It's important to have this context. Without the context, I mean, I'm not even going to be able to get started. But even then once I have it and gathering it in of itself it's challenging just because there's a lot of information to know how to use it how to use it properly access control there's a lot to take into account especially with the advent of LLMs and agents and tools like Genie we started thinking how can we make this so much simpler for users so that you don't have to have all this mental load of sorting through the context that has been gathered together.
So now let me walk through our next attempt at bringing all these different things but thinking about it in the new world, a world with agents and LLMs. So here I can come in and I might ask the same question but maybe I can ask it a little bit more in a more natural language way you know. So what are the products that correlate with um repeat uh customer visits? So now here's this at we at this point we've identified that this is a semantic question.
So instead of going to our standard search we're going to go into a more semantic search. So at this point um data hub is interacting with uh its own data hub agent and data hub is opened up APIs via an MCP server so that the agent is able to interact directly with data hub. So this agent is now asking data hub questions. It's going to start doing that search because we're we've learned so much about how to make search relevant.
We're able to actually provide this agent now with not just any old result about customer visits and customer usage and products but the top ones. All right. So the next thing that you can see it doing here is it also is looking at other types of insights that data hub's providing. So you were seeing earlier humans are able to look at queries. Well agents are too. And so in this case from in this particular case the agent's able to identify that there's a third table that should be pulled in because of a join.
So now it's actually expanded its query beyond what is just matching keywords but by using things like uh analyzing join patterns and common query patterns. It's able to expand that search out again by interacting having the agent interact directly with data and ask data questions rather than the human. Below that you can see that the agent's also pulling in context from lineage. So in this case we have upstream insights from production warehouses and production databases.
Like maybe we've got uh column names. A lot of times there's really high quality documentation on a production database but that might not have made it into the warehouse. You know, similarly, we see a lot of times that there's a lot of really powerful insights like metric definitions that are stored in BI tools. So, I did see that um data bricks just announced their own metrics catalog and that's going to help in terms of consolidating this information.
But if there's going to be BI tools out there, there's always going to be context locked away in those BI tools that can be pulled in. So the agent's be able to pull in these different definitions, pull in context from lineage, common join patterns, and then at this point it can provision a package of context that it can hand over to Genie. So we can push that down to Genie and Genie is now not just uh working completely in isolation on its own with the context that it has initially provided, but it gets to work with all of the context that we were providing to it as well.
So again, we're able to supercharge Genie, let it out of the bottle, and now it's able to answer much more powerful questions. Like here, we're going to get a list of those products that correlate with repeat visits, but we're not leaving on its own. And things like what are the product definitions? What are the human language versions of these products, not just their IDs? Um what does it actually mean to define a repeat customer?
At what point does a customer become a repeat customer? G doesn't have to try to figure this out on its own, but it's getting provided all that context from data hub. Again, we think this is just incredibly exciting and really the future for how people are going to start interacting with data. So, as since data hub sits in the center, it has the ability to both gather all this context and then federate it out. So, for example, we also are exploring interacting with other systems as well as data bricks.
So you don't have to have your end users figure out which tool to log into, figure out how to go qu do semantic queries in those and provide semantic layers in theirs. But your end users even without knowing how to construct a SQL query without knowing where your data is stored, they're going to be able to come into this one front door, ask questions, and instantly get answers. All right, so I'm going to come back to the slides just for one last wrapup.
So we did the demo and I just want to talk through again what you saw kind might have felt a little bit like a magic trick. So how did we make this work? So the human comes in and asks that natural language question. Now this can be something either to uh an existing uh LLM agentic tool like claw desktop cursor we support those or it could be directly to data hub and data hubs agent. At that point the question is going to be received and the agent's going to start working.
That's when it will start making calls to data hub's MCP server which exposes out all the different tools that data hub can do. So things like searching, looking at lineage, accessing downstream tools like Genie or other warehouses uh agentic querying tools. So now the agent will start interacting uh with data hubs MCP server. First it's going to ask questions like hey here's uh terms that I'm looking for. Then it might start to ask questions like all right I found a couple results. are these the trustworthy ones.
You know, prove to me that these are trustworthy. After that, it can start to do that query expansion that you were seeing. And then once it feels like it's got a set of here's the tables that we want, this is how we're going to do our query. At that point, it can forward that question down for the last mile to Genie, have that natural language query executed and parsed and then return the result right to the user. And the thing that's really exciting about this is Genie is just one of these tools that we can interact with.
As long as something's open to interacting agentically or via an MCP server, it can be plugged into this mix and we can continue to bring more context to that. So, thank you so much for joining. That's the end. Um, if you want to learn more and see the magic, we have a booth E13 317 uh over that way. So, swing by if you want to talk more or see a demo. Um, and also make sure to fill out the survey about this talk if you liked it.
So, thank you so much for joining.

Learn more about the Databricks Data and AI platform.