Skip to main content

Agentic Data Foundation: Building Enterprise AI at Scale

Summary

  • The Agentic Data Foundation is Databricks' new infrastructure layer designed to support thousands of AI agents that need access to the latest enterprise data at massive scale, with all data stored once in open formats and governance flowing through all workloads.
  • Databricks announced three new capabilities: LakeBase (managed PostgreSQL on the lake), ELT App (CDC-based ingestion bridging operational databases to the lakehouse), and LakeHouse RT (direct agent querying of warehouse data without copying).
  • Together these capabilities remove the complexity and latency that has made building AI applications difficult, so agents can focus on business logic rather than data infrastructure management.

Agentic Data Foundation: Building Enterprise AI at Scale

Watch: Agentic Data Foundation: Building Enterprise AI at Scale
Building thousands of production AI agents and applications requires a new approach to data infrastructure. Agents need context from all your data, at massive scale, without latency or data duplication. The Agentic Data Foundation is built on a unified data platform where all engines are optimized for their workload, all data sits once in open formats, and governance flows through all of it.
Databricks has announced three new capabilities: LakeBase, a Postgres database on the lake with infinite scale for agent and application development; ELT App, which brings data from thousands of operational databases into the lake immediately and transparently; and LakeHouse RT, which lets agents query your warehouse data directly without copying. Together, these remove the complexity and latency that has made building AI apps difficult, so your data is always ready and your agents can focus on business logic.
🤝

Chapters

FAQs

What is the Agentic Data Foundation on Databricks?

The Agentic Data Foundation is the data infrastructure layer Databricks built to support enterprises running thousands of AI agents, providing massive-scale data access without latency or data duplication. It unifies data engineering, warehousing, and data science on a single platform where all data lives in open formats with governance and permissions flowing through every workload.

What is LakeHouse RT and what problem does it solve for agents?

LakeHouse RT lets AI agents query data stored in your data warehouse directly, without first copying it to another system. This removes a key source of latency and complexity for agents that need access to up-to-date operational data, eliminating the need for a separate ETL step before agents can act on current information.

What is ELT App and how does it differ from traditional ETL?

ELT App uses change data capture (CDC) to bring data from operational databases into the lakehouse immediately and transparently, without the batch delays of traditional ETL pipelines. The approach means changes in source systems are reflected in the lake in near real time, giving agents access to current data rather than yesterday's snapshot.

Why does the scale of future AI agents require a new data foundation?

As AI-accelerated code generation scales up, Databricks describes a future where enterprises run tens of thousands of agents simultaneously to help run business processes and assist every employee. Each of those agents needs the latest data at high fidelity and high concurrency, which is why the underlying data foundation must be designed for massive scale from the start rather than adapted incrementally.

Full transcript

[00:19] All right, welcome back to the Summit live broadcast here at Moscone Center. Uh, Ali Kaplan and Maria, this is our first time we're co-hosting together. Ooh, cannot wait for this. Super excited and Shanku, welcome. Welcome. Thank you. It's exciting to be here.
[00:34] Want to hear your your role and and what what you do here. Yes, so I'm VP of product at Databricks. Uh, I've been here about 4 years and uh been involved in a lot of things, but lately really kind of thinking about what do we need to do to build this kind
[00:52] of foundation on top of which all these agents and apps are going to run and bring value to customers. So that I my team owns lake base, Databricks apps uh and a few other things. Yeah. Yeah, so to follow up on this, basically
[01:07] what I do in a minute I'm talking about AI with all our customers and about agents, right? And the first thing is, okay, where is the data? And now we're talking about this new data foundation layer. Can you tell us a little bit more about that? Yeah, so you know, I think a couple of
[01:23] things that we're seeing like everyone is trying to get value out of AI and the way you do that is building apps and agents, right? Like that's even at Databricks. We used to show you notebooks, now we build apps. And every one of those apps has agents inside. And
[01:38] so that's kind of the one thing, but then the second thing is because of AI, people are just building way more of these. Like we talked about this earlier, I think Nikita shared it like in the next 12 months we're going to create more code across the world than in human history, right? And so like
[01:55] there're going to be thousands of these agents and the way that they're going to run business processes, they're going to help every human and the way that they they do well is by having context and that context comes from like the latest data. And so you
[02:12] need a platform on top of which you get all that context and you get it at massive scale with tens of thousands more agents than we have today. And so a lot of this kind of it's funny it's funny name but like the Agentic Data
[02:29] Foundation is the foundation on top of which you can build agents because it's a data platform that gives you massive scale and removes all the complexity and and latency so you can get the best data. Yeah, that name makes a lot of sense since it's all about data and AI.
[02:47] You need the data and the AI but what's fascinating is, you know, from the audience probably about half of them are new. This is the first time learning really about Data Bricks but a lot of them were here in the very first Spark Summit but what I've seen love to hear
[03:02] your impressions of what happened in the last year but for from my undertaking people were trying it out maybe two years ago, maybe production maybe not a year ago but now they're like scaling out massively.
[03:18] So like what what are some of the opportunities? Like how can you better scale today than you could Yeah, I mean I I think like what's what's been amazing is I think AI is helping build a AI and it's building these agents, right? Now like the cost of coding these, putting them
[03:34] into production has gone way down and so um what what we're now seeing is this kind of like explosion of people building agents, applications, putting them in production. You see this in like the amazing success with Data
[03:50] Bricks apps, you see this with how people are using the agent platform, all the things people are doing with Genie. Um but what's interesting is it's sitting on top of all this like innovation is sitting on top of the data platform that was kind of like
[04:06] built over the years. Like everything started with your data and databases and then when warehousing came in and data science came in and machine learning came in like it was a challenge of how to get that data out. And what's interesting is data
[04:23] bricks even from that spark data that you're talking about has been all about trying to unify these workloads so that people have to do less work and they get the right data in the right workloads that they need. So first we combine data
[04:38] science and data engineering then with lake house we like brought in the world of data warehousing into that and and then now we're taking some new steps that that we announced today and it all sits on top of you know your data in the
[04:54] lake in open formats that you can use it with like any any engine and any tool. Okay, so then you described on that we have a lot of agents being built right from many different by coding tools. We have a lot of different data in different systems that we are trying
[05:11] to simplify. Yeah. Why is this important for agents though and if we think about the simplification what are the layers of the simplification at play? I mean agents need context right and the the challenge and I think Ken talked about it really well in terms of like
[05:27] from the business perspective is all the context lies in a lot of places in the organization and so you know some of the first steps of that is just bringing in all the all the data that's in your like slack messages, your email, your other systems. So we built like lake flow
[05:43] connect that makes it easy to bring all the different data in and because of that Genie gets better, right? Um we've created, you know, if you've got like sensors in your organization in your in your footprint that are kind of collecting real-time data, we've created
[05:59] this thing called Zero Bus Connect. Yes. It will like let you bring in, you know, real-time data without having to like build some complicated pipeline. But then the biggest source for a lot of companies, like we talked to companies all the time that have tens of thousands of databases, they're creating new data
[06:16] data all the time and previously they had to build these CDC pipelines and we were joking like in the keynote like CDC kind of should stand for continuous data corruption. Like schemas change, pipelines fail.
[06:33] And what we want to what we're now doing with what we call ELT app is making that possible, like finally bridging those two worlds and letting so any data in a Postgres database in lake shows up for use with analytics, for
[06:49] your agents, for your machine learning pipelines in almost immediately, right? And there's no CDC, there's no data failure, there's no like schema changes breaking you. Um we just take care of that. That's a big deal. I used to be an
[07:04] Oracle DBA and application, but sometimes companies have thousands of tables, tens of thousands of tables, hundreds of thousands of hundreds of thousands or millions. Now there's hundreds of thousands of apps. Like it's scale is crazy. And then all of this is just so simple.
[07:20] I know you haven't tried it yet. So Data Bricks makes it so much easy to get started. Yeah. What do you think of Holly's Holly's demo? It was it was amazing. She did a great job. What did you think? No, no, I loved it. I mean, I love Holly
[07:36] on stage in general, right? And I said and a fun story is she was the co-host. She was doing the demo and And she ran over here, I saw her, yeah. the funniest thing and she did great. So, we have just a couple minutes left. Uh
[07:51] and then you can ask the final question, but you know, where can people go to learn more? So, so the two big things that we kind of launched around this Agentyc Data Foundation, the first one is this ELT app thing, which starts with
[08:07] LakeBase. LakeBase is our Postgres database, it runs on the lake, it scales infinitely, uh it it's it has a great developer workflow, so you can bring your agents to work and build applications on top of it. And the nice thing is with ELT app,
[08:23] all that data is going to show up in your lake for your analytics and your agents. And all you have to do to do that is start using LakeBase. And you know, we kind of joke like when it comes out and we roll out ELT app, you'll just be turning off your pipelines. That's all you have to do. Now, the other thing
[08:38] that we launched is this I I would say it's the biggest thing that we've done for the lake house since the lake house and that's called LakeHouse RT. Like it was thought to be impossible to like break that barrier where you could use data directly from the warehouse without
[08:54] copying it out. And now with LakeHouse RT, you can do that. So, operational analytics apps, agents can just use data from the warehouse directly. Okay. Okay, so then to to summarize on what we're saying is that we have one data layer as always on top different engines optimized for
[09:10] different tasks. All of them can be fetched and used by agents. And this is the whole simplification. One copy of data, no copying things around, single governance, but then you get best workload performance with the agent the engines that you pick. That's basically at the core of what ADF is.
[09:26] Super happy. All right, well, thank you so much and we're going to go back to the breakout room. Thank you so much for being here.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.