How Data Warehousing Changes in the World of Agents

Summary

Agents query data differently than humans—executing more parallel loops, iterating over results, and exploring metadata—which has already driven a 10-fold increase in data warehouse concurrency requirements with a further 100-fold increase expected as agentic deployment scales.
The shift from decision support to autonomous decision making raises the stakes: when agents act directly on data rather than presenting it to humans, stale data or incorrect results translate immediately into incorrect business actions with no human checkpoint.
The Databricks Data and AI platform response includes Delta Live Tables for always-on streaming ingestion, Lakeflow with Enzyme for AI-powered incremental refresh, Photon for consistent low-latency analytics at agent concurrency levels, and Unity Catalog lineage to audit every agentic action.

How Data Warehousing Changes in the World of Agents

The rise of agentic applications is reshaping what we demand from data warehouses. Agents don't operate on yesterday's batch — they require fresh, consistent data with sub-second latency to reason, decide, and act in real time. In this video, we examine why traditional data warehousing patterns, built around scheduled ETL and human-speed concurrency and latency, fall short when the consumer of your data is an army of agents making thousands of decisions per minute. We address the challenges this shift introduces: tighter freshness SLAs, evolving governance when agents —not people — query your data, and the complexity of always-on pipelines. We then explore how real-time lakehouse capabilities — streaming ingestion, incremental materialization, high concurrency and low-latency query serving, and unified governance — support agentic applications at scale. this video offers a framework to make your lakehouse real-time and agent-ready.
Talk By: Ruwen Henning Hess, Director, Product Management, Databricks ;

Chapters

FAQs

How do agentic applications change data warehouse concurrency requirements?

Unlike human users who query data at human speeds, agents execute queries in parallel loops and iterate rapidly over results. This behavioral difference has already driven a 10-fold increase in concurrency requirements at Databricks, with a further 100-fold increase expected as agentic application deployment scales across enterprises.

Why does removing humans from the decision loop raise the stakes for data freshness?

When a human reviews a dashboard before acting, they can apply judgment to potentially stale or incorrect data. When an agent acts autonomously, stale or incorrect data translates directly into incorrect business decisions with no human checkpoint, making sub-second data freshness a reliability requirement rather than a performance optimization.

What are the three design principles for an agent-ready data warehouse on Databricks?

The three principles are freshness (ensuring data is current enough for agents to reason accurately), scale (supporting the high concurrency and parallel query patterns agents generate), and control (maintaining governance and auditability through Unity Catalog so every agentic action is traceable). These map to specific Databricks platform capabilities like Delta Live Tables, Photon, and Unity Catalog lineage.

What is Lakeflow with Enzyme and how does it address agent data freshness?

Lakeflow with Enzyme is an AI-powered incremental refresh capability on the Databricks Data and AI platform that automatically determines which portions of a pipeline need re-execution when source data changes. Rather than running full pipeline refreshes on a schedule, it incrementally updates only what is necessary—enabling fresher data with lower compute cost for agents that need near-real-time accuracy.

Full transcript

[00:07] Good afternoon. My name is Rueben Hass. Thanks for joining us today. Uh we're excited to share with you how we think data warehousing is evolving in the world of agents. Before I start, this presentation may contain some forward-looking statements.
[00:21] They're accurate to the best of the information at this time. Uh but they don't represent a guarantee of future outcomes. Agents are fundamentally changing the way we're running businesses, the way we're building infrastructure for
[00:34] businesses. Among other things, they've led to an extraordinary growth of software out software development output. Uh by some accounts, the amount of software being coded, being shipped in
[00:46] the coming year will exceed what has been shipped in the previous years combined. Similarly, it's driving usage of data warehousing to unprecedented levels. We've already
[00:59] seen a 10-fold increase in concurrency requirement. We expect another 100-fold increase in the coming years. The impact goes a lot further than just concurrency and and query volume. There's a lot of impact in terms of the
[01:12] nature of queries and the nature of data, too. So, there's changing user behavior. Users, empowered by agents, are querying more and they're querying different data. They're querying data they're not as familiar with.
[01:24] The same time, agents as consumers are different. You see a lot more parallel execution, more looping, more iteration. And there's more metadata exploration than you'd have from a pure human user.
[01:36] And then finally, for businesses that are comfortable driving direct business action on agents, the role is actually changing. The stack is now no longer decision support. It's moving into actual decision making. You're moving from dashboards to decisions.
[01:52] That in turn leads to agents exacerbating a lot of challenges that we've already been facing in data warehousing. First, on developer agility and in the maintenance burden. Velocity is critical if you're facing an innovation wave as we are now with
[02:04] agents. But at the same time, you have a rapid proliferation of apps and of pipelines that's adding to a maintenance burden that we already face. Second, performance and scale.
[02:16] Agents can only reason about the data that they can see. But there's an ever-increasing amount of data and there's an ever-increasing demand on freshness. At the same time, at the sort of data warehouse execution level, the latency
[02:29] dependency that agent looks can have increases the requirements on on on the latency. At the same time that you have higher concurrency requirements. And then finally, government Sorry, governance and risk management.
[02:42] Pulling humans out of the loop and moving agents to decision-makers increases the risk profile. That's exacerbated by an increase in pipelines and apps generated by agents. All of that leads to a situation where
[02:55] the conventional approach of creating a site stack is no longer viable. Each challenge isn't new, but each one of them is moving from friction to fracture.
[03:09] To rise to the challenge, you need to tackle each aspect head-on. You have to build for agility, moving to one source of truth, and one pipeline. And you have to move to something where you have reusable data products.
[03:21] You have to build for scale. You have to think about freshness SLAs in the sense of like from the birth of data all the way through ingestion, transformation, surveying, and then the decision-making by an agent, how long
[03:33] does it How long can that take and how long should that take? And then building for scale at the sense of real-time workloads, the concurrency requirements, the latency requirements against the full data.
[03:46] Finally, you have to build for control, starting with governance from the ground up. And using open data formats to provide to prevent lock-in.
[04:00] We approach this with an end-to-end data stack that unifies real-time ingestion, processing, storage, and serving on an open platform. We achieve this with three major components. Real-time ingestion and transformation with Zero Bus Ingest and Spark
[04:12] Declarative Pipelines that eliminate duplicate systems for real-time ingestion ETL. Real-time Lakehouse that can run queries at millisecond performance and massive scale directly on your lake.
[04:24] And Unity Catalog for unified governance. It's not working here. Sorry. Let's have a look at what this No, now it's working.
[04:56] Okay, that works. Let's have a look at what this looks like. Data producers push data into Delta tables or Iceberg tables with Zero Bus Ingest where you can use Spark Declarative Pipelines for real-time transformation.
[05:08] And at the last step, Lakehouse RT reads that data from Delta tables directly and serves them to absent agents at low latency and high concurrency. So, you have one pipeline and that results in lower latency. You
[05:21] move from batch pipelines to continuously fresh data and analytics. Lower complexity. You eliminated a lot of duplicate systems for streaming ETL and serving. And lower cost, both in terms of maintenance overhead and in terms of
[05:33] data copies that are no longer needed in terms of systems and clusters that are no longer needed. Let's cover ingest, processing, and serving a bit more in detail. Zero Bus Ingest allows producers to push
[05:45] data into governed Unity Catalog tables without without requiring a separate message bus in the middle. Allowing for near real-time rights and high throughput. This streamlined ingestion infrastructure eliminates data hops and
[05:57] removes complexity and cost. It's a serverless connector and it automatically scales and handles incoming connections. Any app can integrate with Zero Bus
[06:09] with ingest SDKs or communicate with via the APIs. So, you create a table on the Databricks platform and then you just start pushing data. There's no cluster setup, there's no maintenance.
[06:21] Zero Zero Bus ingest is always on and always ready to accept your requests.
[06:38] There we go. All right. Once the data is in the tables, for real-time transformations we have materialized views. There's three benefits here. They accelerate queries because results are precomputed. At the same time they're
[06:50] incrementally refreshed with enzyme, sorry, an AI-powered optimizer that processes only data as it that has changed. So, it avoids expensive recomputes.
[07:03] And finally they're declarative and fully governed. You write an SQL query and the engine handles the refresh and you get fully Unity Catalog lineage and access control. So, it's not just a cached query.
[07:16] It's a governed incrementally maintained ETL pipeline. Write SQL, you get the pipeline. And real-time serving is enabled by Lakehouse RT, our new real-time data
[07:28] warehouse powered by the new Photon engine. It delivers consistently fast performance uh tens of microseconds milliseconds and low latency floor. And it can serve thousands of users of
[07:40] users and agents without lag or contention. You don't need to make a copy of the data. It runs natively on the open lakehouse formats like Iceberg and Delta. And everything is governed using the
[07:53] Unity Catalog. The resulting architecture lets agents work at top speed on on the best data while minimizing complexity and cost. There are no extra pipelines,
[08:06] so that eliminates the risk, but also the cost of maintenance and extra data movement. There are no data duplication. Query directly at the source. There's no subsets. There's There's maximum freshness. There's no extra hop for for some side stack or anything like that.
[08:20] And there's no data inertia. Agents move at top speed with a complete full context data.
[08:32] Come on. All right. All of this is running on the same open format tables you're used to and governed by Unity Catalog. There's zero copies, zero data drift, and a unified policy. You have one rule and it works across the all of the data.
[08:45] And finally, you have complete lineage and auditability for agents. So, now you can unleash your most ambitious AI use cases with confidence.
[09:00] There's never been a better time to be a builder, but there's also never been a more challenging time for data platforms. We tackle this by unifying real-time ingestion, processing, and serving in a single open platform.
[09:12] By building for agility, scale, and control from the outset, your data warehouse can thrive in the world of agents. Thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.