Sponsored by: IBM | Extend your Databricks Architecture with Real-time Context

Summary

Stale, non-real-time data is identified as the root cause of AI project failure across enterprise deployments, and IBM's watsonx.data enables zero-copy unification of Databricks with hybrid data estates including legacy, sovereign, and cloud systems.
The architecture delivers real-time context to AI agents from structured and unstructured data across hybrid data landscapes without requiring migration, using open format support and Iceberg and Delta interoperability.
IBM's integration capabilities include data masking and security for streaming sensitive fields, agentic RAG on streaming data, and multi-cloud agent data access patterns for enterprise-scale deployments.

Chapters

FAQs

What causes most AI project failures according to IBM?

IBM and Gartner research cited in this video identify stale, inaccurate, and non-real-time data as the root cause of failure for many AI projects. When agents and analytics workflows operate on data that does not reflect the current state of the business, the decisions they produce are unreliable regardless of model capability.

How does IBM watsonx.data extend the Databricks Data and AI platform?

watsonx.data enables zero-copy unification of Databricks with the broader enterprise data estate, including legacy systems, sovereign data environments, and multi-cloud sources, without requiring migration. This allows organizations to leverage their existing Databricks investment while extending AI agent context to data that lives outside the lakehouse.

What is agentic RAG on streaming data and why does it matter?

Agentic RAG on streaming data means that AI agents can retrieve context from data that is continuously updated in real time, rather than from static snapshots ingested in batch. This is essential for use cases like customer service, operational monitoring, and fraud detection, where agents need the most current information to make reliable decisions.

How does IBM handle governance for sensitive data in hybrid deployments?

IBM's architecture includes data masking and security controls that apply to streaming sensitive fields, ensuring that agents accessing real-time data do not expose protected information. Governance policies travel with query execution across the hybrid data estate, maintaining consistent access controls regardless of which system holds the data.

Full transcript

[00:08] what we want to uh talk to you about today. And first, let me introduce myself. I'm Ed Anov. Uh I am uh the the chief product officer for data at IBM. And um uh we're really happy to be uh here at this conference, a sponsor of
[00:21] it. Um uh we have a uh great uh partnership with data bicks and uh a lot of what I'm going to talk about today is how you extend um your uh data bricks uh
[00:36] investments and architecture across uh the entirety of the data state and most importantly do so um leveraging the real time data uh that uh uh our architecture
[00:51] uh enables. And this real time um component um is is really important when we go and uh survey our customers. Um when we talk to
[01:03] analysts, when we look at uh the general level of readiness for being able to go and uh drive real time data uh into your
[01:15] analytics, into your decision-m processes, um what we find is that um uh the majority of folks out there um are still in the very early stages of making
[01:27] that happen. Um yet we see time and again that this is critical for being able to go and and actually make these projects succeed. And when we look at the key
[01:40] uh requirements for success in particularly leveraging this data in AI contexts um we see or at least um through some of our own work and uh what
[01:52] Gardner has seen uh is that this is the root cause of failure for many projects is uh AI data AI ready data that is is not real time that is not accurate that's not up toate and so Um the way we
[02:08] look at this is that um that we need to solve these issues and IBM has been working with a number of customers to do exactly that. And so um what we see is
[02:22] that whether it is for uh predictive analytics um for being able to uh reduce uh customer downtime, whether it is for increasing customer satisfaction
[02:34] um whether it is for for driving incremental uh revenue um or driving faster time to market for a variety of products that coupling real-time data across the data state into your
[02:48] projects. um can make a significant u material difference in the outcome of those projects. So what we're doing today is sharing some of the insights on what that means from both a planning and
[03:01] architecture standpoint. So um you know all of you have gone uh here presumably you're here at the databicks conference you've made a a pretty
[03:13] significant um uh investment of of time and uh knowledge and experience in being able to go and use data bricks in uh and
[03:25] the capabilities it brings to bear in accelerating these projects. Agentic AI is bringing a lot of new requirements to the table. You saw a lot of these this morning. Um, but what we think is that
[03:37] providing a more uh holistic access to data across the enterprise ends up being the missing link in a lot of those situations. And so when we look at that
[03:49] that real time connectivity not just connecting it um you know through the the means of moving the data whether uh whether through zero copy means or whether through uh real-time
[04:03] streaming um coupling it with governance that allows you to craft policies that travel with where the queries where
[04:15] where execution happens. happens um being able to do that in a hybrid fashion. So the reality is for many of us the data state spans uh legacy systems of record. It spans um uh
[04:29] self-managed deployments, sovereign deployments as well as what's in the cloud. Um and being able to go and and for operational operationalizing that um becomes essential. And so that's really
[04:43] how uh how that influences um uh our view of that. The first piece of how we try to approach that is through this zero copy unification.
[04:55] Um what we've seen has been that over um for really the last decade but particularly this has been accelerating um we've seen this movement towards zero
[05:07] copy um open formats iceberg delta sharing other mechanisms for being able to go and make it possible for data to
[05:20] not have to move but be able to uh bring the compute to the data in a variety of conference. We've expanded and extended the IBM ecosystem and you can see many of the pieces of what we bring to the table to embrace that. Watsonx data uh
[05:36] which is um our platform suite for enabling this allows us to leverage um and provide an optimization layer that allows you to go and make this data
[05:49] ready to your cloud data platforms. Uh this is really the basic pieces of how we think about um connecting this. This is the starting point probably for a lot
[06:03] of your projects that are involved in these hybrid use cases. However, where things start to get interesting on that is when you add real time to the picture. And now particularly in these
[06:15] AI use cases, what this te's up is this idea of real time context engineering. So what I just showed you was really kind of how we think about or how we've thought about the architecture. But um
[06:27] as probably most of you know at this point earlier this year we added uh Confluent to the portfolio. And so Confluent complements that architecture in some really exciting ways because
[06:40] what we now are able to do is give you the mechanisms to connect to your systems in real time and as you're going to see in a minute uh bridge the access of that data into uh more tabular
[06:54] oriented uh access patterns. So um what you see on this slide is really the basis of of how we think about uh this
[07:06] architecture. What you can see on the left um is that we are able to go and ingest this data from a variety of sources. Um we're able to basically
[07:18] connect to them through uh the streaming architecture. we're allowed to we're able to process it in place and we're able to apply policies at the point of connection and in the stream. Um the
[07:31] result of that is that we are able to go and create reusable real-time data that we can then deliver and make use of within whether it is analytical use cases or whether it's actually uh
[07:44] agentic use cases. Part of the means by which we do this is capabilities like table flow uh that allow us to take this and bridge it into uh iceberg tables
[07:56] allow us to bridge it into uh data bricks. And so um this really brings you the best of both worlds in terms of taking this real time data uh and making it available to consumers throughout
[08:08] your architecture. So let's look at what that um uh makes possible. And if we look at the requirements for what you are looking at
[08:22] with the design of your agents, you're really looking at at three key areas. You're looking at going and saying first of all can you get uh timely access to
[08:34] the entirety of your data state in an interactive fashion um where you are able to go and quickly have uh as the agent as the agent is building out its context um be able to request and have
[08:48] delivered to it uh the data that it needs across um across the hybrid data estate. We need to have rich semantics around it. And so this has been one of the key areas that we've worked very
[09:00] closely with Confluent to be able to do is to make sure that this real- time data is also uh carrying the semantic information uh around uh the meanings of this data that's being transported. And
[09:14] again the other piece of it really excited about has been what we've done to unify the governance layer uh with what Confluent brings to the table. So um what this gives you is a real-time context layer um that when you go and
[09:29] and whatever agentic infrastructure architecture you're building around um is able to leverage this. So as we do this um there are a number of elements that we bring together to do
[09:42] this. So I've spent a lot of time talking about uh the first two pieces uh of this at a high level. Um this represents the components that have been part of the Confluent uh product set
[09:56] feature set but what we've done has been to package that up in Watson X data um and give you a very uh streamline mechanism for again being able to deploy
[10:08] this and have a connected data layer uh that goes and and connects all of your systems of record uh into whatever analytics platform you're looking at one of the elements that
[10:21] makes this so flexible is what we talk about as this composable data layer. And so what sits within that um is a variety of different open-source engines as well
[10:35] as key partner technologies. So within that you have uh real time operational data that comes from data stacks um which is the Cassandra based technology that we use. if so if you're adding
[10:48] scale out data as well as things like GPU accelerated um technologies for being able to go and do uh these queries on your on premise data uh in an
[11:01] accelerated way. Um since these are all part of this same real-time operational context you're able to go and leverage these uh however you see fit. So I've talked a lot about this governance
[11:15] component um as a key requirement here. Let me talk a little bit about what that means. Um so the first thing when we talk about governance at scale the definition of
[11:27] scale in this particular case is really that um first of all obviously in terms of of size and scale of of the data under management but it becomes really important for thinking about scale in
[11:40] the scale of your heterogeneous architecture. So live policies means that we are able to go and apply these at the point of decision. Um this is something that becomes really important
[11:52] when you are connecting to systems with sensitive data before it gets on the stream before it gets uh into the tables before it gets to the agents. Um policies are enforced at the points of
[12:04] the connections. Um the fact I alluded to this before the semantic understanding in many cases your policy infrastructure has not evolved to take meaning into consideration. It applies
[12:18] very literal rules and it will miss um and in many cases your agents will be able to leverage this to get around rigid policies. Remember the agents and everything that that um we see in these
[12:32] systems. The agent wants to provide the user uh with the best answer that it can not necessarily from the perspective of of being fully cognizant of of the
[12:45] policy regime. name it's operating under. It may actually um work around policies or technicalities. And so if your policy enforcement is not semantic based, then it becomes possible for data
[12:57] that shouldn't be brought into agent results to happen. The audit trail piece and again this is where having something that allows you to have auditability across the entire data state whether it
[13:10] is your cloud-based systems um or your self-managed ones becomes so important. um being able to trace what's happened whether that's being used for uh for for manual review or into or being fed into
[13:23] a secondary secondary agentic system becomes very important. And finally again that moment of action compliance. Um these are what we mean when we talk about this comprehensive governance
[13:36] mechanism for being able to use this type of data. So, so one of the things we often see is that people take this challenge of how do I get to heterrogenous data as simply a
[13:50] connection architecture problem. Um they go and miss the point that it actually ends up being uh a governance and policy problem. Not just connecting to this data but figuring out how to uh how to
[14:04] do so in a safe way. Um, and so let me double click on this last piece on the hybrid data state. And so this ends up being a really key part of why so many
[14:18] um uh of our customers partner with us in terms of applying uh data bricks um uh uh or building out data bricks architectures because what we enable is
[14:31] the access to uh data that might be sitting in the mainframe um might be operating within sovereign data requirements um and we are able able to do this in ways that leverage our zero
[14:45] copy investment. Um the end result on this is that you're able to basically build on top of this uh this data bricks foundation. Um, some of the key pieces here, leveraging the
[14:57] open D uh lakehouse architecture, being able to go and complement that with what we do from a federated hybrid access, being able to go and leverage what we do
[15:09] in terms of being able to build an enterprisewide context graph for leveraging semantic content and being able to get to and one of the important pieces of what all this enables is a lot
[15:21] of the things that we bring to the table from unstructured ed data retrieval. So when we talk about this agentic rag, what we see is that often times um it's
[15:33] not just uh how do we leverage the analytics data, a lot of the key inputs for the genic reasoning as well as the content that then gets um generated as a response often lives in data that sits
[15:49] in the unstructured components of your data state. Um and this agentic rag piece becomes incredibly important. We've done a lot of work on this with open rag uh which is is our agentic
[16:01] stack on top of open search. So if this is an area that you're doing um uh a lot of work on in terms of of open search and leverage leveraging that within your
[16:13] architecture. Uh I'd encourage you to take a look at that. We actually have some of the folks from um from our open search team uh here with us uh today. Um final point there, governance, which I I
[16:25] think I've already spent a bunch of time on. Um so um one of the things or or really sort of the the um uh you know the the pieces around that
[16:38] that uh that that I think become actually let me just I think I just hit the the wrong slide advance. um uh oops
[16:50] um uh was uh
[17:03] we somehow uh did not uh update our slides. I think I missed the call out. Uh uh so one of the things that that we wanted to be able to do was um uh have a conversation uh with um I have uh Eric
[17:16] Duffield here who uh who from IBM consulting who uh uh is uh uh has helped on a lot of these implementations and really wanted to to bring him up and uh
[17:29] and talk a little bit about how we see a lot of these applied uh in terms of our uh our customer deployments. And again, I uh I apologize for skipping past your slide on that, but but it looks like uh
[17:41] it looks like there was there was an update glitch. So, um Eric, you have been um involved in many many of um uh deployments actually just from by way of background. Uh do you want to introduce
[17:54] yourself and and and how you came into IBM? >> You bet. Hi, everyone. Um I don't know if I'm on yet. I'm number two. Here we go. My name is Eric Duffield. I lead uh data transformation for IBM consulting which is the other part of the IBM
[18:07] business. He's the chief product officer in IBM technology which makes all things great in AI and data. Um and so in consulting we deal across platforms across our ecosystems of partners to help large organizations really address
[18:19] the innovation that they want at scale and we're trusted um to find success for our customers. And so I came to IBM about a year and a half ago. I was a CEO and founder of a firm in the data space
[18:31] that was acquired by IBM. I've got 30 years in this industry. Really part of the last three generations of technology and so a lot of um what we're doing now is rebuilding and rethinking what was
[18:43] created in prior generations of my work. So one of the things that becomes uh really important in these deploy these uh uh projects has been thinking about
[18:55] hybrid and hybrid by design. So one of the things I could hope was hoping we could uh chat
[19:07] hybrid essential for being able to scale these projects? Obviously people start with um oftent times a pure cloud approach but very quickly start to go and say how can how can we build this out? There's no doubt cloud technologies
[19:20] and cloud architectures changed everything that we do, but it's what we're seeing now is the maturity and the evolution is circling back to probably the largest data estates that we have out there. How many in the room have some sort of mainframe that's part of
[19:33] your data estate? We've got a few and I imagine it's it's significant for those that do. Um, how many have some sort of on-prem data that you you do integrate or you'd love to have part of your data state and part of
[19:45] your ecosystem? Okay. when we when cloud architecture started to change about 10 11 years ago um everyone moved up there and it was all about and I built companies around migrating data to the cloud but now what we're seeing in the
[19:58] evolution of design patterns is circling back to what really is exabytes worth of data that exists on mainframes. So, as you sit in this room and your your cell phones are pinging, you're texting, you're getting messages, you're running
[20:10] transactions on mainframe servers right now. When you swipe your credit card, um when you book your flight home, it's all happening on mainframes. And there's a 97% chance it's probably an IBM mainframe. And so that creates exabytes
[20:24] worth of data out in the ecosystem that companies need to have as part of their agentic future. And so now that we've moved to the cloud, we've moved a lot of workloads to the cloud. We're integrating core platforms like SAP. We're circling back to how do we access
[20:36] that data because it's incredibly performant, very efficient, highly scalable what's happening on those. And so they need to be part of how we think about our business and how agents understand our world and make decisions
[20:48] and integrate with. And so that's that's very critical. So hybrid is not only multicloud, which most of our customers are multicloud um or many of them, but hybrid's also your on-prem or mainframe
[21:01] data existing and in the context of that cloud data that's essential. And so you're able to look at your entire data state and create truly that world model that exists around how you think about how you operate, how you execute your
[21:14] business. And so hybrid now is one of the most common design patterns and it's basically circling back how do we integrate with that and some key technologies have advanced to allow us to do that and I think and you saw it a
[21:26] bit in in the Watsonx data the um the adoption of iceberg and open table formats are allowing us to do that and create a plane where we can integrate this onrem and and mainframe data with
[21:38] our cloud estates and have it governed and managed and accessible where you can run local um workloads on the mainframe or on your premises, you can run them in the cloud and the cloud can reach down and access your mainframe data and
[21:51] leverage it too. So, it's it's essential for many businesses. >> One of the things that that um uh you know that that I touched on in the content there was really sort of thinking about
[22:03] uh the governance strategy on that. What are some of the things that typically go into
[22:16] a day one is that day zero day one or a day two challenge? >> It is very early on. In fact, the the there's two things that I have in almost every conversation with an enterprise right now because they're the two
[22:28] missing pieces of architecture that you didn't need three years ago. Um one it's your knowledge graph. Call it your world model, your context engine. There's a million names for it. Semantic layer.
[22:40] And you all know since you're in this room how those were built up over the past. Every BI tool you implemented, you launched its own semantic layer. Every new in database instance, you created its own semantic layer. And semantics
[22:52] have been theoretical, not actual or physical. And there certainly aren't usable by agents in how most of us built them for the past 30 years for me. Um, many of you uh uh much more recently.
[23:04] But so there's the knowledge graph and then there's um governance in the context of runtime. So most people are still trying to finish the catalog project. How many here use a data catalog of some variety? And are how
[23:16] many need one and don't even have a catalog? A few. So um that's kind of the state. If you look at large enterprises, it's they're finishing up their you name the vendor uh big data catalog project and
[23:31] um it's cumbersome. And what happened is
[23:43] curate. And so um that cataloging now and governance is not just about catalog. It's moved really to runtime policy enforcement and um policies actually being created and managed by
[23:55] agents. And so your creation of policies and the management of policies is agentic driven now. And that's a whole new world. People haven't thought of it in that context. And so it is a key understanding. Now if you take that and
[24:07] say okay if I have one cloud database, one cloud instance, I can get my mind around that. Now what if I have a mainframe? What if I have some on-prem datas? What if I'm multicloud? What if I have data sovereignty issues? The
[24:20] complexity here compounds because agents create work for other agents. Those policies have to be executed and understood and known when to run within this. It is a completely addressable
[24:32] problem, but it is highly complex and it's one we haven't had to think about in the prior generations of our architecture patterns. I guess the last thing I would ask you is how much has um uh how much have have things changed as
[24:45] you've now been able to go and uh uh and add uh you know a stronger cap backbone from a real-time standpoint with Confluent. So what I um Confluent has been part of
[24:58] enterprise architecture pattern for years. That's why it's successful. That's why IBM acquired it.
[25:11] Your agents not only external data in ingestion but just data within your network, multiloud, crosscloud.
[25:23] So real-time context engine enables the agent to understand. It's machine readable. So agents actually like it and can traverse it and apply it. And that runtime governance is the same thing. It's it's addressed for a machine, not
[25:36] like a big old catalog software that we would buy and um probably IBM probably has one too um that you would humans would go and curate and kind of label and design things. This is agent um readable policies. So runtime governance
[25:49] and that runtime context engine are really to me some of the most valuable pieces. You can create you know Kafka and Flink streams on your own if you're really handy. I wouldn't do it but that's just me. Um or you can use data
[26:01] bricks uh to build it. Great. That's a great pattern too. So when you think Confluent I would say don't sleep on the real-time context engine and the runtime governance because those are the two big patterns or architectural elements that
[26:13] most people are missing and it is enabled in the context of real time. if you're running Confluence's been running these real-time um flows for a long time and so they understood it ahead of time and they've really built quite a robust capability in there. So it's core to our
[26:26] architectural patterns. Yeah. No, I think that's definitely the case. And I think um what most people end up doing is spending a lot of time um you know very fixated at that top level uh agentic front end but they don't
[26:38] appreciate and it's really cool to see it in action when you when you sort of open up the thinking view in one of these where you see how many um messages for example that it'll show it'll throw into the real-time context engine uh in
[26:52] terms of getting to the right uh right piece of data. Well, thank you Eric. was uh very helpful in bringing the perspective as um uh you know and this is a key part I I think many of you partner with with IBM and IBM consulting
[27:05] and just um we're really excited in terms of of the number of these uh it's safe to say a considerable number of implementations that we're we're involved in on this. So >> yeah, >> you bet.
[27:17] >> Great. Thank you. >> Thank you. So
[27:31] things that um that we brought to bear here, which is putting these systems together operationally requires you to uh to bring these components together to be able to have
[27:44] an enterprisewide uh data connection platform. um that brings these capabilities. You saw us emphasize these points again and again. They're not justing points. They're core requirements for your architecture to
[27:56] succeed. Um we go and look at and there we go. That was the problem. We had the slides uh out of order. Um uh when we go and and look at what uh
[28:11] what that landscape looks like, you want to bring a lot of tools to bear. Uh the great thing is that data architecture now is increasingly open, open standards, open connectivity at the
[28:23] agentic layer through things like MCP uh skills uh the at at your data level at your data tier with zero copy with streaming. Um and with this
[28:36] architecture, what we're able to do is take what you're doing and bridge across the entirety of your data state. And so, um, my takeaway for you should be that as you're thinking about that, as
[28:48] you're, uh, looking at scaling these projects, um, hopefully this has given you, um, quite a bit to think about in terms of bringing real time, uh, real-time context, real-time governance, uh, to the table. Few last things to
[29:01] leave you with. Um, again, we do know what we're talking about and we're not just the ones who say it. as a few of the the the things from uh from Gartner in terms of some of the components we bring to bear. Um and uh and then you
[29:15] know final outcome hopefully uh hopefully we um uh have communicated that being able to get you to faster faster decisions faster better time to value uh and being able to lower the
[29:27] cost of this uh these are the goals for all the projects we deliver and being able to bring that in a real-time way across the data state uh is uh is the outcome we want to help drive. So, uh,
[29:39] thanks everyone. Here's ways you can get started. Um, and, uh, and and please do, uh, thank you everyone for coming out today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.