Sponsored by: IBM | Scale AI Agents from Prototype to Production into Enterprise Systems

Summary

IBM's watsonx Orchestrate serves as an enterprise AI control plane that provides visibility, governance, and financial accountability over the hundreds to thousands of AI agents organizations are deploying across platforms including Databricks, SAP, and Salesforce.
IBM's Agent Development Lifecycle (ADLC) framework addresses the prototype-to-production gap by incorporating security policies, red teaming, compliance enforcement, continuous evaluation, and model upgrade management alongside low-code agent building tools.
A live demo shows a fraud detection agent integrating Databricks and ServiceNow through the watsonx Orchestrate governance layer, illustrating how existing agents from different platforms can be connected and governed without rebuilding them.

Chapters

FAQs

What is the prototype-to-production gap in enterprise AI?

The prototype-to-production gap is the disconnect between how quickly engineers can build AI agents and how cautious senior leaders are about deploying those agents in production without governance, visibility, and security review processes. IBM's experience shows organizations frequently have 700 to 1,200 agents deployed with no centralized way to track, govern, or financially account for them.

What is IBM's Agent Development Lifecycle (ADLC)?

The ADLC is IBM's framework for moving AI agents from creation through production safely, incorporating security policy enforcement, red teaming, compliance checks, continuous evaluation, and model upgrade management. It is designed to give enterprises a structured, repeatable process for shipping AI agents that meet governance and regulatory requirements.

How does IBM watsonx Orchestrate connect to Databricks?

IBM watsonx Orchestrate acts as a control plane that can connect to agents already built on the Databricks Data and AI platform alongside agents from SAP, Salesforce, Azure, and open-source frameworks. This video includes a live demo of a fraud detection agent that integrates the Databricks Data and AI platform and ServiceNow through the watsonx Orchestrate governance layer.

How does IBM help organizations manage token costs and AI ROI in production?

IBM watsonx Orchestrate provides visibility into token costs and latency at the individual agent level, giving leaders financial accountability over AI deployments rather than just a count of agents running. This visibility is designed to help organizations answer whether their AI investments are generating measurable returns and identify agents that are consuming budget without delivering value.

Full transcript

[00:09] Hi everybody. Um, and welcome to the scaling AI agents from prototype into production. Uh, my name is Victor Povar. I'm a Watson X leader for the Americas. I work with clients all across North and South America. And a common question I
[00:22] have, you know, I work with a work with a lot of leaders and they come in and they say, "We have a gap. We have interns. We have young engineers who build amazing prototypes in a week. You
[00:34] know, sometimes they can build things in five minutes, but we have the senior leaders who are terrified of putting those things in production. And we have this gap of it's easy to build and you can build any way you want. The
[00:46] developers can build things with their favorite tools. They use codeex, they use GitHub copilot. We have IBM Bob. It's easy to build things, right? If you are in customer care, there are solutions for customer care. If you're
[01:00] working with your platform providers, SAP, Salesforce, data bricks, everybody has pre-built agents on the platform. And there's also agents for c for personal productivity, right? Like you
[01:13] can use clobot or you can use other tools. So it's easy to build things and we're seeing this in the enterprise. Um there is scaling of agents. I walk sometimes into corporations and they have 700, they have a,000, they have
[01:25] 1,200 agents. But the problem is it's not that it's hard to build and we can also see it by the adoption within the enterprise of 96%. The problem is really this red line is while they have things
[01:39] built, it's they don't have a visibility on it. I was just with a number of clients in Sa Paulo last week and they're like, "Well, we have agents with SAP, we have agents with Service Now, we have agents with data bricks, we have
[01:52] agents built on Azure, we have agents built in open source, we have no idea what we have where, we have no idea how do we govern them. We have no idea how do we tie those things together. And if a regulations come out or if I have to
[02:05] go into a security review, I have no idea how those things tie together. And then if things evolve, it makes it really hard for me. So what I really need is I need to be able to connect all
[02:17] these things together, ensure that they comply with our policies, ensure that there is no vulnerabilities, there's no latency, and ensure that at the end of the day, this makes sense for me
[02:29] financially. I'm still in the business of making money, right? So I'm seeing the tokens go away. I'm seeing developers work on this and provide this but I'm missing the ROI. So while we say
[02:43] it's easy to build agents where IBM and the Watson X platform are focusing is on everything else that's really hard everything else that the enterprise really needs. So connecting to your back-end systems and that also means
[02:55] connecting to your existing agents right like you already have agents build. It's not let's come and let's rebuild things. It's how can I make better advantage of the agents I have and connect them together. When I have gaps, how can I
[03:08] build those gaps quicker? Making sure I have robust evaluation frameworks. Right? Things continue to evolve. We're still very early on the journey of agents. Uh the models are changing every week if not every two weeks. There's new
[03:22] frameworks, there's new protocols. So, how can I make sure that I'm set up for it? And by the way, I need red teaming out of the box. I need to make sure I'm testing for vulnerabilities so I'm not deploying an agent that's going to make
[03:35] the headlines for all the wrong reasons. Right? We need to make sure that we incorporate existing security protocols and policies. Right? As the enterprise is setting up policies, how can we make sure we leverage them out of the box and
[03:47] we're not in a situation where I have some policies against my Service Now agents. I have some other ones in Azure. I have something built in code when I'm building with opensource and I have things scattered all over the place and
[03:59] I need to validate and audit and verify. I can't really do it together. We need to make sure these agents are working together. And then finally, as we're deploying things into production, we need to have a good view of actually
[04:11] what's happening in production and be able to optimize and continuously improve those. Um so we are not in a position where we'll sit down in a research lab and we'll figure out like what's the best
[04:24] way of things and build it. What we said is let's take our biggest clients and let's build it together with them. And as we build this thing together with them we actually published a white paper. There is a QR code to the white
[04:36] paper. Um we work closely with Entropic 2. it's built um quite a bit with MCP is like how do we actually not just quickly build this because we can't build it quickly. If you think about MCP servers,
[04:48] there's about 40,000 MCP servers. Those are growing very rapidly. But how do we secure it and how do we actually make sure that we are able to scale it? So we develop the ADLC. So for those who know
[05:01] the SDLC, the software development life cycle, we develop the agent development life cycle. on actually what do we need um to make sure that we're able to scale build it quickly and then deploy it even
[05:13] faster. And we actually broke it down into two areas. The area on the top, the area in the light blue and the area on the bottom. So the area on the top is actually what you need while you're developing things, while you're building
[05:26] things. And then as you deploy into production, and there is a different set of criterias that you actually need when you're deploying it to production. And more specifically, here are some of the areas what you actually need as you're moving from design to testing and
[05:39] then as you're moving from deployment to monitoring, right? So, you need to make sure that everybody can build in your enterprise because that's the only way you scale to your entire enterprise. So,
[05:51] we have low code no code tools for building for a line of business users. There's canvases that allow you to have determinism. A lot of the banks, a lot of the insurance company, a lot of the companies who are pushing this in
[06:03] production, they're looking at their agents and say Agentic is awesome. That's the future. But today, I need some determinism. You know, if I'm transferring money from one account to the other account, I need to make sure
[06:16] that 100% of the time it follows the same flow and everything works the same way. So, we allow you to have the determinism. Um, a lot of focus is on evaluation, making sure that as you're building your solutions, as you're
[06:29] developing things, you're able to quickly retest things. And we'll show you how that works. And then once we move into the deployment, this is where you need a control plane. Control plane that tells you what's going on within
[06:41] your solution, but a control plane that can also act upon things, right? If things are not working the way they're supposed to, you need to make sure you have a kill switch that can disable an agent that can automatically disable a
[06:53] tool and then guide an administrator of the platform through the through the process of how to reenable those functionalities. So let's take a look. Uh so this is our control plane. Um within the control
[07:06] plane, you can build your agents at the top and kind of quickly move through it. But it tells you if there is any alerts, anything that you need to take a look right now, anything that's just not working properly and needs to be addressed. We can see what's happening
[07:19] with agents which controls um you can see if there is evaluations, if there's failed messages at a glance and it's meant to be interactive. It's meant to be clicked upon and executed. So as agents are failing you can go in and you
[07:32] can address and see where the failures are see the type of failures that are occurring um how the system is being used and then if you do have negative feedback you can drill into them and actually see what that negative feedback
[07:45] that can be somebody actually clicking on a negative feedback or it can be implied people just reasking similar question which is an indication that they didn't get the answer they're looking for. And by the way, who likes looking at traces? Traces are not easy
[07:59] to look at. So they've introduced a easy way to actually see how an Asian is behaving.
[08:17] >> Sorry. So a trace will actually give you the information of all the tools and all the collaborate agents that you have and as a specific conversation is walking through it, it'll show you how that's performing, it'll show you any errors
[08:30] and any collaborators as you move through it. So again, as you're building things so you can easily debug things and find a way how you deploy into production and address again from there, you can then go in and modify things as
[08:43] you need to on the control plane. So for specific if your agents expire or if your tokens expire you can go you can make those changes and you can move forward with your solutions.
[09:01] So the agents that we have within the control plane are not meant to be built on the control like we are able to build agents within the platform but we also able take take agents build in open source and run them on the platform or
[09:13] the most strategic part is actually taking your third party agents agents building platform such as data bricks or service now and bring them on a platform and have everything working together and making sure you have a consistent set of
[09:26] policies that you can deploy across all of those agents. And then as you're testing things, as you're debugging things, you can save everything as a test right in the platform. You can take a conversation that occurred in production and then add
[09:38] that to your evaluation framework. That way, you easily can build your evaluation test set and then you can test it out. When you're testing things, you're not looking that the answer is the same. You're looking that the path,
[09:50] the reasoning that the agent took is in line with your expectations. So, did they call the right agents? Did they were called in the right order? Did they call the right tools? Did they miss any tool calls? Did you extract the right payload from those tools? And did the
[10:04] final answer was in line with the answer that you're expecting? Right? It's not going to be the same every time because you have LMS working on it, but is the path and the reasoning and the thought is the same. Now, as I mentioned, the
[10:16] key to it is having an open ecosystem, right? If you build agents with data bricks, that's great. Let's leverage them. If you build them with copilot, let's leverage them. If you have existing automations, right? If you have existing RPAs, let's let you build and
[10:30] bring that into the ecosystem. U the LLMs, they will change. They're changing every week, maybe every two weeks, right? So, how can we allow you to bring everything into it? Um the goal is to
[10:42] have an open ecosystem where your UI is decomposed from the control plane. The control plane knows who the agent who who is actually interacting with a specific agent collecting to existing tools on the platform or brought from
[10:55] externally on the platform and having everything brought together and giving you full visibility of what's happening behind the system. And in addition to that what we did is we recently uh concluded the acquisition
[11:07] of Confluent. Um Confluent booth is just over there. So Confluent does the data streaming right? So regardless of where your data actually resides is how can we stream that through the platform and then trigger agents here and then how
[11:21] can we build that quickly. So IBM Bob is IBM's coding assistant similar to clock code GitHub copilot. What we do is is we automatically pick the best model based on the question that you're asking right
[11:34] you're not going to be picking and choosing large language models versus small language models but actually applying that and how can we move it forward. So here we have an example of a credit card of a bank actually leveraging the solution. So they have
[11:47] credit card transactions flowing through the system and then they need to identify if there is fraud in a specific credit card transaction. So the left side is things in confluent the right things are things built on the platform.
[12:00] Everything is built by IBM Bob and we can in real time detect fraud and then act upon it. So here we have a simulation of credit card transactions moving through the system. We are automatically identifying
[12:12] the fraud on those systems and then when a specific fraud is detected, we're opening a service now incident to determine that there is fraud on a specific transaction, we are putting a
[12:24] block on that credit card transaction and then we are notifying the user through both an email and SMS. Now if you think of what's possible within other industry, so you have a lot of IoT sensors within the healthcare industry.
[12:37] We have a lot of clients leveraging this in retail. So think of if you have um ordering of groceries and then somebody goes a picker tries to get those groceries for you and an item is not
[12:50] available you can get a notification to determine how to substitute that item. So again it's taking your data flowing it through the system and then having agents and again the power of those agents is they can be built anywhere
[13:02] they can be your uh agents big built with a data brick system they can be built on Azure they can be built in open source actually having comprehensive solutions that address your requirements.
[13:19] Uh so this is the um the specific agent the way it looks that was built on the previous screen and then finally is we're leveraging IBM Bob to actually build the entire solution. So everything that you saw today was built by IBM Bob u in this specific demo is we're taking
[13:33] the previous solution um we're first building the agent. So we're going to ask Bob to actually go and build the agent for us. uh it's going to take the specification and it's going to go and build that and once it's built that it's
[13:47] going to move on to building the confluent piece. Um so as we're moving here to wrap up it's really the goal is to move forward on leveraging the ROI and improving the ROI
[13:59] for that session. If you're going to be keep rebuilding it in different platform you're not able to move forward and scale on that session. The goal is how can you leverage what you already built? whether it's agents, whether it's
[14:12] existing automation, whether it's existing RPAs, bring it together, give you visibility over it from the moment that you build the use case to the moment you deployed in production with the controls. Um, and tie it in all
[14:25] together. Right? So, here I think we're building the agent. Uh, we're going to go we're going to deploy that agent and then scale it out.
[14:56] Um, so I think I think we're going to wrap up the session here. I think we're going to be available for Q&A. Um, thank you for joining. Um, thanks to you for coming over here. And again, if you're looking to really move forward from building things to really getting an ROI
[15:09] and bringing your whole ecosystem into one place that's governed and centered, would love to have to continue the conversation with you. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.