Sponsored By: Monte Carlo | Building the Foundation for Trusted AI in Regulated Industries

Summary

In regulated industries like financial services, AI deployment failures are not just technical—they break compliance, erode model trust, and have serious career and business consequences, making the bar for production-ready AI fundamentally higher.
Nasdaq built a Databricks-native AI stack that ingests data once across business units and uses it to power use cases including a research agent for entity mapping and compliance, choosing Databricks for its governance capabilities.
This video covers AI observability, data lineage, governance frameworks designed for regulators, the role of governance committees in evaluating AI vendors, and practical lessons from what Barr Moses and Lenny Rosenfeld would do differently.

Chapters

FAQs

Why is AI deployment harder in regulated industries like financial services?

In regulated industries, data quality failures don't just break pipelines—they break compliance, erode trust in models, and can end careers. This video explains that the bar for 'production-ready' AI is fundamentally different when data touches trades, audits, or personally identifiable information, requiring stricter governance and observability than typical enterprise AI projects.

What AI use cases has Nasdaq built on Databricks?

Nasdaq built a research agent for entity mapping and compliance tasks, along with a broader data platform that ingests data once and makes it available across business units. According to this video, Nasdaq chose a Databricks-native AI stack specifically for its governance capabilities and the ability to maintain lineage and auditability at scale.

How does Nasdaq evaluate AI vendors and models for regulated use?

Nasdaq established a governance committee responsible for evaluating AI vendors and models before deployment. This video describes the committee's role in assessing risk, compliance fit, and reliability, reflecting a deliberate, structured approach to AI adoption rather than moving fast and dealing with issues after deployment.

What is Monte Carlo's role in trusted AI for financial services?

Monte Carlo provides AI observability and data quality monitoring that helps enterprises like Nasdaq detect and resolve data issues before they affect AI models or compliance workflows. This video positions Monte Carlo as a partner for enterprises that need self-improving, self-healing data pipelines as the foundation for trusted AI.

Full transcript

[00:07] So Lenny and I, this isn't the first time that we speak and Lenny and I have already started a tradition where we start all of our talks with a audience selfie. So I hope you guys will indulge us and join us in this selfie here if that's okay. >> Make a silly face or >> Exactly. Make it look like yeah, exactly. Say data and
[00:20] AI. >> Data. >> Awesome. >> Thank you. >> Can't break the tradition. Hi everyone. Thank you so much for joining us. My name is Barr Moses. I'm
[00:32] the CEO and co-founder of Monte Carlo. Monte Carlo has a pleasure of partnering with over 500 enterprises like Lenny at Nasdaq to help deliver trusted AI. And we know that at the heart of AI is
[00:44] really good trusted data. And I think all of that just becomes a lot more complicated in a regulated industry. We'll talk about some self-improving, self-healing agents. We'll talk about specific use cases.
[00:56] We'll talk about what's the role of humans. How do we handle risk and compliance teams? All of that. We have a very sort of meaty, meaty list of topics for today. But maybe just to get started, um
[01:09] we'll ask Lenny, who's VP of data science and software engineering at Nasdaq, to give a brief introduction of yourself and maybe a little bit about, you know, what does data mean at Nasdaq? Data and AI. Don't I think Nasdaq is sort of a, you know, brand that everyone
[01:22] knows maybe most recently with the IPO of SpaceX actually last week if anyone was tracking that. That was >> Largest IPO ever. So thank you to everybody at the company that made that happen. >> There you go.
[01:34] So Lenny, maybe just introduce yourself a little bit. >> Sure. So Lenny Rosenfeld, I'm based in Boston. I work for Nasdaq. I've been there 11 years in various roles through there, through the groups. And really at Nasdaq, we are
[01:46] we're known for the stock exchange, but we're much more than a stock exchange. At the heart we're really a technology company. Everything from, you know, running, you know, 100% always on servers, doing cloud, doing AI. But at the heart
[01:58] we're really a data company. Data powers everything we do just in terms of getting ready for the SpaceX IPO, we had many many war rooms and war games and tabletop exercises, but it was all based on data. It was all based on facts and how we might scale our systems and how
[02:11] we might prepare for certain events. And so at the at the heart of that that really led us to be one of the most, you know, resilient, one of the most trusted players in in financial services and in software as well. So, been at Nasdaq for 11 years,
[02:24] started out in one business unit that merged with another, etc. The previous role I had was actually the CTO of the data business. So, quick show of hands, who's ever bought or sold a stock here? So, thank you. You or your brokerage is
[02:36] my customer. So, I thank you. >>
[02:48] doing is we've traditionally grown through acquisition. And there's a lot of duplicative things that we've done. Ingress of data, land data, clean data, build AI agents, or build AI and machine learning. We realized we're doing that three, four,
[03:00] five times. In fact, there were business units that were actually ingressing the exact same data set over and over again, paying multiple fees, and if there was an incident, both business units were responsible for that. And so,
[03:12] we started to to really um find these patterns and look for ways of doing it once. And so, the data pattern, the data platform today is really about ingressing data once, having these capabilities for these various business
[03:25] units to transform, normalize, clean their data, and then they can service their customer. They can they can actually handle the last mile transactions much better. And so, that's a big focus for us today. And a big big part of that is AI as well, right? And
[03:37] that's everything from how do we build agents to shorten the time to value of delivering data to our customers, improving the data quality, improving system observability, and then data observability. And then on top of all that, productivity. You know,
[03:49] productivity for to software engineers that are building capabilities, and then productivity for our data engineers that are building these data products and servicing them. So, a little bit about us. >> Yeah, a lot to talk about.
[04:01] A very meaty. Let's get started with maybe sort of a provocative question. We'll we'll we'll go right there. >> Sure. Sure. >> Um, you know, you sort of sit at the intersection of data science, software engineering, and
[04:14] one of the most regulated environments on the planet. And when you know, when I I I get to speak with many
[04:32] And is that fear rational? >> Uh, God, people have fear fear fear for a lot of things. I think a lot of things are are maybe related to maybe job displacement could be one, right? Not yet proven how much if it will. I know there's a lot of sort of fear right now about, you know, entry-level jobs
[04:45] being wiped out, which, you know, who's going to work the next generation? So, that's one. I think the other one is trust, right? And that's really based on, you know, who's who's who has the ability to make decisions, right? I
[04:57] think that humans are still accountable for decisions. That's number one, at least at Nasdaq, right? So, for instance, um, AI agents do not deploy software to production for us. We allow agents to build software. We allow agents to do some of the mundane tasks, write
[05:09] documentation, do testing, but humans at the end of the day are still responsible for verifying and deploying software. So, it's really where where's that trust placed? I think the other thing that's that that strikes a lot of fear is along
[05:21] responsibility, but it's explainability and understandability, right? Is do I really at a deep level understand what the agents are doing, right? And that really comes from like, are the outcomes clear, right? Are the outcomes clear for humans? Are the outcomes clear for for
[05:33] agents? And then, explainability, right? Is can I explain to regulators or can I even explain to members of my own staff or infosec or compliance? And do they understand, right, the
[05:45] outcomes that these agents took or the actions that these agents took. So, So, I think the fear is is it's real. Right. But, you need to put guardrails in place. Um that's that's the big thing and we're doing that now um to help every function, right? Whether it's
[05:57] you know, software engineers that are using uh AI to write code or again, data engineers uh when they're working with complex business logic, right? To be able to uh understand the outcomes, to be able to validate, you know, did the agents do
[06:10] what we needed them to do? Um so, I think the fear the fear is real and justified. >> Yeah. So, let's break that answer down into a couple things. You mentioned at the beginning, you know, when do we automate away and when do we actually
[06:22] have humans in the loop? And I think that's a big question. Like, what is the role of AI? What's the role of humans? When it comes to financial specific decisions, what's the decision criteria? Where do you draw the line? You said software doesn't get pushed or does it
[06:34] get written when there's a set of like, what's the set of criteria to make that decision? >> Yeah, I think the easiest one is is the workflow or is the decision deterministic or not. Right. And traditionally, even before AI, if if the
[06:46] workflow is well-defined, well-understood, and it is a deterministic it executes the same way every single time, that's a good good candidate for AI. It's a good candidate to to try to automate, you know, and make sure that even using some of these
[06:58] non-deterministic methods get deterministic outcomes. Um where we uh apply humans and and, you know, it's not really about removing humans in the loop. It's removing those mundane, wrote, repetitive tasks from
[07:10] humans so that we can deploy them on on higher value. Um so, like I'll give you for instance, in the data engineering space, um many of the data products that we onboard, we we license, you know, hundreds of data products if you
[07:22] know, thousands of data products. Um many of them are really just about bringing them into our environment and very simple level of normalization, right? Maybe just standardizing some of the column names. Um maybe just doing some light validation. Like, the vendor
[07:34] says there's 10 years of data, is there really 10 years of data, right? And so, those are are great candidates for us to deploy AI against um because it's fairly low risk and a human can validate it. Um, if there's really complex business logic, like if we're building out an
[07:46] alternative data product, uh, for instance, looking at global credit card transactions and trying to understand how they may move uh, publicly traded companies based on sales, you know, we may use AI to to build those pipelines, um, but at the end of
[07:59] the day that that business logic and that outcome is validated by experts. It's validated by the data scientists and quantitative finance professionals um, to make sure those alpha those beta generating products really do say what they are. Um, because at the end of the day that that is a reputation. Um, that
[08:12] is what we're selling. Um, so that's how one of the ways we've differentiated our decided where we can have more uh, more agentic flow or or maybe more human loop. >> Makes sense. And as you're building this, how involved are your legal, risk,
[08:25] and compliance team? How do you have this discussion with them? What do you actually show them to say, "I would like for this to be humanified and this should be agentified." Um, what does that actually look like? >> Yeah. Yeah, we we have a fairly rigorous
[08:37] process you would imagine. Um, so if you know, kind of rewind 18 months ago, many companies when LLMs really blew up and everybody wanted to use them, maybe 2 years ago now, um, there's either kind of the frontier or the wild west YOLO, like, you're a startup, do whatever,
[08:50] just move forward. Um, but many of the regulated companies were like, "No. Don't use it. It's shut off. You know, you can't install it on your desktop." Whatever the case is. Uh, we took a different approach. Uh, we have a governance committee, right? So, every use case is first put forth through the
[09:02] governance committee and that's evaluating who the vendors are, what the models are, um, what what the classification of the data is, um, how do you validate it, how do you prove that, you know, um, it's going to do what it is, how do you test it, what are the guardrails. Um, and so through this
[09:15] process and that process started a little bit a little bit slow, but it's it it's gotten much much faster. Um, and that includes infosec, uh, governance, um, and then, you know, kind of management sign off as well. Um,
[09:27] um, and so that's been it's been a good partner, um, in terms of submitting these use cases and then running them through this process to get evaluation and then approval. Um, and then basically what the way the process works is we're approving it for a POC and based on the strength of POC, it goes
[09:40] into a private preview or public preview and then again a final round of approval for for GA. And that's really GA would be to deploy those the outcomes to AI-based outcomes to to
[09:52] the public. And so that has a much higher level of rigor and questions. But it's worked really well. It's it's forced people to ask really hard questions including things like an ROI, right? It's not just hey, let's burn as many tokens as possible, but does this even make sense from a business outcome?
[10:04] Forget the fact that we're using AI. And so that's how people really ask those hard questions to to position the the product in a meaningful way. >> What are some if you don't mind sharing, what are some of the the business outcomes that have been the most
[10:16] successful for AI in particular? And then also, what do you think about token maxing? You're already on the topic. >> Um token maxing. Um some of the I think some of the most successful have been. So we look at AI
[10:28] in two ways, in the product and on the business. On the business is really productivity gains and that's across every single function. Has the has the ability to use tools to to gain productivity gains. I'd say that's been some of the some of the
[10:40] most successful ones just in terms of uplift. >> more efficient. >> Making folks more efficient, making folks more focused for instance. So we all have inboxes that are mile deep, you know, but at the beginning of the day or the end of the day asking what should I focus in on, right? Based on the context
[10:53] of who you are. Um but then we we are starting to release in the product capabilities as well. You know, so one of the first ones that we released and it's not gen AI-based is is dynamic mellow which is a dynamic
[11:06] order type. It was the first AI-powered dynamic order type that we released last year maybe a little bit longer ago. So we are starting to see some of the in the product releases go out. Honestly, it's still tough. It's still
[11:19] tough to to find those million, 10 million, 100 million use cases especially with the maybe shift gears a little bit with the usage-based billing model that we're now seeing coming about
[11:31] you know, with token usage. It's harder to even justify some of these products because yes, you may be able to charge more, but your underlying costs, your cost basis just went way up significantly. So, we are looking at that quite seriously. Then in terms of
[11:43] token maxing, you know, it's one of those things when you everybody heard of one of the hyperscalers having a leaderboard and you know, they they optimize for people appearing on leaderboard and people appear on leaderboard. So, you just have to be careful what you optimize for and what you're incentivizing people to do.
[11:57] You know, we did have we did incentivize it on the engineering side and maybe we'll talk about this a little bit in other topic, things like agent week. Right? We said, great, you know, so go and use agents to try to automate any
[12:09] part of your job. It doesn't matter what it is, but just go and try it. So, it could be everything from writing documentation to writing code to full end-to-end, you know, Ralph Wiggum loops, if you will. So, so, you know, while we're not
[12:21] pushing people to just burn tokens, we are really trying to get people to get comfortable with the technology and figuring out optimal ways of using it. >> So, there's no leaderboard. >> No, there's no leaderboard. No, no, no. >> Just confirming. Um, maybe we can talk
[12:33] about a real authentic system, like actually a real use case that you recently released. And I think it's really interesting. Maybe walk us through like what exactly it is, what's sort of the engineering version, either something that you
[12:45] released more more recently or sort of something from Data Lake Data Lake Co-pilot earlier. >> Yeah, yeah, maybe I'll talk about one that we're actually working on right now. It's in It's an MVP. It'll go It's an internal system, but it impacts quite
[12:57] a few people. So, as I mentioned, we have a a data business and a data platform. So, one of the things we do is we we map and match data. Right? So, we can core data from multiple sources. In some cases, we already have like a master hierarchy, you know, this firm and this data
[13:09] product or this table equals this firm and this one, so on and so forth, right? It It It's a fairly messy process. It's a fairly human intensive process. Even with the best of LLMs, you still need humans, right? Human adjudication. And
[13:22] so, before, this process could take an analyst a content curator essentially minutes or hours on every case, right? Just depending on the complexity, if they've seen it before, if they haven't seen it before. With maybe more traditional deterministic
[13:35] methods like regex, etc. And so, what we've built is a research agent to facilitate that. So, you know, maybe a new data set will come in or updates to data set will come in. We have agents that are mapping the data
[13:49] and then an agent that is actually going off and doing the research on behalf of the end user, right? It's looking at different data sources. It's either looking at data sources, public internet like a grounded search, maybe a database like a master database,
[14:01] etc. And so, right now, while we're in GA, every single mapped, matched entity or entity resolution, a human adjudicator will actually verify, right? So, we've
[14:13] got screens and workflows all built on Databricks to then go in and say, "Yes, I agree with your map." or, "No, I don't agree with your map and here's why. Here's how to fix it." And so, the goal there is to move from, you know, a process that takes one human, and these are
[14:25] thousands, if not tens of thousands, hopefully hundreds of thousands of entities, you know, minutes or hours to really seconds, right? And maybe not at all. Maybe that there's a confidence threshold
[14:37] that the agent can sufficiently meet and and have the the validity, have the rationale, the reasoning, the explainability of why it just passes through and no human is involved. But, today we've got humans reviewing every map,
[14:49] but, you know, basically above a a certain confidence threshold. That's a real use case that's, you know, going to be saving hundreds of thousands of hours in aggregate and it's also driving the,
[15:01] let's say, the the quality of the map and match data itself because we're able to introduce um as many data sources and data stores that are necessary in order to do the mapping and matching
[15:13] and the entity resolution, right? So, you can imagine for humans they can work with a limited number of data sources and every new data source you introduce is an extra step or extra cognitive load. Not quite the same for for for an agent, right? Yes, there's cost involved. Yes,
[15:26] there's time involved. Um, but in the end it's net savings. Um, and then it's really deploying humans. It's not reducing the humans. It's deploying the humans against really hard problems, right? Where we've never seen the data. Right? Or if we want to have a gold source that data, having a human expert,
[15:39] content curator, and many of these folks have spent years in a complex industry um, that really true have deep domain understanding, make those proper selections. To say yes, no, I agree with this match. This is this is true. Um, so
[15:51] not only are we driving quality through humans out human outcomes, um, but we're also driving efficiency um, through just just deploying the agents at scale. >> Yeah. I love that example. Um, and maybe just shifting a little bit to the technical stack behind that. I think you
[16:03] guys are building on Bedrock, PG Vector, Titan Embeddings, LangChain, several others. >> It Surprisingly enough, it's all changed. >> All changed? >> Yeah. Well, so when we when we started the journey, which
[16:15] I think when we started the journey, my memory's a little fuzzy. We were at the end of last year, beginning of this year. Um, and extremely proud of the team. They built it all on Databricks. So, starting from the application layer, um, they built uh, using the Databricks
[16:27] apps. There's a adjudication UI. And it's a simple UI right now. It's going to get more complicated. Um, and so we have a UI that's a form-based or workflow interface. Um, below that we use um, AI Gateway, Mosaic AI Gateway, uh, to do
[16:39] some of the metering, the monitoring. We use MLflow for experimentation. Uh, and we do use LangChain and and LangGraph uh, for for the agent framework. Um, but that's under evaluation right now, where we might go. >> Databricks? >> I mean, not just Databricks, but what
[16:51] Databricks announced um, that we'll see tomorrow in in deeper uh, what they've announced um, in terms of their their meta meta framework as well. Um, of course we use Unity Catalog for governance. Um, and we use uh, Lakehouse as well for some of the the caching and
[17:03] uh, maybe the higher throughput queries. Um, so it's really a purely Databricks native stack. Of course, Monte Carlo uh, for monitoring as well. Um, and then Datadog for system observability um, at the bottom. So, um, yeah, so just in a short of like, you know, three plus
[17:16] months, the stack has already changed. >> Yeah. >> So, it's a little bit scary I'm just continuing chasing the chasing the dragon there a little bit to how how quickly this thing is going to change and do we need to swap out the stack again in three months? Hopefully not. >> Exactly. That was going to be my next
[17:28] question. Like what how do you cuz I think a lot of folks think about the heterogeneity of their environment and sort of not having all of their eggs in one basket. How do you you know, who knows what will happen in three months? How do you sort of are you worried about
[17:40] vendor lock-in or not at all or >> A little bit, right? I mean we we we we we we play strategic bets with vendors and and we think those vendors are responding. I think the most important thing is the data. Right? And really treating data as a product and AI ready
[17:53] data, right? So, is it governable? Is it discoverable? Is it queryable and accessible? That's really key. Um and then on the data as a product side is robust documentation, right? Not just, you know, column descriptions, but really use case. Like what is this data?
[18:06] What is it for? What does it unlock? The deep understanding of the meaning of the data is really important. Um as well as all the metadata behind it. Extremely curious about the semantic or the ontology layer. Um another level of unlock in the deep relationships within
[18:19] the data itself. Um so, that's one way to mitigate vendor lock-in. Um and then of course we've heard with Databricks the rich heritage of open source technologies. Um well, we are using quite a bit of Databricks
[18:31] capabilities, Monte Carlo capabilities. I think they're they're they're safe bets for the long term. >> Okay. >> So, and again as far as the model goes, I mean the models are going to continue to to to iterate and they're going to continue to improve rapidly. Um so, we've got the ability to swap out
[18:44] different models as well. Uh right now we're using the Anthropic family of models, but we evaluated uh the OpenAI family of models as well. Um but really what's important there is the harness. Right? It's that context, the tooling, skills, and and um
[18:57] instructions and all these other uh markdown files today, uh this glue that allows you to divorce the the the LLMs which are going to change those frontier models with your core proprietary logic and what what's
[19:09] meaningful for your company, right? That's the core IP, right, is is the harness. >> Yeah, makes sense. Um, and how are you thinking one of the most important things that I think, you know, first it seems like there's this rush to move from pilot to production,
[19:22] like make sure that you have, you know, this real good use case, like you're saying, you just launched it or launching it, congrats, that itself is a big milestone. The next thing that we see folks ask themselves is like, how do I instrument this thing? Like, how do I make sure that it's working properly?
[19:35] How do I monitor it? How do I observe it? What are some of the challenge like, is it, you know, just the same as sort of engineering and data observability? Is it totally different? What are some of the challenges here? >> Yeah, I think AI doesn't fail spectacularly like a server blowing up
[19:49] or, you know, piece of code that just, you know, stops working because of a change. It fails silently, which is really dangerous. Right, you might not even know it failed or it's just lying to you and it just it looks really right and so we've all had this when you've asked Open AI or a chat GPT or or or Claude,
[20:03] you know, some some question and it turns out like, yeah, that that seems right. I I but you know it's false. So, it the the failure, the silent failures are the most dangerous part. Um, and so I'll come back to the data link so data link is our our data marketplace, our data uh, delivery
[20:16] um, capabilities. Um, and we we're building right now a co-pilot for it, our chat interface. You can ask documentation, help you write code, um, help you look for, you know, use case based semantic search. Um, we started with a list of golden
[20:28] records, right, 100 questions, 200 questions. And just the way we validated it it it you know, it wasn't enough. Like, you know, we we we like we started 50 golden questions, right, cool questions and answers and we validated the responses. That wasn't enough. Then it was 100,
[20:40] then it was more. So, um, it was like just a never-ending list um, of of questions. So, really we we went back and said, okay, what can we do there? And it became guardrails. Like, here's the things that I want you to not do or do. Right, so it became as much
[20:53] the the the the golden questions and answers as kind of the the instructions for how to how to respond, how to behave as well. Um, so that was that was a big learning curve. Um, and that that really did take some time to to fine-tune and then
[21:05] um, you know, that was all internal, so we had a friendly audience that was giving us uh feedback on this this question and answer service. Uh but now we're we're starting to to look at what is it going to take to deploy publicly um to our to our uh
[21:18] paid subscribers. And that that's a whole different answer. Right? Because these are customers that are coming with use cases and they expect a certain level of trust um and response um which they answer by client success and service teams um but we can't scale
[21:30] right with with the number of customers we have. Um so they're they're relying on that trust trust and responses um that we have to provide. >> And I think you know, some of the kind of next frontier if you will in this area is sort of um self-healing or
[21:43] self-improving agents. I feel like that's sort of all the rage these days. Um what does that actually mean? Is that sort of a you know, agent that fails quickly and an agent by the way, side note, my favorite is when AI fails confidently,
[21:56] you know, when it sort of argues with you, "No, I'm right." And you're like, "No, I know this you know, data wouldn't do that." Um but uh you know, what when you think about self-improving agents, like is that you know, what does that actually look like?
[22:09] Is that just a faster human? Are we there? What's what are your thoughts? >> Yeah, and to to be fair, we haven't built yet self-healing true self-healing where you know, just runs autonomously forever. I I think right now self-healing really means better
[22:22] alerting or faster alerting. Right? So, that that is like let's say data pipeline fails, right? To get to a root cause analysis, right? Was it upstream? Was it the pipeline? Was it a change? Um that's starting to happen faster. Um looking for the different ways of And we
[22:34] saw this today with with um uh uh no ops of the zero ops capability um which is okay, it actually can issue an MR. But in the demo that we saw today, you know, it was a human that was prompting it to issue an MR. So, you
[22:46] know, even what Databricks is showing isn't quite yet fully autonomous. Um so, I'm not sure there yet, right? Like what does it really mean for a pipe pile pipeline to fail? Um you know, in in in many cases uh it's the vendor, right? The vendor is sending us bad data for
[22:58] some reason or the vendor isn't sending us data. So, now how do we interact with this third party? Right? How does an agent on behalf of of us interact with a third party, which is extremely difficult. We spent quite a bit of time there when something fails
[23:10] upstream. And at the end of the day, we we do want a human to review it, right? And so when when even if we can get to root cause analysis to a fix to validated pipeline running in UAT, let's say,
[23:22] we're still going to have a human review that and say, "Yes, I'm ready to deploy that now." So so we we don't have an answer for it, but I'm curious actually when you're building software right now, right? You're doing quite a bit of of agent in loop methodologies. I'm curious
[23:34] how are you solving this problem? >> Yeah, I mean in our sort of um early kind of experiments and and internal use cases, we are seeing agents' ability to um interact with a
[23:46] human being who will tell the agent that they got something wrong, and then the agent kind of takes it from there. Identifies what the root cause is, troubleshoots it by calling different agents. So, calling an agent that troubleshoots it, calling an agent that
[23:59] helps create a PR, trouble calling another agent that actually helps execute the code, and then reports back to the human that it's solved. That loop, the first time that I saw that, it was it was pretty magical. I was like,
[24:11] that is actually dramatically better than I thought possible. It's not complete you know, it's not that the human is not in the loop at all because it's through the interaction with the human that the agent can identify that something is wrong. I haven't seen an
[24:23] agent tell an agent, you know, unless it's um actually something that's deterministic or quantitative like for example, the latency is way too low or you know, the token count is sort of dramatically out
[24:36] of bounds, for example. But, you know, if it's a sort of a semantic question or sort of a little bit of a kind of more of a nuanced something here feels off, then you often times needs a human to alert on that. But, the full loop of
[24:48] identify, triage, and fix actually can be done with agents and and it's pretty remarkable. So, I'm I'm bullish on that on that future. Yeah, you know, I think as I think it's really at the end of the day it's an orchestration problem.
[25:01] You know, just just like back in the software days, you know, having a monolith isn't going to solve it. And so, having one agent that has all the context, it's not going to be able to solve it, right? And so, it's going to boil down to a to a two things. One is orchestration, right? What is What is intelligent enough to dispatch? Um you
[25:15] know, go look and see if there's a there's an underlying Amazon issue, right? So, that's a Databricks or sorry, that's a a Datadog MCP call that we make, right? Okay, that's fine. Now, like, okay, let's go talk about Databricks. Is Is Databricks out of resources, right? Or
[25:27] is the pipeline failing there? That's an MCP for Databricks, right? And then that's Okay, and it's a data issue, let's go talk to Monte Carlo and see what's happening there. Um so, we've even just in our way own we're we're al- already using you know, a fleet of of agents or smaller agents
[25:40] that can do that. And then orchestrating that. Um and then what's really important is the context. Right? So, you're going to have context about your own company, what's appropriate there. Um you're going to have you know, context about the group and the product itself. Um and then
[25:52] what's you know, what's um important to um uh to the project itself or maybe the story or or whatever actually caused that that issue. So, or or driving that change. Um and so, if you're able to store that and you're able to make it
[26:04] accessible to these different agents to to uh and then the orchestration layer that can provide the the right context at the right time and they give a higher probability um of of doing of doing that that true autonomous uh deployment development.
[26:16] >> Awesome. And you mentioned MCP earlier. I get a lot of questions from enterprises where actually early and still trying to figure out what's MCP, what do we do with it? Um you know, how do I convince the AI governance
[26:28] committee to let me get past that? Um how is Nasdaq implemented MCP? What's sort of the, you know, easiest kind of ROI or use case that you would go for first? >> Yeah, yeah, it's a great question. And we had our we had our own confusion like
[26:41] I mentioned the the governance committee and they have to approve a remodel and work you know, work workload. Um the biggest question is MCP AI itself, right? Cuz MCP at the end of the day it's JSON. Like it it's JSON instructions files and and it's a server. Like, it's software. It's a
[26:54] deterministic software. So, even in our own teams, we had a lot of confusion. It's like, does this warrant a you know, committee review? Does this not warrant a committee review? But, we like to look at it as you know, we've we've had APIs
[27:06] and protocols and interfaces since the genesis of the company. We're truly a digital company from our beginning. And so, really it was how do we provide interfaces for agents specifically? Because we've got products with REST APIs with robust documentation, SDKs,
[27:19] the whole works. And and agents can work with it. They can you know, read through it. But, this is really around reusability and making things much easier for agents, right? And so, if you think about it, MCP and the tools that you provide, it's another
[27:32] interface. Right? It's sort of just like you know, you if you think about a REST API, that interface is an experience on your product. And MCP has to be thought about as an experience for your agent and treated as such. Right? Just like you would build a screen, right? Or just
[27:45] like you would build a data product with a certain schema. It's another experience that you have to think about. Um So, that's what we've we've started to think about um getting agents better access to our data. Um >> Yeah, makes sense. Soon, there will be
[27:58] more agents than humans accessing this data. I think that's sort of the >> Yeah. Exactly. >> Yeah, the punchline from today, your agents are writing code on Lake Basin than humans. >> Exactly. Exactly. Um so, I want to turn to
[28:10] uh audience questions shortly. But, just before that, one other topic that I think is important and actually not often talked about in in conferences is the organizational reality of making this work. Often times, the hardest part
[28:24] of deploying trusted AI is actually the people and the structure around that and less the technology. Um let's start by talking about the skills gap in particular. Is this something that I hear a lot from folks? You know, we have very eager ML engineers, data engineers,
[28:37] AI engineers. Um but, I don't really have anywhere to send them to get skilled. It's a lot of work to create training and education. AI moves at a at a at a pace that's much faster than we
[28:49] can to actually provide this kind of education. You know, when you look at sort of you know, sort of Nasdaq and you think about sort of AI, like who owns, you know, making sure that folks are upskilled? Who owns, you know, a lot of
[29:02] these parts around sort of the trust of of what folks are building? >> Yeah. Yeah, it's respon- everybody's responsible for it. And so, we take education really seriously. Um one of the things that one of our goals is to have everybody educated um
[29:14] in AI this this year. We've put together a program. Um and we started with engineers because they were the ones that were most receptive um to using the new tools, to experimenting. But now we've started to go I'll call upmarket, right? So, really starting to educate the product teams,
[29:26] right? How to how to have the builder mentality, how to use the tools. Um we started to educate the executives as well. Like, how do you pay for these tools? What's the ROI? What's the risk? Um and so, training is is um it's really a fundamental element for
[29:39] us. It's non-negotiable from there. Um and then so, the next thing is, okay, a thousand flowers blooming, right? We probably all experienced this last year or maybe the beginning of this year is give people the capabilities, give people the tools to experiment, and encourage them to experiment. You know,
[29:51] find out who your your your um your power users are, right? Promote them, promote the use cases, have them share what what's working for them. Also, identify the people that are that are skeptics, right? And really kind of dig in, not penalize them, but really
[30:03] dig in. What what what don't they like about AI, right? What do they fear about AI? And one engineer on my team is like, "I actually took this job and I remained in this job for 20 years because I like writing code. That is what I like to do.
[30:15] Um and so, okay, you know, we have we have an initiative to uh use agents to write more code. Um a certain percentage of our code to be written by agents is a goal of ours this year. Um and he he kind of rejected that. He's like, "No,
[30:27] you're taking away the best part of my job." I'm like, "Okay, well, what what's the part of your job that you hate?" Right? He's like, "Oh, I hate writing tests or documentation or all the other things that engineers don't like doing, responding to emails, meetings." I'm like, "Well, great. Let's use agents there." Right? It doesn't have to be for
[30:40] everything. It's not, you know, one size fits all. So, um so, it's working with kind of an A series and and trying to understand um that um and then and then now moving kind of up the stack of maturity, it's guardrails. Right? It's how do we provide the safety
[30:52] mechanisms, uh whether it's the AI committee, whether it's we're open-sourcing a lot of skills and a lot of the guardrails internally, uh MCP servers, um to facilitate that interaction, right? And so people aren't redoing, they're not reinventing the wheel every time.
[31:04] Right? So, for instance, we have a code review agent, right? That a a large part of our our business unit is using. We started off with many code review agents cuz that was the first thing everybody tried to tackle, uh but we sort of synthesized it down into one. Um and then we have a central team of GenAI uh
[31:17] team that builds um capabilities for us, like the code review agent is a good one. Um so we relied on them to build that code review that then can be uh scaled to to broader use. Um so there isn't one way that we're doing it. Um but uh but at the end of the day, you
[31:30] know, hate to say it, AI isn't AI is a tool. Right? It It's an amazing tool, but it's a tool. Um AI really to me is transformation and culture. Right? It's change management, right? It's how do you change an entire
[31:42] organization to think differently um and to use these tools effectively. Um cuz there'll be more tools, right? This AI is I feel we're really in the beginning of of this massive transformation in tooling and automation. Um and it'll continue to evolve and um the pace of
[31:55] change will improve, but the culture, right? The culture is is slow-moving, and so how do you work on it um changing the culture to to be able to use these tools effectively? >> My bet is a lot of folks in the audience are sort of hands-on folks. And
[32:08] for those, you know, who might not have a top-down initiative or might not have sort of a code review agent available for them, like what advice do you have for folks who want to start building with AI, want to deliver AI that's
[32:20] reliable and trustworthy, um but maybe don't have sort of that top-down support? Um what what what advice do you have for those folks? >> Yeah, I think partner with your organization. Um I you know, don't go rogue, don't just swipe your credit card
[32:32] and buy Open AI and start using it uh your desktop. That's not advice I would give. Um so one thing is obviously you could do this at home, right? I think the price point of Claude $20 a month. Um I don't know what Open AI charges, but it's fairly reasonable. Right? So,
[32:44] that's that's one thing is just just start experimenting with it at your home. Um and don't use it as Google search, right? Just use it as an extension of like an assistant, right? So, if you had an assistant, if you had an an intern, um what would you ask them to do? Right? So, start there like
[32:56] automating some tasks um and getting comfortable with it. Cuz I think once you see the power of of of the the comfort um and the the the capabilities it provides, you're going to start to think about, okay, internally, how might I use this? Right? Um and I'm sure every
[33:09] organization here has rolled out um some form of of co-pilot or some form of agentic tools. Um I'd say experiment if you can, right? If you can safely experiment with your data, do it. Right? Take the 10% project and experiment with
[33:21] it. Um talk to your peers, see what they're doing. Um look for repetitive tasks, I think is an is kind of low-hanging fruit. See what we're doing day in and day out um that you may be able to start in on. Um and so, in one way is if you're not having the organization push down, if you're not
[33:34] having the top-down mandate, it has to be a little bit about the thousand flowers blooming. It's just just start to experiment. Um >> What's your favorite use case in your personal life? >> Uh personal life, I'm a big cyclist. Um and I really have enjoyed working with
[33:47] um Open AI or ChatGPT um sorry, Anthropic now to help me route courses, especially in places I've never been. Um and like Maine or Vermont, I'm I'm from from Boston, so um a lot of dirt roads, a lot of sort of back roads. Um
[33:59] and I've used it to actually help me plan routes, um do water stops and other things. And it's been actually kind of cool. Um and so, there's a big faith a leap of faith that I have to trust that it actually Yeah, make it home, right? Um but it uh
[34:11] yeah, it's it's only got me into trouble a few times, but it >>
[34:25] journey that you wish someone had told you before you started? >> Oh, that's a that's a good one. I I I don't know if we got necessarily anything wrong. Well, we're Nasdaq, we never get anything wrong, but no.
[34:37] I don't think we got anything wrong. I think it was, you know, early on the pace of change was so quick, right? There was a new model being released or a new agentic framework being released or some some new shiny toy being released and sort of getting trapped with the the churn of
[34:51] What about this? What about this? Like how, you know, and as opposed to maybe going a little bit deeper on something. You know, we're still with LangSmith and LangGraph. That was one of the tools we used. I mean, that was That was an early an early vendor
[35:04] or an early open source project and we've stuck with it. But, I think that chasing the models I think won't won't get you there. You know, the the pace of change is going to continue to increase. >> Very tempting. Yeah. If you'd like to speak more with Lenny or myself, please
[35:17] stop by booth number 206. I want to say a huge thank you to Lenny for spending the time and to you all for joining. >> And one last point, we have a booth here as well. Please visit them in the expo. Data Link and Investment. You can look at all of our data products
[35:29] and and some of our products are for financial professionals as well. So, thank you. >> Amazing.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.