Building MCP servers for agentic access to the Databricks lakehouse

Summary

Optiver, a global market maker with highly technical users, built a set of MCP servers and skills to give AI agents reliable, governed access to Databricks databases, job deployment, and complex workflows, recognizing that infrastructure building blocks must remain stable even as orchestration layers rapidly evolve.
The architecture closes four gaps: broker-based authentication for multi-service agent credentials, a Database MCP for efficient schema inspection without triggering API rate limit cascades, persistent Python kernels for iterative development, and Spark UI summaries for closed-loop performance feedback.
These building blocks power four production use cases at Optiver—external data ingestion with contract analysis, researcher pipelines for hypothesis testing, investigation boards for incident response, and distributed experiments at petabyte scale.

Building MCP servers for agentic access to the Databricks lakehouse

As AI-driven applications accelerate and users increasingly delegate work to agents, the platform infrastructure beneath matters more than the orchestration layer above. Optiver built a set of MCP servers and skills that provide agents the context Databricks does not expose by default, enabling seamless agent access to databases, job deployment, and complex workflows at scale.
Learn Optiver's architecture for agent-native platforms: broker-based authentication across multiple services, Database MCP for efficient schema inspection avoiding API rate limit cascades, persistent Python kernels for iterative development, and closed-loop feedback systems using Spark UI summaries. Explore four production use cases: external data ingestion with contract analysis, researcher pipelines for hypothesis testing, investigation boards for incident response, and distributed experiments at petabyte scale. this video reveals the building blocks that remain stable as orchestration layers evolve.
🤝

Chapters

FAQs

Why did Optiver build custom MCP servers instead of relying on existing orchestration tooling?

Optiver found that no matter how sophisticated the orchestration layer, agents still need reliable ways to connect to databases, authenticate across services, and inspect schemas—capabilities that existing tooling did not expose by default. Custom MCP servers provided stable, production-grade building blocks that remain useful regardless of which orchestration framework is in use.

What is the Database MCP and why does it matter?

The Database MCP is a custom server Optiver built to let agents efficiently inspect database schemas without making individual API calls per table or column. Without it, agents enumerating schemas trigger cascading rate limit failures; the Database MCP batches and caches schema information to prevent this and reduce latency.

How does Optiver use persistent Python kernels for agent development?

Rather than starting a fresh Python process for every agent interaction, Optiver maintains persistent kernels that preserve state between calls. This enables iterative development patterns where agents refine code and data transformations incrementally, significantly reducing latency and compute overhead compared to stateless execution.

What production use cases did Optiver build on their MCP foundation?

Optiver built four production use cases: external data ingestion with automated contract analysis, researcher pipelines that enable hypothesis testing without manual data engineering, investigation boards for rapid incident response, and distributed experiments that process data at petabyte scale.

Full transcript

[00:08] Hello everyone, I'm Kelvin. Welcome to the talk. Before I get started, just some heads up like for the content today. I have skipped through some of the low-level technical details just so I have time to cover more part of our AI stack. But if you're interested, feel free to stick around and ask me questions afterward. I'll be here.
[00:26] Cool. So, let's get started. So over the past year, for the past 12 months, I'm sure that we can all feel that like AI usage has been just growing so fast, moving so quickly and octopus is the same. The space moves so fast that what
[00:42] consider as best practice just keep changing like it's hard to catch up. In fact, when I was starting to prepare for this talk back in February, what I had in mind was just simply sharing some of our fancy use cases. easy talk. But then
[00:58] as I was preparing and then I looked through the last 12 months what happened in the space like 12 months ago like people were using like things like copilot and then we move on to cursor and then clone and then we moved to CLI toolings and
[01:15] then like codeex and all that and then people start thinking high level like agent orchestration harness and now we have harness of harness. So I don't know how this will go in like three or six month time. So but I do want to present something that would stick around for longer right something
[01:30] that would not get outdated in like six month time. So I look through all the transition like think about what actually stick behind like what actually just still there and then I realized no matter how fancy your orchestration layer is your harnesses like the agents
[01:48] still need to know how to connect to a database how to post on confluence like how to do all that. So like the building blocks behind all the use case to stick around. So this is what this talk is going to be about. We're going to dig into each of the use cases and then look
[02:03] through the building blocks. Hopefully, that's something that you can just take home and start building yourself and apply to your own stack. So, here's our agenda for today. Part one, we'll talk about the shift in who our user are and what that means for us
[02:21] as a platform team. And part two, we'll talk about like the gaps that get opened up after all these new users and how do we address each gaps. And last, we'll still talk about our use cases after we have a strong foundation. What are the
[02:36] use case that we build on top? Before we dig any deeper, let's talk about who are we. We are Optifa and Optifa is a global market maker. We are founded in Amsterdam 40 years ago. We have around
[02:52] 2,000 people and 11 office around the globe. We provide liquidity and in all the exchange around the world with our own capital and our own risk. And there are two things I want to highlight here. One most of our datab all of our user
[03:08] data users is our colleagues is our internal like OT employees. They are highly technical individuals. They love custom setups. two in Optifa drix is one of the platform but we have more platforms and these two will explain
[03:25] some of our design choices later on and as for myself I'm in the le house team we run the database platform and we help the users we build toolings to help the users leverage of that platform better and for the last two years or so my high
[03:42] level role has been similar like I sit between the platform and the users to help them work on the platform better like the researchers, traders, and some developers. And now there's like a new type of user that came along.
[03:58] So who are our new users? Let me show you some of the numbers that made us pay attentions. So the graph here shows that the monthly active user in different platforms in Optifer with the red line show the AI
[04:14] user account and the dark line show the data ones. We can see that like over the last 12 months they're both growing rapidly and again two things to pay attention here. First look at the AI user account like Optifer use AI for various reasons
[04:31] is one of them but not only not the only reason that explain why the AI user count is much higher than the database ones and what it means is that the native entry point for our AI product would be sitting outside of data bricks naturally because we need to connect to
[04:47] different platforms and second this show the overlap of the two users So 12 months ago this time around like onethird of the data users use AI and now basically every one of them. What it
[05:04] means is that now the integration between the two platform has to be seamless. So let's look at how. So as the overlap goes up, what it means is that now people are accessing data
[05:21] bricks through an agent in a terminal. It's not like just someone clicking through the UI anymore. What accessing the platform is now a model that need clear structure context and like for people who pick up like
[05:37] data bricks after they start using AI, they might not see the UI at all. like all they will see might be just like a login page once every like few days and that's it. So this is the shift like with the usage pattern shift like as a
[05:53] platform team we also need a change. So here comes part two the main topic of today. If agent are now our new users how do we address that need?
[06:11] Before we dive into each case, I just want to quickly go through some of the terms I will use throughout this talk. Like I know some of you guys already most of you guys already know about this, but just want to make sure we are speaking the same language. So first MCP essentially it's just like a server. It could be running in local or remotely but it expose the
[06:28] tools for agents to use. The important thing is that like the session doesn't hold a handle like it's like some server sitting outside the agent sessions and you can call it a cross session and the state will be persisted.
[06:47] Okay. Skills. So consider skills as like a package knowhow on some specific task. So it have all the instruction definitions and knowledge you need and for for an agent to need to perform a task. It sometime come with some template or scripts and all that but it's all for the specific task
[07:03] and context window. Think of this as agents active memory. So basically it doesn't include all the like the knowledge about the worlds and all that those are baked into the model but everything you put in since you start the task will be in the context window.
[07:19] And usually an agent works much better when the context is focused and relevant. Just think of it like a human brain like you would like you wouldn't work too great if you have too much on your mind when you perform like a task. So that's how aging works
[07:35] and there's like a closed loop design concept. So we're talking about feedback loop here. So um so on contrast like what like a open loop is is like typically if you fire off a prompt the agent stop working and then you check
[07:50] the output you flat the output is tell them what's wrong and then go back. That's like the open loop system. But a lot of the time like the thing you actually validate the agent can do it themsself if you provide enough tools. So that like what you get is you send a prompt the agent start working in a loop
[08:07] in a circle and by the time it actually tells you hey I'm done there's a good chance that the result is actually now good like you don't have to go through another iteration. And throughout this talk I will go through a few of different feedback or sensor system like where we can like
[08:24] achieve this closed loop design. Cool. So let's get started. Just a little bit of background on what did we choose to build on. So when we first started working on the integration, we consider some native toolings
[08:40] like Genie and Genie Space MCP. They are really good and easy to use but we hit some limitation. Remember when I say our users are technical and we have more like platform than just data bricks. We usually use AI
[08:55] in terminal and for genie it mostly live in the UI and we tried the genie space MCP but back then when we are trying it they have some data set limitation which is I think it's like 100 which is too low for us like when I say like our user
[09:11] are highly technical individuals these are deal breakers for them but instead we look into the lower level features like things like unity catalog and job are actually really Good for agentic workflow like unity catalog provide a
[09:28] really strong permissioning models and workspace isolations which means that no matter how loose I set my agent to be like it can never mess up with the production data and job like every job is been up their own cluster full isolation noisy neighbor issues
[09:45] combining these two is essentially like um sandbox on cloud for your data flows these two are what makes me feel safe as a platform admin So these are the foundation that we choose to build on and also worth mentioning like we evaluated this like a while back. So since then Genie gone a
[10:02] long way. So I would encourage you guys to try out and make your own judgment. And the way I want to approach this is by walking you through a typical user journey. So usually it goes like this.
[10:17] Authenticate, inspect, develop, deploy, and then the feedback loop. At each step we will discuss what an agent actually need and how do we address the gap. And remember like everything that I will show you here there are two goals. It
[10:34] will be empowering the agent in two ways. One we want to take human out the loop as much as possible so that we don't need someone that babysitting every step along the way. Two, we want to provide better context for the agent just so the contacts are
[10:50] relevant and they can give us some actual answer instead of like just a confident guess. So less babysitting better context. Keep this two in mind.
[11:06] And the first step is authenticate because obviously you can't do anything on the platform until you have proven who you are. So like for in human just logging into the database restace from a UI these are easy you just open the browser just click it and then you're
[11:22] set. Unfortunately for us our deaf machine is not in the same machine as our browser. So we need to do one more step. We need to pop forward so that our death machine can now talk to the browser and authenticate through it. The
[11:38] first time usually works uh but like the token would expire. So in a few days time you do it again and now there's like a good chance that you will hit that annoying pot conflict issues like because the pot has been claimed
[11:54] sometimes by yourself from like a sessions a few days ago. Worse if it is like a share machine and it was someone else. Now this was like a long running issues like but never quite big enough for us to fix it because only power user ever
[12:10] hit it. Most of the user just stay in the UI and it was just fine. So why are we talking about this now? Because in this people need the agent to access data bricks instead of themsel
[12:26] and everyone use agent now. So now this little issues became like the like an issues in everyone's critical path. So now we need to solve it. So how? So here comes the offroer.
[12:41] Basically it's a service that we register on data bricks as like a trusted service. So now we authenticate through the broker instead of data bricks. So this way no we don't have to do any port forward dance anymore. just strike to the broker and it's done and it
[12:58] actually solved a second deeper problem at the same time and which was the original goal of this offroer. This is just like a lucky coincident. How does a B actually add as a user? So the knife answer is like just use service principle. But then if you think
[13:15] about how it actually works, if you want to assign a service principle to a bot, what you need is like you need a supererset of all the permission for people who are going to use this bot, which is probably not what you want. Instead, people like the bot can now
[13:32] authenticate through the offroer and grab the user credential instead because it hold all the off tokens. And it's not just for data bricks. It works for Confluence, Slacks, GitHub, all the other things. Picture this. If
[13:48] you just send a DM like just send a message on Slack, t a bot and then hey, it's like, hey, can you do help me do XY Z? It's like all right, it send you back a link like that just ask you to sign in, click once, you're set, and now it can query data bricks. It can afterwards
[14:06] it can post a result to your conference page like it can raise a PR for you and even reply Slack in other threat. So it can do basically do everything that you can do. So that's pretty cool right? So once you are authenticated
[14:23] now the people need to like the agent need to read the data you need to inspect the table like get the schema right so that it can actually write code that doesn't fail. So that's why we built a database MCP just for that.
[14:39] Now you might ask like if the terminal like if the agent already have the terminal access why don't we just use the direct CLI and let me make this concrete with a true story. So imagine you are now
[14:57] Kelvin in like a on like a Friday afternoon 5:00 pm. You're probably thinking like it's end of the week whether you should go for a drink or go back home and then alert start going off like across the entire firms like half of the data
[15:13] bricks workflow just fell out of no reason and you start picking and you found out hey looks like some API limit issues has occurred. and you sign, you sit down, start digging, try to find something in the system
[15:30] table and it's not there because it's not in the system table at all and there's no way and you just have to go through the support process and find out what it is. Eventually after a few hours you find it cool out someone some of our
[15:45] users thinking hey it's Friday I don't I don't need to work for the weekend but my agent can so um they spin up some agents perform some tasks the problem is they don't spin up one agents they spin up like 50 because they can like every
[16:02] agent go through the same process right so user mentioned this table let's look at where the table is so I go through the naive process of listing catalog, listing schema, listing tables. All right, try to find it. So one prompt turned into like 20 30 API calls for one
[16:19] agent and you multiply by 50 multiply by the retries that after it hit the errors. That's pretty rough. It basically bring down the entire workspace. So the question is, do you want to be telephone on that Friday afternoon?
[16:37] If you don't, you probably need something like this. So what we do right now is we route all the user requests to a centralized MCP and we do two things inside. One, we swap those low limit API calls to some
[16:53] high limit equivalent if we can find one. So for the things like our listing schema, listing tables, you can probably do it through SQL instead. It's might be slower, but it's more stable. for those that we can't find an equivalent like uh like yeah like other APIs we just simply
[17:10] add a rate limiter there so that way one person's mistake wouldn't just bring down the entire space of course in a perfect world if the API limit doesn't exist I'm I'm happy to just package this into like a
[17:26] scale and then people can just use it locally but yeah that's in the future so Now, now the agent know how to how the data looks like probably like yeah it can start writing some code. You can already
[17:42] do that these days like you can write some code and execute it and then you see hey is it good and then it start again just keep iterating like this. So what's the problem? The problem is it's really slow because
[17:57] uh every single time you need to start from top to bottom if you want to change something, you want to ask like follow-up questions, it just literally run the whole thing again. And it also feels like a black box. So when you're debugging, it's pretty
[18:12] hard. So what do we do instead? It's like hey if they can't hold the sessions why don't we build a MCP that can hold the Python handles for them. So with this MCP what the agent now do instead is like tell the MCP what Python
[18:30] code you want to write they write it for you instead and they have like a persistent Python kernel just iterate it one cell at a time. So now the agent don't have to rerun the whole script all it will run as just one cell. So the iteration loop is much faster. No more
[18:46] you're saving compute and token cost. And also when the agent is done, you can basically just ask follow-up questions with probably running the whole thing. And let me show you how it works. So you see the like I input like prom
[19:04] and it starting it should start writing now. Now yes now. Okay. So if you look carefully you can see that the agent just write code and execute it one by one. Every time I write before it write new cell it will actually execute the original one
[19:20] as well. So it means that the previous cell are now becoming the context for later cell. So it can just keep building on top. And what's more important is that after you build it, if it hits some error, which yes, right here, and then
[19:37] you can just fix yourself easily. And after they're done, they we can just ask follow-up questions and they wouldn't rerun the whole thing. They would just add a new cell. So the iteration loop now is much more faster and more
[19:54] reliable. Okay. Cool. So now we can write code. We need to write better code. We need to teach the agent like how do we do some things in a specific way so that it don't make
[20:10] silly mistake. So we have skills for that. We have two type of skills. One is for some specific task like building dashboard. For building dashboard we have some really complex workflow to make sure
[20:25] they do the right thing. Basically it goes like this. The user tell them a prompt. All right. So basically I want to build a dashboard like this and then it start drafting the queries and then it enter the first close loop. It ask the dex mcb for hey does my query looks
[20:42] good? Is the data looks good or is my query efficient? Like it fetch the query plans evaluate itself a few times until the query looks performance and then you exit the loop and go to the next one. And now like we enter the deployment process where we enter another closed
[20:59] loop which is it try to deploy to data bricks and see if it actually show up on the dashboard. There are two feedback loop we use here. We first fetch logs from the CI so that all right if there are any deployment error go fix it and if it get through
[21:15] the deployment now we spin up a playright MCP just to look at hey is it actually showing up on the dashboard. If it does cool like the dashboard now are ready for human to evaluate. So that way we ensure like basically one
[21:31] prompt can just turn into a dashboard. And the other thing is that like we need to provide the agent some knowledge about the things that they don't know about like um some proprietary knowledge that they are not trained for. It could be from data bricks. It could be from
[21:48] ourself like they there's no way they know how photon works because like they always keep it as like a secret and also they they don't know how our workspace is set up and what our business knowledge is about and for things like photon we actually really need a skills
[22:05] because like if we're not using it right we're paying three times for nothing. So we we need to get it right. So the way we do it is by combining a few things. We pulled the official database document for what we can know. It's like some high level understanding and then we
[22:21] pulled a photon research paper for some low-level understanding and then combining with our own learning throughout all these years of like paying extra course and then we got the skills. So now the agent know how to write proper fold on code. There are some
[22:38] caveat which I will cover in later sessions but that's basically our skills. And now we write pretty good code. Now we get to the deploy step. Database gave us a great way of deploying jobs. They call database SS
[22:56] bundle we call dep. So raw dep is like probably couple hundred lines of yaml. So agent can get it right if you give them the correct feedback loop like from CIS and deploy they can figure it out but at what cost? like they can keep
[23:13] iterating and fixing themselves and the moment they get through the CI cool it might be like our 10 loop or something and also like it might not align with the house rule that like the defaults that you want them to be like it might be the wrong policy that they're using
[23:29] like no alert different workspaces this could go like really wrong so what we did is we built a Python wrapper just for that so it's this idea is simple we Let's make it like a decorated format so that like the dependency can be auto
[23:46] detected and what's more important is that like because now the building yaml step is like a deterministic script. So we can just inject all our house rule in this in this time and it shrink what the agent need to get right. All you need to
[24:03] change is the framework that allow them to change and naturally the functions and all that like the functions signatures comments become the native context of what the agent know about. This is really about goals like we shrink what the agent need to get right
[24:19] so that it can much easier get it right. The two obvious benefit one is like for human to reviewing the code it's also way less review instead of looking at the YAML you just look at like a small much smaller Python snippet and then you
[24:34] see that's probably looks right and the second is like the agent will not drift away from standard easily because the standard isn't something the agent need to remember it just apply automatically
[24:52] and now we're at the last that the feedback loop. So since the start of spark or start of spark UI that has been the source of truth of the question of why do why is my job so slow before all this agent things like what
[25:07] we do is like all right I go look at the spot UI and find out what's slow and then fix the cooks and then agent came along and now we start thinking all right how should we fit the context to agent instead because obviously no one want to keep taking spot UI like Now we
[25:24] have agents. We want them to take it instead. So we can already do that in some ways. We can copy the spot plan, paste it in there, and it works, but also it's like not programmatic, right? So it's not scalable. So we try to find
[25:39] somewhere is like more programmatic so that agent can just do it instead of us. So we find a tricky way of doing it. So first like humans still need to be involved. We need to spin up a spot history server and then uh we open the
[25:56] browser. We click on the developer tools. We steal a cookie out of the browser, pass it on to the agent and then hey now the agent can act as like our browser. It's great, right? Um our security team doesn't think so. And uh
[26:13] it's like a workflow that basically really hard to productionize. Think about like when someone try to use this MCP, it's like, "Hey, can you fetch this for me?" And the agent reply, "Yes, can you give me your token first? This feels more like a fishing attempt
[26:29] than like a agent actually trying to help you. So there's no way that's going to work." So we keep building and then um we realize cool actually now that an agent need to read it instead of a human, we don't
[26:45] actually need the UI at all. So why do we border spin up the server? Everything we need is in the event logs. So let's do that instead and that's what we did. We built a spot UI MCP which is like a server where agent only need to hand
[27:02] them a job URL and then it grab the URL go look for the cluster pull the logs from locally like it could be gigabyte inside. So we have to pass in Russ just so it's much quicker and look at the
[27:17] difference. Left hand side is what you would get in like the copy and pasting workflow or like using the cheeky way of like just hitting the API directly. You'll get the raw plan which it works but not that great because you're essentially relying on LM to aggregate
[27:35] all the metrics for you but it sometimes got it right but sometime it don't and the result is not deterministic instead. Now since we got all the information within like like the rust script we can just aggregate them beforehand. So we tell the we tell the agents like a
[27:50] summary of like how this spark job going like it's like some utilization rate some hostage like are there any spills and all that and then we give them a long side with a handle. It's like hey if you want more information for this stage call me back on that all this information is caged so
[28:08] it will be quick. So this way we ensure like the context like after after go through all the plans right we don't use up too much context in the agent so that it will be really relevant and target for what we actually need to address.
[28:23] So this is our main feedback loop and that's our user journey and now after we fix all the gaps let's look at what are the use case that we actually built on top
[28:40] the first one I want to go through with you is our external data to give you some background external data here means that like the data we purchase from vendor so we basically need we need a lot of them and the process usually goes like this. It goes with some email
[28:56] conversation. It's like, "Hey, I want your data." And then they reply with like a contract which we need to review a few times. Go through every clause so they're not scamming us. And then we agree to the terms and then now what we get some data documents on like how do
[29:11] we connect to the database? How what is the schema looks like and all that and this is the first part of the pipeline or first half of the pipeline. And that doesn't sound too bad, right? But what about this? We get like probably 300 emails per week
[29:28] for that. Like it's to be honest, it's not all contracts and data documents, right? But we need to figure out like a human need to go through and reason about all these and it is quite draining. It quickly become a operational nightmare. So like no one want to work like that.
[29:44] So they build a pipeline just for that. We need to pull agent into the picture. So the first step with it is we simply ingest all the email to a data table and then for those with attachment we pass
[30:01] it we pass it with like the database function like there's like a AI pass document function which we can pass the documents into a semistructured format. So after it's passed we can now search for it and classify it for like two
[30:16] different docs. So the two things we care about here is contract and the data data documents. For contract after we pass it since it is now in like a semiructured format agent can read each of the clause and then just like reason about it like do we need to pay
[30:32] attention here? No like do we need to pay attention here and we highlight all the clause that need human attention and then human just review it on the UI instead. This reduce like 80% of the human burden just make it so much faster. Humans still act as like the final call but like now they don't have
[30:48] to pay attention on those like ECQL and then the data docs we just spin up a agent to look at the data documents pull all the information we need for connecting to the database like schemas and connection details and
[31:03] then now we can ingest it to like now we have like a good way to set up injection workflow to optivo and after the injection here comes the cool So like auto vendor just like to call the
[31:20] things slightly different format like sometime they want to call like exchange sometime they want to call like tickers but we have our preferred way of calling things like we want to call it makes we want to call it simple sometime they want to put like month in front of date which is nonsense we need to fix that so
[31:37] we need to um so we spin up agents to write some DVD code just for that like it's by leveraging the DB and SQL warehouse quickly can generate like a simple transformation pipeline which just clean the schema up quickly and then back field quickly as well. So now
[31:54] we have like a really clean table for researchers to use it and talking about researchers let's talk about our agentic research pipeline as well. So this is now basically what they do every day like they come into the
[32:10] office they open their nicely set up terminal ask them like a question like basically this is something that I want to dig through today it doesn't have to be super sophisticated prompt because it's nicely set up agent so it know where to fetch more context
[32:27] so it come up with a plan change a little things like and then just let it go loose and then you go off to a different task like come back in 15 minutes. It's done. It's like there's like a notebook there with some
[32:42] summaries, some data, some plot and then you can look at it, see if you like it. If you don't, ask some follow-up questions and then now you're you're happy with it. Then you can either publish it as like dashboard or just make it like a schedule job. Every step
[32:58] either way like either way you do it like it will be in Git version control. And to be honest, this was the same workflow that researchers been doing ever since like before the AI agent era. It's just that now it's just much faster. Now the bottleneck is like not
[33:14] how quick you can implement a notebook. Now the now the bottleneck is like how quick can you come up with a hypothesis and let's look at the third use case. So we have something we call investigation
[33:29] board. What we do is we just have a bot that got it just connect to all the platform we have in auto so that it can do the investigation for us instead. So this is what it will looks like in real life. So nowadays every time there's like a data
[33:45] pipeline fail we usually get a slack alert for it and then like the spot will react to it like as first line of defense. So we'll go fetch all the information you can find, put up a summary and before the engineer actually come online it gave you like a PR like
[34:02] say hey this is something I think like you should fix it like that and then the engineer came along they can stay on slack like just look at it look at the discuss like how should we fix it and then suggest changes to the uh to the bot and then you can fix it looks good merge it and then just click off the
[34:18] back as well. So this pretty much sounds like the genie of that they've been announcing just that we have our own for basically all platforms
[34:34] and the last one is like data brick experiment. So in Optiva we have some pabyte level tables and you know like when when we are using it we found some issues with it like every time we try to read it seems like there's like a fixed overhead that's
[34:50] like a minute or two like every time no matter how big or small your query is so we want to fix that and we know what's the issue actually like we know it's like a classic checkpoint issues but uh how do we fix that like it's just too many ways we can try like there are so many things we need to go through like
[35:07] like does it have something to do with the read engines or writer or checkpoint version? How should we check pointer? How many stacks should we collect? Like how many commit does it affect? So all these there like just too many things. So it's pretty hard to just come up with
[35:23] a solution just by hand right like so what you need to do is just run an experiment on it which I did. So sometime last year I tried to do this experiment by hand because back then agent is not that popular yet. So uh setting up the experiment is it's okay
[35:40] like it I can get it done like just write some tables and then run some benchmarking and I got the result. Resinging about the result is the hard part. It's literally a nightmare consider like because what we need to do is not just compare the workoff time. We need to look at the spot UI see hey does
[35:57] it actually get pushed down to this filter. Is it actually pruning enough? And if you don't know yet, like getting the spot UI for fetching the checkpoint file is even harder than just looking at the spot UI job. So um so I park it
[36:13] after I find some initial findings like it was useful but not the full picture. And then I come across this like a few months ago. Again this time around I have like a nicely setup agent terminal. So what I did is yeah just put agent
[36:29] into it. The first step the hypothesis like I'm not the only one that come up with the hypothesis now like now I get to um set up an agent session hook it up with some open source spark and delta repo start debating with them ask them
[36:44] to validate my hypothesis before they actually spin up the experiment even come up with some new ones and then after I'm happy with that just let it kickstart it'll be in a loop and then just keep running and running until all the experiment is done and then the cool part is like I don't have to read about
[37:01] each individual one of them like it just write a nice summary on confluence and then I can just read that instead. So it's much easier for human to process and it actually help us find quite a few traps along the way because like we we do need that because we don't want to we pour a pabyte table more than one time
[37:19] and that's basically what's our use case and let's talk about what's still broken from all this. The first as I mentioned like we built some photon skills right but skills get updated because database is doing their job like they are making photon better and every time they make
[37:36] it better our version got st and there's no way for us to just keep track of that second like the API limit as I mentioned it would be a really cool thing if they have that so consider this a data brick wish list
[37:52] so it would be cool if they have a knowledge base that just cover like some papachi launch like photon like not asking for trade secret but some high level understanding would be cool and also some agent grade API limit like hopefully next year they will announce
[38:08] serless API which will just solve this and that's basically it so happy for question but if you're interested in building these kind of things with us feel free to just check our job board like we have a open culture and uh happy
[38:25] for questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.