Skip to main content

The Hitchhiker's Guide to Delta Lake Streaming in an Agentic Universe

Summary

  • Streaming best practices: Drawing from co-authoring The Delta Lake Definitive Guide, this video covers practical lessons for writing generic streaming applications and orchestrating complex data workflows.
  • Agentic-first approach: Learn how to leverage Delta Lake metadata for enabling agentic pipelines and workflow automation, positioning streaming applications for the age of autonomous AI agents.
  • Delta clustering and optimization: Discover best practices for Delta clustering—including liquid and Z-order techniques—and practical tips for building resilient streaming pipelines on the Databricks Data and AI platform.

The Hitchhiker's Guide to Delta Lake Streaming in an Agentic Universe

Watch: The Hitchhiker's Guide to Delta Lake Streaming in an Agentic Universe
Talk by Scott Haines, Distinguished Software Engineer, Nike
As data engineering continues to evolve the shift from batch-oriented to streaming-first has become standard across the enterprise. The reality is these changes have been taking shape for the past decade — we just now also happen to be standing on the precipice of true disruption through automation, the likes of which we could only dream about before. Yes, AI Agents and LLMs are already a large part of our daily lives, but we (as data engineers) are ultimately on the frontlines ensuring that the future of AI is powered by consistent, just-in-time data — and Delta Lake is critical to help us get there. this video will provide you with best practices learned the hard way by one of the authors of The Delta Lake Definitive Guide including: Guide to writing generic applications as components Workflow automation tips and tricks Tips and tricks for Delta clustering (liquid, z-order, and classic) Future facing: Leveraging metadata for agentic pipelines and workflow automation
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why is Delta Lake important for streaming?

Delta Lake provides ACID guarantees, schema enforcement and time travel on streaming data, ensuring reliable and auditable data pipelines on the Databricks Data and AI platform.

How do I build generic streaming components?

The video discusses architectural patterns for writing reusable streaming components that can be composed into larger workflows rather than monolithic applications.

What is Delta Lake?

This video is part of the Hitchhiker's Guide to Delta Lake streaming series. Delta Lake provides ACID transactions, schema enforcement, and time-travel capabilities for Apache Spark data lakes on the Databricks Data and AI platform. In an agentic universe, Delta Lake streaming enables reliable real-time data pipelines with tool calling and advanced features for AI agent workflows.

Is Delta Lake dead?

This video demonstrates that Delta Lake remains central to modern data architectures, particularly for agentic AI workflows. Rather than being legacy technology, Delta Lake continues to evolve with new capabilities like streaming enhancements and tool calling support. The speaker, part of the Spark and Delta Lake community for 10 years, emphasizes Delta Lake's continuing relevance in the Databricks Data and AI platform ecosystem.

What is Delta clustering?

Delta clustering includes techniques like Z-order and liquid clustering that physically organize data for faster queries, improving performance of streaming analytics workloads.

Full transcript

Um well, thank you everybody for coming at 8 a. m. Um for all the engineers in the room, I apologize as well. Um it's very early, especially to kind of kick off things. Um but Um I've been part of the Spark and data uh Delta Tele community for about like 10 years. um uh wrote a book, did a bunch of other cool stuff. Um and this is the second kind of series in like the Hitchhiker's Guide to Delta Lake streaming. Um we're just actually now adding like tool calling and some other stuff to it.
Um so I'll just kind of get started. Um a little bit I'm like I can't talk. It's 8 in the morning. A little bit about me. I'm a distinguished software engineer at Nike. Um I've got a colleague over there, Bridger. What's up, dude? Um and I just enjoy teaching, mentoring. um a lot of the things that people do is like they get later on in their careers. Um just teaching people who can still, you know, handle a full workday to get their job done um very well.
Um and I've also just been working on like huge distributed systems since about 2008. All right, so into what the session is. Um does anybody here write anything on Kubernetes? All right, awesome. We got a couple people over there. Um there's something called crossplane and it's all about like compositions. Um compositions essentially allow you to create applications that can be essentially built via just configuration.
Um so we're going to walk through an entire application that is configuration based and then we're going to walk through an entire application that is a generated um and kind of try and find like the connections and corlaries between the two. Um all right, cool. that actually animated which is nice. All right, so starting off we're just going to talk about how do you create a generic application? What does that mean?
Um and what does it kind of mean to build things based on components versus just building things as like a stream of consciousness in a notebook. So in a nutshell um anybody who's written a streaming application before has probably seen something very similar to this. Um if you who here still write Spark applications? Awesome. Good. Um, so these classes will make some sense to you as well. Um, so with any kind of streaming application, you have a source.
That source, in this case, we're going to just be talking about Delta Lake, but it can be anything that can be connected uh to a data stream. Um, so we have data stream reader. That's great. That gives us bytes, creates data, creates data frames. We've got transformations, we've got a data stream reader, and essentially, if you think about like a ripple, we're essentially going through, we're reading, we're evaluating, we're printing, and we're looping back again.
And this goes on forever. Um, the interesting thing about all of this is that almost every application you see is essentially just a UDF. So, you can create components that make it really easy to create config driven pipelines. And so, the only thing you have to really worry about is how do I create transforms? And transforms are pretty easy as long as you understand the schema of what it is that you're actually trying to read.
Um, and so along those lines, like um, if you've ever called a transform function on a data frame, it's also chainable. And so there's a lot of things you can do just by doing a chain of transformations. But you don't have to really worry about what's coming from upstream or where is it actually going if that's all config driven. You don't have to even really worry about it too much. Um, and that's kind of where a lot of these like, you know, base application patterns come from.
This is essentially like the foundation of how you build like an agentic pipeline. Um, so moving over there. So it all everything just kind of depends on like the state of the application. So if you're just reading and writing, then like you're essentially just having like an extract load pipeline. And so like you're missing like the T or the small T. And some people just kind of call that small T as well. Going back to the base applications, um, if anyone's ever uh written applications in a shared environment um, and had anybody else rewrite or overwrite checkpoints, it can be painful.
Um, if you've ever written an application, you've accidentally blown away your checkpoints, it can be painful. Um there's a lot of things you can do to actually optimize the way that you write your checkpoints. Um and so in this application over here for brevity, can't really see it on the bottom because like we have the subtitles. Um but my footnote says pass is just basically used to be able to put this all on a slide without having like an entire class.
Um so at Nike we worked on something called Picassio which is essentially a way of creating generic pipelines. Um, and so those are all based on like streaming applications that took essentially like Spark configuration to go autowire themselves. So if you're like familiar with like uh Drop Wizard, Spring, anything else like that, that's sort of the same idea where you're just autowiring applications all based on configuration.
So what changes the least is a lot of times configuration. What's most important a lot of times is configuration versus like magic numbers and things like that. Um, so everything starts off with a streaming app. Um, so if I have a streaming app, I don't know if you can see this in the back. I chose green, which is probably a bad idea. Um, but essentially if we're re if we're reusing any kind of like Spark application config, so I'll have like Spark app name Delta to Delta app, Spark app version, so I can keep track of things for monitoring.
Um, if I want to take, you know, a look at a new checkpoint path and I'm using Unity catalog, I can create a Unity catalog volume to put all of my actual checkpoint data in. So as my application is streaming, if I have any kind of stateful operations or transforms or anything else like that, um I can always get back to exactly where I was before. Um and then the interesting thing too is that a lot of times like when you're running a streaming application, whether it's Delta Lake or anything else, um you also have a checkpoint version.
And so if you want to blow away a checkpoint version because you're doing a triage job, you might want to split at a certain point in time, copy everything from a volume and start in two directions. Um and there's a lot of different things you can do with that as well. So being able to have everything be config driven means that when you get paged at 3 in the morning, you can basically just go bump a version number, start a job to go do like a triage, and then everything else can keep on running without you having to think about it.
Um, agents are also good for that. Um, so in this case, we've got a streaming app, which is like the base of our application. These are just kind of first principles. A streaming app needs to create a like we need to generate a stream. So what exactly is that? We're reading stream with options. Then we're going to do something with that. It needs a format. Um, so if you've ever used Spark before, got a read stream. It's got a format for Delta.
Now we're reading from Delta. That's awesome. If we want to write to Delta, we're using Delta format again. It's pretty easy. So it's all part of like the kind of boring boiler plate. Um, so source components, right? Um, I don't know if you can see this from the back. Um, I'll post the actual slides after, but um, in this case, we've got a source generator. Um, it's got a config prefix because this is a config driven application.
It's got a table because it's associated with an actual upstream delta lake table. We want to go generate some options. We want to go essentially be able to autowire this just based off of configuration. And then at the end I also want to be able to go create a data stream reader. Um so for the data stream reader class it's essentially what you can actually composite. So it doesn't do anything until you actually call load or you call table on it.
At that point you have a data frame that's a streaming dataf frame but before that you have a promise of something. Um so we like promises of something. So it allows us to do something different with it if we want to. Um so in this case under generally no one can see the red but uh over here it's like self app generate read stream generate options format delta. This little block right here. Cool. Now we're going to create a streaming application that is a delta based application.
So going back to the actual presentation we're creating delta streamer. We are doing delta streaming. Sorry, it's 8:00. Um, cool. So, other than that, um, just kind of, uh, piggybacking off of like kind of common things that people see in, uh, Spark applications like two data frame. Yay. All right. So, right now we have a base application. The base application gives us our configuration. It also gives us our Spark session, allows us to piggyback off of that to do interesting things.
It also gives us an interface to go and generate streaming sources. It gives us an interface for creating uh, streaming syncs as well. There's also streaming options. Um, so I helped write the Delta Lake definitive guide. Um, it's free if you don't want to pay O' a lot of money for it. Um, if you just go to delta. io, you can download the PDF for free. Um, so we go over essentially like all the options that are available for like a, you know, what's the streaming source, what can it do, what does all this actually mean?
Like if you try and encapsulate like an entire book in a session, usually you run out of time. Um, so some of the things that are most interesting like for any kind of streaming application is essentially rate limiting. It's like how fast do I run out of memory or how fast do I not run out of memory. Um, so there's also like runtime ignorance. So if I'm running a job and I don't care about changes, I don't care about deletes, I can keep my job running essentially not ever worrying about what's actually happening upstream until I hit like a schema change that breaks my application.
Um, and so being able to do that allows your applications to continuously run. Um, two years ago, um, I gave the same session, which is just the hitchhiker's guide to Delta Lake streaming. Um, it's all the same Jupyter notebooks that are used for this session as well. Um, and that goes through everything you'd ever want to do with Delta Lake streaming, um, prior to like the agentic world we live in right now. Um, cool.
So, so far, any other kind of questions at this point? Perfect. That's great. Um, all right. So Delta Lake, if you've ever used it, same as iceberg, you have the ability to time travel. You can go back in time, you can go back to an earlier version of your application. For people who've been doing data engineering for a long time, uh triage is difficult, but if you can just roll back to a specific version, that's really great.
Um, and it's another thing too where if you want to outsource the ability to go and find the last good state, last good version of like a application, it's another thing that is uh very good for like an agentic, you know, flow. It's like go find me my last, you know, safe state. Take me back to it. All right. So, moving on from there. So, engineering engineers in general are lazy. Um because we're lazy, we like to do things where we maybe will take a lot more time to go and actually build something that will then help us be even lazier in the future.
So we spend a lot of time being extra lazy, which is kind of a weird thing. Um so if I have a source generator, I've got a config prefix. Um this is essentially just the implementation um of like my source generator. So it takes options, it rolls those options up into an actual class, and that class has a bunch of different uh capabilities as well. So we want to do as little as possible. We want to remove as much human error as possible as well.
And we also want to be able to have uh interesting defaults. So I don't have defaults right over here, but anything that would be applied as a source option for your streaming pipeline that is common in most cases would go into my option defaults. Anything else would just be overridden and then just get merged together. And so we can take a look at that stuff as well. So down over here, I'm not providing like these functions, but generate options dictionary.
I'm taking my config prefix, which is like spark app source delta, taking my options defaults, and then I'm mixing in what I have actually on the spark session as well. So then we're just doing like a merge override for that. All right, so that's pretty easy so far. As I said, it's basically boilerplate. Um, do as little as possible and then go create cool things. So your application like as boring as this actually looks is essentially just a series of different configurations and then you just apply transformations.
So I've got a source, it's delta, it's got a table that has a catalog, that table has a schema, that table has a name. Awesome. If I want to keep my state around, I need a checkpoint path. Awesome. That's great. I have a sync that's delta that has a catalog, a schema, and a name. This stuff's not very difficult. It's essentially just us keeping track of these variables in a safe spot so that we can do an application release and go on with our lives.
Um so the big important part right goes back to still just the transformations themselves. So again we'll save questions for the end. Okay. Um so the big thing over here so I want to transform my source and then I can transform and do as many things as I want. So leave this up to your imagination. Everybody's project is different. Every transformation that we're doing is different based on the goals of our applications.
So that's awesome. So focus on what's important. I don't know if the pointer showing up, but on the bottom, focus on what you're doing and what you want to do versus just wiring up an application because that's the most boring part of your entire job. All right, cool. So we're going to finish up the writer now. So this is the other side of the other boiler plate. So so far from my diagram we've gone through the source we've gone through the streaming reader we've then applied pretend transformations that could be any transformation you actually want to bring to the table could be udf it could be a SQL um you know expression that doesn't really matter um at the end of the day your right stream also takes options those options can be configured as well and that allows you to control your application without having to really do any real manual work so in the same way that you don't use magic numbers in your applications when you're writing like actual like you know source controlled software.
The same thing over here can be controlled through uh just configuration. So config driven development sounds boring but it takes away a lot of the heartache of you know when things go wrong. So this probably doesn't read well from the back. Um but if I have a data stream writer and I want to actually write that data stream out in this case I want to go create a stream that's writing to a delta table. Um, I pretty much need everything that I talked about before for the other kind of boring side of the application.
So, uh, up in the top, you can't probably see it underneath like this cool like, uh, like I don't know, curtain. Um, but we've got streaming app which goes back to like our application context. We've got a streaming prefix. So, Spark app sync delta. We got a table name that we're going to generate based off of like what we actually have in our session. And then we're going to read a data frame in either at the time of our class generation or we're going to go do it uh when we have a couple of these different functions over here.
So we've got a from data frame. So in a lot of cases you'll see two data frame. So from data frame just absorbs a data frame and then creates a streaming pipeline which is kind of a fun practice. So um it's kind of self-explanatory. Uh takes a checkpoint location so we can save our state. Um and then that saves to whatever checkpoint path you have uh be it a volume or be it like an S3 bucket etc. Um so from dataf frame awesome we're going to go read it we're going to take it from the application and we're going to crash that's essentially the only options we have.
Um then the really only important part over here that matters is I'm going to go create a data stream writer that has an app. I'm going to generate this right stream. I'm going to generate my options. I'm not thinking about any of this stuff. All the stuff is kind of self-explanatory and it's config driven as well. So, uh the other interesting thing over here is like if you think about like what it is that a streaming application does, um it either goes as fast as it can, it goes and operates once with a trigger or it operates many times with a trigger um until we run out of data.
So, you've got like available now processing for triggers. It's a lot of different fun options. Um and so if those are also configurations, then you don't have to think about setting that. So if you have an application that you want to run as fast as possible because you're trying to catch up, you can do that. If you want to slow it down, you can do that all through the configuration. And again, it's kind of boring. You're just like, you know, you're moving knobs around.
Um but functions autot trigger, functions auto partition cluster by um also just takes configuration as well to apply to the actual uh end pipeline. Um and this is all still boilerplate. So again focusing on transformations is the only thing that actually is important because everything else is boilerplate. So kind of in conclusion like data sources can be configured in a generic way. It's not super exciting at like 8:18 in the morning to think of the fact that you know we're looking at boilerplate and we're looking at configuration.
Um, but it's really interesting if you have to create a bunch of pipelines and you're doing all the same thing over and over and over again to be able to just configure all of that um to just be something that essentially runs itself. Um, and then like your applications themselves essentially look like this, right? I've got an app. I've got a streaming dataf frame. I'm going to transform it and do something with it. Then I've got a streaming query that I'm going to run.
And in this case, I'm calling to table, which means it's going to write out to a managed table in like Unity catalog. Then anything outside of here can sit there and basically run. We can wait termination until our stream dies. Uh we can inject monitoring and anything else using Spark streams um to be able to get into like the other streaming queries like in within our application as well. All right, see if this one works.
Cool. Never mind. I thought it was on a different slide. All right, so assembling the pieces, right? So what we looked at so far was how to build an application that's essentially just a composite of configuration so that you focus on transformations being the transformations are fairly simple for an agent to figure out then you're kind of outsourcing most of the work to either configuration and guard rails or to an agent that can go and then do your work for you.
Um I think about that as like an agent that's doing my work for me. Um so you know 2. 0 you know, whatever else we would we actually want to call it. Um, so agents help us along our way. And so what we're going to look at for like the use case is essentially like assisting us to find important table uh important information about our tables. Um, and then this other stuff is more kind of food for thought. So if you have an agent that knows a lot about the context of your tables, your application, anything else, you can generate, you know, clean test data, which allows you to do stuff where you're not just reading prod data back into your actual applications. um figuring out the right syntax for specific things without having to look it up.
Um if you want to do lateral joins, ask it to do it for you and that's great and there's just so much more. Um all right, so with that said, um for anybody who's older like I am, we have this wonderful game called. In the very beginning, um you get a sword. In this case, we're getting Delta just because the conversation's on Delta. Uh this could be an OpenAI logo, this could be Anthropics logo, this could be Iceberg's logo.
It doesn't really matter. Um, but you're getting something that's going to help you along your way, right? So, cool. Thanks, PowerPoint. It's a nice one. Um, all right. So, workflow automation with agents. So, this part's also fun. The last part was kind of fun, but it's more boilerplate, so it's kind of boring. Um, so who's written a agentic tool? All right, awesome. Got a few people in here. We're going to go through what tools are, how to use tools, then how to incorporate tools.
Um, there's also going to be like a gigantic bit. ly So, if you want to go take a look at like the Jupyter notebook versus taking a look at my really crappy slides, um you can actually scroll through like on GitHub and it renders really nicely as well. Um all right, so moving on. Um last time I did this whole entire talk, there's a lot more references to the actual Hitchhiker's Guide to the Galaxy. This time there's essentially just one real reference to the Hitchhiker's Guide to the Galaxy.
So, um does anybody know the answer to this? All right, awesome. Cool. So, it's tools and the number 42. Um, woo, good joke. All right, awesome. So, moving on. Um, I spent way much um like way longer than I wanted to um trying to find um a schematic of a watch. Um I used midjourney for like three hours trying to get like a cross-section. I wanted to see like exploding gears and it didn't work as as I wanted to. But I did find this kind of interesting u you know picture I think of a Rolex, right?
Um, and so the interesting thing is that if you think about like an agentic tool in the same way that you think about a watch, you can think about a tool as like essentially like a component. And so, you know, is it a gear? Is it a screw? What is it doing, right? It's essentially an atomic unit that does something, but it doesn't really matter on its own. It only matters when it's all assembled together. And so, if we think about that, right, there's a reason for all of this.
Like, our LMS are incredibly smart. So, I have this little kind of quote that I came up with. Um, LMS know a ton, but sometimes they distinguish they struggle to distinguish between a ton as being a unit of measurement or a ton as an expression. So, it's like, I had a ton of fun. It's like, oh, cool. You were weighed down. It's like, that's not correct. Um, and so context is always king when you're trying to create essentially yourself or, you know, a better version of yourself that, you know, can do work, you know, while you're in meetings or something else like that as well.
Um, so context essentially is a it's providing a guiding light to a system that doesn't really see like it can see but it's just not seeing the same way that we are and so we have to kind of lead it along its way. Um, so LMS understand language and patterns really well. So we must just act on the part of you know providing essentially memory to that system. So um I think most people in here are data engineers or they're like within kind of the data engineering you know track within the companies that they work for.
I am as well. Um the interesting thing is that a lot of data engineers also came from like a software engineering background. Um tools, tool calling, everything else is essentially just APIs. We're creating APIs that are very well described so that we can create a system that understands and recognizes what it is that we're trying to accomplish. And so we're going to take a look at that right now. So hold on. I'm gonna have this disgusting rockstar.
Hold on one sec. Gross. Okay. All right. So, langraph 101. Has anyone heard of lang chain? Awesome. So, langraph is part of lang chain. Um, I think it's like a you now the supererset of like a lot of like the base tooling that came from just original lang chain. Makes it easier to kind of keep track of memory. U memory and agentic systems are essentially the way that you keep track of short and you know long-term things.
Um, if you think about the conversation we've been having, we went through this entire session so far. We're about 24 minutes into it. In that 24 minutes, we've talked about building applications. Many people probably forgot what the first slides are. It doesn't matter. Um, and it's the same way when you think about like an LLM and how it's actually thinking and working with us. Things only matter for a few kind of like epochs back.
So if you think about this essentially as like a sliding window of memory, um memory in an LLM for your actual session is essentially just being uh tracked via tokens. So I have a certain number of tokens before I'm going to start forgetting which is kind of the same thing we do. You know, we get caffeine, we get something else, you know, we boost our tokens, our life goes on. Um so anyways, what is a tool, right? A tool is essentially an API or a function.
Um, and it really kind of depends also on whether or not we are embedding a tool or if we're using a remote tool. Um, so people have probably heard of MCP like model context protocol. That's just a remote way of doing essentially the same thing as having like a, you know, a local API. But also in this example, I'm using the data bricks like workspace SDK. So that's already my SDK. It's already my API. This stuff is pretty much interchangeable at this point.
Awesome. So, what are we actually doing here? Does anybody use Unity catalog? Awesome. So, I don't have to explain Unity catalog, which is great. So, Unity catalog has a bunch of different cataloges. Your cataloges are essentially context for schemas. Your schemas are now essentially a more bound context for a collection of tables, volumes, uh, models, everything else. And that just kind of keeps on growing. So, for now, we're just going to think about that as a series of tables, right?
But we have this hierarchy. That hierarchy is also good for memory because I don't have to think about everything. I can contextually only keep track of the last few things that I'm thinking about. And so in kind of having the conversation about tools and tool calling, this felt like an appropriate analogy um as we started going through building up a system that understands and recognizes context and hierarchy. So tools just require a lot of descriptive metadata.
Um, if you've written like Swagger APIs or any other APIs in general, if you want someone to use it, you have to document it. Um, that allows them to use it without actually creating a meeting. Um, if you work at a big company, you'll have a lot of meetings. It's hard to get your job done if you're always in meetings. So, if people don't have to ask every time they want to use something, that's great. It's the same thing with an agent.
If my agentic tool doesn't have to really describe a lot and continuously get things wrong, then the LM is happy, I'm happy, and the cost and cont like cost for context doesn't go up too high. So, if I want to talk about a catalog, right, um, all I have to do is basically describe it. So, I don't know if people can see the red in the back. Maybe it doesn't show up, but so in like with any API, we've got a request, we've got a response, and that's highly structured.
And there's a really big reason for the structure itself. The structure allows us to tell our tool calling LLM what everything means. So if I've got a base model that's a list catalogs request and I've got max results because I want to actually limit the number of results then I can tell it it's the number of schemas to return from this request which is a typo because I copied this yesterday from the schemas example.
This is supposed to say cataloges but that's okay because it doesn't really matter. So forget schemas call it cataloges. So great. So we've got a field my default is one. I've got a description. I want things greater than zero. I want them less than 100. I've now told the OLM exactly what it can do to use this max results max results argument. I also have a response. That response just has a list of cataloges. These cataloges are just strings, but the description is the name of the base catalog.
Yay. This is getting fun. Everyone's probably having a good time. Tool list catalog. Description returns a list of base cataloges individually. I like to add more punctuation. It's fun. Um I think the M enjoys it as well. So individually each catalog name can be used as uh the parent catalog reference for the schema or table discovery. These are all important things because essentially like if you think about like you know how did Google learn back in the day for SEO was through descriptive metadata.
Our tools are essentially describing what they do using descriptive metadata. The one thing most people don't usually have in Unity catalog is descriptive metadata. And so this is a really good forcing function to be like if you want to you know automate your job or automate part of your job then provide more descriptive metadata and then it'll get easier for you to automate your job. All right. So, this is pretty much it, right?
I've got a name of a tool. I've got a description. I've got a schema because I don't want this to just be like random bytes, right? I want this to actually be a typed response that can then be parsed and read very easily by the M because it understands patterns very well. Um, and we're returning this directly. Yay. That's awesome. So, if I want to go list cataloges, right, we've done all the same thing over here. We've described what it is that we want to actually call on our list catalog and we're taking a max results option.
Option is one. Our default is one. This is pretty basic stuff right here. Um, so this one returns an iterator over here. When we call workspace client catalogs list, that used to be over here. So, ignore this line. So, we're limiting it to a max number of results. we're trying this and if we fail, we're going to return a tool exception so that um our agent actually knows that we failed the tool call versus us just being like this was awesome.
It worked out really well and then kind of moving on. So it needs to understand what it is that occurred. Okay, so speeding up automated data pipeline. Um this part um if you want to go read along on your phone or anything else like that, hopefully this is big enough to kind of show up uh for the QR code. So just leave it up for just a sec. Um, and while I leave that up, we'll just talk about what it is that we are doing.
So, uh, pointer, I don't know where that is. There it is. Okay. So, what did we need? We needed access to Unity catalog. Cool. We got that. It's also part of our list cataloges request and response. So, tools that do the following are what we need. So, if people have ever worked with product, product managers tell people about what things should do. Um, engineers also come up with things to do. Um, for us, we have to think about how was it that we explain to an entity.
I'm just going to call it an entity because I don't know what else to call it. But how do I explain to, you know, a foundational model that I want it to do a series of things? I'm not going to tell it how to do a series of things. I'm just going to give it options to try and figure it out on its own. It's much more intuitive than we are. So, you be like, here's all these cool APIs. And it's like, I got it. I found a cool way through this.
And so for this example of creating like a typical data pipeline, usually you have a source. You know where your source exists and you know the schema of your source and other kind of descriptive metadata. So you can then go and create transformations on that source data. Can join it with other things to accomplish other things. Um so you need a catalog. So we have list cataloges. Check. We need to be able to list the schemas of a catalog.
Cool. We need to be able to list the tables of a schema in a catalog. Um I was very explicit over here where it's like we're essentially just kind of taking steps down a tree and so everything that we're doing we're providing essentially similar context and similar descriptive metadata. I know it's boring like this is like it's very basic stuff but catalog schemas tables and then if I want to find out more about a table I want to go get um just a gigantic amount of detailed information.
So I want to understand what are the table properties. I want to understand understand actual comments on columns. I want to understand what it is a table's purpose is. So if you think about like discovering things, um, our LLMs are discovering what it is that we intend to do with something based on the metadata that we provide. So if we don't provide anything in Unity catalog, then we actually have to go and read the tables to summarize what it is that the table actually does, which is a lot more expensive than just using metadata to accomplish the same thing.
So moving on, we're going to continue building up these tools. Um, these are also part of like the um they are part of the Jupyter notebook. Um, and they also work which is cool and I'll show a video at the very end as well. So, we've got list schema request all the same stuff. Um, name of the parent catalog, you need a catalog containing a schema, max results. Uh, the important thing with a lot of these is that because things are pattern matching, don't just keep on changing like the name of descriptions.
If you're going to use like things like max results, continue to call it the same thing. Contextually, it's a lot easier for the LM to remember if it's exactly the same thing. It's going to control the same process. That's easy. So, list schemas returns a list of schema names if they exist from the associated Unity catalog name. Takes a schema request. This stuff is all rinse and repeat. So, again, most of the stuff that we do is boilerplate.
And so, we can automate a lot of our boilerplate by creating tools that then do those things for us as well. Um, so schema takes a workplace client schema list, great catalog name, including this stuff. Yay. That's awesome. So we're going to go now return a response. This stuff's really I'm just going to kind of power through this part. So list tables, same thing, right? What is it that we're doing? Well, for us to go and find a table in a catalog, we need a catalog name.
We need a schema name. We also need max results. Again, we catalog schema table. Yay. All the same kind of, you know, basic stuff. Like if you think about it and you're creating a notebook or you're creating anything else like you essentially go back to the same kind of six properties that we looked at when we were looking at like the generic compositable um data application we essentially just have a pointer to where data is and then we have a pointer to where data is going and that part's pretty basic.
So, yay, all this stuff. So, workspace client, bunch of different options, uh, this stuff's all very well documented, so I'm not going to go into it as well, but um, if you want to reduce the amount of data that comes back from the SDK, you can do that. So, you can essentially trim the amount of context that the OLM needs to retain in memory. Um, and that's pretty much that. So, these have been fairly simple up until now.
Now, we're introducing what no one can probably see in the back. let us describe it. Um, so we've got table column. I want to be able to describe to the LLM the notion of a column like what is a column in a table. It understands it already but I want to give it a little bit more metadata. So in this case I have a column. It's got a name which is a string. It's also got a SQL type so that I can be essentially type safe and think about what it is that you know what it is that this column is.
Is it just a string? Is it an integer? Is it a time stamp? Is it a time stamp with nanos? Like there's a lot of different things that things can be and so the more contextually relevant we are the easier it is for our applications to write themselves. Um so I've got a table column with a bunch of information. I've got comments. Yay. Is it nullable or not? So then we can say I only want to take a look at fields that are you know say you're creating a machine learning pipeline.
Anything else like that? I only care about you know things that can't be nullable. Um that's awesome. We can read all that stuff from the uh table uh API as well. So table request, table of response, all boilerplate, fun information about the columns, comments, everything else like that. Again, all we're doing is saying, how do I use this thing? What does it do? How does it represent something that I want from the machine?
So tool get table details provides metadata about a table within Unity catalog. The details include t table name, comment, description, columns, cluster or partition information. Yay, it's kind of like that sing along with the bouncing little thing. Um, so get table details. All this stuff is pretty much rinse and repeat. We've seen this before. Uh, this is essentially the only kind of difficult uh function over here.
So we're getting information about our table. We're bringing all sorts of metadata with us. Um, this will come in about maybe like 2 megabytes for a table. So there's a ton of information and you have to trim that information down. It all depends on how you're actually tuning um your tools themselves. But in this case um we're finally using this column table column over here. So we're saying I want to return table detail responses that has a full table name.
It's got a comment on the table. Then for the columns it takes the actual named table column. So type safety is great because type safety enables our tools to be able to have additional context within just like the response. So in a lot of cases if you're just returning like a blob of text there's no real kind of structure to it. And so again, LMS enjoy pattern matching and they enjoy structure. So the more structure we can provide, the better everything else is.
Okay, fruits of our labor. All right, I'm almost done anyways. Three minutes left. So this video is going to run for two minutes. Um, so you're a senior principal engineer. You specialize in data intensive applications and stream processing at scale. Uh, I don't know how to pause this. I'm going to go back for a sec. Hold on. First time doing the video versus just like failing at doing a live demo. So I didn't explain this part yet.
So um given that an LLM is essentially everything trained on everything, we need to be able to prod uh provide context to that LLM. So system prompt is essentially saying like I want you to be this, right? If you've ever used like a midjourney or anything else like that, it's like, you know, it's like, you know, act like a graphic designer and go create me a logo like this. That's essentially setting the context so it has a subset of information that it actually can lean on.
And so you're a senior principal engineer. You specialize in data intensive applications and screen processing at scale. You prefer to have short conversations and your brief responses contain mostly code examples. Kind of like my slides. This is how I like to act. So I wanted to create something that acted a lot like me. So you're an expert in Python programming language. All code examples must utilize Python. Essentially all we're doing is like setting like some guardrails.
Like um if you just arbitrarily um create a system prompt and say just do whatever you want to. A lot of times you'll get back scoly. You might get some Go code back. Spark has go um with Spark connect but I don't want to use Go for this application. And so you have to tell it exactly what you want it to do so that you can kind of bound everything in that conversation. So then there's me. So I'm like, "Hey, I just joined the team.
Can you help me find some tables to work with?" And I'll play the video. Uh, also, this is important, too. Um, so it takes 7 seconds to do all of this stuff, and it costs me less than a cent uh to run everything here. So I'll help you explore available tables. Let's start by checking what cataloges are available, and how we can drill down to the schemas and tables. List cataloges. Great. Max results 10. Awesome. Cool.
The tool it found is list cataloges. That's not super difficult. It finds a development catalog in systems tables. Awesome. So, can you show me the first 20 schemas in the development catalog? It understands context from the last state. Goes, cool, that's interesting. So, I'll list the schemas in development catalog. Um, again, all it's doing is essentially adding arguments in two functions and it's getting results. So, I want to list schemas.
Here they are. Development. We've got this behavior events. We've got the CDP experimentation, Nike team, innovation, templates, bunch of other stuff. Can I also trim down a bunch of other things because this is from like uh Nike systems. So, thank you. Using the same parent catalog, can you show me the first 10 tables under the consumer behavior events? Same par parent catalog doesn't actually have a template variable, but it understands it contextually.
So, it says development consumer behavior consumer behavior events max results 10. Cool. I'm going to go list those tables. So, it figures out exactly what it needs to do. Finds that I have one table over here. And that's awesome. So, next I go, okay, cool. Thank you. Uh always be kind, right? So using the same parent catalog and current schema, can you find more details about inspect table? If you've used Python and you forget an F, you get something like this versus an actual injected template variable.
But don't worry because it understands. It's like, oh, you left a placeholder. I'm like, oh, thank you for that. Uh, in your request based on previous results, I can see there's only one table in the schema. Do you want to use it? Sure. Let's see what you recommend. I don't want to do anything. Um, so I'll get the details for this table from this catalog. Awesome. Get table details. And over here we just have that gigantic mess of data that I was talking about before.
So trimming this context window is important um because there is a lot of information that comes back. Um so if you give a ton of information to the LLM and you don't trim it down, you're going to essentially run out of tokens and it's going to lose memory essentially potentially like halfway through uh the context of the conversation you're having. Luckily in this case, it doesn't matter because it still works. Um, so show me how to use that response as a streaming data source in Pispark.
Um, so this goes source table. I did it. Uh, consumer uh, behavior blah blah blah. I'm trying to keep up with the video, which is really hard. So, um, so goes, it creates a checkpoint location, merges the schema, does all this other stuff. Tells me other things that I can do, which is nice. Uh, can we add simple transformations? I want to explicitly call out column names. So, it knows column names because we created actual structured responses.
So, now I can go through and it can create all my column names. And the nice thing too is that it's going to also normalize anything that's actually nested as well. Um, and I didn't have to say anything about that. And so, um, that's essentially it. So, create a couple tools, provide the tools to an LLM, um, and then start to automate your life. Um, so closing thoughts for that. So, data pipelines are not that hard to make, which we saw in the very beginning of the presentation.
Using config- driven software development engineering design practice uh, patterns takes us 90% of the way anyways. So then getting aic assistance can be amazing if like what we're doing is essentially like automating like the tedium, right? It's like I don't want to go into Unity catalog and keep on thinking about this stuff. So just tell me what I should be doing with that and then I'm going to go focus on transformations.
Um so context is short, right? So it's a tough memory is a tough nut to crack. Um do you want to keep things in short-term memory or do you want to actually start to put that into a vector database? That's there's a lot more conversations about this going on this week. So just keep that in mind for food for thought. So providing all the right tools can take a long time especially when you're thinking about kind of what level what hierarchy you want to work at.
Um so the one thing that this didn't go into was like if in the notebook environment I start giving access to the tool to go write I need to be able to say what's my current working directory. Um how do I go and create a new directory? How do I go and like you know essentially do other operations and actually run code on my behalf? Um if you've used cloud code or anything else before it will run things on your behalf. you'll say, "Go create a new branch from this repo and go do my work for me."
Um, it's not exactly that easy. Sometimes it can be, but um, when you're doing the tool calling yourself, it's a little bit more safe, right? Because you don't, you know, in other cases, you don't know exactly how it's working. So, if you have to work with like, you know, uh, corporate security or anything else like that, you can say, "Oh, but don't worry, you know, this is all bound to the context of the Jupyter notebooks that are running inside my Docker container.
It's using my bearer token to connect to Unity catalog. It's using my permissions to go do all that work. That's an easier conversation to have. Um, and then cheers. So long and thanks for all the fish. So, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.