Latest innovations in Genie Code: The Future of Agentic Data Work

Summary

Genie Code has evolved from a chatbot interface into a fully autonomous agent that plans and executes data and AI workflows across all of Databricks, with 75% of Databricks code now authored by Genie Code.
Unity Catalog Ontology integration gives Genie Code precise grounding in a user's actual tables and schemas, enabling accuracy in analytical work that generic coding agents cannot match.
The Genie ZeroOps capability monitors production pipelines in the background and automatically triages and fixes failures—such as schema typos—before the data engineering team even notices an issue.

Latest innovations in Genie Code: The Future of Agentic Data Work

AI coding assistants have changed how developers write code, but most stop at suggestions. Genie Code is an autonomous AI agent designed specifically for data teams. It doesn't just help you write code. It plans and executes full workflows, and it monitors your production systems in the background, triaging and fixing pipeline failures before your team even notices. In this video, we'll walk through the newest capabilities in Genie Code and how they change the way data work gets done. We'll cover: - What’s new in Genie Code - How it builds and maintains production data and AI workflows end to end, from pipeline creation to model serving observability - How Unity Catalog integration gives Genie Code the enterprise context, governance, and accuracy that general-purpose agents lack - Best practices for working with Genie Code as an autonomous partner, including Agent Skills and persistent memory
Talk By: Weston Hutchins, Director, Product Management, Databricks ;

Chapters

FAQs

How is Genie Code different from a general-purpose coding assistant?

Genie Code is purpose-built for data work on Databricks, with deep integration into Unity Catalog that grounds its outputs in a user's actual tables, schemas, and business context. Unlike generic coding agents, it can plan and execute full analytical workflows—from data ingestion to model serving observability—rather than just generating code snippets.

What does it mean that 75% of Databricks code is now authored by Genie Code?

This statistic, shared in this video, reflects how much of the code written by Databricks engineers and users is now generated or assisted by Genie Code rather than written entirely by hand. It signals a step-function improvement in the product's capability and adoption over the past three to six months.

What are Agent Skills in Genie Code?

Agent Skills are on-demand, reusable task templates that Genie Code can call to perform complex, multi-step operations in a consistent way across different projects. They allow teams to codify recurring data engineering and analytics workflows so that Genie Code can execute them reliably without re-deriving the approach each time.

How does Genie ZeroOps work and what can it fix automatically?

Genie ZeroOps runs in the background monitoring production pipelines for failures, and when it detects an issue it automatically triages the root cause and applies a fix without requiring human intervention. This video demonstrates a live example where a schema typo causes a pipeline failure that Genie ZeroOps catches and corrects on stage.

Full transcript

[00:07] Awesome. Thank you for the wonderful introduction. So, my name is Weston. This is also my fourth Data and AI Summit. Coming up on 4 years at Databricks. In
[00:19] fact, this was me about 3 years ago on the keynote stage giving the first ever demo of the Databricks Assistant, which is now Genie code. You can even
[00:32] see for those that have been long-time Databricks customers, this was our old notebook UI. It's evolved quite a bit since then. We had different logos, different names. We love changing brand names. That's just what we do.
[00:45] But this was fun. I really got to I was really excited. So, I've been around working on Assistant and now Genie code since its inception. And for those that haven't used it recently,
[00:59] it's undergone quite a bit of change even in the last 3 months. So, we published this blog post around March. Genie code this is introducing Genie code. Now, some folks were saying, well,
[01:11] it was Databricks Assistant, you just changed the name. Not actually. So, this was a pretty large product launch for us. We were unveiling a lot of new functionality at once. And the biggest thing for us was that we took Genie code
[01:24] from something that just worked as a chatbot interface to an autonomous agent that works across all of Databricks. So, for for me like the biggest thing that I have seen from customers that are using
[01:38] Genie code lately at least at this conference is just how much better it's gotten in the last 3 to 6 months. Like step function improvement mostly based on the harness work that we've been doing just to make Genie code work across all of Databricks. So, for you
[01:53] know, for what we just what we What we went out to do is design a tool specifically for data work. Generic coding agents work pretty well for generating code, but we wanted something that data scientists, machine
[02:05] learning engineers, data engineers, even business analysts can pick up and use inside of our product. We really wanted to set our goal of automating a lot of asset creation inside Databricks, which
[02:17] I'll show you in the demo. And then we also wanted something that's grounded in Unity Catalog data. In fact, I find I use all the tools. I use every LLM provider's coding harness out there. I love experimenting.
[02:30] I do find that when I reach for Genie code and Genie specifically, it's really when I want to do an analytical question. Something that's grounded in my tables and my catalog, understands the types of information that I'm trying to do. I will often pull up a notebook
[02:42] and just do a data science analysis using Genie code. And then also as part of this session, I'm going to give you a sneak preview of Genie Zero Ops, which we announced at the keynote. So it's part of the Genie family,
[02:54] um built on a lot of the Genie code platform that I'll show off. This is really our story for proactive monitoring and maintaining your your data assets and pipelines in production. Um so, as I said, like what have we what
[03:08] have we been working on for the last, you know, year or two, three years, but like more recently, how has the Databricks Assistant now Genie code changed? And for us it really is integrating all of these product surfaces. Um we have been working with
[03:22] all of the teams across Databricks and Databricks for those that don't know is a pretty large product. You can do a lot of things in it. We needed to do more than just be able to read information. We also had to be able to mutate these surfaces, write out information, understand how to create
[03:35] pipelines, work within their editor surface. So we'll see a dashboard demo later. But for me, this is really the power. We want to integrate every part of Databricks into Genie code. You can ask it to do something and it just
[03:47] works. And I just pulled these quotes from Reddit. You can find them pretty much all over the place, but people when we announced this and and I get this feedback every single week. It's just hey, this has
[03:59] gotten so much better. So, if you haven't used it, please try it out. I often tell folks, they ask like what can I do? I said, just ask Genie code. It will tell you what what it can do and what it can't do. Um and
[04:11] being the product manager for this product for this area, it's like been really exciting to see you know, Genie usage has grown 10x over the last year. Uh it's used by 90% of Databricks
[04:23] customers, which is probably why when the they asked who has not used Genie code, there weren't that many people that raised their hand. Everyone is using this product. Uh and we reached in in a great
[04:35] milestone about 75% of the code authored in Databricks is now through Genie code rather than humans. Um so, I want to talk a little bit about you know, what makes Genie code special. And for me, I mentioned I really reach
[04:49] for it for analytical questions. This is all about context and that's been a lot of the theme of this conference in general, but I always tell folks like Genie code is data smart. It understands the tables. Um and you know, as we
[05:01] mentioned in the intro, we've been benchmarking this on a bunch of analytical questions. Stuff our users are actually using. Even internally to Databricks, we look at the questions that our data team is asking and that our PMs are asking and we put that into
[05:14] our benchmarks and we run it and we have found that it just delivers superior accuracy, mainly because of all the context that it has. Um and a big thing is that Genie code, just like Genie 1 and Genie spaces, this
[05:27] builds on the same platform. So, you get all of the great features from Genie 1 inside Genie code through the ontology layer. Um now, what is the ontology layer? What makes this special? Um
[05:40] the reality is we are creating knowledge snippets from all of the data assets and the usage information that you have. So, we look at tables and metric views and queries and notebooks um and your connected apps and we pull
[05:53] out definitions like, "Oh, fiscal year starts in February instead of in January." Um or when I look at, you know, this particular region, these are the countries that I need to include in
[06:05] it, which can be different depending on whatever your corporation is. Um we've even started to connect it via MCPs to uh things like GitHub and Slack and Jira um and pull those in and create these
[06:17] little stores of information that power Genie. Um this is all done automatically for you um and I like to sort of show this this diagram right here where uh
[06:29] it's built on top of Unity Catalog semantics. So, this is the semantic layer is something that you all control. Um you can go and add in your own metrics definitions that power dashboards. You uh can create glossary
[06:42] of business definitions. This was just announced. You have domains. That's also a new feature uh for business-specific terms and organization of data assets. The anthology layer happens on top of that.
[06:54] Uh this happens uh it it is automatically um looking over your your semantics, uh pulling out these knowledge snippets, feeding that into Genie as part of the query operation and really just makes this thing much more powerful.
[07:06] Um and like I said, the more you use it, the better it gets. Uh so, if you are using Unity Catalog, I always tell folks, go in there, add all of your AI comments into uh every table that you
[07:19] have, add descriptions. Uh it really does make the product that much better. Uh and with that, I'm going to jump into a demo. Now, full disclosure, when I was demoing this earlier, it did
[07:32] break. So, we're going to pray for the demo gods. If not, I do have a backup video and we will see what happens. Um but the first thing I wanted to show, actually I'll I'll start here from from the home page. Um well, actually, I'll
[07:45] even go back to the real home page. We we've added two two entry points. One is uh the Genie code sidebar that you're familiar with, and we also now have this code button that you can ask uh in the middle. But, we recently launched this
[07:59] full-page experience, and I'm going to kick this off, and I'll talk through it in a second. Let me blow this up one more. Um but I've been doing a lot of demos of Genie
[08:11] code. Genie code's getting pretty good at understanding what it wants to do. I thought it would be fun. I actually came up with this last night. I was like, "Wouldn't it be fun if Genie code came up with its own demo for today's session?"
[08:23] Um so, I asked it, I'll scroll back up, I asked it, "I'm giving a talk at Data and AI Summit. I want to showcase your functionality, your abilities. I want to use this table." This is just the standard Wonder Brick sample table
[08:36] that's like a Airbnb-style booking platform. Uh what are you going to come up with? Um so, let's like give it a bit more room here. Uh and then by the way, every time I've run through this, it has come up with something different. So, this is a pure
[08:48] choose-your-own-adventure. Um we will see what works and what doesn't. Uh all right, so we have data discovery, SQL execution and analytics, inline visualizations, render chart,
[09:00] uh multi-table joins, AI functions. Okay, that's a new one. I haven't seen that one yet, but let's see. Let's go with it. Uh and then we're going to create a dashboard. Uh and then it tells me all the tables
[09:12] that are available. So, it's already found those tables. I think what I'll do, I do want it to show off creating a notebook. So, I'll say um uh instead of step four, let's create a notebook.
[09:25] Um and so, we'll give it these instructions, and then we will we will see what comes out. Um so, uh once I kick this off, it's updated the plan.
[09:37] Let's go ahead and start executing the demo. And this really is live code. This is using uh one of our um demo surfaces, but this is a real data um do it live
[09:50] here. Um so, I want to talk through some of the interfaces. You'll notice that I went into full page mode. Why did I do that? Well, we want Genie code to be the command center for all your threads. Today, it
[10:02] works really well if you have it as a side companion inside of a notebook or inside of a dashboard, but um I will say, "Please execute until
[10:14] you finish the notebook." Sometimes these things can take a little bit of time, so I'm just going to try to get it ahead of us. Um but we want it to be your command center. We want it to be the place that you go to kick off all sorts of data
[10:26] asset creation, from pipelines to dashboards to AI functions to model serving endpoints. You can use it from uh this panel. So, we do expect the full page experience to increasingly become the default. The other thing is um we uh
[10:41] have created a plan mode. So, if you are doing a long demo like this, um you can ask Genie code to um create a a plan for you, and we will verify that plan before you go and execute it. This is especially useful if you're doing a
[10:53] bunch of analysis or creating data learning uh machine learning models. You want to make sure you get it right before you kick off um a timely and expensive operation. So, um it will go through the plan. It'll start walking through what's been completed and what
[11:05] hasn't. Now, um uh I'm going to go and show off what it actually started to do. Well, let's actually just take a look. So, the first thing is that um and you can actually
[11:19] click to expand and see the thoughts about what happened here. But it went and found the right tables. Now, I know this was a trivial example. I picked Wonder Bricks, our sample data set, but I've done this across many of our
[11:32] internal data sets for looking at telemetry of the product. It just knows what's being used across your organization. It's the best way I can explain it. I ask for finance tables, I ask for usage tables,
[11:44] I've asked for logs information. It always surfaces the right tables because it knows the ones that are frequently used by other people in your organization. So for this, it found the Wonder Brick's table.
[11:56] It starts analyzing it as well. So if I open up some of these steps, the really cool thing about Genie code that not many people know is that this is a notebook behind the
[12:09] scenes. If you've used any chat product, you know if you squint a little bit, they all sort of look like notebooks. They have cells, they have responses, they are now adding code execution. It
[12:23] turns out Data Bricks, we have a fantastic notebook product. It's what I what I've been working on for the last four years here. And so we've brought that in to do all of the underpinnings of Genie code. It does syntax colorization, it does
[12:37] code execution in Python and SQL. It does state persistence across browser tabs. Eventually we were looking at adding collaboration so like comments that are available in notebooks. You want to add it to Genie code as well and the ability
[12:49] to share these. But these are persistent assets. They live inside Data Bricks. You can go back and visit these old threads. At the end of the day, these are just notebook files. Including the fact that you can go and
[13:01] you know, filter these just like you would in any other notebook. So the first thing that Genie code does is it samples the data and understands the columns. So it takes a look at a few of the columns. It understands what strings
[13:13] are in there, what data formats are in, and it starts planning it. This is and it uses the same visualization and rendering output that you would see in a notebook. So you can actually have
[13:25] it go execute your favorite open source HTML library for a gene or you can use the built-in data bricks visualization as well. So, right here what it's showing is that
[13:37] it's creating visualizations. It's gone through this. Now it's ready to go and create a notebook. Which you'll see it also gives me the ability to select the asset. So, I'm going to go ahead and like move
[13:51] this out so I have it on my left-hand side. Um and we'll accept all of the edits here. But what this also like really makes it easy. You can go through, you can review the lines of code that were edited. You'll notice that it created it, but it
[14:04] hasn't run it yet. So, run this notebook first. Um this is really cool. Like I have actually found myself using Genie code to create clusters for me or execute
[14:16] notebooks or well, it hasn't started up my my code cells yet. So, the other thing it's very good at is recovering from mistakes. So, often times you it'll try to do something, it will fail, and then it will try again and succeed. So, in this
[14:29] particular instance I had not started up my compute yet. It figured that out. If you go and look at my history of some of the chats, I said, "Hey, start this cluster up." Get it pre-warmed for the demo. But
[14:41] you'll notice that it went and created about five or six insights for me using this using our samples data and our property analysis. So, while this is executing, I did want to
[14:53] share a few other things that we have added to Genie code. Um Let me actually make sure that we just continue with this after it's done. So,
[15:05] continue with the demo. Well, actually we'll see if it prompts me with something. It is fully operational like a battle star. Okay.
[15:22] Continue with the AI functions. We're going to go see it's going to create an AI function. Um, while it's doing that, I wanted to show off a few things. So, um, what we have done inside, uh, inside Genie code, let me actually pull this back and I'll go into this screen over
[15:36] here, is, uh, a few different personalization and extensibility options. I'm going to skip this actions one for a second. One is we've come pre-configured with, uh, about eight MCP servers with many more
[15:50] in beta and preview that are coming on the way. You can also come in here and add your own custom server, one that's hosted in Unity catalog. You can host a Genie space in here, but effectively any endpoint that you have, um, can be used
[16:03] as an MCP server. I have found by far the most useful ones to be, uh, GitHub and Jira for some of my interactions. Um, I will actually take screenshots of
[16:15] the Databricks product surface and it will go and look at my source code repository, where those buttons are instrumented. I will ask it to figure out the click-through rate or build a funnel for my end users to see where I'm users are dropping off in the Genie code
[16:27] flow, and it knows where my logs are stored inside Databricks and it will stitch that together into an super powerful for these end-to-end analysis. Um, and then the other thing is we've brought together instructions and skills. So, for instructions, these, uh,
[16:42] I always tell folks like instructions are sort of your always-on commands that you give. So, in this particular example, I seeded it with, uh, brand palette that I use for the Wonder Bricks demo. It's something that's just a little bit different. Um, and if you go
[16:55] through the Genie code thread, you will notice that, and maybe we'll see it in the dashboard creation step, that it uses the brand palette from, uh, these instructions in there. The other thing is that this is a field engineering workspace. They have added a
[17:07] bunch of instructions for the workspace on policies and how to set these things up responsibly and like when to delete things. Um, so, Genie code will look through these instructions and admins
[17:20] can configure uh Genie code to respect style guidelines and other best practices from inside your organization. And then the other thing is skills. Skills are uh these on-demand tasks.
[17:32] Uh like I have a skill for synthetic data generation that I use to create Spark data frames. Um it has uh it's just markdown files. You can import them. We're working on syncing skills from your local machine up into the Genie code space. But the
[17:45] same types of extensibility you would have in other products we bring to the Genie code platform as well. All right, so let me go back to the demo and we'll see where we are at. Yes,
[17:57] everything looks good. It's now created an AI summary uh an AI asset. Um a analyze AI sentiment functions correctly. Okay. We'll show that off and then uh it's added to the notebook. But while
[18:09] it's doing that, let's go ahead and kick off the dashboard creation cuz this step typically does take the longest. Um and we'll see how that works. Um and let's scroll down to see if it added
[18:22] my AI sentiment. Okay, yeah. So, what this did was we have a bunch of property reviews for Wonder Bricks. It was like, "Hey, you might want to actually run this through an AI classification to get sentiment analysis
[18:34] for the reviews that your users left on your uh on your um bed and breakfast place." So, unfortunately, uh this very first one is not a good negative sentiment, but there are some positive ones in here as well. But just shows you that you can use a lot of the
[18:47] tools available out of the box from AI functions. Genie code knows about those. Uh we prime the experience with it. Um and so, it's just really powerful. Like the whole the whole goal of Genie code is that you can come in here. You don't
[18:59] really need to go and configure any skills if you don't want to. Um those are for like really personal uh actions. Everything is baked in. We are constantly adding skills to our repository of just how to take action
[19:11] across different places. And um and yeah, this is like the the full end-to-end demo. Let's see if it starts creating these dashboards. Um the other thing I wanted to call out
[19:23] uh cuz it it comes up a lot when I talk to customers is um this approval step. So, we added something called auto approve, and auto approve means that rather than surfacing a giant would you
[19:36] like to allow this operation for every time we prompt someone in case of a you know, a destructive operation, we um we actually use an LLM to classify the intent behind it. And if we deem it as a
[19:49] safe action, maybe you're just reading SQL data, you don't need to approve every one of those. Um we will only pop up an approval step when we reach something that is potentially destructive, something that you want to actually review before you hit okay.
[20:01] Maybe it is, you know, like deleting a job. We will definitely want to prompt on that before we do it. And we have found that um it is actually way better for customers that are for people that use it because the most important ones
[20:13] are surfaced to them un- unlike a lot of noise where you get like a 100-line Python file that you need to go and approve. Um okay, so we're continuing with the dashboard creation. Um so, that's one. And then the other thing I also like to
[20:26] show off as well is if you click this button, um I always recommend that people seed Genie code with extra references whenever they can. So, you can uh at
[20:38] reference notebooks, you can at at reference other tables, queries. It'll list sort of most recently used tables that you've used in here. Um you don't always need to do this. I find it does a really good job on its own. But for me,
[20:51] one of the things I I find myself doing quite a bit is pasting in the URL of a dashboard uh when I'm doing an analysis and say, "Hey, open up a notebook. Here's the dashboard that I want to use the source queries from. Pull those out, create a notebook, um and then start to
[21:04] do some deep-level analysis." And that just gives Genie code a little bit more of a template to work off from. Um and eventually those get uh pulled into the ontology layer as well. The more you sort of reference these existing assets.
[21:17] Um Okay, it's creating local metric views. I'm going to come back to this demo cuz sometimes the dashboard does take a while to create and uh I'm going to go through a few more slides and then we're going to come back to it and then I'm
[21:29] going to demo Genie zero apps. Okay. So, that was like a whirlwind tour of Genie code. Um I want to show a few other things that I didn't demo, uh
[21:41] stuff that you can try out after this. Uh we showed exploratory data analysis. So, this is just doing some EDA patterns inside of an interactive notebook. Um machine learning, this is something we talked a lot about Genie code for ML
[21:54] today. Uh Genie code is really good at training models. Uh there's a demo I do where I train an XGBoost model over um the price uh for the host listings. It's amazing at that. We've integrated it
[22:06] with MLflow. We've more recently integrated it with some of the model serving endpoints that will go and actively debug uh if there are failures and traces. Um data engineering, this is something Genie code is excellent at. Um it
[22:20] creates Spark declarative pipelines, streaming tables, materialized views. Um and more importantly, can auto add its own data quality rules based off of um uh the types of data that it finds and the the checks that you'll want to see
[22:32] as a production engineering pipeline. Uh jobs is something that just went to public preview. Genie code for jobs just went to public preview recently. So, you'll probably have to opt into this, but we now have full job support in
[22:44] Genie code. You can ask it to schedule notebooks, Python files. Uh you can edit and configure different jobs. Uh and then you can also diagnose uh any jobs that failed directly um through the
[22:56] Genie code side panel. Um and dashboards, we'll see. Uh it's uh it's chugging away, but I did want to mention two other things with dashboards. Uh, the first is that you can now import Tableau and Power BI
[23:09] dashboards using this built-in import, uh, BI tool that we have available. And, uh, as part of private preview, this will be available soon. Uh, we are giving folks the ability to offer relationship models in some of the
[23:21] semantic models through Genie code as well. Um, it already works with Genie spaces. Um, but that's, uh, something we're going to be adding to dashboards. All right, so coming soon. This is sort of a sneak preview of some of the
[23:33] features that are going to be rolling out in, honestly, the next couple weeks. So, you won't have to wait too long for this. Um, the very first one is parallel agent execution. Um, we want to be able to have Genie code kick off multiple
[23:46] threads at once. Right now, you're really limited to a single thread at a time. Uh, no more. You will be able to spin out multiple threads. They each run in their own compute layer, um, so they don't conflict with each other.
[23:58] Um, and then the thing that I'm most excited about is Genie code scheduled tasks. Uh, so, how many of you walk around carrying laptops uh, trying to make sure that you don't close your lid while your agent is
[24:11] executing something. Um, I do it all the time with Genie code. It's frustrating. Um, we are moving to, uh, background agent execution. And as part of that, we are also giving folks the ability to We're going to roll out this ability to schedule tasks. Um, so you can set up
[24:25] nightly analysis, uh, to send you a morning, uh, summary of uh, any data that you want to analyzed and sent to your inbox in the morning. Um, you can have a morning Jira report that summarizes any open tickets for you. Um,
[24:37] this is the same sort of scheduled task feature that you will see in Genie 1 as well. Um, it will also respond to triggered events. Uh, so we will be, uh, shortly exposing, um, some of this functionality through APIs that you can
[24:50] use to go and trigger these uh, respond to events and then trigger workflows and pipelines off of that. Um, okay, some quick tips and tricks. This is uh, um, nothing too crazy here,
[25:02] but I have found that um, I like to give some advice for folks that are using Genie code for the first time to be a little bit more productive. Um, the the number one advice I can give is that use declarative automation bundles whenever
[25:14] you can. Um, for those that haven't used DABs, this is our YAML syntax that I describes how to deploy assets inside Databricks. The biggest benefit to using Genie code with this is you have reproducible artifacts.
[25:26] Um, so you can check these into get, you can version control them. Uh, Genie code operates really, really well off of those. You can actually have Genie code generate DABs for you. Um, so much so that, you know, around the time that we launched Genie code, which was around
[25:38] here, we have seen DABs DAB usage grow quite a bit over the last couple months. So, I expect this trend to continue as more folks use Genie code with DABs. The second thing is
[25:50] personalize your Genie code, set up your MCP connections. It gets so much more powerful when you do this. Um, these MCP connectors are all managed and governed through Unity Catalog. So, if you are an admin, you can go and
[26:03] configure which ones are enabled for your workspace and which ones uh, require admin approval. Um, and you can even add your own custom MCP points, but I have found that the the amount of functionality that Genie code can get
[26:15] with these MCP servers goes up quite a bit. Um, and then of course I talked about personalization through instructions and skills. Super useful. Set up a workspace policy where everyone gets the same instruction and skills. And we have some other nice features
[26:27] rolling out about how to manage those that um, you know, I can talk about after this this session. And then of course I I mentioned this while I was demoing it, but um, reference your past analysis. So, uh, try to go and find
[26:39] some notebooks uh, that have uh, other folks on your team have done before. Send those into Genie code. Um, it just makes the product that much better. All right. Um, Genie zero ops. So, I have about 13
[26:52] minutes left. Uh I might not need all that time, but I want to give, you know, a a bit of a tease about Genie ZeroOps, which uh we announced at Data and AI Summit and is going to private preview um shortly. But, really Genie ZeroOps
[27:07] came out of this vision of taking Genie code, which is really useful for new asset generation and code generation, and applying the lens of a typical data engineer who spends most of their time
[27:20] managing and fixing already existing pipelines rather than creating new ones. And so, they are built on the same platform. Um they use Genie code under the hood. Um and we were when we were talking to our data engineers, we found
[27:33] out 50% of their time was spent on maintenance. And it doesn't matter if you can code 10x faster, it's mainly about fixing pipeline problems. Uh and so, that's
[27:46] where Genie ZeroOps really comes in. Um it's designed to be this automation and monitoring layer that sits on top of your data assets. Uh we're starting with Lake Flow pipelines, but eventually we are going to move to uh other types of
[27:59] features in the ML space like feature engineering, retraining models with drift detection. And you can imagine this being applied to other uh platforms inside Databricks. And the real promise of Genie ZeroOps
[28:11] is, you know, filter the signal from the noise, don't get paged at 2:00 in the morning, use Genie ZeroOps to manage that inbound triage triage these issues and only surface the ones of highest importance. Obviously, automate the
[28:24] toil. So, like, apply the fixes automatically, um and do it in a safe way. So, do it in a sandbox, safe validation, never touch production code. Uh and then finally, we want to make this really adaptable and personalizable
[28:36] to you. So, keep on making sure that it does um that it learns from from past interactions and stores those in memory. All right. With that, I'm going to jump over to a demo, but first let's see. Hey, look at
[28:49] that. All right, we have our dashboard. It did create it. It used my Wonder Bricks brand palette. Let's accept these. I can probably even ask it to to publish it. Publish this.
[29:01] And uh we'll see if it does that, but sometimes dashboard creation does take a while. But yeah, I have found I use this all the time to just create dashboards. We have little widgets here for working through the flow. We have a dashboard.
[29:14] Drum roll, please. All right, and as they load we now have a published dashboard created by Genie code directly on stage
[29:26] from its own mind, the demo that itself created. I love it. Um all right, I'm going to jump over to zero ops real quick. Um all right, so zero ops. Once again, this is the same interface as Genie code, but
[29:39] now you have this little zero ops inbox off to the left over here. Um and the first thing I will do is set up monitoring, so enable monitoring on this product. Uh let's go.
[29:52] There we go. Um and it'll ask so like, "Hey, I'll help you set up this configuration. Do you want to do jobs and schemas owned by me, all jobs owned by me, or all schemas owned by me?" Let's just This is a pretty small workspace. What it's doing
[30:05] right now is it's going and scanning all of your jobs and putting this in a YAML file so that you can actually configure Genie zero ops. It's more than just a you know a interactive mechanism. We want to be able to remember these and store these in a source controlled
[30:18] version so you can understand the rules. Um but I come back here and you'll notice that it's analyzing my workspace. Let me zoom in a little bit more so you can see it. Um for this workspace it's like I said, it's pretty small. I have
[30:31] uh nine monitored jobs and pipelines and what it's looking for is anything that has failed. Um and so in In particular one um it's noticed that one of my
[30:44] pipelines has been failing. It was reported. Let's go ahead and open it. And uh this is really the power of Zero Ops. So, if you were to go and replay this before
[30:56] Zero Ops, what I would do as a data engineer is I would come in here. I would have to scroll down. I'm sorry for the zoomed-in cramped UI, but I have to scroll down and see that my pipeline, which was executing green before,
[31:09] is now not. Um I'd have to go and click all the way down here. Hang on. I lost my focus. Well, my scroll stopped working. Let me
[31:21] try it. Let me just refresh this. Uh I There we go. Um I would have to go and click on here, click on a recent run, scroll all the
[31:33] way to the problem in my pipeline, um see a bunch of errors, and realize and then I'd have to go and debug the fix. Um this happens to be a pretty trivial example. Um for instance, my developer went and
[31:47] accidentally changed one of the properties in the table, the schema is out of date. Um so, what did Genie Zero Ops do? Well, it went and found the typo. It corrected it based off of the actual schema in the table. It uh
[32:00] created a PR for me. And I think most importantly, it validated it in a sandbox. Um And so, I can ask like what is the sandbox? Uh and really this is our This is
[32:12] leveraging a lot of the built-in features from Spark and from Unity Catalog. So, what we do as part of the Genie Zero Ops principles is that we will never touch your production data sets. We always want you to go through a proper CICD pipeline.
[32:25] Um and so, but we do need a place to run and execute code to test these fixes while they're being worked on in the background. And so, what we do is we create shallow clones of your data sets. We validate them. And then when you
[32:37] approve the changes and you can even view the results of those the operations on those shallow clones, it can go and push into your environment. So, I can come up here, I can create a pull request. This is just a chatbot like
[32:49] Genie code. You can chat with it like anything else. Um and it will go and create a PR, go and fix, and deploy that into your pipeline. Um for a more, you know, interesting example, imagine you had a workspace
[33:01] that was like way more complicated. You wake up with 94 jobs that are monitoring and a bunch of things that uh that you want to fix. Um let me just filter to I'll filter to the highest severity ones
[33:13] and I'll filter to just the open ones since I think we've resolved a few of these. Um but in this particular example, we've set up this pipeline. Uh it's a Wonder Brick's pipeline that is looking um at revenue tracking for
[33:26] bookings. And my goodness, something happened. Uh 46% of our revenue went down, which is a bad thing. Now, it's either a a real application problem, it's a real behavior change problem with
[33:39] our customers, or most likely it's a data pipeline problem. Um and the most interesting thing about this is that the pipeline's actually green. The pipeline is running just fine, but there's a
[33:51] logic error. Um and so anomaly detection caught this, and this is one of the built-in features of Zero Ops. We enable data quality monitoring. It will go and figure out the signals and figure out the tables that are causing things that
[34:04] do not look correct, um and automatically surface those results. And the coolest thing about this is um you can inline view all the lineage. You can see all of the downstream and upstream tables that are affected by
[34:16] this. Um you can view a root cause analysis, um and then you can see the proposed fix. Uh and so for this particular example, um we were actually dropping tables
[34:29] through this new website feature called last-minute bookings. We didn't capture that. Our data pipeline was failing and so everyone that got last minute bookings was just dropping out of the pipeline. So revenue did not drop, we just weren't catching this new use case.
[34:42] Um but my favorite part is that Genie code not only proposes a fix but recommends a test to apply along with it. So we have a new unit test to add to
[34:54] our test fixtures. Um it did that for you automatically. It validated it. You can you know, ask it follow-up questions um like what is downstream? You can ask it to add more tests to your system. But
[35:06] the idea here is that this shouldn't happen again. We've caught it. We will obviously hit other issues in the future, but for the most part, we're going to be constantly fixing those and then validating them so that you don't have to worry about this stuff in the
[35:18] future. Uh great. That is Genie Zero Ops. And yes, you can clap. Thank you. >>
[35:36] >> Uh I'll leave this up, but just to recap, you know, uh I really want to hear feedback from you all on how you're using Genie code. Once again, it's been pretty humbling to walk around the conference and talk to everybody that's been using it and just
[35:49] raves about how much more productive they are with it. I talked to a customer yesterday. He's like, I was going through a POC, you know, with Genie code and it was the mother of all POCs. We budgeted 3 weeks
[36:01] and my team had a demo working in about 2 hours using Genie code, sucking in all of our data from our pipeline, building Spark declarative pipelines, and seeing the output. He was absolutely floored. I love hearing stories like that. Um please come and grab me, come and talk
[36:13] to me about that. Here's the invitation to join the Zero Ops private preview preview. You can read a bit more about it on our blog. Um and then yeah, that's Genie code plus Genie Zero Ops now um as
[36:25] the of the Genie family. Uh thank you so much and I'll be up across over the stage. Um, I'll probably take them outside just because we're running out of time for any additional questions. Yeah, yeah, I'll go back to the QR.
[36:40] Whoop. There you go. All right, thank you all so much. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.