Agentic Data Engineering with Genie Code and Genie ZeroOps

Summary

Genie Code is an autonomous AI partner for data engineering that understands Unity Catalog metadata, semantics, and governance policies, enabling it to take accurate, context-aware actions on Spark Declarative Pipelines and Lakeflow Jobs rather than producing syntactically correct but semantically wrong code.
Genie ZeroOps monitors data pipelines and assets in production, autonomously detecting, diagnosing, and proposing fixes the moment failures happen, with every agent action running sandboxed so production environments stay untouched until a human approves.
This video demonstrates how Genie ZeroOps catches silent data-quality regressions that produce wrong results without triggering an explicit job failure — a failure mode that traditional job monitoring misses entirely.

Agentic Data Engineering with Genie Code and Genie ZeroOps

Genie Code is an autonomous AI partner for data work, with depth across the data engineering lifecycle. We show how builders use it to iterate on Spark Declarative Pipelines and Lakeflow Jobs, accelerating authoring. It knows your data, grounded in Unity Catalog metadata, semantics, and governance policies. This grounding helps Genie Code take accurate and context-aware actions. Beyond authoring, Genie Code keeps workflows healthy, proactively monitoring and maintaining data and AI in production. We discuss how it autonomously detects, diagnoses, and proposes fixes the moment failures happen, with every agent running sandboxed, so production stays untouched.
Talk By: Gal Oshri, Sr. Staff Product Manager, Databricks ; Lennart Kats, Principal Engineer, Databricks ;

Chapters

FAQs

What is Genie Code and how is it different from a general-purpose coding agent?

Genie Code is a data-engineering-specific AI agent grounded in Unity Catalog metadata, semantics, and governance policies, giving it accurate context about your actual data rather than just your source code. General-purpose coding agents can write syntactically correct code that produces wrong results because they lack this understanding of your catalog and data definitions.

What is Genie ZeroOps and what problems does it solve?

Genie ZeroOps is a monitoring and maintenance capability that autonomously detects failures, diagnoses root causes, proposes code fixes, and raises pull requests for human review, all within a sandboxed environment. It addresses the fact that data engineers spend a large portion of their time maintaining pipelines after they are deployed to production.

How does Genie Code help teams build Spark Declarative Pipelines faster?

Genie Code can iterate on pipeline definitions autonomously using an auto-approve mode, pulling in Unity Catalog data sources via Genie Ontology and generating medallion pipeline code grounded in your catalog. Once the pipeline is ready, it can be handed off to Git and deployed to production through standard workflows.

What happens when Genie ZeroOps detects a silent data-quality regression?

When Genie ZeroOps detects that pipeline outputs have changed in a way that indicates a data-quality regression, even without an explicit job failure, it raises an incident in the ZeroOps inbox with its diagnosis and a proposed fix. The fix is validated in a sandboxed environment before being presented to the engineer for approval.

Full transcript

[00:07] Thank you, Alison. Thank you all so much. So, let's get started. So, we have a little bit of fine print here. Uh, let me make sure the setup is working correctly.
[00:21] Uh, quick housekeeping note. If you've been with us, you've seen this a couple of times. We do not actually possess a crystal ball. So, there might be forward-looking statements here. Um, please don't make purchasing decisions based on these statements. Uh, these are
[00:33] all subject to change. Something I care more about is we have an app. Uh, please fill in the survey. We can use your input.
[00:47] With that, let's get started. So, you've seen it. AI agents have completely changed how we do software engineering. >>
[00:59] conference, I don't think this is a a shocker or a big surprise for you. So actually in this talk, what we want to do is is mostly show demos. We're going to show Genie code life in action. We're going to show zero ops life in action. Not
[01:13] have too many slides. Um before that I just want to talk briefly about how is data work actually different from software engineering work. So how is a general purpose coding agent different
[01:25] from what we're doing with genie code and zero ops. First uh data work is not just about software engineering. It's actually many different disciplines, many different signals, a lot of information and then
[01:37] general purpose agents don't excel at this. The big thing though might be um the hard part is actually the data. It's not just the source code. So your general purpose agent might be completely fluent
[01:50] in Python and might write the most beautiful Python pros um that works and runs and is perfect except it produces the wrong data because it uses last quarter's revenue numbers or it turns
[02:03] out in your catalog there's three different definitions of what is a revenue like this. This string is is just unclear. So the answers for for actually understanding your data are actually in the catalog. They're in the semantics of the data. They're not in
[02:16] the source code. They're they're not in the repository. So this is hard for general purpose coding agents. And then the last thing is if you manage to build a data pipeline, if you manage
[02:28] to build a data asset, get that in production, the work doesn't end there. That's have to work. Have to work is after that. like these data assets need constant care in production. Um they
[02:41] will fall over, they will fail as data changes, as software changes. Um they will sometimes silently fail like maybe something changed and upstream there's now new new nulls or something like that
[02:54] and then your data pipeline seems to be green but your data is incorrect. So data work is different and it's difficult to work with which is why we build genie code. So Genie code is an
[03:06] agent specifically designed for data work and it knows your data. So it knows everything that's in a catalog, knows the semantics of your data, knows the governance policies and it helps keeps your data assets
[03:18] healthy. So you've seen in the keynotes, Genie Zero Ops is actually powered by Genie Code and it will proactively look at all of these data products, the data assets in production and keep them healthy.
[03:36] Now, G, we actually have a bunch of different genies, right? I'm told we have a whole family of genies. C. Can you tell us a bit about that? >> Yeah, so you've been hearing about the entire Genie family through the keynotes, like this week, right? So, Genie1 and Genie agents, just to put in
[03:48] perspective, are great for the business users, right? It's when you're talking to data, trying to answer questions about your data. And they do a great job of searching through the catalog, finding the right data, constructing the right queries, and giving an answer that
[04:00] everyone can sort of process and understand. With Genie Code, we are targeting technical users, right? People who are building things with data, building machine learning pipelines, data pipelines, dashboards, and so on. And now this week you're hearing about
[04:13] zero ops where geni code zerops is powered by geni code and it will help monitor those pipelines and fix them when things break in production. So today's session is focused on Genieode
[04:26] and Genie zero ops. And even though Genieode is still quite new, we're already seeing a lot of adoption. You've heard in the keynote yesterday, 60% of layflow pipelines are being authored with Genie Code.
[04:39] But even better, we're hearing really great feedback from our customers. They're telling us that Genieode feels like a glimpse into the future of how data work is done. And we're also seeing that now more code
[04:52] on data bricks is being written by Genieode than by humans. It is a huge productivity accelerant for people, right? Helping them do their jobs three times, four times, 10 times more quickly.
[05:04] But with that, let's switch to a demo. >> Maybe just really quick. All right, really quick, show of hands. How many people are using agents today in their
[05:18] workflows? A lot of you. Awesome. And how many have used Genie Code? Oh, wow. Okay. All right. So, for those that don't use it, you'll learn about Genie Code. And for those that use it, hopefully you learn some new things. So, over here, we're in
[05:32] data bricks in the Lakeflow pipeline editor, right? This is the great a great place to build my Spark declarative pipelines. I'm going to open Genie Code by clicking this button in the top right corner.
[05:44] I make sure that I'm in agent mode. Right, this is where all the new capabilities that we've
[05:57] for fraud detection using Gal's tables customers and transactions. This is Leonard's laptop. That's why I'm specifying Gal's tables.
[06:10] >> Yeah, you might see my name pop up every now and then. >> So, you'll see that Genieode starts by thinking through my request, searching through tables, loading a couple of skills that we'll talk about in a few minutes, but the first thing I want to
[06:22] highlight is that I I was a bit descriptive about which tables I want to use, but I didn't give it the fully qualified table name. Right? It's searching through the catalog, finding the tables that are going to be useful
[06:34] for this task. So, this is something we really invested in, right? We've talked to a lot of customers that said finding the data for my work is often the most boring part of it, right? Finding the the best data to to complete the task.
[06:46] They have to search, they have to talk to colleagues, ask on Slack, and so on. So, we've really invested in Geniode being able to find the right assets for the task and use them effectively. So, we can see that it sort of searched the
[06:59] tables um did another search just to make sure it's finding the right ones and then found my tables in in a schema transactions and customers and is describing them. It's now also telling
[07:12] me here's a proposed medallion architecture pipeline with some ingestion and bronze validation enrichment and silver and then a gold layer. I didn't really give it a lot of information about what I want the pipeline to look like, but I could have,
[07:24] right? I could have provided it an entire spec file to tell it how I want that pipeline to be constructed. But in this case, let's just go with what it proposed. And I can click one of these helpful follow-ups to say proceed with this architecture. Genode is always kind
[07:38] of suggesting follow-ups about what is the next step that might be useful to you along that journey based on the context that you're currently working with. So we'll see it's kind of thinking. It
[07:50] built this plan that it's showing at the top right of Genie Code. It's telling me it's going to read skill files, create pipeline files, write transformation code, and I can follow along as it's progressing in that plan and steer it if
[08:02] it's going in the wrong direction. Right? So I can send follow-up messages and tell it actually you're doing something wrong. Do something else. But for now, let's collapse this so we have a bit more space with this zoomed in
[08:14] mode so people in the back can see. Genieode now went and created a bunch of files. We can see they popped up here in this asset change list. This makes it easy for me to track all the files that
[08:26] Genieode is modifying as I work with them. I can accept and reject the changes or just click around to move between them. But I often leave that kind of review process to the end right
[08:38] after I've worked with Genie Code for a while, completed my task and then I come to to uh review the assets. So let's collapse that as well. So we can see it's updating these files.
[08:50] It's thinking through it all and working on them in parallel. It's now also asking to clean up the empty default file that the pipeline started with. And we can see that for this
[09:02] action, it's actually asking for my permission
[09:14] So I have this option here to click ask every time. And I can choose always allow in this current thread with Genie code or always allow for for all threads. But I actually want to focus on
[09:26] something new that we launched a few weeks ago called auto approve. When I toggle that on, every time the Genie code does an action that typically requires approval, it will actually evaluate on its own looking at what it's
[09:39] trying to do and the prompt that I shared. Right? So, it looks at what I said, proceed with this architecture and my previous conversation, and then decides whether that action matches my intent, whether it's aligned with it. So
[09:53] that's great because we've all heard the stories about agents sort of wanting to delete something that is completely unrelated to what we're working on here. This is an additional check. Obviously it's not 100% but it really works and allows me to step away from what the
[10:06] agent is
[10:18] going instead of just stopping asking for permission. With automotive, it just keeps going. >> Exactly. So, yeah, the the goal, right, like one of our goals is to find the right way to provide a workflow where you can give it a really long and
[10:30] complicated task and then just step away and work on something else. So, while this is running, I want to want to point out a couple of other things. We saw earlier that the agent um created a
[10:42] readme file over here. Let's see why that happened. If I click on the three dots here and then the settings menu, I can see something called instructions. Instructions is a markdown file that
[10:54] I've written which is information that is provided to Genie code on every conversation that I have with it. And we can edit it here. And we can see that one of the the items was in declarative
[11:06] pipelines. All tables must include table properties with a quality set to the medallion layer. So let's open one of the files that I had. And we can see that Genieode added this table properties quality is bronze in
[11:20] it. Right? So the instructions have helped me kind of teach Genieode the way that I want to build these pipelines. There's user instructions which are it's the file that I created about how I want
[11:33] to work with Genieode. But there's also workspace instructions. This is something that the admin can create in the workspace, add additional instructions, and then every user that uses Genie code in the workspace automatically benefits from them, right?
[11:46] So, I don't need to know about my organization's best practices to benefit from them when I'm working with Genie Code. And we can see that over here there's a bunch of kind of shared workspace use and retention policy, various kind of policies that I don't
[11:59] need to know about, but Genie Code will follow them automatically just for me using it here. The second thing I want to show is skills. So while instructions is something that applies to every conversation I've had
[12:11] with Genieode, skills are information for specific workflows that I have. So we can see that really early on when I told Genieode I'm building a fraud pipeline, it read a fraud pipeline
[12:25] standards skill. This skill file has kind of a a front matter where it says the name and the description about company specific conventions for building fraud detection pipelines.
[12:37] Genic code reads all these skill front matters and then based on that decides whether to load the rest of the file which you can see is quite extensive right saying that every time I build a pipeline it should use it should build a
[12:50] readme file in this particular format use certain fraud signals. So, so that's like the full readme that is included there. >> So, the the readme is not included. It's a template telling Genie code how to
[13:02] write that readme, right? So, my company has said every time someone builds a pipeline, you need a readme. No one wants to write their own readmemes. Agents are going to do it for me. So, we can switch back to to Genie Code here.
[13:14] We can see that after it read that skill, it now knows to follow these um these instructions and it created the readme file following kind of that template saying what are the sources,
[13:26] the data sets they created, which signals it used and so on. So those are instructions and skills really instrumental ways to customize Genieode to the way that you want to work. So let's see what happened while
[13:39] we were talking about that. So it it did it deleted that file then it tried running the pipeline did a dry run just to validate that things are working that passed and then the pipeline update ran
[13:51] as well and finished and it's now describing some of the results at the end and reminding me that included the table properties with the quality. So this was actually great. I was expecting something to go wrong somewhere. So I
[14:03] can show that when Genie Code makes a mistake, it can automatically iterate on that mistake and kind of take that next step without me having to act on it. >> I guess that's a demo effect, right? Like nothing ever goes the way you want it or it goes exactly the way you want
[14:15] it. >> In this case, the opposite, right? I thought something would go wrong, but but let's show something else. I'm going to ask it. Is there any other data that might help this pipeline? Again, a very
[14:28] generic question, but what I want to show is that Genieode really understands Uni catalog and the data that I work with. You heard about geneontology as well, where it will pick up additional signals and insights to really help find
[14:41] the right data, solve the the task in as efficiently as possible. And we can see that it quickly it actually found it in the earlier search. It saw another table called merchant fraud profiles with additional signals
[14:56] that might help this pipeline do fraud detection a bit better. So I'm I'm going to skip adding that to this pipeline. But we can see that it would be really easy for me to just click this button and have Genie Code go and add that
[15:10] additional signal, that additional table into this pipeline. The next thing I want to show is while we've relied on Genico to find the assets that I want to work with, I can also click this button or use the add symbol to reference other assets within
[15:24] the workspace. So this could be tables, markdown files, folders, pipelines, jobs, dashboards, and so on. Right? I can tell Genie Code, look at this pipeline that my coworker built. I want
[15:36] to build this new pipeline in the exact same way. Or look at all the notebooks that I worked on last week. I want to build another notebook that kind of follows those same principles or just like summarizes all of those insights.
[15:51] If I click on this plus button over here, can you Yeah, you can see it. Okay. If I click on the plus button at the bottom here, we can see a couple of connectors that we've added natively to Genie Code to make it really easy for
[16:04] you to bring in additional context. Right? Not everything lives in uni catalog and you want to sometimes tell genie code go read my emails go check my calendar go read this QBR doc that I wrote and then help me kind of get the
[16:17] latest insights and and u analysis on that in this case I've enabled the Atlassian connector and I'm going to ask Genie code what Jira tickets are
[16:29] assigned to me while that's thinking I'm going to just quickly show the settings page again where we see these connectors that are pre-built, but I also have the ability
[16:41] to just add custom MCP servers. So there are a couple that are built in for AI search, UC functions, Genie spaces. There's external MCP connect servers
[16:53] where any MCP that you're using with other agents, you can bring in to Genie code by adding it as a Unity catalog connection. You can also set up custom MCP servers by building data bricks apps that expose
[17:06] whatever context or tools you want Geniode to have. In this case, I'm happy with the the Jira connector. So, I'm going to go back here and see what happened. So, it called the Jira uh
[17:19] tool. It had access denied. It said something here doesn't look right. Let me try a different tool to uh to get that information. Right. So this is the auto approve where it's kind of thinking through what aligns with my intent and
[17:32] it's a new feature. We're still sort of tweaking it, but we can see that it eventually found the information that I was looking for. >> It just kept going, right? It didn't get stuck because there was no permission. >> Exactly. Right. So it saw, hey, this
[17:44] this action doesn't seem completely aligned with what you asked. Let me try something else without me having to be there to click kind of allow or reject. All right. So it's saying there are two juro tickets assigned to me. One of them
[17:57] is to create a dashboard with fly delay insights and the other is building a fraud detection pipeline. And it actually kind of put this in context, right? It said that second one is exactly what we just completed, but it's
[18:10] saying the I had to also schedule the pipeline to run weekly. So the pipeline it says the pipeline is built. Would you like me to schedule it? Right? So it's just taking that Jira ticket acting uh on it for me. So I'm confirming I want
[18:23] to schedule this pipeline weekly. And now without me even having to know about kind of LakeFlow jobs and how to schedule assets and so on, it can go and create that job for me running at 8 a.m.
[18:36] on Mondays. I can keep talking to Genie Code to to edit that to change what it's going to do to add more tasks to it or create more complicated jobs. But for now, I'm pretty happy with it. And we could mark the the Jurro ticket as done.
[18:50] So, at this point, I finished authoring the pipeline. Maybe that's the easy part, though, right? I'm going to hand it over to Leonard for the hard part of actually productionizing it and maintaining it. >> Yeah, I'm going to show the hard part, but I'm going to just ask Genie Code to
[19:03] do the hard part for me. >> That's great. >> So, yes, in a few minutes, you basically built a full data pipeline, which is amazing. But this is not a production data pipeline, right? So what I'd like
[19:15] to do is actually take this pipeline and put it in Git repository and have it deployed to production. So what I would like to do is just ask
[19:27] Geniode add this pipeline to Git. Now you should know uh we're working super hard on Genie Code. We're improving it every week. Literally every week. Uh so so last week we were
[19:39] actually working on this feature like add pipeline to Git. This has not landed in production yet. I still want to demo it. So what I figured, so if maybe Genie code doesn't quite understand how to add
[19:52] pipelines to get yet, maybe it's going to improvise. Rather than having it improvise, I can actually build my own custom skill to add pipelines to Git. So that's exactly what I did. So let's let's go look at
[20:04] another skill file. So this is the if I scroll up the add pipeline to Git skill. So I had genie code create a new skill file for me for adding just any pipeline that's in the workspace that's not
[20:17] currently in a git folder to git
[20:29] for me um one very nice touch I found is that it says don't give a generic git lecture and the user ask at this pipeline the n So this is like code added this to the
[20:42] skill. What I mostly asked about is it needs to actually look at the details. What what is in this pipeline? It needs to like list all the files. Um and I figured it would be nice if it looked at the home
[20:54] folder. Are there maybe already git folders in this home folder of the current user? Then the skill can suggest those things, right? And then I I told Uni code it should just ask two questions to the user uh
[21:08] like which folder do you want to use and what name do you want to give this folder inside the git folder and there's a bunch of details here about file formats and so on again I didn't write
[21:20] any of this it's a it's a 200 line scale uh so let's give this a try it's actually one of the most scary things here for me is like is it even
[21:33] going to find my skill, right? Because I'm going to tell it add this pipel and then gen code needs to know based on this preamble that it should load my skill file.
[21:47] So let's see if it's also going to do do exactly the right thing like in your demo. Yes, it's using my custom ad pipeline skill. It's reading the skill. It's reading pipeline details. It's looking at what
[22:00] files are in this folder and treating my home folder. This is going pretty fast. It's thinking about it. And look, yes, it's asking the first question, right? Um, in the skill, I actually told
[22:14] it to use these suggestion bubbles. So, uh, I can just pick one of these choices and I I can pick uh the empty git folder or the the summit folder. It would be kind of like cheat. So, I'm going to
[22:27] empty git folder. And then I think I should ask the follow-up question
[22:40] just reasoning about what I just did and and basically what's in a skill. Um this is where actually has a a brief bit of explanation. So I might know about git I might not know about declarative automation bundles. So declarative
[22:52] automation bundles is the the standard that we have at database for actually writing everything as source code. So we just created a pipeline in UI with DABs. We can make sure that there source code that you can then put in the Git
[23:05] repository and use CI/CD on and all that. So yes, it's using DABs and it should give a name to this this DAB project and it's suggesting uh yeah create a new
[23:19] pipeline yada yada bundle. I was hoping for a nicer name. Like I say in prior demo attempts it gave a name like fraud detection pipeline bundle but this is this is fine.
[23:37] Let's see if it can actually um now take all of my files and put them in the get folder. Cool. So what it actually did here is it generated a bit of Python code. This Python code will take my files and move
[23:50] them into this git folder because GI code just has an execution environment. It can run Python code like this. It can run CLI commands. And since I just wrote this skill file as a markdown file.
[24:03] There's no special tools for adding things to git. Uh it's improvising, right? It's generating Python that they can run. So I'm going to just allow this. I'm sure this code is fine. It's templated based on this skill.
[24:15] Oh, but gen code is actually telling me that hey, it actually reviewed this source code that was generated. It's like a a side agent, like a security agent, tells me that if I run this Python code, it's going to write to SLworkspace, write YLM configurations,
[24:28] uh, copy file tree things. This all sounds kind of like exactly what I want. So, I'm going to just allow this
[24:43] command with grids undefined, not found. Okay. Uh, sorry. Oh, actually is it? I was going to just try it again with I think a slightly different variation of the source code. Let me just try it if it works now.
[24:56] It's analyzing the source code. Run it now. Okay, now it's actually doing something. So, it actually just it generated a
[25:08] Python code for me. It ran into some error in the Python code and then it quickly recovered, right? felt felt, hey, this error in this is Python, I can fix that. Tried it again and it now did run something. It tells me that it added
[25:22] it to this empty git folder in this new pipeline folder and it tells me it can take me there. So, let me just ask it to take me to this git folder and see what it did,
[25:34] right? I should enable your auto mode also. Um, so this is the the Git folder. There's um a new pipeline thing in the kit for I'm going to open this in the editor.
[25:50] No edits have been made in this bundle yet. That's fine. Um and yeah, if you look at this, it's the same bronze, silver, gold Python files. And there's a new thing here that's called a data. Mml file. So this is actually from from dabs. This is a configuration file that
[26:03] says uh yeah you can if you're using a CC system or if you're working with local laptop there's a configuration how to deploy that to development how to deploy to production and the other new thing here that wasn't
[26:16] in the original version is there's a resources folder and this is a source code description so a source controlled description of the pipeline definition. So this is the pipeline configuration
[26:28] inside the git folder. So, it did all of that. Um, I think we should now skip forward, do like a little time skip, and see what it looks like if we had this in production.
[26:48] I actually have that in a tab prepared. So, we're now a week in the future and this pipeline has been running successfully. It's been green, which has been great. Um but then a few days ago it started failing.
[27:01] This happens at least it's failing loudly. So that's nice. But then um as an operator I can look at hey so these recent runs they all failed. Let let me open one of these recent runs. I can uh
[27:13] look at this. So refresh pipeline field. Um I can click all the way into it debug it and so on. Do it manually. Or
[27:25] this is where I could use uh zero ops, right? This is where I could use zero ops to do this for me because what what G's been showing is uh he made a pipeline a full pipeline in a few
[27:37] minutes, right? So you're 10x faster, five, what is it? >> I mean, we're hearing from customers it's it's three, four, 10 times faster, right? Like we built a pipeline that would have probably taken an entire afternoon in the 10 minutes of demoing,
[27:52] right? And obviously like it needs a lot more detail and so on. But you can give that task to gen code. So we could be creating hundreds of pipelines per day. >> Yeah, we have to maintain all of them. >> That's a challenge. >> That is the challenge. Yes. And we're
[28:05] spending so much time on that. And we want to actually spend more time on creating more pipelines, right? Um, this is where we're going to go back to the demo world
[28:22] because this is um I'm going to show ops. So, this is actually a full screen version of Genie Code. Um, is this already in the hands of customers, the full screen version? >> Yeah, we we rolled it out a few weeks ago. It's in beta, so you might need to enable a preview, but you can get this
[28:35] full page Genie Code experience to to kind of work more completely with Genieode and also get the Zero Ops experience in it. >> Yeah, I love how agentcentric it is. But yeah, so so this is a forward-looking
[28:49] version where there's zero ops inbox that's not in beta yet. Um, by default, it's just not configured to do anything. It's it's optin currently. We're talking to customer. Should we enable this always? it does it
[29:02] enable it. Now very core UX principle for us has been um we want to make everything chat based if you can. So there's no model dialogue asking you about how to set it up and lots of options and checkboxes.
[29:15] We'll just say hi I can help you set this up and um in this case it can monitor job schemas for me. Uh in the future we're going to do lots of other things like ML assets. We don't have that in this version. Um I'm going to
[29:27] ask it just monitor all my jobs. I have a bunch of jobs in this workspace. I could also ask it to monitor schemas, but then it's going to take more time. It's going to have more data over time. So, for the demo, I'm going to just look
[29:39] at jobs, >> and I I can uh ambigon a bit. Um, yes. So, it's monitoring all nine jobs for me. Uh, it's going to do an
[29:51] initial scan. So, it's basically going to look at the recent runs. Did anything fail? Uh, might take a few minutes. Uh, I hope it does not. Uh look at the inbox for updates.
[30:03] Okay, so the inbox now shows that uh so far zero incidents. Uh it's monitoring nine jobs, no tables, and it's it's looking at these recent failures, right? Or recent runs.
[30:15] And then yes, it just found the failure that we just saw, right? The fraud detection job has been failing. Let's uh dig into this. So it's failing. It seems to have a typo
[30:28] since the latest uh commit. Um actually links to commit. I'm going to just look at the commit rather than uh read what it came up with.
[30:41] Um this was my commit. Um so apparently I changed the the schedule here, but for some reason I don't know maybe for the demo I made a
[30:53] typo. I changed it from country to home country which was a bad idea because it made the pipeline fail. So let's go back. So what zero ops did is it actually found this. It investigated it without me asking for it, right? It
[31:07] proposed a fix and the fix is just going to change that identifier back and um it actually validated it in a in a sandbox. Um,
[31:19] so you validated this sandbox. That's super awesome because that is super important for agents, right? They need to try things rather than just guess.
[31:33] It tells me it did two read only checks. So apparently it ran a describe query to see does this in the production catalog. Does this column exist? Is it country? Is it home country? And then it did a
[31:45] dry run. This is what you saw in the earlier part of the demo like with STP we can do a dry run where you run a pipeline without actually doing anything and then um in test both operations are read only that is true the big thing is
[31:59] that they run in a sandbox where it's not actually possible to write so even if it tried to write to the production catalog that would be prevented does have tools to create like a shallow copy of the of the data if it wants to write
[32:12] to something um but actual direct rights to production prevent it. It It ultimately all depends on on human and loop review before it makes any changes. So I can click uh create a PR.
[32:25] Um and it has a a fix for me. >> That's awesome. Um I also have a slightly more complicated example actually. So this is a workspace uh that is maybe
[32:39] a bit more typical. Uh I have to say the other day I heard a customer where they they said in our workspaces everything is always green. We never have errors.
[32:51] I also see lots of customers that have workspace that look more like this. So so sometimes jobs fail. Um it could be that after we saw failures here recently those those jobs started working again,
[33:03] right? Could be that some jobs failed but they're actually not important. We don't care about them. Uh it could be that they're running green, but the data is not good. So it's actually very hard
[33:17] to get signal from all of this noise from a dashboard like this. You need to really dig in as a human or ask Genie code, hey, what's going on here? Like uh if I look at recent runs, uh there's a lot of successes, but then this customer
[33:29] 360 field and I can dig in and see maybe it's recovered after or I go to my inbox, right? I go to my zero ops inbox and I see a a prioritized
[33:43] overview of all the things that are important to me, all things I should care about. So, um if I look at the top of this inbox, it actually says there were three dashboards broken from this fan
[33:56] interaction thing, but this is resolved. I think I think below resolves it in the keynote. So, let me change the filter to just show open incidents. And then uh what still demands my
[34:09] attention is the wonder revenue went down almost 50%. >> That sounds pretty bad. >> I think that is super bad. Um so these reports they they start with a
[34:22] summary. So this is a slightly more complicated one than the super stupid typo one that I just showed. Right? This is where uh the pipeline actually stayed green. It was running successfully and then um machine learning based anomaly
[34:34] detection. It's a product that's already available today. It caught this. It caught that. Suddenly the orders dropped. But then the thing about machine learning is sometimes there's false positive, right? But then zero ops
[34:47] used an agent to look at it more closely like is this a real issue? And uh it confirmed this is a real defect. Um it actually tracked it back to a recent code change apparently
[35:00] another code change that caused this. And then what it did is it looked at the impact like is this important? Should should the human even care about this and it told me that hey there's a bunch
[35:13] of downstream aspects like an ex executive KPI dashboard uh popular assets. So it marked it as critical so it appears at the top of my inbox. It root cost it um and it proposed a fix
[35:29] and this is um yeah this is where this is a slightly more complicated instance. So, this is where I might ask you just follow-up questions like, "Hey, what happened here?"
[35:50] And I might ask like, "Uh, did um isn't it just for your bookings?" Um like maybe everyone is at D. Uh like everybody's at
[36:02] the summit, right? So they're not doing any bookings. Actually, let me read what it said here. So it was a code change yada yada drop statement because there were were nulls in the data.
[36:14] Uh, no. And it tells me that the actual upstream bookings, uh, I think this is like in the bronze table, they were unchanged, but there was some drop statement that that caused things to drop.
[36:27] Okay. Um, fine. Um, let me just ask it to create a PR. I think this gives me a lot of confidence in this fix. So, let's let's create a PR. And, um, yeah, it's
[36:39] it even updated my test. This is pretty cool, right? because zero ops caught an issue in production with production data. This test did not catch it. This test did not have a case for this. But now it's suggesting a test
[36:53] with test data that would catch this in the future. So with that um let's briefly go back to slideware. It just shows her ops. Uh
[37:05] there's a four important things that it does. It provides a signal from noise. Um it automates toil. You don't need to constantly press diagnose everywhere. It's just fully proactive. It will
[37:17] safely validate things in production. It's going to run an agent in production safely and it learns from you. I might ask you things like, hey, this wonder thing, isn't that a silly demo pipeline? And
[37:29] just just don't make that critical can and it will store that as in memory. So then for me, it will not show that at the top again. >> Yeah. Do a quick recap. >> Yeah, just to to recap some of what we
[37:41] discussed. So we talked about Genie code and how it's designed for data work. It knows your data and can work with it effectively and it keeps your workflows healthy, right? You can diagnose issues with it and now with zero ops, you can do all of that kind of automatically.
[37:54] So on design for data work, right, we showed that it can find relevant data, construct pipelines, schedule jobs, but really does a lot more. It does it automates all the work that I want to do in data bricks, right? So I really
[38:07] encourage you to try starting your task with Genie Code. see how far you can get with it. Just letting Genie code drive that work. It also knows your data, right? It's grounded in Unity Catalog. It can search
[38:20] and analyze the data for you, figure out how to connect data together, how to use it effectively. And then you can make it your tenur co-orker, right? With skills and instructions, you can tell it kind of remember how I did this workflow.
[38:33] When I work with it, I often start the task with Genieode. I tell it what I'm trying to do. Along the way, maybe I need to guide it a I say I prefer to look at it this way or present the information in a different way. At the end I will tell it remember everything
[38:46] we discussed and create a skill for that workflow. So then the next time around I come to it and I say hey we're doing that fraud detection pipeline work again just use that like it will use that skill automatically build that pipeline
[38:58] in the way that my company wants me to do it. And we also showed connectors and how you can bring in additional context from GitHub, from Atlassian, from from your documents and so on, right? It becomes really powerful because then
[39:10] maybe you don't even start by telling it build a pipeline with so and so layers. You can tell it look at all my geo tickets and just start working on them and present to me the the kind of
[39:22] pipelines and notebooks to review. And I'm putting up here a QR code for requesting an invite to the Xerox private preview. Right? Everything we showed in the pipeline construction, the
[39:36] scheduling and so on is available today. The connectors are in beta, but you can enable that in your workspace. Zero ops will be in private preview soon. You can scan this QR code uh to request an
[39:48] invite. >> How soon? How soon? >> Hopefully next month. So, very soon. And then the second QR code is just for an agentic data engineering blog where we talk about some of the capabilities we discussed today so you can learn more.
[40:01] And with that, thank you so much for your time. I think we're we don't have time for live questions, but we'll be here if you want to come up and talk to us about anything. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.