Genie Code for data science: Work at the speed of your ideas

Summary

Genie Code is an autonomous AI partner built into the Databricks Data and AI platform that handles exploratory data analysis, feature discovery, and machine learning model training so data scientists can focus on creative decisions rather than repetitive mechanics.
Genie Code is grounded in Unity Catalog metadata to help users find the right tables and build accurate queries, and it automatically logs model training runs to MLflow experiments for easy comparison across sessions.
Reusable Skills files let teams encode workflow-specific context for Genie Code, and a Jira connector enables it to pick up tickets and build dashboards to close them, bridging data science work with project management.

Genie Code for data science: Work at the speed of your ideas

What if you could go from a raw dataset to a trained model in a single conversation?
Data scientists spend a lot of time on the repetitive parts of the lifecycle: finding data, wrangling it, writing features, tuning hyperparameters, and not enough time on the creative work that actually moves the needle. Genie Code is an autonomous AI partner built into Databricks that changes the equation. Tell it what you want to explore, and it writes the code, runs the analysis, and iterates with you.
In this video, we’ll walk through how to apply Genie Code across the data science lifecycle. We’ll cover:

How to use Genie Code for exploratory data analysis and feature discovery
How to build and iterate on machine learning models with less manual effort
When to steer and when to let Genie Code drive
Best practices for working effectively with Genie Code as a data science partner

Talk By: Gal Oshri, Sr. Staff Product Manager, Databricks ;

Chapters

FAQs

What is Genie Code for data science?

Genie Code is an autonomous AI coding and data assistant built into the Databricks Data and AI platform for technical practitioners. It can perform exploratory data analysis, discover relevant tables using Unity Catalog metadata, train machine learning models, and automatically log results to MLflow experiments.

How does Genie Code find the right data tables?

Genie Code is grounded in Unity Catalog metadata and uses the signals stored there to identify relevant tables for a given task. This metadata-driven approach reduces the time data scientists spend searching for the right data sources before they can begin analysis or modeling.

What are Genie Code Skills?

Skills are reusable context files that capture workflow-specific information so Genie Code can apply it consistently across sessions. Teams can define Skills to encode knowledge about their data environment, naming conventions, or standard procedures, making Genie Code more effective for recurring tasks.

Does Genie Code integrate with MLflow?

Yes, Genie Code automatically logs machine learning model training runs to MLflow experiments, including hyperparameter sweeps and evaluation metrics. Data scientists can then evaluate and compare runs across multiple training sessions directly within the Databricks Data and AI platform.

Full transcript

[00:08] here on the last day of the conference in the late afternoon. Maybe just before we get started, how many of you are already using agents in your workflows today? Quick show of hands. Quite a few. How many of you are
[00:20] using Genie Code? Awesome. All right. Well, hopefully those of you that are already using it will learn something new and those that aren't will be convinced to try it out. So we're here to talk about Genieode for data science.
[00:33] Just before we get started, there'll be a few things here where I'm showing you something that isn't launched yet. So this is just forward-looking work. And more importantly, we really want to improve this conference in the future.
[00:46] So please submit feedback at the end. All right. So we've all seen that AI agents have changed software engineering already. And that same shift is now changing data work. What does that mean?
[01:00] Well, data work is a bit different from general coding, right? There are different requirements. We think about building machine learning models or data pipelines or dashboards, right? It's not just the code there, right? You have to think about how you build a robust
[01:13] model, one that is high quality and so on. And working with that data can be quite challenging, right? finding the right table to use for a task, building a query that actually works efficiently
[01:27] and finally deploying the model at the end is not actually the end of the work, right? That's the really the beginning in some ways. You need to monitor that model. You have to figure out how things change over time, improve it in the future and so on.
[01:41] So that's why we built Genieode. It is an autonomous AI partner for data work. It is designed for this type of work, right? It's fluent in data science, data engineering, machine learning, and BI. All the stuff that you do in data
[01:53] bricks, Genieode can help you with. It knows your data. It is grounded in Unity catalog metadata and all the other signals that are there to figure out how to answer your questions well, how to
[02:05] build with your data, and so on. It also keeps your workflows healthy. When a job fails in data bricks, you can use Genie code to diagnose that job and fix that issue. And what you heard in the
[02:17] keynotes with Genie zero ops, you'll be able to do that automatically very soon. And even though Genieode is still quite new, we've heard really positive feedback from customers. They're saying things like, "Giode helped us move from
[02:31] raw data to govern tables, tracked models, and a productionready serving endpoint in 90 minutes." We're also already seeing that more code on data bricks is now being written by Genieode than by humans. It is a huge
[02:45] productivity multiplier. We're we're hearing that it's three times, four times, 10 times faster in helping people do their work. And I know you've been hearing about a lot of genies in this conference. I want
[02:59] to just put a bit into perspective the the types of genies. There's genie one and genie agents which are great for business users, right? They help you answer questions about the data. They help you talk to the data really easily.
[03:11] There's genie code which is for technical users. This is what helps you build with data dashboards, pipelines, models. And there's zero ops for maintenance of those pipelines and models. Our focus today is going to be
[03:25] on Genie code. But enough slides. Let's switch over to a demo. I'm starting from a data bricks notebook, the bread and butter for data scientists. and I'm going to click on
[03:38] the genie code button in the top right corner of the of the screen. I open it and I make sure that I'm in agent mode. This is where all the new capabilities and the goodness of agentic workflows come in for Genie Code. And
[03:52] I'm going to start by asking it to find a table for flight delay analysis. Right? We want to do some flight delay exploration here, train a model for it, and so on. And we start by asking Genico
[04:06] to find a table that can help me with this. When we talk to customers, we often find that one of the most tedious parts of doing data work is finding the right table. I see a lot of nods in the in the audience. Finding the right table
[04:18] is a bit tedious, right? You have to search, you have to look at, you have to ask co-workers and so on. So, we really focused on a good agentic search for Genie Code. So it looked through the
[04:30] tables, some of the assets I have in the workspace, some of the queries and reviewed them, right? It can look at the metadata of the tables, the sample data to figure out what is the most useful for the task we have today. And it found
[04:43] this flights table even though there are quite a few flights related tables in this workspace. And it's saying why it's the most useful. It also found some existing
[04:55] notebooks and dashboards that I've already worked with with this data to create, right? So, it can find relevant assets that might be useful for the task I'm going to do. But for now, let's do visualize three insights about this
[05:10] table. Now often if I'm working on a task maybe I have some very particular goal in mind but in this case I'm going to I want to do a free form exploration of the data
[05:23] and agents like Genieode are really good at taking that data and exploring it finding interesting information about it. So let's see what Genieode comes up with here. So we see that it starts by editing the
[05:36] notebook. It added this nice markdown cell to explain what this notebook is going to do. I can review here how it's thinking about it and it already found or it's already planning three
[05:48] visualizations. Departure delay by airline, delay patterns by day of week and where did it go? And monthly cancellation rate trends
[06:01] before executing the code. It's giving me a chance to review it. Right? So, it stopped here to ask for my approval. I'm going to click allow, but I'm going to also show this menu on the left here where it says ask every time. I can
[06:15] actually instead of kind of having to click allow every time it wants to execute code. That becomes really tedious, right? And slow and I don't want to babysit the agent, I can click always allow in the current thread to allow it to run the notebook cells for
[06:28] the duration of this conversation with Genie Code. I can click always allow for other threads as well. But I'm going to use this new tool or new capability that we launched a few weeks ago called auto approve where it will let me just find
[06:43] it where every time it wants to execute a tool, GI code will look at what that tool is trying to do, look at the prompt that I shared and figure out if the intent of of my prompt is aligned with
[06:56] that tool. Right? So, it's finding a really good balance between safety and reviewing what the agent is trying to do and what I actually and kind of reducing how much manual work I have to do to
[07:08] click allow 500 times per day. All right. So, we see that it executed the cells in the notebook. It found a couple of insights. Departure delays by
[07:20] day of week. Seems like Thursday you're you're not it's not the worst if you're flying out today. I hope no one's flying out Sunday. Um, and the delays build throughout the day, right? Earlier in the day, there
[07:32] are fewer delays. It also summarized the insights on the right over here. I want to to show, and we'll come back to this a bit later, this following your Wonder Bricks brand palette, right? It
[07:44] it follows the colors that I wanted to to create all my visualizations with, but we'll come to that. Over here at the bottom, there is the asset change list. It is showing all the cells that Genie Code modified so that I
[07:57] can quickly jump between them and review. I can also kind of accept or reject the changes over here for every cell. But what I typically do is wait until I'm done with the workflow, accumulate all
[08:09] the changes that I want to make, and then at the end I can click accept all or reject all. The next thing I want to do is train an XG Boost model to predict flight
[08:22] departure delays. While that's working, I want to talk about the the wonders brand pallet that that we saw. If I click here on the three dots at the top and then settings,
[08:37] we see there is this user instructions here. Instructions are a markdown file that I have written to provide GI code some information about how I want it to work. These are things that it should
[08:49] always consider when I'm working with it. And one of these is about building declarative pipelines. Another is about visualizations. And for those always use the Wondericks brand palette and then a couple of colors. Right? So without me
[09:02] having to tell Genie code to build the visualizations with those colors, it takes those instructions and uses them automatically. So this is great for any conventions, best practices, or the ways that you want Geniode to work with you. There's
[09:15] also workspace instructions over here. An admin can create a similar markdown file and provide guidance on how everyone's GI code in the workspace should work. So again, without me having
[09:27] to know about instructions, if an admin in the workspace has set that up, I will automatically get all those best practices, which is a great way to kind of get everyone to work in a very similar way.
[09:39] Let's go back here and see what Genie Code is up to. It read some skills, which I'll talk about in a second. It's editing the notebook now. And it created this plan at the top where it's showing the steps
[09:52] that it's going to follow. Right? The prompt that I gave it to train a model is a bit more complicated than the previous one. So, it's tracking the the steps that it's following, and I can follow along. If I see that it's doing something that's incorrect, I can always
[10:05] give it a follow-up message to steer it and point it in another direction. I'll collapse that plan and now talk about the the skills. So we see here this line where it said readick's model
[10:18] card skill. Unlike instructions which are used in every conversation, skills are markdown files that are only used in particular workflows or situations. So, if I switch
[10:30] to this next tab, I'm opening the skill file for the wonders model card. Over here, I have a name at the top and a description. The description is what tells Genode when to load that skill.
[10:44] So, it's saying required for all ML workloads. Genode will only read this top part in every conversation I have with it. And then based on that top part, decide whether to load the rest of the skill
[10:57] file. It's a pretty long one, right? So, it has a model card template with some required fields, an HTML template for how to present it, some formatting guidelines, and so on, right? I don't want to load all of that context into
[11:10] every Genie Code conversation because it'll confuse it, overwhelm it. So, let's come back here and we see actually that Genieode tried
[11:22] to run some cells and hit an error. when it hit that error, it automatically is is thinking through it and now it's modifying the notebook to to get past
[11:34] that issue, right? That ability for Genieode to iterate on issues automatically is what makes it so effective, right? It can look at the results, it can look at the errors and figure out how like what to do next to
[11:47] help me with my task. I haven't typed anything in a few minutes and it's still working to figure out how to train that XG boost model for us. So, it made some edits. It's now rerunning them.
[12:09] I will show you the next thing that we want to talk about. So, I started this this thread by asking Genieode to help me find the assets that I that I want to use for this task. I can also click this at button to
[12:21] reference assets or just type it. So in the future I could say train a model to use and then at a particular table a particular notebook queries pipelines
[12:34] jobs dashboards and so on. I can explicitly reference other assets in the workspace to tell Genode to use the information from those assets to build this new asset. Right? Or I can tell it
[12:46] here's a folder with all the notebooks I worked on last month. help me summarize all the insights or all the the models that I trained there. So, it's really effective to build up those workspace assets and and kind of learn from them,
[13:00] right? And I can kind of bring all these components together. So, in instructions or skills, I can tell Geniode, here's a golden pipeline or here's a golden notebook for how I've trained a model in the past, and I want you to follow it in
[13:13] future tasks that are similar. All right. It is still iterating on training the XG boost model. Let's see how it's going.
[13:33] So the run is started and we can see that it's logging it to MLflow, right? MLFlow is data bricks tool for experiment tracking, logging runs, being able to compare the model training sessions over time.
[13:45] And I didn't have to tell Geniode to use MLflow, right? It has all these data bricks best practices built in so that it automatically logs that run to MLflow so I can review it in the future.
[14:07] While it is training, let's talk about some of the the connectors. So, a lot of the context that I want to bring into Genieode and use in my work might not be in the workspace already, right? There's stuff in on my calendar, in my docs, in
[14:21] my email that I sometimes want to bring into Genieode to to use for my work. When I click this plus button at the bottom left of the Genieode side panel, I can see a couple of connectors that we've made really easy to get started
[14:33] with, right? So I can just click connect and ooth to to Gmail, Calendar, Drive, Slack, and so on. I' I've enabled Atlassian already in
[14:45] this workspace and we'll we'll come back to that in a second. But let's see. It seemed to have finished training the model successfully. You know, one one time I was demoing and it went six like six errors in a row. I was about to give up
[14:58] and then Genie Code finally succeeded. So right like the ability to just iterate until it completes a task is really really powerful. All right. So it trained the model. It
[15:10] created this model card at the end. Right? Model cards are just visual templates that you might your organization might want to follow to present results in a standard way. In this case I've just built a simple HTML
[15:23] template to convey what are the features, the target variable, the performance metrics and so on. And it's telling me that it logged it to MLflow. We can also see here a couple of follow-ups. Genie code is always kind of
[15:36] suggesting what is another step that you can take to to help with the task. Right? So now that I've trained the model, it's suggesting well maybe you want to tune the hyperparameters to actually train a better model. And it's actually stealing my thunder a bit by
[15:49] suggesting the augmentation with weather data because the next question I wanted to ask was what other data is available to improve this model. So this goes back to that unity catalog
[16:02] integration where it can search across the catalog, see which tables are joined together frequently and kind of figure out what might be most relevant to this task. You've heard about geneontology this week, right? right? These new
[16:15] capabilities that make Genie Code and Genie 1 more effective in kind of these types of data workflows. So, we see that it found two relevant tables, weather and holiday events. And
[16:29] it's even telling me how to join those tables, right? The date that joins to flight date. But I don't need to to really think about that because I could just tell Genieode to go and update the model with this new information. But
[16:42] just to keep things moving, let's switch things up and ask Giode to take us to the MLFlow page for this experiment.
[16:54] So far, I've shown Genieode just in the notebook, right? Doing all the data science work that I might want to do here. But really, Genieode is about automating the work that you do across data bricks. It's not just the notebook. So, now we're going to MLFlow. It opened
[17:08] this page where I can see the experiment run that we just finished, the one that failed three minutes ago, and then a couple of other runs that I ran over the last few days trying this out.
[17:20] I'm going to open a new thread here by clicking the button in the top right. And then I'm just going to to show that there are some starter prompts here on various pages of data bricks where Genieode suggests what it can do on that
[17:33] page. So in this case, I'm going to tell it to evaluate my experiment runs. It can look through the runs and the traces and help me figure out how they
[17:46] compare with each other or even how to improve them for the future. So we really invested in capabilities like the ML flow analysis, ML model serving where you can optimize endpoints and many other parts of data bricks that I won't
[17:59] have a chance to show today. data engineering pipelines is another big one where if you're building Spark declarative pipelines, Genicode can really automate a lot of that work as well. So we see that it it's summarizing
[18:11] and it's saying I have 11 runs predicting departure delay minutes it's comparing those runs a couple of metrics and it's giving me some key findings. Augmented features are critical right adding that weather and holiday delay
[18:23] improved the metrics. It's noting that I've been lazy and not doing any hyperparameter variation. Right? So the next step might be to tune those hyperparameters. That's giving me those recommendations a bit more explicitly and making it really easy for
[18:36] me to to just click that follow-up and take that next step. I'm going to switch back to our main thread. Now for the next part, I'm going to ask Geneode to actually we talked
[18:48] about the the connectors earlier with with Atlassian. I'm going to ask it are there any Jira tickets assigned to me. Now I'm showing these kind of built
[19:02] built-in connectors that are really easy to to set up, but you can go back to settings and there's the add server option to connect any MCP server that you have or that you're using with other agents. There are a few that are built
[19:15] into data bricks. Uni catalog functions, AI search and genie spaces. You can build a data bricks app to be a custom MCP server. So anything that you can expose in that app to you can
[19:28] connect to Genie code and have it use it in these workflows. And then for external MCP servers, you can set up a Unity catalog connection to bring those MCP servers into Genieode.
[19:45] So coming back here, we see that it searched Jira for my tickets and it found two of them. One is to create a dashboard with flight delay insights and the other is to build a fraud detection pipeline. It's immediately contextualizing right and saying that
[19:58] first one is related to the work that we're currently doing. The flight delay visualization and model we just built could be the foundation for that dashboard and it's giving me just that follow-up to build that dashboard. So I
[20:10] click on that and now we see that Genie code goes and creates a dashboard for me
[20:22] where I can see it in this asset change list at the bottom. So we have that notebook that we were editing earlier and the dashboard and now it's opening the dashboard.
[20:36] How many of you have used AIBI dashboards in data bricks so far? quite a few. Yeah. So, this is one of the like people's favorite parts, right? It's just so visual the way that Geniode can build the dashboard for me without me having to do much. So, let's just see
[20:50] that it's it's on its way. It's saying once it's built, it will mark my J ticket as done. But, this can take a few minutes. So, I'm going to switch tabs while it's thinking
[21:02] back to the databicks homepage and the Genie Code side panel. So far, we've looked at Geniode as kind of a side panel experience, right? It's something that I'm working with to help me with my notebook, with my MLFlow experiments, or
[21:14] with my dashboard. But as people are shifting more and more of their work to agents, we've been iterating on that experience. And we recently launched a full page agent command center experience with Genieode. Right, I
[21:27] clicked that maximize button in the top left of the Genieode side panel. Over here, we have a lot more space to work with Genieode, right? I can ask it to investigate a particular table. Let's
[21:40] say the holiday events table. And once I get started, I can really see this in the full page. On the left, I can see all the threads that I'm working with and more quickly switch between them. So, I can start a new chat. Actually,
[21:52] let me just auto approve this one. I can start a new chat and then tell it to create a notebook with three visualizations about
[22:04] flights and then have both of these threads running in parallel. Right? So once I've enabled Genico to to kind of do one of my tasks automatically at a time, I can
[22:17] now take a step back and have multiple tasks running at a time and just check back in on them periodically to see how they're doing. And by the way, I'm clicking auto approve multiple times here. You can go
[22:30] to settings, make that the default. That's what I usually do, but just for demo purposes, it's easier to to start it that way. All right. So, so here right in this thread, Genie Code is working on a notebook that it opened to the side.
[22:43] It's a bit squished because I'm trying to zoom in and and show people in the back what what's happening, but I can switch back to this thread and see how it's doing in terms of profiling the holiday events table and the other queries that it is running or just go
[22:56] back to other threads I've run before. In this mode, it's also much easier to kind of access the customizations, right? all the skills, the instructions, and the MCP servers that we talked about
[23:08] earlier, right? So, it's really important to set these up to give Genic Code all the context and capabilities so it can do your work for you. And we see that's already starting to populate that notebook over here.
[23:21] I'm going to switch tabs, see how our dashboard is doing. It is still thinking all right, but it's preparing the data sets and now it is compacting the conversation. Right? So, we've had that
[23:33] thread for quite a while. It's reached a point where it needs to summarize a bit about what has happened so far. Um, so I can continue that conversation, right? It's keeping only the most important information in its context window.
[23:53] Let's check back here. My notebook has almost finished running. All right. So, that finished. And now maybe as a final step for this one, I'll say now I want to run this analysis every week on
[24:05] Monday at 7 a.m. Right? Data bricks has LakeFlow jobs, a really powerful way to kind of schedule workflows and it's all integrated into Genieode. So I can just ask it to create a job or even if I
[24:18] don't know about jobs, right? I could just say I want to run this analysis every day and it can create that job without me having to know about it. and even kind of change the schedule time zone or add an email alert on the
[24:30] failure. And it looks like our other thread is done as well. There was a check mark here uh explaining the the holiday events table, right? So that multitasking is super powerful.
[24:44] All right, dashboard is still thinking. Maybe we'll come back to this. We'll switch back to the slides.
[24:59] So the one thing that I couldn't show today in the demo, but it's coming really soon is automations. What I've shown so far is I'm working with Genie Code interactively, right? I'm at my laptop telling it look at my Jira tickets, help me kind of create
[25:12] dashboards, help me train models. But very soon we'll have Genie Code run things on a scheduled basis, right? So I can tell it every morning look at all my Jira tickets that are open and assigned to me and just go start working on them
[25:25] right build that notebook build that model and by the time that I wake up I can just review all those notebooks switching between them quickly to make sure that it's doing the right thing right so the what I do is more focused
[25:37] on like guiding the agent a bit and reviewing the work that it's done focusing on the insights and the learnings and kind of figuring out what are the right questions to ask what are the right tasks to perform So this is one image of what it that
[25:50] will look like, right? I'll be able to to look at some of the the automations that I've set up. There are a couple of suggested ones all within that full page Genieode experience.
[26:05] So to recap a couple things that we've talked about one, we've shown how Genieode can help with so many of the things that you do in data bricks. So what I'd really encourage you is to try starting your next task from that genie code side panel or full page experience,
[26:18] right? Describe to it what you want to do in probably in more detail than what I did here, right? You can give it a spec or an a markdown file. You can point it at your Google doc to tell it to build what you described there,
[26:31] right? But just try to drive that workflow as much as possible through the agent. But obviously like it needs to learn how you work, right? So that's what we talk about with making it a tenured co-orker.
[26:44] Instructions are a way to teach it how to perform kind of all the tasks that you do and kind of general conventions around that. Skills are ways to to perform specific workflows. And then the connectors help you bring in that
[26:57] additional context and capabilities. So the way that I work, right, is I start the task with the agent. And along the way, if it gets something wrong, I I just tell it, no, I want that cell to look like this, or I prefer to explore
[27:10] five different models at a time, or I prefer that the dashboard always has a summary at the top. And then at the end of that conversation, once I'm happy with the final artifacts, I tell Genico to create a skill to remember that
[27:23] workflow. And then it creates that skill for me. And the next time I start that workflow, Genica will recognize that it should use that skill and then just perform that task a lot more autonomously.
[27:39] All right, so before we wrap up, let's see. All right, just in time. See, it ran into a couple of issues. It it had one seven out of eight widgets created successfully, but it looks like it now is done. So let
[27:51] me zoom out a bit and we see kind of a nice row at the top with a couple of the metrics and then some of the widgets around delays throughout the day, monthly cancellation rate and so on. And
[28:04] obviously from here I can ask it to publish a dashboard. I can iterate on it. I can tell it how I want that dashboard to really look. All right. So with that I just want to say thank you for listening. I hope
[28:16] Genie code will be useful for you and definitely let us know if you have any feedback. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.