Building End-to-End ML Systems With Genie Code

Summary

This video explores how Genie Code integrates with the full machine learning lifecycle on Databricks — from exploratory data analysis and feature engineering through model development, evaluation, deployment, and production monitoring.
Genie Code uses Unity Catalog metadata, business semantics, governance policies, and data lineage to provide context-aware ML assistance, enabling data scientists to move faster and work with greater confidence on complex ML projects.
A live demo builds a fraud detection model end to end using Genie Code, covering training, model serving, endpoint validation, and predictions, with best practices for customizing Genie Code with team-specific skills and instructions.

Building End-to-End ML Systems With Genie Code

Coding agents are transforming how data science teams build and operate machine learning systems. In this video, we'll explore how teams use Genie Code across the ML lifecycle—across EDA, feature engineering, model development, evaluation, deployment, and monitoring. We'll also provide an inside look at how Genie Code was built and optimized for ML workflows. Genie Code understands your data through Unity Catalog and leverages metadata, business semantics, governance policies, and lineage to provide context-aware assistance. Deep integration across the Databricks ML stack enables teams to move faster and work with greater confidence. We'll then look ahead at how Genie Code is evolving from a coding assistant into an agent that can help deploy, serve, and operate ML systems in production. You'll also learn how to customize Genie Code for your ML team through skills and instructions, as well as other tips and tricks to get the most out of Genie Code. Attendees will leave with practical examples of how coding agents are reshaping machine learning and actionable guidance for applying these capabilities to their own ML projects.
Talk By: Nemanja Petrović, Principal Software Engineer, Databricks ; Tejas Sundaresan, Staff Product Manager, Databricks ;

Chapters

FAQs

What is Genie Code for machine learning in Databricks?

Genie Code is an AI coding agent embedded in the Databricks workspace that understands ML context through Unity Catalog metadata, business semantics, and data lineage. This video shows how it assists with the full ML lifecycle — from exploratory data analysis and feature engineering through training, deployment, and production monitoring.

How does Genie Code support automated model evaluation?

Genie Code supports automated model evaluation using scoring rubrics and LLM-as-judge approaches that are calibrated to align with human evaluation scores. This video explains how teams can set up evaluation pipelines that reflect business-defined quality standards rather than relying solely on generic benchmark metrics.

How do you customize Genie Code for an ML team?

Genie Code can be configured with team-specific skills and instructions that encode the conventions, tools, and domain knowledge of your ML organization. This video shares best practices for customizing Genie Code to follow your team's modeling patterns and deliver more relevant, context-aware assistance across projects.

What ML operations tasks can Genie Code handle in production?

Beyond development assistance, Genie Code is evolving into a production ML operator capable of monitoring model endpoints, detecting performance degradation, and triggering alerts or remediation workflows. This video previews how it can help teams manage deployed ML systems without constant manual oversight.

Full transcript

[00:07] Hi everyone, nice to meet you all. My name is Tis. I'm a product manager at data bricks. >> I'm Nemana, principal engineer at data bricks. >> And we're very very excited to share about Genieode and how you can use it to
[00:19] build end-to-end machine learning systems on data bricks. Uh I think many of you might have seen the keynote announcement we made on Genieode for ML. And this is uh a next level of detail on how Genieode works under the hood to help you with machine learning projects,
[00:32] why we're so excited about how it'll accelerate the ML life cycle. And then we're going to cover a live demo as well. And just as a standard, safe harbor, our standard language on forward-looking statements.
[00:45] And another reminder, please uh complete your surveys. Your feedback on these presentations are super helpful for us to inform the content of next year's content. So please uh we really appreciate your feedback.
[00:58] And so we'll cover five topics today. So one is we'll talk about why agents have really kind of changed how machine learning development is happening. And then based on kind of that discussion we'll also introduce Genie code for
[01:10] machine learning. Uh we'll and uh in the middle we'll do a quick demo kind of covering training to serving using Genie code. Uh then we're going to cover some best practices that we're seeing customers adopt around Genie code for their machine learning teams and
[01:22] projects and then we're going to give a quick preview on what's coming next. So first why is now like uh just the agentic kind of moment for machine
[01:34] learning today you know after 10 years of growth in the industry ML is everywhere. It's there in fraud detection. It's in scoring and revenue forecasting customer health prediction. Every organization on the planet has
[01:48] many many machine learning applications that they either could develop or are already developed and so it's ubiquitous across the industry. But we're also when we speak with customers around their machine learning
[02:00] needs, one clear pain point continues to come up which is that they have a giant backlog of ideas and experiments they would love to try but don't have the bandwidth to actually pursue them. And just here's a standard example. You
[02:13] know, say I'm an e-commerce store. Maybe I have a customer LTV prediction out and maybe a sales forecast to help report on numbers to the executive team. But there's so many models that I haven't
[02:25] been able to create. I haven't been able to create a ranking model for the homepage or figure out how to dynamically set promotions to drive more value in the business. Maybe there's fraud on my system and I'm not able to detect it as effectively as I'd like.
[02:37] There are stockouts in my supply chain. there's potential customer turnurn that I'm not tracking and trying to reduce. And lastly, maybe I'm not optimizing my pricing. So, every single ML organization has many, many more ideas
[02:50] they would love to try and operationalize, but they're not able to. And that's where we really think agents can help. But the work to actually operationalize machine learning, and Mike touched upon this in the keynote, it goes way beyond
[03:03] just the code you're developing. you know the ML code and this is the paper from 2015 referenced in the keynote uh talking about the hidden technical debt of machine learning systems. The actual code to encode an sklearn classifier or
[03:16] a pietorched transformer model is really small compared to the broader work needed to actually operationalize machine learning with high quality. You have to figure out how to do data collection, data verification, feature
[03:28] engineering, figure out how to make sure that you serve these models effectively, how you're monitoring your serving to avoid drift. And each one of these components is almost as uh work intensive if not more than just setting
[03:40] up the training code. And so that's why when you think about uh accelerating machine learning with agents, there's three levels of context that need to be brought into the agent
[03:52] that coding uh just raw code alone does not contain. So one is context on the data. So you want your agents to really understand the quality, the freshness, the integrity of the data you will be
[04:04] training on because oftentimes you can get a machine learning model that looks sort of right, but then the features themselves might leak information, might not handle handle class imbalance correctly, and therefore the quality of the result isn't what's needed to
[04:17] actually get a model into production. The second is that models are getting larger and more complex and so you need the agent to actually have context on the infrastructure it's operating on. And so for example, uh if you're running
[04:29] say a PyTorch model, it'll need GPUs, but how do you know that you're using the GPUs effectively or you're using the right number or the right type? How do you know that your model that's getting served is actually reliably handling node failures and so forth? So there's a
[04:42] lot of context on the infrastructure that an agent needs to actually effectuate the full machine learning life cycle. And then lastly is on your team's workflows. So every team works slightly differently but every team also
[04:55] has really really important workflows that need to be done on time. So a great example would be drift detection which we shared in the zero ops demo and keynote. You want to make sure that the agents that are operating with your machine learning infrastructure know
[05:08] when you actually need to trigger a new retraining run based on drift or other key things that you have in your workflow.
[05:20] And so with that uh these three context areas in mind, we're super excited to introduce Genieode for machine learning. So Genieode for ML has three kind of large components that address each of these challenges. One is we've upgraded
[05:32] the intelligence for Genieode particularly for machine learning engineering. So it has all the best practices for how to do machine learning on data bricks built in. Second is it's natively integrated with both Unity
[05:45] catalog, the broader data bicks data platform, but also every single machine learning infrastructure component on data bricks. Whether that's your training clusters or your serverless or ML flow, your serving endpoints, your
[05:58] feature store, all of these components now seamlessly work with Genie code. So you have an agent that can actually operate across systems and not just be tied to a single system. And then lastly, it's p powered by geneontology,
[06:10] which Ken introduced yesterday. Geneontology helps ensure that these agents are really focusing on using the best quality data. They're not spending tokens just searching a really, really
[06:22] complex data landscape and they're really targeted towards getting the highquality rich signal that you need to make machine learning really effective in the enterprise. And just a little bit more about
[06:34] geneontology. So you can imagine it's not just about the code you're working on. It's about the governance principles behind your data, the business metrics, you know, something like LTV, how do we make sure we're getting the right prediction for the right metric that
[06:47] your business actually uses and so forth. Uh how what experiments you've run before, how do you compare results across experiments? So there's a broader tapestry and ontology of everything about how your machine learning team
[06:59] operates that Genie Code actually understands because of this ontology. And so with that I'll actually turn it over to Nemana to share about how we built Genieode. Thank you Tros.
[07:16] So um one of the things about Geniode is the level of integration uh with all the surfaces and the data on the data bricks. So Giode the way it was built is spanning the whole ML life cycle. If you start with a bookish way of doing
[07:29] machine learning and if you can enumerate all the steps which are needed starting with exploratory data analysis all the way to deploying your model and doing MLOps, Genode is aware of all those steps. So it's spanning multiple
[07:42] data brick surfaces. It's integrated not only with notebooks but also with ML flow and model serving endpoints and it also spans the best ML practices. So here is just a small sample of the
[07:55] surfaces on the data bricks that GI code is integrated with. It has a very tight integration with MLflow. So MLflow knows about your runs and the models and practices. It's integrated with unity
[08:07] catalog and the unity catalog you get the control and the governance and also your data which is unity catalog is subject to many other queries
[08:19] from the SQL from the notebooks from other processes within data bricks and from that collective knowledge we know the semantics of the table and semantics of the columns. So later that can help
[08:32] us very much in feature engineering and understanding what the columns are. Once the model is trained, it's really useless unless you deploy it. So we have a model serving endpoints where you can
[08:44] deploy your models and they can be served on CPUs or GPUs. They have auxiliary inference tables where you can put and log all of your requests which can help you later on in
[08:56] MLOps. We support serverless CPU, serverless GPU or your own custo custom clusters and actually GI code has a lot of intelligence and lot of uh hints that
[09:09] provides to the users suggesting the best type of the compute depending on your data and the workflow that you're working with. We are tightly integrated with the feature store and that's a declarative feature store. So in most
[09:21] cases you don't need to actually write your features. you're describing what is the desired outcome. So the feature simply becomes an expression built by us. And finally if you need to change
[09:33] anything in your data engineering in your pipelines lakeflow is as the agent is also the part of the of the genic code. So uh a couple of examples over here. Genie code can help you build
[09:47] features and we hear from many of you that not counting ML operations some of you are spending 60 70 80% of a time searching for the new features your tables which are holding your
[09:59] training data are not living in isolation so because you query them from SQL or from dashboards and we have the whole history of how those tables have been used we have built very rich
[10:11] metadata on a top of those delta tables So we have a very good understanding what each of the columns potential feature in those tables represent.
[10:23] We also have horizontal awareness. So we understand what are other Python files notebooks within your workspace. We also understand what your repository is. So, Geniode as the agent can learn
[10:38] what are the best practices or the commonalities that your teams are having and incorporate them when it's generating the code or doing the feature engineering. It's tightly coupled with MLflow.
[10:51] So it's going to be evaluating deploying the models the way that you are doing and every model run is going to be evaluated using uh the same tests that you're
[11:03] accustomed to use or the same test that you are that you're running. Uh once the model is trained with genic code is making it really really easy.
[11:16] The most difficult remaining part are actually the evolations. You don't want to push the model which is not really living up to your standards.
[11:28] regressing over the time. And there are many many reasons why your model is going to be regressing. Imagine there is a change in a frontier model which is powering gen code that's going to affect the quality of the model which is being trained. Or maybe that
[11:41] model is being throttled in a certain part of a day. So the quality of the model goes down. That means that the quality of the output the genic code is returning is also going to go down. Then you may change many of the things in
[11:53] your skills in your prompts in your tooling and you want to know at any given time if the quality of your model is regressing. You don't want to use anecdotal evidences. You want to have something which is more systematic. So
[12:05] the way we're going to do this is we're going to start with a scoring criteria or rubrics written in plain English saying what the good model looks like along multiple directions. What is the
[12:17] bad model? What is the average model? What is the outstanding model? And we gave those scoring criteria, those rubrics to the human raers who then produced the adilations, the scores
[12:30] together with justifications for those scores. And because this process is very very cumbersome, LLM judges are not nice to have. They are the must nowadays. So what we did,
[12:43] we gave as you would do it the same scoring criteria, the same rubrics to LLM judges who would then produce their own scores and justifications. And this right path is basically a must today
[12:55] considering how with genic code the level of authoring of machine learning models is going to increase. the number of models that you're going to generate is just going to be so hard that no level of human evaluation only is going
[13:08] to be able to scale up. But the problem which appears in that is misalignment between human and machine scores or justifications. Simply LLM judges are missing the human
[13:21] level of nuances when they're doing the evaluations and then those scores are not really in sync. So you don't really feel comfortable to deploy this right path. So the what we did here is that we
[13:34] decided to separate the scoring rubrics for the humans and for the machines. Initially they're going to be the same and as we've noticed the ultimate scores and justifications are going to diverge.
[13:46] And then in collaboration between data bricks and UC Berkeley two very powerful techniques were developed in the last several years. One is called DSPI, another one is called GAPA or Jeepa
[13:58] which are prompt evolution, prompt optimization techniques. So here on the right side there is actually a loop, a feedback loop that once we notice the misalignment between humans and the machines. We can go and iterate and
[14:12] change and evolve the scoring rubrics for the machines until ultimately we reach the alignment. At that moment left and right tower here in this evolation pipeline are the same. So you can remove
[14:25] the humans at least for a while from this evolation pipeline and rely only on LM judges. But what we noticed is that these two techniques even very very powerful require huge number of training samples
[14:38] and doing human evalations on these notebooks is one probably of the most difficult human evalation task in computer science. So not only we have to evaluate the quality of the code,
[14:50] we also have to evaluate if the best ML practices have been followed. If ML training is congruent with the type of the data, one methodology and one algorithm is going to work for one
[15:02] data distribution and the one is going to be working for another data distribution. And ultimately when the model is trained, we want to evaluate if that model is really really doing well. So that task is extremely cumbersome and
[15:14] in most cases people would not be able to evaluate more than 50 or 100 of such notebooks in any reasonable time for teams of the reasonable size.
[15:26] But there is another innovation which came out of data bricks specifically from MLflow team. This is a new methodology for aligning judges with the humans which is called memaligned. It was released a couple of months ago. There is a good blog post about that.
[15:39] It's open sourced. It's a Python library which is part of MLflow. So you can use it at any time. The key design criteria for meline is that it must produce the alignment
[15:51] between LLM judges in the humans using a very small number of samples which is exactly the use case that we have. And here is the ultimately the result.
[16:03] We are measuring the mean absolute error between the humans and the LLM judges before applying memine and after applying mammal line and the lower is
[16:16] the better. So these dots which are marked in gray are the level of misalignment before we apply malign and then in most cases those dots have moved to the left to the lower region which
[16:29] means we achieve tremendously higher level of alignment between humans and judges using the MML line. Once this happens you can be reasonably confident that your LLM judges are going to be
[16:41] doing the right job. So you can have CI/CD, you can have real-time alerting. you can be pretty sure that judges will catch any of the regressions and then over the time possibly there's going to be
[16:53] another level of divergence and then you can do another quick round of alignment and have judges go back uh to this full alignment. So which means we are also trying to help you with one of the most
[17:07] tedious and interactable tasks which is evaluation of the code and appropriate algorithms and finally uh the models that we produced and thankfully because of the memo line we have this very very
[17:20] powerful technique so how we can train and tune LLM judges to do this instead of the humans.
[17:32] So here's one of the quotes because it understands production ML workflows on data bricks genic code help us move from raw data to govern tables tracked models and a production ready serving endpoint in 90 minutes. That's a
[17:46] quote for a real customer. And now I'm going to give the mic back to TAS who's going to show you a demo GI code in action. >> Thanks Dana. So, always fun live demos
[17:59] with agents are a little bit tricky, but we're gonna try. Uh, can everyone see my screen here? Perfect. I'm going to zoom in. So, here we have a pretty standard machine learning application trying to identify
[18:13] fraud in credit card transactions. And of course, this is all synthetic data. Uh, so we have data like transaction date, time, the card uh amount used, uh you know, what merchant category and so
[18:25] forth. And so with this data in mind, let's go ahead and start using Genieode to actually build a machine learning model that can do this for us. So I'm going to open up a notebook. Uh and here you can see Genie Code is on the side
[18:37] just there to help me write this code. So I'm going to actually go in and ask Genie Code to train a classification model. And in this case, actually I can even specify exactly the table I need.
[18:49] So I can go in and just say I want the transactions data to detect fraud. And so this is a great example where Genieode has the full context of your Unity catalog spec. So you have uh you
[19:01] know that you're using and building this model on govern data. So we're going tohead go ahead and start having Geniode run. And let's probe a little bit into what it's thinking. So you can see the first thing it does is it reads a skill
[19:14] that's oriented around machine learning. And you can see it's loaded the skill details and the machine learning skill is invoked. And this is exactly how we embed a lot of best practices for machine learning on uh within data
[19:26] bricks directly within Genie code. So you can see the steps it's taking. It's going to go ahead and run initial exploratory data analysis. It's going to do feature engineering. It's going to train the model and then evaluate and
[19:38] then register it all to MLflow. So all the key steps that uh the data scientist has to do on data bricks is now scaffolded within Genie code.
[19:50] And the other thing you'll see is that it's actually creating and getting a sample of the data to actually understand it better. And this is how it helps shape its plan on how it actually approaches the machine learning problem. So a great example and this and you know just for the purposes of time actually
[20:02] pre-ran this 10 minutes before the uh presentation here you can see what we've done is we're training this classification model and the first thing that it actually orients around that's specific to this task is class imbalance. And that's because it
[20:15] knows that fraud detection is a very very tricky needle in a haststack problem. You might have like 1% 2% fraudulent transactions over 100%. And so you need to have a classifier that
[20:27] actually takes that class imbalance into account. And so that's what it's doing in this case. It's making sure that the model that we're creating actually reflects the core business task that we're trying to do. So it's done a lot uh and I won't go into all of it, but
[20:39] you can see it's gone ahead and trained this full model. It's actually hill climbed a little bit to identify an opportunity to improve the performance and it's done that for us as well and you can see it's summarized our key findings here. So you know this model is
[20:52] good but not great but you can see we at least have an initial kind of scaffold up and running and on top of that it's also produced a lot of analyses for us too by default. So one for example is it's produced a
[21:05] feature correlation metrics because another thing we really care about within fraud detection is okay is there any leakage between a one of our features and the actual label itself and it's kind of gone ahead and verified that that's not the case
[21:17] and you can see it's fully trained the model it's set up the training and test split for us and then it's you know used MLflow it's actually written all of our ML flow code for us and um you can see everything is up and running now so
[21:29] that's step one you know we've authored the code that goes into training this But Genie code is not just oriented around the notebook itself, but it it's available to you across the full ML life cycle. So here it created a MLflow
[21:41] experiment. Let's actually go in and see what that experiment was. So here it's actually pre already created all the MLflow code to track these metrics. And now I can actually use Genie Code directly within MLFlow to suggest
[21:53] improvements to this model and suggest improvements to how we evaluate it. So here I've asked Uni code to actually analyze this experiment and let me know what other metrics I should be checking for a credit card transactions model.
[22:05] You can see it analyzes the three different runs we did and identifies a few interesting metrics that I hadn't considered before. So one is fraud actually matters not just from a core individual row level but like how much
[22:17] aggregate fraud fraud is my team facing. And so you can imagine a new metric I should consider is dollar- weighted fraud detection rate. It also identifies specific metrics that are great for, you know, uh, highly
[22:30] skewed data. So things like breer scores and so forth. And so it identifies, you know, prioritizes two or three uh changes for me. And even within this exact MLFlow context, I can go in and just say, hey, let's add one of these to
[22:42] my notebook. And you can see it goes back and because it has that full ontology for what notebook this machine learning experiment was running on, it goes in and just edits it for me. So it's seamless. I don't have to switch tabs. I can just get everything up and
[22:55] running very quickly. So this is on the experimentation side. But then every model not just has to be trained but then it has to be deployed. And so I've gone ahead right before this talk and deployed it to model serving.
[23:10] And you can see here essentially I also ran a few sample requests against this endpoint. You can see right here I just uh deployed this fraud detection model and then send a bunch of requests. And Genieode can actually go into my live serving endpoint and help me analyze the
[23:23] performance and identify optimizations. So I ran this analyze endpoint function. It has suggested a few optimizations and you can see some examples. One is that okay this traffic is you know just a few requests here. So it's relatively low.
[23:35] I've clearly overprovisioned this endpoint because you know I'm only at 1% traffic. Uh and so these are just examples where Genieode because it has full context over the model, the infrastructure and the full ontology of
[23:48] how you work within data bricks can actually handle the entire machine learning experience end to end and be there as an assistant across all of them.
[24:02] So we'll dive back into slides and I'll turn it over back to Namja to talk about a few best practices with Genieode. So a few things that GI code for ML is allowing you is to
[24:15] customize and automate your work the most uh the the way you want it. So one of the things that we provide our integration with NCP servers. So for example if you want your model to alert you if something goes wrong and alert
[24:27] you in Slack you can do it through this MCP Slack server. If you want to pull any of the business semantics which is needed to build your features or train your model out of Google Docs, you can
[24:39] also do this from out of Google Docs. And if you want to track the issues or have the tickets in Jira on your favorite system, you can also do that. So Genicode engine code for ML are providing these integrations with multiple MCP servers.
[24:52] When you're developing code or models, very often you're going to have some of your best practices. for example, the code style or the favorite libraries that you're using.
[25:04] All of those you can put into the markup files, instruction files, and you can upload them into your GI code environment. And then if you want to extend the intelligence of Genode with
[25:16] specific skills which are just a markup files which can be loaded upon demand when needed, you can also build your own skills and upload them to Genie Code. And that way you can either augment Genode or change the behavior that suits
[25:30] your needs when it's needed. And finally, Geniode is having a very good awareness of your workspace, of your environment, of your compute and runtime, all of
[25:42] those contextes. In this case, what we are showing is that by typing at sign, it pops up a new window for autocomplete and it has a full awareness of your cataloges and schemas and tables and
[25:55] other artifacts that you're having in your workspace. So you don't have to switch the tabs and look for that particular table. You can start typing and as you start typing these choices are going to be further further uh
[26:07] refined. It knows what type of the runtime you're currently on. It knows what type of cluster you're currently running which is attached to your notebook and it also knows a lot about your Python environment. So with all of
[26:21] this it can either make the choices or provide you the hints how you can move into the into the in the right direction. So these are some of the best practices of how you can customize and uh build your own fl bring your own
[26:33] flavor into the genie code uh in a very um systematic and repeatable way for you and um other of your colleagues. And with this I'm going back to Tjas who's going to be talking about the future.
[26:46] Yes. So what's coming ahead? So we talked about building and authoring today, but building is only really half the loop that goes into machine learning. And we also talked a
[26:58] little bit about this in the keynote. And so we're really excited to announce is genie zero ops for machine learning. So what it can do is especially as you're authoring, you know, you start authoring a lot more models now with agents. You need then agents to actually
[27:11] go in and operate them and maintain them for you. And so first is it watches every model in production continuously. So you have the ability to essentially set alerts on model metrics and KPIs on infrastructure and Genie Zero Ops is
[27:24] there just to detect any issue and start actually investigating it for you. So it goes in and actually identifies opportunities to diagnose to repair infrastructure. And then lastly, you can think of it as a as central hub for all
[27:37] of your operational metrics around your machine learning project. So there you know you have agents that can individually inspect every metric and track the progress of each. And so we shared a little bit about that in the demo at keynote. So we're really excited
[27:49] for this to come out later this summer. And the other key area for us uh going forward is how do we expand Genie code and Genie into machine learning functions that anyone in your
[28:02] organization can query. And so for example, we've already launched forecasting within Genie and that was demoed at keynote yesterday. And we have many more machine learning functions that are coming to the Genie experience within data bricks. So anyone in your
[28:14] organization can leverage the power of these models. So thanks everyone for attending the talk today. So just a few kind of key things to recap. One is agents are
[28:26] really accelerating the ML life cycle and so everyone should be using them to help. Uh second is Genie code for machine learning is now generally available. And then thirdly is Genie zero ops for ML is coming soon. So thank you all for attending the talk today.
[28:39] Um, here's a QR code that links to Genieode documentation and we're really excited to see what all you will build with Genie Code. Thank you all.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.