Skip to main content

Genie Code in action - An end-to-end demo.

Summary

  • Autonomous Workflow Execution: Genie Code builds end-to-end ML workflows from a single prompt, handling data exploration, model training, pipeline development, and dashboard creation.
  • Context Awareness: Genie Code maintains full context across notebooks, tables, pipelines, and dashboards, understanding your data and providing informed suggestions.
  • Production Insight Delivery: Move from raw data to production insights faster by leveraging Genie Code's multi-step planning and execution across the Databricks Data and AI platform.

Genie Code in action - An end-to-end demo.

Watch: Genie Code in action - An end-to-end demo.
In this demo, Gal Oshri, Product Manager at Databricks, shows how Genie Code acts as an autonomous AI agent for data engineering, data science, and analytics workflows.
Starting with a single prompt, Genie Code explores datasets, trains a machine learning model, builds a Lakeflow Spark Declarative Pipeline, and creates an AI/BI dashboard. The entire workflow happens from one UI panel inside the Databricks workspace, with the agent maintaining full context across notebooks, pipelines, and dashboards.
This end-to-end walkthrough demonstrates how Genie Code helps data teams move from raw data to production insights faster by planning and executing complex workflows across the Databricks Data and AI platform.
What you’ll see in this demo

  • Discover and explore datasets using natural language
  • Train and evaluate machine learning models
  • Create a Lakeflow Spark Declarative Pipeline
  • Build an AI/BI dashboard for monitoring results
  • Execute multi-step workflows from a single AI agent interface

Watch how Genie Code brings agentic AI to real data work across engineering, science, and analytics.
To learn more about Genie Code:

Chapters

FAQs

What is an agent plan in Genie Code and why is it useful?

An agent plan tracks where Genie Code is progressing in a long-running task, allowing you to follow along and see what it's doing. It's shown in the top right of the UI and makes it easy to monitor multi-hour workloads.

How does Genie Code handle machine learning model training?

In the demo, Genie Code trains an XGBoost model to predict property prices, loads data, does exploratory analysis, performs hyperparameter tuning to find optimal parameters, and logs results to MLflow.

Can Genie Code build multiple models and compare them?

Yes. The demo shows Genie Code comparing XGBoost with Random Forest models. You can ask it to skip certain steps (like hyperparameter tuning) to speed up comparison workflows.

How does Genie Code create dashboards from your analysis?

After completing data exploration and ML analysis, Genie Code creates AI/BI dashboards to visualize insights. It maintains context across all notebooks and results, building dashboards that reflect your analysis.

Full transcript

0:00 Data exploration

Genie code is an autonomous AI partner for data teams. We're going to see how it helps me across data exploration, machine learning, pipeline development, and visualizing insights in a dashboard. We start from an empty notebook with Genie code open. I can reach it by clicking this button in the top right. Let's start by asking for four insights on the Wonder Bricks properties table. You can see that by using the at symbol, I can easily search across assets including tables, notebooks, but also dashboards and others to make it easy for me to select them.
I'm going to also specify that I want to see visualizations. Genie code starts by reading the table structure, understanding the metadata, the columns available, and so on, and also gathering some sample data to help it with this exploration. We see that it created an agent plan towards the top right. This allows it to track where it's progressing in the task and makes it easy for us to follow along as well. This is really great for long-running tasks where the agent is working over several hours so that I can see what it's doing.
It added the first insight to the notebook where it's doing some setup on the Wonder Bricks palette and the first insight. And before executing them, it gives me a chance to review. I can click always allow in the current thread to speed things up a bit. And we see the first insight has executed. Price distribution by property type, new listings by property type, and so on. It has renamed the notebook to convey what we've done here.
And at the end of its response, it summarizes the insights in the chat with links to the cells clearly marking all the different insights that it has gathered. Now we're done with data exploration, let's move to machine learning. Now, before I switch over to the notebook, I want to show you how I built it. So, this is a different tab where I prompted Genie code to train an XGBoost model to predict the price from the properties table and join it with the destinations table.

2:26 Machine learning

It loaded the data, did some exploratory analysis, did some hyperparameter tuning to find the best model parameters, and then trained the model and logged it to MLflow. So, switching back to our original tab, I'm going to open that notebook over here, the properties price prediction. I'm going to say in this notebook, compare the model with random forest. Skip hyperparameter tuning just so that it goes a little bit faster.
Right, so we ideally want it to pick up from the XGBoost analysis that was already done and then do some comparison with a different way to train the model. We can see that Genie code added two cells at the end of the notebook. One for training and evaluating random forest and the final one for model comparison. And it creates some nice visualizations to compare them as well as a clear table in the chat at the end to tell us that XGBoost wins on all the test metrics.
Awesome. So, from here I can ask Genie code to go and deploy the model. I can ask it to do further exploration or even go to a deployed model and help me configure it, optimizing it based on the production traffic that we're seeing. But now I want to switch over to some data engineering. Now, switching tabs again, I want to show a Spark declarative pipeline that I built earlier, again with Genie code. I prompted it to create a medallion architecture pipeline with the samples Wonder Bricks tables.

4:33 Data engineering

I've briefly defined what should be in the bronze, silver, and gold layers and asked it to run the pipeline. And we can see that it created the tables, it validated them, and ran them. It also included this table properties where for each layer it includes the gold, bronze, or silver as the quality. I want to take a a pause here and explain instructions in Genie code. Now, over here I'll open this instructions file that I accessed through the settings.
Instructions are a way for me to teach Genie code about my preferences or conventions. So, here I have the Wonder Bricks brand palette. This is what we saw earlier. For all the visualizations, it set this up and used the colors in the visualizations. For declarative pipelines, it's also asked to include table properties with the quality set to the medallion layer. This is a great way to kind of establish these conventions across all the work that I do.
And but this is just the personal preferences. There's also workspace instructions that an admin can set up to ensure everyone in the workspace kind of follows the same best practices. But let's switch back to the pipeline and I'll switch to the tab where we were training the model. I'll ask to navigate to the Wonder Bricks pipeline editor page so we can continue working on that pipeline and build it out a bit further.

6:14 Feature engineering

And now we're in the pipeline editor. So, this is the right place for me to build Spark declarative pipelines, but I'm still within the same Genie code thread. So, it maintains the context and knows what we've done so far. So, I can now ask it to add a gold feature table for the price prediction model and run it. We can see that it added the gold price prediction features file and now it's updating it with the Spark declarative pipelines code to create it.
It's asking for permission before running or dry running the pipeline to validate things. And now it's asking to run the pipeline update just on the one data set that we created here. Right, so it's automatically being efficient even though I didn't kind of have to specify to just focus on that. So, it looks like the gold feature table is now live and Genie code has summarized the features and the sources that it created in this table.
And from here I can add expectations, I can preview the feature table data, and so on. But now I want to switch over to building a dashboard with some of the insights from the data that we've created here. So, I'm actually going to go and create a new dashboard. I'm going to close the filters tab just to give us a bit more space. And I'm going to ask Genie code to create a dashboard with five visualizations from the gold tables in our pipeline.

8:46 Dashboard creation

We can see that it added a nice header and some charts visualizing insights from our gold tables. And now we see a summary of all the insights. And Genie code has completed its work. So this is just a glimpse into what Genie code can do across data bricks.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.