Skip to main content

Genie Code: Your autonomous AI partner for data engineering, science and analytics.

Summary

  • One Prompt, Full Pipeline: Genie Code builds an entire Medallion architecture (Bronze → Silver → Gold) from a single natural language prompt, handling all SQL and boilerplate.
  • Automated Framework: Spark Declarative Pipelines manage dependencies, retries, scheduling, and orchestration automatically, so you focus on business logic instead of DAG maintenance.
  • Complete Data Workflow: Auto-Ingestion, data transformations, ML model training, and dashboard creation all happen from one agent interface with full context across the Databricks Data and AI platform.

Genie Code: Your autonomous AI partner for data engineering, science and analytics.

Watch: Genie Code: Your autonomous AI partner for data engineering, science and analytics.
In this demo, Frank Munz goes deep on how Genie Code goes from "helpful autocomplete" to something that genuinely understands your catalog, your data, and what the engineers and the business actually needs.
What you'll learn:

  • One Prompt, Full Pipeline — Frank builds an entire Medallion architecture (Bronze → Silver → Gold) from scratch. Just natural language. No boilerplate.
  • Spark Declarative Pipelines (SDP) — Dependencies, retries, scheduling… all handled. You stop babysitting your DAGs and let the framework do the plumbing.
  • Auto-Ingestion — Genie configures Auto Loader for nested JSON and flattens the structure automatically. If you've ever wrestled with deeply nested payloads, you'll appreciate this.
  • AI/BI Dashboards — Instant analytics straight from your lakehouse. And the "Ask Genie" feature turns vague stakeholder questions into queries that return actual answers.
  • Extensibility — Hook in MCP Servers and custom Markdown Skills to push your agent well beyond the defaults.

To learn more about Genie Code:

Chapters

FAQs

What is Spark Declarative Pipelines and why is it suited for Genie Code?

Spark Declarative Pipelines (SDP) is a framework where you define what transformations you want, and the platform handles dependencies, concurrency, retries, security, and scheduling. This declarative approach is a natural fit for an AI agent because it expresses intent rather than implementation.

Can Genie Code automatically flatten complex nested JSON structures?

Yes. Genie Code configures Auto Loader for nested JSON payloads and automatically flattens the structure. This is shown in the demo where customer data with nested address fields is properly ingested and flattened.

What capabilities does Genie Code have beyond data engineering?

This video demonstrates Genie Code building end-to-end workflows: data exploration, ML model training, pipeline development, and AI/BI dashboard creation. It maintains full context across all assets in the Databricks Data and AI platform.

How can you extend Genie Code's capabilities?

Genie Code supports extensibility through MCP Servers and custom Markdown Skills, allowing you to push the agent well beyond default capabilities for specialized use cases.

Full transcript

0:00 Introduction to Genie Code

Hey, I'm, tech marketing at Databricks. Do you know, 2025 was the year of the AI agents, but 2026 is when they actually showed up. At Databricks, we've been building AI-powered tooling for data engineers, data scientists, and analysts, and all of that work has now converged into one product, Genie code. The previous assistant was an LLM at its core, but that helped with the smaller and more isolated tasks. Genie code is something entirely different.
It's not only a tool that you use, it's more a collaborator that understands your data and your catalog and your context. Today, I will focus on the data engineering perspective. I want to show you Genie code working with Spark declarative pipelines, SDP, which turns out to be a natural fit for an agent because of SDP's expressiveness, where you define what you want, and then the platform handles the how, dependencies, concurrency, retries, uh security, and scheduling.
So, it's a declarative approach. And here's the plan. I'll give Genie code a single prompt to build a complete pipeline from scratch. That means auto loader with proper JSON. It means creating a medallion architecture, and it also means creating streaming tables and materialized views wherever they make sense. You name it, and it will be ready to run. So, Genie code acts like a data engineer who's done this a thousand times and who knows all the toolings and all the settings.
Let's kick this off. So, we start in the Databricks workspace. I click on new ETL pipeline, and then you see right away this create pipeline with AI button and it's actually pre-configured. It remembers the last data set. Create a pipeline that reads data from but I want to change that prompt. I want a more challenging uh example for you guys. So, I want to ingest from a volume. I just give it a sloppy name. That's not the exact name and I want to analyze different customer tiers.

1:48 Creating the Data Pipeline

So, it's a data set with customer tiers. And let me switch to the data set. That's VS Code. With my JSON data set, it's a list of customers. If you see each of those customers, it's synthetic data. Um the address itself, it's its own little mini structure. It needs to be flattened. There's a loyalty tier. And there is a total spent per customer. So, I kicked this off. It's actually now starting to create my pipeline.
It gives it a proper name, tier analytics. It's looking for the catalog and schema. I actually found the correct volume name even though I didn't specify the underscores. It tells me about the schema and it's actually even reading now already um the data in that volume to get an impression of how that data looks like. It's a volume. It could be any kind of data. But it understands it's this customer data. It understands the structure.
You see here it's identifying strings and uh doubles. And then it says, "Okay, let me create you this medallion architecture. Bronze for ingestion, silver for flattening, and then the gold layer for aggregation. And I'm using streaming tables and materialized views." And then it says, "Okay, I can do all three pipeline files in parallel." And this is what is happening right now in the pipeline editor on the left-hand side.

3:24 Pipeline Development and Run

You see this navigation bar where we see the files, to Python files, bronze, silver, and gold as they are being generated right now. And now it's updating the files with the proper content. So, we might see that update later if we look into this into these files. Let it think for a second. And it's starting a dry run. So, once it's completed the transformation files, it's going to verify if that is really working. It's asking for permission.
It's actually asking you for any step for permission, but I already told it, "Well, don't ask me that many times." So, this is why you don't see that question too often in that video. It's doing the dry run now, and you see the data sets are available. Dry run means it's not writing uh data. It's not creating new data sets. And then I clicked on, well, start the pipeline because I want to see the pipeline. And while the pipeline is starting up, let's look at these files.
You see this is the change. I'll just accept it. It's using import pipelines from PySpark. So, it's proper Apache SDP DBT DP table. Sorry about that. DB table uh for streaming tables, and then it's reading the data using autoloader. And look at this, it's properly configuring autoloader for multi-line uh it specifies the volume. Lot of manual work if you want to do this uh yourself. Silver layer using a materialized view uh for the flattening.
Look at this, how beautifully it it accesses the address, address. street, address. city. So, that works perfectly well. Never told it it needs to do that, it just figured that out itself. And then the gold tier for the um aggregation. I think that's pretty impressive. That's the pipeline, the pipeline graph. We can run the pipeline, navigate to the code, get the data preview again. That's the bronze layer. Remember that was like the JSON input that we had that was read by the bronze layer.

5:49 Data Transformation Results

Um this is the gold layer. We see the loyalty tiers. Silver, gold, platinum. Let's sort this by revenue. Yeah, you see the total revenue is the highest where the average spend is the highest. That makes sense. That's a good news. Isn't that amazing? That was Genie code generating a full pipeline. Everything from auto loader specification and configuration to medallion architecture, testing it, and running it. Now, there are so many more things we can convert Python to SQL uh transformation.
We can explain the data flow. Use it for Kafka, using it for additional MVs for analytics. Convert the old DLT style to the new Spark declarative pipeline syntax. Try it yourself. Well, I told you this would be about data engineering, but I can't wait to show you this as well. You can also use it for dashboards. Here I'm creating a dashboard based on my gold table. I just say, "Look, give me a dashboard for analytics on top of the gold table."

6:57 Dashboard Generation

And it already kicks it off. I allow it to do it. That's the AI BI dashboard canvas. And this is what it comes up with. That's the graphical representation of our gold table. Let me change that graph just to show you it's possible. And that's the published dashboard. Now, in that dashboard, the cool thing is it's not only the static data that you see in the dashboard. Again, we can click on this ask Genie button and then I can basically ask it any question related to the available data sets.

7:31 Querying Dashboard Data

Um, so I could say like this high-level manager kind of question, uh, what is what is how much better is the platinum tier versus the gold tier? It's a very vague question like how would you say better? Like how do you specify better? I still ask that question. Let's see what it comes up with. If you look at the data in the middle, you can kind of guess what is the relation from platinum to gold. It's analyzing the different tiers, platinum and gold.
Looking for customer counts and offers 3. 4 times more valuable as platinum. That's a pretty good business answer in my opinion for a pretty vague question that I asked the system. So, I like it. I'll give it a thumbs up. Okay. So, if you work in data engineering, data science, analytics, you should have a good idea by now how Genie code supports you in, you know, this mechanical work that, you know, just keeps you busy, but there is more to Genie code.

8:42 Extending Genie Code

You can extend it with MCP server. Could be an external third-party MCP server. It could be a custom one hosted as a Databricks app. And all this tooling is discovered dynamically at runtime. And then you can write your own skills. You can create a skill. md file with a step-by-step instruction in markdown. And again, it's loaded automatically, uh, when it's relevant. Having said that, I want to say thank you for joining me today and, uh, talk to you soon.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.