Streaming NASA Data with Lakeflow and Agent Bricks: A Data Engineer's Guide to Databricks AI

Summary

This video walks through a streaming application on the Databricks Data and AI platform that ingests live NASA cosmic event alerts from NASA's GCN public Kafka topic using Lakeflow Spark Declarative Pipelines.
Agent Bricks Knowledge Assistant condensed what previously required 370 lines of custom RAG code into a much simpler configuration, dramatically reducing development time for an AI-powered knowledge base about cosmic events.
The second half compares Time to Value tools including Genie Code, Claude Code, and the AI Dev Kit, each capable of taking a data engineer from zero to a working AI application in days rather than months.

Streaming NASA Data with Lakeflow and Agent Bricks: A Data Engineer's Guide to Databricks AI

This Data + AI Summit session walks you through a streaming application that analyzes live NASA space data using Lakeflow and Agent Bricks. Lakeflow is the foundation layer of every AI application. I explain how Lakeflow Spark Declarative Pipelines (SDP) ingests from one of the very few public Kafka topics that exist and how it works in symbiosis with Agent Bricks to turn real-time NASA circulars about cosmic events into an AI-powered knowledge base with Databricks Apps UI. The AI stack covers AI Gateway, guardrails, Vector Search, agent evaluation, and MLflow tracing, all from a data engineer's perspective. The second part focuses on Time to Value. I used several Databricks AI tools: Genie Code, Claude Code, the AI Dev Kit, and others. Each capable of taking you from zero to a working application in days rather than months. I share the trade-offs and best practices learned. Grab the full application from GitHub afterward — and play with data from exploding supernovas.
Talk By: Frank Munz, Principal TM Engineer, Databricks ;

Chapters

FAQs

What is NASA's GCN and why is it used in this Databricks streaming demo?

NASA's GCN (General Coordinates Network) publishes news circulars about cosmic events such as exploding supernovas and is one of the few publicly accessible Kafka topics. This makes it an ideal real-world data source for demonstrating Lakeflow Spark Declarative Pipeline ingestion in a streaming AI application.

How does Agent Bricks replace custom RAG code in this architecture?

Agent Bricks Knowledge Assistant provides a fully managed, configurable knowledge base that replaces approximately 370 lines of custom RAG code covering chunking, vector database setup, and retrieval logic. The simplification lets data engineers focus on pipeline and business logic rather than infrastructure plumbing.

What AI tools does Frank Munz compare for Time to Value on Databricks?

The video compares Genie Code, Claude Code, the AI Dev Kit, and other Databricks AI tools based on trade-offs and best practices learned while rebuilding the NASA streaming application. Each tool is evaluated on how quickly a data engineer can go from zero to a working AI application.

What AI capabilities are integrated within the Lakeflow pipeline itself?

The Lakeflow Spark Declarative Pipeline uses Databricks AI functions to classify, extract, and translate content as data flows from the Kafka topic. AI Gateway, guardrails, Vector Search, agent evaluation, and MLflow tracing are also part of the overall AI stack described in this video.

Full transcript

[00:09] Thank you. All right, few more people coming. Good afternoon everyone. Thank you for joining um this presentation. Um this presentation actually goes back to two and a half years ago. I've been to a
[00:22] Kafka conference and um Judith from NASA was doing a presentation about how they use Kafka topics for astronomy and I was watching this presentation and I thought
[00:35] like wow this is so amazing and it lasted like 40 minutes and then after 40 40 minutes I thought like okay but this is just you know the data coming in and being in a topic and how NASA is using
[00:47] it but how about all the rest you know how about the data process processing and the cleaning and the data analytics and the AI you want to apply on top of that and I thought like this is like
[00:59] missing like this is for me the most important part so I thought like um I should do a talk like where I do this other 95% that is interesting to me now the big disappointment is I've done that
[01:12] talk already so I don't know why you're here I've done that talk two years ago and that talk was like at the time when everyone one was talking about rack and rack applications and um so I I did this
[01:25] talk like how using this NASA data with a rack application for AI and then two years later like earlier this year I thought like we should revisit that because so many things have changed and
[01:38] then I submitted it for the summit I got accepted and now I'm here um when I prepared for that talk it was actually even more dramatic than I thought so the improvements have um been bigger than I
[01:51] expected. So that's a bit setting the stage for what's happening. So this is you can't sue me for any forward-looking statements and um so yeah two years ago when I did this presentation um I built
[02:04] such a rack application to explore this NASA data. I'll tell you a bit more like they they they use a small um they use a system which is called GCN for circular
[02:16] for news updates where people can send news updates and I wanted to convert this to a rack application did that I got help from a friend from a colleague of mine um who did a very different
[02:28] project but he was like look I have two notebooks and you can use those two notebooks and they should help you a lot. So I used his notebooks and I thought like wow if I had to do those two notebooks myself and uh you see it
[02:42] was like 36,000 circulars are ingested. So if I had to do this notebook myself with the vector database and you know you have a huge amount of data you need to chunk the data into pieces you need to decide how big those pieces are. Then
[02:55] you need to embed them into this vector space and then you have to create the vector database and then you need the chat application etc. So I felt like if I had to do this myself, it would have taken me maybe a week or two. And then I
[03:09] was very um ungrateful. I thought like now I have this notebook, but looking at this notebook, I had to change like I don't know two three lines out of 370. I thought like wow deii people this is
[03:23] so much boilerplate. And all this happened before we even had um ancient bricks. So, and then Asian bricks came not because of my posting that I put on LinkedIn. Um, but it came. It's also not
[03:36] because people would listen to me. Um, but the the gut feeling that what we did two years ago um can be replaced by something much better these days, be it
[03:48] agent breaks or be genie um today. Um was kind of right. Okay. So, I want to give you a little bit more background. Um it's all about this GCN. GCN stands
[04:00] for general coordinates network. GCN it's actually two parts uh two very interesting parts. So in this talk like two years ago I focused a lot on this space part. So the space part means
[04:13] there is stuff happening in space that we normally don't see with our eyes. There's like supernovas exploding. There's gamma rays happening. But there's also other things happening that you that are very difficult to detect
[04:25] where they built like a a huge array of detectors around a cubic kilometer of ice. So literally and Antarct in Antarctica they kind of designed this
[04:37] cubic kilometer of ice and they put like thousands of detectors around it to detect those nutrinos. Now the story behind this is it's from machines capturing those events and it's built
[04:49] for machines so that if something is going on like if there's a supernova exploding one of these satellites might detect it it puts it into a Kafka topic and then all the other like the um the
[05:02] satellites and all the telescopes have time possibly and rearrange readjust and point to this event if they have time if they think the priority is high enough. So that's the mechanical part. But then there's also a more human part where
[05:16] people say look we just measure this and we think this is very exciting and we want to share the news with you. It's more like like email basically. So it sounds like email and um this is the part that I want to focus on today. Also
[05:30] want to focus on this today because it has changed a lot. Um when I looked into this email human part two years ago, it was like you could download these things and then I downloaded them and I thought
[05:42] like that's very uncool you know to download them as a zip file and put them in a datab bricks volume which is a UC governed um folder at the end and now they're provided as uh in a Kafka topic.
[05:54] So I thought this is much cooler. I can access them via Kafka and I wanted to try that and then I wanted to work with this data and this is what this talk is about. So it's all about GCN this general coordinates network and um the
[06:08] circulus they end up in a topic which is called GCN circular. It's a public Kafka cluster which is like super super unique. Um, normally people like me
[06:20] standing on this side of the uh of the presentation, we always like these live data streams. You know, they're not artificial. It's not a load generator Python script that is running in a notebook, but real life data that we can
[06:34] access. And honestly, there's not that much. There's lots of APIs, but there's almost no public Kafka server because like, yeah, why would you do that with Kafka? Well, NASA did it. They can do whatever they want. They did it and they
[06:48] offer these circulars in a topic which is called GCN.US circulus and they have a quick start guide you get a snippet of code to ingest this data and um so if you use if you build such prototypes if
[07:00] you want to play with lakeflow um you're super welcome to use the same example and um I want to focus on lakeflow a little bit and then um talk about AI how can we analyze this data so you heard it
[07:14] today in the keynote lakeflow it's connect for ingestion it's declarative pipelines for transformations um which is now open sourced as SDP spark declarative pipelines and then chop for orchestration. So everything that is
[07:27] running in production if you talk to one of one of our SAS they will tell you well it's always a chop. So that's the the core thing to make this to put this into production. It's declarative it's easy to write and I think from the
[07:40] keynote I think it was like a nice um comment. the the reason because these spark declarative pipelines are these highlevel declarative is a good reason
[07:53] um for um to make them very usable for AI. So then AI doesn't have to generate a lot of code with a lot of parameters. It basically does something like add table and you have a streaming table. Uh
[08:05] and then it's open sourced so you can use this with pispark uh and ES code if you wanted. Okay. And then you need to get the data in and getting data from Kafka into into something always seems
[08:18] to be complicated. I remember in my previous life I did a tutorial once like 40 minutes getting started with some vendors Kafka uh service and then week
[08:30] after week after week people kept writing me emails and like Frank you told us about the back end about the Kafka cluster and the topics and the partitions and the replication and all this but how do I write a client to get
[08:42] this in now here in data bricks if you do pipelines you can do this in SQL which sounds like wow how what you do in SQL a connection to Kafka and it's basically create streaming table and
[08:55] then from stream read Kafka and then of course you need all these parameters and I always feel like I need to say I I apologize for all the amount of parameters that you need but this is not related to data bricks this is related
[09:08] to the kind of security that is uh used this ooth um system um yeah and that's it so before I go to knowledge assistant I want to show you the pipeline that I built it's not
[09:21] spectacular and honestly I could have made well I shouldn't say it's not spectacular if four or 500 people are showing up here it's not super complex that's what I want to say um it starts here want to show you the code for that
[09:34] that's actually the SQL code create streaming table that's for these raw um space events and then I basically specify the Kafka server and these circulars and I say well start from the
[09:47] earliest that you can get and then there's this whole lot of what I told you security ooth cassel whatever you need to get this right um two years ago I talked about this um it's very tricky
[09:59] sometimes um these days the AI coding agents can help you a lot two years ago I spend a lot of time getting this right um yeah so that's the the first code snippet let me go back
[10:13] here to the pipeline and then I I mean you read the data from Kafka it's a blob it's a it's a message you know and you want to split it up and this is what I do here with a parsing and um then I
[10:27] thought like I want to use this you know I want to replace this um existing the existing notebooks that I had with all these lines and all the vector database and the chunking and the embedding and
[10:39] the things I barely understood with you know the promise that we gave you last year with agent bricks and then I thought well that's a classical use case for knowledge assistant So I wanted to play with knowledge assistant and knowledge assistant requires um some
[10:54] let's call it metadata or some other some extra fields. So I basically have this file path, file name, file size and that's stored as metadata and that's the only thing um
[11:06] that I need for knowledge assistant and um yeah let's go on let's go to oh by the way this is the forgot to show you this. This is the old notebook um that I
[11:19] used like two years ago to get this into the vector database. And there's another one and it's just a lot of technical things that I honestly barely understood. And then I felt like so
[11:31] narcissistic to tell my my friend and colleague like well your notebook that was cool but I had to change two lines and I think all this you know extra code it should go away. It's this boilerplate code. And he was like thank you. We do
[11:43] this every single day. you know this is our job as a AI engineer and it happened. So what you can do now is you can go to well AI you go to agents and then you
[11:56] click on create a new agent and then we're dealing with well I see it as one you know one text blob we're dealing with unstructured data and I want a knowledge assistant and then I could
[12:10] just create one I already have one so I'm not creating a new one um so it gets a name it has this name already and then it's like where does the you know where's your source like typically unstructured data you want to place it
[12:22] in the volume it's a it's a UC managed folder here um you can do two more things one is you go the old way with a search index and I have to admit I did this first because I I didn't know I
[12:35] don't have to create the search index manually so I created that and I added it here and then then I read the documentation you know it's thisa principle we all think we can build this um bookshelf without reading the the
[12:47] manual I'm not uh not different. So, and then I read the manual and I thought like no no no you can just point it to this you know to this table. So you point it to the table you select a table and it's the streaming table for
[13:00] knowledge assistant and then you confirm and then you click create agent. It takes a few minutes and then you have this agent um should show up here. I'm leave here. Don't want to do this
[13:14] and it's this one. Well, once you have this knowledge assistant agent, it's loading here. You can ask it questions and um now you know where the data is coming
[13:27] from from these space events and you can ask it questions about these space events. You can say um well first of all let's try and talk to it. Um what is the difference in detecting
[13:40] neutrinos versus gamma ray bursts? Not sure if this is working. Oh, look at this. Oh, oh, this is my poor spelling.
[13:52] Come. So, it has like a a set of these circulars. It has a set of news informations and then you ask it about
[14:05] this, you know, what is the difference? How do you detect A versus B? And it it tells you, you know, neutrinos are detected in facilities like this ice cube. Remember when I told you about this cubical one kilometer uh thing they
[14:19] they kind of used in the Antarctica where they put the detectors and then the um spatial resolution time characteristics etc. Okay. So you can ask it about the
[14:31] knowledge and all I literally did was ingest the data. I had a streaming table. I had to add these extra fields you know for the for the location for the source for the size but that was not a big thing. And um yeah, now I can ask
[14:44] it questions. When I did this, I thought like, oh my god, and this is a super short talk. And then I I learned like there is so much more we can do. And actually there's even easier approaches where I think you should know them.
[14:57] Okay, so going back to the slides. Yep. So that was the knowledge assistant part. um two 370 lines I counted two
[15:09] years ago that were in this notebook where I was like not very thankful and just say like ah we should replace this with something you know that automates this now we have it it's knowledge assistant replace basically with one step click on new agent knowledge
[15:23] assistant and of course you have to have the right um uh streaming table with the right format but like I almost had it and then I was like using one of the coding agents I tell you about and just to to
[15:36] finish this and it worked out of the box. So that was a really really pleasant experience and um you can ask it all kind kind of knowledge questions obviously like you should be an astronomer and like not a material like myself and um the other thing is like I
[15:51] realized because I was always thinking about you know this more uh schema approach and SQL so it's really good in context you know this is what it's built for this is also like what the vector um
[16:03] search behind um is is good at but it's not so good in counting So if you ask it about how many gamma ray bursts did you see it's like it's a bit vague. Yeah. So it's more knowledge. Think about knowledge.
[16:15] Okay. And then I think the other thing that if you are a data engineer working with lakeflow in data bricks that you should definitely know because it's so useful it's so powerful are AI functions. And we have a whole lot of
[16:29] them. The most generic one is this AI query where you specify the query yourself and you specify the model that you want to call. So any of these foundation models that we provide, you can use for that. So you define the
[16:41] prompt, you define the model, it's the same API call for any of these models, which is actually quite quite spectacular because sometimes people say, well, couldn't I do the same thing that you show me here on AWS with, you
[16:54] know, something native at AWS? And like, yeah, you could, but like like what if you wanted to use Gemini instead? Well, you can't use Gemini on AWS, but here with this AI query, you can. So actually I I built such queries and then later um
[17:08] I thought like wow now we have this new entropic model and I want to run them against this uh because because I felt like it's providing better quality and I just changed it. So the AI query is the common one and then there is many many
[17:21] specific ones for translation for extraction for summarization and here in my pipeline I used one for um I used a classify uh let's see if I
[17:33] find my pipeline again let's use this one. So I used one for classification and if you click here you see the code of the pipeline and what I did I was
[17:46] cheating a bit like I didn't know what to expect. So I did a like a histogram of like what are the keywords in these messages and neutrinos popped up and gamma ray bursts and gravitational waves and then I thought like okay the rest is
[17:58] other. No I don't want to win a nobel prize in astronomy. I just want to show you these these functions and um yeah so I use this for um classifying these events
[18:10] right and well you can use them in the pipeline. Typically what I do and I I built like almost a handful of these examples. Typically what you want to do of course you want to narrow down your data. you want to go from bronze to
[18:23] silver to gold and then maybe gold plus you know if you have like reduced it to the if you if we use another example if it's more about sales top five sales locations for your company once you have
[18:35] them it's only five rows and then you can say okay now create a marketing campaign using a query and then use um AI translate and translate it to Spanish and French and German but you don't do
[18:48] this on you know 117,000 rows you narrow it first and then you apply the AI model at the end. That's always a good um um practical pattern I would say. Okay. And now Ali was showing
[19:02] this uh this morning. I want to draw your attention on Genie because a lot of the things I was showing you we can already do with Genie and well you can guess everyone was talking about Genie a
[19:15] lot. You will hear more about Genie. There's a lot of development happening with Genie these days and there's alo a bit of confusion I have to admit because we have Genie and then we have Genie one and Genie agents and Genie code and
[19:27] there's actually a few more but these are the three four that we want to understand here. So genie is the umbrella term that covers everything. Um which is kind of interesting because
[19:40] previously we we said genie to what is now genie agents and um genie agents before was genie spaces. So I always told people you can create a genie space which basically means you click on
[19:52] create genie space you add a table which is a structured table with rows and columns and then you can ask questions. So it's like text to SQL in a very very sophisticated way because again Genie
[20:04] understands the metadata from Unity catalog. It understands your usage patterns. You can give it additional instructions like at data bricks for example the um fiscal year doesn't start in January but in February. So if you
[20:17] ask it about Q1 Q2 you might tell it once by the way fiscal year starts in February. Just remember this and you can give it these uh natural language um instructions. So this is what used to be
[20:29] genie spaces and now it's renamed to agents which already tells you it's becoming more generic. And then genie code is the other thing that I basically use on a daily basis. I want to show you that too. And then at the very end, we
[20:42] want to do a small um detour to Genie1, which is more the umbrella thing for the non-technical people, which is a really nice interface um to get started um if
[20:54] you're more on the business side, which we didn't have before that. All right. So, Genieode, cool things you can do with Geniode is one, and nobody's talking about this yet. Maybe you hear it here the first time, you can create a
[21:07] lakeflow job. Now electro job the easiest one is a is a task doing something else you know task running a pipeline but there's very complicated
[21:19] multitask jobs like this one and this one for for fun because I wanted to try and see how capable it is. I said, you know, can you create me a job with a STP pipeline followed by a downstream
[21:32] notebook by a downstream SQL task followed by an if then with a notebook for each branch and it spins off and it just does this. So the whole structure is generated from one prompt and this is
[21:44] like the sometimes the scary thing you know cuz like even for me like uh there been like time at at data bras where like not a lot of people could really build cool demos and cool applications.
[21:56] So I built these kind of jobs showing cool stuff. Um now everyone can type in you know this sentence if you know what you need to type in and then the job just pops up and is ready to work. Um
[22:08] for the pipelines, we were talking about this more often. Actually, I do a workshop now at four o'clock um today, but we also do this uh using Genie code to create pipelines. So, creator STP
[22:20] pipeline that analyzes the sales data from a data set. So, you just pointed to data set. Two years ago, three years ago, I forgot, we created the bake house sample data set which is simulating a
[22:33] global bakery chain which is selling cookies and uses franchises and has sales transactions and just pointed to the sample data set and ask it any sales question you know like give me the top
[22:45] selling locations or just like give me an overview and it creates the whole pipeline from brillingest all kind of ingestion. It would configure autoloader. It knows how to read JSON um how to drill down into
[22:59] structured JSON. Uh it would use um um auto CDC. Uh it uses streaming data, whatever is useful. And then it's joining. Sometimes you need to give it a bit help of what it's supposed to join,
[23:12] what you're interested in, and then it's also generating these tables. I could spin that off. you would see it running um not doing it because it's like I could just comment on what the agent is doing but I recommend you give it a try.
[23:25] A lot of people have tried the pipelines. Not so many people because it's quite newish have tried generating jobs. Um you should try that too. So that was my literally my first attempt
[23:37] to try something more complicated with jobs and it worked like the first time. So I'm super excited. Okay, now coming back to genie spaces um
[23:49] where I said basically for a genie space what you do is you take a table you assign it to a genie space and then you can ask questions about the data in this table in a sophisticated way. Now this
[24:01] is also extended now. So if we go to Genie spaces, here it is. I have one for the GCN Kafka genie.
[24:16] Click on this and you see it is this. So demos one is my catalog. GCN Kafka is the schema and then parse space events is where I have the data coming from
[24:28] Kafka but parsed into into columns. So honestly, I'm treating it now a little bit more in a SQL way with rows and columns and not like one big structure where I put in everything because it's like that's what I want to show. It's a
[24:41] bit more convenient this way and it kind of shows the other side. Um so I just assigned this table and um now of course I can ask it questions like how many
[24:54] events do we have? Let's see what it does. And um one of the things to point out, of course, we could add instructions, you know, tell
[25:07] it what is an event for me. Maybe a neutrino detection is just an event for me. Maybe a gamma array burst is just an event for me. Maybe every single row that comes from Kafka is an event for me. My thinking, every single row is an
[25:19] event. So I don't really care. 713 that sounds reasonable. I think this is the number that comes from the pipeline. Um so this makes sense. I can tell it well. this makes sense and um that's it. The
[25:32] cool thing is um you can see the thinking what it does and you can see the query um that it is uh running. You can see the query here. So when it started with Genie I saw
[25:46] awful lot of non-technical people saying oh my god this is the final AI like this is curing everything this is the solution for everything. I was like guys you should be careful you know you should double check if the result is
[25:58] okay like uh if I would see like three messages here I would say no no no I think like the the pipeline ingested couple of hundred if I would see like three billion I would also say that
[26:10] cannot be right. So I want to see the SQL sometimes to verify if this makes sense. And unlike many other systems that basically have this text input, data output, result output, here you can
[26:22] always see um the code and and verify. Okay, this is not nothing super spectacular. It's nothing um very new. One of the new things is that we can now if we see this as structured data
[26:35] because my um my table is rather structured. I can now also give it unstructured data. So talking about astronomy, um I grabbed some PDF. There
[26:47] is a public astronomy book from the University of Austin in Texas and I had to shorten it a bit. So I downsized it here for data summit. I'll give it this.
[26:59] And it's attaching the PDF. And um I need to go back and cheat a bit because I forgot what cool question we can ask. Um
[27:14] yeah, of course. Now what we can do is we can say look what was the last event that you detected which is like a small circle where some some scientists say look we found this neutrino measurement
[27:26] blah blah blah at this date and time with this energy. Um but now we can do something that combines the factual structured data with the unstructured data. Now the unstructured data you say well I just got it from the internet.
[27:39] But the unstructured data in in in this example could be your company knowledge you know where your CFO CTO CEO compiled something put it in a PDF you load it here and then you combine it with the
[27:51] sales data. I could do something list the last three um events and explain them based on the course material. Let's try that.
[28:04] Is that still Oh, it's still loading. Yeah. Anyway,
[28:17] list the last three events and explain them based on the course material. That went pretty well. Okay, I think it's still digesting the PDF. Uh, but this Oh, thinking complete. So,
[28:31] it it was reading the PDF. It's analyzing the PDF. In the PDF, I actually took the section that explains like, you know, gamma ray bursts and supernovas and white dwarfs and star evolution and all this. So, let's see
[28:45] what it does here. And the cool thing is again combining the data from tables with this unstructured knowledge that you somehow have. Um, and um, yeah, it's doing it thing
[29:00] working well hopefully. Shouldn't
[29:17] take too long. Maybe a few seconds. Maximum half a minute or so. Oh, okay. So, it says I have found these three space events um
[29:30] June 14th. That's two days ago 10:00 and then it's kind of explaining like from this PDF knowledge like how this has happened and it's giving you a bit
[29:44] more context coming uh from the PDF and again you can we can see the sources and we can um also um drill down on how it how it was doing this analysis. Okay. So
[29:56] this was like using our own structured data with unstructured data together in Cheni. Now going back to to my slides,
[30:09] the other thing what you heard a lot about is MCP which is like having a special connection to some third party system which is also what we can do. Um I can show you this from Genie one which
[30:22] is the higher level um higher level non-technical interface or from Genie code. I'm showing it to you from Genie one. So if I go to a
[30:36] actually any um any workspace will do and you add one at the end you get to chie one which is encompassing all the cheny spaces that you have. So if you ask a question here, it would know
[30:48] about, you know, all the 17 Genie spaces that you might have. I only have one for big house and one for the for the NASA data, but in your company, in your enterprise, you might have many. And it's also reaching out for other
[31:00] systems. Um, it might know. So here, if I click on the plus, you see it could use Google Drive and Gmail and calendar and integrate all these third-party
[31:12] systems. You can also click on um more connections and you see there is a search um MCP. So the search MCP um I set it up already um because when I
[31:26] tried that I screwed up and then I thought like I don't want to screw up in front of 500 people. Um what you do is you go to marketplace. A lot of people don't know that. You go to marketplace and then you can say show me um MCP
[31:41] servers. which is explore apps. Where is it? MCP service. This has just changed since two days ago. And then here we go.
[31:53] That's the u.com. It's a MCP server that does search to reach out to the internet. And you just click on install and you give it a name for this connection. Um the host is right already. You can keep the base path. Um
[32:08] I think you should sign up once and create such a beer token beer token and then put it here and then you have this MCP server. Basically the way um I did it. So what shows up here is exactly
[32:23] this u.com MCP server and then of course you can ask it questions which are related to search and are related to the data it can find in the genie spaces
[32:35] which I should say in the genie agent because the spaces was renamed to ages uh agents. So I could say something like um I don't know
[32:48] for the say for the recorded events in the last three month uh can you search if any of
[33:01] these news appeared on the internet? Okay, now I have this twice because I'm not good in dictation and you can spin it off. You have to give it a cue, you know, it needs to understand what it's
[33:14] supposed to do. So if I I tried like it cannot guess that it should search, you know, like search using this u.com search. So I always say explicitly can you search internet and then it kind of typically understands. Yeah, it's
[33:28] sometimes also trying. This is not a real failure. Excuse me. So you see it's uh um kind of the shift we we did from using Genie with a PDF
[33:42] which is like restricted knowledge that you have within your company to Genie one using an MCP server that can reach out to anything and search anything or MCP uh it could be another MCP server
[33:55] doing something completely different um is like it's getting wider and wider. Okay, so this is doing a thing. It says completed web search. Did you see that? So, it's it was really using that um
[34:07] that um MCP server. And all this is tracked by the way with MLflow. So, you could go to MLflow and see exactly um what it's doing, right?
[34:19] And this was that one. And these MCP servers, there's a if you kind of uh inform yourself on the internet, there's also a big discussion on like how safe
[34:31] they are within data bras. They're all governed with Unity Catalog. Um, so if you go to Unity Catalog, you click on catalog and then it's a bit hidden like it took me a while to because I'm not doing this every single day. If you go
[34:43] to catalog explorer connections and then you see your um MCP server and you can assign uh permissions like who can access it um or not. All right. And there's one more thing about Cheni
[34:56] integrations and I already showed you um in the uh in the UI. Let me open a Hang on. Genu integrations.
[35:09] Let me just open a new Gen1. We we can also I always do it the URL way. You also get it here. You can also go here to Genie one. I'm not sure. Like I never learned this. I know it exists
[35:22] but I'm never using it. Okay. Okay. So, this is my Genie1 um uh interface and you see it's also it can also use like uh you know like Gmail and you can turn on Gmail and then
[35:37] I think I'm already connected and I could ask it questions like um did I write any emails related to the GCN project with NESA?
[35:56] I did actually because like when I did this like the the basic talk like with Iraq two years ago, I thought like it would be cool to have the folks from NASA um here on stage and like talking about their Kafka implementation. So
[36:08] it's searching. Let's see if it comes up with something. So I invited them and um I felt like a like a kid in a you know in a in a toy store like like who could you write an email to the White House or
[36:21] Nissa or like the I don't know the king of Spain like let's just try it. So I sent him an email. I said like I heard your presentation at this Kafka um conference. Do you want to be on stage with me talk about um yeah so that was
[36:35] my outreach initial outreach blah blah blah. I'm not showing you all the details and they were super friendly and responded like we would love to come and you want to talk about the Kafka topic for the gamma ray bursts you know what
[36:47] we do we launch a satellite exactly at in this week and they sent me pictures from that satellite so it's a shoe box size and I thought that was like the the cutest no I ever received like you know
[37:01] like sorry I can't make it I have to launch a satellite like okay yeah yep And I think this kind of almost brings me to Oh no,
[37:16] there's two more things I want to talk. Um I told you about GD code. There is another open-source um implementation uh which is called AI defkit and um it
[37:29] kind of seems to do a very similar thing and it does. Um there's still like I I get this question a lot like honestly AI defit is something a project that we started at
[37:42] data bricks s started it because they thought like you know we have all the AI there's like so many things we could do at the same time genie code was developed so genie code is within the data bricks workspace I use it every
[37:55] single day um because sometimes I just want to work in the workspace that's actually most of the time um and then I need extra support like I forgot how to do this cassle configuration for Kafka
[38:08] and I just want to type in you know help me with this or like I forgot how to do a app and I forgot some YAML thing and Genie code is brilliant for that. So it's the in notebook in data bricks UI
[38:20] experience. Um and it's like super cool for pipelines. It's super cool for tasks. It's super cool for applications actually. And then sometimes I have other days when I do a lot of you know like cloud code things and I have my
[38:33] cloud code window and then I feel like but now I want to create you know with all the other things I do with cloud code now I want to use this and create a pipeline and then I use AI defit. So if
[38:47] you click here uh AI defit it's a uh it's a repo on on git. It's super easy to install. It's like one line on the classic
[38:59] install. Yeah, it's this bash, you know, curl from somewhere and then uh run the installer and let me show you. Well, you can basically do a lot of things. Um so let me exit
[39:13] here. So I I ran the installer already. You can install it per project per cloud code project. I did this like seven times and then I felt like the eighth time like again I installed it for a new
[39:26] cloud code project and then you say install it for all my cloud code project. So it can be at the root level or at a project level and now here I'm at the project level actually for this lakeflow agent break thing um that I
[39:38] built. And then what you do is you say clot oops clot and then normal clot um starts up but you have these extra capabilities extra
[39:50] skills and extra MCP servers that let you talk to a workspace environment. So I could say um which workspace am I connected now? That's a very um
[40:05] easy question and runs off. it does its thing. Um, how are we time wise? We start at 50. I think that's it. No, more or less. Yeah. And then you can you can say create a pipeline, create a job. Um, the
[40:19] nice thing is now you're outside of data bricks. You can use do all your cloud code things that you might do anyway. Um, it has more or less the same capabilities. Um, you have the possibility to extend
[40:32] cloud code on the data bricks UI side with extra skills. So you could in theory put these skills into um cloud code. I don't think you need to do that but someone posted about this. So yes, you can do it. You can also extend cloud
[40:45] code with other MCP servers. So if you want you know want to use um sorry if you want to use um Genie code with more MCP servers with extra documentation with extra capabilities you can do that.
[40:58] So it's kind of nice to have both and I think they will um merge more and more. Not saying one of them will go away, but I think from the capabilities um they will kind of become more and more equal.
[41:12] Right? Where are my slides? They're gone.
[41:30] Yep, that was the last one. I'm here for more questions. That's the end of the presentation. I actually hope that every one of you took back like 1.5 new things that you didn't hear before like you know at least one hopefully two on
[41:42] average 1.5 something you want to try at home at at your company um I'm here for more questions I think u we don't have anyone with a microphone so it's better if you come to me and I'm not running
[41:54] away happy to answer more questions thank you for being here you've been a great audience Peace.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.