Bulk Metadata Generation To Bring Genie to Life

Summary

Genie Spaces perform significantly better when Unity Catalog columns have rich metadata descriptions, but most enterprise tables—like those at Optus with 15,000 tables and up to 80 columns each—have no descriptions at all.
A bulk metadata generation tool built by Databricks and Optus uses customizable AI prompts and optional data sampling to produce high-quality metadata descriptions across entire schemas or catalogs with minimal manual effort.
The tool generates structured JSON metadata output that flows through a human review workflow with an audit trail, enabling organizations to bring Genie to life across large, historically undescribed data estates.

Bulk Metadata Generation To Bring Genie to Life

Genie benefits significantly from the presence of column metadata. In this talk we will describe the various components that feed into a high-quality Genie space and review some code you can use to populate the base Unity Catalog column metadata in bulk. The code makes use of customizable prompts and optional data sampling to produce high-quality metadata across entire schemas or even catalogs.
Talk By: Paresh Joshi, AD - Foundation & Capability, Optus ; Scott Eade, Lead Solutions Architect, Databricks ;

Chapters

FAQs

Why does Genie need good metadata to work effectively?

Genie uses column-level metadata to understand the meaning of data when generating SQL at query time. Without descriptive metadata, Genie must infer meaning from column names alone, which leads to less accurate queries and a poorer user experience, particularly for tables with cryptic or abbreviated column names.

What problem does bulk metadata generation solve for large enterprises?

Large enterprises like Optus have data estates with tens of thousands of tables, each containing 40 to 80 columns with no existing metadata descriptions due to decades of organic growth. Writing descriptions manually for every column is impractical, and the bulk metadata generation tool automates this by applying customizable AI prompts across entire schemas or catalogs.

How does the bulk metadata generation tool work?

The tool uses customizable prompts and optional data sampling to generate high-quality metadata descriptions for Unity Catalog columns at scale. It produces structured JSON output that can be reviewed by human data stewards through a built-in audit trail workflow before the metadata is committed to the catalog.

What was the impact of bulk metadata generation at Optus?

Optus operates a data platform with 15,000 tables built up over 30 years of company growth, presenting a significant metadata gap. The bulk metadata generation tool enabled Optus to populate Unity Catalog metadata across their schema at scale, with measured improvements in Genie Space usage and output quality after deployment.

Full transcript

[00:11] Hello everyone. Um, welcome. Uh, thank you for deciding to spend a little bit of time with us here today. Uh, my name is Scott Eid. I'm a solutions architect with, uh, with Data Bricks. I'm from Sydney. Uh, my colleague here, not quite my colleague, but my loyal customer is
[00:25] Pesh Joshy here from Optus. uh and we'll talk a bit about Optus a little bit later. Uh but the subject of our talk today is about uh bringing Genie to life with bulk metadata generation. So we'll
[00:38] talk uh in a bit more detail about that as we go through. So uh thank you all. Let's jump in. So what are we going to talk to uh today? So obviously we've talked you
[00:50] would have heard um Ali talking a lot about Genie, other people talking about Genie. So you've probably been genie to death. So, uh, this is an interesting talk because it's about Genie, but I'm not going to go into Genie at
[01:03] all. Okay? Uh, because we're going to talk about what, uh, it takes to make Genie work well. Um, we'll spend a little bit of time just talking about Optus and their journey on their data platform and so on and and into Genie.
[01:16] Um, the hero of our story is the metadata generation. That's what we're all here to talk about. So, we'll jump into that. Um I've got a a nice little demo in and amongst this uh with a uh a very helpful tool that uh will be
[01:28] available and is available now. Um and we'll talk briefly about the impact of using this uh this type of tool uh at Optus. Um and then just touch upon uh some related a related tool um that you
[01:42] can look at as well to help you improve the quality of your Genie spaces. So, have you heard about this little thing, Genie? Of course you have, right?
[01:54] Um, so I mean, I don't think I really need to describe Genie too much, right? So, you've got a dashboard. It's, you know, it's got a fixed view of what's going on. Genie allows you to ask questions of that data in natural
[02:06] language and get the results and, um, you know, with, uh, innovations like, uh, agent mode, do a deeper a deeper look at that data. really useful tool. Um, opens up the uh that data to, you
[02:20] know, a wider audience uh and does so in a in a really nice way where it's governed and secured and all of the things you've heard us talk about a lot. And I guess the thing here is Genie is
[02:32] everywhere. You've seen us, you know, renaming rename certain things to be Genie. It's a a brand that fits across a whole bunch of things. So, Genie Spaces is what we what we know and love. in fact the focus of uh what what we're
[02:44] here today to talk about. Um we've got Genie 1 as well of course right so uh Genie1 is kind of that next layer above Genie one's going to also benefit from what we're here talking about today
[02:56] which is uh you know improving the metadata behind your um behind your your your data itself. Um, and then of course there's the the spaces API, Genie spaces API which allows you to embed your Genie
[03:09] space into other um other solutions that you're building. And in fact, a lot of the times when people have been using the Genie Spaces API, they've probably been kind of embedding it in you've kind of been embedding it in your own
[03:21] solutions. So I think it was overnight or whatever we we announced a bunch of plugins for common tools that will just do that for you anyway. So you can embed Genie in Teams and in Slack and in uh
[03:34] Copilot. Um uh there's a way to access it via an MCP server and um through iframes, all sorts of ways that you can embed it. So Genie is kind of everywhere.
[03:49] So what is it that makes Genie work well? Um I think there's probably a bunch of uh bunch of sessions here, you know, over this couple of days uh that are talking about that in great depth. Okay. But what makes it uh what makes it good? Well, obviously you're adding
[04:02] tables and things like metric views into your um into your genie spaces. Metric views actually deserve a special call out. They're a really good way of defining the uh the dimensions and
[04:15] measures in your data um such that the genie space can benefit from that additional information you give it. So you can code in things like synonyms and you know table descriptions
[04:28] and so on through metric view. So highly recommend uh that you you look at that. Other things you can do you can uh define your primary keys and foreign keys that informs Genie of the relationships between those tables. It helps it builds joins. Um the table and
[04:42] column descriptions that's what we're here to focus on today. Useful uh useful to inform Genie what's there. There's a bunch of other things you can do. not really the focus of our talk today, but you can uh code in example queries,
[04:55] functions and and so on. Um give it instructions, all those all those other things. And then behind the scenes is also the knowledge store. So there's a a bunch of things that I'm not mentioning here today because they're actually
[05:07] quite quite new. So genie semantics, genieontology, and so on, right? All things that are going to help improve your um improve your genie space. So what's kind of the architecture of
[05:20] Genie? This is a really really really dumbed down simple view of of of what's going on when when Genie does its thing, right? So you've got uh a lovely little smiley face user over here asking a
[05:32] question. What happens? It goes and gets some context. So what is the context? The context is the table descriptions, the column descriptions, and all of the other things that we talked about a moment ago.
[05:44] It takes those it gets any other instructions that we provided in the genie space figures out uh the appropriate question and turns that into a query uh and then goes and sees uh whether or not it can go uh whether it
[05:56] needs to to to prune columns and so on to generate that it's SQL and retrie data brick SQL and get the result. It's doing that in a governed way. So you can only see the data you're allowed to see.
[06:08] Uh and you know it runs the queries and brings back the result. Okay, all that's done with an audit log. very simple view of it but the context really helps things move along. So on that note let me pass over to Pesh
[06:21] for a few minutes to talk about Optus and their journey uh in the data world and and towards Genie. Thanks Scott. Uh hi San Francisco. Uh my name is Paris Jooshi. I'm associate director in Optus.
[06:35] Uh that's the second largest telecom company in Australia. We've been operating for last 30 years in there and we power connections for more than 10 million customers across across
[06:47] Australia and this has been a journey for 30 years where we have grown and and that's where you see 15,000 tables. Now that is an accumulation of all the work that we as an organization have done to
[06:59] service our customers. There's a lot of stories behind the 15,000 uh tables. Now I want you to hold that thought. 15,000 tables. That's incredible. That just shows how much Optus was committed on
[07:13] data journey. It knew what data meant and how it can help us power the services and the products that we give to our our customers. And in 2022, we implemented data bricks uh our unified
[07:26] data platform where the gene of the idea was that we were going to amalgamate all our data warehousing workloads and machine learning workloads into one single platform so that we can have analyst machine learning engineers all
[07:39] of them getting data driving insights from one single platform. >> You're living the lakehouse dream, right? >> Living the lakehouse dream.
[07:53] a a world-class data store to a world-class data assistant? And we found that the answer was genie. But genie requires context. It requires metadata. It requires table definitions. It requires descriptions. It requires
[08:06] meaning behind all the columns for those 15,000 tables. And that is I think a lot of a lot of people in the crowd would be able to relate to that journey in your organization.
[08:21] I love this slide. This basically shows the evolution data platform within Optus. And this is not a sequential thing. As and when we grew, we grew incredibly in last 30 years. And when when we grew, we started to integrate
[08:33] with more than 100 systems. So we were focused in actually building products, building services, connecting customers and we did that and along the line we picked up the best technology that was there in the industry to power us to
[08:47] make the decisions to serve our customers better. So IDW, SDW, EW, ODS, BDP, I'm sure you've got all threeletter acronyms in your organization, but this basically tells us the the story that uh
[09:00] for that in 2025, we asked the question, how can we get our customers or our business users to come and ask a question? A question for what has been my prepaid revenue in last
[09:13] quarter or what has been my churn. how can we get or empower business users to ask simple questions uh via English just in plain English language and that's when we identified an issue where it had
[09:27] been invisible for some time because of all the migrations all the things that we did um that was the metadata and when AI came along it needed context it was missing the context and that's when we
[09:40] started to think about what we can do about it so again what does the meta metadata problem. We have 15,000 tables and give or take Scott what we did a math on this
[09:53] yesterday. >> Oh, I think the average was 40 columns. >> 40 columns. >> Okay. So, between 40 to 80 columns uh on a table on average we have and what we what we saw was that if we had to go
[10:05] hardcode it, how if we have to get data stewards of the world go and actually create description for it, it would take 6,000 hours. we'll be sitting here till three Christmas before we get all the data uh data metadata captured and
[10:18] that's when we said we had the data we had Genie but Genie was running uh flying blind here
[10:30] metadata generation so in I think when when was this databris IQ got launched >> yeah probably 18 months ago something like that yeah >> yeah so database IQ came along and it had a very good uh uh option to generate
[10:42] the the the metadata for the tables and the columns. There was only one problem and the problem was you can do one table at a time, one table at a time only and and that it did a good job. It did a
[10:55] good job. We tried with customer 360 model at the time and it gave an accuracy of 95%. But but there are clear some of the things that that were happening behind the scene like if you look at the column names that we have which is custp
[11:09] mbb flag a generic AI model would not be able to actually all the time find out what that really means. So we had to provide >> I can look at that I know what c okay customer FLG flag the other two no idea
[11:23] no idea. >> So that basically meant that does this customer have an MBB uh plan available. So it was it was really hard like it it it could have done a lot of guessing. What we also realized is that in any
[11:37] organization largely we'll have probably following medallion architectures where you got brown, silver and gold. So your same data is actually replicated in multiple places. Now the approach that we had here would enforce that we are
[11:49] regenerating the same description for the same column again and again and highly likely we wouldn't get the same description consistent description. So we said what what do we do about it and in Optus we have this uh hackathon
[12:02] innovation day that we have we get all all the engineers to have tools down we sit together and see what we can do for future and that's where Scott and I got together and said what can we do to address the problem at scale the scale
[12:15] of 15,000 tables and how can we actually fasttrack because we knew that a AI has come along and how can we uh get more value coming out of the platform that we
[12:29] and and I distinctly remember a conversation. It was myself. Um we have a a DSA named Chung who was also uh with us at the time at Optus. Um and Perez said basically this. We like what's
[12:41] being generated. We find it, you know, pretty damn accurate. Um it's a very very big uplift, but how do we do this for our 15,000 tables? And so at the time we're going, is there an API for
[12:53] this? Uh the answer was no. Um so we couldn't do that. Uh so we had to explore another way. Yeah. So the saw in that statement was deliberate. >> Yeah. >> Yes. So many tables.
[13:07] >> So we sat we sat together that day. We we took some time to say what what what would we need? What would what are the steps in the process? What are the building blocks for us to build a process or an algorithm which would actually help us do at the scale. And
[13:20] what be became clear was that we need to get a list of all the tables and schemas and the column names within our our system but also if there were description that were built information schema today provides that particular information a consolidated view of what
[13:33] is there on the ecosystem that you have. So we had that. The second pillar was AI tooling. Now as I said with the data IQ nothing against it but it still doing a generic AI quering. It was just
[13:46] interpreting the data and it was trying to do a string but it did not have the context of our industry. It did not have the context of the abbreviations uh we have. So we said we're going to this allowed us to actually do it in a
[13:58] flexible way where we could choose our own model. So we experimented with couple of model. We ended up with open AI uh thing. We were able to have a custom prompt that we provided where we added naming standards, abbreviations
[14:11] and synonyms that are specific for our industry like an active base. Active base is very specific to Optus. So if there is a flag going on PST_AC flag, the generic AI is not going to
[14:24] know. So we provided that context in the code that we built. Then we had sampling of the data. So that is something that we added which actually improved the the the outcome of it. And then for the template and yeah and as I said common
[14:38] column names in the medallion architecture the same columns are actually repeated multiple times and we were able to get this process to actually dduplicate it and have a very uh efficient process running for it.
[14:53] So we decided uh this was Chung and myself we could build something to do this. we had the three pieces that Pesh mentioned. We could we could build this um at the time. So this was a little
[15:06] while ago. Um we didn't have the coding tools that we have today, right? And we'll we'll we'll circle that back to that. So um what I did have was I had an intern
[15:30] do it and they implemented something and it was awful. Um, they they clearly didn't understand how Spark and and um, you know, the uh, you know, um, data
[15:42] frame style programming worked, but you know, they gave it a shot and you know, I gave them a pat on the back and said great and then rewrote it all. So, um, I rewrote something. It was pretty much go through that workflow, go and pull in
[15:55] all of the um all of the columns for uh you know was configurable to do a table or a um a database schema or even an entire catalog, right? Go and pull those
[16:08] uh pull those out of the um uh out of the information schema. uh it allowed you to customize the prompt to you know say what what you wanted to use for the uh for the column description and then
[16:21] uh it would dump that into a wonderfully enormous JSON file that uh that was my kind of review process at that time. I said hey some random person in the organization hopefully a data steward
[16:33] that knows what they're doing can somehow edit this giant JSON file. Um and that was that was the review process. Um, and then ultimately we could uh we could pull that into uh into
[16:45] the environment and apply it. Okay. Um, and we built that out. We put that in an internal uh internal repository. Um, I'm not making this one public. It was it was a simple simple little script.
[16:57] Anyone could implement it. Um, but I put that in a internal uh repo and published it. And a few people picked it up. Uh, a few DSAs picked it up and used it at customers and I had a few enhancements come back. Um, and those enhancements
[17:10] were things like, you know, someone added in the table description generator, someone added in the value sampling, which I actually didn't have in in in my one. So, you know, some some good useful things uh happened. Um, but,
[17:22] you know, it wasn't a particularly polished uh uh implementation. But what happened next was the call for papers for DS 2026. Um, so uh my boss is sitting over there.
[17:36] uh we they said hey everyone go and give us some ideas for talks at at DEAS and I went well actually this this kind of works so I contacted uh Pesh and um we decided to to have this talk and I think
[17:50] is a is am I at the right spot to go to the next slide there you go yeah and and so Pes described to me some other things that they had been doing that I didn't even know about so this is what we did so
[18:02] that was a good MVP which it did what it did But as soon as we implemented our platform, it became really clear that we required a human in the loop to actually review the process and those people were
[18:14] sitting in the business. The SMMES were sitting in the business. So we went through said what does that end to end solution look like now that we have an MVP which does does what we wanted to do but it doesn't scale to get things into production. So we said okay let's let's
[18:28] go through the steps of we had the master inventory. We have all the tables in this thing. We built in on top of top of what squad has built in the distinct column name. So all the dduplication logic that we implemented that so that
[18:40] we don't do that again. We had a spe special context that we created for OpenAI uh where we start to generate the generate the the descriptions. But this is one thing that we actually bought
[18:52] across and that is something that that I spoke with uh Scott when this speaking opportunity came along is that we start to to see how we can leverage data bricks apps such a powerful feature integrated with unity catalog. How can
[19:05] we get that to do some of the review process which is currently sitting in a JSON file which which which was the analyst wouldn't know what to do about it. So we said okay how about we spin up a a database apps which allows people to
[19:18] go and review. So we build a app which allows the person who's generating the comments to then allocate via emails some of the columns that they need to review. So we have all review process that we built into the app and then of
[19:32] course it had the auditability of what is approving what is the edits that has been made all of that this information or this app actually produces DDL statements and that DDL statements we deploy into our Azure DevOps and through
[19:44] CI/CD pipeline it goes into production. So we we we looked into it in a very strategic way and I think we are at the right of the end of it where we are start to see the futition of some of the work that we've done and we'll start to implement uh this into production.
[20:02] So I have no idea how they implemented that app but we submitted the talk idea it got accepted and I was with Pesh not that long ago reviewing the deck and he described this app and I said can I see
[20:15] it? Can you send me some screenshots? He sent me some screenshots and I thought, well, I do have Claude code now. I do have Genie code. >> Are you saying you didn't have another intern? >> I didn't have an intern. Well, Claude
[20:27] was my uh my intern. Um, so I threw the screenshots at Claude and then just started having some fun. And so I'll talk a bit more about my process in a minute. Let me talk to the slide for a
[20:39] moment and we'll jump into a demo, show you show you what I've done. And this is in a public repo. uh you know I made it public yesterday um and it's uh an enhancement a massive enhancement over what we originally implemented right so
[20:52] there's a nice UI that sits over the top of things it um it does the column fingerprinting so it contains the LLM costs um you know so the the the level
[21:04] of duplication is kind of all resolved um it caters for things like uh you know abbreviations and suffixes and those sorts of things and provides a solid review phase and a solid audit trail.
[21:17] Okay? Because one of the other things to think about here is who actually has the rights to go in and change those column definitions. You don't want to go, you know, giving that to to everyone. Um, but in this solution, you can pull in
[21:29] the descriptions, give them allocate it out to reviewers and then apply them and and do it in a in a contained manner. Um so that repo is um is public and um
[21:41] I don't give the URL but if you go and search for data bricks metadata generator you'll find it. Uh so let's jump in and have a look. So here is my wonderful databicks
[21:54] workspace. Um, I also have in the codebase, it will go and generate a horrible schema for you that has all of these, you know, the kinds of problems that we were we were hitting, um, which
[22:07] aren't particularly uncommon. And so, here's an example table. We've got a bunch of columns. Uh, we can see at the moment there's no columns. Uh, this is, uh, the data bricks uh, environment, you know, suggesting the column description,
[22:19] but there isn't actually one there. If we go and have a look uh let me let me go and start a warehouse. Um we'll see that there's some values
[22:31] behind this data. Um and you know the reason I'm showing you this is the data is important right so look at this column here with various kind of coded values. That's actually something useful
[22:43] in terms of describing what this table can do. Okay. So nothing nothing uh nothing happening here. Um so this is the uh this is the app that gets deployed from that uh that repo. Um so
[22:57] effectively what this does is allows us to follow a fairly simple process one two three four of extracting the data um generating the columns approving them and applying them. And we can do that in
[23:09] one flow which suits us in this room today. But we can actually do that in a more long- winded way and and invite some reviewers to come in and and and participate in this process so that we can uh you know make sure there's a proper uh proper review phase here. So
[23:24] what we've got up here is we've got our um a selection of a catalog and a schema. I can run this for all of the tables. So um if I go here, it will go
[23:36] and interrogate the information schema and pull in all of the columns for all of the tables in that schema. Okay, so we can see here there's a a bunch of columns in there. Uh sorry, tables. There's actually 33 of them. Uh it's
[23:49] going to go and pull those in and it found uh 376 columns across those 33 tables, but there's actually only 176 unique columns. Okay, so some of those
[24:01] columns are things like audit data columns. Think about that. Your audit data columns are actually common across all of your tables. Um, and then there's also, you know, 33 tables there. So, it can go and and and pull those in. What's
[24:14] going on behind the scenes is it's going and interrogating Unity Catalog's um information schema tables. It's pulling that into a lakebased table. So, we've got a nice responsive uh interface here. So, we're seeing what it's what it's
[24:26] found here. Um, and um, oh, I've got some left from earlier on, but that's fine. Uh, so what I'm going to do here is I'm just going to downscope this, so we're not here for a long time, to a
[24:38] single table. Okay. Um, it happens to be the one that we were looking at a moment ago. Um, what I can do here is I can, um, get it to, uh,
[24:51] I don't know why that's regenerate. Let's do it anyway. So, I can get it to go and generate the column descriptions for uh for this particular table. Um, I'm I'm finding a
[25:04] bug in my own code right at this second. Uh, let me get those out of the way. I'm just going to approve them and run it again
[25:17] and then hopefully I love discover uh discovering bugs in in code I've just taken to production, right? Um, let's go back to all tables and we'll do it for our columns. We'll generate the 33 columns. There you go.
[25:30] That will take a moment longer. >> So, while this is happening, Scott, it's it's it's just another idea comes to my mind. >> Yeah. So, we we are dduplicating largely based on the names of the columns, but won't it be cool? Think about it because
[25:42] because it'll be cool to actually use Unity catalog column lineage. >> Mhm. >> To further find out what are the different columns because from the bronze layer, it could have got renamed to something, but the data is the same. Something to think about. Yeah, it's it's actually a good idea. So, when when
[25:56] you're um vibe coding an app like this, new ideas and new features appear all the time and pop into your head and you throw them on a backlog and add them in. Wouldn't it be great to traverse the lineage and even either either it's
[26:09] something being renamed directly or maybe it's um maybe it's this is part of that and I can actually inform my comment so that in the comment for this table we know that it's the source of something downstream which might be
[26:22] quite useful. Um so yeah lots of things can happen there. So uh this step here it's going and uh generated the uh the 33 column descriptions here. Uh it it seems to have corrected itself now. So I'll go and generate just five of my
[26:35] column descriptions here. Um or yeah that's because I went back to all table. So that's fine. So down below now that I've run that for a few of my columns what I can see is I can see what's uh see what's generated. Actually
[26:49] just want to try that just just no whatever. So what I can see down here is I can see the generated descriptions and uh as I
[27:01] said before I can hand this off to a reviewer as well under this other other tab here but I'll just stick here. So I'm reviewing at the moment my column descriptions. These are the ones that have been generated. This one here invoice number pretty clearly invoice
[27:15] number is actually used across four tables. I can see those four tables and I can be sure that this description that I'm I'm doing here will later be applied consistently across those four tables. Generative AI is non-deterministic. If
[27:28] we were doing this four invocations with four separate invocations, we might actually get different descriptions, right? Um, so I've got what was generated here. Where did that come from? Well, let's jump down here to the
[27:41] template section for a moment. So, in the template section, I have a couple of prompts that I can use. And uh I can have one of these active at a time, but really it's just a standard the standard
[27:53] kind of thing that we do when we're when we're prompting an LLM. And so in here I can say um you know I want to actually sample the data here. That's a useful thing to do in terms of giving it context. Um and uh this is PII aware. So
[28:08] we're not sampling data for columns that are used for row filters or column masks uh or with column masks applied to them. And we're also checking for for PII related tags and we can configure what they are. So um you know we we can edit
[28:22] this to do whatever we want and then and then save it. So this is where the the prompts come from. So we've got a couple to choose from. We can go and edit those and do what we like. The table ones are slightly different. There's actually a third option here down below. So we've
[28:35] got, you know, a couple of variations here. We can make them do whatever whatever we like. But there is an extra one down here for um for a table that we're going to use in Genie. So I don't
[28:47] have the link handy. Maybe I should add it to the deck. Uh but there is a best practices guide for what you should include in your table description for when Genie is being used. And you know the recommendation is that it has five
[29:02] short paragraphs covering what entity or event or relationship each row represents etc etc. So really what I did was I cut and pasted the best practice guidance for what to include in
[29:14] your table definition for a genie space and plugged it into this prompt here. Okay. Um but I don't allow this one to be the default and I'll explain why in a moment. So that's where the templates
[29:27] came from that were used to generate these descriptions here. Now I'm the human reviewer. I can actually come in here and um and edit these um hit the edit button and edit these uh and go um
[29:42] you know invoices are awesome. Whatever. That's a really useful useful comment. Uh but you'll see it's it's flagged that I've uh I've edited that. So I'm the the the
[29:55] human in the loop here. I'm I've looked at those. I'm happy with those and I can approve them. And that's a process once again that we could delegate out to others. If we have a quick look at the tables, um we had the I did that for the
[30:07] whole 33 tables there. And you can see um you know these kind of look reasonable. But let's say we've got one that we want to use for um a a genie space. Uh so what I can actually do is I
[30:20] can quickly jump in here and say actually this table I know is used for genie. Uh, and so I've selected that and it's going to go for that one table and use the genie prompt. Okay. And so you
[30:32] see that takes a moment because it's got to do a little bit of data sampling and you know deal with that more complex prompt. And you'll see here it's uh it's done uh a better job of spelling out what's in there. And depending on what
[30:44] table it is, it might have a few, you know, extra details and some status values, all sorts of things that that kind of can be useful there. Now, the reason I didn't make that default and I don't allow it to be made the default is
[30:56] when I do it even across these 33 tables, I looked at what it produced and I went, hm, there's a bit of hallucination and strangeness going on here. So, I didn't want to apply that default approach across all the tables,
[31:10] but it's reasonably easy to come in and apply that manually for the ones that you know are are useful. Okay, other than that, pretty much the same as what we were we were doing before. So, I'm the human in the loop. I can uh I can edit these and approve them and they get
[31:24] added into a queue. All of these actions that I'm taking are logged into an audit trail here. Okay, so who who did it, what uh table was
[31:36] impacted and what they actually did. Uh so you know the invoice number was the one that I edited, right? I said invoices are awesome or whatever. Um so I could do that. Um, so back here now I
[31:48] can come and um, you know, I've done my review process. I can come and apply these. So I'm just going to hit apply and it will do um, all of the ones that
[32:01] have been approved. And you'll see that number was pretty big. There was like 59 columns or whatever it was there. That's because even though I only generated and approved a small number of columns, they happened to be used across a lot of tables. Okay. Um, so it's going and
[32:15] doing that, depending on how many there are, there's a bit of a a progress bar that can happen there. And once again, everything was captured. Um, so before I go and well, let's jump over
[32:28] here for a minute and refresh this. And what we'll see is I didn't do all of them. So we can see a couple of a couple of column descriptions come in here. I didn't do them all. And we can see our description that that happened in there.
[32:42] Um, and that was probably our our longer one. And so over here in our sample data, um, here I can ask questions about my data. It's kind of a pseudo genie
[32:54] space here, right? I can ask questions about that sample data. So having that metadata is instantly useful even outside of Genie. But what else did we do there? There's
[33:06] one more thing to show you there before we finish up with a couple more slides and that is down here in the reference data. So what we did down here is we said okay well let's help decode the
[33:21] cryptic column names that we have. So what this does is it uh uh allows us to configure a a set of abbreviations and suffixes which might be subtly different including things like duplicates like
[33:35] there's ones in here where uh like cat here for example could be category or it could be category code right um and if a column has one of those it's actually flagged as such so that the
[33:48] human that's doing the review can pay a bit more attention to it. And we've got a little scanner here as well that will go and look at the columns that it's detected and see if there are any abbreviations or suffixes
[34:01] that we haven't actually defined. So we can come in here and go ah this model one is you know or this payment right payment is seems pretty obvious. So I can go add it's actually going to go and suggest what that might be or I can type
[34:14] it in myself. So I can just click that one save it and that will appear and be used next time as further information. So some nice little configuration things and there might be other things we extend in the future.
[34:27] Um apart from that um you know we can uh we can see uh you know the lifetime value that this has done in terms of the the number of um you know columns that have been generated and so on. Um yeah
[34:41] so let's quickly wrap up our slides backup demo. I wanted to talk just for a moment about how that was built, right? Um, as I
[34:54] said, we we've got coding tools now. We didn't have the the intern um at data bricks internally, we have a tool called Isaac. It's just a wrapper around clawed code with a bunch of other services built into it. Um, it's got um uh you
[35:08] know, it uses the the models that are in data bricks via Unity AI gateway. Um, and it actually this this thing was really really really quick to build. Uh, as I said, um, it wasn't even my idea to
[35:22] build an app before this presentation. I was going to show the dumb app that I'd built or the dumb script that I'd built. Uh, but instead I grabbed his screenshots, threw it at at um at Claude and then just started prompting it to do
[35:36] more and more and more because it was kind of fun. Um and then you know ultimately it was what is it about 3 days and 52 commits right that was because that was three days of elapse time not dedicated time and the 52
[35:48] commits was because if ever you use these tools and I highly recommend that you do because they're actually fun you just get I mean I think it's called vibe coding because you get into a bit of a vibe you start going oh what if I did
[36:00] that what if I did this what if I did that and you start adding more and more features so the 52 commits is because I kept adding features and there's a backlog features I didn't implement because I drew a line. I said I can't keep going here, right? Um you could do
[36:14] this yourself. It's very very easy. Okay. Um it's possible to use the models in data bricks via uh we have this little tool called Ude where you can leverage Claude and and and the other uh the other models via data bricks. Um
[36:28] choose your your your coding agent. Uh there's a whole bunch of different ones available. We have a little library called AI devkit which allows you to interface with data bricks really really nicely. highly recommended. Um, or you
[36:40] could actually do this inside data bricks with genie code. Uh, there's an uh there's an app builder thing that you might learn about. I don't think we talked about it today, but let's see what they say tomorrow. Um, and so what
[36:52] what happens here? Basically, we've got our reviewer accessing a a data bricks app. Uh, it's using uh a React front end and a fast API backend. It's leveraging uh a Postgress uh lakebased Postgress
[37:05] database. um that allows us to have a nice snappy user interface, allows us to track the audit trail and so on. Um and also that's where we keep things like our prompt libraries. Uh in terms of
[37:17] interacting with uh Unity catalog, that's going via a SQL warehouse. So we're pulling in our information schema from there. It's where we're applying them. And then um in terms of leveraging the AI models, we're going via Unity AI
[37:30] gateway and um you know, we're using Claude. Fairly simple, straightforward app. Okay, Claude generated this slide as well.
[37:47] Um, if you wanted to do it, where can you do it? I think it was last year we announced data bricks free edition. You can go and fire up data bricks free edition and have a play in there if you want. There's a URL on this slide to a simple repo that we'll build a you know a very simple genie space that you can
[37:59] then you know play around with. So let's talk momentarily about the uh the impact of this. >> So yeah of course our northstar is getting everything 100%. Um currently we
[38:11] set at just around under 20%. Where uh we have some source description we've got some sort of uh table descriptions and column descriptions. Now currently
[38:23] as we speak we are in the process of actually generating it and as I said earlier it's going to take 6,000 hours if we get somebody to do it. But with the process that we have now, it's probably going to take us two weeks to just generate all the data, providing
[38:35] the right context, different things together and just doing a bit of a check. What's going to take time and that's that comes in the enterprise space where getting these comments to be reviewed and approved and that's why the app becomes really handy where we
[38:47] actually democratize the whole review process such that data stewards can review that and week after week we'll start to go and deploy this into production. What we have done in our app also is that we are looking at the
[39:00] number of tables which are actually heavily used, number of columns which are actually heavily uh duplicated and we have prioritized that prioritized that in the generation as well as the review process which means that from
[39:13] very soon very quickly we can start to see the impact of it. So generation is in progress and next we expect that in next one or two quarters we'll we'll be at 80%. And then yeah one thing that that we also want to do is and we are
[39:27] looking at an IT managed space but as you can see that in a federated data delivery model all the businesses also have their sandits of the world or their play areas once we have gone through that we're going to extend this particular thing for all the business
[39:39] users so they can start to actually put in their description because they would want to build genie spaces on their sand tables as well. >> So the review phase that I have here would be more useful than that's your current implementation. Yeah. >> Yeah. Um just really quickly I we're
[39:54] we're basically at time. Um I want to point out this thing uh Genie Workbench. So this is available on GitHub as well. It's the coordinated effort that we have from field engineering at data bricks to help you improve your genie spaces. We
[40:08] had a whole bunch of different initi initiatives. We went that's crazy. That's not going to work. So we focused everything around this one uh this one effort. It will look at your genie space. It will review it and it will run
[40:20] a process, a scientific loop where it baselines, finds issues, proposes, patches and rescores the data so that takes out the guesswork of tuning your genie spaces. And after running it on my
[40:35] little fake data set that I uh that that I have, it pushed my uh my genie space up here. So, one of the things it would be looking at would be the metadata that we generated today. Okay. But it also adds things like benchmarks and so on.
[40:48] But Genie Workbench well worth a look. The code I've shown you today, search for data bricks metadata generator on GitHub. It's a live repo. Wonder about that bug. I'll have a look at that. Um but it's available to you today. Thank
[41:01] you so much for your time. Thank you for also for presenting with me.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.