A–Z of Unity Catalog Business Semantics: Open and Unified Semantics for Agents, Apps and BI

Summary

Unity Catalog's unified semantic layer — spanning business glossaries, Metric Views, and Genie Ontology — provides a single governed source of business definitions that flows automatically to AI agents, applications, and BI tools.
Agent-driven queries are growing exponentially on the Databricks Data and AI platform, and without proper business context AI systems hallucinate company-specific metrics; the semantic layer grounds agent reasoning in certified definitions.
Genie Ontology infers an entity graph from lineage, files, and permissions, while Metric Views decouple physical data definitions from logical business calculations to enable reusable, governed KPIs across the platform.

A–Z of Unity Catalog Business Semantics: Open and Unified Semantics for Agents, Apps and BI

Business semantics is the missing bridge between raw data and real understanding for humans and AI alike. In this comprehensive session, the product team behind Unity Catalog’s business semantics takes you from A to Z: what business semantics is, why it matters and how an open and unified semantic layer transforms the way agents, apps and BI tools consume your data. We’ll start with the foundations, defining and governing business glossaries, tags, and semantic metadata within Unity Catalog. Then we’ll show what’s new: the latest capabilities that make semantics flow seamlessly across AI agents, applications and BI tools from a single governed source. See live demos of how unified semantics improve agent reasoning, enrich app experiences and build dashboard trust. Whether you’re just starting or scaling enterprise-wide, walk away with everything you need to make your data universally understood.
Talk By: Can Efeoglu, Director of Product Management, Databricks ; Jasmeet Jaggi, Sr. Engineering Manager, Databricks ;

Chapters

FAQs

What is business semantics in Unity Catalog?

Business semantics in Unity Catalog is the governed layer of definitions, metrics, and entity relationships that bridges raw data and business understanding for both humans and AI systems. It encompasses business glossaries, Metric Views for reusable KPI calculations, and Genie Ontology for automatically inferred entity graphs.

Why do AI agents need business semantics to answer company questions accurately?

Without business context, AI agents querying company data hallucinate metric definitions and produce inconsistent answers because they lack knowledge of how terms like 'revenue' or 'churn' are specifically defined in the organization's data models. Genie Ontology and Metric Views provide the grounding layer that enables agents to produce accurate, consistent results from governed definitions.

What is Genie Ontology?

Genie Ontology is an automatically inferred entity graph in Unity Catalog that is built from data lineage, source files, and permission structures to model the relationships between business entities in your data estate. It helps AI agents reason accurately about company-specific data by understanding how entities relate without requiring manual graph construction by the governance team.

What are Metric Views and how do they improve BI and AI consistency?

Metric Views are reusable, governed business calculations defined in Unity Catalog that decouple logical KPI definitions from their physical data implementation. By acting as a single certified source for metrics like revenue or active users, Metric Views ensure that Genie queries, BI dashboards, and AI agents all produce consistent answers from the same governed definition.

Full transcript

[00:08] All right. So, I feel like we've been all, you know, bombarding you with semantics, ontology, context all throughout this. So, hopefully I can actually kind of put a lot of these things together and kind of explain like how we kind of starting with how we, you
[00:22] know, see the larger problem. Um like we talked about metrics, we talked about knowledge stores, we talked about ontologies, we kind of talked about automation versus userdefined semantics in the keynote and like later on the
[00:35] genie talk. But how does that all come together? What's the rationale? I'm going to cover that a bit. Then I'm going to help you guys think about like our individual solutions to each of these different problems and kind of walk you over like how they're being
[00:47] solved. And then JMI is going to cover a lot of the deep dive section about like how we're doing modeling, what's new in modeling, how we kind of approaching querying and performance and a lot of those things. And then we'll close it off with a demo.
[01:00] Um, so the first picture I want to or first chart I want to show you is this is actually our internal environment's query counts. Um, you know the y-axis is amount like number of queries per day
[01:12] and that yellow series is actually agent-driven queries. And even in our own internal environments now, you know, it seems like within the next two to three months, we're gonna have an order of magnitude more agent-driven queries
[01:25] happening on the platform compared to like the human-driven ones or job driven ones which are the red one and the green one respectively. And this is not something we're just seeing on our side, right? This is something that I whenever whichever
[01:38] customers I talk to, whichever partners I talk to, this is what we hear from them. And with this comes a new set of problems, right? So you have all these agents, your Claude, your genie,
[01:50] whatever you name it, right? And then everything looks great, right? They're very confident. They give you some answers, but you know, then you'll see, hey, is that the right number? I was like, "Oh, sorry. I get that right. I
[02:02] get that wrong. Actually, I made that up." And this actually is a real example we were going through on an unset up like fresh you know instance one of the third party you know uh harnesses that kind of happen. And this is not a simple
[02:15] problem to solve that's going to have like hey if I do this I'm going to get that right. It's actually like a larger you know concern on like how to approach context in general and what pieces in the stack you need to be able to solve
[02:27] that. And just think you over like a basic question right you know if you want to answer the question about hey what's our revenue per active customer in Q3 um what does the agent need to know to be able to answer that question right it
[02:39] needs to understand what is revenue what is active customer how do I define Q3 like is that a fiscal Q3 regular Q3 if I don't say fiscal is that actually the right thing so there's a bunch of concept entity mapping you know things
[02:53] that you have to do there's a bunch of just in general like definitions that it needs to understand and then there's a bunch of retrieval things about like which tables to how to join the tables how to calculate the exact expressions and all of those things but all your
[03:05] agencies instead of everything I talked about is FCT_revenue daily with some random column names and if you're you know if you did your homework well and if you actually spend the time maybe they'll actually see nice
[03:18] comments on the tables maybe they'll see lineage but in most cases that's not going to be enough to get you like you know 95% plus reliability on all of those questions Um and you know when we look at the
[03:31] market and other players right we kind of see this being reflected for example you might have seen like a recent blog post that's been circulating you know X and LinkedIn right now from entropic about talking like their experience
[03:43] about like building semantics using direct cloud in their case and then how that works for them and when we kind of look at their stuff right they have three types of problems that they're seeing one of them is the
[03:55] concept ambiguity Right? The stuff we talked about, what is revenue, what is Q3, how do I map that? All of those things. The other one is staleness and drift, right? Because once you define these things, they get out of sync very quickly. So like how do you kind of keep
[04:08] them up to date? You cannot expect like a human to be building all of this stuff one by one like every day. It's not going to work. And finally, like once you know the concepts, once you saw the mapping, how do you actually get the query to run well? And in their case,
[04:20] right, what worked well was a combination of things, not just one thing. They swear by having a govern semantic layer that is formal and defined as the basis for the truth, but
[04:32] then being able to connect that metric layer with lineage to other data assets so that they're not just depending on the metric layer, but they're depending on the definitions in the metric layer and the assets that need to happen under there. On top of that, you need all of
[04:45] the governed per domain knowledge to be somehow encoded in there, right? That could be in the form of docs that could be in the vicks and then you need to organize all of these things. Just to give you another example, Uber right
[04:58] it's not the same exact failure modes but you know things that we see on our side too right in their case right it was about like hey how do I map intent to a schema I have 10,000 tables I have many domains of interest I like in the
[05:11] case of Uber for example you might have Uber Eats Uber rights Uber whatever right the new product line they're launching and then the agent doesn't know what I'm talking about whenever I ask these questions and what worked for them in their case was about hey let's
[05:24] partition all of our assets Let's let's partition all of our knowledge into domains. So both data and metadata, organize them around access patterns and retrievalss. Let's get rid of, you know, unnecessary things. And let's make sure
[05:37] there's enough instructions for routing so that the agents know what to get at the right point and what to ignore at the right point. And you know, when we're kind of thinking
[05:49] about all of these things coming together historically, right? Um we saw first the rise of metric stacks right you know the Bob J universes all lab cubes more recently metric layers many
[06:01] different flavors in the BI tool outside of the BI tool at the same time we saw the rise of like you know data dictionaries ontologies knowledge graphs LLM type wikis from Karpati's recent like tweet if you saw that and then
[06:14] separately you have all of this ungoverned stack that you have which is about reasoning over everything else and if you think about it these three pillars are covering How do I make the calculations right deterministically?
[06:26] How do I reason over the concepts so I know which calculations to make? And then finally, right, how do I get context so that I'm I know that I'm doing the right thing for my product, for my for my domain, for my business, which might be stored in a Google Drive,
[06:39] Teams, chat, Slack, somewhere else that's not necessarily formly modeled. So a bit more free form, unstructured. And that's how we look at Unity catalog semantics as a solution, right? How do I bring all of these disperate pieces into
[06:52] a unified, you know, semantic foundation that can be consumed by agents, that can be consumed by SQL, from BI tools, you name it. And just to kind of build this
[07:04] up together a bit, right? Um, so the first thing is open connectivity, right? You need to make sure that you can connect into the data lakeink, you can connect into all these like different federated data sources. That could be
[07:16] your databases. That could be your data warehouses. On a third pillar, you need to make sure that you can connect into the business apps, right? These could be your Slacks, Teams, Salesforce,
[07:28] you know, SK little things. And finally, you know, if you're if you have been a data bricks customer for a while, I'm sure you'll have tons of notebooks, dashboards, SQL queries, a bunch of stuff in your data catalog. So, all
[07:40] these rich workspace assets that already have a lot of this stuff encoded. And then on top of it, right, you need a way to define multiple levels of things. You can kind of think about how we're
[07:52] terming unity semantics as the user control layer, right? This is the layer where you're defining your formal deterministic metrics, your relationships, how things should be calculated. On top of that, you have the
[08:04] new glossery concept that we're adding to Unity catalog. This is where you do things like taxonomy management just general conceptual management and then linking that back to metrics back to data back to dashboards back to genie.
[08:16] So between the glossery and the metrics you can kind of think about it as a concept layer and a formal metrics layer coming together and domains finally on top of that is how you organize all of this stuff. A very basic example is
[08:29] domains could be around business business lines for example sales marketing products. It could be around uh more projectoriented smaller domains but it is a basically a way to partition so that the agents do not have a context
[08:43] overload and then for discovery for your end users discovery for agents you have a grouping and this is basically the user control layer that I talked about. Uh you can obviously bootstrap all of this with AI which I'll cover but this
[08:55] is controlled strictly by you. On top of that is geneontology and you probably heard Ali talk about all of these things already. Genie ontology gets empowered by both the bottom section all the open
[09:08] connectivity stuff we talked about right so it ingests your files your you know your PDFs your business app information from Slack your Google drive stuff all of your data but it also can ingest
[09:20] stuff from the semantics layer and genontology is fully inferred it's all automatic right it is an everarning continuously improving mechanism to determine how different things relate to
[09:32] each other what metric snippets could be useful, what you know terms could be useful, who who in your organization have the authority, right? Who who's mostly right, whose like assets are used the
[09:44] most. So it is a graph of entities, of people, of like data, of everything that you you're feeding into it. Um, and it works in tandem with semantics. You can kind of think about ontology as the
[09:56] inferred layer to get you the breath you need because you cannot obviously do all of this by yourself, right? It's going to take time to be able to do all of these things and then the semantics as the user control layer for you to
[10:08] annotate the truth right without having to do you cannot model everything up front with semantics but which ontology can cover that breath and semantics come in to make sure that you can concisely
[10:20] encode what something means and all of this is obviously served to genie MCP SQL APIs dashboards within and outside of data bricks so you can connect with everything we have internally and it will just work out of the box. And then
[10:33] you can also bring any third party hardness, any third party tool through our GDMCP either at the space level or you know at the account level to access all of these things. Um so now I'm going
[10:45] to quickly go over some of these pieces and then uh leave that back to Jasmine. So on the metric side, this is a generally available product that we already launched on the metric views and
[10:57] we've had tons of improvements which we're going to go into a lot of details on uh in in just meet section. But this is basically where you encode your measure definitions, your reusable dimension definitions, how all of these
[11:09] things relate to each other. You can do all of this with agenting tooling both from migrating from thirdparty tools you starting yourself or you know drag and drop UI and then opt on on top of that
[11:22] you can materialize all of it parts of it aggregates so that you can get the best performance if the underlying data is big and you don't you cannot just access it once and everything we talked about is accessible through third parties as we discussed through seat
[11:35] through through like MCP SQL and all of those APIs glossery which which is something we're announcing at summit is the definitive authorative source for concepts. This is you can kind of think
[11:47] of it as as V keys that you can link to other Vikings. You can link to people, you can link to assets, you can link to notebooks. This is a mechanism for you to basically encode everything that is not a metric or something formal like a
[12:00] SQL expression. Um you can partition these and organize them under domains which I'm going to talk about soon. And the nice thing is you can also use agentic authoring to author all of these
[12:12] things. It can use genontology which I'm going to show to bootstrap. And then we also offer things around drift detection so that if something is out of date it will automatically suggest that you can update. There's ways for you to have
[12:24] stewardship so that not everyone can edit something. Some people can suggest some people can approve the suggestions. So this is basically a crowdsource version with governance uh of how you can kind of go about defining these
[12:37] concepts in your organization and domains as I mentioned kind of brings all of these together. You can think of these as tags or folders in a way right this is an organization and discovery mechanism and the big thing here is you
[12:50] can put unity catalog assets under domains but you can also put workspace assets under domain. So in a typical domain you'll have tables, metrics, genie spaces, dashboards and a bunch of these other things in notebooks for example.
[13:03] Um everything we talked about uh you know as as much as possible we're open sourcing. So the big thing that we've recently open source is metric viewcore implementation and this is actually
[13:15] already open source in spark and unity catalog. Um at the same time we're observing what's happening in the market right for example there's an initiative called open semantics interchange we're a member of that we're looking at other
[13:28] open standards and open like opportunities to be part of open standards for things like the the wiki aspect the concepts aspect there was actually a proposal from Google just like a few days ago that we're evaluating we submitted our own glossery
[13:42] spec into open sharing protocol that we announced earlier so a lot of things are in movement we actively looking at it recipe assured we already open sourced a bunch and then that will be our motto uh going forward in this area and beyond
[13:55] open sourcing right you want to make sure that everything that you're defining in one platform can be accessed in many platforms and this is BI tools this is catalog tools this is other thirdparty modeling tools like Irvin
[14:07] right and you know this is just a quick scheme because that's what I was able to fit in the screen but you know there's more and this is something that you'll probably see us you know improve over the next working our partners.
[14:20] Um, and just to hit on the ontology side very quickly, um, Genieontology as you heard earlier is the inferred layer for you to be able to do a lot of this stuff automatically without having to
[14:33] explicitly define something. Uh and geneontology works by you know having a dynamic secure graph uh automatically constructed from your underlying you know data sources your
[14:45] unity catalog assets both within data bricks and assets outside of data bricks and then it provides security as well as fastness because everything you're in the ontology is aware of your
[14:58] permissions whe at the file level at the object level at the data level. So if you can see something in Geneontology that means you actually have access to this. So we do not leak for example context across different users. And just
[15:10] to kind of show you how like this works right we ingest a unity catalog lineage information semantics information glossery domain information we get the workspace access such as notebooks
[15:22] dashboards pipelines files and other workspace things. We talk to partner tools as we talked about like this could be MCPS or this could be file-based and this gets all fed into the genetology. Then this looks at the different pieces,
[15:36] the people, the permissions, the topics and then does a graph ranking internally to figure out what should the authority and beyond just the permissions right it
[15:48] helps you know resolve conflicts looking at the authority looking at frequency looking at usage looking at who's using it and then this then is inserted automatically when you're talking to Genie or when you're doing something without you having to do an additional
[16:01] action and as I mentioned right partners is very important for us and making sure that you can fuel ontology but not just your database stuff but with but a lot of external content. So beyond just
[16:13] databases we have partners on the application layer and all the tools that you use today. I think I said drive and teams like multiple times today but you kind of get the idea. A lot of this context is not in your databases or data wises but in apps. So that's why we want
[16:25] to really make sure that this stuff is connected pretty easily. And just to give you a quick idea about how all of this comes together with ontology. So this whole inferred semantics idea with the defined semantics right let's say my question
[16:39] now is you know hey why did activation drop last month even though signups went up uh but what is activation what is signups right the ontology or or the agent needs to be able to understand this stuff. So in this example right I
[16:53] already had a concept defined for activation in the glossery right under the product analytics domain I said hey this is activation here are the link assets and that link asset in this case is actually a metric view which tells me
[17:05] exactly what measures I have what dimensions I have so that I know how to exactly do the calculation but maybe the other concept I had in there signups right wasn't something that I actually defined up front but
[17:18] someone in my organization did a notebook where they actually talked talked about the activation funnel and defined it as a concept and kind of gave me some calculations that genetology automatically parsed and learned how to do. So even though I didn't define that
[17:32] up front, I now know that hey because of this notebook here's the definition here's the source I can use to calculate it and now even though that was an unknown concept for me I know about what activation means because that was defined and I know what signup means
[17:44] because defined and I can answer this question holistically and this is just a very basic example but geneontology gives you this ability you can upload it you can download it you can kind of make sure that you can certify some things to
[17:57] give it more trust so it gives you a way to manage breath and depth and authority at the same time which is I think a big part of what we're going with. At this point I want to switch gears and I want to hand it off to Jazmite to walk
[18:09] you over um some of the detailed you know modeling experiences, performance experiences and the query experiences you're building. Just me.
[18:22] >> Hey everyone uh thanks John for a great overview of the three pillars of business semantics. Uh I'm just me. I'm an EM work with John on building some of these capabilities. I'll be doing a deep dive on our metrics layer which is uh quite foundational to getting your
[18:35] numbers right. To start with uh we built this metric view object in Unity catalog which is a view- like object that lets you model your star snowflake schemas help you
[18:47] define the reusable metric definitions. You can define how your user count is done or any business metric and define the SQL expression for that. How it's
[19:00] different from a regular view is uh views typically get computed up front whereas metric views are computed on the fly during the query time. So your non-editive measures do get calculated correctly and you don't have to worry
[19:12] about getting them right. Metric views also offer you ability to encode a lot of metadata. You can put in your uh display names, synonyms, format
[19:24] patterns, instructions and verified queries along with metric views. This helps uh genie like systems to get your Q&A right. And also if you've tried dashboard authoring genie, it builds a
[19:36] dashboard which already has the fit and finish that you expect in terms of display properties. While these objects are great for star snowflake schema, we also uh have
[19:50] introduced a capability called relationships. This helps you model connections between multiple metric views and shared dimension tables. So you could define like a many to one join from multiple metric views. Define
[20:02] complex ratios or calculations. For example, average revenue per user where your total revenue comes from one fact and your user count comes from another one. This feature is live in public preview in dashboards and we'll soon be
[20:15] adding such capabilities into our Unity catalog offering. As we expand the capabilities that metric views can offer, the kind of complex questions it can answer off the
[20:27] bat with determinism. One common pattern we see is level of detail calculation. Often people want to calculate a metric at different generality from their main query and the reason to do that is to do
[20:39] some uh comparisons between different grains. We see three patterns today. Fixed LOD lets you model calculations which are computed across your table. So if you have a table where you have
[20:51] revenue for a customer and you want to calculate lifetime sales for a customer, you can do it through fixed LOD. With course LOD, we let you model things like relative ratios. If you want to do
[21:04] your monthly calculations of how does my each month contribute to my annual sales or quarterly sales, you can model it using course LOD. And we also have a more advanced case where sometimes you want to do a multi-level aggregation to
[21:17] calculate let's say maximum daily sales for a sales rep. Now you first want to calculate your daily sales for a sales rep and then you want to find the maximum value. This is done through finer LOD. Both fixed LOD and course are
[21:31] available in the product today and will soon be launching the finer LOD capabilities. As people have started to adopt metric use more and more, a common pattern is
[21:43] for people to do similar type of analysis across uh the same kind of calculations and typically you do it through templatized SQL or you have different capabilities. What we've done is made
[21:55] our metric views parameterizable. So when you define, you can just have a single canonical definition in your metric view and you can control your input values through a dashboard, a SQL notebook or any surface that you use to
[22:07] consume these objects. Typically people want to do like uh if you want to have your output currency control where you want to see a report to be viewed in a local currency, you typically want to use parameters. If
[22:19] you want to do filter controls where certain users want to only want to see a certain data value, then you use parameters. And also if you're doing comparison analysis between date windows, you can parameterize the dates
[22:31] and then u just control them during your query time. This is also very useful for reporting use cases. Sometimes people want to build a pageentated report and they want to control pageated variables using
[22:44] parameters as well. This is one of the arguably like the most asked feature that we get. How do we do time intelligence? Uh a lot of you are probably familiar with that in
[22:57] PowerBI. Um we've used window functions from SQL and expanded that capability in SQL to help you model a lot of time intelligence in data bricks. U recently we've introduced two key capabilities.
[23:10] One is offset feature. This lets you control the offset of your window calculation. So you can now define calculation of seven days but you can say do it for offset it by seven more
[23:23] days. So you go one week before the calculation. This capability lets you model all kinds of period over period comparisons. You can c you can compare this week to last week, this week from last quarter or
[23:35] this week from last year. All of that is very easy to do with the offset feature. We've also added support for numeric values in range and offset keywords.
[23:48] This is useful for people who don't use Gregorian calendars and often people have custom definitions where they have custom calendars uh with numeric values. You can now model all your 445 454 type
[24:01] calendars in the platform and get all these native intelligence built into it. While these are great capabilities that we have expanded, we've also been
[24:13] spending time in improving our authoring capabilities. Uh we're trying to also cater to a very wide range of uh user base. So people who come in from more traditional BI background often like a
[24:26] UI controlled way of modeling things. So you can now model relationships and definitions and you'll find this UI pretty intuitive and we'll give you a live demo uh later. So you'll get a
[24:38] flavor of for it. But we are also seeing a very wide adoption of agent authoring in our system. As soon as we implement Genie code, we see a lot of people switching to Genie code. So all of these things
[24:50] work out of the back. Uh they are supported. Genie code is intimately aware about all the features we built. So it's able to give you the best latest and greatest guidance on how to model certain business concepts.
[25:03] All of this helps you get to your governed definitions much faster. We've made one more enhancement in this area. Uh we've tried to improve our importing capabilities for bringing semantics from
[25:15] different systems. So you can directly point your PowerBI dashboard to databicks and we'll create a very similar looking dashboard for you. But what we do well here is we try to extract the semantics out of it. So as
[25:28] you see in this chart, you can actually distill semantics from any external BI system, build a single foundational layer which then lets you query them through dashboards, Genie or any third party system or through SQL.
[25:46] These capabilities are in beta. So we're like still uh actively working on it. Would love for you to try it and give us feedback and we're constantly iterating and improving them. This is the last section I want to talk about. While we've improved ability to
[25:59] model stuff, we've improved abilities on what platform can do for you. We're also improving querying capabilities and performance that you get out of the box from the system. If you look at this diagram, one common
[26:12] pattern we see this is a very common problem that often people have their metrics defined in base tables and then they have multiple repeated tables which are pre-agregated at different
[26:24] tularities. Uh this creates a slew of problems. One problem is that people humans and agents need to now reason across multiple tables which look similar have same set of fields but they
[26:37] don't know which one to pick. if they pick the wrong one either they get a query that takes too long and in worst cases it can actually give you the wrong result. The other problems it creates is when you have these many similar tables
[26:49] you also end up creating more complex semantic models where you do a lot of many to many joints and these are typically very hard to reason and every system does a very proprietary job of handling them. So data modelers are very
[27:01] constrained when they have to model some of these things in other tools. In data bricks platform what we've done is we've separated the logical
[27:13] definitions from the physical definitions. So you can just expose a single metric view as a logical definition to your agents to your humans. This is a layer above your gold layer which is what we're calling the
[27:25] platinum layer where you get a very well curated definition space where it's easy for multiple systems or agents to work on. And then we separate the physical tables which are created through
[27:38] materialization in the platform. And we have a very smart engine in between which looks at your queries that are authored on metric views and routes them appropriately to the right materialization.
[27:52] What can you materialize today? Okay, so in a metric view, you can go into the YAML or the UI and you can define a materialization for your source object which lets you model your source query which premputees your source query with
[28:04] all the joins and all the filters that you defined in a metric view. We call this unagregated materialization. You can also pick and choose your commonly used metrics and their dimensions which are very popular. So if
[28:17] you're computing weekly aggregates on certain slices then you can predefine that calculation and the system will create a materialization for you. We're also working on inspecting your query
[28:30] patterns and recommending which materialization will be best for you and help you save uh cost and also improve performance of your system.
[28:43] create a DT pipeline which knows about the metric view definition. It's managed by the system. So anytime you make any changes to access control permission privilege even deleting a material
[28:56] deleting a metric view these materializations are managed by the system and all of these materializations are computed incrementally. That is we don't run the full expensive query. It relies on the enzyme and incrementalization platform in data
[29:09] bricks to make sure that it's cost efficient for you to compute these materializations. How do we help rewrite the queries? There are four broad patterns. The
[29:21] unagregated match is basically anytime you run a query on metric view, we look at if there's a source metric join metric materialization, then we just go to that table as opposed to computing your joins and filters again from
[29:33] scratch. exact matches. If you run a query where you're where you're looking up a metric at a certain grain, if we find something that matches in our materializations, then we just quickly swap it out. These are like basic
[29:46] examples, but we have more sophisticated stuff where we decompose your metrics smartly to figure out if there is something more relevant that can help answer this question faster. A a basic example could be that if you have a
[29:58] average which can be computed of sums and unique counts and you materialize the parts that can help build this answer then we will just directly find those materializations for you instead you instead of you having to compute all
[30:11] metrics ahead of time. The rollup match is also fairly sophisticated. We try to find the smallest possible table. If you've done let's say at date granularity you have weekly level rollups or materialization
[30:25] or you have a monthly level rollup. We're able to find the smallest possible table that can help answer this query so that you get the best performance and cost.
[30:39] This is a big enhance. This is a very different from materialization but this is a big enhancement we've made to metric views. Our goal is to have metric views be more and more close to table like object and we let you query met
[30:51] metric views in SQL. Uh but this feature lets you also join metric views to other metric views or tables. With this capability you get these four big benefits. One, if you're quering
[31:03] metric views in SQL you can write a much more concise SQL. Now earlier you had to write a CTE on each metric view query and then join them together. Now you can just write them select measure one, measure two and join metric one to
[31:16] metric two. The second big advantage you get with this capability is that because you're controlling the join like any other like a regular SQL operator, you can control the join type. So currently
[31:28] you're limited to only left joins in a metric view. But if you control a join during query time, you're able to choose which type of join you want. And this is something that people want to control in certain analysis and not have this
[31:40] restriction to only be limited to one joint type. This also segs sets it up for more advanced analysis. You can do post aggregate joins which lets you do currency conversions. You can easily do
[31:52] chasm traps where you can join multiple metric views in SQL. You can do more dimensional extensions in SQL. If you didn't join a particular dimension table in your metric view definition, you can just do it on the fly when you're
[32:04] writing SQL queries
[32:16] more intimately. They can treat it like a table. So for the for let's say a PowerBI or a Tableau, they can just write a SQL query where they can treat metric view and table symmetrically. This should allow for more complex
[32:29] analysis to be possible in external systems. And I'm sure we are all super excited about the lakehouse RT announcement today. And uh I just want to call out that metric views are supported uh
[32:43] natively in the platform from day zero. So all capabilities in metric views just work from get. At this point I'll jump to a demo.
[33:53] All right. So I'll just maybe show you a quick flavor for what we've built so far and how you can build metric views in the platform. You can go to any schema and you can start creating your metric views. For this demo, we'll probably use a TPCH
[34:06] schema. Now let's say we want to, like I said, there's a UI for it, but we've gotten so comfortable using Genie
[34:18] code. So, I'm just going to stick to Genie code for my demo for a large part. I'll give you an overview of what the UI looks like, but uh bear with me. Help me create a metric view with line
[34:32] item table as source. Join it with orders and supplier table. also include
[34:47] any snowflake joints that and add more dimensional fields.
[35:02] So at this point I've just given a generic like uh I'm trying to build a metric view for my line item table. Uh for people who are familiar with schema we find it relatable. uh GD code will access all of your schema, look at all the tables. It understands all the
[35:15] metadata and query patterns from before. It's smart. It knows your primary key, foreign key relationship. So, it relies on a lot of these signals that are already in the system. And hopefully, it should get what I want.
[35:53] So at this point it's already identified that there's a fact table line item. It can do a direct join to orders and supplier. It also identified orders can be joined to customer which can then be joined to nation then to region in a snowflake pattern and it'll just model
[36:08] all of that. It also works on while it's modeled all these joints uh it also looks at some of the query patterns or common definitions and it's able to come up with measured definitions which might be useful for you. So you don't have to
[36:20] do all of that work from scratch every single time. So if you look at what it came up with, this is the join schema uh which it found out and then it was able to create
[36:35] multiple fields for us. I'll close this preview. As you can see the overview, it created eight different
[36:47] measures and exposed 18 dimensions. It also um created date dimensions, ship month based on common usage where month aggregation was used.
[37:00] If you look at measures, you can actually uh in the UI you can go inspect the expression. You can also look at the the details. You can add windowing to it. You can add other metadata like display name, comment, synonym. You can
[37:12] also provide format patterns which can then which are respected in your AIBI dashboards from uh by itself and then you also get this neat capability to preview a metric.
[37:25] So this is t we've built this capability so that when you're defining a metric if you don't know what the if it's right or wrong you can sometimes look at the numbers to make sure you got it right.
[37:41] So at this point I'll expect accept all of these changes. Let me try to do something more complex. Help me create three more metrics. One for this week's revenue,
[37:57] one for last week, and one for week over week group.
[38:11] I think uh I'm pretty sure it'll figure it out. Uh I'll skip the third part of my demo, but this will at this point this we want to make sure the agents help you not having to worry about the syntax or figuring out any data brick
[38:23] specific details as long as you know the intent of what you're trying to do. The system is able to build these metrics and we will show you a UI for it and you can go into the YAML to inspect to make sure it's doing the right thing. But at
[38:36] least the authoring part agents take away the need for proprietary interface knowledge.
[38:59] So it's just sharing all the details of what it figured out. This week's revenue is shipw week range is current. And it also use this new offset capability and it will figure out how to add that offset. The last thing I was going to talk about I'll just quickly uh say
[39:12] mention it is uh you can also define your parameterized calculations. You can do things like hey create me a metric which is parameterized create a current date window range create a baseline date window range and create a growth metric
[39:24] for me. It will just figure out how to add these par parameters. Also, it does a good job of explaining how you can query these if you're using SQL. And if you go to data bricks native dashboard capability, these things work out of the
[39:37] box. And if you go to Genie, Genie is also aware of all these definitions. So when you're asking a question, it will write get the syntax right, but it'll also know that it can do a more parameterized query. So it can just rely
[39:49] on more of these platform capabilities instead of coming up with a SQL completely on the fly.
[40:03] Let me just show you two more things and then I'll hand it over to John. metric view. I can quickly jump to genie. This creates a genie space. With
[40:16] this metric view, I can ask what is my total revenue by for each ship mode. We need to wrap it up.
[40:30] >> Okay, I think we're at time. I'll skip the remaining part of the demo and I'll call John over. Maybe we can just quickly
[40:43] walk you through uh the points I was going to make was Genie is able to leverage these metrics. So wherever it finds these uh curated definitions it relies on that and in cases when it's not aware it is able to come up with more complex SQL and in dashboards you
[40:57] can actually uh just import these objects and start doing your analysis in a very low code manner which some of your business users may be more familiar with. Yeah, >> just to close off it, I'll show you like as as we're doing the closing remarks
[41:10] like what all of this looks together when you put all of these in a domain and organize it as such. So, as we said, domains is a way for organization. You can access them through the new discover UI that you have and in a domain you can put glossery entries, you can be metrics
[41:22] as just me was showing us you can associate genie spaces, you can tables and then all of these work with each other, right? You can have a page that links to metrics that is accessible by
[41:34] Genie that can be governed user. And as you kind of build your estate of metrics, assets, dashboards, genie spaces, right? You're going to have you want to have ways to discover, you're going to have ways to organize, you're
[41:46] going to want to have ways to make things go faster. And that's what we're trying to build with unit catalog metrics, the domains, the glossery, and the ontology piece to kind of help bring all of this together so that you can
[41:58] have both breath and authoritative depth. And that's all we had for you today.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.