Skip to main content

How to Get the Most Out of Your BI Tools on Databricks

Summary

  • Lakehouse BI Performance: Databricks accelerates query performance through features like Photon optimization, Databricks SQL, and Liquid Clustering, while Publish to Power BI integrates seamlessly with existing BI tools.
  • AI/BI Genie Integration: AI/BI Genie provides natural language interfaces for analytics, enabling users to generate complex SQL and discover insights faster than traditional BI workflows.
  • Migration and Optimization Strategies: This video provides actionable strategies for migrating from data warehouses or optimizing existing setups to maximize BI capabilities and reduce latency.

How to Get the Most Out of Your BI Tools on Databricks

Watch: How to Get the Most Out of Your BI Tools on Databricks
Talk by Kyle Hale, DBSQL Product Specialist, Databricks
Unlock the full potential of your BI tools with Databricks. this video explores how features like Photon, Databricks SQL, Liquid Clustering, AI/BI Genie and Publish to Power BI enhance performance, scalability and user experience. Learn how Databricks accelerates query performance, optimizes data layouts and integrates seamlessly with BI tools. Gain actionable insights and best practices to improve analytics efficiency, reduce latency and drive better decision-making. Whether migrating from a data warehouse or optimizing an existing setup, this talk provides the strategies to elevate your BI capabilities.
Intelligent analytics for everyone: https://www.databricks.com/product/business-intelligence
BI meets AI: https://www.databricks.com/resources/ebook/business-intelligence-meets-ai
Explore the only unified platform for agent systems: https://www.databricks.com/product/artificial-intelligence
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What are the key challenges traditional BI systems face?

Traditional BI systems struggle with slow query performance on large data volumes, scalability issues, high infrastructure costs, complex ETL pipelines, and data silos. Organizations often manage multiple vendors for BI, ETL, and data platforms, increasing complexity and costs.

How does the Databricks lakehouse solve BI challenges?

The Databricks lakehouse combines storage efficiency with compute optimization through features like Photon acceleration, Databricks SQL for queries, and Liquid Clustering for optimized data layouts. These capabilities enable fast queries, easy scalability, and reduced infrastructure costs.

What advantages does AI/BI Genie provide for analytics?

Genie enables natural language queries for analytics, allowing users to generate complex SQL significantly faster than manual writing. This democratizes analytics access, speeds insights discovery, and improves overall BI team productivity on the Databricks Data and AI platform.

How does Databricks integrate with existing BI tools?

Databricks supports publishing analytics results to Power BI and connecting other BI tools directly. This enables organizations to leverage the Databricks lakehouse performance while maintaining existing BI tool investments and user familiarity.

Full transcript

0:00 Intro

Welcome to how to get the most out of your BI tools on data bricks. First, our forward-looking statement. Everything on here says may, will expects could, might, no promises. So, uh don't don't take anything here as word, but we do have some good plans. So, there's plans though. Uh please be sure to complete your surveys. Your feedback has a lot of impact on Data AI Summit content. You'll get a survey for this session and every session you attend.
Open the app, select my surveys, give content, uh, give me good feedback. Appreciate it. Uh, by good feedback, I don't just mean like five stars. I like real feedback. So, thank you. Hi, All right. So, I'm a lead product specialist for Data Brick SQL, uh, here at Data Bricks, uh, focused exclusively on BI, data warehousing, uh, and all things SQL. Uh, it's been my entire career. I started right out of school. Got handed my first job working on Crystal Reports.

0:25 Survey

Never looked back. Uh so 20 years doing this. Uh I know a thing or two. I'm here today to share what I know with you. Uh so why are we here? So we're going to review some keybi challenges and how the lakehouse solves them. Uh we're going to give a very BI friendly data bricks product rundown. Just kind of focus on the things that matter. I'll try to say AI as little as possible today. Uh I will provide some advice. Uh and then I'll answer some questions at the end.

1:07 Agenda

We should have probably good 5 10 minutes at the end for Q&A. So, all right. So, what are the key challenges of traditional BI? Well, there's there's myriad challenges, but these are the ones that kind of come up the most that we see in our customers. Slow query performance with growing data volumes, just handling big data, being able to do, you know, kind of BI is really good kind of within a certain data set and at some point it just kind of starts to fall apart. you have scalability issues and high infrastructure costs that are used to manage whatever scalability you do have if you're on prim in the cloud trying to manage your BI vendors, your ETL vendors, your lakehouse vendors kind of you get a lot of costs associated with that.

1:26 Challenges of Traditional BI

You end up with a lot of complex pipelines. Everybody here has probably done a lot of ETL if you're practitioner um you've probably written some doozies. Uh you end up with data silos a lot where you're trying to pull data from different systems uh kind of bring all that together from a disparate world into your data warehouse and into your BI landscape. Uh and then traditionally BI's been separated pretty heavily from advanced analytics and AI whatever the data scientists were doing over there the machine learning engineers and now the AI engineers they were kind of living off doing their own world again with their own silos their own kind of you know separate governance separate security sometimes copies of your data and they weren't really connected to your BI world so the data lakehouse addresses a lot of these challenges right so first we have the open data lake all raw data any streams speeds feeds structured semi-structured unstructured big small can handle all of your data.

2:31 The Data Lakehouse

And because we put it all into cheap lakehouse storage, we've kind of started to work with reducing costs. We have unified security and governance with Unity catalog. So we don't have a separation now between AI and BI and advanced analytics. Nor do we have the kind of the separation of data silos, right? Like we can put all of our data in here. We can use federation for sources that aren't going to come into the lake alongside the data that's in your lake.
So we can unify all the security and management of access to your data. And then we have all these different engines on top. to allow you to kind of use that same data and that same governance structure for both traditional BI data warehousing load workloads orchestration and obviously AI workloads that are out there. So by unifying all of these and simplifying the architecture we really done is we've kind of said hey for BI folks a lot of those limitations go away especially around scalability performance on large data and combining all your data sources and unifying them into one kind of data catalog.
So there's four key facets of delivering BI on data bricks that I want to focus on today. Uh we're going to talk through all of them. So storage, how you store your data in the lake and the implications of that. Compute, how to do deliver best compute for BI governance, some good tips and tricks on things in data bricks that we offer around governance for BI. And then uh yeah, AI and specifically AI's role in BI, right?

3:38 Overview

So kind of not not kind of like a totally left field thing. And we're going to focus specifically on AI's role in BI. All right. So those are going to come one by time. So first storage. So in data bricks we store our data. We recommend you store it in delta lake but whatever for and when you use delta lake you get one key advantage for this which is like we provide a lot of capabilities in the delta format around optimizing your data as it's stored.

4:09 Storage

So we provide things literally like commands like optimize which distributes your data into the file sizes that you want so that it can be worked well on distributed compute like spark and we also run commands like vacuum. So as you do DML operations like specifically updates and deletes over time we remove data physically from those tables uh and and kind of keep your data neat and clean. We also have analyze so we collect stats on top of your delta lake data and stats are just like indexes in your relational database.
So collecting stats on those helps our engine optimize the queries that it runs against your data in the lakehouse. We have historically we've also had partitioning and zordering and we have a new concept called liquid clustering which I'll talk about on the next slide. But how you lay your data out also can impact query performance and how the engine can actually prune your data efficiently. So you have all these commands at your disposal.
But there's a there's a challenge here which is uh and really quick. How many people here did not brush their teeth this morning? You can raise your hand. I'm the only one really. Okay. Okay. All right. Everybody here's All right. Everybody here's brushing their teeth. Good. So, my dad is a dentist. He is gonna be very proud of you. Uh the problem with brushing your teeth, as you know, if you have kids who don't brush their teeth every day, is adherence.
So, everybody here knows that brushing your teeth is a good thing to do. But most people, not everyone here, but most people don't brush their teeth very well. And as a dad who has a dentist dad, I know very well that's true because he's doing pretty well. Uh and he wouldn't do well if everyone brushed their teeth. Uh so the challenge I have for everybody is that most people don't run these commands. Mo what we did is we actually ran the survey and we ran a study of our customers.
We found that like 75% of our customers never ran optimize ever. They didn't brush their teeth and then they would complain to us about storage costs and query performance and degradation over time. We say have you run optimize? And no they have not. So the metaphor I like to give for this is our new concept of predictive optimization. So we actually do all this for you. It's like brushing your teeth in the middle of the night while you sleep.
So you don't have to worry about this anymore. So this is one thing that we've kind of taken away from you. So we if you turn on predictive optimization, we will automatically optimize all of your files for you predictively. So we don't just do it like we actually look at your data. We run it on when we think it's necessary. So we use an AI algorithm to decide when to optimize your data, when to vacuum your data, when to collect stats on it.
And then with liquid clustering, when to run uh kind of liquid clustering is an incremental thing we'll talk about in the next slide. Those things all come for free. So this happens to you automatically. So my first recommendation is use predictive optimization. And if you do this, we'll brush your teeth at night for you and you don't have to worry about it. And the other thing I'll tell you is if you're in the BI world and you find yourself living in something where you're having to figure out optimize or vacuum or when to run analyze or run, you're doing it wrong.
You've you've messed up somewhere. You should be focused on some dashboard you're building some user customer, you know, like problem that they have and requirements that they want and not be worrying about like are my you know, is my data being stored correctly. When you do this, there's a huge amount of savings. There's a nice little thing that the product team wants me to put in here. There are some benefits to actually doing this anyway.
You should do that at a minimum, but if you do it predictively, we'll just do it for you. It's pretty nice. Okay, so liquid clustering. So, we talked a little bit about how you lay your data out affects query performance and that's in really historically that's been done in two ways. Is one partitioning your data. So you actually took your data and you divided by some set of columns and you said I'm going to put them in individual files and when the user comes and they put a wear clause I'll only read the files that contain the values that are in their wear clause for that particular column or columns.
And the downside to that was if my data query didn't hit any of those then I had to read all the files anyway. And so that kind of like it kind of created this like a little bit of cost. If you really knew the queries exactly you could do it right. for the most part you kind of ended up uh people missing the partition advantage and then creating a disadvantage because they had to read all the files. So next we came up with zordering.
So zordering is kind of a multi-dimensional sorting within files that just tries to put data that's similar close to each other. By doing that in the same way you're able to skip more files and skip data within files when a given query comes out that's discriminating right has a filter on it. um or if even if you're doing a group buy, we can actually distribute that data uh more equitably amongst the workers in a spark cluster like data work SQL uh and perform that query faster.
So those were good, but those were manual. Uh they were usually had to if you wanted to change your logic for partitions was a full rewrite for Zorder was a full rewrite. Uh and they weren't really effective for when you had to do lots of writes. So when we're doing a lot of BI stuff, right, we have these fact tables. We're adding facts like every day, sometimes millions of rows every day. You end up with this thing where every time you would write, you would lose whatever advantage you had at that moment, that point in time before the right of your zordering and your logic.
So as soon as you added rows, now all of a sudden your zordering wasn't effective anymore for all those new rows. So you had to re and cons consistently kind of recmputee this. So what we did was came up with liquid clustering. Liquid clustering is high performance, easy to use, hassle-free. So we actually can automatically select the columns for you for clustering, change them on the fly and you don't have to and it's all done incrementally.
So there's no more full rewrites and as your query patterns change or just even show certain tendencies over time or as you add new data or new columns to your data sets uh it'll just incrementally uh adjust to those. And you can kind of see I'm going to show it in this chart. This one here on the far right the read performance you just look at that little red line that's liquid clustering. So as you add new data, liquid clustering starts to as as it learns your data, it gets a little bit of amplification.
Then it gets a little more stable and over time it gets very very stable and that's read performance. So that means is that it's immune to effectively adding new data or adding even new columns to your data set. It doesn't affect read performance. Whereas with Zorder, every time you wrote new data, it literally kind of had to start over and work its way back down to some sort of stable path. So liquid clustering is great and you should turn it on in all your delta tables.
So my next advice is use liquid clustering. Okay, now you're going, okay, I have to turn on predictive optimization on all my tables. I have to turn on liquid clustering on all my tables. I may have to set columns on my tables. No, you don't have to do any of that. We actually have to do is just use manage tables inside Unity Catalog. If you use these, we do all of that for you as well. We actually auto enable predictive optimization, auto enable liquid clustering.

10:16 Manage Tables

Uh it just works uh as you add new delta capabilities. So when we add the next thing because liquid clustering is not going to be the last thing the last word ever right in organizing your data layout and we may add a new command so you'll automatically get all these and we'll automatically run them for you. We get interoperability with all delta readers. So one thing I hear all the time in our catalog is like oh but you guys create manage tables and they're grids and I don't want to change your tables.
You can actually expose those out as read clones to anybody in any format any compute not just data bricks uh in a nice format that they can go read. So read clones solves for that. uh you can re this these manage tables you totally own and define the bucket. So the storage that they're sitting in the cloud is not it can be owned by data bricks can be managed by data bricks on your behalf but it can also be managed in your own cloud as well.
So this isn't any sort of lock in. So I always hear manage tables are like oh no you're locking me in literally not at all. What we do offer though are these capabilities to automate these things that otherwise you have to do yourself. So you can choose to brush your teeth. you guys are all really good at it here. So that's great. But uh we will do all this for you. Uh and manage tables and really this is the default setting when you set up unity catalog.
So if you go set up a new workspace now you set up unity catalog again we'll offer you a bucket. You can provide your own and by default if you just go and do create table or ingest table or use turn on link flow and bring in a table and you don't change anything else. We'll just write and create a manage table for you. It's pretty great. So the main thing here is use manage tables. This is all baked in. You really don't have to think about this.
And again, BI folks, if you find yourselves like outside of this world using external tables and you're having to run your own optimize scripts and you're having to manage all of your data and you're having to figure out your own sizes because we also have system tables. So, we observe all the manage tables and we'll tell you, hey, this table's growing at rate X. Do you want to do something about it? We'll tell you when we last ran optimize.
But if you do all that yourself, you have to manage all that yourself. Uh, and if you find yourself doing that, I'm not going to come out and say like you're doing all of that wrong, but it's really hard to get right. Uh, it's so hard to get right that most of our customers didn't bother to do it. So we decided to do it for them and we do do it right. Okay, let's talk about compute. All right, so hopefully this is a no-brainer.
You should use data brick SQL. So data brick SQL specifically is for serverless scale. It's designed for high concurrency. It uses Photon by default. You don't have to turn it on and we don't charge you extra for it. I hear that one a lot too. Uh and we have drivers and connectors to your favorite BI tools that can talk to data bricks SQL. So really just I mean the one advice I just want to make sure we really really clear is just use data bricks SQL inside your BI if you're using all-purpose clusters today or you're looking at you know kind of like looking at why use data bricks SQL just know that specifically if you have BI and you have any sort of concurrency that the concurrency model here is literally designed for concurrency the other one is designed for first in first out so when you have mixed workloads like users doing very busy queries and users doing very lightweight queries and you want them to all just like in traffic you want the big trucks and the and the little cars to all go at the same rate of speed and all make it down the road.

12:33 Compute

Um, you need a workload management tool that can do that. That's only available in data bricks SQL. Everywhere else in general purpose is one lane. So if you're stuck behind a big truck, you're just stuck. And if there's too many cars on the road, you can't get on. Tough. You have to wait. Um, so data bricks SQL is really designed for that out of the box, which is what all BI users expect. And the thing I would tell you is like if you're the 51st or 500 first or 5,000 first user of an application, you don't really care.
You want the application to work just as well as you were the first person who ever showed up at it, right? You don't really care that there's 5,000 other users on this tool. You don't care that they're all issuing queries. You don't care that everyone else is watching Netflix. You want to watch Netflix, too. It's the same with data bricks SQL. Data bricks SQL is designed to make sure that your experience, whether you're the first or the last person in to the to the uh application, are getting the same experience.
So, really designed around that. So, just use data SQL. That's a pretty obvious one. Uh and datab SQL in particular has gotten five times faster since April 2022. So what we're looking at here just really quickly is a baseline of queries, BI queries that our customers have ran consistently since 2022, since April of 2022. And we use those as a benchmark. We pulled about 500 of them and we said, "How long did they take to run in April 2022?"

14:10 Data SQL

And then we tracked those same queries because what was the sales last week, right? You ask that every week and you're always figuring that out. So we wanted to say like, "Hey, over time, do these queries get faster?" And what we've seen is like over the years through our performance improvements now that those same queries with a baseline of 100% are down to 20%. We've we've shaded them off and made them five times faster.
We're not going to take it to zero. We're going to take it as low as we can go. We have a lot of performance improvements still to add to this. But one thing I can promise you is that if you're using BI queries over here, you get all this performance. By the way, we've never changed the pricing of data SQL. So all that performance improvement has been free. Um you're about to hear a nice announcement on Thursday about even more performance improvement you get for free.
So, we're we're really excited that like data bricks SQL is consistently getting better. Um, especially repeatable BI queries, we really focus on those. We focus on your problems because those are what 99% of our other customers problems are as well. Okay, materialized views. So, for those of you who don't know what a materialized view is, it is both a view, a definition of some logic along with a pipeline. And that pipeline can be scheduled or run manually to keep that data materialized into a table according to again some either scheduled logic or triggered logic based on the underlying tables.

15:19 materialized views

So you can say hey refresh this every five minutes or an hour or whatever you'd like or you can say hey every time the I add a new order or a new set of orders update this table and materialized views can be and should be incremental. You know, not every logic that you can write in here is what we would call incremental aware, but if you do write something that is incremental aware, then we'll manage this incrementally for you, which is really nice from a transformation for accelerating queries in BI.
So if you have So it's really simple, just kind of define some extra logic on top of your fact tables and dimensions. And then you're going to put this into your BI tool on the front page. So whether it's PowerBI or AIBI or Tableau or any of the other pro you know partner tools that we have out there that are great today you're usually going to have on the front page some kind of analysis that's like high level for everyone to come and consume and usually from there then then go and go they want to go inspect something further on you know subsequent pages in your workbook or the report and so on that page it's a really great opportunity for you to define a materialized view that helps them speed up and accelerate those pages.
So the one advice I want to give you about materialized views is don't start with them. Don't just kind of like go like hey materialized views are going to definitely be part of my architecture and I'm going to start with them and just kind of make sure everybody goes through that path. But what you want to do is use materialized views at the last mile once you figured out here's my dimensional model. Here's the analysis my users want.
Here's what's going to go on the front page of my report that everybody's going to see most of the users all the time and materialize that. And so be very kind of strategic about them, but also use them right there at that time to accelerate that performance because that first user experience again just if we lose those basic PTO principles 80% of users come to the front page, they see what they want to see and they leave.
So if we can accelerate that experience, we've already solved four fifths of our problems and then we can just focus on the longtail. So it's a really great experience for BI users who get materialization who get that performance and especially when you know what's going to be which is really great with BI dashboards because you have this really fixed set of analysis that you're doing right I'm trying to look at sales right for this you know this quarter or whatever it might be I'm to you know look at employee performance or look at operational efficiency I I know exactly what I'm looking at really great opportunity for materialization page two and three I have no idea what I'm looking at let's you know that's where we can fall back to data brick SQL and that scale we were talking about Okay, so let's talk about governance.

17:51 governance

So I I there's a really great paper in uh O' about five years ago. I just took it really to heart said good governance is good discovery and vice versa. And I totally believe this. I believe that if you build a plat so the hardest part in almost any BI person's job like ever is where's the data? How do I get access to it? What does this mean? I don't understand any of this and you know just beg barring and stealing from whatever documentation you have data dictionaries your friend your neighbor you know like you're just trying to find any way you can to understand get access to this data so discovery is really really critical if we can accelerate that if we can make it really easy for a BI person to get started just where is the data and what is it what does this mean that gets them literally half the battle towards building something that might be actually useful for a user so good discovery comes from a lot of different ways but the things I'm going to point out here is like one um you get full uh autocl classification of tags.
So we have a whole concept of tagging. You can tag tables, columns, views, etc. , etc. And then you can use those tags to help people find things quickly. Uh we have auto classification for things like PII. But you can also bring your own tags. You can set tag policies. You can enforce that certain tags have to be set with certain values that users do. And you can use that to inform somebody. Okay, so as long as everybody has a consistent tag policy in your organization, then someone else comes and sees that tag, they can go, aha, this is confidential data, so I should treat it with X or this data is tagged for finance.
I'm in finance, this is for me. Uh, so you'll be able to find all those. So tagging is really, really critical. The other one is AI generated comments. So full again in uni catalog, we find provide full comments on tables, comments and columns and so on. We also have an AI generator that looks at that table and tries to make a good educated guess on based on tables and columns what that particular data set is. And all that's available at the very top bar in our global semantic search.
So when you search for this, we don't just search for the table name or the column name or a notebook uh code value. We also search in this metadata as well. So we'll be able to help you find these things uh more quickly. Uh and then the other piece is we have a table health indicator. So we also provide what's called lakehouse monitoring which is automatic data quality controls on tables. So if you have a table and you're writing to it, you want to check for did any nulls show up or did any values show up that were outside of a certain threshold.
Uh you're able to do that. We have this table health indicator. It'll show you for each one of those checks, did something pass. So if you're a BI user and you're coming in, you're having to do data on electric vehicles. You want to come in and you pull this data and you're like, oh, this is exactly what I need. Oh, but there's two or three failed expectations checks and there's an owner. Now you have something to go do which is go talk to someone.
Now you already found your data. We we solved prep one. Good discovery. We found our data. Now the next thing is uh hey what's up with these data quality issues? Can we address them? And then obviously there's also a request for access. So we also have flow in here directly to request for access. So really really nice kind of features within Unity catalog to help you as a BI user or your BI users kind of get and developers in particular find the data.
Is this for me? There's full relationship diagrams and lineage. you can see where this all came from and how it's being used in other dashboards as well. So, there's kind of a lot of little ways that we expose this and make it really easy to discover. When you build these things to make it easy for those users, you also just implicitly built good governance because you had to set up tag policies and you had to set up requests for access roles.
You had to set up owners. These are important things that you need to do inside cataloges. Cataloges don't do these things for you. These are things you need to do and in doing you will um build good discovery. So, good discovery is good governance and vice versa. So, we have a new product that's just went into public preview. We're going to have a big splashy announcement here about metric views. Uh, definitely use metric views as a semantic layer inside Unity Catalog.

21:18 metric views

So, what are metric views? They are a place where you can define business metrics in one place inside Unity Catalog. We're really excited because attaching it to the catalog opens up a lot of possibilities. you probably have a lot of semantic layers that are connected to your particular BI tools and they're usually pretty good for that BI tool, but attaching to the catalog is really going to open it up so that we can talk to all BI tools and AI tools that are coming and just SQL in general.
So having this in the catalog, having it be centralized and governed just like every other Unity catalog object. There's a certification process. Uh we're also got a really big splashy announcement about data bricks one which is our new kind of consumer hub. Um so that users who are not going to build anything in data bricks but just read from them will have a nice clean experience inside data bicks and one of the key aspects of that is certification.
So you as a BI developer or curator clicks a little button says this particular dashboard is certified or this particular metric view is certified and those users will only have access to those certified assets. So certification is a big part of this as well and then uh you can access this today from our AIB dashboards NIV Genie and SQL but we are planning to start to build bridges out to thirdarty tools as well. We think that the semantic layer is really important.
It should be tied to your catalog. It should be agnostic to the tools themselves. So, we want to build this at the catalog level so we can talk to all these other tools. Um, if you want to check out more like definitely follow the keynotes and there's a big there's a couple of other sessions around metric views. Um, but we really believe semantic layer is important not just for BI but also for AI. So, all these agents and all these assistants are going to live next to your users in BI world.
They're also going to be using these same semantic layers to understand your logic, your rules, your business processes, your glossery, how your users see their data. Those agents need to see that data the same way too to be effective. Okay. Uh we also have deep integrations for PowerBI and Tableau. So we go a little further. These are super elite partners with us. We we love working with them. Uh and so within Unity catalog today, you can go directly from the catalog explorer to publish a data set in PowerBI without having to go to PowerBI desktop.

23:09 PowerBI Tableau

Uh we can publish it right to the service, the full schema including your primary keys and foreign key relationships. Uh we're going to start to add additional features like incremental refresh to that as well. You can even schedule it as a task in the data bricks workflow. So at the end of your ETL process when you're done building your new facts and dimensions or adding your new data to them, you finish your merge, you can then just go and incrementally update that import mode if you're doing PowerBI or any metadata if you made changes uh directly from data bricks.
So really really excited about that. We want to make that experience really seamless to go from your data bricks gold layer from the tables and models that you're building inside data bricks expose them to PowerBI. Similarly in Tableau you can explore Unity catalog data sets from Tableau online with a single click. So there's a click right up there. you can just go from that particular schema that you're on and just go right to Tableau server uh and start to you know build workbooks on top of that data set directly from Tableau.
Uh for the other tools that are not powered behind Tableau, we do have a really great data bricks marketplace. So if you have another tool that you guys use or interested in using, go check out our marketplace. It's right there on the lefth hand side of the navbar inside your workspace. We have a ton of partner connect tools and what this really simplifies is the connection experience. So you're in a workspace already.
You have a user. you have privileges, you have data, what you can do is with all these other tools, you can just go to connect and we automatically kind of propagate. We worked with all these tools to build this connection to propagate that connectivity. So what compute you have, what data you have, what access you have, we'll just automatically go into tools like dbt, uh like thspot, like uh like looker, sigma, other tools that I didn't mention that we don't have the all the deep integrations with.
We're going to have at least a partner connect option for you. Makes it really simple to just connect your data into those tools if you do use those today. Okay, so what is AI's role in BI? This is really important for me because uh so first actually let me get this one out of the way real quick. So we have this really cool feature. You can bring AI functions bring predictions to BI. So BI predictively you know is has been kind of like this weird space.

25:08 AI in BI

We've always had that like gartner curve of hey we can do descriptive analytics what happened and then like someone eventually like diagnostic why did it happen and then there's this like kind of like big divide like predictive what will happen prescriptive do something about it and that like side like it's been there on the BI curve for forever but BI's never delivered on that uh and so this is the f this isn't like the first attempt to kind of bring AI to BI either I don't want to say like oh this is the first ever or anything like that but this is our attempt to make sure that whatever AI governance you have inside Unity catalog so whatever models you're using your data sets that you have um any forecasts you've done, any vector searches that you built, any built-in LLMs that you've used or if you just want to use out of the box open AI or mixtural or any of those or llama uh we want to make those really really easy to bring as SQL functions that you can just put right into your BI tools.
So once you write that as part of your model now your BI tools can consume it. So you absolutely can have scenarios where you're in a tool like PowerBI and you have a hover that when someone hovers, you send some information from PowerBI back to data bricks and spit back out some sort of AI query result from your data sets on documentation or standard operating procedures or let me literally see the transaction on an image because we do multimodal as well.
So there's a lot of really cool things that we're kind of planning in this space where you can extend what you can do with BI to even over to the AI side. And all these features are just kind of available in DBSQL. So they're literally that's the function names kind of right across the top. You just put them in, you give it a few extra parameters and you're on your way. And you can obviously parameterize those with your BI tools and go really really far and do really interesting things.
So highly recommend you guys check those out and start you know thinking about incorporating these to go beyond what's available in your BI tools today uh to AI into predictive and prescriptive. Okay. But this is the other part. So we have a great product called AIB Genie for those of you who aren't aware of it. It's basically our conversational analytics tool. So how do you talk to your data in natural language and then we'll generate SQL on top of it and issue that against your your data set.

27:00 AI Genie

And where we kind of take it further is we go really really beyond just kind of like what's available in your schema, right? We also let you add custom instructions so you can tell it. So the main the challenge with all this always is when the user puts in a prompt, the tool has to understand how to connect that prompt and map it into SQL. And it's really easy if you go in there as you're a BI developer. You just kind of write these queries that are like English language that are barely like one step above SQL.
You're like tell me revenue by state, where country, you know, is it like like everybody who writes natural language queries like that, yeah, your your tools do great. But most of your users are not that sophisticated. They'll just come and ask questions in their language the best way that they know how. They'll use business jargon. They'll use their domains. And you have to be able to translate that into SQL. So we have custom instructions.
We have ways for you to create benchmarks against those custom instructions. So you can write a prompt and you can say this is the query it should produce and then you can ask Genie and see what query it does produce and if it doesn't you can modify those instructions until it does. So we kind of give you this really easy way to test like hey are we capturing business jargon correctly or our users question be answered.
You can also suggest questions right you can provide featured questions. It asks clarifying questions when it's not sure what you meant. Right? So if you ask it a question it goes I'm not really sure how do you define X. Can you tell me how you define it? And the user can add that to the prompt and it'll then start to add that as part of its query. We have trusted functions as well. So it's really really powerful for built and there's two parts of this is like one it can work on top of metric views.
So it can absolutely just use your semantic layer as part of this natural language. It's really really powerful on one hand. The other part is we embed Genie in every dashboard today. So if you have a dashboard and you just build it into AIBI and you just you know build your data set there's always going to be something that happens where you didn't put something on the dashboard that somebody wanted to know about whether it's more detail like down vertically like give me give me more information more transactional level or it's just some particular metric that you just didn't consider.
So Genie is able to augment and bring that other piece in. So Genie can kind of fill in the gaps for whatever you weren't able to deliver on that dashboard due to time constraints. Right? And so that really lends me to my next point and this is kind of like probably the thing the advice I really want to give everybody here is how to use these tools. So if you're in BI, I want you to really focus on this. Use Genie to augment and focus traditional BI.

29:12 How to Use AI Genie

So first let me explain this graph. This graph kind of represents every dashboard in your organization. So everything under the curve is some dashboard that you've built and they're sorted by value. What's value? You guys probably already instantly thought of like the dashboard that's most valuable in your organization and the dashboards that like if you could get rid of tomorrow you just would. So like some people already have an implicit thing.
I'm going to try to give a little definition of it just kind of like starts with first with adoption. How many people use it? Kind of the scope within your organization. The second thing is kind of the scope of the data just like how many questions can it answer? How much data sets available in it? And then probably the last piece is you you know how important is that particular data set because it can be small and only set of users but if it's you know this executive dashboard or the CFO is making you know critical decisions about projects based off that it's pretty valuable.
So it's not like one fit answer but like there's definitely dashboards that fall in the valuable. If we go to that righth hand tail I mean it's just the opposite. It's dashboards that I mean somebody here I built dashboards that nobody used. I definitely have. I've built them. Um, I've definitely built dashboards where the like I got so many requests for more data that like it eventually just became something entirely different from what it was and we had to start over because just like what we built was kind of like too small in scope.
So, we built I've built things that were not valuable. I've definitely built things on the right hand of this curve before. Uh, and I think I learned a little bit about it. But the main thing is you absolutely have dashboards that are like this. And then on top of that, you have unmet demand. So you have this huge thing of there's there's data that's out there that you have never even built a dashboard for or you haven't built it in a way that's useful to actually consume by anybody.
Uh you have forecast right like so a forecast is not even data forecast is a model of synthetic data that's just like simulated on the fly. You may have never built a forecast for some particular you know metric. So you have this huge set of unmet demand backlog. Everybody's got tickets at home that you're ignoring right now at Summit, right? You're going to have to get right back to next week, right? So there's like a huge huge met unmet demand and backlogs. there's this threshold line.
This threshold line is made up. There's no like one there's no magical answer here, but there is going to be a threshold of where traditional BI should focus and that's obviously on the lefth hand side. The things that are most valuable and then there's this long tail this long tail of things that are like of questionable value. I'm not sure if there's value. It might be, but I'm not sure yet. Uh and then there's this unmet demand like, hey, someone really wants this and that's that's really valid.
I just don't have time for it. I've got to prioritize. I've got to pick and choose my battles. I only have so many people, so much time in the day, so many resources. I'm going to focus on things that are valuable. Genie and tools like it can really augment and provide this unmet demand. Now, is it going to be perfect? No. It's not 100% like a solution like where everything becomes instantly valuable. But it gives you an off-ramp for self-service BI because what it says is like I can give you a data set and some instructions.
You can ask it literally anything. So, it can generate dashboards. By the way, Genie can generate visuals as well. So, if the user asks for just like one extra visual, you're like, "No, just go ask Genie for it. I'm busy. I've got things to do. I'm focusing on the lefth hand side. you're on the right hand side. Here's Genie. It's a great tool and we're going to build it for you. So, use Genie and these tools to augment traditional BI so you can focus on the traditional BI that has the most value to most users.
And if you do this really well, like actually Genie is going to really do a great job because Genie really is like super esoteric. It can it can answer the most random questions, very very specific detail questions. It can go down to transaction level uh and back up. Um it can, you know, continue a conversation. It can get to that answer, which is what most people need when they ask for a dashboard. They really didn't need a dashboard. they just wanted an answer and if you can give them something that gives them good answers a lot of times that those requests and that support ticket kind of trickles away.
Uh so I just really want to emphasize that you know like figuring out this threshold like use genie tools to figure out where this threshold is to provide as much value to that longtail as possible and to really just kind of keep your focus on traditional BI because people are always want a dashboard. We need shared understanding like everybody here needs to look at the same number at the same day and agree that like that's our target or that's where we're missing wrong.
Like you still need dashboards. Those are never going to go away. It's just there going to be less of them. And what I hope is that like as people start to figure out this long tale and figure out this augmentation that the concept of self-service BI is is going away. And the real reason is because like no one's going to be by themselves anymore. There is no self-service. You're always going to have now an assistant or a co-pilot or a genie sitting next to you.
So help that genie with semantic layers and materialized views. help them with this other pieces with good discovery on their part so they can help those users because that's where like right now sell services I don't understand this and they call you and then you have to come and sit by their desk and walk them through it anyway so just make Genie do that make Genie part of that solution uh and if you do that then like where we're where AI assisted BI is going to be coming it's going to take over this nice long tail it's going to trickle up but there's always going to be a need for these dashboards and we're going to figure out where that threshold is so then you can focus on where things that are more valuable and you kind of let Genie figure out the rest Okay, so let's just kind of repeat the key takeaways here.

33:56 Summary

Let every brush your teeth and I mean manage your tables. All right, so do that. Uh use day SQL. That's uh should be a duh at this moment. Uh accelerate your dashboards, materialize views. Highly recommend it. Uh definitely just a quick way to get good performance wins. Good dis good governance is good discovery and vice versa. Just really want to focus on if you get discovery right and a lot of times that's where the people see by the way people like governance is also like eating your vegetables, brushing your teeth.
We hear that all the time like discovery is valuable like people were people like oh you've made this discoverable and they didn't realize that you also made it governable in the in the same vein. So use discovery as as a kind of like also like a Trojan horse to get governance in uh semantic layers have never been more important. So BI level absolutely but AI and all these tools that are going to use these as well also are going to rely on semantic layers for understanding.
Uh and then finally self-s service BI is dead. So use AI like genie to give your user superpowers. uh move that right you know move that longtail off to the genie focus on that lefth hand side earn yourself some focus and hopefully this is some good good practices for you guys to take away uh how to use get the most out of your BI tools on data bricks uh with that I got five minutes left any questions all right thanks everybody I really appreciate your

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.