Sponsored by: Microsoft | Leverage the Power of the Microsoft Ecosystem with Azure Databricks

Summary

Microsoft-Native Solution: Azure Databricks is a fully integrated, Microsoft-backed solution rather than just Databricks running on Azure.
Ecosystem Integration: Deep integrations with Power BI, Azure OpenAI, Microsoft Purview, and ADLS enable seamless workflows across Microsoft solutions.
Co-Built Partnership: Continuous collaboration between Databricks and Microsoft ensures optimized performance and aligned product development.

FAQs

What is the difference between Databricks on Azure and Azure Databricks?

Azure Databricks is a fully integrated, native, first-party solution from Microsoft, not just Databricks running on Azure. It represents a co-built partnership between the companies.

What are the key Microsoft integrations with Azure Databricks?

Deep integrations include Power BI for business intelligence, Azure OpenAI for AI capabilities, Microsoft Purview for governance, Azure Data Lake Storage (ADLS), and Foundry.

What is the benefit of the 8-year partnership between Databricks and Microsoft?

The long partnership ensures continuous collaboration on product development, tight integration, optimized performance, and a streamlined support experience.

How does Azure Databricks handle diverse data?

Azure Databricks provides Microsoft-backed promise with the Databricks Data and AI platform's enrichment capabilities, handling structured, unstructured, and any data type.

Full transcript

Um well hi everyone. In my day job I lead the Azure data bricks product group. Uh apart from that I also hike a bunch in Seattle where can we talk about how I was promised sunny weather in California and you all are liars. Okay. This is Seattle weather. What is this June gloom thing you have going on over here? So, I'm actually kind of concerned uh about this as a whole, but I will move past that. Super excited apart from the lack of sunshine to be here to touch on some recent announcements, maybe cover not the basics, maybe the intermediates considering this is an intermediate session. uh chat through what we have upcoming here in the future and maybe early announce some stuff that's supposed to go live tomorrow.
But if you talk to me outside this room about that, I will deny that I said any of that. With that, I love to always ground people on the basics, right? You've heard about datab bricks on cloud provider X, datab bricks on cloud provider Y, but you've probably never seen something like Azure datab bricks, which is really a Microsoftbacked product offering. But what does that really mean, right? Why does it why is it better?
Why should you care? And why do we even have this ongoing partnership of eight years and so many more to come? I'm a math person. I know, I know not everyone likes math. Stay with me over here. In my opinion, you will like this math. It's really the best kind of math. We take the benefits of battle tested scale from Microsoft coupled with the amazing intelligence that we call the data intelligence platform at data bicks.
Combine it together to offer our customers Azure data bricks. This is not just data bricks running on Azure. This is a Microsoft back solution for you. That to me is super cool. Imagine two different companies from two different areas coming together to align on this common mission to serve you, our customers, with the best of the best. Right? We're aligned on this mission. No matter where your data is, what your data looks like, whether it's structured, unstructured, we want to give you that Microsoft back promise with the amazing enrichment that data bricks offers.
And so really, that's data bricks. Now, you're probably like, hey, that equation on an sounds awesome, but what does it actually mean, right? What it means is that we co-build. My friends at data bicks, my colleagues, they're just like any other team. Yeah, they have at Microsoft or at data bricks emails almost at Microsoft emails if I'm being honest, but really you co-build, co-inovate, you co-learn from customers to bring the best of the best over to you.
Now, that comes with what I like to call the Microsoft assurance. What that means is you're familiar with Azure support. It's a one-stop shop. You don't have to go to different vendors to figure it out. We'll figure that out. you can focus on getting that analytics inside the rich insights that you need to whether it's optimize costs, improve revenue, um hit those budget targets that are super aggressive as a whole. Those are all of the kind of things you're able to do.
Same thing Microsoft. We invest over a billion dollar dollars annually in cyber security and privacy and we have almost 9,000 people working on this and you're able to leverage that with the power of Azure data bricks. That is so freaking cool when you think about budgeting, right? You don't have to learn a new system for billing and managing like you do in maybe other cloud providers. With Azure data bricks, it's that same unified support that I was talking about and the billing that comes through.
So think about it. Let's say worst comes to worse, there's some outage or some issue going on. You don't have to say, "Hey, should I go to Microsoft? Should I go to data bricks?" You don't worry about it. We will figure it out. You focus on what's matters, which is getting those keen insights and analytics from your data. And that's really a trifecta, right? you have the scale, you have the a agility, um, and you have the platform as a whole.
Triple threat, big fan, and so you're able to get the most out of your data no matter where it is. Now, I'm not the only one saying this. All of our customers, brands that we all are familiar with, know, love, use, have opinions about are saying this. So when you take those data that data intelligence that I was talking about and you apply it to data warehousing capabilities, we're able to really do worldclass price performance without requiring data movement.
Right? So wherever your data may be and this is what we're hearing from most of our customers. The data estates and the data is fragmented. It's everywhere in so many different places. So instead of spending time getting that data together, you spend time bringing those insights for your organizations for your specific use case. And this is really one of the fastest growing products in history that we've seen. Also, I will say I only had few companies on there because the marketing folks were like the slides had to look pretty.
But in actuality, it's thousands and thousands of customers. Okay. Do we know Microsoft Entry? I see no. Awesome. We love Microsoft Entry, right? It's really for those who may not know, which where have you been if you don't know, but it's a cloud-based identity and access. Thank you. Thank you for laughing at my jokes. Finally, some energy here. My jokes are kind of lame though, so I'll give you that. Um, but really, you can access resources that are external to your company.
So things internal to me, things like Microsoft 365, things like the Azure portal, you have that single sign on experience. You can also use Android ID for internal tools within your company. So things like apps that you run on your internet, unique to you. And that's super cool because it's so widely used, best-in-class as a whole. We know entry, we love entry. Now, previously if you wanted to do access management and identity is really actually let me step back here.
Identity is really the key, right? You want to start with the who can access the data before you do anything with this data. And so that's why I'm focusing on identity a little bit, right? Who can I give access to? And we'll talk about governance as well. But the who is the identity piece here now before some of the in improvements and the in integrations that we did, this was all manual. you had to manually use like a skim sync to say, "Hey, I need to give this person access to this cool dashboard that I made."
Then you add them manually and then you sync that. And just imagine managing that at scale every time a person joins a company, leaves the company, someone's out of uh someone's outside of your company as a whole. Hard to manage, hard to scale, so many extra steps and that's so cumbersome as a whole. But because we have this unique firstparty integration that I was talking to you about Azure datab bricks, what we are able to do is now have this automatic identity management.
This means you already have your groups like I have this little video that's running here, but you already have your groups in enter ID. You can without leaving your Azure data bricks UI and context and portal automatically just use the same groups. There's a new user. You don't have to manually sync. you're really good to go. This is so freaking cool because now you're not sitting there and managing access. Instead, you can take that time that you would otherwise spend on that and focus on the insights that you're gathering from your data.
And that's really this common theme that I will continue to beat on this entire session as a whole. Right? You can protect access to resources and data using strong authentication risk based conditional access policies that you choose. And it doesn't have to be in the single environment as a whole, right? It's a fast, easy, single sign on experience across your different cloud environments as a whole. You can manage identities across all of your applications in a central location.
So unifying that in one single place. So let's say you have a dashboard, a very cool dashboard that you have created in Azure data bricks and you want to share with someone they probably have enter ID already. So you don't have to do much. You're like hey customer you know maybe you had some outage or or you had some celebration event. You're like hey here's a cool dashboard that I built. I'd love to share this data with you.
How would you do that without this? Right? With this you don't even have to think about it. It's just built in for you. again taking time away from who access management and putting it into data as a whole. So unified single click easy to use no extra thought for you. We've taken care of that. Oops. We've taken that care of that on our part. Okay. So now we have the who part of that identity equation. Let's talk a little bit more about protecting and understanding your data.
Because data is bifurcated in so many different warehouses or in so many different locations as a whole. The first thing you want to do is understand what the data is even where it's located and do you even need it. Do you even need to bring it into Azure data bricks as a whole and the key is you want to define things once and then secure them everywhere. You don't want to keep doing it duplicating effort again reducing time to manage govern protect your data focusing on those insights and then eventually building AI agents that we'll talk about as well.
So unity catalog which I think you all should be familiar with really provides a comprehensive governance framework right this goes beyond just access control it to include only data it's about models and code governance so everything enhancing secure uh security discoverability all in one place for you as a whole and that defining once but securing everywhere is key because then as administrators you can only do it once and you're good to go as a Of course, that comes in with the American National Standards of SQL um backed compliance models as a whole.
Again, you don't have to worry about those kinds of things because we'll take care of it for you. We have builtin auditing and lineage. So, you want to understand, okay, who's accessing your data, what did they do, what changes they made, how it's connected to each other. It's it's really pretty cool. But then you also have fine grain access. So maybe um you're not HR so you can't access some salary information or you're not from a different organization part of the organization you can't access specific information because it's tented.
You can have that fine grain level of access control with Unity catalog too. And that's where that governance piece comes through. And then finally I was alluding to this earlier. It's also data discovery. Right? Sometimes we have so much data we don't know where it is. And so that will also help you. But the cool thing is you don't have to get in the data if you don't need it. Right? So wherever your data is, you can leverage the metadata within Unity catalog and say, "Hey, actually, yeah, I do need to run this query.
I may need this data in the future." And only then you can import it or bring it into data bricks. And that's super cool, right? Again, that common mission. No matter where your data is, we want to democratize access to it, enable you to build those insights, reduce time spent on governance, access, management as a whole. We take on that overhead for you. Okay, so we know the who. Hopefully by now we know the what the data is, how we govern access to it.
Now let's talk about actually getting that data in. So let's say you've identified what data is actually needed for you to develop these rich business insights that you need. What what can we do with that and how can we build analytics insights and AI on top of it? So think about a lakehouse and no this is not your friend's lakehouse just to be clear not an actual lakehouse on the lake. We took data lake and data warehousing and calling data data probably is not the best part.
So we took the other two worlds thus lakehouse right. So lakehouse really combines the benefits of data lake and data warehousing as a whole to provide one unified platform for business intelligence analytics and ML. If you're on a team you know how cumbersome hard to manage this may be because you have to adjust data from multiple different formats schemas sources as a whole. Just managing that is actually used to be someone's full-time job.
Right? Again, if you're an organization, you're spending teams and cycles to do these kinds of things. How do you have the time to understand and build insights based on that data? You really don't. It's a full-time job coupled with needing to run CI/CD platforms, updating APIs, managing that. That's actually teams do that as a whole. But with uh Lakehouse and LakeFlow as part of that, no longer have to do that, right?
You have a reliable automated data flow. So we take care of that for you as a whole. That to me is just so cool. You no longer spend time on doing the mundane, the annoying. You don't have to get on call say hey fix this pipeline. Why is there latency issues here? I can't get this data that I need for this presentation for this exact review. Whatever it may be. And as part of LakeFlow, one of the main components that we have is also LakeFlow connect. leak flow connect you can think of it as just point and click it's like hey my data is here I need this based on you know based on the analysis or based on the metadata that I got to know and discovered from unity catalog I actually need to input this data I actually need to ingest this data so I just point and click whether your data is in any database uh maybe it's in SQL server maybe it's unstructured data such as PDFs or excel spreadsheets and shareepoint really whatever it is right it supplements our popular native connectors of ADLS Gen 2, which I'll talk about a little bit more.
Whether it's cues for event hub, you can ingest data from databases, enterprise apps, cloud sources, transform it, and then do those analysis and analytics that you really need and you can confidently deploy them in production and operate them in production with that high performance and scale of course. Okay, so now we have the identity, security, governance. We know how to get the data in. We don't have we don't need someone's full-time's jobs to be just getting the data in.
Now, where does the data rely? We have this common mission across both our companies. We want to empower you no matter where your data relies, right? Of course, that can be Microsoft internal sources. So you can connect to a wide variety of things such as maybe Azure blob storage or SQL databases that I talked about earlier, event hubs, synapse analytics, um, and many more, right? One of my favorite ones is Azure data lake storage.
You probably have heard of Azure data lake storage, but I'm not going to keep saying that. So I'll say ADLS for short. It's really a cloud-based enterprise data lake solution and it's engineered specifically for you to store massive amounts of data in any format to facilitate big data analytical workloads. You can use it to capture data of any type and ingest that data in a single location for easy access as a whole.
You may have heard about ADLS Gen One, right? So think of ADLS Gen 2 or data lakeink storage as taking the benefits of gen one and blob storage all together and then coming with that with the power of Azure storage in one place whether that's managing files um system semantics you also need file level security um all of that at scale that's that's effectively what ADLS is right and we'll talk about gen two but I won't say that as whole but some insider info or something that I really find super We've been working together, data bricks and Microsoft on this journey for over eight years.
So this kind of scale where we started and where we are has just been an incredible journey as a whole. But we were able to co-inovate that together, right? And you as our customers get to benefit from that, which to me again is super cool, right? The world's biggest data estates are really running using ADLS on Azure data bricks as a whole. Without a doubt, I can say this is the most scalable storage for data bricks.
The world's most scalable AI workloads are running on this. And I know you know which ones I'm talking about. Overall, because of that first party integration, because of that first party integration, you can directly read and write data um via Azure data bricks from ADLS Gen 2, right? So again, fast optimized native integration. You still don't have to sit there and connect, waste time managing, connecting all of that stuff.
We take care of it for you with this native integration as a whole. Okay, so that was some internal sources with Microsoft internal sources, I guess. But there's also external sources. I mean, shocking to me that you have data outside of Microsoft. I'm just kidding. You can get data in from external sources as well right the goal that same common mission right variety of data sources no matter where it is maybe it's in hive meta stores maybe you're using terod data maybe it's Salesforce you want the goal again is to unify all of that data give you access visibility view governance to be able to take that data and do something with it what is the point of having all of this data across so many different places where you don't even understand what you can do with it, right?
That's where the power comes in and that's what we want to empower you to do with Azure data bricks. So, we want to support your teams, your organization, regardless of the data format. All of the data formats listed there on the slide have some nuances. Some are different than the other, whether that's delta iceberg. Um, but you don't have to worry about those schema differences as a whole, right? We'll take care of that for you and your data is thus easily discoverable.
Now at Microsoft, our philosophy is simple. We want to empower you with the choice of choice to get your data and get those insights wherever they meet. This is not this is not a wrong slide. This is accurate. So even if your data is in AWS's S3, you're able to access that data through what we call crosscloud governance with Azure data bricks. Think of how powerful that is, right? Zero copy. You don't have to move the data.
You can query that data sitting in Azure data bricks from that same UI but your data will be sitting in AWS's S3 powerful powerful stuff because goes back to that central mission and I feel like this is a good proof point right we're putting action where our words are our money where our mouth is as a whole no matter where your data is we just announced this actually last month and I actually just announced this last month at build uh which is super cool and this is a GA release uh it supports read only features like S3 external tables, volumes, um AWS IM based credentials within Azure as well and there's more to come in this space.
Now you may also have been using SAP as a whole, right? So SAP has the business data cloud or BDC for short as we call it and there's a version of SAP datab bricks uh that's available as part of SAPDC. So shortly here in the fall, early fall hopefully, we will be launching SAP data bricks and bringing that into Azure as a whole within the confines of your organization, you're probably using different SAP data products as a whole.
And so now with the SAP data bricks solution as a whole running on Azure, bringing those advanced AI and capabilities in your environment that's fully integrated. Same thing, you don't have to leave wherever your data is within that confine. you can use within your organization. You're really good to go. You can combine SAP data with your external sources, perform analysis, publish custom data to whatever data products that you're using within SAP Business Cloud.
We're super excited about this and we're working across SAP, Microsoft, and Data Bricks to power this and really bring it to you. Okay, one of my favorite things is really power platform. I'm sure you all have heard about Power Platform. There's many different more angles to Power Platform than I have on this slide. Again, I'm trying to keep these slides pretty. So, I just mentioned PowerBI, Power Apps, Automate, and then virtual agents.
But Power Platform, in case you're not familiar, is a suite of Microsoft tools that empowers users like you to build apps, automate workflows, analyze data, and create websites with low to little code, low to no code almost, actually. And it really helps businesses work together and transform organizations to get those insights from their business. And it gives me immense pleasure to pre-announce. You'll see the official announcements go out tomorrow.
So if someone tells you that I announced it today, please decline and never said anything, but I'm super excited to share our native Power Platform connector. This is new. You can actually already use it even though our official announcement will go out tomorrow. You no longer need to build a custom connector or all of your scripts to connect to PowerBI from Azure Data Bricks, right? And so this new connector is truly a gamecher, not my words.
We'll talk about the quote in a little bit for organizations looking to seamlessly integrate Azure data bricks with Microsoft's low code ecosystem of power platform. The connector bridges the gap between advanced analytics and business processes automation enabling secure native governed all of those things that I was telling you about but real time data access right without the headaches of managing those connectors building those scripts this was actually frustratingly painful to do before this and I wonder how why it took us this long to do this but it's here now and I'm excited because people are already using it right so the UK Power Networks is actually one of the largest uh electricity provider uh in the UK as a whole.
They're already using it. They already love it. Um it helped them get the performance they needed with the agility of low code solutions from the power platform as a whole. And so they talk about how it helped them maintain the momentum across their digitalization strategy enabling that seamless keyword seamless instead of having to redo build custom scripts to get the data that they need to have those dashboards that they need with Power Apps.
Super powerful stuff. Including that we also are announcing enhanced knowledge source. So Asha data bricks within your context understanding your business's context right that's important because outside the context things mean nothing you want to have that context and that's really key so then that context and that data can now be used as an enhanced knowledge source in co-pilot studios so that means that agents can understand reason over data with the context that I was telling you about and the benefits of governance and security as a This integration of course supports OOTH and service principles and all of the authentication and the lovely stuff that I was talking about with Microsoft Enter ID as well.
Again, making sure your data is secure, seamless integration and well-governed as a whole. The real- time knowledge and this you could use this today like you could literally leave the session but don't leave the session um and use this today. Ready to go in in all regions as a whole. super excited about this and there's so much more to come here. Another integration that this has been a while out for a while now so not as new as maybe the first two that I was talking about but really bringing the power of orchestration with lakeflow to powerbi that's really what the powerbi task in in data bricks lakeflow does so you can publish data sets directly from unity catalog to powerbi without any importing or without having to leave the context that you're in so you can push changes to underlying tables their relationships all automatically ly with PowerBI tasks.
You can even automate PowerBI semantic model updates refresh directly from data bricks workflows as a whole. This is cool because you don't have to leave the context that you are in as a whole. Right? So you're in the context in that same UI. You don't have to switch out do what you need to have the seamless integration. We take care of that for you. Really bringing that power of Microsoft data bricks and the ecosystem overall.
Another integration point that I wanted to talk about is data factory, right? So data factory is really a managed cloud service. Um it's built to it's built really for ETL or ELT u and data integration projects as a whole. Similarly that same theme right my common themes across this presentation um we want you to be able to natively orchestrate data workflows using Azure data factory or fabric data factory. This tight integration includes now as of last month actually uh a job activity in Azure data factory.
So you have a data bricks job activity within ADF and FDF fabric data factory as well and making it even easier for you to trigger workflows directly within ADF. So the native integration is key here right we could not have done this if we didn't have Azure data bricks as a whole. And I know I keep underlining that but I hope hitting that point home as a whole. So this native integration uh can execute notebooks, Python codes, any activities which enables organizations like yours um to build scalable data orchestration pipelines, ingest data from various sources and curate that data within that lakehouse architecture as a whole.
Okay, hopefully you're still with me. We've identified the data, we've governed it. Hey um we know where it is. We've ingested the data that we need not all the data. We even went across clouds and took what we needed within based on that. Now we have this rich analysis. We have the dashboard. We have the power apps. We call all of this cool stuff. But it's 2025. Why have you still not talked about AI in this presentation?
All up. Okay. I hear you. I hear you. Let's get to it. So let's talk about now I have all of that data how can I do more with that data using AI. So Azure AI foundry is really a unified platform for enterprise AI operations, model builders, um application development as a whole, right? So let's see how we can now use integration points with Microsoft data bricks to enable these AI workloads or AI agents for you. At build last month, we also announced this connector.
So as enterprises explore to use AI agents for the business for their businesses, one of the biggest thing we see is you want production ready agents and for that to happen you need enterprise ready data with the context of your business. Without that your agents are not going to be as effective. Uh the quality of the agents is going to dramatically be low in comparison as a whole. So this is what we announced last month.
As I said at build uh you can build an agent using Azure AI foundry and bring in data intelligence from data bricks as a whole to power that agent as a whole which is super cool, right? It's a connector. You don't have to do the connecting. We do the hard work. We take the we do the headache. We manage the code. We patch everything that's needed as a whole. You just do one click. You're good to go. you have your agent built based on that amazing data that you've now gathered within Azure data bricks as a whole.
Powerful stuff. Going back to that analysis or underscoring the choice angle, right? Let's say, hey, I don't want to use Azure AI foundry, which is totally fine. A lot of our customers use Azure OpenAI APIs directly, right? Azure OpenAI gives you um the advanced language AI with many models and the security and compliance backed with that responsible AI uh framework which in this modern day and age is really key. So that's the difference really between Azure OpenAI versus just OpenAI as a whole.
Same thing. So here you can have all of those different models uh with the private networking regional availability and the responsible AI framework as a whole backed with the governance all of that fine data intelligence that you've gathered within Azure data bricks as a whole. Okay, finally this one is my favorite one as a whole. The whole point of all of this data gathering, building these insights is going to be useless if everyone can't use it.
You want to democratize it. Anyone should be able to use it. You don't have to be a Microsoft expert. You don't have to be an Azure expert. You don't have to be a data bricks expert. You don't have to be an Azure data bricks expert as a whole, right? Anyone anywhere should be able to say, "Hey, what's the revenue last quarter?" Right? I don't want to sit there and have to become an expert overnight in Azure Data Bricks.
Although I do recommend becoming an expert in Azure data bricks by the way. Uh democratization of data to me is the coolest thing as a whole. So we all know and hopefully love teams. Imagine just being able to ask this bot or an agent just like you do your coworker, hey what was the revenue last quarter? Or hey what is the cost efficiency that we achieved the previous quarter? Or hey who in the sales organization is hitting our targets here?
How can I show them some love? And that's not going to be based on some stale information that someone pulled at some point in time. It's real time right now. You always have access to that data and that's so freaking cool. Um, so you have this Teamsbot framework that you can connect to your Genie space. So Genie, if you don't know about Genie, but Genie is basically like having a super smart co-orker that has access to your data with the context, conversational AI as a fool.
You can ask questions in your natural language, get instantly accurate answers that I was talking about. But now you don't even have to go into Azure database. You can just open Teams and ask questions as a whole. We're going to do so much more in this space. The AI foundry connector that I was talking about, we're also going to make sure that's available as an agent connected through Teams as well. So that's something that's also coming up as a whole and truly truly democratizing that data access as a whole, right?
What is the point of all of this data in all these different locations that you've identified that you're secured if everybody can't use it? I want everyone to use it. We want you all to use it. And that's really coming home with that power over here. This is just an eye chart. I talked about these different integration points of the Microsoft ecosystem. Of course, not everything is going to be used all at once. It's dependent on your use case.
But I feel like this is a helpful chart to say, hey, these are the ways that I can leverage truly that power of the Microsoft ecosystem with Azure data bricks, which in my opinion is the best cloud to run data bricks on if I say so myself. With that, I was told that maybe that I'll have time for questions as a whole. So please feel free to come to the mics if there's questions and then I'll have this data bricks resources slide up here.
You can scan the QR code if you do want to become that overnight expert uh on Azure data bricks. you're welcome to do so. Thank you so much for your time and listening to me. Please come back next time and make sure there's a sun in this sunny state of California. Thank you very much and happy to take questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.