Summit Live: Data Sharing and Collaboration

Summary

Delta Sharing: An open-source approach to share data, AI models, dashboards, and notebooks across clouds and platforms without costly replication.
Clean Rooms: Databricks Clean Rooms provide safe hosting environments for data collaboration across companies without data duplication.
Marketplace: The Databricks Marketplace is an open marketplace for data, analytics, and AI resources enabling governed collaboration.

Summit Live: Data Sharing and Collaboration

Hear more on the latest in data collaboration, which is paramount to unlocking business success. Delta Sharing is an open-source approach to share and govern data, AI models, dashboards, and notebooks across clouds and platforms - without the costly need for replication. Databricks Clean Rooms provide safe hosting environments for data collaboration across companies, also without the costly duplication of data. And the Databricks Marketplace is the open marketplace for all your data, analytics, and AI needs.

Chapters

FAQs

What is Delta Sharing?

Delta Sharing is an open-source approach enabling organizations to share and govern data, AI models, dashboards, and notebooks across clouds without data replication.

How do Clean Rooms enable data collaboration?

Clean Rooms provide safe hosting environments for data collaboration between companies, enabling secure analysis without moving or duplicating data.

What resources are available in the Databricks Marketplace?

The Marketplace offers data, analytics, and AI resources that organizations can discover and securely access with governed permissions.

What cost benefits do these sharing approaches provide?

By eliminating data replication and duplication, these solutions reduce infrastructure costs while maintaining security and governance.

Full transcript

`0:00` Introduction to guests

Hello and welcome. And continuing our theme of absolutely fantastic guests, I am so honored to be able to introduce Zahir Valani, VP of engineering. Uh thank you so much for joining us today. I'm sure you're incredibly busy this week. Uh but welcome. >> Thank you so much for having me, and. Uh and, great job on stage yesterday. >> Thank you very much. Uh well, it was made possible by fantastic engineering teams who make amazing products that I can frankly just stand on stage and chat about.
So uh all the credit goes to the engineering teams. So you have quite a big area at the moment and I understand that you work on kind of delta sharing uh but I don't think that's it. So maybe do you want to talk us through what your teams do? >> I work on three main areas. My team's work on data sharing and collaboration which includes delta sharing, clean rooms and marketplace and then also work on AIBI some amazing demos of that today. >> No small feet. >> Uh and then also work on Unity catalog uh Unity catalog experience as part of it.

`1:01` Data sharing strategies

So >> goodness. So quite a large coverage area then. Um but I think we so we've heard quite a lot about AIB. I think that one's very easy to talk about because it's quite nice to see and to look at. But I think uh the sharing and the collaboration piece, it's difficult to build things that help people do things. It's like kind of a bit squishy and a you know difficult to measure and things like that. So what are the things that you're kind of solving for with data sharing and collaboration and you know how does that come about in terms of what working what to build and how to work with customers on it? >> Yeah.
Yeah. Great question and some of the data sharing collaboration is like so foundational. Uh so today for organizations it's uh not just about being able to analyze data within your own organization. You need to be able to collaborate with your partners with your customers and you need to be able to seamlessly and securely share that data. And so with the data bricks sharing and collaboration platform what we really provide is the foundation is delta sharing and then we have clean rooms and marketplace to really aid in those collaboration uh efforts. >> Yeah.
So I I I think it can't be understated enough that if you are spending kind of months trying to share data with someone like that's months wasted on a project that you didn't have to spend time on, it's really slowing down your ability to iterate on things. And I think once people start sharing data, uh it suddenly kind of unlocks a lot of things for them. I know like as my experience as a data engineer I like I know roughly what's in the warehouse that I'm using and like what I can do with that but I don't really know what else is out there and it's it's difficult to share and once that kind of starts happening then you can start getting more ambitious in the types of projects that you're doing. >> Yeah.
Yeah. And that's that's actually one of the main challenges that we're trying to solve with uh data sharing in the data bricks intelligence platform is today that data can be you know in another uh cloud or even in another platform or even on prem. So how do you even get access to that data and then often times you want to be able to collaborate with your partners on the data but that data has sensitive information. So how can you collaborate in a secure way?
And then of course what you were talking about also was like sometimes you don't even know like what data is out there. So how can I discover it? >> And once you've discovered it and like you want to get access to it you know traditional methods of getting access to that data like flaky ETL pipelines can take you know weeks sometimes to get access to that data. And so that's why with you know technologies like delta sharing we're enabling that to happen in a much more uh seamless way.
And I can talk a little bit about um delta sharing as well. >> Yeah I was going to say like this is like the backbone. It is pervasive if you know what you're looking for things start kind of popping up and you're like oh that works on delta sharing. So off the top of my head so it's definitely clean rooms. It's definitely marketplace. Um there's a few other things in Unity catalog that I think use delta sharing as well.

`3:57` Delta sharing explained

Is it system tables is built on delta sharing? Again, you'll notice it if you look in the background. >> System tables are great for like metadata of everything that's happening in the platform. Monitoring. >> Yeah, it is. But go on. I feel like I've not done it justice. Where else does it live in the platform? >> So, let me start with like what delta sharing is for maybe people who uh uh you know don't don't know so much about it.
So, delta sharing is like the world's first open secure protocol for sharing uh for sharing data. Zero copy. So you're not copying over the data and it works across clouds and across platforms. So that data can live in you know another uh data platform. It doesn't have to be just in data bricks and you can securely share that. There's also a rich ecosystem around delta sharing. So if you're using a BI tool like Tableau or PowerBI, it understands the delta sharing protocol and can get access to that data. or if you're doing it in you know pandas or apache spark it all understands delta sharing so there's this rich ecosystem around delta sharing and we actually g delta sharing like a few years ago here at this at the summit >> spectacular yeah >> um and the momentum the growth in the you know the number of uh customers that are are leveraging delta shar it's been a 300% year-over-year growth just you know in this year alone and that just kind of underscores the need like organizations are really trying to get, you know, AI ready.
You can't just, like I was saying earlier, you can't just use the data within your organization. You need data from your partners. You need from your clients and your customers. And being able to get that using delta sharing in this really seamless way um is really unlocking the value uh of that data for our customers. >> Okay. Then and so you talk about kind of collaboration now. Sometimes it's nice if you can kind of share share via Unity catalog and you can click through and you kind of send through the details and that's wonderful and you can re revoke those kind of tokens as and when the kind of project comes to an end.

`5:58` Clean rooms for collaboration

Um but if you want like the next level up because I know there's probably some people at home that are sat there sweating like I can't just like openly share my data that's not okay. This is where clean rooms comes into it isn't it? So do you want to talk a little bit more about that? >> Yeah I'll talk a little bit about data bricks clean room. So with data bricks clean rooms what collaborators can do like a company can analyze um and collaborate on this data with their partners without having to share that raw data that might have PII.
So you you know create a clean room within the data bricks environment. It works across clouds and platforms. You can pick what cloud you actually want that clean room to work on the collaborators and you can have multiple collaborators now. It's not just two. We have support for up to 10 collaborators can all collaborate in that clean room with that data, run analysis, and you can do it in any language. It's not just SQL.
You can do it in Python. You can run your ML workflows. Um, and all of this is done in a secure way without any of that PII data having to be shared uh across uh organizations. We have some great examples like um Intuit did a session here uh where they're using clean rooms to be able to get their lending partners to be able to analyze and run their ML workflows uh for risk analysis using data without having to share any of the um you know underlying uh sensitive information.
Yeah, I've seen that all different industries. So like retail, CPG where they're cooperative and a little competitive and there's a subset of data that they want to share that's beneficial for the whole collaborative but then they have proprietary you know competitive information. So all of this the all the different techniques make it very easy and more uh collaborative across comp companies and then within the company across departments um for sharing and then like the collaboration you can collaborate on the right things and it makes it so easy.
They ask customers like what they like about the sharing and marketplaces. It just is there. It makes it so easy and they have the trust in the data intelligence platform to handle all of you know all of the auditing and security. >> Yeah. Because in terms of the mechanics so I understand it's also like if I have some data and I share it with can write some scripts some SQL but then I can go and approve it to say like yes this is the kind of thing that you can run.
But there's also a lot of attention to detail that I did not realize until I started poking around with it. Things like you can like rename the table so that it's something that's a bit more kind of customerf facing. So you said kind of the CPG element where people are frenemies basically like they want to collaborate but not too much and they don't want to give away anything. So you can start kind of renaming things to make it look a bit more you know uh kind of desaturated for what it's going out there and you're not kind of accidentally sharing anything.
So yeah, I've been really impressed with that just kind of like the level of collaboration that sorry, collaboration is such a a broad term, but in terms of being able to have like two sets of data and then being able to write some SQL and then agree before you even run it and then having them in terms of the results that you can get from it as well, where do the results go? Where do they stay? Whose catalog does this belong to?
It's it goes into a lot of detail >> and all of those events like all that to your point like all of that is like logged you know in the system table so that you can actually then audit like what is happening um and then there's also work that we're doing around continuing to make those like approval workflows of like you know uh the notebook that's actually being run um and that notebook can have anything like not just SQL but like Python but making sure that uh you know the collaborators are all aware of like what is actually happening um in that clean room for that trust >> and this is what I mean by attention to detail in terms of the like the logging and the auditing of things.
So again if you're in a heavily regulated industry where you have to be very careful about how much you collaborate because there are laws saying you know you can't collaborate too much cuz that's market manipulation um but you know you're trying to do I don't know fraud prevention or something and you can then demonstrate to the regulators like yes we did collaborate but not too much so don't worry about that. Um, and yeah, so it is is a very broad ecosystem at the moment.
So I'm talking about financial things. You mentioned to kind of intuit it, but are there any are there any standout to you as kind of like a very novel use of uh clean rooms from a customer? >> Yeah, so we see it across like a variety of industries like in the keynote uh yesterday, Mastercard talked about, you know, they're leveraging uh clean rooms. We've also seen it in healthcare like where healthcare organizations will like uh share it with their research partners.

`10:24` Real-world use cases

Um there's one customer that we're talking to they're using it for clinical trials being able to leverage clean rooms uh to collaborate with their partners for clinical trials. So it's across uh to what you were saying uh earlier. It's across a variety uh variety of industries that we're seeing um customers really excited about. Healthcare is one I'm excited since you're prolonging uh everyone's lives, saving lives, and that's where collectively you could come up with, you know, more targeted cures but without giving away the specific individual.
So that's collectively one of my favorite examples of sharing and collaboration on on data. >> Yeah. Yeah. So I imagine for this week you've been spending a lot of time speaking to customers. You will have hopefully received lots of nice comments, but also comments about what they would like to have next. So can you tell us anything about like what's on the road map, where you'd like it to go? >> Yeah, for sure. Yeah, we've heard uh a lot of like excitement from customers.

`11:43` Roadmap and future investment

Um and then also, you know, what we can do uh continue to do better in the in the platform, which is always a great sign as well because what we're hearing from customers is already planned in our road map. So that's always gives you uh you know kind of confidence that we're we're focusing on the right things. So there's three big areas of investment for us. Um, one is interoperability. So, mate talked a little bit about that this morning.
Uh, you know, support for iceberg. We're open platform. So, if you create a table in Unity catalog, an iceberg uh table in Unity catalog, you can now share that using delta sharing. Um, if you have a table uh in Unity catalog and you delta share that, Snowflake or any other iceberg compatible client can actually read that delta uh that delta share. So continuing to make sure that we uh remain this open and accessible platform.
The second area of focus for us is around governance. Um you know with Aback with attribute-based access control you can delta share those uh tables as well and making sure that you know we're continuing to I mean that's foundational for uh data bricks intelligent platform governance. So that's another big area of investment for us. And then the third area for investment for us is around scale and simplicity. Um so with delta sharing for example one of the things that we're having we're introducing is this uh storage gateway this uh network gateway to really simplify like when a provider has to share with multiple um recipients or multiple collaborators like you know configuring firewalls like all of that adds complexity and so we're introducing this network gateway to really simplify um that access and scale so you can actually scale it you're not repeatedly doing it uh and so that is really resonating ating with customers um you know around scale the other thing that we're working on is for clean rooms being able to uh like I mentioned have multiple collaborators you're not limited just to two collaborators so these are areas that we're continuing to improve uh and enhance the platform the last thing I will say is you know the data sharing and collaboration is just as rich as our ecosystem and so we're continuing to add new partners new data partners um you know for clean rooms uh identity resolution partners like Axiom like so we're continuing to add these uh you know develop these partnerships and they're all really excited to be part of um you know whether it's the marketplace or leveraging our clean rooms and so that's also a really important part of building out this ecosystem. >> Well, great.
Well, we have less than a minute left. Just want to hear what what are you excited about? Um have you been to any great presentations? Are you giving any? Yeah, there's been so many uh so many presentations about you know AIBI but delta sharing clean rooms there's just so much excitement so it's hard for me to pick um one area but on the theme of you know data sharing and collaboration I think the part that I've been really excited about is uh just some of our partnerships >> um you know a really big one is the SAP and data bricks uh partnership so being able to now get this data that's been in SAP P uh all this enterprise data that's in SAP getting that into your lakehouse and that's all powered by Delta sharing.

`14:19` Closing and partnerships

So the SAP business data cloud um and then the other one there was a session this week that Walmart did on their Cintilla uh platform where they're leveraging uh Delta Sharing to be able to um you know with their retail partners to be able to improve the efficiency in the retail ecosystem. So, these are all like mind-blowing uh you know uh uses of this platform that we're building. >> Great. Well, uh thank you so much.
We're gonna thanks for coming on. We're going to toss to a video and then have mate come on. Thank you. >> Thank you so much. >> Thank you so much. >> We are using data bricks for um democratizing our clinical R&D data and it has helped our team tremendously in accessing data from different sources for their analytics use cases. Immediately it's changed the way that we deliver data. How we deliver results and answers in a much more streamlined way. >> Data bricks has really revolutionized us getting our customers into a fully cloud-based architecture.

`15:34` Customer perspectives

It's really made a difference uh for us to get them into the cloud. >> So I'm a current undergraduate student uh at the University of Notre Dame. I don't have a lot of the expertise that I have on my hands right now or in my brain that uh that can help me achieve the results that I wanted. And so I've been using data bricks to uh help me achieve all all the goals and achievements that me and my team wanted to achieve. >> I'm relatively new to the data brick space.
Currently data science is they have to do a lot more manual work than would be ideal. I'm really excited as we kind of mature and expand our our data bricks platform. It's going to cut down that time significantly.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.