Enabling Sleep Science Research With Databricks and Delta Sharing

Summary

• Cross-Cloud Data Sharing: Sleep Number uses Delta Sharing and Unity Catalog to safely share anonymized datasets across multiple Databricks workspaces and cloud providers.
• Privacy & Compliance: The implementation leverages Databricks Data and AI platform security features to ensure data sharing complies with privacy standards across AWS, Azure and Google Cloud.
• Accelerated Research: By enabling secure data sharing across organizational boundaries, Sleep Number accelerates sleep science insights while maintaining strict data governance controls.

Enabling Sleep Science Research With Databricks and Delta Sharing

Talk by Alexandr Rivlin, Director, Data Platform, Sleep Number Labs ; Sajeev Mayandi, Sr Director Architecture & Data Platform, Sleep Number
Leveraging Databricks as a platform, we facilitate the sharing of anonymized datasets across various Databricks workspaces and accounts, spanning multiple cloud environments such as AWS, Azure, and Google Cloud. This capability, powered by Delta Sharing, extends both within and outside Sleep Number, enabling accelerated insights while ensuring compliance with data security and privacy standards. In this video, we will showcase our architecture and implementation strategy for data sharing, highlighting the use of Databricks’ Unity Catalog and Delta Sharing, along with integration with platforms like Jira, Jenkins, and Terraform to streamline project management and system orchestration.
Here’s more to explore:
Open data sharing for data, analytics, and AI: https://www.databricks.com/product/delta-sharing
A new approach to data sharing: https://www.databricks.com/resources/ebook/a-new-approach-to-data-sharing
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Delta Sharing?

Delta Sharing is a secure data sharing protocol that allows organizations to share data across Databricks workspaces, accounts and cloud providers.

How does Sleep Number ensure privacy?

Sleep Number uses Unity Catalog and Databricks security features to anonymize datasets and control access while sharing data across environments.

What are the technical components?

The architecture integrates Databricks, Unity Catalog, Delta Sharing with infrastructure tools like Jira, Jenkins and Terraform for orchestration.

How does this benefit sleep science research?

By sharing anonymized data securely across workspaces and clouds, Sleep Number accelerates insights from billions of hours of sleep data.

Full transcript

Okay, good afternoon everybody. Uh, I'm a senior director at Sleepnumber taking care of architecture and data platform. And I am a director of data platform at Sleep Number. So today we'll go over our journey of implementing unity catalog and delta sharing in a use case that we used in sleep number that is data sharing. So we shared the data and then as a part of implementation we use unity catalog and delta sharing.
So we'll go over that. So a little bit of agenda. Uh so we'll go over on sleep number then uh some of the data architecture related to sleep number ecosystem that is IOD part and we go over the data sharing use case some solution architecture related to that and then will then run through the implementation migrations and lessons learned as a part of that. So uh when you hear sleep number most of us might be thinking hey this is a bedding company and what are they doing out here in data bricks conference uh well yes we do sell bed but we do sell smart bed and we are wellness technology company uh our bed is equipped with a very high-tech technology uh it helps in controlling the temperature of the bed it controls the firmness of the bed uh it also helps in articulating the bed up and down.
So that is a bed operations but we do also track the sleep information of uh 2. 5 and up million customers and the data for the sleep information come into our data platform. So we have billions of hours of sleep data in our data platform that tells the story why we are here in databicks conference. So little bit of our architecture. Um so to give you a context here, our architecture is a little bit more complex than what you see out here, but I'm tailor it to this particular session.
So uh let's go over first on to this bed. I mean that's that's the our bread and butter. But if you see this is our smart bed and we have a pump that gets clamped beneath it and in that pump we have an NX speed chip which is having a compressed version of a Linux operating system. That's what you could consider it and it is connected to various sensors like pressure sensor, temperature sensors and then articulation. And for articulation we have uh bunch of other sensors but the point here is these sensors are used to control the temperature of the bed.
That is a temperature sensor is to control the temperature of the bed and then pressure sensor is to control the firmness of the bed. Uh then actuators are used for articulation of the bed that it can go up and down. And then if you see the app here is connected to the bed through a Bluetooth and API. So we use app pretty much like a remote. We do also have a remote but we use the app as a remote wherein we use that to do a bed operation.
That means you want to articulate a bed, you want to articulate your partner's bed, control the temperature of each of those bed that you see uh and then control the firmness. So that is about the bed operation. But the data part comes from the sleep information. That means our customers sleep on this bed. So when they sleep on the bed uh the data from the pressure sensors come in it is in an analog form. We converted into digital form and then what we do is we have AI algorithms which are running on an edge and those algorithms basically give biometric information about the customer.
That means we calculate respiratory rate, heart rate and whether you're moving left, right based on actuators and others. So just imagine a person sleeping here 8 hours a day and every night this set of data for every ncond goes into our data platform. So that's where you see this uh streaming line out there uh and eventually it goes into the data platform. So that's where we have billions of hours of sleep data for our 2.
5 plus customers. Um now we do have a lot of other applications which are built on the data platform. Uh but in specific to this session I've just zeroed in on a use case called data sharing. That means we collect the sleep data and we share that data for research purpose. So you see here uh a line that goes from a data platform onto the research partners. So the idea here is to export the data set that we have in terms of sleep to research partners who are interested in doing sleep research.
And then another uh use case specific to data sharing is we basically do a crosscloud communication. We put those data into an Azure platform and then we have built an analytics on it. That's more of a use case. But the idea here in this particular session is that for this particular use case, how did we use Unity catalog? How did we use uh delta sharing? And that's where uh we go into the weeds and that's where will cut in.
But for now, let's go a little bit on an architecture on the data sharing and and then then the rest. So uh again the use case we are talking about the data sharing and there are two categories there uh one is a data download and then one is uh data sharing platform as a service. So data download is in a case where researchers say that hey I want the data set in my vicinity that means in my uh compute environment. So for them we basically collect a data set, send a URL, they go get the data set, do their work.
Uh that's one kind of a researchers. Then there is another kind of researchers wherein they log in into our system. When I say they log into our system, that means the data brick system and then we prepare data sets for them and then they do research for a contract period that they have finish it up and then move on. So these are the two categories and of obviously we have an internal use case where we I already mentioned how we communicate that.
Um so a little bit on on the data download solution architecture. So we use Jira for traceability and onboarding and then uh as as a customer comes in hey they want a data download with say X data set with X age and stuff like that. So we prepare that we provide that through and then for Jira we use traceability and then we use Jenkins as an orchestrator. So if you see Jenkins the first job it does is prepare the data sets that we have because we can't give PII and other information.
So it needs to be anonymized. So all those is done as a part of a data preparation in the data bricks and then we put that into an S3 bucket and clamp an uh cloudfront URL in front of them and then send them an email and with an expiration and others and they click it and voila the data is there in the data set. Simple isn't it? It's so this is one-time use case and researchers uh I would say substantial amount of researchers out here in US want this because the data set needs to be in their environment in order for their research to be approved because researchers get uh an approval based on certain criteria security criteria.
So for them this works perfectly fine. So now conversely let's take a platform data share sharing platform as a service. Uh in this case we have to actually provision a cluster. We have to provision a data set and then obviously we'll you still use Jira and communications and orchestration as a part of it. Uh but this is based on an contract that means some researchers or some companies comes in and say that hey I want to research for 6 months or one year.
So that is a contract period. We provide the data sets they're interested in in in this platform that is built in our our our place and then they log in they do research use the data bricks tool as a part of it and then they move on but at the same time when they move on we have to deprovision it. So platform as a service consist of provisioning as well as deprovisioning. So little bit on the solution part of it is uh again we still use Jira here we use Jenkins as an orchestrator but here Jenkins the first step it does is it's provision the cluster uh and that's where we use data uh terraforms right and then we have to prepare data set that is used by that researchers uh researcher or researchers right and that's why we use unity catalog we have them register into our system.
We send them instructions to basically log in and voila. Then now they are into our system in this form. So if you see on the left side is our data platform on the right side is a data sharing platform and each partners gets its own datab bricks workspace. Right? Each workspace have an conditional access. If you have multiple users, each user will have its own access. base permissions for the data set that they are allowed to see and do analytics.
They could use spark, they could use SQL endpoint and the most important thing is they are separated right. So each partner will get its it's anonymized data. It's the what they are interested in in specifically is uh is is the sleep information about a category of users and that's what they are going to use uh once they get this. Now one thing important to note here is the people the researchers typically using this data set as a data analysis tool.
They build models like algorithms and others and for that these compute uh that's why we provide this compute uh information as a part of the workspace database workspace to them and another important thing here is again they will use delta sharing as a part of this whole solution and the unity catalog and that's where will go very deep into uh on what lessons we learned. So uh as a part of a solution uh not as a part of an implementation to be specific.
What we what we learned was initially we thought hey we will just build a platform and let people come in into our system and do research but what we realized specifically in US many of the researchers have very tight security requirement and would require data sets to be there in their environment and that's why we learned that hey the platform as a service may not work here it will work well in uh in Europe environment environment because that's where a lot of there are fundings and other things available for platform as a service kind of a model but here people do want data in their data set so substantial amount of researchers would be in that would be use that offering the other thing that as we went into that journey of data download option we realized that hey why don't we use DBFS API right database given API uh but what we realized when we started using that is it requires the customer to log again again back uh that violated some of our UX requirements like we do want our researchers to do one-click uh download.
So that means you get a URL you click it and voila you have a data set. Uh so that was one thing that we realized and hence we did not use DBFS API. We ended up uh using or building our own kind of a custom solution with of course some bells and whistles and tools from data bricks. So with that in mind the solution part, architecture part and the business part is done. So I will now hand it over to my friend to move on to the next uh right.
So we talked about lessons learned and I want to point to this datab bricks API. Um the things I will be sharing is related to us using datab bricks APIs and CLIs and we'll talk about maybe less uh expected aspect of migrating to unity catalog and uh delta sharing and you will see it in a moment. So let's go to the next in slide that Sajiv started with here you saw two clouds the AWS cloud and Azure cloud and Sajiv was talking about research partner but the very first and very dear partner to us is this data bricks Azure team that is our own team that has to go through the same exact journey as the external research because the datab bricks in Azure is just as foreign to us consider starting from datab bricks in AWS as another research organization.
So our first journey was to get Unity catalog in AWS and then and given what that enables pilot the whole process of sharing data with our own team in operating in Azure and see that all connections are working and you know all the lessons learned are completed and that will enabled us to then later scale and be confident that uh when real customer will come then there no surprise So let's get back and start looking at the history of how we approached that project.
Our first implementation was in Azure in AWS. Um at slip number AWS is much younger pro project. It's only a few years old and it started as a green field. So whoever worked on it four years ago had luxury to come to brand blank slate build things properly with the frameworks with the right design where things are reusable and modular and there is includes and shared uh metas store um kind of JSON files. Um so that migration was um not really eventful to be honest.
We started uh I'm not going to read the slide you can see it but we started with P. P went very well. Everybody can follow instructions. There's so much documentation. You can create a workspace. You can create meta store. Um you can create table attach it to the meta store. It will be unity catalog managed. So PIC goes extremely well. Next step was to migrate a de dev development workspace. It's a very small workspace.
I'll share secret. Our teams are really small. single digit low singledigit people size teams development workspace um gave us some surprises. UX utility was crashing was not giving right output and uh we saw that migrating production will become different from doing the P. uh visa our friends from datab bricks and got an a senior architect to be with us for few weeks and that was speaking of lessons learned that was very impactful lesson learned.
Um we in a choice of do we give s to datab bricks or another consulting company listing 10 workflows 20 tables 10 notebooks please migrate versus invite somebody who understand u in depth the nuances and can teach our team. uh we selected the second choice and um what we learned may not apply today as much because UCX utility has many many revisions and the things that we saw would not maybe necessarily be there now but um the person who had many implementations came with a clear implementation plan and um kind of configurations and settings done right first time.
So this is the uh lesson we learned in AWS. It took us approximately a month to migrate first set of few tables and then you know pipeline goes on and on. Um so at this point what did we have? We had the Unity catalog in AWS. We had tables migrated to Unity catalog in AWS. We had workflows that could use those tables. Now let's uh go to the Azure side. Azure is little bit different than AWS. I said that AWS is clean slate designed with a framework.
There is medallion architecture. They include files. Azure probably also was clean slate 10 years ago with five generations or more of developers that some of them have never met each other and different teams. So it was a collection of many things. And first thing we saw there is that in Azure you have a tenant. So sleep number is a tenant and we have subscriptions subscriptions A and B in our case it is a lower subscription and production and if you have a datab bricks workspace in a subscription it is tied to that subscription in another subscription will be tied to production subscription as a datab bricks objects like views and tables there is no visibility across subscriptions because they managed by different hive meta stores and there is just no visibility What is the solution?
Obviously when you have data bricks with different um workspaces you would come to projects that require to exchange the data and the solution as you see generation one was well unfortunately let's get access to the blob store both data bricks workspaces can mount to the blob store and use it and it works but it makes people like me really uncomfortable because now we have rather random access to blob store that might have important data So we went to the next step and we created ETL jobs where now we have two blob stores and a very intelligent job to copy from here to here.
So we don't have to grant as much access anymore which is good and it's really good for our friends from data bricks because now we have a job that consumes DBUS and runs every day and if it stops we have a problem right so um um that's not really great idea so when we went to unity catalog unity catalog enables visibility across subscriptions and um datab bricks workspaces workspaces you could see in subscription but not across Um and now wonderful thing you can see uh tables in that meta store if they are in in datab bricks in Azure there is one meta store per region.
So our um implementation happened to be in US west. So we created meta store in US west region. Uh we enabled this two datab bricks workspaces to access that meta store. And now we can see tables uh from one met from one workspace in another workspace. Um that is very I cannot express enough how big deal is it for our maturity and the clarity and removal of unnecessary noise of some job failed um and for our stability of our product.
Um now but enabling metas store is only step one. And there are many videos and there's documentation that you can follow to do same thing as we did. You enable metas store you create table set un catalog manage table everything will work. What I want to focus here is a little bit different. First I want to talk about our specific case of security um approach that is um that was found in Azure. In AWS we have infra team that has necessary access and uh when we talked about creating unity catalog um configuring things like delta share I didn't even want to talk about it I said that wasn't an issue they did it it was not an issue it happened that um in Azure datab bricks a lot of things are really welded deeply in Azure and in this case in slip number specifically permissions to create a meta store require datab bricks account access permissions And that permission was given to a global admin uh site admin IM role.
Um so now in order to create metas store for our P which is not inventful at all we need to engage with the corporate security team and explain to them what we want to do and why we bothering people who has noneless access that entire Azure um configuration access. So all of a sudden our approval cycle, our requirements for documents for ARB became extremely interesting. In our case it took extra months to get through that.
In retrospect I think it's good because that scrutiny forces us to write more things and um those things should be written but just sharing confessing here in big room that um we found it after the fact. So ideally we would have found it before and put it in our project plan. Okay let's move now from this subject to the next and um oops I skipped through the plan. Yeah. So um some of you also operate data bricks right?
So what is a typical team size for users for business analysts for people who write SQL like how many people have teams like less than 10 people that uh okay so few people have small teams. Now how many people have teams less than 20 people writing different SQL? All right you have few. Okay so that is interesting room. Either people have very large teams and I'm jealous of you or you're not telling me the story. Anyway, so say you have between 20 10 and 20 people writing the code.
Um let's assume they've been doing it for few years. How many uh notebooks do you think you find? Like give me guesses. I need three guesses. Um my SAP friends, give me a number, please. in SAP how many reports you find if you have a team of 10 people writing a 100 reports okay so I'll skip through the suspense uh we found over 4,000 notebooks uh that uh now are there and when we at this point we created the P of course we made the table now we have a table and now we're ready to go from a PIC table one with two fields date and quantity to actual data.
And you do it, it triggers a change where table is now no longer referenced with two layers of hierarchy where you have a schema followed by table name. And now that becomes three layers of hierarchy. You need to have a catalog, a schema and a table name. That means that in your SQL you need to prefix your table with a catalog. It's a really trivial procedure. But if you change your source code, you need to deal with it.
You have a new code revision, somebody has to test. Um if there is important report, maybe there's some compliance requirements and all that. So you cannot just go and change. And moreover, you cannot um just make change in the back end and have all the notebooks just die. So then we have this question for our users. I say would you just you know I have project manager TPM and they want me to do it next sprint and you've been around can you change without table like products and can you change all the notebooks that have table products and just prefix it with a catalog uh that will not be a good choice and our users would not like it but the bigger problem is and this is what I think if you talk write down one thing from this it's not that users wouldn't like it is that even users tell you the answer.
That's not a true answer because they don't have knowledge of each of the 4,96 if I remember right notebooks and each of the includes and dependencies between those notebooks. They may tell you something but it is not true. How will you find true answer? You will find true answer for about after about a year because when all of your infrequently running jobs will start failing six months after you change to unity catalog then you live through that cycle and then you find it out and this is what I wanted to center now is um we did different approach we used datab brick CLI and API to do it programmatically and what we did is we expanded the scope first and we found all the code that uses data bricks all one workspace at a time.
We don't have that many workspaces. So you can do one workspace, two workspace, every workspace. You expand all the code then you classify it and contract into the code you need to worry and you worry about this code at a different level. Something's really really has to go through scrutiny. Something goes through quick change. But at the end you can show report uh and that tells you that you did not forget that one temporary notebook that actually is included in some really key report.
Anyways, so we use two approaches to get so first we need to get the source code. What source code do we need to get each and every line of a code that is there? In our case, it was done in two places. workspace has notebooks and I'll show you exactly how we get the notebooks out of workspace and then there is also notebooks in the g uh for which it's really like one liner get clone repo and you have it um after we collected all the source code in one uh Linux directory we can now uh search through it simple tools like uh even grap will tell you with the right uh if you want to use python You can write Python but you can easily find each object each notebook that refers a table you're trying to upgrade.
Now if you do that you can easily then use programmatic code editor set of Python and programmatically stick the catalog in front of the table name without spelling errors without typos without emissions and do it right. After that you can create unit tests because you can run the notebook as it was and the notebook as it is proposed compare the results and have some data quality metrics. But there is a trick and so we put a break here.
If you did that you could run a notebook that I wrote right when we started the project 10 years ago that says drop table users and then create table users with this new fields. And if I run it against production because my role is powerful, it's gonna have bad results. So we further classified all the notebooks as create, update, and delete to be treated differently than just report that select data. All right. Uh so let's take a look in details how we did it.
We used datab bricks APIs that I love. There is a link in documentation that shows APIs provided by datab bricks. They give you access, trivial trivial access to everything you see in datab bricks screen. You want workflow, it's there. You want genie, it's there. You want notebooks, it's all there. Um the link before datab bricks API brings you there. Or you just search and there is one page. And uh all of those um programs work really similar to what you will see in this screen and the next.
Um the rest APIs. um not going to impress you by saying that I have authentication here and there are some predefined URLs that you need to set and we will be using them on the next um function that I'll show you. So with this simple simple setup we can now access objects and recur through them to find what we want. So in the next slide we're going to do exactly this. um right here where the captions are is a first call to the function called list notebooks and it gives you root directory.
So before we actually get here you probably think why are you making such a fast you go to workspace UI and you search and it will give you all the workbooks with this table. Yes indeed it will. you get multiple pages in the output and you will not be able to like replace and test and manage things. So what I was trying to do here search is great it's absolutely necessary it's wonderful but what I was trying to do here is to get hand on the source code so I can analyze it including dependencies because search will not search dependencies if I have run directive run some functions do setup do something which uses table I'm trying to change now that search will not catch that because it looks at one notebook at a time.
Um so back to here um very simple code work through list function takes um set up parameters and get a list of objects returns you array. These are all the objects. We're only interested in two types of objects notebook directory. If it's notebook we do something with a notebook. If it's directory great we call ourselves again with a little longer path. Okay. Uh what happens if it's a notebook? Uh we call another API.
All these APIs are listed. This development is maybe one day development. Uh but the power it gives you is pretty good. Um you call another API, you provide this uh parameters here. So you're asking for download, you're asking format for the source and added the path to the specific notebook that we found in this directory. And uh with B 64 decoding UTF8 right to the file. Now we have it. Okay, great. So now we have this one big directory has lot of text files, their notebooks.
What to do with them? Um then we go to Excel because we can't complete any one project without going to Excel and um it really helps us to classify and to have very specific conversations with the people we need to talk to. Um on the right side you see list of the notebooks. on the left side things we care does this contain drop but it also contains update delete as well right insert also don't want to insert in production something right very simple rag access searches will tell me that if I have table and it recur back and I say update delete drop uh very simple I'll find it out programmatically for 4,000 notebooks reliably contains import notebook has run directive if it's there it's there then you have it contains tables.
That's important one. If it doesn't contain my table, we filter this out. And now after looking at the first three tables, I have a short list of notebooks they can discuss with people like Sajie because they wrote them. And um it makes our conversation focused and simple and we are not asking questions that we know do not have answer at all. So that helped us um to migrate uh tables to unity catalog enable three layer hierarchy and not um trouble our workflows not expect our failures.
So let's find where we are now at this point. Now we can remove blob access and the to manage data in Azure uh between subscriptions. And now let's move now to crosscloud uh collaboration. We already have same thing in AWS. And due to nature of younger project and a much smaller developer team, there was not a big deal with the notebooks. There's a lot fewer notebooks than here. Um and um now to enable the data sharing, we just need to enable delta sharing and create recipients.
The instructions it's real simple and straightforward. You create recipient, you get the key, you put the key into the other workspace. Now you have access and you see the data. So this uh comes to a little bit of a history how we uh managed this cross cloud. Cross cloud was just like the cross subscription S3 access directly was there as because of age there was VM with SFTP. Um sometimes ago we changed that to Azure sync agent.
Azure sync agent is good. It specify source and destination and both in sync. It does very good job. Um it totally takes care of it part. The only thing it doesn't have that you would want based on the delta sharing. You lose complete visibility in access. So if I have data in AWS and it is um important enough to control the access um and I can do that all my IM roles all my logs are there and then it goes to Azure magically through Azure sync agent and appears in Azure but now there is no link to Azure AM roles Azure users and what happens there u same thing in Azure I see this data but where it come from I have no clue it came magically so with delta share that magic disapp appears and now we can grant permissions and have access control to the remote delta shares and uh that problem has been will be solved and that was a big celebration in our team.
So this is a screenshot from Azure um datab bricks uh UI and delta shares re received is this is a AWS table that comes to Azure with no ETL with no shared S3 access and controlled with uh with a code info is a code when we get access to this table here we create um g commit it's reviewed it's monitored log Okay. And um coming back to the big picture, what has happened so far is um by enabling unity catalog, we enable visibility across disjointed um workspaces.
Delta sharing removes um ETL and shared access to blob stores. And on the upper branch you see expanding scope of u cross subscription cross cloud and then finally cross cross company data sharing. The down branch is actually a lot more interesting. Lighthouse monitoring only works with delta with unity catalog workflow migration. Um I'm not sure about others but we do have other workflow tools that cost us uh quite a bit and datab bricks workflow offer very attractive option if you are in unity catalog because all the features you need they are there but only in unity catalog genie um is many more includes genie and other AI tools and well that's our journey thank

Learn more about the Databricks Data and AI platform.