From Largest to Best: How We Transformed Databricks’ Biggest Workspace With Unity Catalog

Summary

• Governance automation: Learn how Databricks transformed its largest internal workspace using Unity Catalog, implementing governance automation through lineage tracking and unified access management.
• Scale considerations: This video reveals best practices for governing hundreds of thousands of tables, implementing consistent policies and building automation that scales across massive lakehouse environments.
• Real-world challenges: Discover how to address data governance failures at scale, maintain trust in data assets and build systems that enforce governance consistently on the Databricks Data and AI platform.

From Largest to Best: How We Transformed Databricks’ Biggest Workspace With Unity Catalog

Talk by Donghan Zhang, Sr. Software Engineer, Databricks ; Li Yang, Senior Engineering Manager, Databricks
Join us as we unveil how we transformed the largest Databricks workspace into the best-in-class lakehouse through Unity Catalog. Discover how we harnessed lineage and unified access management to build ultimate governance automation.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why is data governance important at scale?

As organizations grow their data assets from thousands to hundreds of thousands of tables, manual governance becomes impossible. This video shows how automated governance maintains trust and compliance.

How does Unity Catalog provide governance automation?

Unity Catalog tracks data lineage, enforces access policies across all users and enables automated controls that scale across large workspaces on the Databricks Data and AI platform.

Enterprise AI with built-in data governance

Enterprise AI with built-in data governance ensures that AI systems operate safely and reliably while maintaining data quality, compliance, and auditability at scale. This video showcases how Databricks implemented Unity Catalog across its largest workspace, enabling centralized governance, lineage tracking, and fine-grained access control for thousands of AI and analytics workloads. By embedding governance into the platform from the start, enterprises can accelerate AI development without sacrificing security or regulatory compliance.

Which solutions offer both data protection and data governance?

Solutions that offer both data protection and data governance on the Databricks Data and AI platform provide unified security and policy management across all data assets. This video demonstrates how Unity Catalog combines role-based access control, audit logging, and data lineage with encryption and compliance features to create a comprehensive governance framework. By consolidating protection and governance in one system, enterprises eliminate silos and reduce the operational overhead of managing multiple tools.

Full transcript

Five billion dollar $5 billion. Imagine I hit the biggest powerable jackpot in history and that money was going straight into my bank account and I walked away a billionaire. If that were true, I would be up here happily doing the Elon Musk bizarre dance. though be honest I cannot outweird the real billionaire. So I was daydreaming that5 billion dollars was not the jackpot I hit that money was what Facebook had to pay to the FTC the largest privacy penalty in history. here shows the historical moment.
On July 24th, two, 2019, Facebook, now is known as Meta, was fined by the FTC for $5 million. And does anyone here remember what for? Raise your hand. Anybody know? Okay, it was for Cambridge Analytica scandal where third party developers exploited the Facebook's lacks API data governance and harvest data from more than 30 million users without their consent and use it for political targeting and it ignited global outrage hashtags like delete Facebook, where is?
Trended on social media. Congress was calling. Zuck was sweating. And that $5 billion. Oh, that was just the beginning act. Next up, later in 2019, 40 million fine for inflated video matrix. In 2021, another 650 million settlement for violating the BIPA biometrics information privacy act again in 2023 1. 2 2 billion euro roughly about 1. 3 billion US dollars by the Irish data protection authority for violating the GDPR.
And these were not technical glitches nor operational hiccups. These were real data governance failures. And again and again consumers trust bleached and but now meta survived. And you might be wondering so what? How did they pour through? And actually they're lucky thanks to their massive scale and consumers lock in. And to be honest, people seems to just enjoy watching those cute dogs and the cats videos on Facebook and Instagram.
But in B2B world, it is different. Trust. Trust actually isn't a feature. Trust is the product. I'm senior engineer manager at Data Bricks. And before data breaks I was spending a decade at Meta. I joined Meta in 2013 and left in 2023. Yes, I was there when all those previous headlines news lines hit us. Oh my, it was crazy time. As I recall, it was feeling like trying to fix airplane while flying it and with all the regulators watching over your shoulders and we were scaling fast and had to dealing with all these trust crisis and try to rebuild credibilities and still shipping very fast of those product out.
So when I'm here talking about data governance, it is not from any white paper. I got all those battle scars. And on a fun fact, if you haven't noticed, if you trace the color pattern of all the companies I worked at, red, blue, multiolor, it loops. So now I'm back at the red with data bras clearly. I'm where I meant to be and it's my destiny to give you all all the talk today and I'm joined by my awesome team member Donhan Jang and Donghan is a senior engineer on my team at data brakes.
Before data bricks Donhan built a data platform for AI and analytics at Airbnb. Yes, Airbnb is probably the app you all have used to book a place to stay for maybe a family ski trip to Lake Tahoe, California. You will hear more from Ta uh Donghan uh very soon. But trust me, you are going to be in a real ride. So now I wanted to turn the spotlight on you for a second. Quick poll here. Who here has sensitive data in your lakehouse?
A lot. And how many of you already using Unity catalog? Awesome. Great. More than half. And because here's the thing, the key difference between in control and in crisis often boils down to what do you have done? What have you prepared before things go wrong at data brakes? We believe data governance matters. And here's the kicker. If data governance is hard, people are going to find a workar around. Trust me they will.
I see you knocking and if da d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d d da governance is easy that becomes paved way and people will just naturally follow on. So our mission was simple, make data governance easy for everyone. That is how we transformed our biggest data bras workspace from being just biggest to truly best. About two years ago, there were warning signs. If we didn't act fast, we were heading straight to this a smelly, messy, unpleasant swamp.
We were not in it. And but the risk was real. And at data bras, our ability to enable innovation really depends on how broadly and securely we can enable data access for the company. But we will at risk being stuck in this swamp of data governance pinpoints. I believe some of you guys are going to relate to what I'm going to share. fragmented access different team different organization they manage data access differently and inconsistently and rising security risks we manage mult multi- region multicloud with more and more AI workloads and manual reviews analyst reviews legal reviews privacy reviews security reviews all these reviews just slowing down everybody and evolving regulations, GDPR, HIPP, PAAA, DPDP, you name it.
More are on the way. And you feel like the ground underneath you is constantly shifting. And we didn't have any systematic and trusted access control. This wasn't just a theoretical risk. And what I talked about was already blocking our innovations and slowing all the teams down. While those challenges were real and things were about to accelerate because we were not talking about just any yet another workspace and we were we were dealing we have been dealing one of the biggest workspace on data bras as you can see in the chart about in uh October 2023 in under 18 months our workloads daily workloads surged from about 300,000 to nearly 1.
5 million. More users, more pipelines, more user cases, more AI workloads. All this added to complexity into how we govern our data and our governance model has to evolve and evolve fast otherwise we would have been buried by our own growth. To give you a sense of scale here shows uh where we are with our biggest workspace. We have close to 9,000 monthly active users. It's pretty much everyone in the company uses the workspace we manage.
We have nearly 1. 5 million daily um workloads running as I mentioned previously. And uh we have over 100 pgabyte of data under our management and we have close to 90,000 tables and are still growing at tremendous rate. So this is not a corner case anymore. This is the core of how data bras runs on data bras and at this scale the old governance model just didn't have a chance. So we have to find a new way. So how did we approach this transformation?
We grounded our work in these three simple yet a very powerful guiding principles. The first one we build a guard rails not a gates. We didn't want to block innovations nor even slowing them down. We want to build a paved way so the teams across the company can even move even faster with confidence. The second principle platform investment over exceptions governance shouldn't be living in manual reviews or any one off of the process.
It should be built into the platform by design. The last but not the least, trust through transparency. And we believe the more we share the more we expose usage logs, audit logs, access patterns and our governance posture and the more we can gain and maintain the trust both internally and externally. These three guiding principles guided every decision along the way we make while we building out our data governance platform.
And here I share with you the north star we were aiming for. Imagine data governance is a spectrum. On your far left is data liberal. So in here everyone can access any data as they wish. It's fast but very risky and dangerous. On your far right is data conservative. Here nobody access any data. Very safe but useless. Quick show of hands. Who here lean towards more data liberal? A few. Okay. Now, who lean more towards data conservative?
Relatively more. Okay. So, a lot of you are like us. So, our goal actually was to strike the sweet spot somewhere in the middle. We call it ultimate governance. In here it is personalized and optimal and we make the right access of data easy, fast, seamless and through our paved way. And at the same time we make the wrong data access hard being blocked by design with auditability and accountability. This northstar has shaped how we build our platform.
How we can be able to scaling without slowing down innovation speed. Let's make it real. So now I invite you to imagine with me. You and I, everyone in the room now we are wearing these governance hat. Usually not one hat, multiple hats. when you are in governance role and one day someone from your marketing team pinned you, hey I want to access that redacted customer data and what would you do? You were like well it depends.
You are right. The answer is not a simple yes or no. You have to think through these four questions. What data is the data sensitive where it lives? Does it have user consent to access? Next question. Who is asking? What is the people's role? Which team is this person on? Does these people have clearance? Third question, why they need it? Is it for analytics, for AI model training, for external sharing or just for internal debugging?
And the last question, how they are going to use it? Are they going to use you know our governant platform we build and offer or ad hocly even through their manual or customized whatever notebook and script hacky way is any AI oversight. So we want a system be able to having the answers for all these four questions ready so that the right decision can be made automatically. This is the high velocity, high leverage governance we want and the unity catalog cat catalog was the foundation we built upon which enabled us to build this to fully address the four previous key questions we talked about what, who, why, and how.
We build our governance model around the three core pillars of access. These pillars sit at the heart of our data governance and they are enforced automatically through our governance system. So picture that forming a triangle as showing on the right here. At the top of the triangle is attribute-based access aback. It answers the question well what data here we evaluate contextual attributes of the data it's sensitivity geog geography access um access consent and the bottom right of the triangle is role based access it answers the question who is asking and we want to honor organizational roles and the clearance levels so that we can enforce default uh entitlement.
Bottom left of the triangle is purposebased access control Pback. These answers why do they need it? We define clearly and enforce purposes so that we are not only just know who is asking but also know why behind the ask. So together a bank, RB and the PB bank they form this governance triangle and by all by enforcing all these three access models through our platform we we achieve final grand and dynamic controls. This answers the middle one how they going to use it use our platform.
All this ensures the data access and all the policies is applied consistently and responsibly throughout the entire data data life cycle. A critical part of data enforcement model is what we call privacy zone model. This is a framework where data access aligns with data sensitivity. As you can see after the data get ingest into our workspace the data travels across zones and when they travel those privacy tags come along with them and they got automatically enforced by our platform.
So here how it works in zone one here leaves the data which is raw highly sensitive and with full fidelity in the zone here with very limited access and preapproved purposes very limited and then move to zone two the data here is redacted and attacked PII AI information, personally identifiable information got redacted here. The data is tagged properly and access is purpose found. And then move to the last one, zone three.
Here it is centralized through tagging and aggregation and is very safe for AI and analytical uses. This is the broadest access tier. Most purposes are allowed here. Ensuring access is very secure and compliant and purpose bound. And when someone is accessing your workspace at any of the tier the privacy privacy policies are enforced automatically around all the egress route. So we talked about the framework the governance triangle and the privacy zones. you have to roll it out without breaking things and no throwing teams down.
So that's how we took a phased approach. In the phase one we call it the centralized governance. In here we started establishing by building the foundations and the control. Here and then phase two is pave the ways. Here we focus on making compliance access the default experience for everyone. Then we move to the phase three hub and spoke governance. That's the way we scale. In here we are evolving into a model where we have platform enforcement and also have domain ownership and coupled with AI monitoring so that we can scale safely and fast.
This phase approach allowed us to scale governance without bottlenecks and continuous improve monitoring. So now you have seen the reasons why we had to make the transformation and uh the guiding principles you know through the guiding our transformation and the actions we have taken from the northstar to the architecture to the privacy zones and to this phased rollout and as you can see earlier we showed this before.
We were heading straight into this if we didn't act fast. Luckily, we did. So, today our workspace looks like this. A clean, clear, well structured, full of innovation with us. Can anyone guess which lake is showing this photo? Bingo. Yep. That's right, Lake Tahoe in California. And since we are here in uh Mosone Center, not far away, a few hours driving from uh Lake Tahoe, I think it's a very perfect metaphor. But we don't magically appear in Lake Tahoe.
We are about 200 miles away. So, how we get there? fast and securely. Now, let me welcome my brilliant teammate. He is going to be our tour guide and a driver also. So, let's all hop into his this flashy flurry and cruise towards the beautiful well-governed lakehouse, the Lake Tahoe. Take it away. Now that you have seen u the challenges we faced and where we started and let me take you on the next part of our journey.
How we turned kiosk into clarity and how Unity Kellogg became our solution. As Lee mentioned, let's picture a secured and governed lakehouse as Lake Tahoe. It's beautiful. It's our dream destination. is where we all want to be. But how to get there? That's the real challenge. As you can see on this map, there are different routes to the Lake Tahoe. You could try your own path, take some questionable detours. You may even encounter some cliffs.
Sure, it eventually get you there, but not without a lot of risks, delays, and even stress. You could also fly. It was only a one-hour flight. It's very fast, but it's very expensive. Cost you almost $200. Honestly, it's overkill for most of trips. What if there was a better way? A way just designed for this trip. This highlighted on this map, it's a highway 80. It gets you to Lake Tahoe safely and efficiently. And for us, the highway was Unity catalog.
What makes a great highway isn't just the road itself. It also includes the road signs, speed limits that keep you safe. It also includes the inspection checkpoints that checks your license plate. It is a maps app that shows you where you're going and it also cameras on your car that shows you what's happening in real time. That's all what do I want and that's what exactly Unity catalog provided us. Not just a path but tools, observability and guidance you needed along the way.
That's the infrastructure we needed to travel safely and efficiently. Now let's buckle up and start our trip together. You will see the features built into the highway and how they helped us reach our destination. First stop, data security. On any good highway, what keeps you safe? Lane markers, speed limits, road signs, etc. For us, that was Unity Celloc. Unity catalog provided fine access control and specifically there are three features benefit us.
Number one attribute based access control or aback. It is a way to manage access by evaluating multiple attributes say data attributes like columns or tags or labels or user attributes like who the user is where the user is or even the time of the day. Second, we have dynamic views. It helps us implement it the row level and column level access control directly in SQL. And third, we have audio locks. It locks every user interaction.
Who access what, when, and how just like security cameras. Let me walk you through a real example that how we leverage the APEC at a data at a data bricks. We have large sales team and we generate a huge amount of sales data for each account executive. They should only see data for the accounts they manage. No more no less. Fair, right? But the challenge is how do we enforce the visibility rule? Especially when people move around or account shift hands.
This is exactly where APEC shines. For example, in this notebook, we define a row filter policy called sales account filter. In this policy, it will be applied to all tables in the schema product of sales. And also you can see it reference a UDF called account future UDF which runs at a query time for each user. The function checks row by row whether the account is visible to the user and it simply returns true or false.
We can also specify that the group of users this policy apply to in this case is account executives. Maybe you have a different um policy for different groups. See the engineering department they have less restrictive rule because they want to build some data pipelines on top of it. And last we have match columns using columns. It means for any table if they have a account column tagged as sales sensitive it the policy will be applied to them.
And the real magic of Aback is that this policy is enforced globally and automatically. Unlike the traditional rule level security, you have to config every table individually. For the APEC define the rule once, use it everywhere. Data security is one thing but data privacy is more nuanced. If you think of data security as the guardrails, the ling markers, then privacy would be the inspection checkpoints like this. Privacy isn't just blocking access.
It's maintaining access open access to non-privacy data as well. With the features provided by UD catalog, we are confident to open our access to the nonPI part of the world table with a broader audience without fear of exposure. One of our favorite features is privacy dashboard. It's basically a uptodate view of compliance posture. is like a real time control center and you can ask questions like what table contains PI data which tags are most common and who access the word you are always audit ready and you don't have to be panic when GDPR review hits here's how it works for another feature called data classification we have automated data classification in Unity catalog.
As data lands in our workspace, we automatically scan PI data. No manual review is required. Sensitive fields gets tagged as for example here the name is tagged as name and we have location. We have SSN. All of these columns are tagged as sensitive. Once tacked, the policies will be applied automatically like a row filter or column masks. It's a perfect partner of the APAC feature I just mentioned. Together, they create a visibility over who access what and under what conditions.
That's the beauty of data privacy. And we also talk about data security. And next up is data observability. Now imagine you are driving without maps, Google maps or Apple maps. You have no idea which axis are jammed, where the gas station is and how long it will take you to get there. We were in that situation before unity catalog. It was very hard to identify which table was being used, which teams were missing compute or where inefficiencies were hiding.
Then came system tables. It also another feature provided by Unity catalog. System tables is not is building tables data bricks hosted. You don't have to create separately but it keeps track of operations data like lineage building job history etc. From all those usage logs we were able to get many insights from them. For example, usage insights. We could say which table, which job, which notebook touched each table.
For cost attribution, we identify high cost, high price. And we align them with team ownership. It's just like checking who is spending too much gas. Third, we have data deprecation. For the tables that hadn't been queried for over six months, we either archived or deprecated them. Last lineage. We saw the full path from very upstream sources to downstream metrics and dashboards. It's like a true history of each data set.
Let me share a true but unhappy story. Three weeks ago, about three weeks ago, I was on call when one of our widely used tables got delayed. Instead of being panic, I quickly identify the root cause and it was turned out that um we have upstream table delay. Once we identify the root cause, the very next step was to perform uh impact analysis. Let me pause here and ask how many of you have ever been an encore engineer.
Cool. We have lots of hands-on people today. And for those who have can you think back and give a rust estimate like how long it would take you to do a impact analysis say generate a list of downstream users who access tables for the information like that. How long 30 minutes one hour or even more? Thanks to system tables I was able to complete the impact analysis within just five minutes. Five minutes is the runtime of our notebook and this notebook is for handling async like this.
As you can see here the notebook takes in three parameters table name, oxision level and look back window. And under the hood, the notebook will identify the downstream users by accessing the system tables where the lineage information lives in. And it will get the list of users who access this table within the specified time frame, say 30 days in this example. This level of visibility was a game changer. We were not just governing data anymore.
We were optimizing by reducing cost, by cleaning up unused data assets, by tuning our pipelines. And for the instant case like this, it helps us minimize disruption and quickly efficiently communicate with impacted users. Okay, we talked about security, privacy, observability. Let's talk about future and how we brought future to life. Unity catalog didn't just get us to Lake Tahoe. It's built for the next generation of datadriven experiences.
Think of self-driving cars, but for your data. For us, that was Genie Space. Genpace is no code natural language interface at data bricks that lets business users ask questions like what's what was my revenue. So business users can analyze or explore data by asking questions without writing any SQL. No SQL no coding is needed and it directly connect with the Unity catalog tables. So you that you can get instant insights and visualizations.
It makes self-s serve analytics easy and accessible for everyone. So anyone in your company as long as they have the right permissions, they can ask questions like what is the latest monthly active users or what is the revenue? What was revenue by region last quarter? Genie returns reliable answers and Genie is great but our team made it even better. , can I open source our recipe? Cool. Our secret sauce is metric views.
Let me introduce metric views quickly. Mesh reviews is where we define core business KPIs like revenue, active users. Instead of each team coming up with their own definitions, we have a centralized global place to consolidate those definitions. And metric views not only serve as uh like a single source of truth for metric definitions, but also it is an actual context that we can share with Genie Space. With that extra context, Genie Space was able to provide more accurate consistent answers with us.
And besides, Genie space and the match views were governed by Unity Kellogg. So all the answers were permissioned and you can query them across tools like SQL editor, jobs, notebooks, anywhere. So everyone could get the same and trusted answers. Okay, let's step take a step back and look how we got here. We talked about data security. We introduced the APAC making sure the right people access right data at the right time.
For data privacy, we have automated data classification. So the sensitive information is identified and projected without any manual effort. For data observability, we have system tables provide visibility into lineage, building, job history. And for data intelligence, we have genie space. It empowers different teams to understand and collaborate on core business metrics. All of these innovations brought us to our destination, a trusted, secure, and scalable lakehouse.
This is our real story as the data platform team at data bricks. We are running the largest workspace and we breathe live these challenges every single day. If Unity catalog can handle data governance at our scale, I'm confident that it could work for you as well. The road is paved, the features alive, and the destiny is definitely worth it. Now it's your turn. We invite you to start your trip with us and we look forward to seeing you at Lake Tahoe.
Thank you.

Learn more about the Databricks Data and AI platform.