Managing the Governed Cloud

Summary

• Governance Architecture: Build a comprehensive governance framework across cloud resources using the Databricks Data and AI platform to ensure compliance, security, and data stewardship.
• Multi-Cloud Management: Manage governance policies across hybrid and multi-cloud environments, maintaining consistent security posture and regulatory compliance across diverse infrastructure.
• Data Stewardship: Implement centralized data governance and compliance controls that enable safe, auditable data access while supporting business agility and innovation.

Managing the Governed Cloud

Talk by Johnathan Powell, Staff Data Engineer, General Motors ; Sherri Adame, Enterprise Data Governance Principle, GM
As organizations increasingly adopt Databricks as a unified platform for analytics and AI, ensuring robust data governance becomes critical for compliance, security, and operational efficiency. This presentation will explore the end-to-end framework for governing the Databricks cloud, covering key use cases, foundational governance principles, and scalable automation strategies. We will discuss best practices for metadata, data access, catalog, classification, quality, and lineage, while leveraging automation to streamline enforcement. Attendees will gain insights into best practices and real-world approaches to building a governed data cloud that balances innovation with control.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What does governed cloud mean in this context?

Governed cloud refers to managing data governance, compliance, and security policies across cloud resources on the Databricks Data and AI platform while enabling business teams to move quickly and safely.

How does governance support multi-cloud environments?

This video addresses strategies for maintaining consistent governance policies across hybrid and multi-cloud deployments, ensuring compliance and security regardless of infrastructure location.

end-to-end platform security across data, governance, and AI?

End-to-end security requires integrated controls across multiple layers: data governance (who can access what data), network security (secure connectivity), identity management (authentication/authorization), and compliance monitoring (audit trails). This video explains how Databricks provides an integrated approach: Identity and Access Management controls who can use the platform, Network Isolation restricts connectivity, Encryption protects data in transit and at rest, and audit logging tracks all actions. For AI workloads, this includes monitoring model access, tracking which data trained which models, and maintaining audit trails for regulatory compliance.

Best practices for deploying internal AI apps with data governance

Deploying internal AI applications safely requires: (1) governing data access so apps only see data they're authorized to use, (2) tracking model provenance (which data trained which model), (3) monitoring app behavior for anomalies or misuse, (4) enabling audit trails for compliance. This video shows how Unity Catalog governance on Databricks enables these practices automatically. Apps request data through governed APIs, access is logged, and audit trails remain immutable. This transforms AI app deployment from a security risk to a controlled, compliant process that meets enterprise standards.

How will AI agents change the role of data engineers over the next few years?

AI agents are transforming data engineering from a traditional ETL focus to building "data infrastructure for agents"—ensuring agents have secure, governed, real-time access to accurate business context. This video suggests that data engineers will increasingly focus on: (1) curating high-quality, governed datasets optimized for AI (embeddings, time-series, multimodal), (2) managing real-time data pipelines instead of batch-only ETL, (3) ensuring data governance and compliance for AI access, (4) building feature stores and agent memory systems. Engineers who evolve to support AI workloads will remain central to enterprise infrastructure.

Full transcript

Hi folks, thanks for coming. Um, I work at General Motors. I started within their softwaredefined vehicle vis uh space where I defined a vision for data governance and then took on a role uh for their enterprise data governance team and that's where I sit today and I'm leading the team through like an endto-end implementation for data governance and this is one of our key experts as well. Thank you. Uh my name is JP.
Uh I have roughly about 15 to 17 years of overall IT experience. Uh over the last seven years though my primary focus has been in data governance and the last two and a half years really specialized in data access. So it's nice to meet everyone. Yep. Thanks for coming. We know you guys are like the true data people at this data bricks conference. So I'm going to memorize all your faces, right? And I'm the kind of person that thinks like there are data people and there are people that are not data people.
Like it's quite black and white to me actually. Um so this is the data bricks forward-looking statement piece. Don't forget to complete your surveys and then let's get right to it and talk about how we're shifting left on data governance at um GM. So when we started we said okay the corporate data philosophy is really around how do we get things faster to market and um do it in a governed way and really when we think about faster to market and agility when we looked at the data culture within GM it lagged behind.
So if you think about GM as this huge organization there are teams on the spectrum of data maturity. So how do you crack that nut? We came up with three things to focus on. One, we wanted to shift left. So, insert governance at the beginning of the development life cycle. So, our team sits within the engineering space. So, we partner directly with the engineers and put in some processes and I'll describe what those are.
We also wanted to shift from being reactive to being proactive. So, long gone are the days of, oh, you've delivered this to production. What's the definition for that? Should we put some quality rules on it? How should we protect it? Right? We want to have all those conversations up front so that when it deploys in production, we have the policies in place. And then we have the shift from big data to domain data. I won't spend a lot of time on that.
I think if you're here at the data bricks conference, you are already shifting probably from big data sets to more curated approach to delivering data products. So when we think about governing from design, I talked about us sitting in the engineering space. We play a role data governance plays a role between strategy and analytics and data operations in the data platform. We partner very closely with the operations teams and the platform teams to make sure we understand what the data is and it gets uh registered in our catalog and classified and we understand lineage around it and all of that.
So we can do governance by design but we also play a role with our legal cyber security teams as well as the privacy teams to understand what's the strategy and approach what's what's our perspective or point of view we take on our data and then we of course work with the analytics teams that consume the data from the platforms. So we have five core principles. So everything that our team does really focuses on these things.
One, we wanted everything to be proactive, automated, and adaptable. It can't be an inflexible thing that you can only use with one data set or one data type. It needs to be able to uh you be used across all kinds of data. We also want our data to be easy to over uh easy to discover, understand, and use. So that's a hard thing to do. Uh but we have deployed our Atling catalog to inventory everything within our data bricks environments and pull it in.
We're currently focused now on partnering with our Snowflake teams to pull in their data sets and we started conversations just in the last two weeks to talk to um the folks that use Google Cloud uh and pull in their metadata as well so that data from across the organization is easy to discover. Um the understand piece is difficult. So because we're in the engineering space, uh we have a really good handle on what's happening from an engineering perspective.
So the technical perspective on the data, but the contextual understanding of what that data is and how to use it has been a tough nut to crack. Um so if you feel like that's something you're excelling at your organization, please find me because um we're we're still working on that. We have a lot of metadata and we're enriching it. We score our metadata. We have policies that we deploy automatically, but this conversation with who is the business steward um for us because we're embedded in engineering is uh we're working on that now.
Like we're going full force on that, figuring that out. At our place, we have a product management team and it's the product management team that are defining these things and they really are insulating us from who those business resources are that we need to be getting into our catalog. So uh that's something we're working on today. Data governance must also meet security legal and compliance considerations. So we have regular forums with our legal and cyber teams where we talk through what their requirements are and then we reflect it and practical approaches that we've deployed in our framework our framework.
Uh we also committed to monitoring data usage. So every piece of data that you have out in a production environment carries an amount of risk. If you have a piece of data that's out there that's not being used or not being used as it was originally intended like our push is to bring that transparency to the usage piece not only by people but by systems or applications and then push to have those things shut down that are not delivering value.
So we have a set of usage reports that we're deploying where we look at all the data across data bricks and look at the usage on that and then present like what's what's being highly used to what's not being used. And then the last piece is we have structured standards for metadata where we have YML files protobuff files that we collect from services and applications. uh we ingest that information and pull it in in a structured way and we can understand data ownership right from the inception when it's being released to a production environment.
We register data transfer. So any transfers that are coming in from external third parties or transfers that are going out to external third parties, we register those things and we put a waiting or risk on them as well. And then we're going down the AI path full force. So right now we have an AI governance PC where we're collecting a structured set of metadata. Um we'll collect it from GitHub and a structured YAML file.
We'll snap our data catalog at it and then ingest and automatically read those models as well. So we're uh traveling down the AI governance path. So for us like I said everything has to be proactive. We're looking at it end to end. We don't want to put just a piece out there. We want to put in that endto-end process that really helps us understand uh what's happening with our data sets and you know in five minutes be able to say this is the data this is how it's being used who it's being used by in what country or by what application in what country as well.
So when I look at the tech technical solutions that we have so of course we have accountability and consistency. So we defined a set of processes. We have specific set of roles for um stewardship. Like I said I feel like we've nailed technical stewardship or the um data curators. We have very good relationship with them and we're really exploiting the value of that. Now we're forging down the path of how do we push back further left and get into the business's head to understand the context of our data better.
And the technical solutions we put in place um is metadata management. We have an internal uh internally customdeveloped metadata repository that allows us to connect directly to our GitHub environment and protect our uh code bases that way. It's like a layer in between all these thirdparty tools that we're using. So we can collect protobuff files, uh, YAML files, JSON files, ingest those directly from release branches in GitHub, uh, parse it, rationalize it, and then expose it out to the data catalog.
We also use um I'll go to the bottom one data classification that allows us to uh once once we get the metadata in that's going to be released to production. We classify it with a utility that's platform agnostic. Um it's actually going to release on June 30th. We're in the final bits of testing on that. uh we worked with the audit team as well as our privacy and compliance team to make sure that we're class classifying appropriately according to our standards and that um we're putting that out for all production data because we had an aggressive OKR goal this year to say that 95% of the privacy related data would be classified as so and 100% of the metadata for that CLA those things that are classified as PI data would be um enriched or the metadata would exist for that.
So um I think we're going to make it, you know, fingers crossed. Check with me again in November. I'll tell you how it went. But we're releasing that to production. So those are kind of like the front end of things. We have a data quality platform, Anomalo, that we purchased. It allows us to detect anomalies on our data and then reflect that information back out to our Atling catalog as well as um allow our engineering teams to build complex rules within that tool.
Our data catalog is Atlin as I said um data lineage is also brought into Atlin. When I started on this journey, we spent a lot of time having to answer why are you bringing in an Atlin if there's Unity catalog. And so what we did uh when we talked through that was we really wanted to have a catalog of cataloges, right? Because we have all three uh major cloud platforms, we wanted a catalog that would allow us to look at metadata across all these things.
And we felt like the search functionality in the Atland catalog was better than the search functionality in Unity catalog. And we did time studies against that. And what we found it it was um six times faster to use Atlin to search and find the same information that you would find if you were searching in Unity Catalog. So you can you can use Unity catalog, but if you have a really big complex environment, you may want to consider using something that can really address that complexity in what we feel like is a more complete way.
And then the last piece is data access. So today when we first started now two years ago, um everything was on prem. They had ranger policies and there were thousands of them and nobody quite knew what those ranger policies were doing. So one of the first tools that we selected was auda that allows us to deploy data policies. Um so and JP's going to talk about this in detail so I won't steal the thunder but a very few data policies for the huge vast sets of data that we have.
So um we're on the journey with data access and JPA is like his thing and we'll talk about how that works within the framework itself and then so when we think about where do we fit in with um interactions with our organization. So if you think this is just typical like build cycle right there's a business need and a go no-go decision. So when it gets to development, that's where we kick in. We have our first interaction where we're saying, "Hey, here's the definition of ready for the epics or features that you have that include data."
So we describe to the product management team the acceptance criteria that needs to be in those features or actually in the epics and then the engineering teams pick up those fe pick up and create the features so that um they know what to build against and we we tell them what the expectation is at the get-go and then we go through a certification process. So while they're testing, after they've tested, they'll be deploying a YAML file or a Protobuff file or JSON file to a release branch.
We'll um automatically pick it up and certify it and provide some feedback on that file. If there's metadata in there that's not complete, we will um reflect out to them what the errors are and what standards they have to meet and they resubmit. Um it's just like a regular operation now. And then, you know, it eventually gets released into production and we get to data use. By the time we get to data use, we're set up because we've collected the metadata upfront.
We understand what's being released into production. Um, we've done uh we've understood what the classification of the data is and the policies have been written. Now, when the data hits production, we're in a protected or governed state. And that's where I leave off and JP will talk to us about designing for trust. All right, thank you. All right, so let's just go ahead and get started and jump right into it. Um, I think like she's mentioned a lot about shifting left.
So this idea of shifting data security left is what I will discuss, right? But let's first just start off with some of the challenges and to give you guys historical context here, right? So like two and a half years ago or two years ago when and I started like security was basically an afterthought, right? Which led to very slow data access. And what I mean by that is that the from an engineering standpoint, their main goal and idea is to create data or land data that would essentially support data processes or some sort of application or business processes, right?
And so what what would happen then that would lead to a bunch of tables that exist in this schema structure that will have a mixed bag of or levels of data access which would lead us to manual controls you know which obviously those manual controls do not scale. And so um with that being said, you you now you have this picture of like these three components that are not connected, right? So like you have engineers, you have a security aspect there and you also have a a data access uh component to this.
Okay. So our idea right is in shifting data security to the left primarily evolves around um having upfront conversations with our development teams regarding security data security having them really start to think about you know who's going to be accessing your data what sensitive data will you be storing and um and being able to create those uh clean boundaries boundaries or clean clean schema boundaries so that we can then begin to automate and enforce uh data access policies.
Okay, so just to dive a little bit deeper into some of those components. So tag sync and enforce. So as alluded to uh we we've purchased uh Atlin as our primary data catalog which is essentially used for uh storing our metadata as well as our source of truth. And so our data stewards will then go into Atlan tag the data that's based on their security posture and then those tags would essentially sync to Ammuda which is our data security platform where policies will be uh I guess uh where we apply policies automatically uh based on the tags and attributes assigned to our data sources and uh users and groups.
So I one thing to call out here the only thing that's that's manual today is the attributes on the users and groups. Everything else is is basically um automated today. Next we get into designing for purpose and sensitivity. Uh so we've worked with our engineers engineering teams like showed you guys on that um that one diagram exactly where data governance inserts itself which is in the uh upfront the development part of it.
So our data governance team today we have a uh we play a critical role in schemas being created. So upfront we're asking the engineers okay so what type of sensitive data are you storing? where does that data exist? Who needs access to that data? Um, as as well as documenting that business purpose. And so I I think one thing I I really failed to mention at the beginning was was that regarding like our the manual process for granting data access to these tables.
Uh we wanted to really roll that up a level and look at data access from a schema level. And so the main way and the only way for us to do that was to create those clean schema boundaries to support those automated policies. So here it just gives you guys a overview of how our access request workflow looks like today. Uh so we operate using sale point and uh salepoint is essentially like this air traffic controller right that is going to direct traffic to uh the appropriate approver.
So a user goes into the system requests access to a specific role. That role will then get routed to an approver for a decision to be made. Once the approver approves it, if I could think of a better word for that, um then that user then gets added to the appropriate AD group where they're scanned and synced into our uh Amuda tool where the um where the access will be granted to those groups based on the attributes applied to the uh applied to those groups.
There I am speeding and rushing right through this thing. Let's see. Okay, so just uh very quickly how our um attribute based policies work. So this is a screenshot of what an Amuda policy looks like today. Um, so we utilize their advanced DSL here where we're we were able to call this function called has attributes and essentially what this says is that um it's going to look for a variable called allowed tables on a group or a user and then followed by that are the specific values that will be stored in that variable called allowed tables.
And so for us, it was very critical that we do not manage access at the table level because what that would mean is we would have to list out literally every single table that group would have access to. And so part of again just going back to having those upfront conversations and tough tough conversations with the engineers uh just designing an approach where we could uh deploy access at the schema level. So in this case, the how we would do that is we would list out the database or catalog followed by the dot and then we'll reference the schema uh that they'll need access to and then in order for them to gain access to all tables within that catalog and schema, it'll be a star.
And so the other cool thing about this Amuda policy is that you can write it at any level of granularity. So if you wanted to grant access to all cataloges or all schemas within that within that catalog, it would be catalog. star. star or all tables within that schema. Uh catalog. s schema. star or you can go to more painful route and list out table by table here. Uh so the other the other cool thing about this is that um along this journey of data access we were able to simplify the number of policies that we wanted to deploy.
So we like we're in a uh data mesh model, right? Where we've broken up big data into domains. And so we you could essentially roll out one data access policy to control all of your data because what how you'll restrict the data access is simply by the referencing the database structure there. or you can you know assign a policy per domain like we've we've chosen to do here. And so what we've done is uh created a auditable structure and repeatable process for administering data access.
Whereas in the past in this uh reactive approach here data access was just all over the place, right? So like you can receive email requests for data access or they could follow a process or if your friend was the owner of a schema or table that you needed access to, you could just go over to them and tap them on the shoulder. Okay. Um and I'm actually nearing the end of this thing. So pretty much I just wanted to put a quote up here, right?
Just to reiterate the idea behind security, right? And so like most times when we think about data security, it's like a afterthought, right? Like that final um that final step in this process after you've done all your work. But really what we have to do is we have to be proactive in our approach to security by having those conversations upfront with our data engineers involving our data stewards and uh creating a policy so that trust then becomes scalable, right?
And so like uh part of this process of of doing this upfront work was that you're now just managing the exceptions here, right? So like if you can imagine being able to separate your sensitive versus your nonsensitive data or the different levels of data access then you can pretty much just feder or I'm sorry democratize that data set right by uh auto approving which is what we do now for a lot of our roles. we auto approve the our 80% of the data and then manage the 20% by way of Amuda as well.
So just being able again just being able to just to have those upfront conversations with the data engineers understanding what data they're creating that they're working with who should have access to it allows you to build a much more scalable and robust process here. Then finally, uh, what we've achieved, and we still have a, you know, a long ways to go here, but really, um, our main, one of our main objectives besides just making data access easier, was to be able to answer who has access to our data sets and why.
Um if you can imagine right like being coming to that from that manual process it's very difficult to determine who has access to your data in a timely manner or reasonable manner. I think we were taking maybe two to three days to answer that question. Now we're under you know 20 minutes or to an hour. I'll just say under an hour today. Um access is granted consistently through uh a centralized policydriven controls by using Ammuda.
Amuda really helped out a lot in this space for us. Uh just by um their design and approach into simplifying the number of policies like had mentioned earlier. Uh we were using ranger policies to control data access but now you can simply you know with a minimum set of policies roll out and protect your data in this automated fashion by using those policies there. Uh and then lastly, yeah, audits are faster, cleaner, um and backed by automated, uh enforcements there.
Uh so we are uh so Unity catalog has allowed, you know, uh opened up a lot of those system tables out there for you now to look at to determine who has access to the data to your data. Ammuda's actually doing some work in that space as well. And I think that's that's pretty much it. I don't have anything else. Any questions? Any questions?

Learn more about the Databricks Data and AI platform.