Using Catalogs for a Well-Governed and Efficient Data Ecosystem

Summary

• Catalog governance: Unity Catalog enables data management controls at scale across enterprise data platforms.
• Capital One's strategy: The company leverages Databricks governance capabilities to tackle enterprise-wide data pipeline management challenges.
• Unified architecture: Modern data platforms combine SQL functionality with governance suites to solve data management at scale.

Using Catalogs for a Well-Governed and Efficient Data Ecosystem

Talk by jim Lebonitte, Distinguished Engineer, Capital One ; Kajal Woods, Vice President, Capital One Financial
The ability to enforce data management controls at scale and reduce the effort required to manage data pipelines is critical to operating efficiently. Capital One has scaled its data management capabilities and invested in platforms to help address this need. In the past couple of years, the role of “the catalog” in a data platform architecture has transitioned from just providing SQL to providing a full suite of capabilities that can help solve this problem at scale. This talk will give insight into how Capital One is thinking about leveraging Databricks Unity Catalog to help tackle these challenges.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does Unity Catalog help with data governance at scale?

Unity Catalog provides data management controls that help enforce compliance and reduce effort required to manage data pipelines across the organization. It enables centralized governance policies that scale across hundreds of users and thousands of pipelines.

What challenges does Capital One address with Databricks catalogs?

Capital One uses catalogs to manage data pipelines at scale, enforce data management controls, and reduce operational overhead. The approach helps the company tackle the complexity of managing data across multiple platforms and pipelines.

Why are catalogs critical to modern data architectures?

Catalogs have evolved from providing just SQL access to delivering a full suite of governance and management capabilities. They enable organizations to solve data management challenges at scale while maintaining control over access and compliance.

How does Capital One's approach apply to my organization?

Whether you manage dozens or thousands of data pipelines, centralized governance through Unity Catalog helps enforce policies, reduce manual management overhead, and maintain data quality across your data ecosystem.

Full transcript

Hey everybody. Uh, nice to see you all and thank you to all of our friends that have come to support us. That's super appreciated. Um. I'm a senior distinguished engineer at Capital One. And I lead the enterprise data storage and consumption platforms for Capital One. So nice to see a lot of familiar faces. Looking forward to a great discussion and hope everyone's having a great summit so far. So we're going to talk today about kind of cataloges and I'm saying it plural for a reason as you you'll see as we get into this.
Unity is one of them and then we'll talk about a different type of catalog as well. But so uh as we go this through this first we're going to talk about capital's general tech transformation. It's very relevant for the rest of the conversation. Um then is going to take us through how we can leverage cataloges and platforms in a data architecture. And um I'll then take us through some of the capabilities that enables from a technical perspective and then Kodle is going to finish it off with some how we've actually used some of these capabilities with real business projects and takeaways.
Sound good? Sweet. All right. So, so Capital One um you know our our founder like really prides himself and the company on being a technology first company and I mean we definitely definitely are and this is kind of a timeline that takes you through some of the journey we've been on. So starting in 2014 we migrated to restful APIs across the ecosystem. Um and then we really embraced open source in 2015. Uh so in 2016 we started you know modernizing some of our workloads to the cloud and then 2017 we really started modernizing our data ecosystem as well.
So in 2020 I think everybody has heard at some point that Capital One is fully on cloud. We are so we are fully on AWS. Um and then the other like big things that we've done is we we have launched a commercial part of Capital One called Capital One Software. And if you all haven't seen the expo booth and seen Capital One Slingshot, so definitely go check it out. But we've been building commercial products in the data space now for the past three years and you know the team released data bolt uh this year.
So definitely check that out. It's a cool tokenization product that you know we we actually built that first in e data uh internally and we've now released that partially or selling it partnering with data bricks. So when we did do that move to cloud that I talked about, we started you know not super organized. We we followed a you build it you own it mindset when we moved to cloud and we basically said hey look you know we want to get to cloud want to go as fast as we can.
If we really try to plan this and be super organized it's going to take forever. So we did it we got to cloud by that time frame and said in 2020 since 2020 we've really been trying working to optimize our workloads on cloud now. So you can see here that we we're kind of moving from where we had all of these kind of unique workloads in lines of business and we're now trying to shift it more towards that we're building enterprise products and platforms that have shared capabilities that we can reuse across everything and then lines of businesses can really focus on driving business value and working on like the core business logic of stuff that needs to happen.
So this is also a pretty important concept that you'll see throughout the rest of the talk. And just so you see some of our demand. I mean with the data bricks platform today we're a pretty big customer. We're we're growing but I mean this we're already running you know 500k queries per year 4 million jobs per year. We're 5k users and again this is just the start. We've been you know super thrilled with data bricks and been investing here.
So this is uh you know just to show some of the scale that we're working with. So with that pass off kajel. Thanks. So in this section I'm going to cover a little bit about our business cataloges evolution of our data cataloges and then how the combination of our business and data cataloges really powers the access to our lake and consumption from our lake. So when we think of the business catalog this is really for Capital One as we describe it.
It's the front door to our data. And why do I say it's the front door to our data? Because this is a one-stop shop where all our data producers and consumers come in to register their data. They have various data definitions. Here we do a whole bunch of data management activities that are powered through the registration of data in the business catalog. Subsequently, there are a bunch of data asset creation activities like orchestration of the data and federation of the business catalog and technical catalog, provisioning of our data pipelines, both federated and central pipelines that create assets and eventually uh the access provisioning.
And finally, the reason we also call our business catalog a front door is it allows for data discovery, gathers statistics on data usage, etc. So what are the key things that happen when a user or producer really registers their data set? What do we capture in a business catalog? We capture things like the schema definition, the data set description itself. We capture key information like um data classification. So this could be like your human sensitive data whether your data contains PCI information, NPI information and then we drive a whole bunch of access provisioning based on this data classification allowing the users to see NPI or non-NPI data and come up with a whole bunch of roles and definitions and mappings to these roles.
So uh this is probably like the high level idea. We'll go into detail on how we really orchestrate and the technical implementation. But the key thing to note here is it is the front door entry. We capture data set registration information. We also have a integration with our enterprise ontology tool. Here this ontology tool provides like standardized metadata, data definitions, column names etc. And then we power a whole bunch of services to do data discovery, data observability and data life cycle management through a business catalog. , if you could move to the next slide.
Now when we think of our technical catalog and in general the evolution of technical catalog particularly from Hadoop era to now there is a change in how there's a change in the data architecture and this shift is very reflective of the adoption of Hadoop with the expansion of data and the need for like cheaper data processing and storage. As a result, any technical catalog that was built such as Hive on top typically only allowed you to do schema management allowed users to query this data.
Now think of transition into uh cloud and data in the lake. With the rise of Spark and Presto, there was a need to also for data in the lake to separate the compute and storage. With this we really needed our catalogs to allow for data discovery uh data quality and access control. Fast forward to present day in today's lakehouse architecture data technical data catalogs have a far expanded set of capabilities. They tend to be the central nervous system.
And why do I say that? At this point now not only do we configure data quality rules, we capture partition information there. Most recently, if you've heard of Delta live tables, you do declarative ETL pipelines. We also configure data observability uh collect data lineage and enforce fine grain access control system. And we'll talk more about this as we dive deeper into Unity catalog integration. But holistically, this is the evolution of catalog.
So we went from really just capturing basic schema definition and being able to query to adding a few more capabilities to do data access and control and data quality. And now the next generation of it is really expansion and driving a whole bunch of enterprise services for data management, data provisioning, data quality, tokenization, all of the things that just mentioned earlier as well as part of our product release. you want to move to next one.
So as I mentioned the combination of really the business catalog and technical catalog is really what drives the access and consumption for our lake. So I mentioned initially at data set registration we capture a whole bunch of information as the user or the producer on boards to uh to our catalog. Now in the next step of this process once the data is registered and a data steward approves the data set itself and allows for provisioning access to this data set.
The next thing we do is we actually in the technical catalog we capture the S3 location of this data set and with this we also do provisioning of the access controls associated with this data set. Another key aspect of this is also like data life cycle management. So once the data set is configured uh users have access to this data. There's data set publishing like where you have both streaming and batch data that is published through these pipelines and these pipelines are both centrally managed and fed federated as well.
In centrally managed pipelines, we are able to include our SDKs that enforce again governance and these governance are driven by data that's already registered in the catalog based on your schema, the enforcement of the schema, the tokenization, all of these capabilities that we included as part of the technical catalog itself. Another key function I would say of this data and the lake in the data in the lake is primarily think of life cycle policy.
So we store our data in S3 and as part of the storage of the data in S3 we manage the data in various tiers and the least frequently accessed data is in in a tier that is the cheapest. Now again how do we derive this information? We derive this information when we configure the data set. We apply life cycle policies and this information is also driven by information in our data catalog. So as you can see like the combination of the business catalog which had a lot of information and characteristics about the business data itself the technical catalog which has a lot of information about the physical table names the partition if it's a iceberg or a delta table that information all of this information is captured in the technical catalog and then we have a whole bunch of services that are powered through the technical catalog.
And these include data registration, data pipelines, um data management activities and data observability. So th those are I would say like some key highlights of how we are using the combination of the uh business catalog as well as technical. I'll move on to the next slide. I want to talk in general about our lake architecture and again the role that the technical catalog plays. So at Capital One we have built our own business catalog and we have three technical cataloges and each of these cataloges stay close to our consumption tools.
Data bricks is one of the consumption tools we use today and it is also a way in which we ingest data into the data lake through uh delta uh through delta tables. Now the we our lake actually is open table format. So we support both delta as well as iceberg data and we use data bricks key capability called delta uniform. What this allows us to do is when a user writes the data in the lake in delta format the delta uniform capability correspondingly also generates iceberg metadata and hence we are able to support multiple consumption tools.
We are able to access this data through glue catalog with EMR. We are able to access the data through snowflake and then we are obviously able to access this delta meta data uh and data through data bricks itself. Uh on on top of this lake itself again like I said we have technical cataloges. Each of these cataloges live very close to the consumption tools and we have built federation and orchestration capabilities that sync the data from a business catalog into each of these technical cataloges at runtime allowing users to access again the data in the lake from a tool of their choice also allowing us the capability to be rendered neutral.
Um and then finally we have a whole bunch of management services that I talked about earlier which includes everything from creating uh structure in the lake provisioning access to the data in the lake. We also use some key capabilities like vacuum for file optimization so that we don't have a whole bunch of small bite-sized files. And this is how again the combination of using several key features from data bricks uh allows us the ability to keep our data in open format in the lake and be able to consume from various consumption tools.
With that I'll pass it over to who will talk a little more about our implementation of these services. Awesome. Thank you. So some of the benefits that this architecture g this this architecture gives us quite a few benefits but I'm just going to highlight kind of six of them that you know we use in production today and then like I mentioned before going to talk about some of those use cases but the first one being um one these cataloges enable producers to leverage extractions to reduce disruption for consumers.
So we'll talk through that. Um we it enables thirdparty data sharing without copying data. Uh we are also enabling automation of data asset creation. So all of the stuff in Unity is is automated from uh when producers create data pipelines and all that we'll talk through. We also then have tooling that ensures that data as it gets into the lake matches that stuff in the catalog from a pipeline governance perspective.
Um last but not least or or two more actually the cap we also have capabilities there that allow central platforms to manage data sets on behalf of users. So there's a lot of capabilities that we perform that allow LOBS just to function focus on business logic. And then last but not least we we do are starting to leverage advanced access policies to limit data proliferation. So the first one with uh abstraction. So in a in a kind of legacy data lake you know this was a very common problem that you would see like you would have a data pipeline as version one.
You know the data consumers would be using the data. they'd actually have, you know, paths in their code with S3 and then, hey, look, I I made a big change and now I had to do a breaking change and move my location. This um, you know, there's kind of two ways it would go and it was normally the latter. First, like the producers could try to understand all their consumers and work with them and coordinate deployments and try to make sure that this all happens in a good way.
What the other thing that could happen is the producer just move the location and then the different consumers be like, "Hey, look, I'm missing data." And they'd find out a few days later. So, um that you know, without a catalog, this this and especially with data lakes at scale, this gets very complicated and um you know, it just it really kind of hurts consumption and makes people lose trust in the data. when you now add a catalog in place, this can give the producer many more kind of tools to limit that disruption.
So one um you can leverage views, you can when there are breaking changes, you could have views that still point to different location and use unions. Uh you could also use delta schema evolution to keep the location the same. So, so like the one big one that we're starting to use today is just leveraging cataloges to kind of abstract from the from the S3 location directly. This next one with third party data sharing is is a really big one that we're starting to leverage here at Capital One.
So, I mean data sharing isn't new. We've all been sharing data for some time and we, you know, traditionally have done that by shipping files. So you ship big CSVs, epsidic files, whatever it is. Um, which worked, but I think now there's a few things coming that's going to make that not work. One, data volumes are getting so big, like if you wanted to ship data, it's not even feasely possible sometimes. So, and then on top of that, usually when you need data, you don't always need all of the company's data.
So even if you ship it there, then you'll find the use case is only using a part of it, which you know, It's not great to just ship all of your data somewhere. So, um, with now using Unity catalog to do data sharing, th this mitigates a lot of all of this challenges. So, one, you can support as large data volume as you want. going to walk through a use case later where we have a marketing uh, third party that we're using that we want to have them access all of our customer data at Capital One, but for marketing purposes, which I don't think we could package that up and ship it somewhere.
That would be tough. Um this the other benefits of this too is that you do get real-time data access. So you don't have to wait for the copy and have data pipelines that are delivering it. Uh you do get much more enhanced data consistency and visibility. So you won't have as as these pipelines evolve different kind of files laying around with different schemas. Um and you'll also be able to see you know who accessed it, when they accessed it, stuff like that through through audit logging.
Um and then you'll also get just improved overall data governance. You get you capture lineage from this from these capabilities. You capture what transforms the vendors are doing on your behalf. Um all that type of stuff you get and you'll hear this a lot like so this ends up being less overall complex. I mean all of these things you'll see or hope you see that reducing complexity in our environment. So Kel touched on this quite a bit and then we're going to kind of do a deeper dive into this here.
So our business catalog like she said is the front door to data and when you want to create any data at capital one you're going through this process. So you and this catalog is something that we have built. So like there are it's a little bit different than like a traditional you know elation or business catalog like that because we've actually built orchestration capabilities into this that do catalog federation on behalf of the data producers.
So once they register that data, the actual first step that we do is we assign a a PDS to it that assigns ownership to it and then it makes somebody responsible for it. So that's actually assigned to a business unit. They, you know, if we have findings in that data that need to be remediated, if it starts costing a lot, whatever it is, we have a point of contact now that we can go say, "Hey, please go fix this if you still want it."
So somebody's responsible for it. And then the other big part of that too is we enable users across capital one. Everybody can access data and get access to data without talking to the central platform team. So you can go to that catalog we mentioned request access and then the actual owner of the data set is giving access. It's not a platform team. It's not central team any of that stuff. And then you know like we said this all gets pushed into unity and then unity catalog now is first of all you can be ensured that it's matches exactly what's in our business catalog.
So as users went in there to discover stuff and search when they go into unity and start using notebook it's all consistent. Um and then we also have even more automation configured in this for pipelines to for system access. And the next slide here we'll talk about actually talk about those data pipelines. So we did all that work to make sure we have the data registered. We registered data quality rules and now when the data pipelines are running we go one step further to even ensure that hey the data that's coming through these pipes matches what was registered.
So um we do a few things with this. One is we enforce the schema and the data types and all that stuff. We we have data quality rules that are um calculated in flight and then we also do a few um transformations here. So the tokenization thing is what I mentioned before. This is the databol thing the capital one software is released like that came from here. So when you configure in that catalog hey this is needs to be tokenized.
We handle it on behalf of everybody. Um and then like we said users can go search and they get access to this and it all just works out of the box. Um so we're going to switch gears a little bit now to another part of the platform that this enables. Um so like in that first slide we showed that there was like platform things and LOV things. Um before a lot of the stuff on the right a lot of users just had to do themselves like if they wanted to turn on intelligent tearing if they wanted to do data remediations or any of those things all the yellow bees would have to do that on their data themselves and not handle it centrally as we've been kind of rolling out our new platform um we've been shifting some of that stuff to the right so you can see now that like our lake for example has storage tiering turned on on behalf of users so they don't have to worry about like cost getting out of control there.
Um same with retention. So if if if business users need to configure retention policies on data, we have regulatory stuff like that handle it for them. Um we handle data remediation. We're and that list is continuing to grow on stuff we're doing centrally because we really want to enable our our business users to build what they need to build and not have to be burdened with some of the stuff that you know that we could handle on their behalf. and you know the catalog is super important here because of transaction locking and stuff and stuff like that like we it's really tough to uh do a lot of these things on users behalf you break their data pipelines.
Um so transaction locking there is is another big key feature helps catalogs. Uh all right now last but not least um we we are starting to use advanced access policies for uh data access. So you can see a before and after here. On the left side is kind of how you would handle making curated data sets for a bunch of different consumers in a traditional data lake. Um now on the right side we can keep the one copy of the data and use aback rback and fine grain access controls to make those three different curated data sets without having make to make those copies of the data.
And we we have actually have a use case in the next part. We're gonna walk you through how we how this helps our partnerships business. But um yeah, so these are some of the things that we're doing with cataloges at Capital One. I'll pass it back to here. So uh you heard talk a little bit about uh really how data um from the business catalog and I from business catalog is then orchestrated into our runtime cataloges.
So now think of you're a data consumer. you want access to a certain data set and then you want to go into data bricks use a notebook and do run a whole bunch of queries. All right, swapsy. Uh what you would do is you would go to our business catalog which is again a self-service UI uh capability. You go in there, you search for the data set that you want access to. Once you search for that data set based on at the time of registration and you know we talked about the sensitivity indicators etc whether you have access to the data set or not you apply for access to the data set if you don't already have it.
Then it goes to a uh data steward and this is a workflow. The data steward looks at your role um and looks at your entitlements in our entitlement system and then approves or disapproves your access. Say they approve your access at that point. Now when this data set was originally created in the business catalog if you remember we talked about creating an orchestration capability that created a corresponding entry in the technical cataloges.
So there was a corresponding entry created in Unity catalog as part of the data set creation as an eventbased system. And so now within Unity catalog, we have a mapping of this user has access to these tables and is able to do these types of things. And this is all within the properties within Unity Catalog. So now me as a user could just go in and provision a creator notebook and I'm able to see all of my tables within Unity Catalog.
And so this is a very normal regular use case where our users will come in and do different types of experimentation, analysis, exploratory jobs, all of that and u the views are now available in your data. Uh the next business use case again u and then touched upon this on how we implement fine grain access control policy. So, one of the lines of business at Capital One is a credit card business. Um, and you probably all are familiar with what's in your wallet, right?
So, now as part of the credit card business, we issue private label credit cards as well. What this means is you could have a Kohl's card and you could go and shop with that particular card at Kohl's and you have rewards from there or or any of those partnership cards. So um Capital One's partnership platforms car partnership platforms have built data pipelines that will allow you to configure and onboard these partners under the platform.
So once they are onboarded, there are regulatory requirements that do not allow you to see other partnership data. And so in order to enforce that type of regulatory requirement, we are leveraging unity catalog and particularly fine gra access control where we have views um that sit on top of these tables and we abstract the underlying tables from the users and these users now have specific data that they can access that is very specific to the partnership data that they should be able to see.
So this is another implementation of how we are really using binary frame access control and unity catalog to provision partnership specific data within our credit card business. And one other thing with that one is like if you think about if we didn't have Unity catalog here in fine grain access control that turns into new pipeline for every partner we on board whole bunch of work for those teams and this is this is just really I mean they want to onboard partners fast.
So this is really just helps simplify that speed that up and stuff like that. Um and then uh again touched upon this a little bit. The third use case we want to highlight here is um you you read about the zero copy EDL and data sharing. So again within our marketing teams and marketing platform there is often a need to um create some specific customer segmentation and leverage a um third party like hightouch in this case that actually will create specific audience segmentation files.
Um and this is based on campaigns that the card uh team would come up with either for exploratory AB testing etc to understand uh how how customers are reacting to certain offers and in order to do this they need to share certain specific customer data and then they use this third-party marketing uh platform hightouch that will read or access this data come up with certain audience segmentation And then we are able to they are able to send data to emails through this audience through Salesforce or some other technology and ultimately we are able to bring get any of the outcomes from that email campaign back into Capital One.
And we have instrumented all of this through a combination of unity catalog specific access controls within UC and our one lake and now we are integrating with a third-party hightouch to do marketing campaigns uh for a marketing platform. uh we haven't fully enabled um delta sharing yet within our platform but that will be like the next evolution of this uh architecture. So with that I want to leave you all um with three key takeaways from Gemini based on this.
The first one is to have a catalog strategy for your data pa platform. And this again comes down to how are you going to ensure that your company has source of truth. You're able to adapt to the evolving consumption landscape and federate the data from technical data business catalog to technical catalog and able to power your lakehouse strategy. The second key takeaway is self-service and the ability for data both data producers and data consumers to find the data, access the data, manage the data and largely be self-sufficient without involving central platform teams.
And the third thing I would say most important is building data management controls within the life cycle. You heard talk a little bit about how we have SDKs and we have governance checks built within the SDKs. We also have other types of data management activities I talked about such as life cycle policies, data quality checks outside um life uh intelligent hearing for managing uh data in the lake. And so all of these data management activities really control the entire life cycle of data. everything from registration to pipeline to actually making sure your data is in good standing and trustworthy.
And so with that uh we'll conclude the session and I want to open up for any more questions.

Learn more about the Databricks Data and AI platform.