Sponsored by: AWS | Breaking Data Silos with Multi-Engine Catalog Federation

Summary

Mastercard federated Databricks Unity Catalog to AWS Glue, enabling Apache Spark engines on Amazon EMR to consume data in Apache Iceberg tables registered in Unity Catalog—achieving a single governance source of truth while allowing domain teams to choose Spark, Athena, or Snowflake as their compute engine without vendor lock-in.
The architecture is built on two tenets—open data formats (Apache Iceberg) and no engine lock-in—which allow Mastercard's domain teams to evolve independently while sharing a centralized governance layer that provides ACID reliability and multi-engine interoperability.
A zone architecture of raw, clean, and curated data layers with service principal authentication underpins the federation, turning what was previously a strategic constraint of fragmentation into a governed, future-proof architecture for machine learning, streaming, and data sharing.

Chapters

FAQs

How does Unity Catalog federation with AWS Glue work?

Unity Catalog to AWS Glue federation works by syncing table metadata registered in Databricks Unity Catalog into AWS Glue's catalog, allowing Apache Spark engines on Amazon EMR to query Apache Iceberg tables as if they were native Glue tables. Service principal authentication manages cross-service identity so access governance remains centralized in Unity Catalog even when queries originate from AWS services.

What is the benefit of using Apache Iceberg for multi-engine data architectures?

Apache Iceberg's open table format allows multiple compute engines—including Spark on Databricks, Athena, and Snowflake—to read and write the same tables with ACID reliability and without requiring each engine to maintain its own data copy. Mastercard's architecture uses this property to give domain teams compute engine freedom while ensuring consistent, governed data across the enterprise.

How did Mastercard solve data governance fragmentation?

Mastercard adopted a balanced mesh architecture that centralizes governance in Databricks Unity Catalog while allowing domain teams to use their preferred compute engines autonomously. By treating fragmentation as a strategic constraint rather than just a technical problem, they achieved a single governance source of truth working across Spark, Athena, and Snowflake without forcing teams to abandon their existing investments.

What is domain-driven layered lakehouse design?

Domain-driven layered lakehouse design, as practiced by Mastercard, organizes data into zone layers—raw, clean, and curated—where each zone has specific data quality guarantees and access patterns. Domain teams own and manage data within their zone while the platform enforces governance policies across zones, enabling both autonomy and consistency at enterprise scale.

Full transcript

[00:16] Welcome everyone. Uh good afternoon. Thanks for joining us here at the Data and AI Summit from Databricks. How many of you are attending the Data and AI Summit for the very first time? Oh, that's a sizable number.
[00:30] How many of you are into heavily into analytics or AI? How many of you are really really curious about multi-engine catalogs? >> Yeah.
[00:42] >> Thank you. Okay. Let's get started. Today we are going to talk about breaking data silos with multi-engine catalog federation. So
[00:54] the main presenter would be Ranju. >> Yeah. Hi everyone. I'm Ranju Kumari, uh data uh director of software engineering in Mastercard, and I'm presenting today with Saba.
[01:06] >> I'm Saba Parmeswaran, principal solution architect with AWS. Helping customers on their journey through AI and data. So let's go into the agenda.
[01:18] So what we are going to talk about? We are going to talk about the challenges that typical large organization faces in terms of data, how to have the data shareable, how to have governance around it, how to
[01:30] interoperate between different data organizations or departments or different workloads. I I'm pretty sure most of you might face the same set of challenges. Then we'll talk about like what can be a guiding
[01:43] philosophy in terms of building a solution that really works and delivers for you. We'll kind of go a little more deeper into the actual solution architecture. What are the nuts and bolts that were used to kind of build that architecture?
[01:56] Then we kind of talk about the business results of those and the lessons learned. To start off, a large organization like MasterCard might be facing Okay, on one side they actually have
[02:09] tons of data pouring in and they have a lot of different organizations working on different features or priorities or functionality. So, can you talk us through the data challenges that MasterCard faces? Thank you.
[02:21] >> Sure. MasterCard operate in a complex multi-business environment where team work across different platform, data domains, and tools in their own way engineering standard and velocity.
[02:35] >>
[02:49] or customer-facing. Over the time teams have invested a lot in modern modern technologies or platform to fulfill their business requirement,
[03:01] right? And especially for analytics, AI use cases, or or in today's world any any use cases which is on agents. This These choices which they made to grow or
[03:14] build their use cases drive real value, but they evolve locally or independently. As a result, fragmentation is no more a technical challenge. It become a strategical
[03:27] constraint. It made it harder to apply a centralized governance, cost accountability, operational um operational
[03:40] observability to be a centralized process. It It make it more harder when data need to be bring from different domain and fulfill the use cases. It add more cost
[03:53] or operational risk when you are bringing data from different domain to fulfill one use case. >> Okay. I think like how many of you have like different departments strategizing or kind of
[04:05] building things on different tech stacks. Like one might have a data brick, one might have snowflake, one might have EMR, one might have something else. And it's always a challenge. How do I get the data to flow freely between the
[04:18] teams without the duplicating the same set of data. How do I get through the access controls? How do I make sure I can actually discover the data and use it the right way in the fastest way with all the different
[04:31] pieces as in governance or security and standards. I think that's the challenge. I'm pretty sure most of you like how many of you are from the fintech?
[04:43] Healthcare? Or like how many of you actually have these challenges day-to-day? Okay, let's go into like what is like your guiding philosophy in towards kind of solving this problem.
[04:57] >> Yeah, right. So, the the way you all are facing the issue, same issue was with us. Um, this led to the journey of building an enterprise data foundation to enable interoperability with trust at its core. The goal was not simply
[05:10] integration, um, but to treat data as a product, make it discoverable, shareable, and reusable, um, under strong governance, clear controls, and accountability without slowing team down
[05:22] or centralizing whole thing. >> So, let's uh, talk a little more about what do you want in the final data platform? >> Sure. So, what we have been what what it
[05:35] led us to is a balanced mesh. Um, the idea is to harmonize centralized with decentralized data ownership. It's not a pure data mesh, um, where where everything is fully
[05:48] federated or everything is traditionally centralized like a data warehouse, it's a deliberate balance between federation and centralization. We foster structure between independence and standardization.
[06:01] Product team have their independence over their data, but core capabilities like security, governance, catalog management are centralized. And here's the strategic unlock. This balance makes
[06:15] mesh built on hub and a spoke model. So, enable interconnectness of data assets that open up new data product and commercialization opportunities.
[06:27] Data was previously trapped in one silos, now it is available to be consumed by discovered by different domains and securely at a scale.
[06:40] >> So, what is this domain driven layered lakehouse? >> Have we jumped? >> Yes. >> Okay. Sure. So, as I spoke
[06:52] just just mentioned hub and a spoke model. So, you you see in this architecture there is a central hub hub services which like the centralized services or all the support we build we we serve it as a services and then
[07:05] autonomous is spoke. What is really the centralized hub? It provide the foundation that enable product team or data team or a spoke team to do their work independently.
[07:18] What we have in centralized system, we have security baked in into it. Orchestration template which reusable patterns so so that team can use it they don't use something random.
[07:31] The most important thing iceberg table open format that made it possible to you know have this hub and a spoke model and I will talk more how it open up the opportunity of interoperability between
[07:43] different catalog. Um, the autonomous is spoke, um, data products and application data team own it. Uh, business rules and product pipeline, they are writing it. They are writing the contract definition
[07:56] which was defined as a contract framework into, um, centralized hub. So, basically centralized hub is providing the service and autonomous spoke is consuming those
[08:08] services. >> So, how many of you have something similar or dissimilar patterns? In terms of data governance, data in management. It doesn't gel with you.
[08:20] Having a common fabric for governance, for security, for access and a open standard, open data standard and then building other layers, services so they can consume the things faster, they can
[08:32] ingest or produce for downstream applications much more faster. So, once you're taking care of the basic plumbing needs, so everything is secure, everything is governed, everything is discoverable.
[08:44] Now, go through the tenets of your I think we kind of saw it but let's actually understand the tenets of your data driven lakehouse. >> Sure. So, you are seeing a lot here, uh, five section but I will I will summarize it
[08:56] into two section. Uh, multi, um, engine operation, uh, uh, optionality with interoperability and governance. So, the the first one, this is critical. We
[09:08] deliberately designed the platform to avoid long-term lock-in. We love Databricks, isn't it? That's why we are here, but we use other computer like EMR, Snowflake, Sagemaker, um, and the platform need to work across all of
[09:21] them. Um, you you don't want to be lock-in, um, and how it is possible, um, the way we achieve this, uh, is through open format, Apache Iceberg, REST catalog, um, and APIs. The second,
[09:35] unified governance, regardless of which engine you are using or or querying the data, Mastercard enterprise level trust and compliance standards apply everywhere. One governance model, many
[09:48] engine, um and and you are you are using it. You don't get to bypass governance just because you are querying from Athena or you are querying from Snowflake or you are querying from Databricks. So, it's
[10:00] across umbrella, same governance apply in this architecture. >> So, that's a that's a path and the vision kind of gel with the audience.
[10:12] Like, why is Iceberg open table format? I think it's at the right time kind of came came up at the right time, so people can start delivering value on top of it because
[10:24] it's easy to consume and it's kind of supported by all the vendors, the main vendors in the data analytics space. And it is it can be the lingua franca for also the AI side of things. Okay, let's talk about the Amazon goals that helps build this kind of
[10:37] architecture, the vision. >> Sure. So, again, the goal is one catalog, one copy of data, many compute engine. Um and what make it possible? Apache Iceberg. Um open format, no vendor
[10:51] lock-in. That's the goal here. And works with a Spark, Athena, Snowflake, give us acid transaction, um time travel, schema evaluation. It's all there. You heard Ali today. And optimized for large-scale
[11:04] data set on S3. So, it support the rest catalog protocol and which is the glue of this whole infrastructure. And I will talk more on the architecture, how this is possible, and how you achieve it, but that's the glue. Um Unity catalog rest
[11:18] bridges, this is the centralized um centerpiece, right? EMR um Snowflake other other any other um engine, they they talk communicate through this um
[11:30] um, to, you know, catalog federation or those endpoint to you talk through the data which is governed in a centralized manner. There are few more, like infrastructure and all, definitely it come handy. Um, I am role, all the setup
[11:42] you done do behind the scene or plumbing to make this front working. >> So, like we understand that Iceberg is actually enabling us to kind of have a open data format that can be read by others
[11:56] and who are is a department or the the org that is owning the data can continue to ingest the data, transform the data, enrich the data, while others can discover it using the Unity Catalog and Glue Federation. So, let's talk about
[12:08] the entire architecture end to end. >> Here you go. So, this is the architecture. This is what make us achieve what we have today as an uh, interoperable system. It is
[12:21] divided into three zones. The first zone is the hydration zone. So, data coming from on prem, coming from customer, it land in there in a parquet format and um, it's a raw data. You can say it's a raw
[12:34] data. Then it come the second the zone one. In zone one, what you are seeing is um, raw, cleans, and then you make the business ready gold product. Um,
[12:47] we are using Databricks heavily here um, to do the transformation and whatnot. We have other options like EMR uh, and all, we will discuss. Um, underlying this, if you see zone and
[13:00] two, uh, all stored um, all the data which is stored in here is using Iceberg format and we apply data quality checks, controls, or contracts uh, and metadata
[13:12] management here in this layer two. And importantly, data that is already business ready can bypass that zone one and you see a line which is going to directly to zone two in a consumption.
[13:25] Um, the magic, the glue here is the third layer. You see Unity Catalog and Glue Catalog sitting in the between zone two and zone three, and that's what enabled the interoperability.
[13:38] Unity Catalog is the authoritative catalog in our case, and what we do is Unity Catalog federate with Glue Catalog, and I will talk more how we do that federation. And what it enable is that
[13:51] consumption layer where you are seeing multiple compute um which is like SageMaker, you are on Athena, Glue, wherever you are, right? You are accessing the same data, which is
[14:03] governed. Like the data layer is still same, right? You see zone one, but you have optionality on the consumption layer. And user who are on consumption layer are not aware like what's happening behind the scene, but it is
[14:15] available to them. It's look like a native solution, right? Um >>
[14:28] Airflow, monitoring, all that by default the platform compliance which which is the common layer between all these zones. >> Does this solution gel resonate with the view that we are
[14:41] talking about like Mastercard is able to kind of realize a vision using this kind of architecture where in like the data can come in through multiple different sources. There can be different engines that are processing the data. Ultimately, they
[14:54] all kind of save it as Iceberg format. And then using the federations between Glue, Unity Catalog, or others, they are able to kind of understand and explore and start using it. Like a new AI project might say like I want cleanse
[15:06] data that is ready for this so I can actually upsell something to my credit card holders. I had already gone through this risk and validation. I want to kind of start pushing new data, kind of see like,
[15:18] okay, can I serve this new service for my customers? Boom, you basically have the data that is discoverable inside like, "Okay, what do I do? I need to copy it over? I need to go and talk to this particular team, get access, copy
[15:30] it again? Like, what if something change and whatnot? Like, I have to take care of the sync up and whatnot." So, all these are kind of solved through this kind of integrated solution kind of
[15:42] tied down to Iceberg as an open table format and using the federation between different sources and services. Okay, let's go into a little more about how the federation actually works under
[15:54] the covers between Unity Catalog and Glue. >> Right. So, that's the layer between zone one and zone two. And what are all the components? Um
[16:06] I get most question on this slide because this is the technical slide and the components are Databricks Unity Catalog because that's the core catalog for us or the authoritative catalog. And then how you authenticate between Databricks and Glue
[16:20] or the your other catalog. And then definitely the connection, right? Glue provide a connection and you build that connection. If you have your Glue Catalog and your Databricks Unity Catalog in same AWS environment, you're
[16:34] good. You're connecting on a private network. But if your AWS um your Unity Catalog is sitting in another AWS account and your Glue is in another account, maybe an application account
[16:46] you are running in another AWS, there is an opportunity or or this connection give you private link to which you can use to connect privately between these two AWS account and and you know, do the
[16:58] catalog federation. And the the last layer is the consumption layer where you're using any consumption here. We are showing an example of Athena which is consuming it. There are more component it to it which
[17:10] is like how you authenticate. You use service principal to authenticate. Then you do the permission management or least permission management in Unity Catalog. Um And then you enable the OAuth, which is
[17:23] like short-term token you you create, which is 60 minute or or something like that, and you authenticate through it. The the few of them like I am role and all you all know how it work in AWS. So
[17:35] those are the component which help us enable this flow which you are seeing on the top. >> So I'm pretty sure you must have heard like, "Okay, vendor sold me this stuff, but it's not interoperable with the others. Okay, I have to build in my own
[17:48] layers, the pillars, to exchange tokens. Figure it out like one is a lake formation, one is Unity Catalog. Each one is vending different credentials. How do I interchange between them? Like how do I change the permissions? Like okay, I have read permission on this
[18:01] one. I want to get the equivalent from the other side, right?" How many of you have gone through this friction? Very few. I don't know how others are able to solve this. So we we kind of hear this
[18:14] pain from multiple customers, Mastercard being one. And so we have been working together with Databricks to say like, "How do I interoperate? How do I make the data that is sitting on the glue side viewable by Unity Catalog and vice
[18:26] versa?" So we understand that data and AI no one actually chooses just one silo or or like one vendor to solve everything. There needs to be interoperability. There needs to be this heavy lifting that needs to be removed from the
[18:38] customer. So you are successful and we are also successful in enabling you, right? So that's where we are working towards. I don't know like the previous slide, anyone unders- understood like, "Okay, S3 that is that a special variation of
[18:51] S3 for Iceberg?" Yes. So yeah, S3 table is like a more specialized version of the storage object storage meant for Iceberg. So it's much more performant because it
[19:04] understands that you're not dumping like a video file or a doc file or something else. It is parquet. We can really read it. You can get much more performance. The same way like we have been kind of working towards more optimization, so you get the best speed, the flexibility,
[19:18] the security, the interoperability. Yeah. So, let's look at quickly how the federation works in practice. So, you might actually have data sitting on the Databricks side of the house where
[19:30] it is like governed by Unity Catalogs. So, if someone from Athena or a EMR wants to go and discover the data or view the data in their pipelines, all they have to do is make sure the federation is enabled
[19:42] between the Glue and Unity Catalog, and when you come through, the Unity Catalog kind of surfaces its catalog information onto the Glue side of the house, and then Lake Formation starts reading the credentials that can map to the right read function.
[19:56] Yes. >> Are you seeing any issues with this federation? >> So, it's still kind of work in progress. We have just kind of delivered our side of the deliverables. Like earlier like Lake Formation was like indispensable
[20:08] portion of it when you talk with Glue. So, now we have removed that. We said like, "Okay, just pure IAM, you would be able to integrate." So, Unity Catalog would be able to use a IAM credentials to kind of get the read side of thing.
[20:21] The same way like S3 tables, I don't know how many of you are deep into S3. Like S3 is the straightforward like, "Okay, what are the objects out there? The tables, the prefix." But S3 tables is a little more convoluted or a
[20:33] little more like a hidden link because the metadata is a little different. The actual storage is yet another level. So, we have been working on it. >> What are the benefits of Lake Formation here?
[20:45] >> So, we are kind of working towards all of this. More of a read read way. Yeah. So, this basically enables the right set of consumers, be it either on the AWS side or the Databricks side, to kind of consume the data. So, the federation
[20:58] takes care of it instead of you like, "Okay, how do I map this user to this role?" Everything is kind of taken care for you by the engines. Okay, let's look at a sample pipeline. When a data lands on S3, what what happens under the
[21:11] covers? So, you might actually use Lambda or you might use like auto injection to kind of go through the data discovery. You start processing it. You might do a ETL. You might transform it. And once it is residing on the SC side
[21:23] of the house with in Iceberg format, it's kind of discoverable by anyone else on the other side. A downstream client, it can be like a Athena, it can be EMR, it can be anything or a AI or a Spark engine.
[21:35] Okay. So, now that we have gone through the architecture, so can you talk a little more about the business outcomes? Ultimately, like what did you deliver for your business? >> Sure. Um so, um
[21:49] If you see this this four section, right? So, enterprise scale leverage beyond platform silos. Uh for the first time, data that was logged in one business domain is now discoverable and consumable by other domain without
[22:02] moving it. This is the game changer. Um the the second one is consistent um sorry, faster time to value for our market. One of the example I will take is we had a user who was in one of the
[22:14] other compute engine um and data was getting processed and prepared in data brick. Uh they wanted to use the same data. Their requirement was almost same. Uh but for it was challenge earlier
[22:26] before catalog federation where you had to move the data. That was the option or you again prepare the data and bring it to different um compute. With this solution, that made it easy uh for time to market. They were
[22:39] able to use that data the minute we had the uh federation done, they were able to access the data as it was native to their own compute engine. So, goal is not to move the data. Meet customer
[22:52] where they are. Um >>
[23:07] and hub and spoke model. Um >> So, what about customers who have kind of focused or built on like Delta Lake? So, what is the path for this kind of
[23:19] migration to this particular architecture? >> Sure. So, if you have already data in Delta format and you have processed it in Databricks, Uniformat is a concept in Databricks. You apply Uniformat on your
[23:31] table. You choose the Iceberg format, which is V3 today. You might have heard Ali talking about it, and it convert your um um Delta format table into Iceberg
[23:43] consumable format. So, the minute you run it, it is available to be consumed by an Iceberg engine. >> Okay. So, let's go into the lessons learned and the strategy going forward for you
[23:55] for your own organizations. >> Sure. So, open format are non-negotiable. Apache Iceberg and the REST catalog protocol are the foundation. They are what make multi-engine optionality
[24:08] possible. Data contract are essential. They are the mechanism which build trust between consumer and producer. I'm start with federation. Don't plan for migration.
[24:20] Um Migration is a pain for sure, and try to meet where your customer is, and this solution give you that optionality. Governance has to be baked in, and it should not be an afterthought. Also, and
[24:34] and finally, this is a journey and not a destination. We are still building our platform and trying to get there. >> Oh, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.