Transforming Data Governance for Multimodal Data at Amgen With Databricks

Summary

• Multimodal governance challenge: Amgen is advancing its Enterprise Data Fabric to securely manage sensitive multimodal data including imaging and research information across various formats on the Databricks Data and AI platform.
• Unified approach: Extending Databricks Unity Catalog governance to unstructured multimodal data enables centralized access control, enhanced traceability, consistent access standards, and comprehensive auditing across both structured and unstructured data.
• GenAI standardization: This video shows how implementing comprehensive multimodal data governance enables Amgen to standardize its GenAI projects on the Databricks Data and AI platform while maintaining security and compliance for sensitive biopharmaceutical data.

Transforming Data Governance for Multimodal Data at Amgen With Databricks

Talk by Jaison Dominic, Senior Manager Data & Analytics, Amgen ; Jinesh Kunjumon, Sr Data Engineer, AMGEN
Amgen is advancing its Enterprise Data Fabric to securely manage sensitive multimodal data, such as imaging and research data, across formats.Databricks is already the de facto standard for governance on structured data, and Amgen seeks to extend it for unstructured multi modal data too. This approach will also allow Amgen to standardize its GenAI projects on Databricks. Key priorities include: Centralized data access: establishing a unified, secure access control system Enhanced traceability: implementing detailed processes for transparency and accountability Consistent access standards: ensuring uniform data access privilege experience User support: providing flexible access for diverse stakeholders Comprehensive auditing: enabling thorough permission audits and data usage tracking Learn strategies for implementing a comprehensive multimodal data governance framework using Databricks, as we share our experience on standardizing data governance for GenAI use cases.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

Why is multimodal data governance challenging?

Multimodal data includes structured tables alongside unstructured formats like imaging and research documents. Governing all these data types consistently while maintaining security is complex, especially in regulated industries like biopharmaceuticals.

How does Databricks extend governance to unstructured data?

This video demonstrates how Unity Catalog can be extended beyond structured tables to manage unstructured multimodal data. This provides centralized access control, traceability, and auditing across your entire data estate on the Databricks Data and AI platform.

How do I implement multimodal data governance?

This video shows Amgen's approach using Unity Catalog to extend governance from structured data to multimodal data like images and research files. The strategy involves organizing data at clear granularity levels to avoid permission inheritance issues, using Databricks Volumes for unstructured data, and applying consistent role-based access controls across the Databricks Data and AI platform. This enables governance of both structured and unstructured data under one enterprise data fabric.

What are the key priorities for Amgen's governance framework?

Priorities include centralized secure access, detailed traceability and transparency, consistent access standards across data types, user support for diverse stakeholders, and comprehensive auditing for compliance tracking.

How does this support GenAI standardization?

By standardizing governance across all data types, Amgen can safely leverage its multimodal data for GenAI projects. The Databricks Data and AI platform ensures these projects meet regulatory requirements while maintaining data security.

Full transcript

Hello everybody. Good afternoon. Delighted to be here to talk about this interesting topic. I'm the product lead for enterprise data fabric platforms at Amjen. And with me is He is u our product architect. Let me take a minute to talk about Amjen. Amjen is a pioneer in biofarmaceutical industry and remain a leader today. Biofarmaceutical industry make meaningful impact on patients through the intersection of chemistry, biology, engineering.
And at Amjen, our mission is simple to serve patients and we try to do that with u medicines that treat serious illnesses particularly with medicines that include proteins, monoconal antibodies etc. So producing safe reliable macroolelecules like uh we produce is a complex process and it it has stringent guidelines or regulations from the drug discovery till we get the drug to the patient. So when we talk about bio uh technology it is it is a process by which medicines are produced in living cells and that's where it is different from u traditional pharmaceuticals.
We produce proteins monoconal antibodies on a living cell and that is what is finally delivered to a patient. So it is not just the product that we make. The process is also very complex and it is fully regulated each step of the way and not just the process the data generated at each step of the way is also regulated and that's why we have the we have this topic today. So um let me talk specifically about few things about Amjen. uh we focus on three six uh on six uh therapeutic major therapeutic areas.
Um we serve uh more than uh we directly serve patients uh with serious illness in more than 100 countries and um you know some of the our products are on the screen there. uh we have uh kyprolis for u for the oncology uh therapeutic area repatha for the cardiovascular for inflammation uh prolia for bone health to name a few uh let's take a quick look at our governance journey so far we started with our enterprise data lake 10 years back uh with cloudera onrem then we in uh 2017 2018 18 we migrated to AWS and um AWS for storage and infrastructure and data bricks for compute.
So AWS glue was driving our uh you know that was our catalog driving our governance and we had um our uh datab bricks high metas store integrated with AWS glue and um we tried a third party tool for our fine grain data security but it didn't really work out for us. So when in 2022 when datab bricks announced unity catalog we went allin with unity catalog as our backbone for our data security and governance. And last year when data bricks announced uh datab bricks volumes we expanded unity catalog to uh cover the data security and governance for multimodel data as well.
Um what you see on the screen is a high level landscape of our enterprise um uh you know uh data ecosystem as you see in the middle there that is the enterprise data fabric that's the product team we are part of. So uh uh using the technologies that you tech stack that you see there, we build services which are as part of the enterprise data fabric platform which is used by other platforms like um you know for uh injection into staging uh for consumption for data quality uh master data and also by our AI and data science workbench uh uses uh our services for infrastructure prop provisioning, data access and security.
And uh we have a internal data marketplace which is like the front end for uh enterprise data fabric um where users have the self-service ability to search for a data and also uh request for access for that data. So uh as I was mentioning in the previous slide, we tried to solve the three V's of that traditional uh enterprise data lake. The volume, velocity and variety, right? Volume we solved through uh AWS. Velocity we are solving through data bricks and uh technologies like prophecy, low code technologies.
And the last part of that was variety where we wanted to expand the same kind of uh fine grain data security we have in structured data expanded now to include the multimodel data. So as you see on the screen there uh the data is generated by the various business functions and our core customer segment uh customers are data engineers and data scientists. So our goal is to provide that crossf functional access to data right make that uh data democrat democratized access to data with the right governance and also with the right uh auditing capabilities.
So uh coming to the factors which are which were driving uh the driving factors for our multimodel strategy like many of you might have experienced last year uh there was a surge of AI related projects and applications which required the same kind of stringent regulations on data access for the multimodel data both structured and unstructured data and also uh with the new AI capabilities. We have uh you know we have a trove of uh image data which we also saw the potential for you know getting more insights using the new AI capabilities and the new technologies which are available.
So uh those were the driving factors behind it and our um main challenge uh was to provide like in the previous slide right provide that crossf functional access to unstructured data as well with the proper governance and with a fine grain auditability meaning at any given point of time we can tell that who has access to what and who accessed what. So now will walk you through the uh you know our solution architecture and the design principles we followed in this deployment.
Thank you. Thank you. Hello everyone. I will be covering the technical aspects of multimodel security and governance framework. So let's jump right into the core principles for our framework. First is the centralized governance. By unifying all the data and data type under the umbrella of enterprise data fabric, we are breaking the data silos. We are enabling the transparency and the collaboration across the organization.
Equally important is how you're organizing the data. For structured data, it is pretty straightforward. You are setting the data permission at catalog, schema, table. Even if you want to do more further fine grain access control, you can set the data permission at row or column level. But when you're dealing with multimodel data like images, sensing feed, you have to have a clear strategy how the data is organized and at what level or granularity you want to set the data permission.
You may ask why that is important. For example, you have a root folder and you want to set data permission at root folder level and you want to now set a different set of data permission at subfolder level. So by default whatever the permission that you have set at the root folder level will be inherited by the subfolder level too. So breaking this inheritance of data permission is one issue. Now from datab bricks point of view if you try to create a volume which has already been created at root folder level you won't be even able to create a volume at subfolder level.
Matter of fact if you try to create a volume which is overlapping with the existing volumes or existing external table you won't be able to create that. And that's why you need to be intentional how you are organizing your data and it need to match with the business goal and the uh security needs. If you are not getting this right from the start, you are risking confusion and future rework. So upfront organization is uh non-negotiable.
Standardization is equally important by following unifying naming conventions, standard metadata practices. It help us to set rules at scale and avoid chaos down the line. Data ownership and accountability. Even before the data lands in the centralized governance, we make sure that we identified who is the data owner and who is the data steward. So we can do the management of permission and data enrichment data accessibility.
Once the data has been provisioned to the authorized user, we are making sure that the governed data access should be seamless, fast and simple and secure. Transparency drives trust. So auditability is another cornerstone. So we are making sure that we know that how the access has been provided to who what and when. But we are not just mirroring just tracking the permission. We are also actually monitoring the usage to make sure that there is a full visibility that how the U data is being used which is required for the stringent compliance and governance process that we follow in.
Finally, we have a framework which is built to adapt. So as the datab bricks and the broader tech system ecosystem brings new solution, we will be able to integrate it seamlessly with our framework without disrupting our platform. So now let's see how these principles turn into action. Let me walk you through our reference architecture and what makes it tick. So it all starts with data producer could be generating anything from engine internal data public data uh proprietary data confidential and even the data which has been shared to us by the external data partners.
So you can understand we are dealing with all different data types and it can be text, images, PDF, DICOM files. So you name it, we will be using those of uh those data types. To move all this data from point A to point B, we have built a robust pipeline which focuses on fast and efficient data transfer. Here we are leveraging the data bricks data pipelines, AWS data sync for moving high-speed data transfer. This utility is capable of transferring the data in incremental data injection or delta uh copy or it can also do the full load.
This uh utility is also capable of metadata extraction. The metadata extractor leverages the image processing packages to extract metadata from files like uh images, diccom images and finally and finally funnel this metadata into the centralized metadata store which is used for the data searchability and data discoverability. The framework also support referencing the data especially if you're dealing with massive data which is in pabytes.
We identify and we work very closely with the business also to make sure that we are aligned how the data is organized and what granularity they want to set the data permission and at this point we also identify who will be the data owner to which sets the data ownership clarity. Now let's move to the data consumers. Data consumers can be data engineers, domain leads, AI developers who want to consume this data. They can search for this data by the internal data marketplace which leverages the centralized metadata store.
Once they identified what they need for their work, they can submit a data access request. The approver can decide whether to approve or deny the access which if the request has been approved, it triggers the centralized governance framework. The centralized governance framework do the needed validations and once everything is successfully validated it provisions the access to the users by the execution datab bricks volume ACL or S3 access grand ACL.
Our platform do support both datab bricks users and non-databicks users. For supporting non-databicks users we are leveraging AWS S3 access grand ACL. Now let's see that how the data how we can enable the centralized governance system on the data and how the data consumers can actually get access to this data. It all begins with the data owner bringing a new data which they want to enable the centralized governance on the requ the process is pretty transparent and simple.
They can submit an request with some key details like how the data what the source of the data how it need to be organized what granularity you want to set the permission and importantly who will be the data owner. This is a completely streamline automated process. It make the needed entries to the metadata and security tables and finally onboard the data to the centralized governance and notify the data owner that it has been successfully onboarded.
Once the data has been onboarded, the data consumers can search and discover this data by the centralized metadata store. They can submit a data access request and it basically triggers our streamlined data approval workflow. The data owner get notified and the data owner get all the details and they can review actually the fine grain details how the request has been submitted and by whom and they can decide whether to approve or deny the request.
If the request is approved the centralized governance system get initiated. It validate the request to make sure that all the requesttor or the consumers are actually active engine users. Whoever approved the request is actually the legit data owner. And if this users are not already in databicks, it will also onboard these users automatically to the uh datab bricks account. It provisions the unity catalog entities like external locations volumes for the request.
And finally, it provisions the access to the data consumer by the execution of volume ACL as approved by the data owner. Then it transparency is built in in each and every action which is carried out across the uh data access request workflow. It make sure that we are able to capture who approved the request to what and when and at what level of permission. And once this has been set and all logs have been captured in the metadata and security table, the data consumer is notified that their access is ready.
Now the data consumers can actually go to their respective datab bricks environment and do the research and analysis through the volumes securely which has been provisioned to them. The transparency and the auditing is not stopping there. We are also actually monitoring the actual usage by the users. We are there we are leveraging the datab bricks unity catalog system tables. It's actually help us to make sure that this data is actually right users are consuming the right data.
Now let's see that once the data access is been provisioned to the data consumers how will be the user experience. We have a data which is program and study and we have two user type domain lead and researcher. The domain lead and this data is been organized into volumes at study level. Domain lead is been provisioned access to all the programs and studies whereas the researchers is only provided access to program one and study one.
When when the uh domain lead log into their respective data bricks environment they can they can see that all the volumes as it's shown in the screenshot they can see all the studies and uh programs that have been asso allocated to them. Whereas when the researcher login is only able to see the program one and study one which has been provisioned to uh the user by the centralized governance system. Here a point to note the access has been provisioned to the user entity not to a not to a compute or to a IM role and this is actually give us a huge flexibility and auditability point of view because now we can drill down how each and every user is actually consuming this data and utilizing this data and we get pretty good amount of usage details through the Unity catalog system tables.
So uh basically finally in summary we have built a framework which is not only secure and scalable but easily accessible and fully auditable. On that note that's our solution. Handing. Thanks. Um so the biggest benefit that we got from that deployment is about the ease of use for the users right like now the users can get the both the structured and unstructured data possibly under a single catalog under a single schema.
So the searchability is easier. So is the request for getting access for data is easier and as I was mentioning before about the stringent regulations. So that's another key uh benefit from this deployment right whether um we can we have that fine grain auditability now available for both structured and unstructured data. So we at any given point we can find out who has access to what or who accessed uh what. So um one of the key lessons learned was uh mentioning about that right in the beginning that you need to have a blueprint of where you want to have access for this unstructured data whether in a parent child relationship whether it's at the child level or at the object level right depending on that there may be some tech technical debt on the product we are working with data bricks on uh some of that uh as part of their road map.
So uh that's that's also a a key lessons learned for us and looking forward um we are further strengthening our uh our fine grain data security using attribute based access control Aback and arbback uh which uh data bricks recently announced and we are uh we are uh expanding our fine grain uh uh capability using that uh in our production environment we enabled uh data bricks delta sharing. So it is currently only for the structured data.
We wanted to expand that to unstructured data that's in our road map. Um then uh we also enabled uh uh the lakehouse federation in our production environment. So both lakehouse federation and delta sharing uh is uh reducing the dependency to ETL and you know in the increasing the speed to market for data. So bringing those uh cataloges right the delta sharing catalog and the foreign catalog from the lakehouse federation integrating that with our central catalog is is a is a a core focus area for us and that's our presentation for today.
Thank you for thank you all for showing up this late in the day and uh I think you have good things at meet and greet. So thanks again. Thank you.

Learn more about the Databricks Data and AI platform.