Skip to main content

Leveraging Databricks Unity Catalog for Enhanced Data Governance in Unipol

Summary

  • Multi-Company Data Governance: Unity Catalog enables organizations with complex holding structures to segregate data and manage permissions across multiple subsidiaries.
  • Compliance and Access Control: Precise read-and-write permissions tailored to each workgroup ensure sensitive data protection while enabling authorized access.
  • Unified Governance Solution: Unity Catalog provides a single platform for governance across diverse AWS environments and organizational structures.

Leveraging Databricks Unity Catalog for Enhanced Data Governance in Unipol

Watch: Leveraging Databricks Unity Catalog for Enhanced Data Governance in Unipol
Talk by Beniamino Del Pizzo, Data Platform Manager, Unipol S.p.A. ; Giovanni Cinquepalmi, Senior Data Engineer, Data Reply
In the contemporary landscape of data management, organizations are increasingly faced with the challenges of data segregation, governance and permission management, particularly when operating within complex structures such as holding companies with multiple subsidiaries. Unipol comprises seven subsidiary companies, each with a diverse array of workgroups, leading to a cumulative total of multiple operational groups. This intricate organizational structure necessitates a meticulous approach to data management, particularly regarding the segregation of data and the assignment of precise read-and-write permissions tailored to each workgroup. The challenge lies in ensuring that sensitive data remains protected while enabling seamless access for authorized users. This speech wants to demonstrate how Unity Catalog emerges as a pivotal tool in the daily use of the data platform, offering a unified governance solution that supports data management across diverse AWS environments.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What data governance challenges does Unipol face?

Unipol comprises seven subsidiary companies with diverse workgroups, requiring complex data segregation and precise permission management across multiple operational groups.

How does Unity Catalog help with multi-company data segregation?

Unity Catalog emerges as a pivotal governance tool that supports data management across diverse AWS environments and organizational structures, enabling proper segregation while maintaining authorized access.

compliance and data governance solutions?

This video shares how Unipol leveraged the Databricks Data and AI platform Unity Catalog for comprehensive data governance. The platform provides centralized catalog, access controls, and audit trails for compliance. Unipol achieved robust governance across its data platform supporting regulatory requirements.

data governance platforms for multi-cloud?

This video demonstrates the Databricks Data and AI platform multi-cloud governance capabilities through Unipol case study. Unity Catalog provides consistent governance whether data resides on AWS, Azure, or GCP. Organizations maintain single governance policies across all cloud environments.

What are the top rated AI data governance and compliance solutions for enterprise platforms?

This video showcases the Databricks Data and AI platform as a top solution for enterprise data governance and compliance through the Unipol case study. The platform combines cataloging, lineage, access controls, and audit logging. It helps enterprises meet SOX, GDPR, and other regulatory requirements.

Full transcript

Hi everyone, good morning. Welcome to day two of uh the data and AI summit. Um so this is an extremely powerful session on leveraging data bricks unity catalog uh for enhanced data governance at Uniole. U leading the session are and. Um so hope you guys have a great session. Uh and with this um happy to welcome and to talk about their journey with Unity catalog. Thank you. Thank you. Hi everybody and thank you for joining us today.
Uh we are going to show you how we are leveraging datab bricks unity catalog for enhanced data governance in UNIL. Managing data at scale at Unipol. It's more than just volume because it means to deal with the complexity of a multi company group. Now let's have a look at the agenda of the day. We will start with the background about our company and then we will show you key challenges in data governance in the insurance sector.
Then we will have a look at our solution which is based on datab bricks unity catalog. In the end, we will have a look at how we manage data infrastructure at scale with terraform automation and we will have just some final consideration about key outcomes and lesson learn throughout our journey. I am data platform manager at Unipol and with me is It's a senior data engineer at data reply Italy. So, Unipol is an Italian insurance company and we rank it first in Italy in reputation in no life insurance and the sales network.
But among all that we are the European leader in telematics with the highest number of connected vehicles on the road. The structure of our group reflects our organization. So at our core we are an insurance company with a lot of different companies with different business. Uh but we are evolving our um our ecosystems into three key ecosystem that are mobility, welfare and property in order to provide customers with value added services throughout all um throughout the everyday life.
Recently, UNIPOL is introduced a new strategic framework which substantially evolves around four strategic guidelines. Stronger inducer profitability, faster integrating offering model, distribution network, better tech and people skills. Among all the asset that Tuniple is exploiting to evolve our business, data and analytics plays a key role to reach our strategic goals. But what it means to deal with data tunable.
Let's have just a quick snapshot with us on this project. uh we have been uh data reply which is an Italian consulting company uh focused on data solutions data reply has been our a fundamental partner on this experience now let's have a look at data governance challenge in insurance sector when data lacks governance order is just an illusion and that's why data governance is uh a key factor to manage data at scale. We group data governance challenges in three main categories.
Data management, governance and compliance and trust and innovation. With data management, it's clear important in our in our scenario, the integration of a lot of different data sources that cames from a lot of different companies. And then we will have a look at what this means about governance and compliance. You have to take into account that we are a strong regul regulate regulated field and trust and innovation that go hand in hand because strong data governance is at the foundation for enabling us to manage uh models and I application effectively.
Bad data governance doesn't just cause errors. It can lead to significant consequences. We are a regul regulated field. So we we can get financial uh penalties if we are not compliant and uh bad data can cause us to take poor decision making because data is essential for a datadriven company or at least for what we won't try to be. This can cause loss of trust and reputation and uh worst of all it can cause operational inefficiencies.
So you maybe can have the case where you can uh you have to put data in the right place and it's difficult without a clear governance framework. Now let's have a look at our uh solution. To support data governance at scale across a multi company group, we are sub substantially acting on three different levels. Organization, a new data platform with data mesh principles in mind and we are modernizing our infrastructure by moving to a cloud-based platform with data bricks and AWS.
Let's have a look at what organizations means. Traditionally, our company has been organized with a clear distinction between business area and it department with the data team following in the latter uh falling in the latter. We are now starting to follow team topologies principles. So we have introduced a cloud platform team which is responsible to manage the infrastructure of the company and then we have introduced um we are reshaping uh the uh the data team which is now composed by a data platform team which is responsible to manage the data infrastructure and it gives a standard and best practice within the data platform.
We have organized uh several different delivery team which are organized in domain that trying to reflect the business organization of our company and in the end we have a data governance team which is uh horizontal uh respect to the other teams to delivery teams and it's responsible to manage the data the data product portfolio. This is because we are trying to follow some principles of the data mesh. Uh it's not easy for us because we came from a very different kind of organization.
So we are design a new data platform uh starting from uh a new logical architecture uh because as I already told we are trying to do uh to adopt some mesh principles. So since we are uh we have a lot of different use case and different scenarios we have defined the new logical architecture which is composed by three different subsystems. We have the enterprise data lake that is the subsystem where we ingest data within the data platform.
So it's just for ingestion and it's where we store source aligned data product. Then we have the analytical platform which is composed by two fter more subsystems. The data warehouse and the analytic the advanced analytics platform. The data warehouse is just for standard business intelligence while the advanced analytics platform is for machine learning and artificial intelligence application. In those subsystem we store both aggregated or consumer oriented data product.
Then we have the operational platform and it's the part of our architecture where we are able to proc to process streaming data and uh we in in in that kind of application the data platform itself became an operational system. So we provide other system of the company with data that are consumer oriented data product. We have defined um a central data platform team which is composed by uh a product owner uh that's in the end will be me uh data architect platform engineer and administrators.
So we are trying to follow me principles and even the data platform itself for us is a product. So the data platform team is responsible to uh deploy platform artifact that is a self-service platform that in the company can be used by uh our stream aligning team to realize the product. The evolution and life cycle of this platform is guided by uh federated governance community. This is a virtual team composed by different experts spread across our company.
This team uh goes directly to feed the product backlog of the data platform team. We apply this model to each company of the group of the group. So the data platform team and the federated governance community are entities that live within the parent company that is UNIPOL. Other company of the group can use our platform. Let's say that we release a version of the platform that is 1. 0. In this slide, we see that company X is running it platform and it's provided with its own team of platform administrators.
In reality, smaller company can exploit the administrator of the parent company through a self augmentation model. When a company is use is using um the platform can ask for a new feature or can find a bug during the their application development. In that case that request goes directly to the product backlog of the platform team. Then the data platform team release a new version of the platform and that can be used by the plat the company that has asked for that new feature.
Um it's important to not only not all company must use the same version of the platform at the same time. So we as I already told we treat the platform as a product. Now let's have a look at how we are uh deploying the data platform with datab bricks on AWS. Essentially we have three different environments. The each environments correspond to an AWS account. So the development account, integration test account and production account.
Then we have a fourth account that is the management account and it's where we have the datab bricks uh subscription and the datab bricks account. We manage environments with datab bricks at workspace level. So each workspace can use resources of um a related AWS account. We have uh put in this slide just some service some AWS services. It's not an exclusive list but it's just to give an idea that resources live within AWS while in each workspace we have all the kind of the datab bricks objects.
We deploy with the same model the same platform for each company. This is quite effective because in this way we are able to allow companies to work in a current way in um in accord to the standard that the central team is giving to them and in this way we are also able to uh manage cost effectively. So each company has its own accounts and we can track cost at level company but even at work at workspace level. So each company has uh its dedicated and own accounts.
Now we have a look at the infrastructure of the workspace. We have essentially one workspace for each team and for each environment. Within the workspace uh we have all datab bricks object such as compute classes catalog notebook and g integration. So we provide each team with a g integration in order to version their code. For a data perspective in the AWS account we have three different kind of bucket that are the workspace root bucket that is to deploy the workspace.
So it's it's really a technical bucket. We have the landing bucket to store incoming data and the data bucket that is where we put the real data. Then we use KMS service to encrypt the data within the data bucket and we provide each team with the possibility to use the AWS transfer family protocol in order to uh send or receive data within uh the platform. Now we are going more into details to see the data storage architecture and the bucket layout.
So the landing bucket uh temporarily stores incoming data. It's just a temporary area to store uh data. But when we start to ingest data, we put we put it in the in the data bucket which follows the medallion architecture. We have the classical medallion architecture. But for each layer of the medallion architecture, we have different zones. Starting from the data lake, we can see that within the bronze layer, we have the road zone that is for raw data and conform zone.
That is a zone where we just do data tabization. Then we have the silver layer and it's where we put where we have the standard zone that is just for an answered quality check and the refined zone that is where we store source aligned data product. So in the refined layer you just have data model like near to the source for the data warehouse. We have just one zone for the CL layer that is the source layer, the source zone that is a logical layer composed by a lot a lot of views because we ingest data within the data lake and in the data warehouse we just read the data from the data lake between these these views.
Then the golden layer have the core M presentation zones that are just for classic business intelligence. Then the analytics platform has another different perspective because even in this case we ingest data ever in the data leak but we have a golden layer composed by three different zones. Uh the first one is the prop the prep process layer that's for data preparation and the registry zone is where we train and score our models and the result zone to store the output.
Then the operational platform uh we have just the application zone that is a layer that uh we use to serve other systems or other applications. All those uh architecture is within the data bucket. We have one bucket for each team. So if you remember the previous slide and we name the bucket uh with the name of the team that owns the bucket itself. Then all the zones are just part within the bucket. Now I leave you with Javanni which is going to show you the Unity catalog solution.
Thank you. And now let's dive into what is truly become the foundation of our governance model. Unity catalog govern once and for everywhere. This is the promise. But how do we make it real in a complex organization like Unipole? This is where uni unity catalog come into play. Unity catalog has been a real gamecher for us. But let's take a step back. What is it really? Unity catalog is the datab bricks unified governance solution for managing any AI and data access uh across workspaces and teams. uh Unity catalog provide a single point of control to manage every uh datab bricks access and access like notebook a model and dashboard.
It brings together three main key aspect uh uh centralized governance model fine grade access control and uh audibility all strictly integrated inside the platform. Let's see now the object model of Unity catalog. At the core we have the meta store the central governance layer of Unity catalog. The meta store is what make us uh apply make us possible to apply uh centralized governance policies across all the workspaces and environments.
It's also maintain all the metadata about the data like their location, their uh access and even their lineage. It's also exposed at three level uh three level name spaces uh composed by catalog schemas and the object themselves. On the top we have the catalog along with other uh securable objects and is on the top level when we want to organize data inside the platform. Uh every catalog in contains schemas where they are they can be seen as a uh logical uh container for grouping related data assets and on the bottom we have the object like tables, views, volumes, function and this is the object the user interact with directly.
Now that we know what is the unity catalog, let's see how we have organized cataloges inside the platform. We have two kind of cataloges. Private catalog and the shared cataloges. Each team works on its own private catalog. This is isolate place where they can uh develop, explore and even try without any concern to affecting others. When they are ready to share the data, they publish them on the data platform catalog.
A place where user are usually uh read data in a read mode only and this is act as the official trusted source of data. Uh also the platform catalogs are organized by subsystem and on the bottom we can see the naming convention we are adopting for private catalog. Here we can see information like company team and environment. This is help us to uh to scale our platform in uh inside. Uh let's see this slide again. has already walked you through the different layers of the platform and how they support the data life cycle.
Now I want to show you this slide from a different point of view. I want to show you how we have organized cataloges and schema to represent the data architecture. Let's start from the data lake. Here we have the bronze and silver layer. Inside the bronze layer we have a shared catalog uh that is focused on uh ingestions and here we can see the V schemas that are organized by source company and source systems. On the silver layer we is where we are starting aligning to the logic business.
And here we can see that we have a more granular organization of the cataloges. Here we can see that we have a catalog for every domain and on the refined zone we also have a schema for every data product. Now let's talk about the data warehouse uh subsystem where we uh manage data for traditional BI. Here we have the uh silver and gold layer. On the silver layer, we have a unique uh catalog that is also shared with the gold layer and data organized by data product.
Uh now let's shift our focus to the analytical and operational layer. Here we are in the gold area and here we can see that we have a catalog for the analytics layer and the catalog for the operational layer. Now let's see how we are organized the data inside the data access inside the platform. uh we are using Azure entry id as identity provider and the groups are synchronized inside the platform uh via a shim API. In the platform we have three kind of accounts.
On the top we have the datab bricks account that are used to manage the platform. Here we can see the different roles. We have the account admin and metas store admin that are the role that are used to manage the platform. Also have an architect role that re access in read uh mode on all the platform and also an operational role that uh uh that is focused on the maintenance. After for every workspace we have dedicated roles that are uh workspace admin that is the uh account role dedicated for the management of the workspace.
We also have the deployer developer uh account role that is focused on uh the development of the code and the deployer role that is focus on the CI/CD operations and for analysts that want to interact with the data product we also have a dedicated role in this way they can read and interact product let's say now we have organized the data access inside company. Uh here we can see an example where we have two different company on the company X we have a typical team team one that has a full access on the developer and seat workspaces and the limited access to the production workspace.
Also when we see the cataloges in the bottom we can see that these teams has read write access to data on the dev and seat environment and only a read uh access to the production data. On the right we can see that there is another company with their data and their cataloges and this data are physically separated from uh the other company. So the team one in the company X are not is not able to read interact with that data.
But what if they want to interact to this data? This is where uh data sharing come into play. With data sharing that is an open protocol supported by databicks. uh you can interact to data that are available in another uh datab bricks account and you in you can interact with this data without copying data or duplicate them and using the governance of unity catalog. But if we want to interact with data that are outside the datab bricks we can use lake house federation.
Last federation is a service that make us possible to interact to data that are outside the platform uh using the tools that we already know inside the datab bricks platform. This is helpful for us because this can help us in to interact with data that are uh on the on pre platform during the migration but also when we want to explore data uh in other system and uh when we don't know if we want to ingest or not this data.
Now we have seen how to go how the man the governance works also the data access and the data sharing but now let's see how we have automated everything with by terraform so build apply repeat uh so we want to know how we have automated everything let's see how we are managing the infrastructure at scale inside the unipole here we can see that we have two different cloud platform platform uh with different service catalog.
Sorry. And on the right we can see the cloud platform service catalog where we exposes uh low-level modules uh such as uh uh network module uh storage module and so on. On the left we can see that there is another service catalog the data platform that expose higher level modules for datab bricks like cataloges workspace and so on that internally interact with the second uh uh service catalog for the services. So when a team need a new service inside the platform, they don't need to do it uh programmatically or manually, but they can simply interact with the internal developer portal and the portal is going to interact with the internal service cataloges and provide the component to the team one.
And now back to for the final tools. Thank you. Now let's have our final consideration. Unity catalog enabled us to govern data scale without slowing down innovation. Unity catalog has been really a gamecher for us. We have reached a very effective model with centralized governance and we also can guarantee enhanced collaboration among different team spread across different companies without slowing down innovation. We have framed permission. we have unified access layer that it's really important to manage all data spread across the organization and essentially it's quite important because we I remember that we are a regulated field with unity catalog we can have built builtin audit and lineage so we are able to track data and it flows and how it flows but what have we learned throughout about this journey.
We have some key considerations that we want to share with you with you today. And uh we start from governance first because often data governance is considered just um a constraint but governance brings clarity, compliance and alignment. So it's important to deal with data governance from the first moment when you start to build something new such as a data platform. Clear team boundaries. We are adopting team topologies approach but it's really important to define clear responsibility among different team.
We have stream alignment team platform even two kind of platform team. So the responsibility and the zone of action for each team, it's important to reduce friction and accelerate collaboration. Use an iterative approach. Don't aim to think for the first moment to reach out the best solution, but listen to feedbacks. um work strictly with people that actually use the platform and follow their suggestions because uh we we have experimented that designing iteratively to evolve the solution is more effectively.
Think about scalability from the start. uh that it's quite simple to say but it's important to remember because long-term success starts with scalable design. It's uh difficult to uh scale a solution that was not thinked in that way from the first moment. So when you have to take some decision, it's important to think that how this will affect in the future when the platform will grow. Last but not least, tech alone isn't enough.
And this point it's at my my art because organization and process uh are key factor of success. So unity catalog has been really a powerful tool for us. It allowed us to reach all our goals from a technical point of view. But organization um in my idea comes first because without a clear organization it's difficult to design a solution that will be effective. Thank you. And uh now we have some space for some question if you have

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.