Scale Governance: HelloFresh's Move from IAM to Unity Catalog and Automation

Summary

HelloFresh replaced a fragmented governance model spanning IAM policies, Glue Catalog, and Hive Metastore with Databricks Unity Catalog, using Hive Metastore Federation for zero-downtime migration and Terraform automation for persona-based provisioning across 10,000 corporate users.
Automated grant management through a domain-based governance model with YAML-configured access controls cut data access approval times from 44 days to 15 minutes, eliminating the manual bottleneck that was blocking both analyst productivity and AI adoption.
Proper Unity Catalog governance unlocked Genie spaces and AI analytics capabilities, with 68 percent of HelloFresh's Meta advertising spend now managed by AI — demonstrating that governance enablement directly produces measurable business value.

Scale Governance: HelloFresh's Move from IAM to Unity Catalog and Automation

At HelloFresh's scale, 70,000+ tables generated across decentralized domains with rapid growth threatened to outpace governance. Legacy governance spread access across IAM policies, Glue Catalog, and Hive Metastore with fragmented naming conventions, making data access requests take months to approve and leaving analysts confused about physical access models.
Discover how HelloFresh pivoted to Unity Catalog, replacing brittle IAM policies with a unified privilege model. Learn the blueprint for zero-downtime migration using Hive Metastore Federation, the domain-based governance model that balances autonomy with control, and how Terraform automation enables persona-based provisioning (admin, engineer, analyst, consumer). See how automated grant management cut access requests from 44 days to 15 minutes and enabled 10,000 corporate users. Explore how proper governance unlocked Genie spaces and AI analytics, with 68% of Meta spend now managed by AI.
🤝

Chapters

FAQs

What governance challenges did HelloFresh face before Unity Catalog?

HelloFresh managed governance across IAM policies, Glue Catalog, and Hive Metastore with fragmented naming conventions, creating an environment where data access requests took up to 44 days to approve and analysts were confused about how physical access models mapped to their needs. The scale of 70,000-plus tables growing across decentralized domains made manual governance increasingly unsustainable.

How did HelloFresh migrate to Unity Catalog without downtime?

HelloFresh used Databricks Hive Metastore Federation as a bridge, allowing them to migrate tables and governance policies to Unity Catalog progressively without disrupting active workloads. The migration was complemented by Terraform automation that enforced consistent provisioning patterns and a domain-based governance model that distributed ownership to the teams closest to the data.

What is HelloFresh's Terraform-based governance automation?

HelloFresh built a Terraform-based automation framework that provisions Unity Catalog access based on four defined personas — admin, engineer, analyst, and consumer — using YAML configuration files that describe each domain's access requirements. Domain owners can manage grants for their own data independently through this framework, with PII access and row-level security handled by the central platform team.

What business outcomes did HelloFresh achieve after migrating to Unity Catalog?

HelloFresh reduced data access approval times from 44 days to 15 minutes and enabled 10,000 corporate users through automated governance. The improved data access foundation unlocked Genie spaces and AI analytics capabilities, with 68 percent of the company's Meta advertising spend now managed by AI — a direct business outcome from the governance investment.

Full transcript

[00:08] All right, let's get this show on the road. There's a lot that we want to cover today. Welcome everyone for governing at hyper growth how HelloFresh cut data access from months to minutes specifically with Unity catalog. Um what we want to show you today like we will just do a brief intro and like a bit of business overview in terms of
[00:24] like HelloFresh in general. We will go through what we were using before in terms of like our governance and how we were managing access specifically with us three and with glue. Um then we will show you like how the new setup actually looks like with with Unity catalog and a
[00:39] lot more also in terms of the automation that we built behind that. But first things first, I'm Max. I'm leading what we call foundations at HelloFresh, which is essentially our platform organization. Not only data platform related topics, but also infrastructure core topics like like a
[00:55] bit more in general. And like I've been working in the data space for like more than a decade now like always focusing on platform topics and productionizing like large-scale systems both for data as well as general infrastructure topics including like cool new things like GenAI enablement,
[01:11] which is something that of course a lot of the things that we are building now is something that we are also enabling. And I'm joined today by Adam. And hello everyone. My name is Adam. I was software engineer for over 10 years. And now I am data engineer for
[01:27] the last few and I think this last few is probably five or six years now. Currently working as a senior staff data engineer at HelloFresh and I'm helping Max build data platform from the rest of the HelloFresh. And in my free time I am father of 10 years old
[01:44] daughter and son one and a half years old and I am a big board gamer. Right. Cool. Um just very briefly about HelloFresh itself. Like who of you is using HelloFresh by the way? Hands up. Okay, cool. Who of you was using HelloFresh in the past but not anymore?
[02:01] Okay, I have you memorized. Like I'll talk to you after this. We need to sort this out. No, but like um not a lot of people know that HelloFresh is actually a German company. Um like we went all the way here from from Europe um to to speak to you today. Um not a lot of people also know that we have quite a
[02:16] big selection of different brands including uh Factor for instance, which is also quite big here specifically in the US. Um that also belongs to HelloFresh. Uh but just to also give you some numbers so that you get like a rough impression about like the scale of what we're dealing with. Um we had like like more than 1 and 1/2 uh
[02:32] billion in in revenue just in Q1 this year alone. We have like more than 18,000 people worldwide. Um and like that of course also means there's a lot of data that is produced through that and that we are handling also from the platform perspective through that.
[02:48] But let's jump right into the original governance hell that we were originally coming from. And um just to set the scene a little bit so you get an understanding in terms of the tooling landscape that we were dealing with at that time is generally speaking um HelloFresh is like almost exclusively on AWS. So like we have most
[03:04] of our data in S3. Um we were managing access to that data directly through IAM, right? So that means like like direct policies managing raw access to the files on the S3 level. Um and that is of course like a very complex and also to large extent like very manual
[03:21] process in terms of how we were dealing with that. And then we were also using Glue catalog like for the table management, um which came with very limited capabilities in terms of actual governance around that. And how was that usually then looking like when bringing that back into into Databricks as well?
[03:37] Um we were using the the Hive metastore federation like the the legacy version um to to like essentially make the tables that we had defined in Glue available in Databricks. And then on the Databricks side you were having like instance profiles which were directly connecting to the IAM roles that were defined, um which made it such that we
[03:55] had very limited usage of the Hive Metastore table access, like to to be able to actually manage it that way. Now, how did that setting come along in the first place? Like, what what were the general access patterns that actually emerged
[04:11] First thing first is like like we were always building our data access to be open by default, meaning that anything that is created is generally made available unless there's a very specific reason not to. One of those reasons is of course PII data, um personally identifiable information, um that needs
[04:27] to have be like special protected. That is something where we had like put specific restrictions in place. Um on top of that, we had like a broad variety of tooling, right? Like we built our in-house framework um to to do like data engineering pipelining um couple
[04:42] years ago before things like Lake Flow actually existed. Um and um that is something that is natively running on Spark on EMR. Um we of course do operate with Databricks at large scale even already for the past couple years. Um we still have Snowflake as well as a data warehouse that is sitting next to all of
[04:59] that. And then we had to like manage the access to all of that also across the different tools. And even bigger pain point also across multiple AWS accounts, uh which added up to the pain that we were in. But the worst part was also like that even in terms of the users of the data, there was quite a gap between
[05:16] um the levels of skills that the people were dealing with. Like for instance, analysts had no clue whatsoever how the physical access underneath was actually mentioned managed, but it was something that they had to deal with on a daily basis as well. That that was like breaking a lot of
[05:32] continuously. And depending on like which team you were coming from or which tool you were using, um the access patterns were also very different. And like depending on like the different parts of the org you were talking to, um was always like different points that you needed to go through.
[05:47] Now, how we solved that problem originally was um like on the first side, like on the metadata side, um we had like like Glue Data Catalog as the source of truth. That's where the tables were defined originally. Then we had the Hive Metastore Federation into Databricks. That's how the tables got
[06:02] made available. And then we were relying on on a lot of patterns like naming conventions, for instance, to differentiate between private tables, which is something that only somebody, let's say, within your team is using, so very limited access, but like a lot more freedom in terms of what you can actually do with the data. Um and public
[06:19] tables, which was like something that was supposed to be of higher quality, that is shared more broadly across the organization. Now, translating that also on the physical layer then is um like on the one hand side, like we had global read access to the to the data tools, um like
[06:35] on the account level, like directly defined again like through through access policies, for instance. And then we had like explicit denies for sensitive data, right? So, like we had bucket policies, um and then for some specific use cases also, we had to go like one level more into the details and
[06:51] being even more selective. Like, for instance, um particular teams only getting access to the data from the particular country that they are responsible for. Right? Specifically, when it was like more like the head dealing with customer data, for instance, um and like you are
[07:06] from customer care for the US, let's say, then we also need to make sure you only can see the data from the US. And that is again something that were on the physical level was like managed through like particular table partitions and then like definitions that were only pointing to a particular physical part
[07:22] of the data, um which again was quite complex and like quite a mess. Um cross-account access is of course something that I already Now, legacy governance then broke down into like like essentially two different layers on where access was defined, right? The compute itself had access to
[07:39] the data via the attached policies, um like the IAM roles as well as the Sorry, just just the bucket policies, like the the policies on that front. But, when it came to the actual table definitions, then you had that second layer with the IAM roles and the respective policies
[07:55] that were then also attached to the respective instance profiles um through the definitions that we actually got from the from the data catalog on that front. And like those table permissions were then again like managed on on IAM level. Now, that all for the user again
[08:11] mentioned that the access propagated to the user via the cluster that were that they were using through the IAM roles that were associated with the access to the respective tables, while the physical access was directly mentioned managed on the on the bucket policy side based on the metadata that we had from
[08:28] where those tables were actually running on top Very complicated and something that that we had a lot of pain with, not only because of the complexity, which automatically also results in things getting broken a lot more often. And like some of the pain points that we actually had was that, right? Naming
[08:44] conventions were Like like teams ended up in situation where they were like, "Hey, why have you deleted this like super important asset that I was using all the time?" Well, because you didn't name it properly. Nobody knew where it was actually important, right? The restriction on on the PII data side, like randomly people
[09:00] started deleting data that they were not supposed to be deleted and they reached out to the platform if we can actually restore the data and like fix that back up, right? On the physical layer, the the problems that we had with the access was that people wanted to get access to like certain public tables, let's say, and then all of a sudden they
[09:16] needed to deal with, "Okay, what IAM role they were supposed to use? From where are they actually going to query that?" And if things broke, they were left with like random technical error messages and they had like no idea where that was even coming from, right? Was it denied on the bucket level? Was something wrong
[09:32] with the role that they were assuming? And just like a a of steps that they had to go through and from a cross tool perspective it was even worse because like some systems simply couldn't access certain data because that particular mean of access was not
[09:47] clearly defined or was not technically supported and sometimes we even had cases that like auditors needed to query some data to understand what was going on in the organization but then we had to one more than two very specific systems because otherwise we would have overexposed data to somebody from
[10:03] outside of the organization which was also something Last but not least granting access to somebody took forever. This is like a real use case that we actually had where like there was a request that was raised at some point and then there was like 65
[10:18] messages exchanged back and forth over a period of one and a half months until finally this was resolved and the person got access to the data that they That is something that we really had to address that we really need to get a hold of and that's what Adam will talk
[10:33] to you now about. Thank you very much Max. So before I start show me the hand if you in past set up the Unity connectivity to the to your cloud provider or at least you know what the tools are required to set
[10:49] this up. Not not much. So let me actually jump in in a bit more detail into that. So if we look at the Unity Catalog permission model probably most of you are familiar with like this middle pillar where we have got essentially the direct
[11:06] access to we are giving access to the data tables view volumes but the Unity Catalog permission model covers a lot more. There are various securable objects which can be used to properly protect
[11:21] the infrastructure. So as an example on the left hand side we have got things like service credential storage credential external location which we are going to cover in a bit and there is also many other resources for example for Delta sharing. I can imagine nowadays it will be also
[11:37] used for open sharing which are out of scope of this presentation. And maybe one more important point to note here the middle pillar is also supporting some inheritance concept. And so like schema level grants and permissions are propagated to the tables
[11:52] and views and so on and similarly with the the catalog. And and also this is metastore level concept which means that this is not tied to the particular workspaces. So now let's see how this magic works and why Unity really
[12:09] can access the data which is in underlying cloud provider. So we need to look at this from the two parts what we have in Unity and what in our instance AWS brings for us. So of course we want to protect or give access
[12:24] to appropriate assets which is just metadata on top of physical objects which are in the S3 buckets. Uh then the important concept here is external location. The external location is essentially link between
[12:42] remote S3 bucket or even some prefix inside that but for the simplicity actually also on the HelloFresh we are using like one to one mapping we have got single external location per bucket. And and this link links that bucket with the storage credential and
[12:59] the storage credential is essentially key information what Unity plane which Unity which role Unity plane should use. So we have got IAM role which needs to be specifically configured which trust essentially the Unity
[13:15] catalog compute. And what it means that any request coming from Unity is it by human is it by machine by the service principles will go through this route. Um and there is a bit of quite
[13:31] specific thing happening that as long as a user or machine, uh let's call it them principal cuz this is the idea how Unity represents that, and the principal can be user, service principal, so machine, or group, which
[13:46] can group other users, other service principal, or also other groups. Um whenever they access some table, and they have got the permission to this table, Unity will use the configured external location information to physically access um that data um in uh
[14:05] a AWS in the cloud provider. Uh this sometimes might be quite confusing because the user doesn't necessarily need to have direct grants or permissions on the external location. This is only required if someone would like to read files directly, like read
[14:23] object, write files, read files. Um then the direct grants are required on the external location level. Uh but for most common scenarios, um uh the Unity uh assume that if someone has permissions to table, they are allowed to access uh that uh that data
[14:40] as well. Okay. And the last important concept I wanted to mention what makes it secure, and it's quite quite clever mechanism, is the idea of the tone scope access. Uh so what it means this this role essentially that we are configuring has got broad permission in that bucket.
[14:57] Um and also it's possible to actually hit Unity catalog endpoint to get the permission to the cloud provider uh because the various engines are using this approach, Spark, Snowflake, DuckDB. Um and how this works is Unity catalog is
[15:12] asking AWS, "Give me temporary credentials, but please limit what I can do only to this location." And for example, only if the user wants to do select on the table, that this should only allow to read the objects and not write the objects. This what it makes it extremely secure
[15:29] and we no longer really need to deal with this infrastructure uh complexity. And then and of course this the same patterns are uh in different uh cloud providers as well. So, with this in mind, like what really
[15:44] what it how it benefited us. So, now we can clearly separate human and machine access. Um it's extremely clear who has got access to what tables, to what schema uh because it's no longer like that user has got access to the particular machine
[16:00] and randomly um they have got access to the data they shouldn't have. Uh thanks to the Unity Catalog, it's much easier to to audit the access, audit the queries. Um and also those access permissions are a lot more robust. Now we clearly see that we can give some
[16:15] uh permissions on the schema, on the table and no longer on the bucket using this quite complicated IAM policies uh uh which are completely detached uh from the table information. Um and also as previously mentioned,
[16:32] uh the access is quite easy via multiple query engines. Um Snowflake supports this this way of querying uh open source Spark, DuckDB. Um and there's also quite clear feedback from the user. If something doesn't work, the user gets information. Sorry,
[16:48] insufficient permissions. Like, uh please request the permission for this asset. Um whereas in past uh it was this extremely long conversation loop, why it doesn't work, uh let's see, maybe you are using wrong role, and maybe you are using
[17:03] wrong compute cluster. Um it completely simplifies this with this setup. Um and of course it also unlocks a lot more for us. Uh all the Unity benefits. Um although we are using uh row level filtering, we haven't started using uh
[17:19] yet uh row and column level uh security. Um we have haven't used um Ibach Aback yet. Um we started using Clean Ads a bit, but we are still experimenting. We noticed couple of problems, uh especially cross engine and cross workspaces, but this is still definitely
[17:36] uh something on our radar. And of course, having this proper governance across our assets is the key enablement for AI and for agents. Um so, now in the contrast, another conversation with the user um
[17:51] who couple minutes after we announced that we are releasing this feature, that now you can request the PII access to our tables in our community workspace, um the user was so eager that they have this access in past in our legacy workspace. Now they are requesting um in
[18:08] in the community, and you know, a bit of back and forth, um uh setting this up, explaining to the user what's possible, how the user can access the data. 2 hours later, um user was amazed that yeah, it works. And it even works in the query engine, which
[18:24] is essentially our Data Bricks workspace for ad hoc analytics, which we never could um enable the PII use cases due to the uh complexity of the setup. So, already like a big win for us and for our users.
[18:41] Um okay, so now maybe let's jump into the fun part, how we designed this and how we automated and um how our users uh can quickly benefit from Data from Data Bricks and using our automation. So, we have got two key
[18:56] components, um two mono repos uh to get our teams quickly started. First is Data Bricks automation. And this is entry point for our users to just get on boarded. Uh we coined term domain here, so we are provisioning like
[19:12] for for group of users, for maybe multiple teams, we are calling that domain. This is not Databricks term. I know that Databricks started introducing also domain, this is also our invention. Um and what this repository will do, we are using um Entra uh for SSO, for
[19:29] single sign-on. Um so, we are going to provision uh appropriate groups for access to Databricks. And our users from the domain can use those groups for role-based access to the Databricks. Uh we are going to give the users set of catalogs um
[19:45] from this repository, and we are going to provision them appropriate S3 buckets, essentially the whole infrastructure which I mentioned. So, our user, they don't really need to um care about that. Um they are getting this uh from our infrastructure for free. Um and the
[20:02] second part which is quite important for the user for the users is our place when the users can control the grants. Uh we truly believe in automation, we want to automate as much as possible. Um so, we are also automating our grants. Um and this is the place when we also
[20:18] govern our Glue Data Catalog uh because of course, we cannot lock users away uh from Glue. Um this is the place when users can um manage the access to the domain catalog. Um and this is when we are implementing central features like PII, row-level uh
[20:35] security for central assets. So, let's jump in uh to the automation uh for Databricks. Um and from the user perspective, it's essentially as simple as opening pull request with just couple of line of YAML file. We love YAML uh
[20:51] for things like that. Um and they bring new domain. So, on this example, it's essentially our data platform domain. Um the users can optionally configure additional groups, and those groups can be then used uh to further grant um other access to other
[21:10] assets. So, in this example, we see things like system catalog access. So, the users will be able to see or read system tables. We see quite privileged group which will allow to see all brands in our sensitive
[21:25] tables. We see group dedicated customer PII readers to issue select statement on the customer table. And then we also see brand specific role level filtering for the German brand. And we are going also to like
[21:41] depending on the configuration, set up the initial ownership for those groups. So, the users can fully manage the permissions and the access on their end. And after those groups are created in Databricks,
[21:57] then the second part of automation kicks in. And the users need to come and provision like physical infrastructure which is happening behind the scenes. But again, just couple of configuration elements. Probably most important
[22:12] from this example is actually the target workspace. So, by default, we are offering like common community type of workspace. But for more technical users, there is also option to provision the fully dedicated workspace. Of course, it requires a bit more
[22:28] configuration. But for someone who just wants to start up with Databricks at HelloFresh, it's extremely simple. And also what I wanted to highlight here is we have got this idea of the personas that the users can provision using our automation. By
[22:44] default, we support we bring five different personas. First one is admin. This is not workspace admin. This is not metastore admin. This is just the admin of the domain which gives appropriate admin permissions on the catalog itself. There is engineer persona, Um, a lot
[23:00] less privileged. analysts, even um, less privileged and uh, service account, service principal persona, which uh, we asked our users to use for automation. Um, and the important something we relatively recently introduced, the
[23:15] consumer persona. And the consumer persona can be used for this new way of granting access for the user without them really going into the the old Databricks UI and just use uh, new one. Um,
[23:32] so when they want to like when the domain wants to give access to their business users, uh, they can use this uh, consumer persona. Um, and also um, the users can easily create new service principals. Um, so for example, they want to use some kind
[23:47] of external tools for data quality or maybe other warehouse, um, they can bring this in. Uh, they can configure default group membership uh, to ensure that all access is in in in place. And stepping a bit back, we actually offer a bit more features um, in this
[24:04] repository. Um, there is another securable object called service credential. Um, this is used to essentially get access to like AWS APIs or maybe some external secret engines. Um, we had to implement this even internally
[24:20] for our data platform domain uh, because we wanted to analyze some uh, glue tables and the federated glue catalog was like didn't give us enough information. So we uh, provisioned for ourselves the service credential, which can be essentially used uh, to safely
[24:37] uh, connect to AWS and only get uh, the permissions to get the better data from glue. Um, and all the grants are also in place, so we can clearly see uh, who can invoke those, who can use those services um,
[24:52] and things like that. Additionally, quite important feature um, is the automated group assignment through the group memberships module. Um, all of those custom groups that we created, the users from other domain can come and they can request
[25:08] their users or their principals to be added to those groups. So, for example, um, if, uh, admin from our another domain would like to get access to the system tables, then they can come to us, open pull request to be added to the system reader, uh,
[25:24] group. We will approve that, um, and immediately they will be able to query, uh, those system tables. And And all of that is, um, in this instance, built on top of Terragrunt and essentially Terraform reusable modules, uh, using the Databricks provider.
[25:45] Okay, the other repo, uh, I mentioned used for the, uh, controlling the Unity, um, access control, um, the same idea, uh, the user needs to create some YAML file. So, as an example, um, we have got a couple of grants, uh, for
[26:01] like table level grants, schema level grants. Um, what's important here, we support only schema or like the lowest asset level. Um, we don't support catalog level grants, we don't want teams, domains to share the whole catalog. Uh, we believe
[26:16] that this is often too much, right? Like in your catalog, uh, most likely you have got a lot of intermediate assets, like, uh, so so this is like why we don't allow our users, uh, we want them to we want prevent from this from happening. Um, but also we
[26:31] want this to be quite easy for the users to be used because we don't allow the catalog level permissions to be set up, and of course there is super important permission use catalog, um, so actually the the the users can issue any queries, uh, so we are going to like derive, uh,
[26:48] the required grants. Um, so everything still works. Um, another important concept I wanted to highlight here because as Max mentioned, we were still believe that at HelloFresh, a lot of data assets are helpful and a lot of data assets should
[27:04] be shared. So, we have got this idea of the open access, internal access. Um so, this is essentially the dedicated group that you can use to target the grants and essentially almost every employee will have access to those grants. Um so, again, this is quite
[27:20] useful for the consumer consumer access um and those types of patterns. Um and of course, you could argue that you can achieve the same things using the all users grants. We want to avoid all users grants as much as possible um because
[27:36] sometimes we have got those use cases to bring external users. We want to maybe onboard auditors um and they shouldn't see the tables what they are not directly granted. Um so, instead, this is like our solution to solve those problems. Um we quite
[27:53] dislike the all users grants. Um and also, as I mentioned, um we use this repository internally uh to manage the uh the old all of the Glue Glue assets. Uh so, this was of course something that
[28:10] maybe not really surprised us um but when we federated Glue into Unity, um it requires a lot of work, a lot of governance work because now everything is governed through the Unity, which is a good thing. Um but it creates sometimes a bit of tension, right?
[28:26] Again, we have got a lot of assets open um and only maybe there were like a free deny. Uh there was um in the one workspace, we were using legacy Hive Metastore table access and suddenly, um we truly need to control what is queryable. Um
[28:43] so, yeah, we were going a bit of um forward. We piggybacked on those our naming conventions. Um it could work slightly better, I have to admit, uh because many teams uh haven't followed naming conventions. They were like, "Why I should follow naming convention?" This
[28:59] is good example why naming conventions are are helpful. And there's also, of course, a lot of assets we no longer want to grant permissions. There are schemas with like 10,000 tables. We don't want to open those schemas. Um they are often like a dumping ground for
[29:14] some random test data. Uh so, there's a bit of tug-of-war with the users, but generally speaking, again, if someone has got good use case, it's extremely easy to use our open access group uh and uh grant use schema and select um to
[29:31] appropriate uh glue uh glue asset. Um another use case in that street repository um is something we had to automate it on our end. Um we found this a bit more tricky than initially expecting. Um there is a gap
[29:48] in in native automation what Terraform provider offers. Um so, we had to do a bit of custom scripting, Python Terraform, essentially to solve this uh PI access for various markets and various tables. So,
[30:04] essentially what we ended up creating um is a a groups for every market we support um and then groups per hour uh PI tables. So, in total we have got here around 40 um and our scripts uh generate and attach uh
[30:20] appropriate uh UDF uh filters. Um and also, uh now stepping back, um like, this is what we believe from the governance perspective. As a relatively small team, we don't want to manage the permissions for the users. We don't want
[30:37] to be uh bottleneck for approving the the pull request. Um so, we strongly believe that each domain should own the grants uh that they are issuing on their catalogs. So, in this catalog in this repository, we essentially split
[30:54] um the grants per each domain. And the teams can come, they can claim this domain, they can set themselves as a code owner in the GitHub. If someone is not aware how it works, essentially then in the in GitHub only the members
[31:09] of the teams who is listed as a code owners will be able to approve uh the pull request. Uh so, for example, uh payments team have got I don't know, like some GitHub uh payment team. They have got five members, and only those
[31:25] five members will be able to uh approve pull request um to the payments folder. Um, everyone else approval wouldn't count. Um And so, of course uh this also requires a bit of additional governance because we don't want other teams stealing
[31:44] uh the grants. Uh so, we we have to have some automation to prevent uh foreign domains to be modified from your domain. Uh so, essentially using our naming conventions, you can only um issue grants uh towards your catalog. Um
[32:00] For legacy assets, for new assets, it's a bit more complicated uh because there was no clear naming conventions. So, we we took this approach that we want teams to come to us and claim ownership of some schemas. Uh of course, there could be someone that doesn't really own those
[32:16] schemas which is doing that, but we noticed actually opposite uh problem. Currently, we have got 8,000 YAML code which is configuring the the grants, and we really hope people would come to us and then start uh taking those grants and put them into
[32:32] their folders instead of this being in our data platform. Um, but this worked quite well. Um, I think the teams are um quite agile now and they can fully uh, quickly get the grants. They can give grants uh,
[32:48] where it matters. So, now couple of last closing thoughts uh, from me. Uh, what are the next steps? Maybe couple of gaps in our governance journey. Um, we for sure want to double down on the row column level security. We have got
[33:05] um, quite important and useful tool for our users uh, which brings uh, Kafka topics um, into Iceberg tables in like you know, nearly time. There is like a 10-minutes delay. Um, and some of those Kafka topics uh, contain PII data. So, we want to build
[33:22] uh, for the users quite seamless uh, automation. So, they can also control uh, who has access to those Kafka topics as an Iceberg tables. Um, we want to ensure that this is as easy as possible. Of course, we cannot really build on our existing group based
[33:39] solution because that would be just exploding uh, quite quickly. Um, and we also want to ensure I know there are like a beta features nowadays um, that even this row level security and the column level masking uh, would work cross engines.
[33:55] Uh, so we can actually query this from Snowflake or query from um, Spark open source. Um, it of course creates further complexities because we have gotten additional entry point for the system uh, but we are hoping maybe to configure this at least uh, for users to see
[34:13] anonymized data. Uh, so at least they can uh, see something. But I think the options are um, quite vast in that space. Um, and we also want building on this our mantra of the open data, we want to make um, data sharing
[34:29] as easy as possible. We want to automate this if the data is of quite quality, if it's named properly, we want even users to that that the magic happens and the assets are valuable for everyone. We don't even want our users to go into our um repository uh to open any pull
[34:46] request. Um but at the same time saying that, we want to prevent oversharing the assets. And the example of oversharing assets was this all account users, uh but also maybe uh test data and low quality data. And this is something that we want to uh
[35:01] stop from happening. I think there are no currently um elegant solutions for this. We might end up revoking some grants, uh but we are also observing the space uh what Databricks will bring for us. With that, I would like to hand over to
[35:17] Max. Yes, thank you Adam for the technical walkthrough through what we built here. Um let's let's give that some closing thoughts. Then let's tie that back together in terms of like what were we able now to actually achieve with all of that. Um starting from again like the very technical
[35:33] perspective, basically the tagline of like what we were promising you in the talk in the first place, um going with the data access from months to minutes, right? If I just look at the automation piece and you you see like a tiny screenshot here as well. Like like this now usually takes like less than 3 minutes, right?
[35:51] Add a bit of back and forth to that in terms of the actual user communication and the requests that are being raised there, uh and most users now have their access in 10 to 15 minutes, right? That is so different to the 44 days from the example that we were actually showing.
[36:06] So like like absolute massive impact like on our user base in terms of just dramatically simplifying and giving quick access to data. Now, we made a massive step very recently to actually give default access to all our corporate users to
[36:24] Databricks. Specifically because we're now engaging more also in in in terms of like Genie spaces like sharing of data bricks up sharing of dashboards so that also non-technical users can start getting access to that and that has led to an absolute
[36:39] explosion of the users that we have on the platform which of course makes it even more important that we manage that properly. So now we have like more than I think around 10,000 users that we've created through the automation already just about like 2 months ago or so
[36:55] and over that period of time like we actually saw like a like a 40% surge also in monthly active users from like around 1.5 1.6 data we had before to way more than 2K that we actually have now and that trend is continuing right so like the adoption of the
[37:11] platform is But the most interesting part is of course like what are all those users doing there now we are really tying it back also to a lot of the more cool and fancy features that a lot of folks are also here talking about at the conference specifically user driven Genie spaces
[37:27] right so like now we have at the time of submission which I think was about a week ago 56 Genie spaces that were actively being operated compared to two that we had in February right so like like this is something that is just enabled through
[37:43] the unity setup in the first place but also through the simplification of the access control out of those I would consider like 15 of them to be like at the level of quality that I would consider them production usage we still have quite some that are actually on the dev setup we have some that are actually
[37:59] just used for personal exploration and then of course we have also a bunch that were just recently created that are still work in progress. And last but not least just to quote like one one of those very specific use cases you you might have seen that thing hanging in Western Moscow and Western as well so now you know where that quote is
[38:14] originally coming from um the most important use case that we have running on unity today is how we actually managing our ethics spend for meta because 68% of that is now managed with AI analytics. So, that is like huge huge impacts like
[38:31] that's that's like millions of dollars that are being like processed through that infrastructure already and all of that is only possible because of the things that we have put in place from the platform perspective. That being said, thanks a lot everyone.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.