Skip to main content

Unleashing Data Governance at iFood:Harnessing System Tables and Lineage for Dynamic Tag Propagation

Summary

  • Dynamic Tag Propagation: iFood uses Unity Catalog system tables and data lineage to dynamically propagate classification tags for compliance monitoring.
  • Regulatory Compliance: The system helps organizations meet regulations like LGPD and GDPR by managing sensitive data access and preventing breaches.
  • Scalable Governance Framework: DLT pipelines with system table ingestion and permission monitoring enable proactive risk management at scale.

Unleashing Data Governance at iFood:Harnessing System Tables and Lineage for Dynamic Tag Propagation

Watch: Unleashing Data Governance at iFood:Harnessing System Tables and Lineage for Dynamic Tag Propagation
Talk by Lucas Corsi, Staff Data Engineer, iFood ; Pedro Henrique De Carvalho Teixeira, Head of Data Platform, iFood
With regulations like LGPD (Brazil's General Data Protection Law) and GDPR, managing sensitive data access is critical. this video demonstrates how to leverage Databricks Unity Catalog system tables and data lineage to dynamically propagate classification tags, empowering organizations to monitor governance and ensure compliance. The presentation covers practical steps, including system table usage, data normalization, ingestion with DLT pipelines and classification tag propagation to downstream tables. It also explores permission monitoring with alerts to proactively address governance risks. Designed for advanced audiences, this video offers actionable strategies to strengthen data governance, prevent breaches and avoid regulatory fines while building scalable frameworks for sensitive data management.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is iFood's scale and business model?

iFood is a Brazilian tech company that connects people to meals with over 110 million orders per month, making them the leader in Brazil with millions of monthly users, drivers, and business partners.

How much data does iFood manage?

iFood manages 16 petabytes of data lake, over 15,000 tables in their Unity Catalog, and more than 800,000 machine hours processing data per month.

How did iFood implement comprehensive data governance across their lakehouse?

The company leveraged Databricks system tables to gain visibility into data usage patterns, access patterns, and potential governance violations across their organization. System tables automatically track query lineage, data modifications, and access events, enabling the data governance team to audit who accessed what data and when without implementing custom logging. This approach enabled iFood to establish baseline compliance metrics, identify data quality issues, and enforce governance policies at scale across thousands of tables and hundreds of users.

How do system tables enable data discovery and metadata management?

Databricks system tables automatically capture metadata about all data objects, including creation dates, modification history, ownership, and access patterns without requiring separate metadata management infrastructure. Teams can query these system tables to understand data lineage, identify stale datasets, discover tables relevant to specific business problems, and track data quality metrics over time. This automated metadata collection eliminates manual catalog maintenance while providing comprehensive visibility into the data landscape.

How does iFood balance data governance with enabling analytics and business use?

This balanced governance approach uses system tables to identify valuable data while enforcement policies restrict access to sensitive information, enabling analysts to derive insights without exposing raw personal data. Role-based access control combined with column masking allows the platform to grant broad access to analytical views while protecting truly sensitive information from unauthorized use. This demonstrates how strong governance enables rather than restricts data-driven decision making when properly implemented.

Full transcript

Good afternoon everyone. Uh I'd like to thank you all to be here today. It's really a pleasure. I'm super excited to welcome you guys to this data governance session. We are going to explain how we are using system tables and lineage information for dynamic tag propagation. So I'm the head of data platform at IOUT and I have here with me uh I'm a staff data engineer at IOU. Okay. So the data platform teams take care everything that people uses for analytical purpose and data governance.
It's also uh something that we are responsible for. So uh before I start talking about I um data governance, I'd like to take two step backs and introduce you guys to iFood. So I food is a Brazilian tech company. We and we are not only a delivery company. We are pioneers in Brazil to connect people to meals and to create this uh very seamless uh customer experience. So with more than 110 million orders per month. We are the leaders in Brazil and we connect millions of monthly users to uh drivers to business partners and we operate across all regions in Brazil. uh actually data it's at the heart of everything that we do.
So data feel innovation we feel uh products and how we create this uh ecosystem connecting deliveries drivers customers and restaurants pet shops and so on you name it. uh to give a better perspective of how many data we handle in a daily basis, we have more than 150 proprietary AI models resulting in 14 billions real-time predictions which makes our delivery routes better. We reduce errors and create this uh excellent uh user experience.
We have a 16 pabytes of data lake. We have more than 15,000 tables in our unit catalog. We do a lot of retraining. We have uh more than uh a thousand retraining and model updates per month. In Brazil, we are the top uh one of the top consumers uh data brick consumers. We have more than 8 8 thou 8,000 800,000 machine hours processing data per month and we already observing a huge impact of AI in our revenue. So we estimate about 30% of uh eida impact only by uh AI.
So to dive a little more into our data governance topic which is very important to us because this needs to work within our rules, we have a lot of uh uh uh guidelines uh and policies that people needs to conform to. I'd like to uh pass to explain why we did the dynamic tag propagation. Thank you. So the object of the session today is to explain a little bit uh how we can use the lineage and privilege tables in in catalog system tables and we'll talk a little bit about reactive and proactive security strategies and how we can ensure transparency and also about uh how we can identify who can access which kind of sensitive data. uh even for confidential data.
Uh so also how we can audit and do monitoring and classifying uh the table sensitive tables in the downstream uh in the downstream. So uh data governance is a big challenge uh for us I guess for every uh company and also because it's hard to track uh who can access uh uh sensitive data and what kind of permissions they these users can can have. So uh if you imagine if you have a big team with thousand of users using accessing data with a lot of autonomy to al also for read the data but also for write uh their own data sets.
So it's if you have a one single cing to do that it's impossible to scale this. So it's in part to share uh those kind of responsibilities. So uh so data classification uh is easy to be automated to with PII data but it's very hard to do that with confidential data and I I will explain a little bit more in the next slides and also because for example uh do it to uh this kind of this level of autonomy that the users have uh it's uh sometimes the users uh for forot forget to how for forget to set the appropriate tags for the tables and it's difficult also to track the confination sensitive data in down stream and also when you have more than one workspace the lineage view that you have in data bricks um they don't match um the lineage between different works workspace.
So each workspace has uh their own view of the lineage. So uh for that um what what are system tables in a catalog? So sustain tables basically are delta sharing tables hosted by datab bricks and for our solution that uh we we going to present today uh we use this main uh this this uh those important tables. So uh the lineage table is the to track the downstream and the privilege tables to identify who can access which kind of sensitive data and their privilege and everything related and also the tag tables to in a way to label uh with the classification of PII and etc.
So uh here um is so system tables are a data bricks hosted analytical store of our accounts peritional data found in the system catalog and system tables can be used for historical observability across our account. This is the datab bricks documentation. So uh at first um let's talk uh about uh proposed workflow for tag propagation. Uh at first we need to enrich the data uh normalizing it simplifying it and in which to then pro process that.
So this diagram uh represents um u an overview of the our DT pipeline. So uh as you can see first we have the table lineage here uh we ingest the table lineage and we split in two DT views one call it matched lineage and another m unmatched lineage. The first one is where is where both source and destinations are in the same row and matched lineage. I will explain better uh after and then we we can merge them um together uh back uh join with the table tags from system tables.
Then we have the output table called lineage. In for with this output table we can split into other uh DT tables. One call it edges and vertices to use for our uh graph processing. Uh also the lineage table we can use uh to control the offset uh that we you use uh to next iterations to uh passing uh as start time stamp to do the ingestion incrementally. And also we have this another step. Um uh here we have the skin API and with that we can load uh and generate an output table called it user groups uh that I will explain uh in the next slides.
So at first we need to ingest the dimage system table incrementally using rich uh data gen feed and uh system tables in general supports rich gen feed natively. Uh but we have some limitations. Uh in this case we needed to ingest the data using batches. uh because if we try to ingest the tables uh with streaming and read change feed configuration, this can limit us uh in how we need we need to aggregate the a matched lineage block and also to avoid for example having a a big um uh sorry uh we have a big uh watermark session.
So, so to do that and to better scale this solution, uh we we have this limitation. So you need to do in batches and also uh in DT pipeline the Python API um doesn't support batch reading uh in in in lineage table because it's a uh delta shared table and we need we need in this case to use the SQL API table change that does the same uh effect uh and then we need to ingest the scheme users groups serves principles to rule the skin API.
Um but uh I will explain better. So at first um to uh normalize and simplify the the table lineage ingestion at first we need to identify uh some dirty data that we can have in this table. Uh for example the prefix of S3A, S3N etc. and also uh some uh informations that are not useful for us for example partitions and etc. So as you can see in in here uh we have S3 uh prefixes. Oh come on. We have S3 prefixes and uh partitions with timestamp information that are not useful for us.
So for the lineage view block uh the first block that I show in the diagram uh at first we need also to f filter out irrelevant data where type is dashboard or or new because they are not useful as well. Uh then we need to apply qual um and the in the source table full name column and the source path column to simplify merging them in a single one column uh that we can call a source and do the same for the target. So we can have uh less data to uh we can make our graph processing easier.
So also we need to use uh we can hax patterns to remove dirty data like file name is date formats uh uh hashes etc. And we can replace with a placeholder. And as I said before, we can normalize path schemes. And then we need to remove uh rows where sources and destination are equal to avoid uh cycle lineage in your graph. And then the duplicate the data. So uh as you can see here um we have the S3 prefix normalized and the DT uh timestamps replaced with a placeholder uh that we call it DT and it can be anything else.
Um so for a matched lineage view uh the second block that I show in diagram uh at iPhone in our task orchestrator uh we have 8 hour timeout policy. So job is uh must run um before uh reach 8 hours. So to better track uh potential lineage breaches for example when the users uh when the users uh read uh some data set and then collect them to the memory and then transform back to data frame and write in another place. uh we can try to track the indirect lineages for example um where we know that are the same notebook or the same job gener reading and generating that data.
So in this case um we need to filter only when one of the fields source or destination is empty and the other is fillet and then we can group them with the entities primary key for example workspace ID uh the entity type ID that can represent job or not etc. And then we can group uh with the session window a spark function with a session duration of 19 hours 8 hours from our uh our 8 hour timeout policy and plus one hour for have um a margin of safety.
So here it's possible to understand better how we can group the those kind of events. So as you can see the first two rows here uh have all the same uh information of job ID etc. Oh, and as you can see, Cersei is filling in the first row and destination is filling only in the second. And we have less than 90 hours of difference on event time. So they they are the same session window and the same job, but they are unrelated.
So we can group them together. Uh the third one row is alone. So it's not useful for us to identify who is the what is the origin the source of this table C and the last two rows uh are the same case of the first u but we have another entity ID another run ID. So we have even if uh we have the same time stamp but um in this case is another job and but they are those both rows uh are in the same session. So this is an example of our glT pipeline uh like the gram that I showed before.
So at first we need to consume it uh for user groups uh in order to have the user groups uh table uh the output table we need to consume the skin API listing the users and service principles uh then we can collect those data in memory and convert back to data frame uh also we can have a hard column uh type to identify if it's a user or serves principle only because they are in the same table and uh like the ent and the entity ID uh but in this case we have some limitations because uh the skin API on account level uh returns empty for the groups field so it's necessary to use the skin API on workspace level uh so if you have more than one workspace you need to iterate through the skin API on every workspace that you have.
So here an example of the user group DT pipeline and um this was the user info table contains the relationship between the users and groups um even when they the groups are inherited is not direct and also we use this table after to do the monitoring uh to do uh the alerts and everything else. So this is an example of the output. Uh here we have the hardcoded column to identify the type. Uh we have the name and the male and group information and if is direct or indirect.
So um then we need to uh pro uh we have with the data uh normalized and simplified we can uh go to process the data. Um and we need to generate the edges and vert tables and then we we can load them in graphics with a custom gel uh computation. So in this diagram represents some overview of the graphics uh processing. This is not a DT pipeline. It's a job pipeline. Uh so we have the lineage uh vert and the edges. We use as input of graphics.
Then we do a prejel calculation here. Collect the vertices and then we save uh as final uh table call it table classification. So here is how but we how we can do uh the tag propagation and how it works. Uh in this case for example uh imagine that you have only the A and G nodes uh tagged with some kind of tag call it PII or or something similar. So in this case A is PII, D is public and G is confidential. So in this case every every other everything else is not classified.
So we can infer that uh C nodes and and B nodes um and B node are PII because A is PII. Also the E node is public because D also is public and this case F uh is generated from C and E but in this case PII is more relevant than public. Uh so F node will be public as well. And here uh the last node um is generated from PII and confidential uh classifications. But PII is more relevant than confidential. So we need so we classify as PII.
So uh to do that at least at least we need to have the first table in in the lineage in downstream uh tagged. So if you can tag this the search table uh you can do that and we have a lot of ways to tag tables in any catalog. uh doing it manually or tagging through some automation using a PII auto discover tool. But tools like any kind of uh autodiscocovery tools like data bricks house monitoring in Brazil you may see they work well for PII data but um they don't work well for confidential data because uh this kind of um this kind of data they don't follow patterns.
So the rule is uh of the classification of the confidential is more related to the business. So also uh how we can handle uh in cases that we have the table secured for example the users um set a mask function for example in in in the table. So we can create another tag call it uh for example security and classify this uh this table as manually as public. So as you can see the I node is confidential and the J node is public and in this case the K node was treated and for because of that we can uh consider that L is not anymore um reading a a confidential data.
So the lineage is truncated here and the L node will be public as well because it's not using the confidential anymore. So about pre priel algorithm uh it's a platform for parallel processing of large scale graphs and was developed by Google and it's used it to calculate page rank um and then we have a lot of advantages using graphex like scalability, simplicity and efficiency. So uh about uh the PJ algorithm has some phase like the initialization where the each ver vertex starts with initial value or they and then uh they receive and update the vertex message.
Then they send the message to next neighbors and then they iteration that is repeated until a convergence condition is met. So those are the graphics per gel parameters. Uh this it's from the project's um documentation uh the graphics documentation and al but this is important to mention that edge reaction that controls how messages flow between the nodes. In this case we are using the out mode and the V program is I guess is the core U function that we have because the this function will update the vertex state.
So uh the workflow works like this. Uh we have the initialization. Uh so at first uh the vertex will receive the initial message uh value. The vro applies the first update. the iteration. Then uh the send message function uh will pass the message through uh through the neighbors and then the merge message will combine those messages uh for each vertex and var will update again the state uh based on received message. Then um we have a stop a stop condition when no more messages are sent or the maximum numbers of iterations is reached.
So this is an example of code uh that uh we did uh so it's important to note it that uh here we can have u uh a if clause where the message sensitive uh has an order and the question that I showed about the relevance between PII and confidential and public can be treated here. So this this is most this is the most important part of the code. So also uh here is the send message function. Uh the if uh is important also here because when we need to not send the messages to next nodes, we can use the iterator amp to stop the the propagation.
And here at the merge message if the cursive message uh the from the parent node is more relevant than this the target node. So we can keep the most relevant message and if else we we we not. So this is the an example of the output table that we have. As you can see we have the catalog schema table name and also we have a column call it data classification that can we we will have the label of the table here. uh also about a column called it secured.
For example, when we treated uh the table uh with masking function for example the this security will be true and now about monitoring and how we can do monitoring with that output table. Uh so in this case we have reactive strategies and proactive strategies of security. Sorry. And reactive uh security focuses on respond to and mitigating treats after they occur and proactive um emphasizes prevent treats before they happen.
So uh we can use both strategies to handle u our um uh uh to do our monitoring uh above the table that we generated. So with users in table we have the relationship between users groups and also with the privilege tables we can track uh which kind of table of permission the users have in in their tables. So with that we can generate alerts. Uh we can tag back the tables. Uh we can use tools like discover X from datab bricks to tag to tag tables back.
We can revoke grants and we can apply masking and also it can be used with a a back feature that datab bricks is releasing. So this is an example of um of the alert on on the slack. Uh we are only generating alerts but we are not revoking automatically. So the with that the user can take some action. Um so also it's possible to track uh opening issues. We can do this extra table for example uh with the table privilege we can identify if an user has access and which kind of privilege the user have in in that table and if if doesn't match our rules or our defined rules we can uh track the status as opening security issue or for example.
So why this is important? Um so this allow us to scale the company uh avoid uh uh avoiding having PII or confidential data open to the users and providing more autonomy to users. Uh and it's important to emphasize that because um confidential data doesn't work with the auto discover PII tools. So uh because business users uh will set the that this kind of data is is confidential or not. So uh with that we can create a sense of responsibility uh among users uh because can serve as form of education. uh for example if users uh receives alerts uh they can learn about the concern of having PII and of having uh confidential data exposed to the company.
So we can reduce risks and problems when are detected. Uh applying masking automatically or maybe revoking access uh in reactive security way. So also we can generate audit reports for privac for some other purpose. So uh to finish um uh we have uh some takeaways and recommendations. Uh if you need to users not create downstream tables for example they they don't don't need to have the autonomy to read a PII data or confidential data.
It's important to create a PII workspace uh with catalog binded in read only mode. So they can read the the tables but they can't write uh in another place. So it's important to mention that lineage table from sustain tables has a retention of one year and also the first lineage history ingestion might be heavy and then uh you need you in this case you need to use a a bigger computational cluster to process that and after you can uh decrease the size of the cluster or or even use the serless So also uh because we run in batches you can adjust the job schedule as you need.
Uh so uh and also because currently service does jobs doesn't support don't support Scala. So libraries like graphex and gel graph frames um are not supported because they are um created in Scala. So for to use and process with graphics you need to mix them. And thank you everybody. Feel free to reach us at linkading.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.