ViewShift: Dynamic Policy Enforcement With Spark and SQL Views

Summary

• ViewShift engine: Meet LinkedIn's dynamic policy enforcement system that integrates compliance directly into SQL views and data lake queries.
• Automatic routing: Learn how catalogs automatically resolve table resolutions to compliance-enforcing SQL views tailored to user context.
• Privacy automation: Discover how SQL views honor user consent, preferences, and regulatory requirements without manual intervention.

ViewShift: Dynamic Policy Enforcement With Spark and SQL Views

Talk by Khai Tran, Senior Staff Software Engineer, LinkedIn ; Walaa Moustafa, Senior Staff Software Engineer, LinkedIn
Dynamic policy enforcement is increasingly critical in today's landscape, where data compliance is a top priorities for companies, individuals, and regulators alike. In this talk, Walaa explores how LinkedIn has implemented a robust dynamic policy enforcement engine, ViewShift, and integrated it within its data lake. He will demystify LinkedIn's query engine stack by demonstrating how catalogs can automatically route table resolutions to compliance-enforcing SQL views. These SQL views possess several noteworthy properties: Auto-Generated: Created automatically from declarative data annotations. User-Centric: They honor user-level consent and preferences. Context-Aware: They apply different transformations tailored to specific use cases. Portable: Despite the SQL logic being implemented in a single dialect, it remains accessible across all engines. Join this video to learn how ViewShift helps ensure that compliance is seamlessly integrated into data processing workflows.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is ViewShift and how does it enforce privacy policies?

ViewShift is LinkedIn's dynamic policy enforcement engine that encodes compliance and privacy rules into SQL views. When applications query data through the Databricks Data and AI platform, the system automatically applies the correct transformations based on user permissions, consent, and regulatory requirements.

How are ViewShift SQL views generated?

This video explains that ViewShift views are auto-generated from declarative data annotations. These annotations define which users can see which data and what transformations apply. The system automatically translates these policies into SQL logic that runs on the data lake without slowing queries.

How does ViewShift support user privacy preferences?

This video demonstrates that ViewShift views are user-centric and context-aware. They honor individual user consent preferences and apply different transformations for different use cases. This allows users to maintain control over their data while organizations remain compliant with regulations like privacy laws.

Why is portable SQL logic important for ViewShift?

ViewShift's SQL logic is implemented in a single SQL dialect but remains accessible across all query engines on the Databricks Data and AI platform. This portability ensures consistent policy enforcement whether data is queried through Spark, SQL warehouses, or other tools.

Full transcript

So thanks everyone for joining and um today I'm going to talk about can you hear me? Can everyone hear me? Okay cool. So today we're going to talk about dynamic policy enforcements in data lakes. And before I start just wanted to ask how many of you have dealt with privacy uh either through enforcing it uh or building infrastructure to enforce it or adhering to uh its compliance rules. So many of you awesome. All right.
So more likely than not you have had days when you felt like our staff engineer here where every day he has to keep up with a lot of the um new regulations that come up in that domain. They have to implement them and catch up with them and at the same time they have a growing data lake with a lot of tables and applications that they just need to make things work in it. And this sounds like a very overwhelming problem.
And at the same time, companies that deal with uh user data, they have to uh gain users trust and then proactively implement knobs to put the user in the driver's seat in terms of their compliance and privacy settings. So such companies expose um userfacing settings and knobs such as u privacy dashboards for the user to review everything about their privacy settings and security checkups uh that run on a frequent basis to give the user reports about the status of their account and its security.
In addition to tools such as exporting their data or deleting their data altogether from the website or tuning their ad preferences to state like which advertisers can know what about them or which types of ads they can receive. So although managing compliance might sound like an overwhelming problem, the solution is surprisingly easy. So you just need to build two machines and one machine here is the policy engine on the left hand side and the other machine is the query engine and we probably have many of those in our data links but let's see how to put them together to achieve the required privacy and compliance.
So basically uh what we need to do is to define the policies in a policy language uh that encodes what types of users can see which types of data and what parts of this data as well in addition to the meta data about our tables in the data lake and feed those into the policy engine. The policy engine would take this information and then blend it together to produce SQL views that would encode the logic that needs to be applied to filter out the data that cannot be seen by different applications and by different principles.
Once the views are built then we uh feed them to the query engine which would take them up and apply them on the data and pipelines that are being executed in that query engine achieving compliance for every user in the data league that use those those query engines. Um but if you have noticed in the middle step between the que the um policy engine and the query engine we've used SQL views. So why did we uh think of views and why do we prefer them for enforcing compliance?
There are actually many good features about views. First they are expressive. So with tools like projections, filters and UDFs and drawings, we can implement so many rules and policies to implement requirements that are posed by the policy rules and convert those to SQL constructs. Further, they are portable because we can define the view in one SQL dialect and then execute the view on multiple engines even though they cannot understand that SQL direct through tools that can translate the SQL across dialects and we can see a detail of that down the slides.
They're also modular because they can be a drop in replacements for the tables that are intended to be filtered. As long as we create a view with the same schema as the table and the view filters data from the same table that we want to apply compliance on, then we can have the view become as a drop in replacement for the actual table. Further views are agile because they allow us to create view versions uh and we can roll out new policies with them or fix issues that we've observed in the existing views or roll them back to previous versions.
And this is just a simple view that shows us how to implement row level and actually also column level of a table. So here you see the input table that is uh called T1 and let's assume that there's a settings table where users encode what uh they want to have known about their data and then uh to satisfy each user's requirement individually then we simply join the two tables based on the maybe the user ID and then if the user says uh or allows the data to be seen such uh as expressing the consent equals allow in this in the settings table then we show the data unoffiscated but if not then we offiscate the data and then show it.
So this is an example that shows us how exclusivity in views helps us achieve advanced compliance rules where settings are stored in one table the data itself is stored in another table and then we implement the privacy at each user level. So more details about expressivity. We can see that views enable us to express column level filtering through projections or row uh or column level offiscation through UDFs or row level filtering through wear conditions and filters or actually both hybrid column and level and row level filtering through an a mix between projection projections UDFs and filters.
And not only that, it allows us also to express multiple ways of transforming the same data using multiple views. So you have one table on the left hand side and depending on the use case or the application that is accessing the table, you want to choose between different views that apply different ways of filtering based on what that application is allowed to see. So this was regarding expressivity. This is this slide is more on the portability aspect of views where views are really good because they are metadata that you can store in your metadata engine.
Whether your metadata store is the hive meta store or some iceberg catalog, then you can have your view metadata stored there along with the view text and then use tools such as coral to interoper interoperate the views between multiple metadata stores and multiple dialects and then target engines. In this case, for example, we can define the views in one dialect such as Spark SQL, but then execute the same Spark SQL in multiple engines by transforming the dialect from Spark SQL to the respective engine that the view is supposed to work on.
Here we can run the same view on Spark, Hive, Flink or Trino for example. So just to see an end to- end example of how this works and the general layout of the data lake with uh the views as a compliance um aspect uh and how it plays a role in applying compliance in a data lake. Let's assume that our data link has this set of tables from T1 to T5 and then we build a number of views for each table that offiscates the data in a different way.
So for T1 for example we have the view um U1 which is uh expressing the use case one and then the second view UC2 and the third view UC3. All of these are variations of the same of the of different way of changing data on the same table. So for example, this is the query text for UC1 and let's say we want to roll out those views into the data lakeink and again depending on the context. So our end users will have those scripts.
This is a spark scala script on the left hand side here and there is a bunch of tables used in that and we want to roll out dynamic data masking on those on those tables. So the straightforward way is to go and ask users, hey based on your use case, we want you to stop using T1 and T2 and T3 and so on and use the UC1, UC2 and so on views based on your use case. But migrating views manually is a very human and errorprone process.
We don't want our developers to go and manually replace replace each table with its respective view. Uh it's a slow and expensive process and exposes the context specific view names to the business logic. You you don't really want your user to be exposed to the policy and the encoding of the policy in their script and it makes things hard to evolve like how do we roll out the new regulation or how to roll out the new policy.
We're going to go again and ask our users to change their scripts. And there's a technical issue with that as well. I is that if replacing tables with views works for tables, it's very hard to work for views. So if the user has already used a view in their script, it's going to be very hard to replace that view with a view that applies the compliance on the base tables of that view. So this is where view shift comes into play.
So with view shift we keep the user script as it is untouched. User still runs the same script with the same logic with the same base tables but under the hood we replace the tables with the respective views. So in this case the user job would run on the spark engine for example and spark engine would connect to a metadata store or a qu or a metadata catalog that resolves tables and views but then intercept the regular call when resolving table name.
For example, in this case, we're trying to resolve T1 from the Spark engine to the underlying metadata catalog, which again can be Hive meta store or iceberg rest catalog. And then the catalog would respond based on the context information and what it knows about the application with the proper view name that would apply the privacy and compliance rules onto that table. In this case, the catalog would automatically resolve T1 to T1_U1 for example.
So even though the user sees in their application that they are accessing table one, what what runs on the execution side is actually a different logic that applies and injects the query rewrite on the fly. This principle is also applicable across engines. Trino also has a connector that also depends on metadata stores and allows them to resolve tables and views. Regularly tables would be resolved to tables and views would be resolved to views.
But in this case again we route table calls to view resolutions and from the user side they are working with a SQL resolving tables as how familiar they use them but then under the hood the view is actually being executed. So the benefits of that over human driven migration is that the process is very transparent. It is portable across engines. You can apply like build the infrastructure once and then have it run on multiple engines.
It allows users to use familiar names for tables. And it's more agile because we can e similarly to how we run or execute views to substitute tables. We can have views substitute other views and new versions of views substitute other like existing versions of views or roll existing views back to previous ones. So in terms of the architecture this looks like engines typically look like that including Spark for example.
So awi engine is represented by the dark blue square and query engines typically interface or expose plugins for users to implement the catalog implementation. And the contract there is that the query engine would hand over a table identifier to the metadata catalog and then the metadata catalog would resolve that to an object which can be a table object in the case of table identifier or a view object in the case of a view identifier.
But with viewshift what happens is that we can introduce the viewshift plugin within the engine table and view plugin as a plug-in within a plug-in and then when the call happens to resolve a table identifier from the engine to the table and use plugin Then that plug-in would call into the viewshift plug-in that would maintain the mapping between the tables and the respective views along with the way to resolve them given the context information.
So there is also a context map that is passed to the viewshift plug-in that tells it about the nature of the application and then enables it to resolve the table name to a a particular view name. Then this returns a view identifier to the tables and view plug-in which again naturally resolves the view identifier that was handed by the viewshift plug-in and handed back to the query engine. So this architecture is repeatable across query engine including spark and trino for example and these are the APIs that I just talked about.
So we have uh get view shift view API which allows us to translate a table name along with context information to a view name and then hand it back to the uh engine which is going to use them in the load table and load view APIs. So to see things in an the architecture from an end to- end point of view there lies our policy engine on the left hand side which again has the policy data encoded in a policy language along with the metadata that can be blended together to produce the views and then the query engine would use the views transparently to substitute for table references.
So as an example application of that let's see see on the ground how this happens in spark. So in spark we have these two applications one that is an ads application and the other is a learning application and the expectation from the policy engine is that policies should be applied on the ads use cases only and not necessarily on the learning. This is just a hypothetical example. Then both applications would issue the same query which selects data from the underlying demog demographics table and both also enable just one flag which is enabling the viewshift flag.
But then even though like both issue the same query here the learning application sorry the ads application would get different data compared to the learning application because the compliance kicked in in the learning case. So we can see that the learning the ads application had its table substituted with that view that we have seen at the beginning which offiscates the data and then the result would be different. So we see row one here for example in the ads application has been nullified but it's still plain text in the case of the learning application.
So this work is actually um published in that paper. You can check it out for more details about the implementation details and more details on the design. And this is also a link to the open-source coral GitHub repo where you can learn more about how to map queries between multiple engines and then use views across multiple engines. Thank you.

Learn more about the Databricks Data and AI platform.