Skip to main content

Advanced Governance and Auth With Databricks Apps

Summary

  • Advanced Authorization Models: Explore complex permission and access control patterns for enterprise apps using Unity Catalog on the Databricks Data and AI platform, distinguishing between app and user authorization.
  • On-Behalf-of-User Authentication: Implement agentic applications that enforce user-specific access controls through on-behalf-of-user authentication, enabling agents to act securely within defined user permissions.
  • External Integration Security: Secure external integrations using API-based authentication including personal access tokens (PATs) and OAuth flows, enabling safe data access for third-party applications and services.

Advanced Governance and Auth With Databricks Apps

Watch: Advanced Governance and Auth With Databricks Apps
Talk by Andre Furlan Bueno, Staff Software Engineer, Databricks ; Doug Judice, Software Engineering Manager, Addepar
Explore advanced governance and authentication patterns for building secure, enterprise-grade apps with Databricks Apps. Learn how to configure complex permissions and manage access control using Unity Catalog. We’ll dive into “on-behalf-of-user” authentication — allowing agents to enforce user-specific access controls — and cover API-based authentication, including PATs and OAuth flows for external integrations. We’ll also highlight how Addepar uses these capabilities to securely build and scale applications that handle sensitive financial data. Whether you're building internal tools or customer-facing apps, this video will equip you with the patterns and tools to ensure robust, secure access in your Databricks apps.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

How do apps differ from notebooks in terms of authorization?

According to this video, notebooks run as a single user with a simple authorization model, while apps always run continuously with multiple users and require separate authorization for the app itself and its users.

What is on-behalf-of-user authentication?

This video explains that on-behalf-of-user authentication allows agents and applications to act on behalf of end users, enforcing user-specific access controls within the app's security context on the Databricks Data and AI platform.

How do internal platforms handle authentication and user access?

Databricks Apps implement dual authorization where applications can act on behalf of users or themselves as service principals. Request headers pass user context throughout execution, preventing both developer privilege escalation and user privilege escalation. OAuth consent flows provide governance for intentional escalation scenarios on the Databricks Data and AI platform.

How are user permissions and access controls managed?

Access controls combine user entitlements with application permissions through Unity Catalog. Users can access only the intersection of what they are authorized to do and what the app is authorized to do. Service principals allow apps to access backend systems while maintaining audit trails of user-initiated actions on the Databricks Data and AI platform.

How can I know which user an AI agent is acting on behalf of?

User identity is passed through request headers in each app execution, enabling developers to audit and log which user initiated each agent action. Agents can inspect these headers to understand execution context and enforce user-specific data policies. This transparency supports compliance requirements and enables secure multi-user agent deployment on the Databricks Data and AI platform.

Full transcript

0:00 Introduction

Good afternoon everyone. Uh. I'm the tech lead for Dix Apps. I kind of built the product. This is Yes. Uh. I'm a uh engineering manager at Adapar. Um and I manage our data platform team built on data bricks and AWS. Yeah. And today we're going to talk about governance and authorization authentication with data bricks apps. Uh apps are more complicated than notebooks. Everyone in data bricks is used to notebooks. So we thought about like let's go into the details about this.
Um I'm going to present the conceptual and what we wanted to achieve with the product and then dog is going to present his experience of using it and how he uh he put it to practice in at a park. Sounds good. Uh if you have any questions at the end we will open up for a Q&A. It should be good. This is just so you guys don't trust me for anything that I say. Uh so notebooks versus apps. Um there are different authorization models.
The way notebooks work is that you have a single user script. The authorization model is like here's my laptop. Go and run this thing. Um it runs from beginning to end. It ends that simple all it is. Everyone data bricks know what a pat it's funny that we call it pat token but it's just personal access token. Um apps are slightly different single the app is always running. There are multiple users at the same time in the app and the app itself has other things that it owns goes to the database or not and that is not is so we itself has its own authorization.

1:06 Apps vs. notebooks

So you have the users authorization and the app. Now we're talking a lot about agents. I put the little robot agent there because a lot of apps are becoming agentic. So we have to think about apps as their own users in this context. Um the way I like I like to think is that either during an app you uh server you either act on behalf of the end user of the app or you act on behalf of the app itself. U so kind of like setting up.
So if you see um in a lot of our competitors there's like they try to simplify. I actually did not. I tried to keep it as open as possible. Um let's show you a little execution that is like the basic thing that you can do in an app. You would need those two authorizations that you go to a database read the user session. The user have no business having authorization to your database is the apps database. You use the app service principle for that.
Now you go and create query SQL warehouse. You have to own a unit catalog the authorization. That way you have to act on behalf of the end user because you want to make sure that the data is secure the way from beginning to the end. You get the result set you save you change a little bit the user session with saving the DB using the service principle again. So doing a single request in the app, you use two different authorization models.
And that's very powerful because now you can actually build full-blown applications. They're not toys to dashboards anymore. They are full-blown applications. Um and that's one and because of the multi-user and the identity of the app um you we every app gets its own service principle and also uh we pass on the via headers the information of the specific user because you know the context of the user is only throughout that request and the request headers is how I pass app developers don't need to worry about how theation authentication works and all that they just need to know what they need the app to do um and get the request.
So just a little summary here notebooks is single user group notebooks just runs app doesn't run app starts and then whence it starts to getting requests that's when you need to think to write code to know like I'm going to use the app authorization or the end use authorization. Cool. Um another very important concept in everything that we do in this is about privilege escalation. Um the goal here is to prevent unintended privilege escalation.

4:31 Governance and security

You can be uh a little idealistic and say that we never want privilege escalation. But actually in practice there are a few use cases that you want some privilege escalation. I'll explain what but that's the goal and it's very important concept um any user in the system cannot have access and use apps to be to work around the authorization that they already have apps is not a way to do that uh so and I think apps is also different to notebooks because of the two you have to think about privilege escalation from the developer and the user the developer cannot get more access because of a user logs into the app and the user cannot get more access because it logged in the app.
So it's both ways that we need to think. Normally I see a lot of customers think in only one way but you have to think about both. So how does it work? Uh the principle of list privilege is that developers have a set of permissions. Uh an app has a strict subset of those permissions. an app does not get the whole permission of the user of the creator admin creates an app now the app can change IP aos in the workspace that's not the case should not be the case so only what the app needs to do is grant the permissions to uh the user on the other hand has a not a subset there's a different but there is an intersection of the user's permission on the app permission And in an ideal scenario, an app or the user would only have access to the intersection.
So there is no privilege escalation and that goes from assets to rows, right? So everything um however full intersection is impractical in a lot of the cases and that's why we added um OOTH consent. So I I'm creating an application platform. I'm focusing on the authorization model the developers know that's oath. I don't not going to reinvent the wheel here. There's already a way to do this. But for example, let's say the developers need to create an HR application.
They would not have access to every row in the HR table. Uh but the users of that application might want to have access to rows that the developer never had to. So at some point there is some need for privilege escalation and the key part here is the two is that governance on how that app gets deployed and developed through you know get source code controls and reviews and whatnot and the other is that you need to when you're interacting with application and application wants to act on your behalf you need to be given a chance to think about it especially if you're a power user you know an app comes and say you know Just typical app store stuff.
You start an app in your phone and the app is like I want to use a camera and you're like this is a sudoku app and same not same in data bricks instead of the microphone or whatever it would be like oh this app wants to read your secrets you know like you know give a little thinking about why this app wants to read your secrets. Uh it prevents a few other things too is like you can create little go links or little short links share with someone if there's no consent is kind of like you can do some very nasty stuff like fishing things and I that's just not a thing uh for apps.
Um so in general I think think this is what I wanted to build which is like a powerful authorization model that you can actually create full-blown applications but we have a long way in transparency in data bricks to move from batch notebook authorization to an app authorization. Our model is just starting. I think I think oath is only like two years old in data bricks and IM takes a long time to evolve in a company this big.
So we have a long way to go but that's kind of the direction that we want to go. Now I want to welcome dog to go through what how does it look like in practice. All right awesome. Um hey everybody. Um so how I'm going to kind of go through this um first I do want to explain a bit about what Adapar is. Um we're a financial services company. Um we sell software to um asset wealth managers. And I think that's really important because it highlights how why governance and the kind of authentication model we're talking about is so important because we deal with a lot of sensitive client data.

9:08 Addepar case study

Um I apologize some of these might look a little like marketing but again I think it's important to kind of show our journey. Um then I'm going to talk about the apps that we're building inside Adapar to really facilitate um getting more data into our company which is kind of like our our life and blood. Um so like I mentioned uh Adapar is this is some nice like you know marketing approved screenshots but this is kind of what our product is.
It's a reporting and analytics app. Um we do uh a lot of other kind of products like trading alternatives data management and document ingestion. Um but really what we're trying to do is aggregate a lot of data from a lot of different sources for our clients and present it in ways that they can kind of tell stories to um the kind of cl the clients that they manage. Um just again who are we dealing with when I say like who are Adapar's clients um we service some of the largest kind of banks in the world. um you know and we deal with a lot of international clients as well and that's again very important to how we kind of have to think about approaching using something like data bricks apps because we do have to deal with things like building an EU um you know adapar's global data centers how do we kind of keep ourselves compliant with things like GDPR and any other region that we're kind of scaling to.
Um there's a lot going on in this slide but really again I think it's just important to highlight why we use data bricks in the first place. Um you can see it's kind of central here and I think that's the main takeaway for everything on the outside rim and that's what adapar is. It's all the products we have. It's all the things that we sell. um these are things making us money but at the end of the day um Adapar is basically um our data and data is very kind of intertwined with our DNA and it's something that again we've really kind of consolidated everything onto a central platform and I think you know one thing I've kind of discovered since um we've been working with on apps and everything else and kind of starting to use them is I found that they're very very compelling to operationalize a lot of things that we want to do in a very um you know complex uh ecosystem managing data data.
I'm building more pipelines and getting things in faster. Um, again, you know, kind of this is our view of the world. It's um trying to expand as quickly as we can and build as many feeds as we can. And that's really the hardest problem that Adapar has to solve. Um, we want to expand to different regions and we need to in doing so we need to work with different um they're called custodians and different maybe market data providers that we have to call integrate into our core platform.
To do that we have a very large uh engineering analytics and operations team that kind of manage um 500 plus pipelines every single day kind of working to bring data in transforming it to adapar's common data model and then making it available to our clients. So these are um again because we're doing this globally a tight governance model um with something like UC has been kind of foundational to support this at a global scale.
And this is really what you know our vision for getting the feed build process look like is it's a pizza tracker. We want to be able to make the feed build process so streamlined and so easy that it just has different steps that different people follow in different stages and we end up with a data feed at the end of the day that's ready to get production data into our into our core product. Um, Adapar deals with a lot of very complex ETL kind of uh pipelines.

12:28 Scaling with applications

It's not easy um to go from those 500 different kind of integrations I mentioned to one common data model and building them quickly has been kind of our biggest challenge. Um and one of those challenges I think that's the most you know kind of integral is actually just you know we actually saw today in the keynote how do we have the analytics you know the business domain experts work with data engineers and kind of partner together to build these um and one thing that we've found to kind of uh be a very compelling way to kind of streamline that so far has been data bricks apps and really what we want to do is democratize the way that we build feeds on data bricks using data bricks apps and using again this very kind of tight governance model.
Um, and we're putting pretty much everything kind of consolidating it together into data bricks apps. Um, but that's not exactly where we started, of course, because data bricks apps are fairly new. Um, like was saying, we first started kind of operationalizing and making things easier to do for non-technical users using notebooks, which made a lot of sense at the time. um they're very good as scripts and we really started to get into this habit of multiplying them very quickly.
You can see the face is getting a little bit more frowny. As we kind of made our, you know, bu feed build process more complicated and we had to handle more corner cases, getting different kind of feeds kind of online, we started adding new notebooks so that analysts and again nontechnical users of our platform could do more themselves and it kind of started to multiply like this. Now we're managing production configuration tables using notebooks.
We're using we're driving analytics using notebooks and a bunch of widgets kind of stored at the top and it started to really kind of fall flat and we weren't really sure exactly what to do because we ended up with a you know folder in git that had 10 notebooks new people were getting onboarded they didn't know which ones to go to they also didn't know which ones they were permission to use because we do have a lot of different personas in our you know UC groups and uh bill and uni catalog and everything where some operators would be able to modify production configuration tables.
Um, some were just analysts and wouldn't be able to do that. Um, some were developers or fees engineers wanting to run like a pipeline potentially. And having a bunch of notebooks like this is not a very conducive way to run anything actually. Um, and that's where data bricks apps come into. So what I'm show showing here on the screen actually are some screenshots of our kind of internal data bricks app that we were building to kind of consolidate all of these features together under one roof.

15:15 App features and UX

Um it's a again being deployed to multiple regions. We're expecting it's you know kind of peak kind of usage will be around 50 60 users per day and we're supporting a lot of different personas at once and I'm going to walk through again just because it's interesting. I don't know if uh how many people in this room have built a data bricks app but I want to kind of again give some insight into what we've built. I think that could inspire maybe you know we've seen just how easy it is to make an app these days. uh maybe that can inspire people to help solve problems um at your solve your own challenges that you have at your own company.
So on this slide, we're looking at a um way to do ingestion easily into data bricks um just through a UI. And this is something that again I find very compelling because in our world we have to deal with a lot of different proprietary, you know, different data formats. We deal with JSON files, XML, um CSVs. They have to be parsed differently. We of course all know there's different ways of delivering CSV files and everything.
So we wanted to streamline that ingestion process. Again, what used to be a notebook with I can't even imagine how many different configurations baked into it is now being reinvisioned as a single pane of glass in an app. And one thing that we are actually doing um that I think is, you know, again kind of important to show here, I'm trying to use the laser pointer, but in the bottom right, we actually run a lot of AI and ML processes on top of this data to kind of help analysts move things through the pipeline more quickly.
Again in that idea of there being a pizza tracker, we want to reach uh insights and get to production data as quickly as we can. So we do a lot of automated things like schema mapping um kind of again mapping different proprietary ways of modeling securities uh to our own kind of internal format. And these are things that analysts can just do and run as buttons instead of it being going to another notebook or um you know figuring out which compute to attach to and everything else.
It's a much more streamlined experience. This is another kind of example of something we've built into the app which is um you know behind the scenes um we're using Genie and a lot of other AIBI tools but how do we talk with the data that we have and kind of again get analysts up to speed quickly on what is going on with new data we ingest. It's no surprise maybe that with over 500 feed builds we get a lot of very crappy data.
So we do have to kind of work back and forth with those providers to understand decisions they've made or why something looks a certain way. And we're accelerating the process of kind of getting to the insight of understanding what's wrong using AI faster. And you can kind of see here, it's maybe a little hard to see, but we have a question we're kind of asking like why are there so many transactions with zero cache value.
So that's something that again we're plugging into um SQL warehouses, we're plugging into Genie, and we're figuring out a way to kind of bring everything together into one place um again under one roof as an application to show back to the user. And lastly, again, this is kind of showing off the fact that we have again multiple personas kind of operating under one roof, but we do things like modify configuration mapping tables um in production using again um a person that would be kind of provisioned to do that.
So, an analyst who's working on a feed build and maybe using some of those other tools wouldn't necessarily have the ability to go and modify again this table that we're looking at right here. And we're doing it through a UI. what used to have to be done by uploading a CSV potentially and syncing that to a you know delta table. Instead, we're just giving them kind of one um easy experience to kind of do everything again um immediately just with the UX, which um people have been, you know, I think every one of the people we demo this to and show it to and are rolling out to as like a beta, uh is obsessed with the fact that we have this ability now and it's made so easy um by again on behalf of uh user authentication.
So, it's again very important to us that we have this feature now. In fact, I would say the app that of our dreams that we wanted to build wouldn't have been possible until features like this came out. Um, we definitely have very again tight controls on who can access what type of data. And being able to very easily take a user's token in that request header mentioned and just pass it through to any kind of requests or anything we're doing in data bricks makes it much easier and much safer to do this type of development and make uh these and kind of democratize things and make it things as so self-service as we wanted them to be basically um again this is an example of dealing different geographies but um you know I as a US employee couldn't go and run anything on a SQL warehouse as my user in the EU on client uh EU data um an EU analyst could and these are things that again are kind of showcasing this is one example of many um but we're really showcasing again I think the power of you know we wouldn't be able to do this without on behalf of authentication and until we could do that something like this would have been difficult to build um we can't just give our service principle that is backed by the app permission to a ton of data um it aimlessly in the in uh our unique catalog that would have been a no-go from any kind of governance standpoint.

19:12 Advanced authentication

Um but what is great is that we do have this kind of split head model where we can run some actions as a service principle. So in our app um we want to try to asynchronously collect feedback and we actually do um until uh the announcements kind of yesterday we're managing user session state and UC volumes as well but we can run some actions as a bot basically and that's a service principle backing the data bricks app being able to easily flip and switch between a user running a SQL warehouse getting uh maybe some feedback from like that AIBI genie or anything else and then easily being able to submit feedback for us to submit back to the volume as a um system user is again critical because there are definitely some actions we want to run as that you know system user as that service principle um for like kind of more monitoring auditing collecting things but again that service principle doesn't have any access to our actual data and that's really enabled again us to build this type of app it's enabled us to feel a lot safer about how we roll it out to anybody and um again it's giving us a lot of flexibility in how we develop apps and how we think about them as really complex um all-in-one solutions to drive processes um as analysts and those less technical users at Adabar.
Um so I made this slide maybe a month ago and I think it's already pretty out of date. Um data bricks moves very quickly. Um but I do think it's still kind of important to what challenges we saw at least at the time when we were developing apps and that was difficult state management. um which I think you know will talk about and that's something that I think is being solved with things like lakebase but um we do have as you know apps are growing complexity managing that session state was important it was a little difficult to do with the stream with which is a framework that we were using um and again I mentioned we had a solution we're kind of saving all the stream state as a JSON blob and pushing it to UC volume and loading it was hacky for sure and I think I'm we're going to be very glad to move to something that's like link based um sharing ing apps more easily.

21:47 Current app challenges

I think um I stole this uh it's I didn't come up with this but going from URL to data um again for our entire organization is something that I you know again have a big vision for. I think we're starting with this kind of feed building um data bricks app but my goal is to move to a lot of other problems we have to solve it out of par and it's very hard to imagine um an easier way of an easier way to kind of do URL to data and show insights with these types of more complex apps than again data bricks apps but getting to a point where we can actually let users at aar provision with something like octa which we use for SSO without actually needing to be in the datab bricks account is something that would be convenient And lastly, I think what's always important to us is just more system and audit tables and observability.
Um, as we build much more complex apps and they have different requirements from our users on uptime and bugs and issues, it's going to be very important to make sure that we actually do collect again who is using it, what features are getting more more traction, and um again, how do we make sure that they're actually running and that the user is having a good experience essentially. So that's kind of my portion about how Adapar is using data bricks apps.
Um what we've built and again why um being again a very financial services forward company I think the governance model that we have now with on behalf of has been so critical in our journey of actually being able to build it. Thanks. So like dog would say like and we have lots more to do. Uh the three things that I'm working on right now that are like top of my mind uh is at some point we need to add external IDP integrations.

23:41 Roadmap and future plans

Um we need to make sure that use your users in your organization if you use octa abon github I don't care um don't even need to be in the account of data bricks to be able to use application apps data bricks apps u the thing stopping this is that somehow there needs to be I need to build a system to do a token exchange to a datab bricks uh authorization for data so those users are not known to data bricks what data how does that map to you see?
So I need to do some work on that side. Um there are some some ways to do it but uh it is work. Uh the other thing I need to I really want to do is enforce git optionally today. You can write the file in the in the workspace and just deploy from there. um if you are dealing with some highly sensitive data and applications that are touching some stuff that you you need to have more governance over that and I think one of the best best ways that I can do is just follow the normal standard development process that you guys already have.
So we enforce that and say like you have to have a proper GitHub you have to go through the CI/CD pipelines that you already have and treat source code as of data bricks apps like any other source code in your organization. Um and the third one is we need to have much much tighter controls of the on behalf of ideally the way I want to get this is that the most useful on behalf of authorization will be just the data you don't even need to give authorization to the warehouse for example the warehouse is delegated from the service principle and you can collect it down to the scope to say that this while for this user while using this app the app can only have access to read this catalog or this table go down to that level and mix permissions with the out scopes.
So I want to make yeah so that's going to be a little road map but uh but that's the goal and of course uh there's a lot more that we have to do with audit logs and system tables but not just that I want to make sure that you guys can log your own stuff and go fast your workspace and have your own observability of apps. So we're working on like fast ingestion of open telemetry data. So you can instrument your own app however you want and have your own standards of how to do telemetry with applications.
And like this is top of my mind and I know that um governance and security are very hot topics. So go to town, ask me anything because it's controversial sometimes.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.