Secure External Data Access with Connections, Secrets, and Functions in Unity Catalog

Summary

Modern data platforms require governed access to external databases, APIs, and cloud services, but hardcoded credentials, scattered API calls buried in user code, and lack of visibility create security silos and governance gaps that Unity Catalog Connections, Secrets, and Functions address as a unified solution.
Unity Catalog Connections provide centralized governed access to more than 11 external data sources through Lakehouse Federation, while UC UDFs expose governed functions for any compute type and cross-workspace secret replication policies manage credentials without long-lived secrets in user code.
The architecture enables a zero-trust approach where credentials are injected at query time rather than stored in notebooks, service credentials provide governed access to cloud services, and full lineage and auditability from source to insight are maintained across the entire external access pattern.

Secure External Data Access with Connections, Secrets, and Functions in Unity Catalog

Modern data platforms are rarely islands; they exist in a sprawling ecosystem of external databases, APIs, and third-party services. However, managing the "connective tissue"—JDBC drivers, secret rotation, and custom logic—often creates security silos and governance gaps. In this video, we will hands-on demonstrate how the next generation of Unity Catalog Connections, Secrets and Functions work in concert to centralize external data access, eliminate hardcoded credentials, and deploy governed UDFs across any compute type. Attendees will learn how to build a "zero-trust" data architecture that simplifies the user experience while providing full lineage and auditability from the source to the final insight.
Talk By: Jakob Mund, Staff Product Manager, Databricks ; Vladislav Mantic-Lugo, Senior Product Manager, Databricks ;

Chapters

FAQs

What are the three problems with external data access that this video addresses?

The three problems are governance gaps (connections buried in user code that are invisible to administrators), security risks (hardcoded or long-lived credentials not following best practices), and usability friction (difficulty of integrating external data in a governed, easy-to-use way). Unity Catalog Connections, Secrets, and Functions address each of these problems together.

What is Lakehouse Federation and what external sources does it support?

Lakehouse Federation is Unity Catalog's governed access layer for external data sources, allowing teams to create a connection and a catalog so they can see and query data in target systems without copying it. This video notes it currently supports more than 11 data sources, with connections acting as governed objects that provide visibility and security over all external data access.

How does credential injection at query time improve security?

Rather than storing credentials in notebooks or user code, Unity Catalog injects credentials at query time from a centrally managed secrets store, so users never directly handle sensitive authentication material. This eliminates hardcoded credentials, supports best practices like short-lived credentials managed by systems rather than individuals, and ensures all access is auditable through Unity Catalog.

What are cross-workspace secret replication policies in Unity Catalog?

Cross-workspace secret replication policies allow secrets stored at the workspace level to be replicated and managed consistently across multiple workspaces, with schema-level secret configuration and access control. This enables large organizations to centralize credential management without requiring each workspace to independently manage its own secrets inventory.

Full transcript

[00:08] All right, good morning everyone. Thanks for turning up this uh this early for this talk. Um it will be lightning talk. So very quick uh here to give you an overview of how to achieve secure external access with Unity catalog using
[00:20] connections, secrets and functions. All right. So why is that important? Well, usually your data is not only within data bricks and it's part of the larger
[00:33] data landscape. So your users and especially your agents um will have to reach outside of data bricks to databases, SAS APIs, cloud functions um
[00:45] and yeah because of agentic AI uh this problem will only get bigger over time. All right. Um so the problem is pretty pretty important but why is it a challenge? Well we think for three
[00:58] reasons. First the governance part of it. Um if you have connections specified somewhere in user code if you have API calls from your workloads you might not even see it and if you don't see it you
[01:11] can't govern it. Um and governing at scale it becomes a problem if you have very different many pieces floating around that do this external data access. Second, usually those systems
[01:23] are not free for all. You have to secure them. Um, so you make have to make sure that if you use credentials, any form of credentials, certificates, passwords, pet tokens, um, make sure those are locked down, following best practices,
[01:36] secured. Um, ideally not using long live credentials, um, have it managed by systems um, and not users. So th those are the kind of problems that we're seeing. Um and then governance security
[01:50] is great but you also want to have people reach out to other data integrate them. In order to do this for the best practice you have to make it very easy to use and that's exactly what this talk
[02:02] will be about. We will talk you through how we make those external data access governed secure and easy and we do that um in three parts of the section. The first one is external uh data sources.
[02:16] I'll hand over to Vlado for this um and then later on pass on to go for external services uh as well as talk about how to secure credentials and sensitive information. There you go.
[02:33] >> Yeah. >> Okay. So talking about external data sources can you hear me? We have today already leav lakehouse federation already you kind of know it.
[02:45] Uh it allows you to have a govern access and security to more than 11 data sources. Uh you create a a a connection and then a a a catalog so you can see
[02:57] the data that you have in your um target system. What they bring today is what happens beyond if you want to go beyond those 11 data sources, we bring you JDBC
[03:09] connections which allow you to connect to any data source just using your JDBC driver. Why is this important? Because today if you want to connect to data sources that
[03:21] are beyond those 11 data sources on on Lakehouse Federation, you will need to use for example the Spark GDPC API. But for that you have no connection serverless because you need to bring your own driver and we need to isolate
[03:34] that. Then you also have ungoverned and undefined uh connections in in the notebooks. So you don't have governance and security on it which sometimes happens that people have just um uh
[03:48] passport directly in the notebooks. And to solve all those problems what I bring to you is JDBC connection. How this works? So you create a connection of type JDBC. You bring your driver
[04:02] specific to your data source that you want to connect. You share that connection with anybody in the organization that you want to be able to connect and then they can use it the same way as today they're using their JDBC uh uh Spark JDBC API. Why is this
[04:17] important? You don't have uh you have governance because you can see who is accessing what when. You can control the access. You have passwords that are not running around in notebooks and configuration that have to be copy
[04:29] pasted. You just created one. You can reuse it across any compute including serverless and you have reads and rights. Today Lehouse Federation only enables you uh having uh reads.
[04:44] So this is very good. Now we can connect to any data sources but we need to have a connection to connect to data sources and today connections lives in in metas store. So they are metas store object which means that there is a metas store
[04:57] admin that needs to provide you the access to be able to create um the connections. So you need to have the right permissions. This is a problem because in many big organizations accessing those meta store admin is very
[05:10] hard. So what we want to make it it's it's much easier is that we bring connections to schema level. What this bring what this makes means is that you have a connection that can live similar to a table a model or a function in a
[05:23] schema which is makes it much easier for developers to just create the connection as a self service. You don't have a flat list of thousand of connection because you can self-organize those connection within the right schemas and within the right
[05:35] schemes and very importantly you can scope those connection to workspaces. They're not living right now at a meta store layer which means that it's very hard if you have different workspaces for different team teams to to control
[05:47] it. Very simple. You just uh open your your workspace UI. You choose your schema create connections. For example, right now we are bringing already in beta uh
[06:00] the connections for HTTP. you uh you choose the connection, define the credentials, define the URL and then you can share your connections with anybody in the organization the same way as you today can do it with a table or a UDF.
[06:14] Why is this important? No admin involved. Your team is self-service. This is very easy to discover and search through the uh databicks Unity catalog search engine. You can scope it a
[06:26] workspace and is much easier for you to develop when you're building your agents. uh because connections are used for MCP services. With this now we move to external
[06:38] services and pass it to Jacob. >> Thank you. All right. So Vlado told you all about data sources but what about general services? Well let me start with one class of service a cloud service. A cloud service is anything that is a
[06:50] native service living in your hyperscaler. Um and you're using usually cloud SDKs like B3 for Python to access those. Um you can do this today uh and
[07:02] many people do. What you do is essentially you in initiate a session and you somehow pass in a credential to that one to authenticate. Sometimes it's really just hardcode it in the notebooks. Uh but the more common and
[07:15] better way is that you use workspace secrets for this and you store it somewhere workspace secret or very very common is that you put it in an environment variable um and pass it this way. However, there are two things that we think is a problem with this method.
[07:28] First, those credentials are long lived. So, if one of those get exposed for some reason, uh there's a higher chance that that credential will still be useful. And number two is you don't have
[07:40] visibility into what's going on in the code in terms of access to those external systems. It's hardcoded. You don't have visibility. The only way to find out more or less is when you look into the code and see, okay, this is
[07:52] called an external system. Um so this is very hard to govern to audit uh and comes with a lot of challenge. So our recommendation here is to use UC service credentials. UC service credentials they
[08:05] wrap cloud identities like an IM role or a managed identity on Azure. Uh and bring them into UC as a UC entity. Um what we built here is a very thin native
[08:18] integration with the cloud SDKs. So essentially what you see here in code is you get a cloudspecific object when you want to access a UC service credential. In this case it's named AWS cret um
[08:31] governance happens at this place. Further on you just have your boto3 call. You can make this even easier if you put this AWS cred name in the environment variable. Um it's just a name of thing. There's no credential behind it. Everything is hidden. What's
[08:44] happening under the hood is that um we will check all the permissions whether the user who invokes this has the permissions in unity catalog to use that credential if that user has it um under
[08:56] the hood a short shortlive token will be uh generated and used for all subsequent interaction with the cloud service. So this gives you the the um peace of mind that you're using shortlived
[09:08] credentials. Um it's still super easy because you don't have to do more or less anything. This is handled between data bricks you in the catalog and the cloud vendor. Um and it's fully governed you see. So this will show up as audit
[09:20] events in your audit trails. Um you can control who can actually create those credentials. So you got all the governance in place from uni catalog. All right. So this is how we would expect or recommend to interact with
[09:32] cloud services. Now what about any API really? Well, this is a bit trickier, but our recommendation here is, hey, let's wrap those in UCDFs. Um, so this
[09:44] is a simple example where I'm calling a weather surf weather service, passing in a location, and getting a temperature back. Um, and I've wrapped this in a UC Python UDF. Um, why did I do this? Well,
[09:56] for two reasons. First, it's really easy for users. They can just simply call this uh current temp uh function. It behaves like it appears to be just like a built-in function. So super easy to use. On the one hand, you don't need to
[10:09] manage anything on the user side. But on the governance side, this is wrapped in a UC UDF. So you control those UC UDFs. You can make sure that only who users who have execute can actually call that
[10:22] thing. And this being a UC UDF, you got a couple of advantages. You get audit again. You got the privileges. But what we just also launched, you got you got lineage. So you can see okay which workloads was this UDF called in was it
[10:35] called in for like let's say other table definitions view definitions. So you get the lineage and you see where this is where this is happening. So you got the visibility even into arbitrary APIs. And the pro tip here really is if you use
[10:48] the batch version of this one. So instead of saying this rowby row to an external API you can send a batch of data to an external API. It works the same way um just using pandas under the hood and you get that performance
[11:00] benefit as well. All right, great. So now we talked okay let's talk let's connect to external data let's connect to external services there's one common piece uh among both of those that is very very critical and
[11:12] that is usually authentication requires you to have some form of sensitive information included like certificates, passwords, tokens and so on. So how do we secure those? Well, our answer is
[11:24] what you got in product today. You can use workspace secrets. Those are secret scopes that live in each workspace. Uh you create them using the CLI as you can see here where you create a secret scope
[11:37] and then within a scope you can put a bunch of secrets in and you have a workspace echoes to essentially uh tell who can use a scope or not including all
[11:49] secrets. All right. Um sounds great. But we heard that there are a couple of things that people don't like about this. First, it is bound to a single workspace. So any of those works the
[12:01] secrets will only be available in one workspace. However, authentication information often lives in many many workspaces. So what you do is you replicate those in a couple of different workspaces which is not great obviously
[12:13] from a security point of view. Um the other one is it's a CLI thing only. So you need to download the data bricks CLI to manage those to ex um to create those to see which are available. This is not
[12:25] the most like user friendly way where you can met all your users and make sure that everyone in your organization is actually using secrets. Um and finally this is a workspace object. So workspace admins can read those secrets. You don't
[12:38] have to grant them the permission. It's kind of because I'm works with admin. I can see those. We heard that for some people this is something you want the more of a blind admin scenario. Um, so that's is a problem here as well. So as
[12:50] you probably guessed by the title of the slide, our answer is simple. Let's put secrets in Unity catalog as well. So what does it mean? Well, you got the full thing. You got the UI, you got this, you got SQL, you got APIs like the
[13:03] UC APIs, the rest APIs, you get Terraform, you also get CLI in Unity catalog. Here's a screenshot of essentially this new secrets object living next to function models, volumes, um, and tables. and you can just use the
[13:16] UI to create it. So that's a lot simpler for more users to create it, to discover it, uh you can search for it and so on and so forth. Second, it's living at the meta store level. So you can make the
[13:29] same secret without replication available across workspaces. Of course, you can tie it down to a certain workspace by just using the catalog and binding it to a certain workspace, but if you want to, you can also have this
[13:41] across many workspaces. And finally, you got a per secret permission. So you can grant users on a per secret basis instead of a whole scope permissions. And it's unified in
[13:53] Unity catalog. So there's a create secret. There's a read secret permission and so on and so forth. Let me show you how this works. Let's start with the creation phase. To create a secret, I just choose SQL, but as I said, anything really works. UI, API, Terraform, CLI.
[14:08] So I choose this SQL here just to make it to visualize it a bit. Uh pretty simple, pretty straightforward. What you do is lives in a three name space lives as my pw. How do I share it? Well, as I said, it's
[14:21] a unified permission model with uni catalog. You can just grant someone like Vlado access to that secret. So, so Vlado can now read the secret. I can do ex I can assign this to a group. I can assign this to a service principle. Uh I
[14:35] can do this in the UI again. Uh and this will be true across workspaces wherever has access to. All right. Now, retrieval. Is that more complicated than before? No. The only thing that really changes is before you do it DB utils
[14:49] secrets get and specify the scope and the key of the secret. Now, instead of a scope, you just specify the catalog and the schema. Uh, and it's pretty much the same as before. All right. So, if you
[15:02] have workspace secrets using today, the lift to this one is pretty minimal. All right. So, this is what we got today. There's more stuff coming up now that we actually have secrets in your D
[15:14] catalog. Number one, we're building native integrations for all the cloud secret managers, meaning for AWS secrets manager, for uh Azure key vault and GCP secrets manager. How does that work?
[15:26] Well, you'll be able to choose for a schema whether you want to have data bricks manage secrets where you create those secrets in data bricks. They're stored in data bricks encrypted. You can also bring your own keys or you can switch over to hey I want secrets to be
[15:40] pulled um from my secrets managers that I have in the hyperscalers because those are used for many more systems than just data bricks and we do this zero copy so we only get metadata in the existence of
[15:53] it we create a UC object for it but the value stays in the hyperscala secrets managers we're only pulling this out on demand for retrieval but we're not storing it anywhere in data bricks so that's important for compliance. Um, and
[16:07] it's automatically synced. You create a secret in let's say AWS secrets manager, it automatically after just a short period of time appears in data bricks and you can use it in data bricks. You can put permissions in your catalog on
[16:20] it. We will check those before fetching the secret from the external secret manager. If you wrote it now in one place your secret manager in your cloud it will automatically apply to all the places including data bricks because of
[16:33] this fetching the value life. The second thing that is important is as value as V mentioned is UC connections. So you want to embed those secrets into UC connections. Uh, and what you do
[16:46] today is, if you're familiar with the API to do this, but you'll be able to do it in the UI as well, is you specify your connection details, and there's usually an option clause where you put a
[16:58] couple of options into like host, password, pet token, certificate, and so on. You can now freely choose whether you use a string for an option like you have for host here, or you just move the
[17:11] host key down to a new section called secrets. And any option that you want to use a UC secret for, you can by just moving it down into the secret section and pointing it to a name of a UC secret
[17:24] instead of the actual value of it. What will happen is when we initiate the collection, we will pull that secret in and use that to authenticate or like resolve the host or whatever like your option you're going to use. We'll work
[17:37] together with the external secrets manager as well. All right. So to conclude um we wanted to make external access with Unicata governed,
[17:49] secure and easy. Why is it governed? Well, for governance, use lakehouse federation and UC connections for data. Use EC service credentials and UCUDS for services. Regarding security, if you're
[18:02] using cloud services, use service credentials. you will get shortlived credentials um from the cloud identities and use reus secrets to avoid hard copied or replicated secrets. And
[18:14] finally, why is it easy to use? Well, it's easy to use for users. You get the full discoverability of Unity catalog. More users will be able to use secrets instead of going another path to use credentials. Um you get the full UI for
[18:28] it, easy to manage. You also get deep integrations with things like the cloud SDK. So we made it as easy as possible for you uh to do this and for as many users to actually use it. Uh but it's also simpler for admins. So you don't
[18:41] have to manage it specifically for workspaces. You don't have to keep this in sync. You can manage your objects and data access across workspaces in one permission model from Unity Catalog with the lineage and audit and the
[18:54] auditability that Unity catalog brings you. All right. Thank you very much.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.