Databricks x Palantir | Partnership Deep Dive

Summary

• Bidirectional integration: The partnership enables secure, bidirectional data federation with minimal data movement between Databricks and Palantir platforms.
• Unified governance: Unity Catalog provides centralized governance, while compute pushdown ensures workloads run where data lives.
• Faster value: Joint customers reduce friction between platforms through tighter integration of analytics, workflows, and operational AI.

Databricks x Palantir | Partnership Deep Dive

Together, Databricks + Palantir are accelerating high-impact AI use cases and outcomes.
In this walkthrough, Benjamin Abood and Chad Wahlquist break down how the integration enables:

• Secure, bidirectional data federation with minimal data movement
• Unified governance using Unity Catalog
• Compute pushdown so workloads run where the data lives
• Tighter integration between analytics, workflows, and operational AI

Chapters

• 0:00 Partnership overview
• 2:03 Bidirectional integration
• 5:57 Governance and security
• 8:59 Roadmap and AI workflows
• 10:25 Demonstration and conclusion

FAQs

What does bidirectional integration mean?

Both platforms can read and write data to each other—register data in Databricks with Palantir and vice versa, keeping one copy with unified governance.

How is governance unified between platforms?

Unity Catalog provides centralized governance across both platforms, giving consistent permissions, security, and audit trails regardless of where workloads run.

What is compute pushdown?

Compute pushdown allows workloads to run on whichever platform the data lives on, improving performance and reducing data movement.

What workflows benefit from this partnership?

Analytics workflows, data processing pipelines, and operational AI workloads all benefit by eliminating friction between the two ecosystems.

Full transcript

0:00 Partnership overview

Hi, I'm an architect at Palanteer. >> And hey, I'm an architect at Data Bricks. And today we're going to talk about the data bricks and Palanteer partnership. >> So data bricks and Palunteer better together. What does that mean? Really, we think about this. How do I get to value compute and storage runtimes across my platforms and to really have this seamless integration across the two platforms so that our customers can get to value quicker.
That's really the end goal behind this is remove the friction between the two platforms. >> And I'll I'll add that this is really a customer-driven integration. So uh it should be no surprise that Palunteer and data bricks have many joint customers and sometime around November last got together and were talking hey we have all these joint customers they've been asking me how can Palenter and data bricks work better together you know Palunteer and data bricks both play a pivotal role uh within my ecosystem and right now there's a bit of friction as I'm deploying a platform side by side you know I want to do things like said I want to reduce data movement I want to keep one copy of my data between the two platforms I want secure data sharing between the two platforms.
I want uh more flexibility on choosing which tool I'm using from which platform when. And so as we had this request from many joint customers uh we got together and said hey why don't we build a partnership? Let's build some first class product integrations to allow our customers to achieve faster time to value through before then and our partnership is centered around these four pillars of product integration which are data federation, governance integration, compute push down or compute federation and AI and workflow integration. >> Awesome.

2:03 Bidirectional integration

Yeah. So let's get some more let's get into some of the details of like what does this look like across the platforms. Maybe we can jump to the next slide here. So you know the idea here of like how do we build this birectional integration I think that's a key part of this partnership is people are like oh you can connect here and pull data here uh really this is the synchronous birectional integration backward and forward so I can you know have data in data bricks and register it in palunteer I can create data in palanteer register in data bricks and move those back and forth seamlessly all with the unified catalog and governance around it um and really having that interoperability story that people want is, you know, reducing the amount of times I'm having to move and copy and create pipelines.
All of that is just slowing you down. And so, this is really kind of how we'll look across this. Uh, but the end goal is to enable the use cases that are adding value across our business and getting there quicker by removing that friction. >> Awesome. So, what is the multimodal data plane? Uh, you know, if you look at Palanter's website, you'll see a lot of references to MMD or multimodal data plane recently. And what this means is any storage, any this is a birectional integration.
So there's going to be some data that lives in Foundry. You know, your workflow data or you know, maybe Palunteer has a great connector for some data source you're connecting to. That data is uh in Foundry, but you want to access that data in data bricks. Well, as of recently, you can now register Palunteer's iceberg rest catalog directly in data bricks. So now I can access that data from datab bricks without the need to move the data.
And the other direction is, you know, you're using data bricks, your open data lakehouse, and maybe you've ingested a bunch of different data sources. You've built out your medallion architecture, but I actually want to use that data within Foundry for my workflows or for mytology applications. You can register that data within Foundry via virtual table. So again, I can access that data without the need to actually physically move data across the wire.
Yeah, I think it's pretty powerful and people like start to understand like what is the difference of like why do I want to have this in data bricks and ontology here and all these things and I think the the key differentiator and where we see the value is hey if I've already got all my data together and I'm pipelining I'm doing this this work in data bricks now I want to build operational applications and I I want to model into the ontology the d not only the data but also the logic and the actions to be really be able to drive these integrated workflows across your business.
So now this brings the best of both worlds together between data bricks and palunteer and our our customers can go build these operational applications have this also powered on the back end by data bricks powering the compute and storage for your data across your enterprise. I think that's it's a pretty important point about what is the difference between these layers and like how do we utilize these in the best in the best way possible to get those synergies.
Um, so we're seeing a lot of customers see a lot of success with this exact kind of methodology of I've built out these pipelines. The cool part and we'll show some of this later in a demo where I can actually author a pipeline in Palunteer even and actually have the storage and compute running even in low code tools inside of the platform. So the ability to actually author in Palunteer and select the runtime v data bricks is a pretty cool and innovative thing. >> Yeah.
And this is this is really awesome. You know, you might have Palunteer Foundry experts operating within your organization and maybe your data is being federated from data bricks via that virtual table. Well, your Palanteer experts can continue to author their pipelines and pipeline builder and code repositories, but they can actually run the compute where the data lives. As you're federating the data, let's also federate the compute.
So, again, it's this going back to the concept of minimal data movement. uh you know the ability to interoperate between the platforms and really build a bunch of product seamless. >> Yeah, maybe you want to hit also on and we just talk about kind of the security piece of this as well and some of the integration with Unity catalog. You want to hit on that? >> Yeah, definitely. So our our governance integration today is centered around the connection connect to I'm going to choose whatever catalog whatever schemas whatever tables and uh I'm going to give that connection on the foundry side credentials to connect to data bricks and so we recommend using service principle and workload identity federation and what this will look like on the data brick side is I'll create a service principle on the data brick side I will give that service principle access to you know whatever data assets I wish to federate to Palunteer authenticates with data bricks it will authenticate as that service principle um and use workload identity federation.

5:57 Governance and security

So we don't need any secrets for uh access to data bricks APIs and then any data access from the foundry side will be permissioned according to that service principle and the way uh the way the data federation actually works is unity catalog can vent credentials to the underlying uh to the underlying data assets. And so you know if you have your data assets let's say living in S3 somewhere um Foundry will reach out to Unity catalog you know using that service principle for authentication and say hey Unity catalog there's some data asset uh you know at this catalog this schema this table name that I'm aware of that I would like to federate that I would like to use within Foundry and direct read and write assets So now Foundry can directly read and write to that blob storage to that S3 bucket where your Delta or iceberg or paret files actually live. >> Yeah.
And that's a huge boost for performance by having the direct access to the where the data is instead of having multiple layers I need to go through. Now having that unified in Unity to be able to vend those out and actually get direct access to the federated data. It's a great thing for performance um across the board. Then >> absolutely and I'll say of our joint customers using this integration, we have a bit over 100 customers using the integration today, the majority of them are using this virtual table or data federation capability. >> Yeah, that's pretty cool.
And I think that brings up the other point too is like you can actually back an ontology object with a table sitting in data bricks. So the fact that I can have a data bricks table backing ontology objects I also can do rightback. So a lot of times when we're creating these operational applications, part of this is I want to capture part of that feedback in the workflow instead of now me landing that in Foundry and I got to create another pipeline and move it to data bicks. just write that back straight to data bicks.
And if I have to do reporting or other workflows on top of there out of data bricks, boom, it's right there. Um, and so having that integration starts to remove all these layers as we're talking of now I'm not having to create a pipeline to bring data in to build an ontology object and another one to move it back to data bricks. I've eliminated those extra steps and that's just less things to break, faster time to value, which is great for our customers. >> Absolutely.

8:59 Roadmap and AI workflows

All right. So, yeah, let's get into this. So, jumping jumping into this next slide, uh, this is our product road map, and I'm not going to read through each of the bullets on this slide. Um, but really what I want to point out here is over the past six months since we announced our partnership, I believe on March 13th, we have built a bunch of product integration work and a lot of this has been customer-driven or customer-led.
So, uh, we had our sort of initial set of ideas. Hey, what are the things that our joint customers care about the data federation, governance integration, compute push down or compute federation and AI and workflow integration and since then we have been responding rapidly to the feedback from our customers. So, as our joint customers have come to us and said, "Hey, this part of the integration works really well. Here's something I would like to see on the road map."
We're the rest of our product road map look like?" >> Yeah. your data science team has built a model in data bricks, you can actually just register that in Foundry and use it as part of your workflows. And so having that even if you want to have it hosted in inside of data bricks and use it in Foundry or you can bring the model into Foundry and host it, you have that optionality across your architecture. So really making as seamless as possible to get value out of both platforms together.
All right, so should we take a look at what this looks like um in a demo? >> Yeah, it's great. All right, so this is Palanteer Foundry. This is our data connection uh setup and so you can see we have a data bricks uh connection already set up. So when we talk about this we have batch sync is how we would traditionally pull data from the environment. So we connect to data bricks pulling data into palunteer foundry virtual tables is the new stuff that we're talking about here.

10:25 Demonstration and conclusion

So this is where I'm registering data from data bricks in the unity catalog straight into palunteer so we can use it. So I'll hit create new virtual table. So this is actually going to go read the schema information across datab bricks. Uh, and so we'll get into this environment here. And we're going to search for DBX customer clone here for this demo. And so this is giving us a preview of the data from data bricks. And I can hit add virtual table.
And I can hit create. Now I've already done that for the demo here. As you can see, I have this table here. Again, it just looks like a Palanteer Foundry data set. Um, but there's a few few key differentiators. So we have the sources data bricks. We also have some table details. So this is a datab bricks source. It's a JDBC connection. So this is actually going through the warehouse to the blob store. So actually this is where data bricks is actually serving up the data to giving us the vended credentials from Unity catalog and giving us the location in S3.
So there's different ways that we can access it and you can see those details in here. So if we look at this, I can actually go hit open in data bicks. So having an open a datab bricks button right inside of Foundry is pretty cool. Um and here is my table. So this is that exact table in Foundry and I can see all of the data bricks environment around it. So something with this. save. And when we hit external, so this would be a batch pipeline or running externally.
So this is where I actually want to have the storage to have it run in my datab bricks environment. Um and so let's pick this as our source. And voila. So now there is the table. You can see this is saying external. Um and so this is our customer clone. So let's just do some basic transform to it inside of Foundry here. We'll do a filter uh rows and let's pick which uh column that we're going to want to filter on. Um let's see.
I think there's a state in here somewhere. All right. And we're going And we'll hit apply. And so if we come back and close here. So this is the cool thing about building here now is like I can even see the previews. So this is actually going back to data bricks using the transform getting the result set and will actually give me previews along my pipeline as I build. let's say I want to do something else here like I want to actually create a new table and so this is where I can actually select >> here.
And we're going to do the default. This is our demo here. And it's going to be a manage delta. Could be manage iceberg or external. this compute and storage is going to be pushed down to data bricks. So if we hit save uh and I'll hit deploy here and looks like we got an error. Import those. All right, the reference. Cool. DBX demo and. Um and we can even then look at the deployment history as we normally would. And so we can look at the build report inside of Palanteer Foundry like you're used to.
Uh but this is actually executing in data bricks. And >> so this is a really awesome feature and just kind of wrap it all together. You know showed the data bricks connector. We had that table sitting in data bricks. We federated that data asset to foundry. So we picked that customer clone data set. We didn't ingest the data. We just virtualized or federated that data asset to foundry. transformation on the data but that compute ran on data bricks serverless compute and the output data set was written directly back to unity catalog.
So again there was no data movement here. Uh we read the data from data bricks we used data bricks compute to run compute where the data live you know at the data bricks storage and we wrote that output data set directly back to unity catalog. a really really cool feature where you know you can leverage uh all the tools you're used to in Palunteer Foundry but using whatever compute whatever storage uh you want to. >> Yeah, this is pretty cool.
Like I can see the logs as they're happening here in platform. I can see the build just like I would. So it really is a first class citizen inside of Foundry. Uh and then let's go actually update our uh catalog here. Let's see if we can find our table. Uh let's just I'm assuming no one else has a a table with in there. There we go. And here's our table with just California on it. So there you can see from end. Now there could be multiple tables and I could have joined multiple things together in my pipeline.
But the idea here is I now have that capability to author here and push down that entire thing utilizing data bricks for the comput and storage. It's it's proposition by making this so seamless that really is not just like a bolt-on or an afterdot. This isn't just a JDBC connection to another data source. This is a firstass integration of Palunteer and data bricks. >> Yeah, absolutely. Thanks. It's always a pleasure.

Learn more about the Databricks Data and AI platform.