February Databricks Updates: Lakebase Autoscaling, Knowledge Assistant AI, and Sharing to Iceberg

Summary

New Connectors: Databricks released multiple new integrations including Jira, Confluence, Microsoft Dynamics 365, Salesforce with incremental loads, Meta Ads, Excel, NetSuite, and PostgreSQL connectors in public preview and beta.
Knowledge Assistant: This feature is now generally available, providing AI-powered assistance within the Databricks Data and AI platform to help teams leverage data more effectively.
Lakebase Autoscaling: New autoscaling capabilities in Lakebase enable automatic resource management, optimizing performance and cost efficiency for lakehouse workloads.

February Databricks Updates: Lakebase Autoscaling, Knowledge Assistant AI, and Sharing to Iceberg

Databricks platform release notes: https://docs.databricks.com/aws/en/release-notes/product
Want some pre made data? Check out Caspers Kitchen for a 4 click deployment here: https://github.com/databricks-solutions/caspers-kitchens
Loved this podcast but wanted it on spotify? https://open.spotify.com/show/1HVAzoA2A9nHfayMlRS81i
Hated this podcast? Why not replace us with an RSS feed: https://docs.databricks.com/aws/en/release-notes/#databricks-release-notes-feed

Chapters

FAQs

What new connectors are available in Databricks?

This video covers multiple new connectors released by Databricks, including Jira, Confluence, Microsoft Dynamics 365, Salesforce with incremental loads, Meta Ads, Excel, NetSuite, and PostgreSQL. Many are available in public preview or beta.

Is Knowledge Assistant generally available?

Yes, according to this video, Knowledge Assistant is now generally available (GA) in the Databricks Data and AI platform.

What is Lakebase Autoscaling?

Lakebase Autoscaling is a new feature that enables automatic resource scaling within the Databricks platform, helping optimize performance and manage costs.

What is Delta Sharing to Iceberg?

This video discusses Delta Sharing to Iceberg, a capability that allows sharing data between Delta and Iceberg formats within the lakehouse.

Full transcript

`0:00` Intro

Beautiful. >> Yeah. >> Oh my god. It's a 10 out of 10. I think we got a 10 experts on data and AI, but separately, not so much. >> This is the podcast where we talk about our new favorite features in data bricks and then try to shoehorn them into one architecture to see if it's actually realistic. And here are our favorite features from the last 60 days because we do have Christmas as well. And I think to start us off, uh, am I going first or are you going first? >> You're going first.
Give us some data. >> Our excuse last time was that we didn't want to do 30 days because people are going to have a break and they're not going to be delivering that much over the holidays. And it turns out that the actually the connectors team has been doing a massive amount of work since we have all been off. So, we've got a bunch of new connectors that are available for people to use. A lot of them are either in betas or public previews.

`0:36` New Connectors in Databricks

Um, but that does mean that you can go ahead and get started with them. So, first of all, we have a Microsoft Dynamics 365 public preview. Uh, we've got a Jira connector. We've got a Confluence connector. Uh, the Salesforce connector has been improved as well, and you can now do incremental loads, which is fantastic because those tables are quite big. Uh we've got a meta ads connector which is in beta as is the ability to read Excel files is also in beta.
We've got a netswuite connector and we also have uh postcress SQL as well. >> Oh my god, I need to go register for every one of these things so that I can >> Okay. Uh so just draw all of them. Um so I actually do have a little demo uh to show you kind of how to get started with these. Right. I'm going to head to my previews tab uh which is here. I just click on my little icon here. Uh and if I type in Jira, um I've got a link flow connector for Jira here as well.
I'm just going to turn this on. I might have to reconnect the page. Uh now there is a bit of setup that you need to do uh within the back end of Jira as well. So this is the kind of app uh the developer console I stand corrected. Uh and here I have um some settings that I need to set up. So I do need to configure this Ajira API. Uh, and then when I go on this, this does a link to things about kind of scopes that I need.
There's some classic scopes to set up and then some granular scopes. Once I've done that, I then go to my settings and I get all of the things that I need for my console. I am hoping that it's not going to h I've got a client ID. I guess I'm blurring that out later. Um, but then we have these connection settings here. And this is what we're going to then use to go ahead and make our connection. So now when I head back to here, um if I go and create a connection, so if I go to my catalog, I drop down, I create a connection.
Um I should be able to call this uh Jira test, the connection type. And I'm really hoping Oh, there's our Dynamics 365 that's popped up. And lo and behold, here is my uh Jira uh connection. Oh, >> first time. >> Uh, I technically did this earlier today and clearly I've not putting in our details that we had from our developer console. Uh, and then we go ahead and we sign in with Jura and we do our oorthth um authentication um for Jira.
Sadly, this is actually blocked from a data bricks perspective, so I can't show you any further than this once you've kind of copied and pasted everything in. Um, but then once you've got that, you've then got the ability to connect Jura uh and get all of that wonderful Jura information out and into your lakehouse. , can I ask you to put it very simply on our architecture? because I do have a bit of idea of what we can do with our infrastructure this month. >> All right, so let's do a Jira.
Um, there's also a matching confluence one as well that you set up in exactly the same way as well. >> Uh, so I have an idea for what we're going to do for our February features that we have in data bricks. So, what happens in February? Valentine's Day. And what I think we should be doing is using Jira tickets to play matchmaker to play Cupid. And we are going to play matchmaker with every single employee at Data Bricks based off of their Jira tickets.
What do you think? >> Pretty spicy, not going to lie. Uh here. that person who like beautifully well articulates their jury tickets in terms of problem statement >> Claude. Yeah. Yeah. Very good. Super articulate. >> They're obviously a perfect couple. Couldn't be me because I'm like this button is broke. Please fix. So, oh maybe I am also matched with someone I don't know that uses a lot of emojis with things. It's the next big idea.
Like we could make some really beautiful relations happen like this. >> Okay. Yeah, I'm down. Let's Let's find out. We need to build it first and uh let's see >> let's see where the holy has to go on some very specific training after this episode. >> Let's take a look at data bricks assistant in the last 60 days. So three things in the databicks assistant. First thing in the data bricks assistant is that it's now live on the documentation.

`5:15` Databricks Assisstant

Uh so docs. databicks. com. Actually let me just show you. I can just do that one. docs. databicks. com. Boom. And now on the bottom right here you have ask assistant. And this is the same assistant that's available in the data bricks platform. Obviously, it's not like the agent mode because you it can't edit the documentation, which I wish it could. That would be nice. want is to use the assistant in the platform. That's typically how you how you use it.
But, it doesn't hurt to have it here. It's kind of a a standard in the industry now that docs have some kind of chat type interface with them. So that's the first piece on the assistant. Second piece on the assistant is that the agent mode that can actually interact with your workspace is now widely available in public preview. And with that also comes support for something called skills. Do you know what skills are? >> Uh are they the things that I lack?
They're my skills. Yeah, agent capabilities. Stop laughing. Stop laughing. This is serious business. I'm talking about skill. I'm trying to upskill you right now and you're just laughing at me. skill issue. Go. >> So, skills are an open standard. It's something that was originally uh developed at Anthropic and now it's open standard donated to uh open source. So anybody and everybody can uh use and extend it and uh the whole community can work on it.
Um but it's essentially a set of instructions uh scripts and assets like any arbitrary assets that are bundled together in a folder called skills. You can actually extend the abilities of your agent to do custom things. Now if that sounds a little bit like MCPS and tools, um that's because they are pretty similar. It's like this universe is still kind of I would say fluid and everything's crossing over each other and it's you know there's all kinds of different ways to achieve basically the same thing.
But the key benefit is that uh with skills is that it doesn't load everything into your context at once, right? So skills can be broken down into like separate text files, MD markdown files. And by doing that um the agent can progressively look for more skills or more details of a skill um as long as they're broken down um by file and then iteratively add them to the context to explore and find what it needs to do. So, um, how that differs from like a huge file like you have a claw. md or agents or llm's. ext or, you know, there's like a million of these different protocols.
Now, um, how that differs from that is that those files are loaded in kind of all at once. So, that pollutes your context and so you're using more tokens, your agent's getting more confused about what it might need to do. Um, and it's not really portable. So skills are essentially that same text file, that big text file that includes everything for your agent, but broken down uh into folders, uh scripts, some some additional assets and individual markdown files within that skill.
So very exciting. Those skills are now available in the data bicks platform. So in data bicks, you can actually go to your workspace that I have here and you can do this assistant. So it's kind of a hidden file. Um in this assistant, you have something called skills. And in here, for instance, we're still in the universe. I made a trivia thing in a Skills MD file for trivia, saying, "Hey, uh, has a secret location in Los Angeles."
So, when you go to the agent and you ask it something um about, say, "Hey, give me some trivia." It'll actually load this skill dynamically and then give you that response. Hey, by the way, Caspers has a secret location in Los Angeles. a trivial example, but you can use it for deployment scripts, anything you need for your personal workflow. You can continuously add skills to this workspace um and essentially build up the intelligence for the agent so that um you can do your work better. >> Okay, interesting.
So, could I use skills to make my Jira dating app less of an HR nightmare? >> It's for more of a development like when you're developing in data bricks, right? and you and you have something some task that you need to accomplish. Let's say you're always restarting a job that has some dependencies and there's like a certain order of things that you might need to do that kind of thing. So that's really where a skill comes in handy. >> So could I instead have it more on like um this is my git structure and this is where you go for this and then ignore this because this is just like Terraform templates and that's kind of abstract.
But instead I need you to go in here. >> Mhm. >> Okay. Then so um we've not really kind of loaded anything in yet. development skills and that's like a repo structure. This also connects. I guess this also has to load in some data. We'll shove that in here into Delta. >> You put an arrow through my heart. >> Yeah, I did. Yeah, it's on theme. I guess this kind of like sits around is really stretching it. Okay. I think that's that's maybe the best way to recommend to consider is having kind of like part of the screen for like dev tools and part of the screen for the architect.
You know what I mean? Like two different universes over overarchitected. >> I just it's perfect. I can't believe you're criticizing it. >> All right. I think it looks good. So we can use skills to help us build this Jura connector better. Tell me what's next. that is uh for each match uh within spark declarative pipelines. So this has been something that has been um yeah missing I'm going to go ahead and say it for a while now uh from declarative pipelines.

`10:57` forEachBatch() in SDP

Uh, and this has uh been a little bit tricky if you wanted to do something like a uh a custom sync like a custom destination um for your Spark declarative pipelines, but it also meant things like merge into you are unable to do just anything that you needed to do in that kind of like microbatch approach. Uh, and especially if you're used to a lot more control because you're used to kind of classic Spark pipelines, this is going to really open things up for those more complex use cases.
So if you've been using spark declarative pipelines and you're like hey you know I like the easy stuff and that helps me and kind of like get quickly on my feet but the moment I need to do any kind of serious processing this is a bit of a problem. This is where that for each batch comes in for spark declarative pipelines. So I do have a little example. >> Amazing. >> I love this. This has been such a pain um to not have because if you're doing any serious work for a very long time for each batch is like your saving grace.
So that's amazing. Show me. >> Yeah. So this is my kind of declarative pipelines that I've set up. Again, if you've also not seen this in a while, this UI has had a massive overhaul. And so what I've basically done here is I'm pretending that I've got some refund recommendations and I want to merge into um my updated table to say actually I would like you to overwrite kind of like with the latest things. Now this wouldn't have been possible in a streaming scenario beforehand because it's got that merge into here.
So, I've got this for each batch sync um which I'm creating here. Um I've got my target and I'm just kind of updating it. Don't look too closely at the syntax because it doesn't really make that much sense. Um but then I've got my append flow that comes in after it after it. And then I'm kind of running the stream um to write out afterwards. And so not only do I have down here I've got my um my streaming table which you you know seen before, but this is the new thing that is new.
So we've got this merge into sync um which is happening here and it's not written any data to it so it's not come out um but we do have this new syntax or this new kind of example here of a sync um of what it is uh being saved into. >> So now the question is how do we use this in our world? All right well we have some data the jur connectors coming in in it delta table. I kind of want to use the I feel like for each for each batch is really like if we're writing delta to delta hopefully we already have all the tools that we need right it's kind of like natively in the spark world so what would be an external service or something that we would write out to maybe we can do anything right we can we like where do we want to send the data to thing or a mythology thing so he'll live in the cloud >> he'll live in the cloud loud.
Okay. I like this. I like this. We're going to keep going. Uh, okay. And then I don't know how to draw a cloud. Oh. Oh, wow. I was going to say the heart is the cloud, but I like it. You're going to love this. >> And Cupid is here, >> right? Okay. Yeah. Yeah. Okay. And so, we're sending these. >> We don't know yet what. I don't know what yet, right? cuz what's coming in the Jira connector like I don't even know what the data is. >> You get all the information about all of the tickets um and the details on there but you also get things like the comments that are on them.
Did people react with emojis? You also have things like the um you know the names of everyone, the profiles of everyone, uh what their pictures are like, all the kind of movement that you get from kind of jury tickets. >> But in our use case, what's what what we have? So we're matching people based on their jury usage. Uh, yeah, why not? super hot right now and add some some a little bit little bit more here and say that we have foundation models during this for each batch thing that we're doing.
You started it. >> Yeah, I did. Um, I I do think maybe uh Cupid uh he is in the cloud, but I think he should also be like a a JDPC sync. Uh so that is something that is uh unsupported at the moment in kind of vanilla smart declarative pipelines but now with for each batch you can write to JDBC. >> Oh okay cool beautiful. >> Yeah. >> Oh my god it's a 10 out of 10. I think we got a 10 out of 10 architecture for the first time. >> We'll get fired but it's a 10 out of 10. um and Claude 4.

`15:38` Foundation Models in Databricks

5 from Anthropic are now and these are like super hot. Everybody's I think uh Claude's having like a resurgence. Claude Code is having this whole resurgence online. Everybody's obsessed with it. Um it it looks like it's AGI. I don't think it's AGI, of course, but it's it is pretty awesome. It's pretty awesome to use and it's really fun to use. In any case, it's now supported on the foundation model APIs uh in data bricks and that's really cool.
Um, and actually that's how I use it. So I I'll just quickly show you how this setup looks. And in this terminal, uh, if I just go to Claude, boom. Uh, yeah, let's just start it here. And you can see here that I'm using data bricks cla 4. 5. And the way that this actually is possible um is because if I just go to uh clock settings JSON and we will blur all this out of course naturally. Um but here you can actually see uh how you set this up.
So you have an environment. You can set the model that you want. You include uh the base URL which is your actual workspace serving endpoints. And then this one's a little bit custom. This is what you need to pay attention to. So serving serving endpoints enthropic um and then whichever the enth you can use any of the entropic models that data brick supports um and once you have this set up then you are ready to go and claude works with the data bricks foundation model API.
Now this is where I this is where I start to get the real um kind of benefit from it. I actually have the inference table set up for um this endpoint, right? So I can go back and if I go to data and I go to opus 4. 5 or sonnet can do sample data here and this will take a second to load. So everything all my usage that I'm using through cloud code is being logged in this inference table that's set up in data bricks. So I can go back and connect all these things and do all kinds of custom stuff on my previous conversations.
So I can actually come in here and do say like hey you know what are the common things that I've been doing or that have been going on over this uh over the last period of like you know whatever the last 10 conversations that I've had you know can you suggest any other uh like tools that should be created um or skills I should say um that should be created uh in order to uh things that you don't have to redo or relearn.
So you can imagine this working in a situation where you have like a big team and everybody's using some version of uh of some kind of coding tool. You can bring together all these conversation histories and kind of sus out what are the common patterns what's happening um and then create tools or first you get some kind of insight what's going on and then second you can derive some new tools or or things like that. It doesn't have to be more AI stuff but it's um it's just really useful to take a look at.
So, I know the foundation models API the API is technically not new, but I could have this uh with uh GPT. Um, sorry, is it 3. 5 or 5? It's 5. 3, isn't it? Right. >> 5 It's 5. 2. >> 5. 2. all. Uh so before these development skills uh for our repo structure, I think we should also use uh like the code for this and maybe use that to make like an onboarding guide >> for Cupid. Cupid knows everything. >> Uh for the dev team uh who are also going to visit our iceberg which is pretty sweet.

`19:26` Delta Sharing to Iceberg

So uh this is my little diagram on my demo that I've got here. So the first thing that we need to do is disable deletion vectors because that is not supported with delta sharing at the moment and that is set as default for our workspace that we've got here. Uh the next thing is turning this into an iceberg table. Uh so that we we now have as iceberg compatible and this means that it can be read by a uh iceberg client whether that is something that you've hosted yourself or whether that's something that's part of another platform uh elsewhere.
Uh, now that I've got this, we can see that we do have um some iceberg uh details in here. And that is now an enabled format on my delta table. Uh, and then finally, I can just go ahead and I can add this to the share. Um, and that will go in there. And um, I know it doesn't look very impressive, but it works. Uh, which is quite nice. for an iceberg. So, who else is in the cupid universe? >> Maybe we should send this to HR to be like, by the way, here are the people that we've matched.
We're going to email them soon. Do you want like a quick look? >> So, we need to write out a new table. >> like we did with the like we did with that one the one that you just demonstrated. >> Yeah. So, you don't have to like read in something and then write it out again. You can just modify that existing table as it is in place. It's just a change to the metadata. uh and then share that out further. >> Okay, so let's put HR on this thing.
Let's start let's start easy. >> Yeah, >> human resources. >> Such a horrible name for a team, isn't it? >> It's like the people team, right? Yeah, people rebrand. I like the people team. People team's better name. I guess this one, this table, I'm going to keep it simple. >> Yeah, >> right. This will work. How this actually does it is through can I do delta share as iceberg. >> So HR they'll have an iceberg client for reading that >> that uses iceberg.
How's that? That makes it clear. Oh man, human resources cold as ice. It's too wide. Yeah. >> Okay. Siri be wireless. >> >> say the names. >> It'll be a whole team. This is my last thing for this episode. Um and it is the knowledge assistant uh of agent bricks is now in generally available in generally available. It's generally available. Um, which is really exciting. And, um, I think we've mentioned it in the past on this show, but uh, this time I I do want to show you actually what it looks like.

`22:06` Knowledge Assistant is GA

Here we are in our data bricks. Oh, I'm in the wrong window. bricks platform um that we were in before. And on the lefth hand side here we have the agents tab. And in the agents tab you can actually see uh this agent bricks tab within the agents tab. And we have this location operations bot. It's a knowledge assistant that I already created. I'm going to go into that in a second because this is the actual one that's already running.
So I want to show you how to build it. So to build a knowledge assistant um simply from docs that already exist. And a knowledge assistant really is just a like rag, right? So retrieval augmented uh generation. So it it makes it really really easy to create rag chat bots um without really writing any code at all. Right? So you come in here, you say here's what the bot is, here's the name of the bot, here's what the bot is.
You can go here and select a source for some kind of uh data for that bot. So this is the rag aspect. Um you can select UC files or you can select a vector search index. In our case, I'll I'll show you how we can use UC files. So I can select a source and I have a volume that has a docs folder and that docs folder has the operations manual for. So this is the actual uh manual for the employees at the actual location.
So I can hit confirm and we can give it a name all this kind of stuff and then we hit create agent and then it takes some time and it automatically creates a bot for us. I won't click it here because we already have it running. So I'm going to leave this page. Show you location operations bot. And here you can see it's already been synced successfully with all the op all the docs that are in this volume. And then I can ask it things that are in that document.
So for instance, hey, what do I do with a late order? It's really, really, really, really late. I'm panicking. I'm panicking. I'm at location. I don't know what's going on. It's really late. Um, so we already got our first token. So you can see that it did this. Um, and it gave us a thought chain, right? So we can actually see what it did in our in this case, all it did is search the docs um, that it has available. So this is kind of the rag aspect.
Um, and then you see it says, hey, log the delay, handle the communication, and it actually also gives us the exact location of what it's referencing, right? So everything's built into this already. So actual citation. So hey, this is cited from page three and this is also cited from page three and it gives you the little excerpt that it's actually using. So knowledge assistant live um generally available on the data bricks platform.
Um I think this is an amazing tool. You basically don't need any coding ability anymore to make something useful. All you do is just feed it the docs that you have um and you're good to go. >> Sweet. Okay then. Uh, so I guess to have this working, I'm going to need a bunch of docs somewhere, aren't I? >> You could also use vector search. You did see that. >> I don't have a vector search at the moment either. Uh, >> you can just >> I I could make >> That's the beauty about drawing stuff.
You can just spawn it. >> Okay then. All right. So, I I think I'm going to have vector search here. Uh, actually, maybe also did like a custom writer instead from our for each match. So we can also have this go into a vector search as well. >> I like that. Yeah. >> Yeah. Um and so from our vector search, this is not following the rules of left to right, is it? Never mind. Um from our vector set, >> it's all relative >> knowledge assistant.
Sorry, what is the specific bit that's new about it? >> It's now generally available. It's gone through its betas. It's gone through its previews. And now it's GA, baby. >> Properly GA. Lovely. The thing that comes out the end of that is a is a little chatbot uh that you can use or you can use it as part of something bigger as well >> and it's available like it's available as a serving endpoint, right? So so it's you can use it through APIs, you can use it in the playground immediately, you can use it as part of a multi- aent.
So one of the things that agent brings has is a multi-agent orchestrator. So you can orchestrate multiple agents for one specific task. So you can include them in there as well. So, >> go into my big agent as part of it and uh there's some other stuff that's come in and there's some other stuff that's going to come out. Okay. What do we think so far? >> Yeah. Question question is the is looks good. >> Yeah. I mean, we're a forward thinking company.
We're using AI. We don't know what for yet, but we are using AI. >> Cool. Yeah. >> And my last feature actually might help you uh with this uh and that is uh autoscaling for lakebase. And now this is in public preview. Uh this means that you've got the ability to scale to zero or you can scale up quite quickly, but you can also scale to zero as well. So no more choosing capacity units uh for your lakebase instances. And there's a few other things that come with this as well.

`27:05` Lakebase Autoscaling

So the autoscaling also supports alles uh access control lists um and you also can connect to it with the SQL editor for readr access as well. So just like you would with kind of uh provisioned uh lick base, you also get those things um with the autoscaling version as well if you want to go and have a play around with it. Uh so step one, you do have to enable it uh in the previews. Uh in terms of where to find it, so I normally go to compute and then lakebase provisioned.
Um and then it's got this handy little link here saying lakebase autoscaling. And what you will notice uh is that it has its own brand new uh UI for you to be able to use uh that you might not have seen before. So if I go ahead and I create a new project uh and I want to have something custom custom uh so I could have something for kind of like all right for cupid app uh you get the postcrist version and then there's literally very little that you can go ahead and choose in terms of the settings here.
Uh so um the production and then I've gone ahead and I have created that uh and this will go ahead and do some setup for us uh whilst it's setting up. But you get quite a lot of metrics here um in terms of that. And then if you want to go ahead and connect to it, uh there is this button up here. Uh and this will show you how to go ahead and connect or give you the things that you need to connect your app or whatever it is that you're using uh late for um to go ahead and do that.
So you do have this new interface here which I really like. Uh it's quite nice. >> It's amazing. I mean I feel like you didn't even mention it, but look how fast you have a database up and running. >> Yeah. Uh like that's what what more can you say? Um >> yeah so I guess maybe you should also talk about like the benefits of lake base if you've never used it before. >> Your future you tell me. I don't know. >> Uh so yeah so lakebase was announced over the summer.
It is the ability to have Postgress uh and set up your own Postgress to have an app or equivalent uh using Postgress and one of the best things about having it with data bricks is a couple of things. One it is right next to your data. So in terms of syncing that data in there, getting that data back out again, uh that is something that's incredibly low overhead compared to if you tried to do this yourself, you also have the ability to do things like branching as well to be able to take a snapshot at point in time.
So that if you want to do some testing, but you don't want to have to um you know test in production, which is obviously a terrible idea, you can branch it. Uh and that makes it very useful. And finally, you have separated storage compute as well. So, um, if you don't need that compute, you can turn it off and the storage is still there. You don't necessarily have to destroy that. So, um, this is part of the autoscaling piece.
It's just kind of an extension of that last point. >> Amazing. I know exactly how we're going to use this. >> Excellent. >> And here's what happened. Cupid got back to us and he said, "JBC is cool, but >> I'm going to need an app to really work on the matching here." So, we're going to do as you might expect, we're going to add the Cupid app. >> Okay. >> And then underneath the Cupid app, Lakebase, right? So, we're going to use data bricks apps to make the Cupid app.
Underlying that Cupid app, we have Lakebase running with all of its wonderful features as you've uh elaborated. And into this Lakebase app, >> boom. >> Can we not have our big agent with a big question mark? We not have that. >> Oh. Oh, we're having it too. Don't worry. >> Right. Okay. but I I wanted to draw the line between this Delta table and the lakebase app on purpose. You actually demoed this at uh data and AI summit last year.
So we can reverse ETL as they call it. Um so common ETL is going from something like an operational store like Postgress, MySQL, blah blah blah into you know delta tables or parquet or whatever you whatever you're using. Uh reverse ETL is the opposite, right? So we have these delta, we have all the stuff from the Jura connector in this delta table. That's our source of truth. We're going to reverse ETL that into the lakebase instance.
Uh we're also going to do all the agent stuff, all the recommendations, and that's going to expose it to the Cupid app. And then Cupid is going to be super happy with the Cupid app. All right. What do you think? >> Uh made? This makes total sense. We This is the first time we've had the actual end user fully as the focus of our architecture. >> That is really nice actually uh to be able to deliver something that could be useful to someone >> to a fake fictional thing in the sky. >> Yeah.

`31:52` Rating

Next time we'll move on to like real users. So, um out of 10, how would you score this,? >> Eight. >> An eight. Okay then. Yeah. No, I mean I feel like there's there's maybe a few things in here that are a little bit optional. So that delta share is iceberg it feels like and it's an extra step, but if HR are an iceberg team that refused to use Delta, then maybe that is actually the right thing for us to do. Yeah, I mean I feel like it's a little bit convoluted and we missed a few steps of kind of classic ETL in here, but I guess if we've got Spark declarative pipelines, uh we could also do a few steps in there as well.
So yeah, I you know, in terms of architecture, I agree. I think this is uh probably a seven. I mean, in terms of use case, it's obviously like minus 10, but in terms of architecture, I want to say like a seven. >> In terms of use case, minus. What? What are you talking about? You got no love in your heart. Just keep it, baby. >> All right. And on that note then, um, what do you think about this slightly unhinged example that we have come up with?
I will put a link to the, uh, documentation in the description below. I also have a bit of news from myself and as well. And quite a lot of you have messaged us to say that you really like this, but actually you kind of wish that it was on Spotify. So, we are looking to move a lot of this to Spotify. We'll still be on YouTube, but we're also moving it to Spotify, and I'll put a link to that in the description as well.
So, if you'd prefer to join us there, you can do now as well. Uh, with that in mind, thank you so Testing. Testing. 1 2 3. >> Okay. Lake flow. Yeah, I like that. Should we come up with a like a datab bricks tongue twister? >> What's difficult? Lots of L's. Lots of L's. >> L's. This is actually something we can ask. Chachi BT. Tongue twister datab brick technologies. Go >> do it. Do it. Do it. >> Tongue twister datab bricks.

`33:53` What even is this

Buzzwords. >> Oh, wow. Okay. Uh, I don't know if I can do this. Staple Spark stream synchronized skewed session statistics. Unity catalog universally unifies users usage and underlying assets. workloads. Weave warehousing workflows and workloads without whiplash. Photon powered predicate push down processes. Pabyte scale park delta live tables. Dynamically derived dependable downstream data sets. All right, enough messing around.
Welcome to the world's number one podcast with and

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.