Low-Latency Data Mesh with Lakebase and Iceberg on Databricks

Summary

Warner Music Group faced a data trust crisis where multiple systems returned different stream counts for the same artists, caused by 600 pipelines and billions of daily rows with no quality checks or single source of truth, prompting a complete architectural redesign on Databricks.
WMG built a production data mesh with five foundations: data quality metrics, data explainability with golden tests, compositional and interoperable data products, an engine-agnostic layer with 1.7 petabytes in Apache Iceberg enabling Snowflake interoperability, and Lakebase for low-latency operational serving.
Lakebase reduced serving latency from 3 seconds to 130 milliseconds, enabling natural language queries through a Postgres-compatible interface, agentic AI workflows with operational memory, and real-time API serving for artist analytics and label operations.

Low-Latency Data Mesh with Lakebase and Iceberg on Databricks

Warner Music Group faced a data trust crisis: multiple systems returned different answers for stream counts, making KPIs impossible to set. With 600 pipelines, billions of daily rows, and legacy architecture without quality checks, the organization lacked a single source of truth. Leadership demanded AI capabilities while the company struggled with data quality.
WMG built a production data mesh on Databricks with five foundations: quality metrics, explanability, compositional products, engine-agnostic Iceberg, and Lakebase for low-latency serving. Learn how declarative Palumi architecture manages data products, Iceberg enables Snowflake interoperability, and Lakebase cuts latency from 3 seconds to 130ms. Discover how this platform powers natural language queries, AI agents, and operational APIs.
🤝

Chapters

FAQs

What was Warner Music Group's data trust crisis?

WMG had 600 pipelines processing billions of daily rows with no data quality checks, causing multiple systems to return different stream counts for the same artists. This made it impossible to set reliable KPIs or trust the data powering decisions across labels, rosters, and revenue management.

What is Lakebase and how does WMG use it for low-latency serving?

Lakebase is a Databricks-managed PostgreSQL service that provides sub-second operational query performance. WMG uses it to sync curated data from their lakehouse into Lakebase, reducing API serving latency from 3 seconds to 130 milliseconds and enabling natural language queries through a standard Postgres-compatible interface.

Why did WMG choose Apache Iceberg as their engine-agnostic data layer?

WMG uses Iceberg as an open table format to store 1.7 petabytes of data, enabling Snowflake users to query the same tables that Databricks pipelines write to. This eliminates data duplication and gives different teams access to consistent, governed data regardless of which query engine they prefer.

What is the declarative Pulumi architecture WMG uses for data products?

Pulumi is an infrastructure-as-code tool WMG adopted to define and deploy data products declaratively, where each product is described as code specifying its dependencies, quality rules, and interoperability contracts. This makes the data mesh reproducible and version-controlled rather than manually managed across hundreds of pipelines.

Full transcript

[00:09] Well, hello and uh welcome. Uh I'll try to stay away from that speaker. I'm told it has echo, but uh we'll see how it goes. Uh obligatory forward-looking statement. Uh read at your leisure. Um so yeah, if if you're here on a Thursday afternoon, thank you for that. I realize you probably had chilled beverages you could be attending to and things are
[00:24] shutting down and all that. So if you're here and mostly functional, thank you. That's really I really appreciate that. So, uh, what I'm going to be talking about today is sort of our arch just an overview of our architecture. Um, I think there's a lot of like you probably covered a lot of the capabilities
[00:39] everywhere else and I actually just wanted to show you kind of what we're actually doing. So, there's a lot of content in the slides. Obviously, we've got 30 plus minutes to go over. not going to dive too deep into a lot of it, but I hope if if there's just something that maybe you can take away, it's maybe
[00:55] some inspiration if you're newer in there or maybe you're further ahead on it. Um, one thing about WMG Warner Music Group is we've been pretty leading edge in both iceberg and lakebase. Um, so we've worked with their teams a lot. It's been super exciting and so I just wanted to share a bit about what we're
[01:11] doing um and jump into that. So just a little bit about Warner Music Group. You probably have heard about of WMG, maybe not. If you haven't heard about it, you've definitely listened to the music since uh WMG has uh 75 years of music history. Um so maybe your favorite
[01:28] artists actually are are are WMG artists back in the day or even now. Um and Warner Music Group is home to like a number of amazing labels. Um and the mission is to turn dreams into stardom and audiences into fans. So we are
[01:43] basically here to try to really help our artists succeed and um I think it's a really interesting exciting time with especially with AI and everything that's happening there um and the technology that we have to help our artists. It's super exciting. So um WMG tech where do
[01:59] I fit? I'm a senior director of engineering within WMG global tech. That means that I lead the data and insights organization. So I own all of our data bricks infrastructure. I own all of our interoperable data, analytical services, all of our analytical data, all of our
[02:14] dimensional data. That's like rosters and tracks. So I basically own our entire data space. Um, and I support our BI and RM analyst teams. And I support our labels. I support our global functions. So they are always coming to us asking for data and AI requests constantly. It's pretty exciting. It's
[02:30] pretty fast-pac. and we've got a lot of teams that are working on a lot of different systems helping these in a very eclectic diverse environment uh with a lot of different competing concerns. So that's kind of the the mode we operate in. And so I'll jump into what I'm going to talk about today. So
[02:47] this is sort of like my my highlight of the things I'm going to go over like so I'm just going to talk a little bit about the challenge we kind of faced. We've got a lot more challenges but I'm sure it's not new to anyone fundamentally data trust. Um I want to give you a bit of an architectural vision like what is the end state we
[03:02] were kind of going for. Um how we actually went about establishing better data quality and trust which is a work in progress. Uh we're not there yet. It's it's always a moving target but um it's it's coming together. And then how we're doing our data mesh approach um which I'll talk about in in in a little
[03:19] bit more detail on how we do that. And then we'll come around to lakebase and and how that fits in because that was a pretty gamechanging capability for us in this environment. uh as well as iceberg. So we kind of got to go the full stack. You got to go from low latency OLTB serving through from interoperable data
[03:36] from other engines. So and I'll talk a little bit about what we're doing with AI in this environment. So a bunch of stuff got to cover a bunch um and hopefully you find some of it interesting. So let's start with data quality and explanability. So this is the main problem kind of that we had. I just kind of this is almost a direct
[03:52] quote. Uh, why do I have three systems telling me three different answers for how many streams this artist has had since her release? And uh, how in the world can I set a KPI for stream growth when I don't know what that number is? She's laughing because she knows it's true. Uh, this is exactly the type of problem we face. And then they come and they're like, Mike, what's going on? Uh,
[04:10] what's true? Right? And they have an existential crisis, an epistemological crisis because you can't know how many streams you've actually had. Um, so what's where this kind of arrived was pipelines at WMG were originally built without quality checks. So there's
[04:26] hundreds of them that are just churning through data in just kind of a spit data out. So like that's kind of what this tech has entered into. That leads to this kind of like fog of war. You got 600 pipelines, billions of rows of data, and then people have built a whole bunch of ad hoc ways of looking at that data
[04:42] and queries, and it's it's really tough to know what's true. Uh then on top of that that data is really hard to actually access. If you have to query an artist's total streams across say a specific demographic going all the way back to when streams started in I don't know it was 2010 2012 um that's a giant
[04:59] query to to do and so you have to look over hundreds of terabytes of data to pull that. It takes a little while and so this is a challenging thing for us to deal with. And then of course at the same time now as I'm sure all of you are experiencing if you're in a tech enabling org uh your non- tech org is
[05:15] asking you how they can be like faster with AI. And so we have this onslaught of things coming at us being like hey how can we support these teams with AI while fixing data quality while they can interrogate data with natural language which just amplifies and force multiplies the data quality problem you had in the first place. Right? So there
[05:31] we go. Now if you're in that boat then you know we can commiserate together and all that. Okay. So this is a bit of our architectural northstar and then I'll jump into the five foundations that we're using to kind of fix it. So the idea is we want fast consistent data and this is sort of starting from the
[05:47] bottom. So we pull data from other engines like snowflake and other uh other sources via iceberg which I can talk about that goes into our data mesh in managed in unity catalog. We serve it through lakebased to our apps and AI. That's the idea. And this is actually a
[06:04] a a pretty realistic architecture diagram of what's happening. It's a little small. Apologies. Uh if I was a marketeteer, you'd have better slides. I'm not. I'm an engineer. So here there's uh the business users go through the API gateway. Now marketer wouldn't have put API gateway in there. So there you go. That goes down to the what we
[06:20] call our insights mixer. That is our analytical serving plane. So it's an API surface that it's kind of inspired by the idea of an ads mixer. Like can you just ask it for an insight and it gives it to you? And of course you know with natural language interfaces this is truly a possibility at this point. Um
[06:36] and then that goes down to a set of sort of domain services uh that use uh data bricks SQL and lakebase variously for either analytical queries or operational queries. So the idea there is that our serving is primarily lakebase. We get a decentralized data mesh and unity
[06:52] catalog and it's interoperable. So I'll talk a little bit about that but let's jump into the first foundation data quality. So, one thing that I've kind of learned is that a lot of data quality problems are actually explainability problems. They're not necessarily a data
[07:07] quality problem to begin with. People say the data is bad, but what a business user when they say it says that often times they don't have a baseline anyway. Sometimes they do and and and maybe in your environment they do more. What they really want to know is how did this number arrive? So, what we did was we
[07:24] came up with something we called the data quality system instability score. And I'll show you a an example of that next. And what it is, it's it's an aggregate score weighted for data quality metrics where we have a single number that can tell us if something is going wrong at some level in a weighted
[07:40] way with our data. And that at least tells us like along the main stuff like correction rate, null rows ratio, invalid rows, consistency, all of that if something just fundamentally basic is wrong. Now, you may already have that. Well, a lot of our pipelines had none of that. So, so this is actually we're
[07:57] we've introduced this this paradigm and and it's given us a lot more confidence that hey, we are actually doing the right thing with the data. Now that we have that right thing with the data, we can help explain what's going on with it because we know we're not screwing it up, right? Because that's the first
[08:13] thing like you can't explain it if you're screwing it up partway through. The second thing there is like deep cost observability. So we've we've really wired in a lot of cost observability across engines. So we have a really really good handle on like what's costing money where and uh and we keep a
[08:29] really close eye on it. So that allows us to make principled decisions about where to invest and where to run things and how to run them. So we've actually we don't have ballooning costs on the uh compute side which is really awesome. We've actually got a pretty good handle on it. Um and now with our data product
[08:45] platform this quality approach is integrated as a first class citizen into every data product. So I'll show you an example. This is maybe more interesting. This is actually a dashboard of one of our data products. Um, we call it fact audio streaming. And what you're seeing there is that there's a bunch of DSPs. We call them a digital service provider.
[09:02] That's like SoundCloud or Spotify. And you can see we've got scores for each of those from 0 to 100. And uh, if this is actually all clickable through a data bricks dashboard, but you can drill in and you can actually look at all of those different quality metrics for each DSP in real time. And we weigh it by
[09:19] size of DSP. like Spotify is huge, you know, SoundCloud smaller. So when you look at that, you can actually go and weigh the the uh totals and and you see that total number at the top 34.72. It's sorry, it's a little hard to see with contrast, but the legend there, it
[09:34] it as that number climbs, it's getting worse. So we have a weighted way of saying at a glance across all of this data, is something materially happening with it? So if a small DSP has a big problem in one area, it won't hit that much because that's not going to have much business impact. But if like a big
[09:50] DSP has a moderate problem, that score is going to surge and we get it. We can see it right away. It's a lot easier for us to manage this across dozens 100 DSPs. Um, so that's just one example of what we're doing. And this we would apply to all our data products. So we
[10:05] get to do this at every data product. This helps tell us that we're doing we aren't screwing up the data while we are transforming it. That allows us to back the problem up into okay are are the semantics right? Do we get that properly? This comes to data explanability. Um, so
[10:20] as I said earlier, data quality problems are often really explanability problems in my experience. Um, and data is wrong can only be validated if you have a baseline. So what we've actually done is we've got three things going on here for this. One is golden tests. You're probably already using them already. If
[10:35] you're not, use them. But basically what we do is we actually go and take real cases usually just like a historical or back test and we actually go check the numbers manually and we work with our business partners to make sure they make sense to them and then we set up a test
[10:50] and we run that test on the pipeline and we make sure that the pipeline always produces that answer. And so we have got a lot of these and these tell us that hey we validated this. You looked at the golden test number with us. We know this is true. We know the transforms are working. So we can extrapolate that to
[11:07] the other data because you can't exhaustively test data. That's like literally impossible. So you have to use a combination of like uh essentially unit and and regression testing combined with golden tests. The other thing is like we're really working a lot more with our BI and partners and this has to
[11:23] do with the data mesh idea to under to define the semantics because like what is a release date? Um in music there's not one release date. A release date is a vector and there can even be releases before a release date. So it's really complicated. So when you're asking like
[11:39] how many songs has an artist how many plays has an artist had that is like a crazy non-trivial question. It's like an existential metaphysical question of the universe in my world. And so you really have to define that stuff with your business partners who are actually using it to make a decision like what do they
[11:54] mean by it? Right? The other thing we've got is we've kicked off this really fun thing lately, our data steer code, where we actually can come together with memos across our business function. We decide on this. We're like, this is what source of streams means. This is what we say when we mean release. We all agree on
[12:11] it. We have our BI partners helping to champion that around the business and tech helps to implement it and and we all get along and it's all happy and and all of that. So uh that's just an example of some of them. API level golden tests like retrieve track with start date and end
[12:27] date parameters and it returns the appropriate tracks. So we do stuff like this. This is just like a little tiny snapshot of API level golden tests on our insights mixer which is serving on top of our data mesh. Okay, let's jump into the data mesh. There's two aspects to it. The compositional and the interoperable. So I'll kind of rush
[12:43] through this a little bit. So if you're not familiar with a data mesh, the way you can go read the book. It's an awesome book. Highly recommend it. The way I would put it as a software engineer is it's taking the concepts and principles of software engineering and applying them into data engineering. So what it is is it's taking domain driven
[12:59] design which has been around since the 2000s. It's taking encapsulation which has been around since the 1960s with object-oriented programming and it's taking these ideas of of software and putting them into data. So instead of having layers of data with a whole bunch
[13:14] of queries running on them transforming it, you are trying to encapsulate which I have there a a data into a product where all the things for that particular domain representation of data are in one thing. So what that gives you
[13:30] is composition. So rather than a single medallion in your lakehouse, what you have is each data product is a full medallion complete with potentially APIs and it is the source of truth for that representation. And what that gives you
[13:45] is a bunch of Lego blocks that you can assemble together and put them together in a stack of reusability. And so the way that you start to handle quality and explanability is you extract and bubble out data products which we've done so that you aren't duplicating those transformations. Set aside the compute
[14:02] cost. As soon as you duplicate you've got aggregation window timings, everything's different. So a compositional data mesh is a really powerful way to make sure you're just doing it in one spot and you don't have a twoclock problem. Um with that there's this what we've done is we've taken a few things from it. We can't fully do
[14:18] this. A data mesh would have like domain teams like a business team like finance building their own data products. We're starting to get there with AI, but really we have tech kind of driving this a lot and just kind of doing virtual matrixing with our business teams. So we
[14:33] can't do the full thing, not the full decentralized ownership. We're taking a lot of the principles within a data mesh. You've got source aligned, aggregate, and consumer aligned data products. And that's a really that's a really good paradigm for the types of Lego blocks that you're building. A source aligned more fits the source.
[14:49] Like say data from Spotify for us would would be represented in a streaming data product. In the middle, aggregate data products are those ones you bubble out that you reuse and they become a self-standing representation in Lego block. And then on the at the top of it, you've got consumer aligned and those are the ones that are representing the
[15:06] actual customers uh uh domain representation. And you can have data products that do all three in one. That's that's what we've got those two. So we have a lot of teams building data products um in my org data foundations, inside services, marketing and enablements and we have um we have now
[15:22] more non- tech teams starting to do this as well. So I'm going to talk about the data product platform. It's a that we've built. It's a declarative Palumi based platform and it essentially realizes data products declaratively and lakebased serving declaratively which is pretty cool. So um this is an example.
[15:39] This graph here is one that we we kind of put together to show where business intelligence and tech would own different aspects of the data product stack. Um so you can see it's got uh consumer aligned aggregate and source aligned data products. Uh we because
[15:55] source align typically require a lot more data engineering expertise. Tech owns those. But we kind of have it blurs as it gets higher and we really want other teams to own more of the the um consumerf facing products and we we have some really cool stuff happening around our business where teams are building
[16:10] out really cool semantic definitions on on our engines. So we found data mesh is more helpful than a medallion and helps us to represent these sources of truth. So each product is each is at least a medallion. And um the reason for that is I put a note there. If teams are just
[16:26] responsible for a bronze layer, they tend to be very disconnected from the value and use cases. And that leads to poor explanability and low trust, right? Because the they're they're always just serving ad hoc queries without context. You need to build data like a product
[16:41] where a product owner actually drives use cases that vertically drill through your stack so that you build things that make sense. So that's what we're doing rather than horizontal buildouts of layers, vertical drill down of use cases. Okay, this is an example of an
[16:57] actual data product, our streaming data product. This is just one inside one data product. So you can see it goes from S3. It's primitized for DSP and environments. It has trans a silver and a gold layer all in one. It it does further unioning and downstream aggregates. So this and it has data
[17:14] quality aggregates that flow off of it. This whole thing is in data bricks um except for the the left part which is um you can see snowflake analytics workbench. We actually use iceberg to federate the data in and then this data products there. So this is a really neat idea of data mesh compositional and
[17:29] interoperable which is the next thing I'm going to talk about. So again, uh actually that contrast worked a little better than I thought it would. Okay, so the interoperable data mesh, what you're looking at here is is a snapshot high level of our actual data mesh. So on the bottom there, you have our data product
[17:46] platform which is realized via Palumi. So we actually declaratively like a Terraform cloud for type thing. If you're not familiar with Palumi, we actually declaratively realize a DAG or uh this this directed graph for our data products. So we have a whole bunch of
[18:03] data products with sources there that feed into other composable data products that end up in like consumer ones like campaign planner and audience insights and performance insights which are serving our apps. So this is the flow of data through the stack. You see all those arrows, there's lots of them in
[18:20] the top two layers, but they go one direction and that's key. So that's it's unidirectional, which is great. And that allows us to actually um have a reusable uh set of products. And you'll notice streaming can both be used as a consumer product, but it can also be used as a
[18:36] feeder to consumer products, which is cool. this already um but uh data products are defined in provider code. So it's
[18:52] actually a codebased definition and we actually declare that via Palumi. So this whole stack comes together declaratively. Everything's versioned together. Um which is great. Um and dependencies are actually inferred from the input configuration. So there's a
[19:07] bunch of sources with a data product. You can just say like here's what I need and it pulls those sources in. So it's really cool. You get access to those and the platform realizes it for you. So, it's a code native data mesh with explicit contracts. All right, I'll keep moving.
[19:22] Um, oh, oh, yeah, that's the one I wanted. Um, dependency graph. I actually talked about this one already, too, where the platform is essentially constructing a graph, validating it, and then it's able to traverse that graph in the appropriate dependency order. So,
[19:38] that that enables like a very deterministic sequence of processing through the the the pipeline. So that's how we've kind of orchestrated it. Um we have flexibility in our platform with like what compute strategy we're going to choose. So whether it's serverless or
[19:54] like a traditional cluster. So that allows us to like change that on a per data product basis. Um we have we're able to configure the job scheduling for these data products as well as we can use uh SDP and and um Delta Lake to actually uh flow data through as well.
[20:12] So I'll keep moving to the iceberg part. So what we're doing and I I talked about this previously at another at reinvent but we've created an engine agnostic data layer. So we have I just checked yesterday because I wanted to confirm. I think we have about 1.7 pabytes in
[20:27] iceberg right now shared between Snowflake and data bricks. So and it's working really well. Uh which is pretty cool. They're all V3. We're doing V3 now. Um and uh so that's what I mean by we're cutting edge. we jump into the stuff quickly. Um, and
[20:43] all of our tables in uh, uh, we actually have materialized views. We've been working with data bricks on that, but we've got MVs now through iceberg, but our our main tables are delta tables. And this is actually how we have them configured. Uh, you'll see it says v2. We actually have v3 now. And then um,
[20:59] we've enabled iceberg format. So what's great about that is we actually can share this data in an engine agnostic data layer where our analysts who are primarily uh running on snowflake through their BI can actually use stuff from our data mesh which we house on data bricks and similarly a lot of our snowflake data we federate over and now
[21:16] we can do all this cool stuff on data bricks with it which is really neat. So this gives us like a really great way to do it and this is built into our data product platform. So we're able to actually realize this uh declaratively. So it's zero copy. We don't have to run feder federated queries and do double
[21:31] compute and uh it makes our data mesh available to the entire company which is really awesome. And like I said we have a lot of data. We get billions of rows a day flowing through this system and we have pabytes in there. Okay, this I'll talk about this in the fifth pillar, but
[21:47] I wanted to bring it up in the context of our data platform because lakebase we built in as a fundamental uh first class primitive in our data product platform. So what that means is that you can provision lakebased declaratively with our Palumi platform. So a developer can
[22:02] just boom boom boom lakebase boom and it will programmatically realize the lakebased serving for that table. Um which is really awesome. So that gives us the ability to say like what actually is in a synced table. Um in that case for us it's the source table primary key
[22:18] scheduling policy. We're still doing like full syncs at the moment. We do have triggered enabled but we haven't applied that yet. Um and so we're using sort of like chronype scheduling for this in our context. Um but that that allows us to actually manage lakebase
[22:34] along with our data mesh in one versioned environment which is really cool and it means that lakebase can actually be logically a part of our data products. So we're able to serve through that. So I'll talk about that a bit more later. Um this is just I I sort of
[22:50] mentioned this in the versioning thing but uh we actually have pretty uh strict sync table updates. So like core fields are immutable. So we often have to recreate. Um but the immutable fields include the source, the primary key, scheduling policy, and the logical DB. So that that makes sure that while we're
[23:07] doing syncing, we're not messing something up. Um in the current mode that we have, we'll be moving to a more triggered mode and uh taking advantage of some of the the nice announcements that Lakebased team has made this week as well, which are which are pretty cool. Um one thing about our system, we
[23:22] actually have a custom approach to incrementalism. uh we're not primarily using Delta Lake incrementalism. We actually created our own liquid clustering based incrementalism approach. Uh and that was because the way that a lot of updates come in from DSPs require highly parameterized
[23:38] updates that could be either replace or inserts or appends. So we needed a way to be able to sort of like whether whether truncate patch or increment or upsert pretty dynamically different sections of our streaming data. So we kind of came up with our own
[23:54] incrementalismbbased approach for that. Uh it uses Delta Lake tables but it's not uh using uh the the native incrementalism. So I thought I'd mention that. Um it uses sedd type one merge semantics but uh it's basically using kind of this watermark and start stop dates and trying to apply the data in
[24:10] intelligently based on what a whole bunch of DSPs are giving us. Um okay so bringing it all together in this section we get an engine agnostic data layer. It's governed in Unity catalog which is pretty cool. Uh with an optional lakebased enablement where desired. So
[24:26] this is all like available in the data product platform. We have strong contracts on all of that stuff. We don't have to worry about like configurations drifting. It's actually all strongly versioned and encode. And this gives us really good interoperability because we can actually pull data from different
[24:43] engines and seamlessly serve it up via low latency serving up through lakebase. So this stack is actually kind of kind of interesting. Um, and we we actually have cases where data goes back and forth between engines a few times, which you'd think is crazy. It probably is, but we're doing it and it works pretty
[24:59] well and it's not expensive. So that's awesome. Um, and so yeah, we call kind of this delta native with iceberg compatible table strategy for now. You can imagine we are very excited about V4 when it all comes together. So we're we're very there very very excited about
[25:16] that. So why is the data so slow? Uh that is something that we got a lot because when an analyst or a uh marketing person goes and uses one of our analytical tools and they're like why is this query taking two minutes to run and they they lose interest by the
[25:32] time it's done and they build their own queries to go and figure it out and then you get sprawled data. So this is where low latency serving really makes a difference. So I'll give you a bit of our history of lakebase here. one year ago today about today I don't know when the data bricks summit was last time we were actually
[25:48] building APIs for serving insights to applications um and they were directly reading from data bricks DLT so it was extremely slow like 20 to 30 second for a query time um it's not not great and so the aggregations are fairly complex like our
[26:03] one of our streaming tables we we do have it we do break it out into different aggregates to make it more manageable but it's like 22 terabytes across like artist release to date country demographic when you actually denormalize the whole thing and you actually have to kind of do that in order to track certain like identifiers
[26:19] all the way through to the end. You you can't keep it fully normalized while doing that type of aggregation. So in May 2025 we actually started building our own low latency Aurora Postgress based serving solution. We designed the thing. We started building it and we estimated a ton of work. And then Lake
[26:36] Pace Lake Base was announced I think in in preview in uh I think it was preview June 2025. And so we saw that we're like wait let's uh let's just stop what we're doing and take a look. And we did a quick test and we were able to serve up with via low latency operational serving
[26:53] within like minutes. And we're like this just changed the game. So we just literally stopped working on everything we were doing there. salvaged all the time back, went all in on Lakebase on it. Um, and that actually uh that actually really freed us up to do a lot more. So, we started with foundational
[27:09] lakebased infra, we adopted sync tables, and then we're now starting to get into triggered as well as we build out more uh more sophisticated cases. So, actually, I jumped ahead here. We found lakebase initially lacks some programmatic functionality, but we
[27:25] manually worked around it. Um, and we actually put our API plane on Lake Base and as capacities increased 8 terabytes, 16 terabytes, more aggregates have moved into Lakebase. So, we've been able to put more and more stuff in there. So, what we saw, I just have a little actual
[27:40] snapshot of one of our little latency graphs. Uh, we went from like you can see there, that was just a 3 second query time down to 130 milliseconds instantly with this. So, and that was basically free to do, right? um because
[27:56] we didn't have to do all the engineering to make that happen. Uh we just connected the table, boom, done. Um and that has enabled a lot of performance improvements across our systems and uh we're super excited about where it's going. So this is actually what we have
[28:12] running on lakebase today on our data mesh. So you recall earlier I'd mentioned some of these products, audience insights, campaign planner, performance insights. So these are actually the sync tables that are running through lakebase and their primary keys. And what we're doing there is that um the this is just that graph
[28:28] you see on the side there is actually the uh it's a little architectural snapshot of how the data product platform we have declaratively realizes that data product stack with lakebase so that it's all nice and declarative and
[28:43] programmatic uh to be able to serve up from these data products. So that's there's three different data products there that are connected to lakebased declaratively. So, um, this makes it really nice and elegant and easy for us. Not that there hasn't been hiccups and challenges along the way and we're
[28:59] always talking to Lakebased team and that type of stuff. Uh, but we've achieved uh something pretty stable and pretty fast and we're using it more and more uh every every iteration. So, I I just threw up here for interest sake like kind of what we made configurable versus what we made immutable when we're
[29:15] talking about like whether we would have to reync a whole table or just change something that's configurable. Um, so I won't dive into all that, but it was kind of an interesting thing that I thought would be neat to share for for people who are a little bit more interested in what's going on with our columns and keys in Lake Base. So if
[29:31] you're interested in that, you can always check out the deck after. Um, so here's here's some of the benefits here. Like we got a clear abstraction layer in here. It was really nice to be able to have a data product platform that's realizing these declarative data
[29:48] products that then have this optional lakebased serving. So that's really great. Um it also has enabled us to actually manage security very well because our Palumi based stack can ensure that the same security is applied to everything. Not that we don't have governance problems but in this case we
[30:03] can actually declare the governance and it's really solid and obviously Unity makes sure it's synced between uh lakebase and um unity. So, uh, we actually have explicit control over when we're refreshing. Um, which is good. So,
[30:18] because we're using sync mode right now, more so than triggered. Um, and we have Delta first platform compatibility. So, like our upstream tables are Delta with iceberg compatibility, which lets us pull data from other engines. And then lakebase handles the OOLTP serving cases
[30:34] in this kind of full stack setup. Um, so that gives us like the ability to get a full stack going without duplication with compositional data products that can be used analytically and operationally kind of where we land. Um,
[30:50] so I wanted to just show like the architecture here for uh what we call our insights mixer. So this is our API plane for serving up data. Uh, and insights mixer is basically serving up via lakebase. And so you can kind of see
[31:06] there that uh in the middle there that middle box is sorry it's a bit small but it says insights mixer API layer which is like a fast API and ECS Fargate. What we have is it has a feature flag capability for us to toggle whether
[31:21] we're going to serve directly from data bricks SQL and delta tables or lakebase. So that's those two on the bottom. So for each API, the team has the ability to go like lakebase on, lakebase off for each API and that and it's it's like that cool like we can change it which is
[31:36] great because as we're able to test or we have big aggregates we're like oh this isn't working so well but we can use it on these other ones we now have the ability to like dynamically uh use lakebase on on tables as we're developing. So that's kind of what's what's sort of the the main path there to to kind of note is we can actually
[31:52] choose whether we're serving it analytically or operationally um via the API which is good as the tech evolves. So that's the repositories choose the data source and we do so that lets us still do warehouse style analytical
[32:08] queries um while at the same time we can apply low latency querying to cases wherever we can to to benefit it. Um, so yeah, that's pretty uh that's a pretty neat system that I think we've been able to set up there. And um finally, I I've
[32:23] been talking about this the whole time, but it's it's worth noting like Lakebase is not uh it's not merely a separate thing that we've tacked on, but because we have our buil built-in declarative data product platform that we've created, Lakebase is actually like a whole serving extension within our data
[32:41] mesh that's native to our data mesh. And so that gives us a really elegant way to provide operational serving and it prepares us pretty well for low latency serving in an AI use case. So that's really interesting. Um, and it's been it's been a lot of fun. So that's an
[32:56] actual API call uh to the insights mixer. Um, and it uh it returns like some interesting things about superlisters and whatnot, but that would be served by lakebase uh and it would return within tens of mills. So that's pretty cool. Um the final one I just
[33:12] wanted to talk about was like the AI natural language data queries. This um this is kind of the next frontier but we're doing a lot of work here where we have like a internal natural language interface on our data. So what's what is interesting is lakebase is now actually
[33:29] already the default Postgress DB for our natural language serving over our data. And what was really actually interesting about it was the engineering team chose it not for low latency serving. They actually just chose it because they wanted an operational database
[33:45] and uh and so they chose that in data bricks because their data was already there. And so this was kind of the first time we actually used it just as a Postgress database which I actually think is super cool and makes that data automatically available uh in in an analytical environment. So that's they
[34:03] just needed one they chose it and uh it's working which is really cool. So they were able to turn it into their operational memory layer rather than just analytical serving. So we've got both cases. We're doing like readonly analytical serving and we've got this operational layer. And we're I think we're going to be building a lot more of that that stuff as we go forward.
[34:20] So I wanted to talk a little bit about that in a bit more detail like how the agentic AI uses lakebase. So what happens and I think I actually have a diagram for it on the next one, but obviously the user will type in a question. The AI turns that prompt into like structured intents. that in turn
[34:37] gets stored stores lakebased then stores the durable session state like tasks retries errors results progress and uh then that's all synced with data bricks and unity catalog for for governance so we get this uh sort of like workflow
[34:52] ledger in lakebase for all of our natural language querying that we can use to understand how to improve uh natural language uh agentic uh um queries. So that's really awesome because it allows us to make things uh
[35:07] not only like inspectable um but we can also resume it enables like polling and retries and um gives us a lot more explanability as we learn how to do natural language quering over our data. Um and I don't know if I yeah here's the
[35:24] here's the graph for it. So this is just sort of a a diagram of of that what I just described. Um I won't go through that in any detail here with the time constraint but you can kind of see how that data is flowing through to lakebase uh postgress in the bottom there. Um and
[35:40] that's used for operational management of our data during uh natural language queries over our data. Um I will say I don't before I jump to the outcomes one thing that we are going we are looking at right now is actually uh genie enabled um lakebased serving. So that's
[35:58] a pretty exciting thing. So, um, we're going to start to use Lakebase for essentially AI, direct AI query, not merely the operational memory for our agents. We're going to use it actually to serve up data to agents while they're querying. And what that will allow us to
[36:14] do both in Genie and uh, outside of Genie is answer like cut down the total overall perceived latency for our natural language queries significantly. because if we can shave 45 seconds off uh an analytical streaming query, then maybe the other 30 seconds isn't so
[36:30] painful while it's like walking through its like analysis, right? So, um that's that's sort of one of the really cool things that's coming. Um and we're going to keep leading into that. Um so, outcomes and recommendations, start with some outcomes. Uh what we've got here is like much faster serving speed, like way
[36:48] faster than we've ever had it before at Warner Music Group. Um, we've actually improved trust in various parts of our data. This is still a long journey. We have a long ways to go on it because we have a lot of different data areas, but we've actually been able to improve trust where people are like, okay, this number actually makes sense. Um, we also
[37:05] have this really awesome flexible engine agnostic data layer and that's been really really cool. Um, and it's actually allowed us to use data from other engines in data bricks which uh, you know, would have been a lot harder. you would have had to do federated queries and a lot of copies and
[37:20] duplicate storage and everything. Um, and so, uh, we're able to use both, uh, Snowflake managed and external iceberg. And then, uh, we're using Uniform and Delta Lake, uh, to have a birectional iceberg back and forth between the two. And that gives data bricks access to
[37:36] essentially all of our Snowflake. Well, we've got a lot more data in Snowflake that isn't federated yet, but whenever we need it, we can easily federate it. And so, more and more our engine agnostic data layer is building up. Um, what I recommend is like if you're in a situation where you're trying to
[37:52] make this better, uh, just be really pragmatic. Like you can't fix everything. So pick one thing and fix that. Um, make it just a little bit better. Like every big complex system has come together through a series of incremental steps. So sometimes as
[38:07] engineers, I think we like to boil the ocean. We like to see the whole thing come together and build a giant design dock and everything. Start with a product use case, make the data better for that. learn from that, extract, repeat. Um, we I mean I highly recommend if you're in a dual engine environment,
[38:23] whether that's like with other engines like GCP or you have to share data with third parties a lot, iceberg is a really good option for that. We're finding it very powerful and it allows like this birectional interop. And so like that's enabled us a lot of flexibility with where where our data is living and how
[38:40] to get it. Um, and then the final thing is obviously like apps and chatbots need to be fast. So for us, we're we're building in operational serving underneath what would normally be an analytical query for our uh chat bots and whatnot. All right, so that's a
[38:55] little tour. I hope it was somewhat inspiring and maybe some interesting thoughts there for you. And uh that is it. I am done. They want a survey slide. So fill out a survey. Um and uh yeah uh thank you. Thank you for coming.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.