Sponsored by: E6Data | The Hidden Strategic Leverage in Open Lakehouse Architectures

Summary

This video argues that open lakehouse architectures—pioneered by Databricks with Delta, Iceberg, and decoupled compute—give enterprises hidden strategic leverage including workload portability, compute optionality, and the ability to swap infrastructure without large migrations.
Vishnu Vasanth of e6data walks through a total cost of ownership framework spanning infrastructure, people, and licensing, showing where organizations inadvertently re-create lock-in at the application and governance layers even on open stacks.
The video introduces a pilot-prove-swap operating model that lets teams benchmark and change compute engines without disrupting governance or storage, preserving full flexibility as the AI and analytics landscape evolves.

Chapters

FAQs

What is open lakehouse architecture and why does it matter?

Open lakehouse architecture separates storage, catalog, and compute into independently governed layers using open formats like Delta and Iceberg, so organizations are not locked into a single vendor for every operation. This video explains that Databricks helped establish this model as the industry standard, moving enterprises away from the closed, tightly bundled data warehouse model.

How can enterprises avoid re-creating vendor lock-in on an open data platform?

This video warns that organizations can inadvertently re-lock themselves at the application or governance layer even after adopting an open storage and format stack. Vishnu Vasanth recommends auditing each layer—storage, catalog, compute, application, and governance—to ensure each layer remains genuinely replaceable.

What is the TCO framework for open lakehouse architectures presented by e6data?

The framework breaks total cost of ownership into three buckets: infrastructure spend, people costs, and license and tooling budget across the full technology chain. This video uses this lens to show where organizations typically overspend and where open architecture creates room to optimize.

What is the pilot-prove-swap operating model for compute in open architectures?

The pilot-prove-swap model means selecting a new compute engine, running a controlled benchmark against production workloads, and replacing the existing engine only once performance and cost targets are confirmed. This video presents this as the standard operating procedure for teams that want to take advantage of compute flexibility without risking production stability.

Full transcript

[00:09] Hi. Thank you all for coming. Um This is uh Vishnu. Uh he's the CEO of E6 data talking about the power powerful leverage that the open lakehouse architecture gives you.
[00:21] >> Yep. Um Thank you for making it. I know it's the last day of the event and uh uh everyone's like overstimulated, exhausted, and excited by all of the the cool things that you've heard. Um I'll
[00:34] keep this talk quick uh and sort of focus on aspects that I think haven't yet been fully covered uh during the course of the past few days. Uh and I'll try and zoom out uh and take
[00:46] a slightly meta or an architectural lens to the problem. Uh and really go into some unusual uh findings that we have based on what open architectures give us, right? Like and we call this the hidden strategic
[00:59] uh hidden strategic leverage that your open architecture already gives you uh and and benefits that you can get from it. So, the good news is that the closed data is behind us, right? So,
[01:13] uh we were sold this model for decades. Everything was clumped together, tightly controlled and gated, uh and you had this closed storage, closed catalog, closed everything, which meant that you were essentially held hostage and forced
[01:26] to use what your one compute provider or your one database provider gave you. Now, as this has been decoupled, uh you know, and because of this closed model, you had heavy restrictions, which
[01:39] created its own problems, lack of usability, uh lack of leverage, and grossly limited adoption of analytics and AI. Um Databricks with the concept of the lakehouse uh for close to like half a
[01:52] decade or a decade now are sort of breaking this closed model, right? So, they reimagine what uh closed data warehouses look like for cloud-native architectures, you know, and object store native stacks.
[02:04] They pioneered open formats like Delta and now Iceberg, you know, created basically catalog categories like, you know, the governance that the governance and catalog layer that, you
[02:17] know, is decoupled from the storage layer. And really like introduce this concept that hey, you can use multiple compute engines that can coexist within this sort of open garden, right? Like as opposed to a walled garden. Others then
[02:30] meant that you could decouple your applications from the underlying engine and from the underlying governance and from the underlying storage. So, lots of flexibility got opened up here.
[02:42] And we sort of now take this for granted, right? Like this is now table stakes. Now, let's look forward. What this did is, you know, what we see here, right? Like we've been coming here for some years. 5-6 years ago this event had like 1,000
[02:56] people. This year there's 30,000 people, which correlates strongly with the growth in adoption and the explosion in adoption that we've seen. We're somewhere here, right? Like 26, 27 and we're starting to see the early
[03:08] signs of an exponential growth in adoption. Right? So, what does this mean? This is a good thing. We're getting value from this. Our businesses now depend on the data that your teams provide as a lifeblood of all intelligence,
[03:20] analytics, and AI. But with all good things comes a problem. This is cloud economics, right? Like consumption, dry I mean, more you consume, the more, you know, data you work on, the heavier or more complex your
[03:32] computations, the more your consumption. Right? So, this really introduces some worrying signs that I believe will start to peak next year and reach boiling point maybe next year or the year after,
[03:44] right? Like but the early signs that you see is that, you know, it's common knowledge that the compute spend within most companies is growing like like multiple times as fast as the company's top line. It immediately catches the eye
[03:57] of top management and I'd be surprised if any of us haven't sort of had these concerns sort of raised to us. Um and what we're realizing especially from the big tech uh sort of
[04:10] uh moves in the last like quarters is that there's some math that none of us can sort of shake off, right? Like as much uh you know, excitement that there is there's something that's fixed and predictable and those are budgets
[04:23] uh and that it turns out that that budget is not necessarily an infra budget or a people budget or a license budget, it's everything across that TCO chain. And uh uh while you know, adoption is great, we need to find ways to squeeze you know,
[04:36] derive efficiency and continue to keep leverage. Uh and I don't I do believe that even as this uh increase in spend is sort of uh percolating down to your uh
[04:49] uh to your teams, it's still not priced in the impact that agents will have on this, right? Like because they will 10x demand. So, to look at what we can do about this, let's see where this cost actually comes from, right? Like so if we zoom out like
[05:02] and abstract out the names and the jargon, there's really like a few things that we spend money on, compute, storage, uh services and so on. Uh everything that you really spend money on if you like look at the 90% of
[05:14] your bill, it's compute. Uh it's compute to get data in, it's compute to process data, it's compute to serve data and the tokens that sort of wrap these and agent I mean that that serve agentic applications on top of
[05:26] this stuff, right? And it it makes sense because compute is that essential link that connects or transforms data to any form of value like intelligence, right? So if the demand for intelligence is is
[05:38] like exploding, the demand for compute is exploding. So no mystery there, but it's just nice to get this little bit of a zoomed out view. Um and there's this you know, like um
[05:50] there's this sort of false trade-off right now that you know, keeps us trapped, right? Like so, if um you know, if the temperature is rising and we still feel trapped, like we have to ask ourselves why.
[06:02] Um, what we like is on the left, right? Like uh central governance, a single pane of glass, a single security posture, right? Like democratized data through unity governance and so on. We like that
[06:15] single pane of glass for visibility, for forecasting, attribution, uh you know, govern like you know, cost controls and so on. It's also simple, right? Um, uh but on the other side,
[06:27] we assume that automatically implies that we have to use one compute for everything, right? Like and this talk is really to uh subtly challenge that assumption uh and recognize that that might be uh
[06:41] the unlock that really gets you the best like of both worlds, right? Like so, um what you know, this uh status quo means is that we assume this trade-off exists, we're on open foundations, uh but you
[06:54] know, we uh this this false trade-off really keeps us trapped. So, when there is the cost explosion, uh you know, we assume we have no options or no leverage. So, this is sort of like my analogy for what like how we're
[07:07] operating today, right? Like you you are architecturally free, but I don't think you're taking advantage of that freedom because of like what you perceive uh uh you know, is the reality, right? Like it's like a powerful horse tied to this
[07:19] flimsy plastic chair uh that uh is is sort of holding you back. Um, so really, why not the best of both worlds, right? Like why not central governance? Why not like uh you know, centralized security? Why not that
[07:32] single pane of lineage and so on? And the choice of compute for each workload, right? Like pick the like horses for courses, um you know, achieve deeper ROI while still not migrating from these
[07:45] foundations that you've like invested so much and so so much uh, you know, effort and and care in sort of laying down, right? Like so don't really migrate. So that's really uh, the message that
[07:57] uh, you know, I was hoping to convey. Uh, and now the question is really how do how do you do this, right? Like in practice. So your foundations are really your open lakehouse, your Unity governance and so on. Uh, and the flexibility really comes
[08:10] from workload specific compute. Right? Uh, and this really means a lot of leverage, uh, you know, for your uh, teams, for your leadership uh, in getting the value out of data and everything that it powers, as well as
[08:23] like keeping uh, flexibility in your hand. Uh, the interesting thing is that we already do this, right? Like we, you know, our teams do this in our AI stack, right?
[08:37] Like so you, every time a new model provider comes up that says that, "Hey, I I offer like superior performance." We have like really good efficient ways to test that out, prove that is the case and swap one for the other, right? So
[08:50] you what doesn't change is the stuff that you own, the layer at the top, application, value and so on. So the business logic, the semantics, the evals and observability. So so you that that's your domain IP or your company's IP.
[09:04] What also doesn't change is the cool things that you've got from Databricks, right? Like security, governance, auditability, that single pane of glass uh, and so on. But what you can change is the model provider, right? Like there's always a flavor of the season,
[09:16] it it seems there. And similarly, you know, you you know, we we do this in our AI stack, but we haven't fully internalized that we can do this in our application stack.
[09:29] So um, this is sort of like what I think you'll see increasingly, right? Like which is an example of sort of what we do, right? Like so uh, a Databricks and Unity Native uh, independent compute engine that can plug into your existing
[09:42] lakehouse stack, works with what you have, so no application changes, no governance changes, no security changes, no data movement, sits within your infosec posture. And you can selectively turn it on for
[09:54] the workloads that are really hurting you, right? So, this is this is something that's possible today courtesy of the open ecosystem that Databricks has created and courtesy of you choosing
[10:06] mindfully to sort of build on those open foundations, right? So, what could this give you for workloads that are particularly expensive? It could save you between 50 to 80% on compute costs while preserving your applications, your
[10:18] foundations, and everything else, right? And if ever something better comes along, you can swap back out to it just like you do with swapping model endpoints, right? So, you keep your compute when by making sure they're
[10:30] giving you the best you know, best in breed for that workload. Um so, this is sort of what this looks like in practice, right? You could run the two compute instances that you're choosing between for an hour, right? On
[10:45] your actual workload, on your actual data foundations, and look at the difference in price and performance. So, let's say you know, V6 is the one in purple, so over the course of an hour you'd look at how much credit consumption like if and
[10:59] you look at the difference in the area under the two curves, that's really the savings on credit or cost consumption, right? So, you don't have to imagine this, you don't have to take anyone's word for it. You could try it out without like any changes, and then see that if the savings are
[11:13] achieved, you can just swap one out without like that being a one-way door of any sort. Um so, yeah, so this is sort of what the world that we're advocating for, right? Like so, open architectures is the
[11:26] enabler, but really if we stay stuck with single engine or single vendor compute, it negates the benefits completely. What you need is open architectures with a heterogeneity of compute with you having the power and
[11:38] flexibility to decide. Right? So, this it turns out also gives you a ton of leverage, right? Like you don't feel trapped. And so, essentially it's architecturally
[11:51] open, but is it economically open? So, that's kind of the the concept that we're really sort of advocating for, right? Like so, follow the money. Where is the spend coming from? It's from compute. Look at where is
[12:03] the friction, right? Like and encourage your teams not to undo the openness that you've achieved by like locking uh you know, by going into choices that again tightly couple the application layer and the governance layer and the compute
[12:16] layer. That that's a recipe again to go back. Right? And then finally, like pilot, right? Like you run this for an hour, a week, a month, prove out in production, and then and then swap out. And then if something better comes along
[12:28] again, you can swap back with no changes or no disruption. Right? So, this is, you know, try I breezed through a lot of the talk since I know it's the last day, but this is essentially the gist of it.
[12:40] I would love to get any questions that you have or also talk right after the session.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.