The Full Databricks App Experience

Summary

Advancing Analytics built a production-grade agentic Lakeflow generation app entirely on the Databricks Data and AI platform, using a LangGraph loop with foundational models to profile Unity Catalog tables, design pipelines, validate transformations, and generate deployable Lakeflow artifacts from Silver to Gold.
Lakebase serves as the OLTP backend for application state, specifications, and user preferences, providing the millisecond-latency read performance that interactive apps require — something Unity Catalog's Delta tables are not designed to deliver.
The talk argues that engineers building internal tools for themselves — not just dashboards for business users — produce some of the most valuable Databricks Apps, and that Genie Code removes the traditional app maintenance objection by making iterative updates low-cost.

The Full Databricks App Experience

What does a real Databricks App look like in production? In this video, we walk through an internal tool we use every day: an agentic Lakeflow generation system built entirely on the Databricks platform. Starting from raw Unity Catalog tables, the app uses foundational models orchestrated in a LangGraph loop to profile data, design pipelines, validate transformations and generate fully deployable Lakeflow artefacts from Silver through to Gold. We’ll show how application state, specifications, and user preferences are managed in Lakebase, enabling a seamless, iterative build experience inside Databricks. Forget the quick toy POC Apps you've been seeing. This is a practical blueprint for building intelligent, stateful data applications natively on the lakehouse.
Talk By: Simon Whiteley, CTO, Advancing Analytics ;

Chapters

FAQs

What is the agentic Lakeflow generation app and how does it work?

The app is an internal engineering tool that takes raw Unity Catalog tables and uses a LangGraph orchestration loop with Databricks foundational models to automatically profile data, design pipeline structures, validate transformations, and generate fully deployable Lakeflow artifacts from Silver through to Gold. Application state, user specifications, and preferences are persisted in Lakebase for a seamless iterative build experience.

Why is Lakebase needed instead of Delta tables as the backend for Databricks Apps?

Delta tables and Unity Catalog are optimized for analytical workloads and are not designed for the millisecond-latency reads that interactive applications require. Lakebase provides OLTP-style performance for reading application state, session data, and user preferences — the rapid read-write pattern that makes apps feel responsive rather than slow to a developer audience.

Should Databricks Apps only be built for business users?

This video argues the opposite — engineers building tools for themselves produce some of the most valuable Databricks Apps. A pipeline visualizer, governance tooling, and an agentic generation system are all high-value internal developer tools that benefit from the same Databricks App infrastructure without requiring a simplified UI for non-technical end users.

How does Genie Code address the app maintenance objection for engineers?

A common reason engineers avoid building apps is concern about the long-term cost of keeping the interface in sync as underlying data and platform capabilities evolve. This video explains that Genie Code removes this objection by enabling developers to update and regenerate app components through natural language, reducing the ongoing maintenance burden significantly.

Full transcript

[00:08] Hello. My name's Solomon. I'm I'm obviously not from around here. I want to talk about apps. I'm going to talk about Databricks apps specifically. I want to start with a question. Who has built an app for a business user?
[00:20] And then there's the more important question. Who's built an app for themselves? There's not as many people. There's some people. That's good. I'm That's kind of where I wanted to go with this. So, I'll talk a little bit about architectures, some of the components. But, honestly, I
[00:33] wrote this session 6 months ago, and a lot of those big decisions are kind of just standard now. And so, I want to talk about more why we're building apps and why we think about it and kind of how it how we're building into things.
[00:45] So, like you'll have seen the keynote. You'll have seen in there a lot of the messages is we build this kind of thing. You know, if you if you go and listen to the big feature announcements, and you saw kind of all the things about live coding an app, we're basically using it
[00:58] as a glorified dashboard. We're saying, "I'm going to build an app so I can give it to a business user with some charts, with a a little button I can press to edit some data, with some kind of interactivity." But, it's all about
[01:10] addressing a low-code user's problem. It's about saying, you know, an app an app's for someone who can't build something themselves. We're going to wrap it, make it nice, and give it to them, right? Or at least that's what a lot of the messaging's around.
[01:22] A lot of the messaging that I hear from my team is we don't need an app, we're technical. And I disagree. Um so, it's that's the the kind of problem. But, we'll talk a little bit more about apps and kind of how we think
[01:34] about building it, certainly some of the choices that you still have available. But, there's the whole architecture of what an app looks like. If you've ever done any web development, nothing's changed. And you'll have a bunch of things that we need to kind of decide.
[01:47] So, front end, you've kind of got a choice between low-code, aka Python, or like React, a bit of a JavaScript TypeScript. And so, like loads of the apps that people start with, because they're
[02:00] dealing with the nice basic I want to make a dashboard, I need a button, Streamlit so easy for that. Gradio, Flask, that kind of stuff. But you'll soon reach an edge. You'll be trying to do something a little bit different, a little bit interesting.
[02:13] Streamlit won't necessarily do that out of the box. So you have a choice going, are you building a nice plug-and-play, little bit of Python, wrapping a framework, and accept the limitations? Or are you actually building a fully-fledged, build it from scratch web
[02:26] application? In which case you can build whatever the hell you want. But it's just a little bit more complex, or at least when you first start. So you need to decide on the front end. The back end, most people have now settled. The idea is Fast API, it's
[02:40] Python. All of my data engineers love Python, they don't like Java. So Node.js, they're not as big fans of using JavaScript, but if you look at the App Kit that Databricks have put out, builds it in Node.js. So you got a decision, do you want to
[02:52] take a nice easy wrapper framework, but accept that your devs aren't really going to know what the code does? And does that matter if they're vibe coding it? Are we Are we at that point where we happily go, I don't need to look at the
[03:04] code. Cloud looks at the code, Genie code looks at the code, why need to? So there's some decisions around actually what you're doing, but it's starting to standardize. Plugging things in is getting easier and
[03:16] easier. Saying I want to plug it into a SQL warehouse, I want to plug it into an agent, model serving, Genie code. Whereas that all used to be, you need to build a custom integration, you need to wrap something in the API, you need to
[03:28] plug it all in manually. It's a enable these functions, plug into a standardized wrapper, and then you can use it. So it's all getting to be a lot easier than it used to be. And we've seen so many people make the
[03:40] the pro- the the mistake down at the bottom. My data's in Unity Catalog, right? I like Delta. I've spent years convincing people to move into a lake, use Delta or Iceberg to store all data. And now we're saying, "Actually,
[03:53] you should use lakehouse for that. Why you shouldn't use the thing that we've been convincing people to use." And that comes down to a latency problem. It comes down to a if I want a user to click a button and update a record and that to happen like
[04:05] milliseconds, Unity Catalog and Delta's not milliseconds. Because it's a lakehouse system. You've got a checkpoint to write, you need to compress the data, write it down, add it to the parquet, update the checkpoint to say you finished writing it. There's
[04:17] just some parallel hops that has to happen. Lakehouse, you don't have that. So, it's a decision going, "Am I doing a reporting app?" In which case I'm doing a big chunky analytical query. Yeah, that's going to come from Unity Catalog. That is a SQL query. That's the stuff
[04:30] that we're used to. Am I trying to build a little OLTP app? Am I trying to update data and have it just happen immediately? That's a lakehouse decision. So, you've got a slightly different data architecture than you might be used to.
[04:43] And a lot of it again has come down to this argument of going, "Well, who's the user? These guys? My exact sponsors, decision makers, someone in operations, someone planning something." A lot of that tends to be that nice combination. Get to the
[04:56] end of my lakehouse, get all my data ready, build a little app layer to help them actually consume that data. And occasionally, we might go fancy and we'll do something for a data governance person cuz data governance is cool
[05:08] again. Now we can call it an ontology. Hello. Yeah, there's interesting things about that. The thing that we really don't see enough is building an app for a data engineer. Building an app for a platform
[05:20] administrator. Building an app for me to make my work easier. And I deserve that. We all deserve to make our lives easier. So, I want to show you some things that is essentially what we've been building. Cuz this is what I keep hearing from my
[05:33] data engineers. "I'm too technical. I like scripts. I want to live in the code. Can't we just build a CLI? We don't need a UI for anything, right?" That's what they told me. So, not going to go too much in detail to this, but building out lakehouse
[05:46] pipelines, wrapping it in a an agentic harness, so it does all the decision making, it spits out some pipelines, all fun stuff. You don't need any UI for that, right? That's just a bit of code. You can open it in a notebook, you can
[05:58] hit go. So, we built that. It's great. And kind of I had fairly straightforward architecture. Had a notebook with a bunch of cells and some parameters. Bit of Python, bit of LangChain behind the scenes, normal Spark notebook stuff.
[06:12] Used some Spark, used some Databricks LLMs. Stored all the files in the workspace. Cuz that is what I knew. I'm an engineer, I'm like, I want to have a nice file layout. So, getting my guys to use it,
[06:24] they open this up. They're like, "Yeah, this is nice. I'm happy here. This is my UI. I've got a few cells, bunch of code, some parameters. I can go and have a look at the files." And that journey of me going, "Surely, surely this needs a UI.
[06:37] Surely this is just going to be nicer if we put something on it." And you get the traditional objections. You get the objections that are grounded in the history of if you had to build a web application, everything slows down.
[06:49] And that's kind of an a conception that we've all got in the back of our heads of going, "Web application is slow. UI slows things down. You can't be as agile once you've got a UI in place." So, I heard, "Yeah, but we're technical
[07:01] users. It'll slow us down. It's harder to use than a CLI." Madness. It's difficult to maintain. What the moment you've actually encapsulated something in a UI, if you change the schema, uh you've then got a downstream
[07:13] dependency overhead, you have to change the UI, and you end up in this world of maintenance and overheads. Well, at least that's the objections that I was having. But as soon as you do that, certainly for my engineers, well, they could visualize the pipeline that they
[07:25] actually developed. They didn't have to create it in their brains. You can do a quick, you know, GitHub-style diff comparison. We can have it as a conversational LLM, cuz everyone likes an LLM these days. You can have validation.
[07:37] Like previously, if they wanted to make a quick tweak, they They open a JSON file, manually edit it, run it and see if it works. Whereas I can just put it inside a form and have some validation going, "No, that doesn't work."
[07:49] All the stuff that we built a UI for, right? But they didn't want to cuz this conception of it's slow. Because it's overhead, cuz it takes time to maintain. So, throwing something quick together. This is a Databricks app that I quickly dive over to.
[08:03] So, I've got my apps experience. I've got a load of apps that are deployed. I can dive into an existing app. And yes, I can make it pretty and user-friendly and give people a load of stuff. That is just That's pure psychology and manipulation.
[08:16] But saying what as an engineer, I'm going to dive into a pipeline that's being developed. I want to go and have a look at what's actually being done. Let's Is Lakehouse going to work with my internet? We will see. Ah, there we go.
[08:29] Yes, we're doing things like just a quick pipeline visualizer, so I can dig into something and say, "Does this actually work?" Dig into some details. That's a nice way to kind of for someone to get an idea of what they've built out.
[08:41] Rather than reading raw JSON, you can dive into something and just show it a little bit nicer. We can see what details have been done. Fine. A little code viewer. You can go and see what code's been done. Again,
[08:53] like the difference between I'm living in code, I'm opening a file, I'm having to manually look at output. I'm a developer, so I can kind of figure it out. So, they're going, "Well, this is just a lot easier." And so, all the developers have now
[09:05] switched and they all just use this because it makes life easier. The whole point being they hadn't even considered building an app for themselves. They had this whole conception and building that took couple of days.
[09:17] Because we can now all these tools that we've got coming out, all of whether you build it in code, whether you use App Kit, whether you use the new thing announced this morning of the whole Genie app Vibe coding experience,
[09:29] all of this speeds things up. To build a UI and say, "Oh, quick refinement, quick tweak." That's a couple of prompts if you put the right guardrails and the right processes in the right working place. So, that's my main thing is going to
[09:41] just build apps. And if you're building an app for yourself, you're going to get better at building apps for other people. So, it's a nice starting point. It's like the whole idea of dog fooding, right? If you're trying to build build a good tool, if you want your user to have a good
[09:53] experience, use the tool yourself, and then you're going to know a lot more about how to make that a good experience. So, lessons from going on that journey and having built a whole pile of apps.
[10:05] I mean, certainly, as we mentioned earlier, the storage mechanism, like storing things in JSON files didn't make sense. Storing it in Unity Catalog, the default one every data engineer goes to is just far too slow.
[10:17] Storing it in Lake Base, fast, cheap, and easy. Like, who's using Lake Base currently? Oh, that's not a lot of hands in the room. Like, honestly, I mean, if you've used Postgres before, it's really,
[10:29] really straightforward. But, it is as quick and easy. I made a quick demo of it for a YouTube video, and it's the most disappointing demo you've ever seen. Cuz it takes about 5 seconds to go, "New database. Uh there's a database there."
[10:41] That's That's it. It should be underwhelming cuz it's just a database that comes up instantly, scales, branches, all of that good stuff. And the development experience of being able to branch and go, "Oh, I need to do
[10:54] a feature branch. I need to try changing this." Create a branch, takes less than a second, and then you've got a separate branch you can do some testing on, and then you trash it. So, the idea of like a database is ephemeral, the same as just creating a
[11:06] feature branch in Git, really easy to work with, and just builds into the agility of how you actually build these apps out. A lot of it is just getting used to some of these ways of working. So, every app we tend to build, you'll
[11:19] end up having some kind of Lake Base usage. That could be for some operational store, some operational data they're working with that they need to edit. It could be the uh the agentic state where you've got conversational LLM and it needs to somewhere as a
[11:31] scratch pad. Going to go into lake base using like that way. If you got some open telemetry data, you want to store that somewhere. There's loads of these bits and pieces. Almost every app is almost like joining together. There's the app surface,
[11:43] there's a lake base. They come together fairly fairly regularly. Been a while since we built an app without lake base behind the scenes. The front end as we said, Streamlit's okay. You can do a lot more funky stuff
[11:55] in React, but you're embracing the joy that is vibe coding some stuff. And economy means there's an updated architecture. Which is usually for us it's a React app. Now, the first React app we built took a
[12:07] little while. Took a little while to actually get the process of the patterns and things in place. But the whole point being, once you built one, just make it a template. Pull back the individual customization you've done, and then you need a new app. Copy the template, adjust it for
[12:20] the new things you need. Copy the template, adjust it. And that's what they were talking about this morning, that kind of the app dev kit is essentially that. And the whole vibe coding experience of coming into app spaces just gives you so
[12:32] much more functionality there. So, currently if you're doing it, you kind of build your own. You kind of build your own default app and you just start from that as a starting point each time. So, you don't have to reinvent the wheel. Going forward, it's going to be easier
[12:44] than that. It's going to be opening new app page, type in a few prompts about what you want, and it'll start to customize it and tweak it based on that pre-built, pre-connected idea. And again, a lot of the marketing, all of the messaging is that's to enable
[12:57] your business users. But I have it in your mind, just so many things going, "Ah, there's an app for that." It's almost like back in the day every developer would end up just having piles of PowerShell scripts, piles of Python scripts that they keep as a
[13:09] utility pockets for what they do. It's the same for apps, especially now that they'll scale down to zero, so you're not paying for it. Currently, if you have the app turned on, it's going to cost you money and that detracts. Um deployment.
[13:22] I mean, deployment is uh used to be awkward. You You to use the Databricks CLI. There's the whole sync script, there's a deploy script, takes a little while. But apps now, you can just deploy straight from Git.
[13:34] So essentially, you create a new app, you point it at the Git repo, you tell it which branch. Whenever you hit deploy, it goes, fetches the latest copy from that branch at a particular commit, pulls onto the app server, spins up.
[13:46] It's getting easier and easier and easier to do all of this stuff. Just try it is the main thing. And that's yeah, the the cut point is there. Everything's basically the same.
[13:59] All the different app versions that we come out with is some combination of a React front end, Fast API for our back end, Lake base sitting behind, bunch of connectors, be it SQL GD, models, pipelines, whatever that is.
[14:11] Maybe a sync back to UC. Just again, makes it easier. Oh, for whatever the connectors you're doing, there's a whole bunch of pre-canned. So if you find yourself building an app
[14:23] and you're manually coding in the Unity Catalog API, you're manually coding in secret connectors, you're not doing it right. There's a lot of these things that are pre-canned that will manage the security, the connectivity, a kind of a default way of working,
[14:36] which if you're using the app kit, these things are baked in already. So don't feel like you should start with a completely blank Git repo and build everything at an empire from scratch. There's all of this stuff pre-canned, pre-built. If they then change the protocols,
[14:49] it's going to change anyway cuz it's pre-canned into what you've built. So it's just a start with the baseline. Easy. And then yeah, the way we work now, Pantheon is our big uh lake flow generator. We need a new app. Uh we basically take
[15:02] off the uh the Pantheon bits, we add in the next bit. We need a new app, we take off of the uh functionality, we add in the next bit. They all share like 70% of the code in common, but for lots and lots of different kind of tasks that we're
[15:14] doing. I mean, if I have time, I can show you quickly. So that same that same baseline, the same baseline that we had in here. Uh I was talking to my team and they were trying to get over some problems in
[15:26] uh in Unity Catalog. So, as I work, hey. What problems are we having? Well, we don't know what's in Unity Catalog. Like we we can go and look at a particular table. And we can see the lineage for a particular table.
[15:38] But like we don't know everything that's in there. Uh and then what's the hottest term? Everyone loves an ontology. Built an ontology. But it took a couple of days. Just getting this together starting from that template.
[15:51] So, I built a little pipeline. It scrapes all of your data assets. You can go and see it. Most fun part is going in and saying, "I want to see what concepts I've got. I want to see what what the overlap of different ontological layers
[16:03] are. And you can build some funky things." But that you couldn't build that in Streamlit. So, to get some of the the nicer the more funky the more reactive components you need React.
[16:15] Now, Streamlit you can have custom components. But those custom components are written in React. So, if you've gone down that path, you might as well choose React for everything. But again, this they are the same app. But just with some of the pages taken
[16:28] out and then a different skin put over the top to do a different bit of functionality. You don't need to rebuild all of the SQL connector, the inference model, all of that stuff. We had the same problem where someone came to us and said, "Have you got a a master data
[16:41] management app?" Building a little master data management app to wrap manage reference data. I want the user to go in, update a record, do some fuzzy matching to it, publish that back to Unity Catalog. It's just another app.
[16:53] Plugs into the same stuff. Again, just just finding these different reasons to let people come in and have this UI experience. Trying to do that level of give someone a SQL like a an Excel style
[17:06] work around in a grid with an approval workflow. Those kind of things you could try and do in code, but it's a little bit awkward. I'm just going, "Well, actually again,
[17:18] takes a couple of days to get an app together once you put in the work to get that baseline. So I guess that's my It's almost my counter marketing to a lot of the marketing that Databricks are doing of going, "All this stuff is great. Apps are great for business users.
[17:31] It's great for getting this data out in front of people who wouldn't do it normally themselves. But actually there's so many more things we can just build an app for. And then obviously announcements this morning we saw loads of things going on.
[17:43] So one of the biggest complaints with Databricks apps is once you deploy it, once it's turned on it's just turned on. There is no scaling. There is no If you turn it off and turn it back on, it takes like 3, 4, 5 minutes to turn back on.
[17:55] That's not really good enough. Which means that people just left it turned on. Or we ended up back what we used to do in the old days, have like PowerShell scripts, Python scripts set to turn it off at a certain day, turn it on on a certain day. It's It's like going back
[18:08] to the start of Azure. Auto scale fixes all of that. Again, like a lot of the compute we see is going to turn up, turn down. Scale to zero is coming. So we'll see all of that baked in. And that just makes it
[18:20] if you built an app and you turn it on and it you only charge us you for the 10 minutes a day that you use it, there's no cost to having it there elsewhere, there's no reason not to just start building these apps and maintaining it. There is a support overhead. Someone has to look at the code.
[18:33] But yeah, it makes it so much easier. App spaces are huge. We saw there again this morning that idea of going, "I want to create an admin environment for people to go into to build the apps. I want I want you guys to be able to build an app, but you're only allowed to
[18:46] use SQL. I don't want you using all the LLMs cuz you're going to charge me lots of money. But actually giving people that workspace and giving people that control to say what they're allowed to do just making it a bit more mature. That and like more enterprise readiness of
[18:58] like how you delegate responsibility for what kind of app you're allowed to build. All of those functionalities that are coming out. App kit, that's the one that we developed that's basically a load of cloud skills that you can then use to build things, being built into
[19:10] this Genie app builder. So more and more you'll just open up a page, kind of go through that same if you're vibe coding if you're using like AI augmented development currently, like the good pattern to get into is kind of
[19:23] like the plan, implement, review, test kind of thing. Get an architectural plan written up front. That'll then make a plan, review the plan, actually look at the plan, and go through that process. And we're starting to see some of those things baked into that Genie app builder
[19:36] to kind of get people to describe what you want, describe the experience, actually vibe it on the fly a bit like Repl.it, Lovable, those things. We're going to see more maturity coming in. Currently, it's more like a pro dev experience and you'll do it in GitHub
[19:48] and VS Code. You'll use your editor of choice. But that is going to start getting a way, way easier. Sound good? Cool. I mean, that is all I want to talk to you about. Essentially, a lot of those things hopefully some of those things
[20:00] sound obvious. Like Base React, those things 6 months ago, they were decisions that people were facing going, "What Where do we even start?" More and more, you don't have those decisions. It's just a way of working.
[20:12] Get started, start vibing it, and it will work. Cool. I think I need to clear off the stage so we can do Q&A outside. Okay. We'll do Q&A over there. So, thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.