Skip to main content

Building Agentic Apps on the Databricks Platform: Architecture, Considerations, and Key Patterns

Summary

  • Agentic applications differ from traditional CRUD apps because they reason over data, take real-world actions such as filing orders or support tickets, interact with other agents, and maintain persistent state—requiring fundamentally different infrastructure.
  • Four core requirements for agentic infrastructure are consistency, governance, idempotency, and observability, which map directly to Lakebase, Unity Catalog, MLflow, and Lakewatch on the Databricks Data and AI platform.
  • A key pitfall is storing agent state in-process or mocking the database during development; production-grade agentic apps require a transactionally consistent operational store and safe rollback via database branching.

Building Agentic Apps on the Databricks Platform: Architecture, Considerations, and Key Patterns

Watch: Building Agentic Apps on the Databricks Platform: Architecture, Considerations, and Key Patterns
Agentic applications are fundamentally different from traditional software: they don't just retrieve data, they reason over it, take actions, and update state in response to the world around them. That makes the database layer a critical design decision, not an afterthought. When agents need to read fresh operational data, write back results, and coordinate across concurrent tasks, the infrastructure underneath determines what's actually possible. this video covers what makes agentic apps so powerful, what can go wrong without the right foundation, and how the Databricks Platform addresses the core requirements. You'll learn how Lakebase provides the low-latency, transactionally consistent operational store that agents need to act on live data, how Unity Catalog enforces governance across agent actions and the data they touch, and how Databricks Apps simplifies deployment and scaling. We'll also walk through a reference architecture that shows how these pieces fit together, from agent orchestration and tool use to state management, observability, and safe rollback using database branching.
Talk By: Yassine Essawabi, Senior Resident Solutions Architect, Databricks ;

Chapters

FAQs

What makes agentic applications different from traditional apps?

Traditional CRUD applications simply read and write data with no persistent state between requests. Agentic applications reason about data, interact with other agents and external systems, and take real-world actions—requiring infrastructure that supports consistency, governance, idempotency, and observability.

What is the reference architecture for agentic apps on Databricks?

The architecture includes Lakebase for low-latency transactional state management, Unity Catalog for governance across agent actions and data, MLflow for observability, and Lakewatch for AI governance. Databricks Apps handles deployment and scaling of the application layer.

Why do agentic apps need Lakebase instead of an in-process store?

Agents that coordinate across concurrent tasks or need to recover from failures require ACID transactions and idempotent writes that an in-process store cannot provide. Lakebase offers the transactional consistency and database branching needed for safe rollback and concurrent agent coordination.

What are the key pitfalls when building agentic applications on Databricks?

The two highlighted pitfalls are storing agent state in-process rather than in a persistent, transactionally consistent store, and mocking the database during development in ways that mask production failure modes. Using Lakebase from the start, with Unity Catalog governance and MLflow observability, helps teams avoid these issues.

Full transcript

[00:08] Uh yeah. Hi everyone. My name is Yassine. I am staff forward and deployed engineer at Databricks. I've been at Databricks for 6 years now. I work across Singapore, Paris, and now I'm based in Spain. Uh yeah, super excited
[00:22] to be presenting at Data and AI Summit again and of course super grateful for everyone who decided to attend the this session. Uh so, the purpose of this session of this
[00:36] workshop is to leave you with the answer to one key question. If you are building an application today, if you are building an agentic application today, something that does not just read data,
[00:50] read write data, but something that reasons about that this data, that takes actions, that interacts with other agents, other systems in your in your platform, what is the right architecture
[01:03] underneath to build this type of use cases, right? So, uh we will try to follow uh Sorry, one second.
[01:20] We will try to follow a total structure starting with the agentic shift to explain the difference between the traditional applications that we have today and the new uh trend of building
[01:32] agentic applications on Databricks. And we in this section we will cover the different requirements and the different challenges that we see with most of our customers. And then we will try to map
[01:46] the Databricks platform features to these requirements and to these challenges. And after this, we will go through a reference architecture to see how we can build this on Databricks and using
[01:59] Databricks features. And we will cover the key patterns and pitfalls that we see with most of our customers when we are working with these customers to build agentic
[02:11] applications on Databricks. And of course we will finish the the session with key takeaways that you need to keep in mind when designing your agentic applications.
[02:23] At the end of the session I will leave you with my LinkedIn profile. You can reach out if you want the slides. I can share with you the the slide deck.
[02:36] Okay. So a traditional crude applications that you are familiar with is an application that reads data and writes data and that's it. There is no state. There is no back end where we keep track of what's
[02:49] happening in that application. And moving to an agentic application in or an agentic systems, we don't just read the data and write the data. The agent will
[03:01] reason about this data. It will interact with other agents. It will take different actions. For example, file a support ticket, make a request, file an order for the customer and so
[03:13] on. We will see few examples on this. And the if I go to the next slide to see a map like a one-to-one or a side-by-side comparison between the
[03:26] two, you see that the traditional crude application is like stateless between the between the calls and the data the database is the back end store of your applications. So we read data, write the
[03:39] data, and the process is done. On the right side, we have our agentic system or agentic app where agents they will perceive the world. They will reason about the data that they are interacting with. And then they will
[03:52] take some decision or they will act on these data and on these requests. Then write the results and of course interact with different systems, either MCP servers or other agents or other
[04:06] components of your platform. And every iteration of this agentic app, we will mutate the state, we will change the state. So, we need to keep in mind that we need to pick the right system or
[04:19] the right product for the back-end state of our application. So, what's other key requirements before going and talking about the Databricks components? What are the key
[04:32] requirements for an agentic application? First thing is we need a fresh operational state. The agent does not should not use data from yesterday,
[04:44] should not use data snapshot data, should not use old or outdated data because they will make wrong and outdated results. It needs fresh recent data to
[04:56] use when making decisions. We also need transactional rights because when working with agents, they will as I mentioned they will make a lot of they will take a lot of decisions, they will
[05:08] make a lot of changes to your in your systems. And we need to make sure that when the agents make any action, we need the transaction to be atomic, to be fully committed or not
[05:21] committed at all, right? And the next one is concurrency coordination. In our agentic systems, we will have different agents working at the same time and we don't want these
[05:33] systems to uh overlap or to run into each others work and changes in the database or on our system. And the last one is safe rollback.
[05:46] Uh I think most of us familiar or works before with different agents and the agent might make different uh or wrong decisions or make
[05:58] mistakes in uh as part of the uh their the the actions taken by the agents and we need to have a a way to implement a safe rollback uh and go back to the previous state of
[06:13] our system or our application. So, if these are the key uh requirements for the agentic systems, what actually break uh your agentic system today compared to
[06:26] or what breaks the back end of your crude application today if you want to use it for agentic systems? First one is stale reads as I mentioned. Uh this is very important because
[06:39] uh if we have like old outdated data, the agent does not just make wrong decisions, it confidently make wrong decisions using that data. And if we don't have atomic or acid
[06:52] supports, uh no transactions, then the if the transaction is not fully committed, we don't have uh a guarantee or a consistent state of
[07:05] what the agent is doing or the different agents are doing on our application. The next one is uh vector using vectors store as a state. This is one of the common pitfalls that I see with
[07:17] customers and with uh this kind of decisions, like we don't have any idempotency on the system, we don't have the right performance or the right low latency
[07:30] requirement for our systems, and the agents will might might take a lot of time uh to get the required data or get the required context to act on this uh context. And the last one that I
[07:43] mentioned earlier uh around rollback, this is also a key challenge that uh we see in our agentic systems where whenever the agent makes a mistake or makes a wrong decision, there's no
[07:55] support for uh rollback. So, discussing all these requirements and discussing all these challenges, what I want you to keep in mind is that when building an agentic system,
[08:08] the issue or the challenge is not the LLM, it's not the model that you are using, but it's the architecture underneath where you need to cover different requirements to make sure that the architecture itself is performant
[08:21] and resilient, and then you can easily switch between models and compare the performance of the model itself.
[08:33] So, here we can go a little bit deeper to list all the different challenges that we see, uh but uh if I go briefly on these, like we have stateful context that we mentioned earlier, no safe rollback supports, uh the performance issues when
[08:46] using like vector store, the split between OLTP and OLAP, and you end up with data silos, like you read from a system and you write to another system, and it becomes an overhead to manage,
[08:59] uh ungoverned actions uh if you're not leveraging Unity Catalog to make sure that the agents, whatever the agent is doing, is govern fully governed by uh Unity Catalog. We will come back to this
[09:12] uh in our next slide. And then we have, of course, the runway costs, and I see this with a lot of customers where they're working on POCs, prototypes using different uh agents or different LLMs, and at the
[09:26] end of the month or at the end of the quarter, you might end up with uh cost spike or uh a huge bill that you need to address.
[09:39] So, we discussed that the actual problem, we discussed the requirements, and we discussed the challenges with the traditional crude applications. Uh what I want us to do now is try to break these down into different pillars,
[09:51] and for each pillar, discuss what's the key uh like uh product or offering from Databricks that maps to these requirements when designing your architecture.
[10:03] So, what we can do is we can break it down into different primitives. Let's try again. Okay. So, in total, we should have seven to eight primitives uh starting with Lakehouse as
[10:18] operational store. So, you have a managed Postgres uh performance, you have asset uh support, uh atomic transactions, you have branching support in order to quickly prototype and work
[10:31] on different uh approaches in your agentic system. Uh we will talk later about the uh Lakehouse, the new Lakehouse search that will be announced later this week, that allows you to retrieve data, uh allows
[10:44] the agent to retrieve data with low latency uh support. Uh next we have Unity Catalog as the governance layer or the governance plane. Whatever you're doing in uh in your
[10:57] architecture or in your system needs to be fully governed by the exact same uh model. So, you define your ACLs either RBA or ABA, uh you define your
[11:09] permissions on not just the tables, not just the catalogs, uh it can be the models, the endpoints, the uh the Unity Catalog functions, MCP servers, the skills, everything that you build as
[11:22] part of that architecture should be fully governed by by the exact same governance model. So, that when you start onboarding new use cases, they will adopt the same exact uh governance model.
[11:34] And next we have Unity AI Gateway and Databricks apps. For me, these are my favorite like two of the most exciting features that
[11:46] were announced this during summit. Unity AI Gateway allows you to govern the spend AI spend across different business units, different teams,
[11:58] different use cases by defining budget policies for these for these teams. And also you have the ability to define guardrails.
[12:10] When we talk about responsible AI, you want the to make sure that the model of the agent system is not biased, is not harmful. So you can do this as part of the AI Gateway as well.
[12:22] And next we have Databricks apps and the new announced app space that allows you to define like a container with certain configuration, for example, the list of
[12:35] tables that can be used, the list of catalogs, the authentication system or option that can be used. And then you grant access or access to certain users to use this app
[12:48] space. So you don't define the configuration at the app anymore. You define it once and allow the users to quickly prototype and build their applications.
[13:06] Okay. And next we have Agent Bricks and Omnigen that also was announced during summit that allows you or gives you the features required to for authoring and orchestration. So you can leverage different LLMs, you can leverage different frameworks
[13:20] in order to build and design your agentic system. You don't need to stick to a specific framework. It just gives you like the structure to design your AI agents. And two of the new announced
[13:35] uh features that I will discuss later in the in the session, we have agent memory and agent sandbox. Agent memory allows you to be uh to have uh governed and uh
[13:48] uh scope down session and memory that can be shared across users, and I can do an import later on and use the same session or the same conversation from another uh
[14:00] system or another uh app and avoid restart starting from scratch every time I'm working on my on my use case. And agent sandbox allows you to quickly work
[14:12] on your prototypes and on your use cases in a safe uh uh and governed uh space. For example, uh you define or you don't allow uh internet access, you you define
[14:24] a specific token to be used, and it's like a uh down scoped uh uh system or environment that can be used by user to uh work on their prototypes. And of course, MLflow 3 for everything
[14:37] around observability and evaluation. Okay, so starting with Go back to the previous slide.
[14:55] Okay, so starting with Lake base as the managed Postgres uh for agents for building your agents on Databricks. First, we have this uh the it's safe by construction, so we have
[15:07] asset transaction supports or asset rights. Uh we have uh idempotency uh keys, so retry the retries from the agents become uh a silent no operations. So, and you need
[15:20] can make sure that the agent if it's retrying certain action or certain operation, it does not break uh the state. Uh we have the MVCC isolation. So, if you I
[15:32] have multiple agents working at the same time, there is full isolation between these agents and they will not uh run into each others uh actions or uh changes. Uh we have of course we mentioned
[15:45] branching for instant rollback for any risky runs or for again for prototyping in a separate environment without duplicating the whole data into another uh environment.
[15:58] Uh next we have built for the hot path or the low latency performance uh and we have the possibilities to sync the tables from Unity Catalog to Lakehouse or from Lakehouse back to
[16:11] Unity Catalog for analytics uh use cases. So, the agentic systems uh or the agentic system can use Lakehouse as the operational store and then I can write back the data or the output from the
[16:25] agents back to Unity Catalog for analytics uh use cases. Uh why are they into the platform? This makes uh makes it easier to uh interact
[16:38] with the rest of the features that we will discuss. Uh we mentioned Unity Catalog, we mentioned the AI Gateway. Everything is works seamlessly together. So, if I define the governance with Unity Catalog, I can have the exact same
[16:51] governance model through uh for uh for Lakehouse as well.
[17:03] Okay, and the three new announcements from this week. First, we have the Lakehouse search. So, this will be uh hybrid full-text search that can be used by your agentic system to retrieve the data, retrieve the context from uh from
[17:17] Lakebase. Uh the fully managed cross-region and cross uh cloud disaster recovery we don't need anymore any DIY uh uh setup in order to
[17:31] fail back or like uh to back up our Lakebase uh store into another region or into another uh zone. And we also have the real-time analytics uh that
[17:44] gives you the low latency performance required for the all the requirements uh we mentioned and the challenges we uh mentioned earlier.
[17:58] Then we have Unity Catalog. So, Unity Catalog in summary, we have first one plane for everything. Any assets you have as part of your agentic systems, either tables, models, use uh
[18:10] uh UDFs, functions, agents, everything is governed using the exact same governance model with Unity Catalog. You have uh RBA for role-based access control on the table and catalog uh catalogs of schema
[18:24] levels. You have ABAC if you want to use govern tag and make sure that uh the agent only access the data that it has permissions to access. Uh next we have uh the audit and lineage
[18:37] and discovery. So, when I call, for example, I'm using a chatbot, I build an agentic system and it's like a chatbot application. When a a user uses the uh application,
[18:51] any permissions or any action made by the uh the user and by the agent will use the identity of the user, and that identity will flow through Unity Catalog to make sure that the user has the right access
[19:05] needed or not. And also any action that the agent does will be tracked automatically on uh system tables and table lineage to to make sure what the agent
[19:17] has done as part of that request for the user.
[19:29] Okay, so for Unity Catalog a lot of new features have been announced during Summit. Some of them will be announced later on public posts. But just to go briefly on
[19:42] these new features, we can take the hard spend caps. As I mentioned earlier, you can define budget by teams or by business units to make sure that you control the the spend. The contextual budget policies,
[20:02] the integration with Lake Watch so that you can have an AI security monitoring and govern or monitor and audit what the agent is doing as part of your
[20:14] agenting system.
[20:26] Okay, so next we have Databricks apps that allows you to ship not just the the the infrastructure underneath, but the whole agentic application that you are building. And here we have different features. I will
[20:39] not go through all the the features, but Databricks apps give you the possibility to use different frameworks to build your apps. We talked about the new vibe coding capabilities with Databricks apps
[20:53] in order to quickly build your use case or your app using Databricks. We have like full support with Databricks asset bundles in order to package the code for your application
[21:07] and deploy it in production and in other environments. Authentication supports, we have different authentication systems, either you are using a service principle
[21:19] that's that is created automatically when you create an app, or you can use on behalf of so that when the agent does any action or takes any action in
[21:32] the process, it will assume the identity of the user that's making the call. Okay. So, on the deployment side, I briefly talked
[21:45] about these two new capabilities before. We have agent memory managed per user and cross-session memory, and also agent sandbox that
[21:57] allows you to deploy and test your systems in a secure and isolated environment.
[22:41] Okay. So, next we have ML Flow as the last components or the last pillar of the building your agentic apps on Databricks that allows you to have full capabilities to author, to observe, and to improve your agentic systems. But, to
[22:56] keep it simple, we have three main components. You have the author once, deploy anywhere that allows you to use any type of framework to build your agentic system and your
[23:08] architecture. We have the monitoring and the trustability on MLflow so that you can see what the agent has done what before taking any action and
[23:22] the whole reasoning behind any action taken by the by the agents. And the last one is the closing the quality loop and this is where I mentioned earlier the
[23:34] building responsible AI or building responsible agents on Databricks. You can define your guardrails, we have things like correctness, safety, guardness of the agents and also
[23:47] use cases for example avoiding biased agents when taking actions. I delivered another talk on responsible AI that you can also check and gives you
[24:00] like a deep dive on how to build responsible AI or responsible agents on on Databricks.
[24:14] Okay, so now I think we can try to put all these pieces together to see how the architecture end-to-end will look on on Databricks. So at the top you can see the user
[24:27] that will try to access the application that we built for our multi-agent system and it's using its own identity and the token generated for for the user the auth token. And then that request will go
[24:41] through Databricks apps and for Databricks apps either we're using app service where we using micro apps that was also announced yesterday. And in the middle layer this is where we
[24:54] have agent bricks or omnigents as the orchestration and authoring system that allows you to build and design your architecture, your agentic AI architecture. And on the tools layer, we have
[25:08] different options. So, you either have your calling different MCP servers, you're calling some skills, or you're using UC functions, or the newly newly announced Lakehouse search to
[25:21] retrieve the required data for for the agents. And on the data layer, this is where you have Lakehouse as the backing store of your application, both for the actual
[25:33] data, but also for the memory of that agentic system. And in the last layer, as you can see, we have MLflow 3 and we have Unity Catalog. And
[25:47] here, the idea is that we want to, as I mentioned, govern everything, all components and all pillars of our agentic application using the same governance model.
[26:04] Okay. So, if we break down that architecture and that flow into three main components, first one, we have the read path, the write-back path, and then the safe rollback path. So, for the read
[26:17] the read path, we have the user who opens the application using OAuth token to access the that application. And next, we have the app will call the agent endpoints where we deployed our
[26:29] agents. The agent will invoke different tools, either Lakehouse search, UC Unity Catalog functions, or other tools, MCP servers,
[26:41] as we mentioned before. And then, it will return the grounded answer, and all everything that the agent did will be tracked and traced on MLflow. And as before returning the answer, this is
[26:54] where you can have your guardrails or your like another agent that will make sure that the answer is safe and it's not biased and so on. The The next step or the next
[27:07] Yeah, the next step is the write back path. So, the agent will decide what action needs to be done. For example, updating an order, filing a ticket, sending an email, or another action
[27:20] as part of your agent system. And it will decide which function is required or which skill or which MCP server or databases are required to fulfill that action.
[27:32] And then it writes back the like it will have a dedicated function to write back the result into lakehouse and you have the asset supports asset transaction support that we mentioned earlier.
[27:44] And once we have the data written, we can also uh mirror the outputs from the agents, the lakehouse outputs from the agents back through Unity Catalog tables for
[27:57] analytics analytical use cases if needed. And the last one is mainly around safe rollback, how we can safely and securely test
[28:09] our use cases, but also how to be able to safely go back to the previous state if the agent made a mistake or made unauthorized
[28:22] action. Okay, so I think first uh elements that we can cover as part of best practices is where should we store
[28:37] and which tool should we use for storing the state, the agentic system state. Uh on lakehouse and agent memory, you can use either Lakebase or agent memory for the
[28:52] conversation summary, for the task ledger, keeping track of all the tasks that uh agent are doing is doing. And you should avoid using vector store
[29:05] as a state. Uh you should avoid, for example, in-process Python in order to save the state of the agentic system. Uh avoid system prompts like
[29:18] uh very long system prompts when designing your system. And of course, the last one is to avoid object storage as a store for your for your state.
[29:34] Okay, so we also talked about authentication for the user to log in and use the app. And in summary, we have two possible possible option. First is the application service principal that is
[29:47] created when we create a new app. And the second one is on behalf of user. So, the first one allows you to give the users the permissions to use
[29:59] that service principal. So, any request will assume the identity of the service principal. And the agent also would use the identity of the service principal. But, if you have
[30:12] like, for example, you have ABAC permissions or role-level permissions and access permissions on Unity Catalog, and you want to make sure that users only access the data that they're
[30:25] allowed to use, then you will define this governance model on Unity Catalog. And then you can use on behalf of user. So, when I try to when I request something from the
[30:37] application, it will use my identity, not the identity of the service principal. and then if an agent tries to read some data, Unity Catalog will check if the user has the permissions to read
[30:50] that data or not. It will not use the service principle. And it's also important because when you do the audits and observability and monitoring of your system, you can
[31:02] see what each user is doing as part of the agentic system and what kind of data they're trying to access.
[31:19] Okay, so the last section we we should cover is like a summary of the key patterns and the key pitfalls that I see with a lot of customers today. The patterns, of course, you need to
[31:31] avoid using personal access access token and use the short-lived OAuth tokens. Everything should be governed using Unity Catalog.
[31:49] You can you should use up a spaces that were announced this week in order to have like a secure and governed way of building applications at scale instead of doing it at the application application level.
[32:02] And the last one is that when designing and building agentic system, you should avoid mocking you should avoid mocking the the the database. You should mock the LLM and you can create a branch of
[32:16] the database using Lake Base to avoid duplication the data and you can just use that branch for your prototyping. And the key pitfalls is forgotten SP or
[32:29] service principal grants. I see a lot of customers that use the default service principle of the application, but with time, you forget that that service principle was created. If you don't have
[32:42] governed and you don't have audibility and auditing and monitoring, you might forget that the service principle is being used by some users. The second one is
[32:56] the pitfall that I see the most with customers using vector store as a state. And
[33:08] Yeah, so the last one is around observability. I think the key pitfalls would be using vector store as a as a back end, the lack of AI spend and AI budgets when you're building your application. This is super
[33:22] important, but now with AI gateway, you're able to define these these budgets. And also another one is the lack of traceability or monitoring when building the system. And these
[33:36] three items or three topics is something that you should think of before building the use case, not after building the use case. So you should build the foundation first for these use cases, and then you can start onboarding users
[33:50] to work on these use cases. Okay, so just go right to the key takeaways.
[34:07] To give some time for a Q&A. So if we try to summarize the full session in five key takeaways, we have first, you need to use lake base as the back end store.
[34:21] Use Unity catalog to control Use Unity catalog to control and govern access of all your data asset and data estates. And also use Unity AI Gateway to manage
[34:34] the cost and control access to the for example the foundation models that are used as part of your application. Use agent memory and agent sandbox for
[34:48] cross-session memory and for secure environment to deploy and test your prototype your use cases. And the last one that I mentioned earlier
[35:00] don't mock the database, don't duplicate the data from your database. You can just mock the LLM and the agentic system and use a separate branch so that you can securely
[35:12] test your your changes. All right, with that that's all folks. You can find the QR code if you want to reach out on LinkedIn or if you want to
[35:25] receive the slides and yeah, if you have any questions happy to answer these questions or we can also discuss after the session. Thank you. >>

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.