The Full Stack of Innovation: Building Data and AI Products With Databricks Apps

Summary

Full-stack app development: Ivan and Giran cover frameworks (Streamlit, Dash, Gradio for Python; React/Vue for JavaScript), development workflows, debugging and testing strategies for Databricks Apps.
CI/CD and deployment: Learn how Databricks Asset Bundles automate building, testing and deploying applications—reducing manual steps and enabling continuous deployment for rapid iteration.
Scalability and observability: this video addresses performance optimization, multi-instance architectures, monitoring with OpenTelemetry, and best practices for production-grade Databricks Apps.

The Full Stack of Innovation: Building Data and AI Products With Databricks Apps

Talk by Giran Moodley, Regional Architect, Africa & Middle East, Databricks ; Ivan Trusov, Senior Specialist Solutions Architect, Databricks
In this deep-dive technical session, Ivan Trusov (Sr. SSA @ Databricks) and Giran Moodley (SA @ Databricks) — will explore the full-stack development of Databricks Apps, covering everything from frameworks to deployment. We’ll walk through essential topics, including: Frameworks & tooling — Pythonic (Dash, Streamlit, Gradio) vs. JS + Python stack Development lifecycle — Debugging, issue resolution and best practices Testing — Unit, integration and load testing strategies CI/CD & deployment — Automating with Databricks Asset Bundles Monitoring & observability — OpenTelemetry, metrics collection and analysis Expect a highly practical session with several live demos, showcasing the development loop, testing workflows and CI/CD automation. Whether you’re building internal tools or AI-powered products, this talk will equip you with the knowledge to ship robust, scalable Databricks Apps.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

What frameworks does Databricks Apps support?

Python-based frameworks include Streamlit, Dash and Gradio for rapid UI development. JavaScript teams can use React, Vue or other frameworks with Python backends for data connectivity.

How do Databricks Asset Bundles enable CI/CD?

Asset Bundles define your application infrastructure, tests, and deployment logic in YAML. They automate building, running tests, and deploying apps—enabling continuous deployment without manual steps.

What is the full stack for building data and AI products?

This video presents Databricks Apps as the full stack for building data and AI products, combining the data foundation with AI capabilities and user-facing interfaces. The full stack includes data ingestion through lakeflow, unified governance with Unity Catalog, feature engineering, model training, and deployment all within the Databricks Data and AI platform. Databricks Apps enable teams to build complete products that serve internal users, customers, or both without integrating disparate systems.

How do I build production AI applications on Databricks?

Building production AI applications on the Databricks Data and AI platform involves combining data ingestion, governance, feature engineering, and serving into a unified stack. This video demonstrates how Databricks Apps provide tools for creating user interfaces, integrating with your data and models, and deploying production-grade applications. The platform handles infrastructure management, scaling, and monitoring so teams can focus on building features that deliver business value.

What is Databricks Apps and why would I use it?

Databricks Apps is a feature of the Databricks Data and AI platform that enables you to build and deploy data and AI products with integrated user interfaces. This video shows how Apps eliminate the need to build separate front-end applications by providing a framework that connects directly to your data, AI models, and business logic. Apps allow teams to ship products faster by reducing integration complexity and enabling rapid iteration with automatic scaling and reliability built in.

Full transcript

`0:00` Introduction to Databricks Apps

Hello everyone. I hope you can clearly hear us. Gan, how does it work for you? Yes. Yes, I think so. Cool. Was uh good. Hope it's the last session for the day for everybody and uh happy hours. One session away. Yeah. Um so um our today's talk is about um the datab bricks apps a new capability of datab bricks which evolved over the last year starting from a small private preview into a fullscale product surface and uh in this talk we would like to cover a lot of details uh prior to that uh we must work a second I was clicking the wrong way.
Yeah. So a quick forwardlooking statement from the legal perspective you know it's not a big promise right so we may say some things say are anticipated okay good um who we are trus I'm a senior specialist solutions architect in the datab bricks I emia team I'm physically located in Berlin um and still would be happy to connect with um you know our colleagues and customers all across the world my main topics of knowledge are uh unity catalog serverless security and datab bricks apps uh and I was always interested in building the applications and uh when this uh capability appeared in our platform I was really happy with that giran to you hello everybody bit like the band Jan juran with a g so hopefully everyone here is a sim labon fan I'm also a solution architect here at data bicks covering a variety of customers uh mainly across middle east and Africa And previously to that I spent the best part of a a better part of a decade building and deploying mission critical apps for a variety of different customers.
So it's interesting to go bit full circle uh learning about data and air and how it scales and how we're evolving and combining these two worlds together. So it's very much a a passion project of mine and very excited to be here today to speak to you folks as well. Cool. Okay. Um, let's uh Jeran, would you like to talk a little bit about what we would like to and just a bit of a caveat as well because when we ran this preview internally, this presentation, I know I don't want to offend anybody here.
I know some people are fans, but uh, you know, there, you know, my superman. I don't care what anybody says, but hopefully we're going to take you here from, uh, the left to the right. So, for those in the folks, maybe I could a show of hands. Huya has a uh an u maybe a background in in data engineering data analysts for I assume I assume majority of people here are which is fantastic um okay for those in the room who here comes from maybe a software engineering background doing a bit of front end typescript okay so there's a bit mixture of bag here that's great news because today we don't want to make any assumptions so again whether you're starting your journey with building uh your applications uh we're going to be giving you kind of a a a level grounding upwards So the idea here is to give you a bit of a talk in terms of how do we build this applications with data bricks and then also give you uh sufficient knowledge to think about okay what what what does it mean for your organization and how can we enable for future use cases with data bricks apps.
Yeah. Um in terms of agenda real quick so we're going to do a quick intro then we're going to talk about the frameworks and tooling that you can use to build applications. We're going to run through the topic of development life cycle. Uh then we talk about the very important piece of apps which is integration with other data bricks you know interfaces and services. Um indeed if you have an app you want to test it before showing it to your end users and we going to cover the topic too.

`3:21` Understanding Databricks Apps

And when you tested it you want to deploy it. So the question is how and we're going to talk on that topic too indeed. And then we're going to touch upon best practices as well as monitoring and observability and give you a glimpse into the road map. Cool. Now this piece goes fully. Yeah, thank you. Um, so let's kick into here. So one of the things that we always get asked is you know what do we actually consider what is a data and AI application and the fundamental you could textbook terminology it's about an uh an application which allows uh a non-technical let's say business users to interface with something through an interactive means.
Uh I would say everybody here today is a user of a data and a application. So I'm sure if you are probably using some ride sharing maybe like Uber or Lift there today you know if you're tracking in terms of your ETA that's using AI and machine learning underneath the hood to determine like how far your ride is and then also surfacing that in a way that makes you informed of those decisions along the way. Um the key thing with this is um it's about connecting some of the very complex uh logic in the back end to something that's a lot easier to face and move forward.
And the data bricks um there's kind of two journeys which we've seen so far. So I'm sure here today everyone loves a notebook. Everyone loves getting you know maybe those who may be less familiar with it. But again notebook is really the heart and soul of data bricks. So when data bricks actually started its journey, one of the things they wanted to do was put a a notebook a Jupyter style notebook in the cloud and how to manage that for data science teams.
But over times, you know, it's it's seen a lot of love, I've seen a lot of development, but as you probably know, I don't really want to be giving an a dashboard to my CEO or or the CFO, right? They want uh sorry, a notebook to from this perspective. If you look at uh capabilities like AI, for example, that's a really good way of being able to take some of that data and make it a little bit more accessible. But the key thing here is like it's very much fixed in its nature.
So it'll show data as needed, but there's no flexibility in terms of uh how we can edit and update that. And that's really where apps come into play. When we think about an application, it's that full customization, that flexibility that allows us to interact and update in real time. But to get there, um you know, we noticed that there's several challenges with it. Number one is around the complexity of it. So again, a lot of people who are might start developing with different frameworks.
If you don't know where to start, you may start looking at the web and looking through a whole variety of different technologies. Uh and then actually getting into production uh is a bit of a challenge. Time to delivery is very very important because with organizations like we have here today there's always a need to update and iterate quickly and then sometimes you might take shortcuts and eventually that means like there might be some security concerns that arise from it because you know people are putting credentials uh without even thinking about it that gets leaked and then you know uh we're opening ourselves to litigation and other financial implications and this is a really interesting journey.
So if you ever go on a website called roadmap. sh um gives you a career path of all potential you know tech uh career paths out there. So this is the one from the to start off as a fullstack developer. So let's take a journey into to how we would do that. Okay. Number one is the language of the web. So you can't escape things like JavaScript, CSS and HTML. That's the founding uh foundation for any web application. But on top of that, we may want to start learning about additional technologies.
So for example, we want to be able to look at the wrapping of this using things like node uh being able to kind of connect to specific databases like postgress for more that operational things. or even lakebased now that it's been announced. uh and then also secure that using your own um security model last you know you've been then hit in the face with hey I don't work in infrastructure why why am I being asked to then deploy this on on on Amazon or Azio or even GCP right that's sort of multicloud uh way of de-risking so setting up things in relation to v-netss firewalls security certificates you know that's it's not something you would consider doing uh when you embark this journey and you know going from here to there to keep going and then you're just as confused as right?
But surely there's got to be ability to do this. Now, how can we improve the experience for the bunch of you know lovely folks that we have in the room here today. Okay. And this is really where data bricks apps comes into play. Okay. For us, it's about not just democratizing data and AI, but also enabling you to get to building these data and AI applications as quickly as possible and giving you time to value so that you can delight your internal users as well as your customers.
So, data bricks apps um again very easy to build and get started. We provide out of the box uh frameworks and we're going to show you that in a bit more detail today. Very easy for us to do uh operate. So again using that serverless model we can deploy things um near elastic uh compute and the ability then to kind of share that uh information out. So again integrated with your workspace there today as well as your identity provider and to be secure as well.
So if you're using your data in unity catalog again that's going to respect some things and man is going to talk about this in a bit more detail and as well as including things around ex uh accessing APIs and so forth and other data brick services. Again really key to this is historically data bricks is being used by data practitioners including yourselves in the room but now we're extending that even further to software engineers because what we've seen in a general trend uh in the market is that we're bringing applications a lot closer to the data rather than bringing data to the applications and what I mean by that is as a full stock engineer myself is historically these two uh let's say domains were treated in silos again you have your separate stack for building and creating ETL pipelines to curate a data product and then you bring that and copying that data over into an operational system.
But now again as you seen in the keynote earlier it's unifying that data separating the compute and storage a single copy of that data which you then get access for both analytical as well as operational purposes right and just as a bit of a small subset uh of applications which we're enabling organizations today. It could be a simple dashboard creating a chat application to even go even more full scale advanced data analysis and understanding and how that works um through an agentic process there as well and we'll be walking through some of these use cases as well uh in further detail.
Perfect. But you know this sounds really amazing um but how do I actually start to consider what applications are in terms of tooling? Okay, I'm going to hand it over to talk about that now. Sure. Thank you so much Gan. So uh frameworks and tooling um there are so many like uh you know always disputes about engineers which tool to use this or that. The cool thing is that uh would it be Python or JavaScript or TypeScript or any specific framework they all can be hosted on data bricks as soon as this is a web application.

`10:43` Frameworks and Tooling

But from the overview perspective you know just to understand right um we consider usually grouping these apps into three groups. So one is apps that are built on pythonic frameworks such as dash streamlid and some other new like like reflex for example. Um or you can build your application in a JS front end and then plug it together with your Python back end. In this case um you will be doing like static side generation and then your uh your Python in the back end will serve the data to your front end.
And um a very very new announcement that came out out today at the keynote. We also will support a NodeJS runtime and a NodeJS back end. So you will be able to run a purely NodeJS applications on data bricks apps. Cool. But the question is uh do I like need an app or like which which tool should I use? When should I use what? Um and we were thinking about this. I came up with this very simple concept. So take a look.
If you want uh like you know no code low code data visualization which only has like you know some filters and minimal user input then you probably would just go with dashboard. If you want your user to have a chat interface towards your data with minimal like you know customization then probably gen is something that would be already quite sufficient for your case. But indeed, you know, large organizations, enterprise scale organizations require more advancements, require more custom options.
And in this case, you're going towards the data bricks apps territory for a quick start. And in case if your team has more skills towards Python, you probably would prefer one of the um one of the frameworks listed here. Shiny for Python, Gradio, Reflex, Streamlit, Plotly, and Dash meaning um those would be your choices. Uh however, if you want to go further and you know if you want to have a very custom application or have specific uh UI heavy components, you would probably look towards building your front end in JS or TypeScript and having your back end in uh having your back end in Python or even making everything in Node if you obtain these skills.
Cool. Um but how do we start quickly? Can I can I just you know just spin it up and you know start it. Um any of you can open your laptop right now, log into your datab bricks workspace and uh we counted this in five clicks you will get a running datab bricks application like no joke we specifically counted the amount of clicks it is five clicks um and we provide a set of templates for you just you know to kickstart um but indeed you may have your own organizational templates which enforce specific code styles or choice of technologies and we also support this good um million-dollar question.
So, do we do this in Python or do we do this in JavaScript? Right? And um we consider the arguments for this in the following way. If your team is primarily, you know, like Python oriented. If most of your application is server side logic with, you know, just a set of user inputs, then probably Pythonic framework would be something that fits in your scenario. and they still can be extended with JS JSX code for you know custom visualizations.
On the other hand, if your team has you know experience in JS or very good at uh you know by coding with V0 and something like this then you may consider using JS on the front end then generating a static asset out of this and serving this together with your application. Right? In uh in the practice we see that JS native applications may have better client side performance especially for you know heavy UI operations like when you need to do some 3D visualizations.
Um and uh they provide you know more styling capabilities with custom CSS options styles and Figma and everything. Um a very powerful combination that we explore at like know we see this with a lot of the customers is the following concept. So you have a single datab bricks app and it has two routes. One route closes the UI topic and it uses static files and these static files are generated from the uh generated from the UI uh generated from the front- end framework like classical combination of uh white, react and typescript for example. uh and then you have the API route and this API can you know can provide data in many many formats can be rest graphql arrow and then you have clients who connect to this API uh one more thing that we quite frequently observe is the so-called pure API uh applications in this scenario you don't even have any UI you just want to serve your business logic in a certain specific way and uh for this Python with fast API and you know you can easily deploy this on data bricks a very very convenient choice.
We have seen this a lot because this allows you to combine certain blocks of business logics under under specific API routes and given the fact that we now have lakebase if you have missed the announcement. You practically can build a full scale application on data bricks which will be able to serve thousands of users at the same time. Good. Um, it was all slides until now, but we're here not only for slides, right? Uh, so let's take a look at this in action.
Okay. So, how would that look for us? Uh, oop, sorry. Where is that one? First tab. First tab. First tab. This one. Yeah. Cool. Yeah. So, let's take a look at what we can do with uh React on the full stack. If you don't know where is this located, this is map of Germany with uh data served from the datab bricks SQL and uh it has digitalization level of seven square meters. So this is the type of application you can build using datab bricks apps and datab bricks SQL.

`16:48` Demonstration: Map App

Um this data is taken from uh HDX humanitarian data exchange provided by meta. I must mention this. Um and as you can see you know I can have all kinds of customizations. I can have all kinds of you know like interactivity on this map. So it's not you know just like a generated static image or something like this. It is an actual application which you know we can also check check the styles out and so on and so forth how such things are built.
Let me quickly uh demonstrate. So here is a quick example. This is the actual code behind this app. This app is by the way open source. You can find it on GitHub. find just search GitHub terramatria and you will find this application. So this is exactly how it works. On the front end we have a piece of TypeScript which was actually generated by JGPT. I'm really bad at Typescript and it just sends an API request towards the back end and our back end serves uh serves a set of um uh practically it executes the query based on the incoming connection and then sends the data back to the client using arrow format.
What is cool is that since we're using arrow, no need for double serialization dialization. We're just directly streaming the data down into the front end. And this allows us to easily work with this data. As I've said, this is open source. You can run it yourselves and, you know, build your application out of this quick example. Good. Okay. Um, where do we have the slides? Here. Yeah. So, um, with that, that was the piece of the demo for you.
We're going to have a couple more, don't worry. Um, and after looking at this, you're like, okay, this looks cool. How can I do it myself? And for that, you will need a little bit of introduction into the development life cycle. And Jiren, would you mind helping? Yes, of course. Thank you, um, so what I'd like to do is take a bit of a groundup approach to this. And this is definitely that amazing future vision in terms of again how we're enabling customers to build that full scale data application combining the power of data bricks as you know uh through an easy layer but okay for many of you out there like how do we embark that journey so one of the things I want to talk about today is around maybe looking at some of the pythonic uh deployment with dash um I'm going to talk a little bit about terms of the actual environment itself like how it's actually created what runs behind the scene and then give you an idea when things go wrong and what are some of the things that we need to look out for because again ideally we're like one click deployment everything works but as you know we need to iterate and make sure things are operational as needed.

`19:05` Development Lifecycle

Okay. So in terms of the development life cycle, so this is a bit of a simplified version where you have your application code that's written in Python today again and we also will will be supporting node uh as an environment. But you have your application code there today. You also have a YAML file which tells you how the code is uh is going to run or its entry point. Um and also some additional configuration. So if you're declaring things like environment variables as well as secrets uh that can be deployed there as well. and your dependency file.
So, data bricks provides a set of runtime out of the box. But if you've got some additional custom libraries uh that you want to use, even if it's something that you've built u you know through your own internal private repository, we can we can pull that in for you to use within the application itself. Once that's available, you then need to push that publish that to your local workspace. So again, we're using synchronization there.
Ivon's also going to be talking a little bit about from like how do we industrialize that using a CI/CD approach. So um we'll we'll be talking about that but as a as a as a general purpose be uploaded to the workspace. Um you then need to create an application slot. So this is going to be a a specific unit of work. Again it's it's a it's a type of compute that we have developed in data bricks where that application can run.
Think of it as just like a Kubernetes pod uh for lack of better word. And then again it's how we deploy it. So you take that code, deploy it, install all of it, uh dependencies and then it's served through HTTPS. Okay. And just to give you an idea as well uh between this again we support different uh types of environments and again about that synchronization of the code having the latest changes or depending on what what kind of branch we'll go there a little bit more detail.
And just to give you a feel um again for those who are maybe not as familiar with application development uh we're going to be walking through this uh right now we will take our template which is off the shelf. We can modify it as needed and then have a quick development loop iteratively as well. So again here's a an example where we'll walk through this. Okay. So with that in mind let's go and create a databicks app from the template.
So just going to take a look here. So we'll just drag all that up across. Okay, perfect. So just going to go here to my dash application. So So underneath compute, um, so you might see this at the moment. Uh, we have a list of compute types. So you're probably well familiar with using things like, you know, classic um, compute as well as a SQL warehouse. There's a new one here that says apps. uh this should be available in your environment for I think majority of users here today um subject to region availability but um this now becomes available what I want to do here is click on create app and I can either you know for more advanced users so we'll do the latest create things using custom uh application code but for now again I'm a new I'm a new user at data bricks or uh to or developing applications I just want a quick uh starter introduction so as you can see here we support all of the common uh frameworks as mentioned.

`22:06` Demonstration: Taxi Data App

So, Dash, Flask, radio, shiny, and stream lit. In this case, we're going to be clicking on dash for our data application. Um, in this workflow, uh, what's really important about this is apps can provide a set of isolation like any other type of securable asset within data bricks today. So, it's not that case that we spin up an app and everyone has access to it. we can make sure apps are using the resources as permitted in control to put the guardrails and control of your organization in place.
So in this case I'm going to specify a specific warehouse. Um I then want to give it a set of scope. So this is really really key as well when we when we do some of the on behalf of authentication workflows and in this case I'm going to give it an app say UI dash okay and it's critical as well because we are running on a serverless um uh control plane uh we're able to also apply certain tags and budget policies accordingly with that.
So in this case I'm going to install and let that go through and therefore now we've already deployed it. Initially it's going to take a couple of minutes just to make sure we actually provision the application code but you know just like a cooking show I've got already something baked in the oven. So uh just to give you an idea here I'm going to go once that's available and you can see here that I've got an application running um as I mentioned it's part of your data bricks state so it's a new capability but because it's already working with your workspace you have a secured uh endpoint which we then can click in it actually tells you where the applications then been deployed to so when we make changes uh it's then available um and also the resources associated with that so with that in mind I'm going to take a look at the application here which I've got running.
So in this case we're taking of I think everyone's probably seen this the taxi it's a surprise surprise but it's it's just an idea of taking the data set and being able to visualize it but on the back of this as well we're actually doing sort of a prediction a classical machine learning algo. In this case, let's say I wanted to look at a particular time. Well, let's say I'm looking a pickup code from 003 to 002. Maybe I want to do 00 22, right?
So, I want to do a prediction. So, what this means is I can go fire into our model serving endpoint and it'll give us an update. So, maybe the F is 99 already, but just to give you an idea again that data has been queried through. Actually, no, it's updated. See, it went from 99 to 12. So we've done a prediction on there already. So in this case, you know, it's it's showing all of that taxi data set that we had available.
And just as an example to this, so you can see I've got the taxi fair distribution as the um the title, maybe I want to change some things on the fly, right? So I'll take a look at the code. Uh I'll bring that up so you can see that structure that I mentioned. I'm going to go into the application. Um and again, this is also open source. So this is available for you to use to configure as needed. But this just provides an entry point to it.

`25:58` App Compute and Debugging

Um and let's say for example then I wanted to trigger something like taxifair distribution or say something like map for example. So I'll save that down and then I'm going to go into u this UI here and click deploy. Uh and in near real time it's going to pull in the latest code changes. And because we're already running compute, you can see we've just done it in a matter of seconds. And again, this is actually happening live.
And you can see I've changed the title already. So you can see how quickly it can go from ideation to production in a very short space of time. Okay, that in mind now that we've seen the dash app, uh important to think about the where's this actually running. So I mentioned around this runs on the data brick serverless compute. So just to give you an idea today each application slot we're giving you two vCPUs two cores as well as 6 GB of RAM today and what this means is um this can run again uh a very lightweight application and something to bear in mind with uh a lot of the folks here in the room today is I want to make this kind of crystal clear in the fact that data bricks apps is really intended to be the lightweight orchestration and UI framework.
It's not intended to do heavyweight uh you know computation lifts things you would traditionally think about you know spark and photon. What we do encourage is users to actually push that compute down into serverless into the jobs into things like model serving endpoints if you want to call your LLMs as well as your SQL warehouse to do those works and even now that we've introduced lakebased if you want to do operational work you can integrate there.
So again I wouldn't want to do within the application. It's just take the request from the from the user then offload it to the appropriate compute as needed. Key thing here as well um again similar to the database runtime we provide a secured hard hardened Linux environment. So all of the security libraries in place. Uh we also s provide you again Python 3. 11 um out of the box. We will we will be upgrading as well as as it see fits and we provide an isolated virtual environment.
So the requirements file that you provide if you've got any additional libraries that's going to in be installed and then the application code will run. Key thing here as well is that code will be um bring up from the workspace. So again as that changes like I just did on the fly it's very easy for you to deploy um and make that available. Key thing here um is around again when things go wrong what are some of the things that we can do with this and I know even you want to maybe touch upon some of this stuff as well.
So practically um we can kind of group the problems into three uh into three blocks. First one is code failures. As of now we cannot like immediately fix logical issues in your code but you all know how to do this nowadays right assistant compiler anything like this would be able to help you. Um it is really recommended to test the application locally before deploying and we have a whole section about this today. um verbose logging really would help you and like monitoring and observability and also talking about this like closer to the end of our talk today.
Second thing that usually you know somehow happens wrongly is the environment configuration and um you know use if you're using Python use pyantic settings to verify the validity of your settings at the moment of application startup not at the moment when user hits the application right um if you're using JavaScript or uh TypeScript applications they sometimes have some issues with course that probably could be the reason of the problems please do not use personal access tokens do not put them in the code.
Do not get them from the secrets. Application is automatically configured to use specific tokens and user tokens will be provided from the request header. I will show this in a moment. But in any case, do not use personal access tokens. No need to put them in secrets. No need to write them into the environment variables. Um yeah, and in terms of startup, just make sure that the startup command you are running is correct. and uh avoid using relative paths because the paths in the container are different.
So always uh use like you know uh proper relativation to the root of the project. Okay, that would that would help you. Uh good move on. Um yeah, so this is an interesting one. So we've now got the application running. Um again now that it's on data bricks, why wouldn't I want to bring you know some of the services that we have available? I think even you were touching up on this as well if you I'm going to take care of this.

`30:34` Integrations and API Access

So yeah, um applications like practically it's just static content if they are not filled with data and the question is okay so what data can I plug in and the answer is well pretty much any data that you would have on datab bricks and even in the external services. So first of all like you know just datab bricks native services like SQL warehouse spark compute workflows vector search serving endpoints uh anything can be easily plugged in and worked with through the datab bricks SDK for Python and there is also one coming for JavaScript TypeScript you can trigger workflows you can embed dashboards you can write events and messages uh and rows into the lake base so you have all types of options outside of that you can also reach out to the cloud provider services if that's of your need for example if I don't know maybe like to trigger a function outside of datab bricks or something like this and indeed you can reaching out you can reach out to the external APIs if that's required uh the external APIs however are subject to serverless egress gateway so you know um apps are under the common security pattern of uh of datab brick serverless and there are certain network boundaries such security agress gateway apps will comply to it.
Um now let's talk a little bit about okay but which user will do the actions like will it be the service principle will it be the user identity what exactly is happening there and this is the topic that many people don't get and I'm really happy to present this to you there is a whole separate session from who is like you know the main engineer behind the apps about the OBO and how it works in detail I would like to focus on specifics of like usage right so by default when the app is created there is a service principle created for it and usually the service principle can be used to perform some actions which are outside of the user context.
For example, you can expose an um a UI which will customly create like prepared workspaces for your departments. In this case, this action should be properly executed under the service principle and this is where you will use it. Um and you can grant this service principle access to the resources. Okay. But sometimes you want to access something in user context like a table or send a query to an agent or execute something and see the result exactly as the user would like to see it.
And in this scenario you would like to use something that we call Obo on behalf of and this is really easy to use inside the datab bricks app. One more thing to mention is that you can have two datab bricks clients in the same runtime in the same uh application. one can use the service principle identity and another one will use the client identity. Okay. So how OBO works in this perspective practically in the following way.
Um when you are sending your request to the app uh it can come from two sources. It can either come to the API like through through curl or rest API or python or it can go through the browser via cookie. If it comes from the browser it's a UI authentication. we perform that for you. If it comes from the like you know curl or any type of programmatical client it still hits the control plane we will still do the verification.
Uh it's just a different way of providing the identity. So what we will do on the control plane if the user is correct if the token is correct and token is correctly scoped because applications have access to different scopes then we will pass this information. and we will generate a temporary token and we will also tell you who is the current user reaching out to your application and it will go directly into your application as a header.
There are two headers. One is called x4ared access token another one xvared username if I'm not mistaken. So what you can do is that like in this code example this is an example for fast API but I'm sure you can adopt it for anything. um we initiate the workspace client based on the current request and then we execute the query based on this uh based on this uh information and we can just look up the current user and return the user details like a quick example.
Okay. Um how we can use datab bricks uh SDK other SDKs with OB here is a quick example on the left side where we can query a specific table using the user identity. So we receive the token, we pass this token into the SQL. connect function and then we execute the we execute a specific query and then we return the result back to the user. Another case is for example I want to reach out to a serving endpoint for example to a chat endpoint or LLM endpoint.
In this case again I'm receiving the token initiate the workspace client with it and perform certain actions on the datab brick side. Important thing you can use both service uh service principle based workspace client and user in the same step. So for example your business logic requires one action under automated user another action under like user under user context you can combine you can do both at the same time.
Okay. Um so assuming we implemented all of this cool logic now our application really has you know cool connections to the datab bricks SDK and API. We we elevated the user experience. There's just one small thing before we need to test it like how it works, how it looks, how does is it fast, can it load the handle um can can it handle the load sorry so if there will be many many incoming requests how will this work at this moment in time we would like to do some application testing Jan back to you thank you so this particular segment I'm going to cover just a little bit of a foundation knowledge um so we'll talk a little bit about the testing pyramid um everyone loves a good diagram.

`36:08` Testing Strategies

So, we'll break that down. I'll talk you about some of the things that we can do from uh you various different testing strategies. We won't cover all as you can imagine today, but we'll just give you a little bit of a grounding in terms of what um some things that you can do today. Uh and then I'll actually demonstrate them uh from what we've uh been talked about as mentioned. So I don't know if many people have seen this this diagram or an encountation of it but the testing pyramid just gives us a way of visually describing uh testing strategies for your application and your organization.
So similar to we start from the bottom and then we go to the top. Uh I would recommend starting very much to what we deem as a unit test. So going from top to bottom we refer to this in terms of complexity and cost. And what I mean by that is at the bottom these tests, these unit tests are individual pieces of work that tests individual pieces of functionality. They're very easy to implement. They isolated to uh identifying individual pieces of unit of work for a particular function and then they can be run very very quickly to run.
So we're not expecting these types of tests to take minutes or hours. Um once you have your test covered for individual pieces of functions, you may then want to start looking at how these services integrate with each other and therefore they deemed as integration test. So let's say for example, it might not just be let's say calling an API. It may be then I call the API and then I may want to upon perform some additional kind of translation on that API and then push it out into another system.
Well, that's now integrating two or three different units of work. So how do we make sure we cover that? And lastly at the talk we go into more what we call system or integration test or even end to end sometimes it's it's sort of called and that's actually testing the validity of your entire system. So I'm a user I log in I may want to start doing my query. I may then want to understand what the data looks like and as well maybe I want to export that to Excel because that's where all end.
But that's just to give you an idea of some of the things that we're um how we were enabling organizations to do their testing and I'll I'll cover that as well. One thing to bear in mind is it's great to have a functional application but um there's a question on okay how do we move things or how do we get things to production and how does it work in a production setting and this is really where low testing comes into play it's like you know as you scale your application not just taking it from your desktop and making sure if you've got hundreds or even thousands of users in your organization can it perform can it meet things like SLAs if especially if you've got a critical application how can you identify and make sure it works as intended so again I'll cover that in a little bit more detail in terms of the testing frameworks used today.
Uh again, if you are writing things with Python, um you probably might be familiar with things such as called Piest. Um there's also the inbuilt unit test as well. Uh which is again piest is wrapping around that. But Piest is the de facto way of you know testing your Python application. You may then want to use things like Faker. So Faker is a cross-purpose library that allows you to generate synthetic data. So um again if you generate loads do a test against that uh it'll be pretty pretty important and then as you moving to the sort of system test as I mentioned the end to end testing and you want to click on the user interface um you could use a library like playright which is actually cross functional and I'll walk you through that on the middle side if you start looking at the Java uh script ecosystem it has an analogous things there too.

`39:27` Demonstration: Testing

So unit testing gest uh is pretty much used for functions as well as components. If you want to start looking at integration even behavioral type of testing you have a more abstractive framework with jasmine or enzyme. There's a variety out there uh but these are kind of the more common ones that are used and similar that end to end testing playrights also supported in JS but we also have Cypress there as well uh to simulate um user interactivity in the browser and the last final stretch is making sure it works in production so again we'll be looking at things like locust for load testing that's a Python very specific library but if you come from the Java or SE world gatling and jet are also credible um capabilities you can use as Okay.
So, with that in mind, let me switch over to my demonstration app and I'll I'll walk you through Sony's capabilities. So, I have uh actually let's just bring up the application here. So, I close that out. So, um yeah, don't shoot the messenger. So, in this particular application, I have a simple, you know, task manager like a to-do list effectively. So it's a crowd application that I can use to insert task, edit it, update and delete uh so forth.
So just to give you an idea, if I want to add a specific task, I can say like to do uh social media, I can't even spell and post, you know, I've got that there. That's enabled. You know, it's it's visible. Um it does what it says in the tin. Uh if I take a look at the application code. Okay. So what I'm running here is a flask application um that's served in real time. Um this is the specific function call for creating.
So as I'm sending a request to through to through to through the browser uh it's hitting this endpoint. Uh it's picking up a local database session. In this case I'm actually using SQL Alchemy. Swapping out whatever database you want. Um I've got a data model here that was created in Pyantic. So again, uh ways of being consistent and then I'm committing that to the sort of database. I've introduced a time sleep here just to say look this is what could happen when you're doing an actual database, but the principle still applies.
If I flick over to my test suite here, um you can see that I've defined a class. So this is Python specific. So uh sorry, pi test specific. So every uh test suite will start with a capital like you know class name uh test capital T and the relevant test individual pieces of functionality has a test underscore. So in this case I want to create a a task. I'm firing a response to my task API and then I'm also doing an assertion to make sure that I'm getting the right response.
It's interesting at the moment because even though this says client post and the API, I'm actually not running the API. I'm actually using a um you know a methodology called dependency injection. You might have heard this is called a mock or fixture in other realms of work. But the idea here is when you want to test uh your code sometimes you may not have access to that environment. But what you want to do is test the implementation of that code.
So I'm saying if this client was running what would the response look like? So in this case when I'm doing um this application here you know in this case I'm returning this JSON structure and as well as a return code of 2011 to signify I've actually entered something into the database. If I did something like modify this and then run the test suite again it will give me a failure. uh from that perspective if I take a little bit further in terms of uh so that's looking at a kind of more of a unit test at the moment if we want to look at things in relation to uh user testing as as well uh we can flick over to play right so I'm going to take a look here and look at the user journeys API so just to give you an idea is uh I'll take a look at uh adding the task journey here.
So again, another type of thing where we're actually able to control spin up a browser session and then on behalf of the browser go in do certain changes and then see what the results of that uh to do. Um maybe to help demonstrate this because there's a lot of code is I'm going to kickstart that uh test suite here at the moment. So what I'm doing my hand is off the keyboard by the way you can see here. Uh and also not doing anything guys so it's it's not magic.
So um yeah as you can see the um the test suite is actually going in spinning up a browser session looking through my application and doing these critical user journeys. So in this case again I want to create a task I want to modify it then I want to delete it and wipe it down and does it render the page. So again that's an example of that working in action. Assuming those user journeys work, I then want to, you know, test this in anger.

`44:37` Deployment and CI/CD

And this is really where I want to uh use things like locust. So if I go over to my um testing here um so locust what I can do in this stage is this allows us to simulate user sessions um and do a variety of those user journeys that I talked about. Uh but then I have the ability to then scale up over a specific defined time period. And this could be really handy because you know I think a lot of you here today have probably seen uh through various different application testing is you know it could be 8:00 in the morning whole you whole bunch of uh you analysts or users access an application and then you know they're having that spinning wheel or hamster wheel because they're unable to access the page.
And you know these are the type of things that if we've done load testing before then we're able to identify those bottlenecks um and then fix them. But just to give you an idea. So let's say uh I've got this application now and I'm giving it internally to 100 users. I may say you know over a um over a period of um you 10 users at a time. I may want to see like how does this perform over a 30- secondond period. So in this case I can go ahead and spawn those tasks.
So in the background Python is spinning up individual threads. It's then hitting that those those API. So this is now doing a live test. So the application is running and you can see like you know over 60 requests are hitting the server at the moment. Um and you know if there's a case have we even scaled this further and we starting to see failure rates then it's a case of us understanding you know have we sized up the machine accordingly um and so forth.
But it looks again we've we've done a very similar sim u simple um simulation here but as you can see you know nearly 90 requests per minute um hitting the task creating new uh uh you know task in in task manager and if I hit refresh here for a second you can see I've got 600 tasks or nearly 600 tasks have been generated as part of the load test as well. Okay. Um so with that in mind um assuming we've tested the application uh we've proven that it's worked now how do we actually move this from the development uh to a production state and Ivon's going to talk about that process in more detail.
Thank you. Thank you. Yeah assuming you're the guy who tested and prepared the application and I'm and now I'm just you know just a DevOps guy who needs to deploy all of this somehow just throw it out into prod. So quick question here. Who of you have ever written a CI pipeline? Oh, that's great. Okay, I will just skip myself. That's better than I I was expecting actually. No, no. Okay, that's that's great. Okay. And who of you have ever had a CI pipeline fail like in production critically when you're trying to roll out that bug fix life?
Okay, that happens. That happens. Yeah. Uh my example probably shouldn't do this. I hope at least, you know, but demo effect could be with me. Um so for those who are completely off the chart right uh what's happening with CI/CD nowadays uh two pieces there is continuous integration and continuous delivery or continuous deployment there is a big fight about how things should be called I will not go into that um in our terminology CI is everything that happens when you write the code build the code and run the tests any kind of tests and then staging/production is when you roll out the application to actual user And this these are two substantial and different parts of your pipeline.

`48:09` Demonstration: CI/CD Pipeline

Okay. Um how do we uh help with this CI/CD journey? So um datab bricks asset bundles are a standard for datab bricks deployments when it comes to deploying data products right. I mean if you want to deploy infrastructure you can consider using terraform but usually the data teams who actually develop the data products they rarely would like to write terraform because it's a completely different skill set and a different way of doing things.
The cool thing is that datab bricks asset bundles fully support databicks apps. So what you can do is that you can describe your app definition including the resources, including tags, including names and everything. And this will allow you to easily deploy your um um to easily deploy uh your data bricks app multiple times through the CI pipeline. So you can deploy your application first time to staging then to production, right?
So you can like roll it out on staging, get some user like feedback and then if everything is fine, you go towards production. The cool thing of bundles in this perspective is that applications rarely leave on their own. They usually connected to some jobs which populate the data, some SQL, some models and so on and so forth. You can conveniently describe these things in the datab bricks asset bundle and then connect them all together in a single file.
Um one thing about the format I personally don't like YAML uh unfortunately we have datab bricks uh bundles SDK in Python. So you can describe all these things in Python and they will be easily deployed. Okay. Now assuming we have written this uh this definition right now we would like to understand how our CI pipeline would look like. So first of all your application code must be in some kind of git storage. It is an absolute like must.
All things should be there like it should be there for versioning should be there for code control and many many other things. Uh which provider do you use is completely up to you and your organization. You know GitHub bucket there are many many options. All the git providers have some kind of CI builtin. Uh and uh what you need to do as a developer is that you need to describe your uh application and its dependencies in a dub format. uh in the formats of YAML or Python, right?
And then you need to use your datab brick CLI inside your CI/CD pipelines. Logically I usually like if I'm being asked so what is CI what is CD? How do we split this? I usually recommend to think of it in the following way. Um um so on the greeny side here right um what happens is that developer pushes something into the feature branch and we will install the dependencies run unit tests uh deploy the new version and run integration tests and all this usually happens in the testing workspace or like several set of workspaces and here we are interested in collecting as much as possible you know coverage metrics locus test results uh playbrite can do video capture So you can you know watch why the pipeline has failed and so on and so forth.
And all this goes in the feature branches and multiple features getting merged into main one by one and at some moment in time you decide or manager comes and says we need a release now. Um which case happens more often is up to you. Um but you decide that it is the time to do the actual like deployment to production. And the usual scenario is that we are using a release. We create a release and this creation of a release triggers a c a cd pipeline.
So the uh the continuous deployment pipeline and it usually takes the pre-built package or builds the package and deploys the new application version but now towards the production workspace and maybe it you know collects some release pre-release information. Um what do we have in terms of tooling for this? So first of all Python projects I mean you can use requirements txt nobody will limit you in doing that. Um but uh we see a lot of UV adoption uh recently.
Uh we are also planning to introduce native UV support for datab bricks apps. So you know if you're choosing the package manager UV is probably the best choice. Um you can also use like other pro other project managers uh package managers in Python but you will probably need to write like a small custom hook uh to generate the requirements txt. Um you can use the self-hosted pipe registries and pull and put your application code in there and then install it when you're starting the app.
Um and just a common recommendation, right? So you can split like your code into like the actual application logic and just common sets of things that can be reused like the domain model for example and so on and so forth. Um, in terms of the CI/CD setup, currently the proposed way is that you use datab bricks CLI and then you use DABs to deploy your application to several workspaces and you can use variables to uh kind of specify application behavior differently depending on the different uh stage. for example like in production use this catalog in development use this catalog or this schema or this model and so on and so forth.
Um if uh you have datab bricks network access policies in this case you would like to use the self-hosted runners runners which are running inside your network and they reach out to datab bricks. Um we're also planning to introduce something which um you know can remind you of something like versel when you have a hook which is constantly on your repository and as soon as you do a push it automatically you know triggers everything into data bricks and this will be kind of a managed experience more or less uh this is on the way um not there yet unfortunately um okay so now I'm going to do what uh nobody likes to do is to make a release when there are many people standing and looking at you at the same time.
No pressure. Yeah, but I hope I I will be successful in this task. Uh so um this is my application running on data bricks. This is a really really really simple application. What it has is it's just one uh UI interface which shows that this was deployed automatically and the current version of this deployment. So what I can do now is that I can go and just create a new release. Let's let's increment this. Let's assume we added some cool feature or something like this.

`55:02` Scalability Best Practices

Let's hit this. So, it's going to have the same title. I will leave the comments blank. Uh and then I this is not a prayer release. So, I'm going to publish my release. And what's going to happen is that this will trigger a CI a CD pipeline. So, this is a CD pipeline now because it's the deployment pipeline. And uh we can quickly take a look at the code of this pipeline uh while it is uh running. It's going to take like 30 seconds or something.
So what this pipeline is doing um is practically you know configure the secrets, install UV, build the package, deploy to data bricks and run the new version of the app. This is really like a no-brainer. Uh very very straightforward uh thing that is happening here. If we take a look at our actions now, it has been already finished. I was talking too long. Um, and if everything is fine, then refreshing this page should give me one should give me 019.
Are we good? We are good. Perfect. Uh, yeah. So, um, this uh repository by the way that I'm showing to you right now is also open sourced. We will add the this repository and all the repositories you have seen today into the presentation which will be given out to you afterwards. Uh, okay. Good. So this was the story about the CI/CD piece and now imagine that you tested it all nicely, you deployed it and now application grows, complexity grows the um you know you have more and more users coming and you want to scale it out.
You need scalability because uh um you know because as soon as your application getting popular, you know, you will have a lot of users. Anecdotally, we have an internal application which has more than 2,000 users and we have other customers where we also talking thousands on data bricks apps. So, uh what do we have in terms of best practices for apps scalability? Um first of all in terms of you know development and deployment um before deploying run your application locally with the correct Python version.
Check that everything is working fine. Do not use pat tokens like completely um use uh connect to data bricks using o this is really simple this can be set up pretty much in every scenario and use the testing frameworks as we have demonstrated um as gear have demonstrated you know that's how you can do it um yeah in terms of the deployment use dabs because it's simpler and more straightforward have a CI pipeline to run your unit tests have a CD pipeline to perform the deployment And really a big thing to recommend um have separate datab bricks workspaces one for dev one for prod or like dev test prod and deploy your application in a staging way right uh if you need to act on behalf of the user's identity inside your app don't try to grant all permissions to the service principle instead use obo in your application this is relatively simple thing to do you just receive the header initiate the client and perform what needs to be Okay, in terms of runtime and performance, well, first of all, we're well aware that, you know, two CPUs, six GB of memory is probably not the most biggest amount of compute we provide.
We're going to work on that and we will talk about this in the road map piece. Uh but uh you know even from these two CPUs, you can actually get quite significant results. But we need to apply a couple of rules here. Well, first of all, if you are using the DBSQL connector, it has a sync operations and it's really recommended to use them because in this case um like practically if you use a non-sync operation, it just takes the whole core like completely one process takes one core and um you know if you have two users you're at capacity.
If you have three users, somebody's going to wait, right? If you have a UI with heavy customizations like you know 3D visuals and things like this, you probably would like to use JSJS uh and uh Python in the back end. Why is that? It's not because data bricks apps are built differently. It is because of the way how Python frameworks work in terms of front-end code. So what is a Python front-end framework? It's practically every single user is a separate websocket connection which constantly opened and everything that user is doing on the client is sent in the form of uh in the form of message back and forth.
Uh this works nicely for many use cases but if you have let's say like uh 3,000 users something like this uh you may have performance issues especially if the application doesn't support running several of workers. Okay. Um, so what do we want to say with this is like don't use Python. No, wrong statement. It's just that if your application is planned to be scaled out to let's say like 2,000 users simultaneously. Uh, just apply some common sense in terms of you know how you run it and use a synchronicity where it is possible.
Um, and this is exactly about like small detail of implementation. If you have a long running operation, do not like do not keep a running core around it constantly. Instead, create an async task and let it just check the status periodically. In fast API, there is the background tasks that are very convenient for this for this task. You don't need to keep a constant connection to check the status. You can do this like in a status pool fashion when once per 15 milliseconds you kind of check the status of the query and send it back to the front end for example.

`1:01:04` Monitoring and Observability

Okay. In terms of you know just common sense uh Python projects have a problem that uh versions especially if you use just requirements. txt you are allowed not to pin the dependencies. This can lead to so many problems especially for long-term running projects. It is really recommended to pin your package versions using uv or any like you know uh package manager of your choice. Um if you are going for pythonic front ends always run your application through uv cornorn goonic corn or something like this.
Why? Because if you run just one single instance of the application and one user you know may take all up the memory or may crash the process it will be crashed for all. If it's a UV corn per each user there is a single lightweight thread and if one user fails the thread well not a problem we will just restart and the next user will receive the expected you know the expected experience of the application. So process managers like UVORN fast API works nicely with Uicorn uh flask uses Goonicorn if I'm not mistaken.
Yeah. So you know you can have a combination you can have a mix there. Yeah. Um okay now let's assume that I have been following these instructions I tested my application I made my CI/CD pipeline everything is nice and shiny and I want to understand how the users are doing like you know what they're doing in the app how do we monitor this how do I observe these things and at this moment I don't know anything but I can only call gear to help me thank you okay so in this section we're going to talk a little bit about monitoring and observability and uh uh We'll dig into it from the perspective of uh what are some of the capabilities that we have in data bricks to apps today.
I'll also touch a bit of a sneak preview in terms of what's coming um around this particular area and then talk some of the capabilities of you know how do we then improve uh the application through this lens um you know as Dr. Dre said you know what's the difference um between uh monitoring and observability. So just to kind of break it down to the folks in the room, monitoring really focuses on the what and when. So it's kind of the historic look back.
So an event has happened, something's been written to the log. Um and it's identifying things which are known. Okay. So and this is really around I wouldn't say uh reactive but it it is for lack of better word, right? So something let's say an application crash what do you do? Check your SIS log. you see you see it pop up. Observability on the other hand is this sort of new paradigm you know especially of the movement from the DevOps that have happened in the late 2000s.
It's so it's answering the why and the how. So it's for those sort of unknown and unknown knowns if that makes sense. So it's peering into the application itself and identifying especially when things might start to occur and especially the change what we're seeing breaking down a monolithic application into a microser framework as you probably seen in individual computers is being able to kind of trace uh the events throughout these systems.
So if you if you're starting to um go down this route and you might have been doing it with some of your peers there today, uh this is really where observability shines is being able to answer these questions. And it's worth pointing out together, you know, gives you a full view. So you don't use either or you you really need to have both to construct the view and so and also give you a feel of how you can best improve on on the life cycle of your application.
So again really critical is that monitoring will tell you something that has happened wrong or observability is focusing on the what and the why and the how. Okay, when it comes to monitoring applications in data bricks there today uh we do provide some out of the box tools. So application logs um again as we you've probably seen the application definition where we've got the configuration there's another tab there for application logs and this shows you things piping off from the standard output.
So if you do something like a print or use a logger library by the way this is I would say you know uh to everybody out there please use a logger. I cannot I cannot see how many people have just like you've written print statements and they don't even know what that print statement is but if you're if I could have one takeaway it's just making sure that you have this in place where you've got a clear time stamp you've got the right information level uh in place especially when things are critical and it allows you to identify issues a lot clearer and if you look at the box below uh this is actually providing a standard log which you can then search in the UI uh just something to bear in mind as well these are logs are written from most recent and reverse chron chronological order.
So that kind of threw me away as an application developer because I'm usually usually using you know tail f- log but now it's it now it makes sense because you want the most recent and most critical thing in your view pane. So just something to bear in mind moving forward. Um again really important is that when these apps are ephemeral so if you sorry these logs are ephemeral so when you shut down the application and reboot it it's completely we don't have a history of session of that.
And if you are uh using secrets through the configuration those will be redacted. So again we don't want people spilling that out. Um there is a question people have been asked is how do we actually publish that logs into into data bricks or through delta. Um there's no native way of doing that but there are ways which we can give you mechanisms to push that into a table as well. So that's just about customizing the logger to then either push that through a volume or even write to delta table as well.
So we have options in place. Uh another thing that we do again uh assuming a lot of users but I don't want to make any assumptions because they are dangerous but um if you're using unity catalog there today you would also come through a capability called systems tables which is effectively a way of providing native observability into data bricks and as part of that we're actually instrumenting a variety of surfaces especially in relation to things like autolog. we can tell you which users logged into the application, how long they've logged into application um and and some of the things that they've tried to do with that.
So, especially again for more highly regulated uh you know customers here in in the room today, we help provide that out of the box and that can actually be tied very closely with our security framework as well. Uh if you've come across that so again ways of just giving you that visibility of who's having ownership and what not. One thing I wanted to talk about is around getting to the observability piece and I wanted to focus specifically on open telemetry here and then for you for those who aren't familiar open telemetry is an open-source observability framework for collecting telemetry data and I think what's really important as for an organization like data bricks is we're built on opensource open standard and open toolings.
So for us to talk about this is very important here today because we want you to regardless of whatever monitoring tool you're using. So some of you might be using graphfana data dog splunk open telemetry allows you to create that agnostic framework. So if you decide to use another tool or you want to expose that data even into data bricks for consumption that's uh that's all possible yet today. So this something when we're walking through the demonstration I'm just want to kind of call it out there.
It's combining these best practices just help you again own your own data, own your own IP. For those who may be familiar like the core tenants of observability are sort of three things. So logs you will probably quite familiar with already. There's a second piece called called metrics and also tracing. And in this particular uh slide is just talking a little bit about how do we actually see that journey of that request.
So we refer to this as a uh a span. So something that covers let's say uh the life cycle. So let's say for example a user clicks on click to basket. So that's going to call an API uh in the web UI. That web API then might call a a particular database and then that database may write to a particular data source. So it's going through various different transfers to actually make that happen. uh but important is actually understanding down to the wire where that fundamental piece of failures might happen and tracing allows us to pinpoint exactly in the stack or where that might go wrong.
So and especially if we're moving to that micros service framework where you've got APIs calling other APIs and you're orchestrating that um it can be very difficult if you're just looking at it from log. So this is why tracing becomes very very important as well. metrics. Again, another very critical thing to observability. Um, not just looking at terms of users, how much uh sessions are hitting, you know, particular endpoint, but may we may need to start looking at things in relation to um the application code itself.
So, you know, what is kind of my throughput to a database for example, I can create a gauge or a counter to do that. And it's interesting is like if you're starting to see, you know, coming to the when you're doing things like load testing, you may want to create certain metrics to identify where the bottleneck might be because it might be the case that again um you know sometimes we may be even limited by vertical scaling.
So it doesn't matter how much we scale up a database if we can only push certain amount of connections on that particular node. We may then need to start scaling horizontally for example to support that. But we can only understand that if we start tracking metrics along with other uh observability points like maybe we're looking at I network stats as well as CPU and so forth. So again it's just about collecting as much as information as possible to bring that clearer picture into into place.

`1:11:04` Roadmap and Summary

When it comes to dashboarding and alerting again all rules I say read XL but S3 is really critical. Um so when you're collecting this telemetry data using open telemetry there's a few options you can either send it to directly to a third party um provider uh as well uh to whatever endpoints that they they take it. Another option is building your own uh telemetry stack. So uh number of customers uh that I've worked with as well collect this through a collector.
So that could be typically like a a a Docker container, Kubernetes cluster that um all of your applications are firing telemetry then to is then collected in a in a in a database that can then be visualized through any other tool there as well. Um really critical to especially how data bricks works is you can actually export this telemetry information directly to S3. And what's great about S3 folks is that again we can create things like delta table straight off of that.
So that information is available we can extract it for further use downstream. Um and again when that information is available in delta you know we have out of the box capabilities there. So again you've probably seen things like dashboarding where we can do quicker graph and again understanding how users operating maybe through natural language through genie as well. Okay. Um something as well that we've seen and uh this is a bit of the current iteration but just to give you a glimpse into the future as well.
We're actually redoing our alerting capabilities on data bricks. So we're going to be offering more sophisticated ways of not just understanding when jobs and other comput types go wrong but actually if you're doing things like quering or other sort of observability we can then alert and send that information out through a web hook or even if you've got things like page duty for example um you can take a look at that real time.
So again uh if there's an issue with the application again we want to move from that reactive to a proactive approach there as well. Okay. Um so in this stage what I'm going to do uh for the sort of last segment is talk about okay that application I created earlier with the low testing what does it look like when I've instrumented it and how can I look at what some of the data points that I'm getting. Okay. So if I go back to my application code, right?
So I'm just going to take a look here at uh let's say create task. So for example, in the create task u this is on the flask side by the way. So you can see I've got a request or in this case I'm creating a gauge or a counter. So I'm saying in this case take an observability point that I'm actually adding to the skew and then want to say something called um a trace context. So in this case I can actually extract things I can extract unique information from my front end.
So I actually do have front end observability the same open telemetry information available. So let's say I have a specific request ID that request ID can be translated right across the stack. um when it comes to spanning so this case is that's when I set up that parent kind of tree where I can start to instrument and what's great about open telemetry is um this is a very easy uh you know SDK you just simply just use the decorator here u sorry the context manager to say create the task um as a span and then you can set specific attributes here so in this case I can get the request ID from the front end um so this is unique as well to when you log into data bricks apps this this head is available um and then push information in relation to uh that request.
So in this case when I'm creating a task I'm saying this is the title I can have that logged in and then being pushed to uh to open telemetry. You would have seen here as well um tracing the info. So in this case I'm also sending that information back to u the telemetry tool. So this is where I talked about you know combining both the logging as well as the observability data there as well um to give you that full step and really critical as well if there are errors we do want to encapsulate them.
So in this case I'm also using an exception block. Um so it's it's really key to us to understand that behavior. Um and what I've done here is I'm going to go into the application. So again I'm making some changes. So let's say I'm just doing now a delete all. I'm I'm flushing that out. So it's now I've now dqed that. I'm actually going to look at my observability tool here. So in this case I'm just going to click on reload.
I'm using uh up trace here as the as the system. But again you can swap it for anything. If you've got again a data graphana it really doesn't matter at the end of the day like it's it's all on a standardized uh tool. But really critical to how this works is you know we're tracking uh again very deeper information in relation to um how the our application is running. So if I take a look at um you know uh looking at uh the front end task actually maybe I want to let's let's bring the hour back.
So let's do the last uh 6 hours. Okay. So let's take a look at you can see again when we were doing sort of the load testing earlier um again this is really critical to see how your system is is operational. I'm going to take a look at um backend task here. Okay. So let's take a look at when I wanted to um create this um when I wanted to create a task. So bear me a second. Okay. So it's around there. Okay. So this is again where that sort of tracing comes into play.
When I create um you know I'll send a request to the front end. I'm issuing a post request underneath that I'm then saying uh create task. So that's the individual you know pieces of functionality in in the in in the Python core and then I'm actually you can see even further going into the database. So this is that the connection to my C my database instance and what's happening. I've even got an instrumentation on the database itself.
So if it's a case of like hang on I'm been able to send all of these users through load testing but I'm not getting any rights we may need to identify exactly where is it something in in my implementation of the code or is it something in the database I need to configure but you can only do that if you understand the telemetry information and again this is why tracing becomes so important to that. So this is again a bluffer whistle swap tour but um again this moves beyond just looking at application logs uh and observability is really powerful uh to help enable this.
Um with that in mind uh I think we're on time. So I think we may want to talk about you know what's next and yeah uh for the team. Sure. So um assuming you developed your application, deployed it, love it and you thinking okay what will data bricks do in the future to make my application experience as a developer or as a user better. Um so let's summarize this and give you some glimpses into the road map. First of all, when yes when no for data bricks apps yes uh in case if it's an exploration for your data if you have any data connected services database APIs chat bots ra lm endpoints uh MCP servers custom ones uh you know complex custom visualizations practically you know anything related to this and also a quite important thing is when you want to expose a curated set of databris API I to specific users or specific user groups where we want fit right now.
Mobile apps um a little bit not not our area at the moment. Let's say uh static co hosting static content just a little bit too costly. You know if your content is static you probably can just serve it from you know from GitHub pages or something like this. It's not the case for data bricks apps. Although I've seen customers who are serving uh like they have several projects, they all have DBT docs and they just serving them all inside the same single application possible. uh not yet we are there but there will be a possibility in the apps at some moment in time to make your application have fully public facing you know so you have a URL you can send this URL to like on the net and click there click through and get you know kind of login into the application uh yeah um in terms of uh sorry where I am okay in terms of you know let's recap what we were talking about First applications are GA.
GA as in full support, full production and uh things like this. GA also means that we have added most of the regions uh for the serverless support. So you know you can easily scale uh you can easily run up in not only just you know Europe central or something like this but in all regions almost all regions. Um and the idea of apps is that we take the data and AI capabilities of your uh of your platform and you we give you the tool to prepare a curated experience for the end business users usually.
Um and it's a great tool for several sides. So if you are a data engineer and ML engineer, you can just quickly write a simple application in stream. If you are a software engineer, you need something more complex, more profound, you still can use, you know, your well-known tool stack like Node. js, Python, uh, and with Lakebase also posgress. So, you know, kind of in the light of the recent announcement. Um, and if you're from the business side and you think, okay, so why would I consider like using apps or not using apps?
What would be like, you know, benefits? There are three things. Apps are easy to build. They are simply they're simple to operate and you can easily scale them in terms of you know amount of users um and um to get the data uh to get the value from your business data from your AI services um building a fast and flexible UI and having the chance to quickly experiment on your own is a great opportunity to you know to drive business value out of this.
Good. Uh Gan, would you mind talking a little bit? Absolutely. Thanks a lot. So I think um in terms of the you know what's now, what's next and what's towards the future I want to kind of break it down into these verses from three themes. So I think scalability is uh you know one of the big questions that you know especially in our internal chats and lots of our customers have have pushed is like oh you know two CPU 6 GB of memory come on you know it's like you know my uh my uh you know my what's that what's that piece of device that you just uh you plug in no blueberry raspberry that's yeah it's a long day folks long day right it's like you know my raspberry's got more you know resources that I'm like okay yeah I get I get it.
So to address this concern, we'll be introducing more like t-shirt based sizing. So we'll start from kind of a small, medium, large and so on. So similar to what you've seen with um you know the SQL warehouses. This also includes not just vertical scaling but as well as horizontal. And what we we want to do is especially for users who have let's say uh a dynamic structure where they need to scale up and scale down and be that flexibility.
Um this is something that we're considering as well as part of it. it's very easy for us to scale up up our infrastructure. Um, so just want to put it out there. The real difficulty is actually how do we determine when to shut it down. So what that means is uh we're looking at ways of again if a user is not logged in a particular time period, maybe you want to spin it down to again help you manage your costs in your organization when it comes to observability.
So I know I'll walk through this in a very peacemeal way but something to bear in mind moving forward as part of the new release is we actually enable uh observability out of the box. So the instrumentation that you do uh can still be done but instead of you shipping things off either locally or doing to a third party service you can just send that directly to data bricks and that'll be available for you to natively query as needed.
So again, just focus on instrumenting your app and we'll take care of the other heavy lifting for you. When it comes to again monitoring um how the application is functioning so like CPU, memory and networking, I know it's not the most attractive thing to talk about, but again when you're deploying things which are critical, you want to know that fast feedback and see how your app is operating. You need this level of information.
Um key thing as well is again it is GA to today and we do have SLAs but again for organizations who are supporting highly regulated highly critical workloads we want to give as much uptime as possible. So again number of ways which we're bolstering the service. Um there was also a question that keeps constantly getting asked around uh when it comes to availability and we're actually also ging when it comes apart from maybe I think I think federal government uh or gov cloud but we do have you know enhanced capabilities now.
So if you got hippo workloads PCIDSS and so forth that is supported on data bricks app. So I just wanted to kind of throw it out there if you weren't aware of ready and this comes down to the governance piece because at the end of the day you're using unity catalog for uh data there today you're using your entry your single sign signal um of applications we then want to take this further move it into even the codebased uh application so looking at ways which we can use effectively treating your deployment and your and your application as code in its entirety so I would say it's an evolution of dabs to this so dabs will still be used need to h but what we want to do is say look here's um if you ever use things like Azure DevOps you've got your code there you've got a button to say click to deploy from that code so again you can use things like branching and so forth but again natively to data bricks um systems table so we've got ordered systems out of the box but when it comes to additional workflows in terms of that CPU that telemetry information that's something we're thinking about moving into systems tables again giving you greater visibility included out of the box again something that you don't need to write to Delta yourself.
And really critical around is how do we then not just enable users to create applications, but how do you do it in a reliable way at scale. So if you're an organization that has a very specific way of creating applications, whether that means use a specific framework. This is the way you define how your components layout looks like. This is how you integrate with services. Well, now you can create a template in your organization and have that available for users to pull from that template.
So, they're not deviating against different standards that they might just fire from the internet. You know, there's guardrails in place which you can trust. And this kind of ties nicely into things that we're announcing around the marketplace. So, stay tuned for that uh very very soon. Something to point out as well is um and I know um the team have done an amazing job trying to hobble this together and I appreciate you know it's it's going through the phase around the documentation but uh we've got an amazing colleague uh by the name of Pascal Vogle.
He's done a fantastic job with to rehaul some of that documentation and put the application cookbook. So at the moment it's it's on on open repository but we're looking at bringing that today. uh they provided recipes out of the box in terms of like let's say connecting to Unity catalog volume writing how do I query this data how do I query a model serving endpoint and things you would use as a common uh let's say stepping stone to building your application uh there as well um the documentation as I mentioned is getting rehold um so we're going to simplify a lot of great things especially being GA now so as you seen it was very easy for us to create an app with you know two or three different steps uh we'll have that reflected uh there's also a demonstration Again, we'll provide these uh slides for your for your user as well as the GitHub links as well.
But yeah, that's available for you to consume. And I think a lot of things as well um again we've covered a great deal in a short space of time. But if there's things which you feel you want to dive a little bit deeper onto, then please reach out to your account team. Um, and if there's things that where, you know, Ivon and I can, you know, join you with the right stakeholders on your side, then, you know, they're able to kind of facilitate that and help accelerate some of the use cases that you're thinking about as well.
Awesome. I don't know about the surveys, but to summarize on this. So, thank you so much guys for attending our session. This is officially the ending part of it. When you will be out, you will get the complete your surveys request. If you liked what we did, please let us know, you know, or if you didn't, then please let us know, too. Um, thank you so much for your attention. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.