Enterprise Observability with ZeroBus and OTEL at Scale with Databricks

Summary

This video presents an end-to-end enterprise observability architecture using OpenTelemetry and ZeroBus to ingest telemetry streams—traces, metrics, and logs—into Delta Lake for scalable storage and long-term retention.
Joining telemetry data with business data in Databricks allows teams to correlate slow requests or errors with specific enterprise customers, unlocking insights that siloed proprietary tools cannot provide.
A custom Databricks App built on the Lakehouse delivers flame graph visualization, distributed trace exploration, and correlated metrics and logs in a unified interface, rivaling proprietary observability platforms at no extra licensing cost.

Enterprise Observability with ZeroBus and OTEL at Scale with Databricks

OpenTelemetry has become the industry standard for collecting traces, metrics, and logs from modern distributed systems. However, operating observability platforms at enterprise scale often requires proprietary tools and fragmented data pipelines. In this video, we present an end-to-end architecture for enterprise observability using OpenTelemetry and ZeroBus with Databricks as the analytics and visualization backbone. We will show how OpenTelemetry telemetry streams can be ingested through ZeroBus and stored in Delta Lake, enabling scalable processing and long-term retention of traces, metrics, and logs. We will demonstrate how to build a custom Databricks App to visualize flame graphs, explore distributed traces, and correlate metrics and logs in a unified interface - delivering capabilities similar to proprietary platforms but powered by the Lakehouse.
Talk By: Aleksandar Dragojevic, Staff Software Engineer, FDE, Databricks

Chapters

FAQs

What is ZeroBus and how does it work with OpenTelemetry?

ZeroBus is a routing layer that receives OpenTelemetry telemetry streams and directs them into Databricks Delta Lake for storage and analysis. This approach eliminates dependency on proprietary observability tools by using the Lakehouse as the analytics backbone.

How does storing telemetry in Delta Lake improve enterprise observability?

Storing OpenTelemetry data in Delta Lake allows teams to join observability signals with business data, enabling queries such as identifying which enterprise customers are generating slow requests. This correlation is typically impossible in isolated proprietary observability tools.

What does the custom Databricks App for observability provide?

The custom Databricks App delivers flame graph visualization, distributed trace exploration, and correlated metrics and log views in a unified interface. It matches capabilities typically found only in expensive proprietary platforms, built natively on the Lakehouse without additional licensing costs.

How does Genie integrate with the enterprise observability architecture?

Genie can be used to query telemetry data stored in Delta Lake and set up alerts based on observability metrics. The demo in this video shows sub-second query performance on seven days of road telemetry data, making real-time anomaly detection practical at enterprise scale.

Full transcript

[00:10] Good afternoon everyone and welcome. I hope you enjoyed your lunch and if you had a party last evening such as myself, I hope you are more awake than I am. So in that case hopefully get this is this is entertaining. So um welcome. I'm
[00:22] Alex. I am a staff software engineer as part of our field engineering organization. So FD specifically and today we'll be talking about enterprise obserability on data ricks. Yeah.
[00:35] So let's maybe start with a bit of history or where we are or how we got here. So you know observ observability is a very difficult story a very very long story right and usually it means I
[00:49] want to know what happens when it happens and why it happened and how it started here what I want to call a stone age almost three decades ago even more right so at that time we were usually
[01:02] you know just you keep running one server somewhere usually you know maybe in your basement or if you're lucky you're colloccated it's very simple it's very monolytic So you typically have just one or two apps running on the same
[01:14] physical hardware. That can be your like web server or it could be your database usually boat or mailer or something else. And you know the monitoring at that point was very simple like is the thing on? If it's on, cool, it works. If it's not, you know, someone has to go
[01:27] and push a button like usually at 3:00 a.m. it it happens and it happened a lot at the time. or you know at some point if you're really lucky you get an SSH so you don't have to drive to a customer site and go back after they told you
[01:40] three times it's already on but it wasn't right so you know at that point that that that was the observability and you have to actually debug something you go to a server like open the console like login tell the log that you need and figure out what's wrong and what's
[01:52] broken and usually something very mundane like disc going out or something like that so then we advance a bit you like early 2000s
[02:04] here you know things getting a bit more complex already. So you know at this point your software grows you have more customers. um you have to run more things at the same time and at this point you know you
[02:16] want to know like yeah obviously if the compute is on but also at some point like does it actually healthy like what's the CPU usage is it choking on requests like is your database full is
[02:28] the disks full so you know then you know we have graphs and if you lived in that era this should look familiar to you like this is Nagios and you know you register all your like different servers
[02:41] couldn't be one but Usually at this point it was like you have your web server, your database server, your backups, mailer whatnot you were running at the time but still usually on the physical hardware somewhere if uh that
[02:54] was the standard at the time. So a bit further things get a bit more complex at this point. So you know like
[03:06] these early 2008s let's say so that's about the time that also Facebook became a thing right and we were already getting to cloud environments at that time so you know you wouldn't typically just go and provision physical server
[03:18] even though you still do that but you maybe you rent a VPS and you run it different different things on it right um but at that point like you know your hardware works hardware is healthy u but then you get concerned about the
[03:30] applications that run on it so yeah you know maybe Today you have a thousand customers but a year ago you had 100. So you got to figure out like how can you get the most out of that hardware. So that usually
[03:43] means some sort of a profiling. Uh so getting into your application code figuring out what is slow, what stops the system or chokes everything. And here we come to something called APM or the application performance monitoring.
[03:57] So what that means is that you would go put some sort of like an agent in your I don't know Spring app or something like that and that would continuously profile and send the information back to somewhere. So you can go through it, you can check latencies, number of requests,
[04:11] you can figure out the error rates, why is it failing so you can keep bit more of a live monitoring of a system, how things evolve, right? So you you know you have a problem and that this kind of helps you zero in where the problem is.
[04:26] But you know, you see the trend here like the complexity seems to be growing and growing all the time. And this is still relatively simple like you know there's not that many moving parts. Everything is pretty much still fixed.
[04:38] You know you have maybe different services but they don't scale up and down and they don't need to handle like these various various loads from your from your services or your customers.
[04:50] So where we are today uh is the fun part. So at some point we figured out like you know the future of the commerce is on
[05:02] internet. The future of everything is very much on internet. So all of these services that you end up building you know e-commerce social networks uh platforms whatever you know it's it's out there and more and more people are
[05:15] using this every time. So now it's not really unheard of that you have, you know, tens of thousands of customers every hour buying things on your web store and at some point you know we figured out hey there's a bit of a
[05:27] complexity here and we want to manage that. So we came up with microservices for example now your big monolith that you had before is three different services. So for example, I don't know, you are building a web shop. So you have
[05:40] inventory to check things. There's order processing. There's notifications. There's many many different things going on. That used to be one application before, but now like every time someone wants to actually do a purchase of something, they would hit one API and
[05:54] that API hits another API and maybe it breaks somewhere along the way and you don't know why. And all the time it doesn't work or why is it slow? Your company's losing money effectively, right?
[06:06] So here we already see these new tools developing to help you with that. So I mean I don't know if you use like tiny trace or data dog these were like the tools that are still are the tools uh to
[06:18] deal with this like distributed tracing complexity right. So you can actually go and track your requests sent to end. So from the moment it hits the web server to the moment it goes to the database and then back obviously you know you can
[06:32] trace actually go and trace to the point when someone click the button in the UI what happened to the entire user flow how fast it is what's the latency what's the request throughput so pretty much everything right it works great still works great you also have some free
[06:45] tools I think like Jerger or whatever I think some graphana tools are also this today to help you with distributed tracing so it works nice. Um, it's also very expensive uh if you're at scale. I
[06:58] mean, if you're very small, it doesn't matter. But like in companies that I worked for, it's like 200 servers and 300 services running on that needs to sync this telemetry somewhere and it gets really expensive real fast. Um, so
[07:12] these tools were built but also at some point the open telemetry standard emerged at that point. So how it works so far is that we had uh different let's call it very vendor specific agents and
[07:25] tools to sync this telemetry. So you you know you would go configure it set up an agent with your service that agent ships your telemetry your logs your metrics your traces usually their own store and maybe you have one of them maybe you
[07:37] have more of them. Uh so basically open telemetry was emerged actually because it got uh made from two different projects. So it kind of emerged to try to streamline that. So you don't have to configure three different things. You
[07:50] can just configure one and have it yeah ship ship the data somewhere else. So um basically like all of these tools that we had I mean can be naggios can be
[08:02] something else can be graphana data dog datase. So all of them were built at a time where the problem was the problem needed to be solved. it was simpler or more complex and today it's it's a more complex and those tools are still still
[08:16] great. I mean we still use it every day every time. So here uh we're not going to talk necessarily about replacing them but more like supplementing them to see where it where it makes sense. Uh and one very important point is that none of
[08:29] these tools were really built for the world where your telemetry also sits alongside your business data and I will tell you why uh that's it's very important to look into. So basically to summarize it, the open telemetry gave us
[08:41] a freedom of collection but now we also need the freedom of destination because at the end of the day usually this telemetry data that you send it goes in a vacuum you just get the insights but if you want to keep it for three years
[08:53] that's going to be a very big bill at the end of the month. So here we are today and the question is like what if your observability data lived where your business data lives
[09:05] right? So usually today you are one of these two groups. You can be group A where you you know you're fairly advanced. You have running a big system. You're a great customer. You want run a web shop that serves a millions of
[09:18] people per day and you know your data is maybe on data bricks but your telemetry is at some vendor like XYZ and usually they don't meet and the problem becomes like if you want to ask thing like I had
[09:32] an incident last night you know it impacted my America region like what's the amount of revenue that I lost due to that incident and it's can be very difficult to answer that question
[09:44] depending on how things are laid out. So today, you know, it's it's not a hard question to ask, but the problem is like that data lives in two different places and it's very hard to join them. Or if you're in group B, u you're not that
[09:58] advanced, you might have some logs somewhere shipping and usually when something breaks and someone sends you a Slack message, you go and have a look at that or you have some dashboards here and there that someone also has a look into. uh probably not maintained. Uh but
[10:13] you know like the same applies right depending on which stage you are you know you probably the more data you get in more data retained as your company grows as you grow it kind of makes sense to keep everything close together.
[10:26] So how does this actually look in practice as this is like usual story that you have today. So you know I mean depending again depending on the size of your company you have your infrastructure you have applications cloud containers AI stuff whatnot and
[10:40] usually that's ends up in one or more places and usually there's a significant overlap between that so effectively it's a fan out you know your logs might end up in one vendor or another vendor or
[10:52] your S3 buckets uh your metrics usually end up in graphana or somewhere else and you know like it's no longer just one tool it's like four or five different tools that you have to coordinate access
[11:04] to actually figure out what's going on at any given time. Unless you're very lucky and you can can afford this really good stuff with a big bill that comes at the end of the month, then yeah, maybe it's usually everything is in one place. But from my experience, it wasn't as
[11:17] simple as that usually. And you know this is very organic uh architecture. So usually you start small, you start with very simple dashboards and then over time you build on top of that and these
[11:29] patterns kind of emerge over time, you know. So as more of data you collect, you try to do things better, but you end up with four or five different tools that kind of overlap. So the problem here
[11:44] is let's maybe focus on once you have your data already and currently as I said they're very siloed and you can't really get the data out of them if you wanted
[11:56] to ship somewhere else. Maybe for some of them yes but usually it's very proprietary. So who are your users for observability for your telemetry? So you know usually for the guys so for example for SRE or on call they're usually
[12:09] interested in like hey I want to keep things running it needs to be healthy it needs to be fast it doesn't need to break and for that you know I need some metrics for the uptime for the request rates for the latencies maybe some logs
[12:21] or you know your typical developer they're usually concerned about the apps so you know they track their own app telemetry they look at the logs when something breaks again they can that can be another thing that can be like usually in open search or somewhere
[12:34] else, right? Or you know platform security teams that take care of the logs but not really the logs that you take care of. They usually want like the security access logs etc. Again third tool now the problem here becomes you
[12:48] know there's in order to actually figure out where the issue is like you have to go three levels deep like if I'm application de developer and I see my service misbehaving it's not always obvious where the issue is. I mean I
[13:00] could see requests failing but maybe the reason for the request failing is a database issue that's like three layers above and I don't see why. Second problem is that one I mentioned
[13:12] before is that there is no connection to your business data at all. So for example, you already run a CRM somewhere like within your company usually with data bricks right so there are some tables about the users your financial data purchases orders your typical
[13:24] analytics if you're very advanced and for example you run a commerce platform you have experiments you know you can put that also like your typical revenue stuff so anything that is operationally important
[13:36] um and like this today like these two are completely separate I mean unless you go and create very specific pipes to move one to the other or back which is what usually happens. It's very difficult to connect them together.
[13:50] So now the question is like is this really how we want things to work and the answer would be usually no if my clicker would work. Okay. So um so let's maybe try to
[14:04] simplify things a bit uh or just paint a picture how things could work if we were in ideal world. So you have still have the same sources and like in observability there's like what we call
[14:16] these three pillars of observability that would be your metrics traces and logs. So metrics is your you know number data could be requests per second could be amount of requests latencies could
[14:29] also be your business uh information like you know the number of orders number of stock in inventory it can can be anything right. Um logs are just typical text logs. Sometimes structured,
[14:41] sometimes unstructured, can be errors printed. It can be debug logs, info logs. So that usually goes somewhere, helps you diagnose things when the things go wrong. So usually not something you discard, but you have to put it somewhere. And usually maybe you
[14:53] don't retain it for very long, but it needs to be there. And the last one is traces. And this is fairly I would say new, usually related to the APM world. So performance monitoring. So the traces help you understand how things move through the system, what components are
[15:06] touched. So if I go and make a request to an API and it goes that request goes to hits the database and gives you a response back I will have a trace that says for each of those components this is what happened this was the latency
[15:19] this was the result of it and you store those and you can actually usually the use case here is to figure out uh if things going slow why they're slow and you can for example see the slice that
[15:32] says your database requests are now like taking two seconds so it helps Once you figure out like for specific requests what happens. So the story here is still the same. Um so how do you move this is
[15:46] we can leverage the open telemetry. So you can connect source the data that you already have with the old user. It's an agent that you installed somewhere either alongside your application or you know as part of the application that you
[15:58] deploy. And now if you want to make it work on data bricks there is a zerobus. Uh so if you don't know what zerobus is datab bricks way of
[16:11] basically being the ingestion sync for any kind of data. So it supports like you can use it to write from kafka but in our case it also supports open telemetry as a format. So how it works is basically you would just go and point
[16:25] your open telemetry collector to data bricks and just sync the data directly into delta tables. So at that point once you have that working what you get is kind of this picture. So you have your
[16:37] business data on the top but you can also directly join your observability data at the same time because at the end of the day they just live in the same delta tables in the same catalog in the same environment. So everything just
[16:49] gets flushed into into one let's call it bit more uniform system and obviously this doesn't mean that you have to rip out and replace everything like the good thing about open telemetry is it's very
[17:01] routable right you don't need any tools you can just go to your configuration for your agent and tell it hey so far I'm pushing things to graphana hey let's also push it to data bricks it's a configuration change right it's there's nothing for you to instrument anymore
[17:14] you just point it to a second place and it just gets written written to So how this query would actually look like? So so I mentioned like we can we can
[17:26] join the business data with operational data. So let's say there's a you know someone asks me like uh for any purchase requests for my enterprise tier customers in the last
[17:38] five minutes show me the ones that have like requests that last over two seconds which usually leads to the bounce in users. Right? So they see the page loads, nothing loads, they go back. Right? So now this query becomes very
[17:51] simple, right? You know, you just take your sales data from your account, you join it with the telemetry information, you just filter on some attributes and in this case it would be your customer ID and you're basically done. You know, you just pull out all the requests and
[18:04] you can maybe feed that into some sort of a system downstream to, you know, run this query as an alert, right? And every few minutes new list pops up and if something goes wrong you use that for alerting for anomaly detection and any
[18:18] time you actually need to try to understand what's going on in the system and how it's actually any kind of incident impacts your operations or your business now it becomes very simple query you don't have to shuffle data anymore between systems and you know
[18:31] just derive your goal tables move them to observability do the alerting there not you actually do it in in data directly
[18:44] So what does this architecture unlock? So first one yeah you seen the joints between your business and telemetry which is fine you know we can ask many many different questions and now you have you know you can see can come and ask like for a daily report of the impacts per region per incident if you
[18:58] have them you can pull that out easily it doesn't take two weeks to build so it's very easy and you know those drive your operational insights you can also feed somewhere else or fairly new thing
[19:11] on the let's call it on the market these days it's AI is you know you already have the data that's really accessible to agents into the same structure. So you can build an agent or have someone build it for you that just goes through
[19:23] your, you know, observability data, scans it, checks the latest one, looks for anomalies. It maybe even fixes it if you give it access to that. So you know, you don't have to pay for another tool to do that. You can just build your own.
[19:36] I wouldn't say simply, but you know, you can get started. It's very very hard field. And if you already use some sort of a AI or ML models for example very very
[19:48] common in like business and sales and retail that drives your decision- making like uh what to recommend the recommended agent customers you can feed your observability data there. So you know like maybe for your customer is in
[20:00] South America like if they see the latency going up and it takes a while to load pictures hey maybe now we have that signal to feed into our training so maybe we recommend them something else right so it's kind of supplements your
[20:14] existing picture and the best thing is like so far I mean you would have asked me like okay but it's in delta table like that takes a while you know like if it's usually you know observability is like terabytes
[20:27] data and if you want to scan for last month it takes a while to do that. Usually we don't usually take last 24 hours and aggregate in distinct buckets but it used to be like it's still normal delta tables and up until two days ago
[20:40] these queries would still work but maybe they take 30 seconds maybe they take a few minutes but now we have Raiden as well or realtime rake house uh which makes things a bit snappier right so
[20:52] this your AI agents doesn't have to wait 30 seconds for reply you can just spin 10 of them you just hit this end point every half a second or something, get the results back and do something with that information. So, and it it scales
[21:05] quite well, right? So, when we were testing this uh specific case, I think it was like few terabytes of data and like hundreds of requests per per second just running uh
[21:18] running these queries and getting the results back. Usually just because dashboard and alerting that was the use case, but fairly fairly good implementation. So how building this would look like? So
[21:30] you know we seen the architecture. So you probably if you haven't used open telemetry uh this also might actually look simple because it is simple but if you already use it you probably already know uh where and how it works.
[21:50] So oops. Okay. Yeah. So typically I pulled two examples here. So one of them for JVM, the other one is for Python apps. So a lot of the open telemetry works through agents. It's an agent, not the
[22:03] AI agent, but the agent is a thing you inject somewhere to look through your application. So in Java, that's basically just running your service with the agent attached or in Python, it's very similar. You just run it through a wrapper and the instrumentation works
[22:16] out of the box actually. So usually you don't have to configure anything. Uh in case of Java, like if you use Spring, you know, it will instrument your uh HTTP stack. If you instrument JDBC stack or JPI if you have it you can add custom
[22:29] stuff as well the same for Python. So it does work out of the box and provided everything is configured right you should just see telemetry flowing and you can also add your own like custom metrics custom traces or logging everything gets picked up out of the box
[22:41] for majority of the software and stacks that that work today. So then you go need to configure this collector and it's again fairly simple u you just point it to a zerobus endpoint.
[22:55] So you give it a workspace ID and the region in case of like wherever your data sits give it authentication and the only actual custom thing here is the zerobust table name and that should point to a table in your unity catalog
[23:09] where you want the data to sit right. So no more than that. There is a few steps before this. So if you actually go and look through the official docs, it will tell you how to create these tables because they need a like very specific structure and partitioning. But besides
[23:22] that, once that is done, you can just put the data in and it it's usually it's just append only tables anyway. So it's very fast and once everything starts flowing the your collector will just natively write
[23:34] through zero bus to data bricks. It usually takes a few seconds to land the events into tables before they're queriable. Uh but after that you can just go and since you have the delta tables you know you can just write your
[23:46] own queries point agent to it write uh alerts rules automations and you can build on top of that whatever whatever you want so that whatever you want can be many different
[23:58] things these days it's mostly agents but if you want to build some custom UI to pull this I will show that to you as part of the demo later or can be some custom tools you know just pull off some statistics errors or you know just feed
[24:12] it to your ML pipeline and take that for for training. So the important part is like since open telemetry is very open standard nothing really changes if you already use it. So you just repoint it somewhere else. Um if you don't have it
[24:26] it's fairly easy to configure. Uh but the only thing that you change is configuration right. So the good thing about zero bus is that there's usually no double hop right. So
[24:39] you can just go and point it directly. Usually when it comes to telemetry depending on how you let's say architect things sometimes it goes to public endpoint depending on what the what the customer is or sometimes maybe there's
[24:52] also Kafka in between that syncs the telemetry somewhere else. You don't have to do that. You can just point it to an endpoint and just happily accept it and sync it for you in the in the table. So it's fairly fairly simple, right?
[25:04] So let's see how that works in practice a bit. Right. So I I will show you a small application. It's a demo application, right? So there will be like a cookie
[25:17] shop simulation like there will be some virtual orders of cookies being being made. So you know uh customers will be purchasing them. Let's see. Yeah. So these virtual
[25:29] customers are trying to make orders for cookies, right? So it's an application that I built just just for the sake of this demo. It's running. It's a spring boot back end and there's just being simulated orders, right? So if you see like on the top like you can go through
[25:43] and just explore the logs you see being emitted. If I scroll down a bit just finding one that looks interesting and here I found one that says insufficient stock for a specific cookie. Right? So I can go and click it. And if I click it,
[25:56] it actually gives me the entire pipeline or the entire path. And this should be very familiar if you ever use any sort of APM. Just gives you the entire request end to end uh with all the logs,
[26:08] all the metrics, gives you the information, gives you the queries, gives you the HTTP stack, pretty much everything that you need. And you're probably seeing something like this. Or you can also just go and navigate to the logs related to that specific request
[26:21] already, you know, because in the case of the open telemetry, everything is kind of related by this trace ID concept. So anything that happens if it shares the trace ID it's part of the same effectively same trace of events
[26:34] that happened. So you know we can just go and explore quick two things and this is actually working directly on dataix right is not using any third party tools uh just sourcing it. The app itself is custom and we will open source it if you want to use it after just contributing
[26:47] it further. Uh but it is just running on raid and it's just delta tables. So nothing nothing really custom besides the UI here. And I think over time you know things things will improve uh as as
[27:00] we add a few things but you know in a few weeks we'll probably be able to see this in data brick labs and just install it and run it as on data bricks as well just to play things out. But I did mention agents as well.
[27:13] So, you know, we have Genie, right? And sometimes, I mean, I know I like clicking through the UI. Some people don't. Some people like to chat. So, let's ask Genie to actually do something with this data. And I will ask it to go
[27:26] through the logs that we ingested and plot out of stock errors. And here I'm actually intentionally using logs and not metrics because metrics we already have. I'm syncing the amount of cookies that we have and the inventory. But I
[27:39] wanted to make it a bit harder to actually figure out uh what's going on and have agent try to figure out cuz I didn't prompt it. Hey, this is the specific field you have to look. It's actually needs to go and look for it and
[27:52] to figure out what's the my logging structure, how the logs look like and how this pattern actually emerges. Obviously, in the real case, you you would do that, but for the sake of the demo, I wanted to make it a bit bit more challenging. So, it's going to look
[28:05] through the logs and it says it's only has actually 24 minutes. So, logs because at the time I recorded this demo, I just run the app for to get the data in. So, it's going to go and figure out and put some data for us at the
[28:20] chart soon. And let's give it a moment. Yeah. So basically it's going to summarize
[28:33] everything and it says it found a pattern that says the log line says insufficient stock for cookie of type X that we have the user asked for x amount of cookies but we have zero in inventory
[28:45] right and it will happily also plot us all the errors as as they appear right in the uh on the chart right so theory how it looks like so once the recording
[28:57] scrolls a bit back so yeah basically says these arrows follow the specific patterns and it will pull out from the logs directly like these are the air counts for very specifics and I didn't have to do anything here I mean this is just based on the telemetry from the
[29:09] application itself that's coming so it's not even something that configures just goes through the logs and figures out what needs to be done and obviously you can run this on schedule right it's not
[29:21] something that you can type in the genie and you can actually go and ask it to create alert for this very specific query and now since it's powered by Raiden you can run this query in a few hundred milliseconds or even shorter. And if there are results, you can just
[29:33] power that to pop some alerting or somewhere like push this as a signal for something else or some other consumer.
[29:49] and this never works with recordings. So, where does this actually leave you or what does this save you? So, first one, it's uh obviously money, right? So the thing is like we have all these very very good tools for observability and
[30:01] they're they're very nice uh but usually if you want to retain your data for more than like two weeks it gets very expensive right not always but usually right that's the case. So for your
[30:13] immediate operational needs, they're civily good. But if you want to have like a long-term analysis, you still need to sync that data somewhere else. And this is makes this makes it very simple to do that. Even if you don't leverage like Genie or any kind of
[30:25] custom UIs, if you just want to have data somewhere like in data bricks for long-term analysis, you can just do that. It doesn't really cost you anything besides the storage, which is I promise you much cheaper on S3 than you would pay someone else. And the second
[30:38] one obviously is the join that we've seen, right? It's very easy to join your business data to your operational data in case you need to build some sort of operational reports how it impacts your business. And you know the third one you
[30:51] know today you know this tool is popular tomorrow some other tool is popular. If you use open telemetry you can just repoint it at some point in the future to something else wherever you need to be again it's configuration change and your telemetry will look exactly the
[31:03] same you know just keeps things a bit more flexible. So with that I would say just few things to remember. Uh so your telemetry is just data at the end of the day. It's a very specialized data but it's still a
[31:17] thing that lives on a disk somewhere. So why wouldn't you try to put it as close to all your other data if you can? uh you know like if you are in a very big retail company like keeping these
[31:29] metrics uh means a lot when you try to improve the business over time because you need to know what happened and what will happen and shouldn't lose it right and the migration if you're using open telem
[31:42] today you know just repoint it to somewhere else uh like configure data bricks tables that you need to configure point it there and it will land data you can please you know play around explore see how it works for you it should be fairly performant you uh just syncs it
[31:55] should be able to sync like gigabytes of data easily. No, no issues even if you run like very big systems. And you know the last one is the operational speed. Uh with the Raiden being uh announced and introduced two days ago, these
[32:08] queries become very fast. Even with a large amount of data, you know, like this usual delta or quering on delta was very slow. Now that issue basically doesn't exist anymore unless you have
[32:20] very very let's call it very complex joints which in this case is not the case. uh you know you can just use this to build any kind of dashboard and it will just uh pop everything off very fast even you have like if you scan last
[32:33] seven days of road telemetry without obligation it will just be be very snappy and with that uh we are at the end of the talk so if you have any questions we will be taking them either by the podium or outside as time permits
[32:48] uh enjoy rest of your day and I hope this was uh very useful thank

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.