Managing Databricks at Scale

Summary

• Scalable Data Management: T-Mobile manages hundreds of terabytes of daily data from 5G networks using Databricks' scalable architecture with Delta Lake and auto-scaling clusters.
• Optimized Performance: Performance optimization through data partitioning and caching ensures efficient query execution and cost management across massive data volumes.
• Unified Governance: Comprehensive data governance with Unity Catalog empowers T-Mobile to innovate and streamline operations while maximizing data impact on the Databricks Data and AI platform.

Managing Databricks at Scale

Talk by Vikas Ranjan, Senior Manager, Network Data & AI, T-Mobile
T-Mobile’s leadership in 5G innovation and its rapid growth in the fixed wireless business have led to an exponential increase in data, reaching 100s of terabytes daily. this video explores how T-Mobile uses Databricks to manage this data efficiently, focusing on scalable architecture with Delta Lake, auto-scaling clusters, performance optimization through data partitioning and caching and comprehensive data governance with Unity Catalog. Additionally, it covers cost management, collaborative tools and AI-driven productivity tools, highlighting how these strategies empower T-Mobile to innovate, streamline operations and maximize data impact across network optimization, supporting the community, energy management and more.
Here’s more to explore:
Unified and open governance for data and AI: https://www.databricks.com/product/unity-catalog
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does Databricks handle 100s of terabytes daily?

T-Mobile uses scalable Databricks architecture with Delta Lake, auto-scaling clusters and performance optimization through partitioning and caching.

What are the benefits of auto-scaling clusters?

Auto-scaling clusters automatically adjust compute resources based on demand, optimizing performance and costs for variable workloads.

How does Unity Catalog improve data governance?

Unity Catalog provides comprehensive data governance including access control, data classification and lineage tracking across all assets.

What are T-Mobile's primary use cases?

T-Mobile applies this platform for network optimization, energy management, community support and data-driven productivity across the organization.

Full transcript

All right, good afternoon everyone and welcome to the datab bricks data and AI summit. Uh this has been my fourth time attending this summit and every year I've been fascinated and blown by about the innovation data bricks team is bringing to the customers like us and taking the bar higher and higher. I do want to say thank you first of all to you guys in the beginning not at the end because I know it's a lunch time and it's really really hard to come to a session when you are hungry and you know thinking about all the fancy foods you're going to go with your partners and ecosystem so the lunch is still on them not on us you can still enjoy the lunch afterwards hopefully good and better but thank you so much again for coming and joining today so today I'm going to talk about managing data bricks at scale uh a small introduction about myself. uh from T-Mobile and in the sessions today I'm going to talk about our data journey at T-Mobile for the past many many years.
A little bit of introduction about myself. This picture is actually AI generated if you notice that's not who I am but uh I'm part of uh network data and a organization in T-Mobile. uh current role as senior manager fortunately and unfortunately I've been working in data space for 19 years and I feel like boy this is crazy work man not easy thing to do um in this 19 years uh I'm responsible for managing a team of platform engineers data engineers data scientists and app developers this can cannot be a harder job than this when you have all kind of smart crazy people working with your team and challenging you every Today I'm very very passionate about how can we make the use of data for good.
Not just uh using data to have a bottom line or topline impact but how can we make data useful for people. How can we bring a positive impact to with the data to the society outside of work I come from Seattle a beautiful Pacific Northwest. If you guys have not visited this is the only time to visit Seattle. beautiful, sunny, not gloomy, dark and rainy and windy like San Francisco. That's the place you guys should visit.
Now, um I have a very uh intriguing son, 12-year-old son who actually wants to change the world, which I've been trying for 100 years in my 45 year age. So we have a lot of interesting discussion about how can we change the world make it better beautiful awesome with and without data and he's like a big time Wikipedia guy and then he uses uh citation links to make sure Wikipedia has not helping and then we discuss that to make sure you know it's all good.
So that's a little bit about myself. We at T-Mobile are passionate about only one thing. We are passionate. We are working hard all the times to keep our customers in the center of everything we do. That's our ethos and our mission is to be the best at word in connecting our customers to their world. Everybody does that. What's so special? What's so different? The way we are doing that is in the words of our uh CEO, we are thriving ourselves to become a deeply data- enabled AIdriven digital first company. not just bring the data together but bring the make it digital, make it real and then use AI on the top of it uh to provide the best experience to our customers.
How do we maximize the value? What is the day in the life for our team here? Since we are part of the network organization, we have a philosophy and a vision of uh connected data and connected intelligence. What does that really means to us? What do we do every day at work when I'm not talking here and trying to, you know, share some of the things we do at work here? We are a network company first. Our focus is to build the best network.
I hope all of you guys are using the best network we have in the country. If not, you can reach out to some of our executives here and they can provide you some really, really good plans, great plans and correct equity afterwards. But when we talk about building the best network, we built the best 5G network, the first in the country. Over the last many years, we launched some really, really innovative services. Recently, we were the first organization in the country to launch a dedicated network for you know the early responders, firefighters.
We launched slices that enable many many businesses not just telos across to provide their services in the best network. So bring the ecosystem together on the best network. You can have the best network but what does a customer needs? A customer needs only one thing best experience. How do you make the experience the best? We look at every single signal on the network of our customers. We have 140 million subscribers and more after this session and we built some very robust, very scalable, very customer experience course looking at what is the experience of a customer on network.
It's not an easy thing to do when you look at 140 million subscribers who are making tons of calls leveraging tons of different applications on our network generating tons and tons of data for us. They are always moving. You know, we are here today. We are going to be back to Seattle tomorrow. Some of you basically go back. So, we bring in this notion of what we call as a uh customer-driven coverage. We provide the customer-driven coverage where we're building the new tower with the best experience by looking at the customer experience on where they are at what time and where.
On the top of it, we launch the home internet services. We were the first one to launch that in tumo to provide the experience not just to connecting and making the calls but also basically able people are able to use the home internet services all the times. The area I'm so passionate about as I said earlier is data for good. You know, I feel like we are very privileged to live in this world where many of us don't have to think about, you know, going to a shop and buying a expensive dress or going to a really good place to eat a fine dining.
Not everybody can afford that. Not everybody has that, you know, exclusive things they can afford or think of in their life. So we have built tons of solutions on where we provide a connectivity to all especially to the underserved people especially to the people where now connectivity of phone has become the fourth essential ingredients of life right people can first thing I do in the morning is I want to check my phone right then before I do everything else but not everybody can do that so you need to give those I call a basic privileges of people right so that they can you know reach out to their family talk to people do everything Recently uh we were the first one to launch u a satellite service and I think we took the we provided our customers through the really really bad time through the LA wildfire.
We were the first one not to provide this service just to our own customers but anybody who was impacted there to me that is the real impact we bring to the society. To me everything else we're talking about here is nothing. If we are not able to help the people sitting in this room, the people who are trying to reach out to somebody else, the people who are struck somebody else, that means nothing to us. Humanity first, data second, right?
We work with our law enforcement or legal agencies, providing tens of uh thousands of API calls throughout the day on a person went hiking, not able to reach their family, went missing. For me, there's nothing better than having somebody meet with a person who has gone missing, right? When you have the movement of bring those people together, I feel like the life is done. That's the purpose of life. And then last but not the least, we are living in the world of AI.
What is the most essential commodity or what is the most sought after commodity in the world of AI is energy, right? Uh I've been reading I think we are all reading tons and tons of articles from Samiteman. There was a one I was reading last night where we were saying, "Hey, we're going to optimize energy more. is going to be 120% cheaper than what it was last year and further having that energy. There's only one mother god we have which is earth right we are privileged to be born here live here we have built solutions in house we are collecting tons and tons of uh data from our IoT devices looking at all our data centers across the country around 80 plus data centers looking at 15 minute telemetry millions of data points to see how can we not only optimize our capital efficiencies but also make sure that we are not overprovisioning and underprovisioning our resources which in turn is ensuring that we are using energy to the best.
That is what we do in a day of our life. Right? And there there are many many more things we do to make these things really happen. And I can't be thankful enough to my team, my organization, my leadership here for us to bring these opportunities to make the real changes in the with what we do. Let's look a little bit into our data and AI platform journey. This journey has taken us a lot of time. If somebody says that you did this in two years, I bet you they're lying.
Unless they are the smartest and the greatest in the world, which are very few, right? The journey for us has been almost a decade of a journey. I still remember the days like many many of us who come from the you know age more than uh 20 there are very few in the room here. Uh we have been in the world of databases. We went through data warehouses. Many of us went from data warehouse is to a big data a notion in early 2010 2013.
This is how we started our journey of what we called as building a hadu platform which I'm pretty sure many of you guys probably have have experience to work with and deal with. At that time for us two terabyte was a big number. I I was like man this is two type of data. How in the hell I'm going to really process two ter data? It was a challenge. Hadoop was painful. It was not easy. And then guess what? Thanks to all of our customers who have been there for us for so long, they're like, "Man, I don't want to go to the web search.
I want to go to my phone and buy something. I want to browse something. I want to do something. Good for you guys. Tough for me. It keep adding my data exponentially." In within one year, freaking I got like a 10x more data to deal with. And then my boss is saying, ", you always come back and ask me like $5 million more to buy something new. Why the hell you cannot make this run better and cheaper? It's just some 10x more data. it's only 20 terabytas, you can do it.
I'm like, yes, boss. During that time, we went from a big data processing to more of building insights with the traditional reporting layer, which many of us still use every day, to more of a statistical analysis. Over the next two years, I don't know how this word has really happened to us, man. Sometime I feel like that this numbers doesn't make any sense. You're going from 20 terabytes to 11 freaking terabytes. Like I have to still look at my calculator to see how many users are there in 20 terabytes.
And then on the top of it in 2019 around that time 2016 onwards everybody was talking about AWS. Come on this data is nothing because you can throw everything into AWS it will magically do everything right. You will have this data in one place you have analytics in one place. Apply machine learning you know all the buzzwords you know in 2017 15 onwards everything is doable. Put the data it's very simple very easy. Absolutely.
Yes sir. And we tried that we went into a journey of cloud modernization not a lift and shift but basically kind of green fielding or brownfielding your architecture with only 200 terabytes of data. In between our customers continue to grow right and then it continued to become more and more hightech for us and towards the early 2020 mid 2020 we had one of the biggest merger in the telos and uh we got T-Mobile and Sprint merged together everybody was celebrating we were super happy we were like we're going to give our customers 10x more value you know we're going to make their life awesome good better and then I'm sitting in my room I'm going to get data for 35 million subscribers how I'm freaking going to handle this data when I'm already having struggle to handle like 200 terabytes of data that time thank you it become 400 terabytes in no time like this terabytes keep moving was like a challenge for us and then the top of it everybody wanted to know oh you got this data now you have T-Mobile customers now we have Sprint customers we need to merge our network we need to look at the pre and post experience of our customers who were on the print network before now they're on the T-Mobile network god save me I was thinking that and then it's like oh we have this print platforms you know which we don't have to use it again.
We want to do a platform optimization, cashization, you know, all kind of, you know, the big words we heard from our leadership, cost optimization, value optimization. And then I'm like, yes boss, yes, boss. And Vicas, you have six months. Thank you so much. Here you go again. It took six months, took us actually one year and a half, which was still awesome for us. And then we were thinking, we are stable guys. Now we can process 400 terabyt to 600 terabytes of data.
And guess what happened in 2024, 2023? All of a sudden, everybody started hearing about this word called tragy. I'm like, "Holy cow, man. I'm like getting older." You know, I need a technology that can be there for 20 years, not every five years. Now, leadership come back and again, we have a wide variety of unstructured data, right? We have been traditionally looking at structured data to look at customer experience.
Now, we need to look at the whole life cycle. Bringing the structured and structured data together. And then we started leveraging or started playing with OpenAI models on Azure. We did some work with data bricks on leveraging some of their models they were providing with LMA 3 and other at that time. And then we started building this new word of AIdriven or genriven for the lack of better words analytical platform with only 600 terabytes of data.
And then people are like oh it's very easy because I want to know my experience. Put my phone number there. You have my consent. You have privacy approvals. Put my phone number there. Put a simple query in intelligent bot which everybody has used with the rack solutions. I want to know everything I did in the last three weeks on the network. I'm like yes sir. You made 50,000 calls but I have to look at 500 billion records.
You made you use five apps but I have to scan one ter5 terabyte of a compressed data set. And on the top of it, you want me to tell you recommendations also. So that was 2024 and then now we are in 2025 and we are processing on an average of around 700 terabytes of raw network logs on a daily basis. And guess what? Now everybody's talking about agents. You know we were joking earlier this morning. Maybe next time we do a agentic presentation and demo where we talk in one of our native languages and we will present do everything real time translation for different people who are sitting in this audience.
So the sky's is the limit right for all of us. Very fascinating time very exciting time at the same time this journey has lots and lots of interesting opportunities and pain point challenges. This is a a lay of the land of our current data ecosystem. Um today I used to be very proud of this number few years ago. 80 plus pabyte of network logs. Today I'm actually look at that as opportunity not as a proud movement. It's like how can I make it better?
How can we make it more streamlined? We run our ecosystem as a hybrid or a multi cloud ecosystem. you know uh we run a lot of analytics on Hadoop tons of tons of our work we do on uh data bricks using the lakehouse architecture we do some reporting analytics on snowflake on the top of it we have built some custom solutions on on kubernetes because not everything can fit into everything so you need to look at the best architecture for everything else internally we are supporting around 5,000 users across our network uh organization not just the network engineers and then uh we try our best to keep is ecosystem secured and governed.
Um on the right hand side as you can see on the chart here pretty much for us everything is network. We look at a customer from a network view and we look at a network from a customer view and it's a very interesting challenge to solve like when you're looking at the same data two different ways for a general person all you need is a handset device and make a call or do something. For us there is a call translation happening between the CDRs.
It looks at the RAM network. It then looks at the device. Do you have the right device? There's tons of things you have to do behind the scene to make the seamless experience to your customers. Everything is primarily a combination of from a storage layer is a combination of uh ADLS genufs as a story layer. Um and then we have what we call as a domain driven sort of a connected architecture or data mish strategy where we use our bronze and silver following the architecture medillion on our uh Azure data bricks and then we write some of our code layer into snowflake and on the top of it uh we have built a open source data catalog using data hub which gels very well with both snowflake catalog as well as unity catalog and then tons of different machine learning components on the top of it.
It is so easy. I think we should do we should go and run. I mean I'm like dude we go from 20 terabytes to like 70 terabytes. What's the problem? There's no problem. Unfortunately, there are lots of challenges, right? Um, I think all of you guys, I hope you guys are all data practitioners or look at data one way or another. These challenges are not just limited to us. When we look at a data ecosystem, a data estate or data platforms, some of these challenges are common to everybody.
The first and foremost is you've got some data, you need to process the data. And I'm going to double click on each of these in the subsequent slides. The second challenge is always governance. Negative ecosystems have become so critical, so important. The organizations are focused more on data now than ever before, you need to have very robust governance in place. Data reliability, if there is anybody in this room says they don't agree with data reliability, I would love to learn more how we can be not be reliable and then still provide analytics.
Uh cost, we always want to do more with less and more and more and less and less. How do we really make it happen? And last but not the skill. The skills are changing every day. How do you keep your team always on top of the latest technologies to build new solutions? The first challenge for us has been uh what we call as how to process and challenge data at scale. When we talk about a network data sets, these data sets are fairly different in nature.
There's a lot of unstructured data that comes in very telco proprietary formats. uh there is not out of box connectors or solution available to provide the data there comes in um different velocity and velocity of the data for us as I talked about earlier we're processing almost around 700 terabytes of raw data per day the data is not really structured streaming or event-driven architecture network logs are coming from the telos providers like Nokia words and they still provide us data in files and these files are unfortunately not five files a day or 10 files a These files are unfortunately only 30 40 million files coming every minute sometimes even low latency to us and then some of our data sets we have processing every day we're talking about one of the data sets around 1.
5 trillion records per day and to make it even more fun that data actually comes in a file format there's not a even data coming to a cafka how are we solving these problems today with data bricks as you can see this is a very common architecture uh you know you have a batch processing and you have a streaming processing. So we are leveraging database both for patterns as well as batch and streaming. Uh we have tons of data coming throughout the day.
In some cases we have a file being generated every minute. In other cases we have 50,000 files coming in every 15 minutes. In other cases we have around two million records coming per second. That's a fairly large volume of data that we have to deal with. So the way we build our solutions were we definitely leverage um in the beginning we started building all our solutions with the data bricks for batch processing where we're processing data within 15 minutes to all the way to a one day.
Somebody says everything is real time. That is a real word that doesn't exist today, right? It's about right time intelligence, not real time intelligence. And at the top of it, uh over the last year and a half, we built Spark structure streaming pipelines with ADLS and lot of other ecosystem components to process around 100 200 billion uh network locks per day. The second challenge I think we can all talk about is how do you make all this work and how do you manage this?
How do you really have a robust governance in place? Some of you guys who have been dealing in the data bricks or any data platforms from a platform side perspective, I'm sure you can all relate to that. Managing platform in today's world is very very cumbersome and painful. Everybody wants everything now. They want all permissions. They want all access to every data sets and they want to have all the latest and the greatest tools like yesterday and they're like why are you worried about?
I'm going to pay you the bill, right? Do it. you guys are too slow for a platform team. They're like, Vakas, I need your help to prioritize, man. Everybody wants it everything now. I can only do that much. I only have two hands and two eyes and two legs and whatever. You know, I can't do it. And then on top of it, everybody's creating their own workspaces in data bricks. You know, everybody wants to have like I said, right?
I want everything there, latest and greatest. When it comes to troubleshooting, it becomes so painful and so cumbersome because not everybody's a smart data engineer or a smart power user. Then they are knocking back at the door of the database platform ads and saying, "Hey, I need your help, man. You are my best friend today. So, I need everything to fix." You know, we are living in the word of uh the too little and too less and too more word.
What is the right balance? What is the right permissions? What's the right data do we need to provide to people? There's a lot more concerned about privacy now than ever before. We want to make sure people have right permissions, right controls and checks in place. And last but not the least, I have been a victim of this many many times myself. I got a new use case. I want to do something EDA. I created a notebook. After one month, I forgot the notebook.
I got another use case. I created another notebook. And then I one day we went back and there are like 50,000 notebooks running all over the place. Some of those are not been touched, not been used. Some of those jobs running that are ever been consumed again. It's just become a mess to manage. How did we solve this problem? In the older days, as you can see, many of you guys who are familiar with the notion of data mesh or what we call as a connected data.
People will come from finance or marketing and say, I want to run a campaign. I needed network data. They'll come from care and say customer is complaining and they're saying we have a really bad network. Unfortunately, it's not. It's a device issue. Anyways, they wanted data and what we were all doing, we have our data engineers creating reports every day, scanning like a terabyte or a multi- terabyte tables and then creating extracts and sending the extracts to down some systems and then it was like oh you're delayed there SLS miss and all those things.
So it was very very difficult for us to manage these platforms. You know there was a lot of data duplication which in turn lot of data wastage and the cost wastage right people were always chasing the data than actually looking at solving the problem. What we really did was we leveraged unity catalog as the glue to bring a a holistic cohesive data governance platform. And if you guys are not familiar with Unity catalog, I would say in the last six years of our journey on database, that is probably one of the best thing, the best technology, the best tool I have used because I have a very small nimble team of three platform engineers.
Right now, if I have to give access to people, the rules are very well defined, the policies are defined, the governance is put in place, you have alerting in place. So, it just makes your life so difficult. I mean now my engineers are telling me because can we do more work because there's like you know 50 people not pinging me every day to give me access you know how can we do more I'm like okay can you make unity catalog agentic they said yeah we'll talk to database so that is the awesome part of unity catalog that's the value we see from unity catalog uh when we are dealing with thousands of users every day in our platforms the next problem is data reliability we all say garbage in garbage out we always say we want to do AI and we want to change the And then next day many of us have read those news that agent or some AI solution data hallucination the company ended up paying $100 million penalty or many more penalties.
You don't have a strong data foundation. You don't have a reliable data. That's a problem. If your data is not refreshing time, if your data is not available in time, if you don't have all your KPIs vetted, you know, that has always been a challenge. And last but not the least, what is this data about? If your data is changing every day, you need to have an illiteracy in place. You know, other than people asking to ASME with the same question million times.
That problem has been compounded to many many business uh solutions and problems we have dealt on every day. How did we solve this problem? It took us some time. You know, everything we thought was good. The next day we get a ping from somebody. Hey, I thought we making a change in the Seattle market, but I don't see the data coming from Seattle market and my customers getting impacted, right? because we launched a new service.
Uh we started working on something what we call internally as a early warning system which was essentially leveraging anomaly detection and time series data to understand your data pipelines. Is your data coming the same based on your patterns of data, right? Are you finding any anomaly data than before? It is not just looking at the business KPIs. It is about loading of your data, availability of your data, refresh of your data, right?
Making sure that you have critical KPIs that you are monitoring more than not before, right? We built a log analytics dashboard because we do everything on Azure. So we integrated everything through a framework. Everything was being monitored through 15 minute, five minute latency on the loging dashboards. We leverage Azure actions basically for setting up our alerts based on the severity of the issues. Created a very comp comprehensive runbooks basically that provides you the action.
So your devops team are not just reaching out to themes but they're taking the action based on what is in the uh runbook document. So there's a lot of work we have done. Are we starting today? Absolutely not. There's a lot to be done and I'm so excited what what the technology is going uh to see how we can leverage more and more of you know building what I call as agentic data management ecosystem cost challenges uh I'm sure many of you guys have dealt with this almost on a regular basis I want to share a story we started our journey in databix in 2019 of July we got a brand new environment I had my three super smart guys it's like hey there's a white room and we have to build the whole platform here right and some of us knew how to spell data bricks at that time that's all we knew great partnership with data bricks I was in a vacation in uh in Vegas with family just to make it clear with family right in December 4 months into our journey I looked at my cost alert for the previous day I spent 4x more money in that one day as compared to what I spent in the last two months first I thought maybe I went to some casino last night I was drunk And I checked my bank account.
Thankfully, whatever money I had in my bank account was there. Then I called my engineer. I'm like, "What the heck, dude? You're the smartest and the brightest I have in this team. What the heck is going on? How are you spending like tens of thousands of dollars in one day?" And we were all doing stuff from data for years. He said, ", I wanted to run a scale test. I took all the CPUs that were available for that particular workload and I ran it."
I'm like okay can you come with me to my senior director vis who's sitting this room and tell him what why we did that so we don't get the beating I'll share the other example how these things are very complicated right um we thought we learned everything five years back we were so happy oh we know how to manage the cost come on all of a sudden a year or so ago we were using a wampool when you use a wpool it's kind of like a always own database cluster right and one of the super smart guy working under tremendous time He had a min and a max worker as the same size.
So your min is 2 210 and your max is 210 and my PHOPS team which again happens to be only one person in me right after one one week we looking at our weekly alert and see where the hell we spending this money man this is crazy and then it was baffling to us we were running these jobs it was only using DBUS there was no VM utilization it took us like literally 10 days to figure out what happened and all you have to do is chain them into like two nodes and max to whatever you wanted to have but it was so painful.
Um I'm sure many of you guys know when you run data bricks there's DBUS and there are various different things data bricks guys make our life so easy you have interactive clusters you have all purpose clusters job clusters you know serverless compute right I mean amazing things these guys do hell man that's amazingness makes it difficult life for us um we have to always be on our toes on like where are we really spending you know we look at DBs and we look at um VMs and then as the next build we There is a premium storage.
Spark is memory based. Spill happens. The memory spill disk happens to be a premium storage by default. There's a small component of network. So many pieces which are not really known to everybody until you start really getting into connecting the ROS to be able to understand where the cost is. How are we solving this problem? Uh I would say if you guys have not done that, make tagging as your best friends. Tags has to be there everywhere.
Every application, everything you do has to be tacked. Leverage cluster policies. You know, in fact, I would say even force policies sometimes harder than what people think. Do not give them the free word. There's no free word. You know, define the policies. Give them the right type of compute. You know, make sure you leverage um auto termination features, right? Make sure you are basically not running your production jobs into a interactive or all process clusters.
Use job clusters. Right? In our organization uh one of the thing we do really really well is we have a weekly architecture review of all the new workloads and a lot of technical depth we have over the years where we go back and reook at architecture and say where is the waste what are we doing differently when we're building a new architecture look at cost as a component of the architecture. Don't just look at the cost as after effect of you know you do something and then we'll figure out later.
Last but not the least reward your engineers who are doing a good job. people who are building best practices the more you award them the more value they will bring to the organization and the more other people will look at them as inspiration so reward those guys you know appreciate those guys call those guys you know in our organization we have a monthly engineer of the month award and we're not looking at only data engineers or machine learning engineers who are building cool stuff we looking at platform guys who are actually making life easy for everybody else um skill set challenges I think we'll all agree every day there's a new technology every there's a new model there's a new AI Everything is a new agent.
Holy cow, how do we keep a track on that? Right? Every day I go and tell my team, hey, let's do a you know something I want to see tomorrow. There's AI available in data bricks. There is AI available in Azure. Right? There are so many tools now. It's moving very fast, very rapidly. And with that, unfortunately, is increasing the expectations for everybody. You have everything in your hand. Just freaking do it. You are smart people.
I'm like, yes, sir. Right. How are we doing that? So, we started leveraging data bricks AI assistant. The first thing I did was actually I found out that tool very useful to fix our data literacy issues. We leverage a lot of recommendations from database AI assistant to basically uh update our catalog right there in some of our network data sets we get 500 700 800 columns and they're really hard to understand. It's smart enough for you to provide you a basic definition of what those column means so you don't have to go and look at your SMBs and then get vetted by them right.
Um absolutely for our new engineers you know the junior engineers who are joining our team right they may not have worked at database in their previous experiences they may not know spark so the AI assistance is really really helping us every day to look at our code completion you know help with troubleshooting you know help with debugging our code and then it really really provides real-time insights and code optimization and I think they are really really taking this forward so it costs almost nothing unless datab changed the pricing model yesterday right So leverage those things that are available to you every day.
In the end, I will summarize um some of the best practices learning. When you're building your solution for scale, you need to start designing from the scale from day one. I was just talking to the database guys on a data bricks workload we have. It has grown 45% in the last four months. And I was telling them, guys, it's going to go another 45% in the next three months. And I don't want to wait for the next three months to spend another $100,000 on that pipeline.
I want to build that scale today right because when a data grows another 50% 6 months my solution is optimally built believing the value and it's also cost efficient I didn't talk a lot about automation is the key but I think all our data engineers our software engineers we always follow the principles of CI/CD bring CI/CD from day one right have automation in place not just for the CICD for engineering also but for your ML pipelines you know use ML flow for your data engineering pipelines use CI/CD for platform build your policies using infrastructure as a code, right?
Um, data bricks keeps bringing new solutions every day. Always stay on the top of it. If your team is not supporting you, uh, which they always do, create wonderful data with partners. Look at Delta, look at SQL functions. You know, SQL serverless SQL has been a great tool for us. Now we're looking at serverless compute. So, always stay in touch with the new changes they're bringing in very rapidly. Data quality and data reliability.
Again, like anything else, you cannot look at data quality after the damage is done. You need to build your solution in a place to prevent and minimize those data quality issues from the beginning. Logging, alerting, and monitoring for operational teams, DevOps teams is it's it's I would call this as like you know, you're building a car, but you need to also make sure your car car goes long way. So, if you're looking at signals on the car, take an action, right?
Unity get lock for access controls, data sharing. I didn't talk much about data sharing but data sharing with the delta sharing that database has done has not just improved the cost but has accelerated collaboration and innovation right one thing I always tell people I've seen developers they are very very sometimes not in a way to write code they want to write good code but they don't want to write lot of documentation right make sure you have time in your dev cycles in your sprint planning to write good documentation that is going to take you forward when you're building best practices when you're building create solutions when you're looking at future not just for today but also a year from now build the knowledge base and now you have geni leverage geni to build those solutions and knowledge bases for your devs and last but not the least we always say people are your biggest asset invest in your people all the times in our organization we do quarterly hacker hackathons you know we always believe in the philosophy of eat your own dog food first before enable to others right have ongoing trainings and workshops right attend the sessions right ask people right link LinkedIn is the biggest community where I learn every day.
Leverage that free word which we don't have before. With that, thank you so much for your time. Uh I'm open for any questions.

Learn more about the Databricks Data and AI platform.