Databricks as the Backbone of MLOps: From Orchestration to Inference

Summary

• Integrated MLOps Backbone: Globe Telecom uses Databricks Workflows as the backbone for orchestrating the entire ML lifecycle—from development through inference to monitoring.
• Automated Pipeline Deployment: Streamlined processes automate model development, inference, and continuous monitoring, ensuring reliability and reducing manual overhead in production.
• Operationalized ML at Scale: Implementing ML best practices and governance through Databricks enables organizations to move beyond experimentation to reliable, enterprise-grade ML systems.

Databricks as the Backbone of MLOps: From Orchestration to Inference

Talk by Cyd Kristoff Redelosa, Senior Expert - MLOps, Globe Telecom ; Reinier Veral, Asst. Director MLOps, Globe Telecom
As machine learning (ML) models scale in complexity and impact, organizations must establish a robust MLOps foundation to ensure seamless model deployment, monitoring and retraining. In this video, we’ll share how we leverage Databricks as the backbone of our MLOps ecosystem — handling everything from workflow orchestration to large-scale inference. We’ll walk through our journey of transitioning from fragmented workflows to an integrated, scalable system powered by Databricks Workflows. You’ll learn how we built an automated pipeline that streamlines model development, inference and monitoring while ensuring reliability in production. We’ll also discuss key challenges we faced, lessons learned and best practices for organizations looking to operationalize ML with Databricks.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does Globe Telecom use Databricks as their MLOps backbone?

Globe Telecom leverages Databricks Workflows to orchestrate their entire MLOps ecosystem, handling workflow orchestration, large-scale inference, and continuous monitoring. This unified platform enables the company to systematically manage the complete ML lifecycle in production.

What challenges did Globe Telecom face transitioning from fragmented workflows?

Previously, Globe Telecom relied on disconnected tools and manual processes for ML development, deployment, and monitoring. This fragmentation created inefficiencies, reliability issues, and made it difficult to enforce governance across ML operations.

What is automated inference in Databricks MLOps?

Automated inference refers to deploying trained models to production where they automatically generate predictions on new data at scale. Databricks enables reliable, monitored inference pipelines that ensure model performance and consistency in production.

What are key lessons learned for MLOps success?

Key lessons include establishing clear workflows, implementing comprehensive monitoring, automating model retraining, ensuring governance throughout the ML lifecycle, and focusing on reliability and scalability rather than just model accuracy alone.

Full transcript

So hello everyone. We will be talking about data bricks as the backbone of MLOPS. So we're from Globe Telecom and we're happy to see you this uh this afternoon. Okay, just a bit about my career journey before the industry. I was in the academ so I was an assistant professor handling subjects like software subjects like C language, assembly language and some of the other hardware subjects. And then also I was leading the R&D the tech R&D.
So we developed some applications for our school then it's Mapua University but then I journeed on to the industry starting with Juken Software Philippines. It's a Japanese company and we were involved in testing um engine control units of cars. So who here has the Honda cars raise hands. So most probably we tested those especially uh the automatic cars the electric cars I think and then of course uh I moved on to Nokia they started an R&D office in the Philippines.
So I started as a senior software engineer and we're we were handling site uh software. So it's like uh the ops of the uh site man site uh hardware. So uh I also handled side competence development management uh incorporating of course the latest tech and learning on the side but sadly R& closed in the Philippines. It was transferred to another Asian country. Maybe you could guess. What that is? I have a short stint with quasset.
So this is asset management, still software development and currently I'm working in globe. So this is the first time that I was handling or involved in data and AI. I'm now leading a group uh an MLPS group. We are 10 in the group and we're so happy working in globe. Okay, what's the game plan for this afternoon? So, I want to introduce you to more details about Globe, who we are, and uh how AI matters to a country like ours.
We're from the Philippines, by the way. I will show you uh the journey uh in ML from the traditional to what we have now. Then we will show you the proof of our POCs and projects that we did experimentations that we've done to streamline our processes and improve or um create impact as well. Then we will go under the hood seeing the architecture and the improvements done and lastly the road ahead we will do starting this year up to next year CICD a govern governed feature store and scaling to Gen AI because we have those use use cases as well.
Okay. So let me start with the Philippines. A show of pants. Who here knows where the Philippines is located? Oh, quite a lot. Yes, you get a price. So, Philippines is the home of. You know? Yes. So, he's probably one of the brightest boxer right now. Of course, Mayweather will not agree. And you know, he's a senator now and soon maybe he will be the president of the Philippines. So, you're not alone in the USA. So the Philippines is a vibrant um mobile first nation of over 110 million people spread across about 7,600 islands.
Can you imagine? Can you imagine the logistics there of Telco with one of the highest social media usage rates? Okay. So if you remember the Philippines was also the SMS capital of the world like 20 years ago. We were number one in texting and right now we're number one in social media. A lot of content developers move to the Philippines because it generates those views from videos from uh from well from reading uh the reviews and etc.
So it present both a challenge and an opportunity for digital transformation espcially for Globe Telecom. So Globe Telecom was born in 1974 uh from a small radio company and it has since evolved into Techco. So aside from telco, we have other uh solutions provided to our customers. Then joined in 1993. We grew into the top mobile network. We are now number one in the Philippines for mobile now serving over 60 million subscribers.
But Globe is more than just a telco now. As I've said, we're a digital ecosystem from fintech, hech, edge and using the cloud and driven by data and scaled with AI. Of course, globe is redefining what it means to be a tech company in Southeast Asia. And under the leadership of our new CEO, we're going further expanding digital infrastructure driving financial inclusion through GCash or Fintech app and delivering everyday innovation for the Filipino people or subscribers.
Okay, just a bit of the platform ML platforms that uh has continuously evolved for us. Maybe you can relate with this. You follow the same journey with us alongside the technology landscape. So we started with SAS during that time pre 2021 on premise was the in thing which was the which ideal at that time because it was stable. It was rules based and business friendly but as big data and open source libraries gain traction we move to Cloudera CDSW orchid straight using CDSW during that time we leverage its distributed compute capabilities for Python based models.
Then came the cloud shift. So we all know this. We adopted snowflake for scalable data storage, sag managed model training and airfow for our orchestration. But you know it gives us flexibility but created some fragmentation in the process across the different platforms. Now we're bring it bringing it all together in data bricks unifying everything in data bricks but still there's some snowflake there for data lake combining orchestration training and inferencing into one platform that's collaborative production grade and fully cloud native let me now go to globe's journey to knowing our customers It's always been our croed to understand our customers 360°.
Starting in 2017 we began with a traditional segmentation things like predicting affluence the life stage of our subscribers. From there we deepen our view through behaviors like multisim use uh content interests household roles and even fixed mobile convergences by 2022 we introduce psychographic segmentation understanding not just what people do but how they think is expanded into ecosystem level insights in 2023 identifying shared affinities across fintech health and education and in 2024 we have integrated uh globe group wide digital IDs across services like consulta MD which is healthtech and GCash which is fintech enabling us to deliver through 360 customer understanding which helps us also serve our customers better.
Now I will talk about some of the use cases that creating impact for us as a company. First being interest AI. So interest AI is our use case for understanding our mobile subscribers or what the mobile subscribers care about. Based on the websites they visit, we use models like universal sentence encoding and mark of chains to categorize this interests, detect emerging trends and group users. into behavioral segments.
The data the data powers personalize marketing at scale. So it helps out with the marketing helps tailer offers given to our subscribers in their lifestyle. content integrated into our broader AI stock for customer value and targeting use cases. Second was one is the next best offer. So it transforms how we engage customers by delivering AI driven personalized promotion with the highest likelihood of accepting the promos.
Instead of relying on manual business rules, the model applies contextual bandits, a reinforcement learning technique to learn which offers resonate best with each of our subscribers. It pulls from rich data sources including subscription behavior, offer features and target based dynamics. Since launch it has delivered around 7 million quarterly ROI driven by smart offer matching and the key benefit including boosting campaign effectiveness improving customer retention with timely offers and expanding targeting scope beyond the rigid rule rule sets And the last one is airtime loans propensity.
Okay. So this use case predicts which subscribers are most likely to repay airtime loans. So we offer loans to our subscribers and using behavioral location and top up data with models like logistic regression. So simple models and gradient boosting we are able to boost revenue by around000 monthly through smarter smarter risk assessment expanding eligibility to high potential borrowers and drive retention by keeping users engaged throughout the whole process of the rewards and loans.
Okay, this is our MLOPS maturity model. When we started uh we created this basing it into or from the existing maturity model of Microsoft and other companies as well. We tailor it to our own processes and this maps how our capabilities evolve as we scale both in value and process maturity. We began in 2024 at level 1. So during this time um we have assets like ML pipelines. We already have that. But we started creating model canvases that outlines or summarizes what models or use cases we have and we place some monitoring in place.
Okay. So our 2024 goal then was to reach level 3 which is automated enabling the retraining process. So we created some POCs for that feature store and centralized registries model registries that reduce manual work and we reach that end of 2024 and looking ahead to we aim to uh reach of course the highest level which is continuously improved where models adapt over time retrain automatically and are optimized iteratively unlock unlocking maximum value for the business and with data bricks we know that this is possible before moving to data bricks we run into real challenges with our previous stock on the MLOP side we faced platform overhead package limitation and minimal logging from airflow which is a pain point for model development tools felt really clunky auto ml was restrictive and resources were limited which slowed us down so these issues rebuild a clear need a more scalable cost effective andollaborative machine learning environment to how we build andy ours This pain became a catalyst for change.
One that data bricks helped out address. Okay. Databs gave us more flexibility with collaborative notebooks across Python SQL compared to the legacy platform that we used to have. Even the standalone Jupiter setup. It also streamlined auto ML and experiment tracking with ML ML flow built in natively while the legacy platform requires more manual steps for big data processing data brak spark engine proved far more scalable than EMR which we use in our older models and uh most importantly Data bricks enabled realtime multi-user collaboration, speeding up teamwork and delivery.
In short, data bricks allowed us to simplify our workflows, scale better and collaborate faster. Okay, so let me give you SID for the next segment. Hello. Hello. Can everyone can everyone hear me? So thanks R. So Rain talked about the bigger picture. So now let's talk about our hands experience and how we actually tested data bricks on the ground. But before that let me talk a little bit about myself. So I began my career in academia where I earned uh BS applied physics and I majored in instrumentation physics.
So that's where I studied um network analysis of traffic analysis zones. I use Uber data to do some resource allocation studies and then in 2019 right the day after graduation I started working already so no rest. Um, I worked as a data scientist in United Health Group R&D. So that's where I studied the effects of the COVID pandemic on CKD patients. So that's where I got my hands on experience of um ML with high impact.
So then in 2022 I actually joined Globe as a data scientist or in Globe terms as a model development expert. And then I think 2 years after I transitioned to being an MLOPS expert and now I am an MLOPS senior expert and I lead a team of three people. So um before I head on to the actual hands and experience I just wanted to share a little bit something not related to my career cuuse um they say when you try to give when you give a talk you should try to connect with your audience.
So um I like to share two things. So the first thing is this is my first summit. So who among here is um first timers. Okay. At least we connect in that part. So second part, who here loves cats? Cats. Cat lovers. Okay. So, now we're connected. So, we're good. Okay. So, next, um, so when we got the opportunity to pilot data bricks, we didn't go in with toy data or just sample notebooks. So, we wanted to see if it could handle real production workloads with actual business impact.
And the first test, one of our biggest, the prepaid CLV pipeline. So basically the prepaid CLV pipeline predicts the customer life time value of prepaid subscribers in the Philippines. So for this use case, we were working with over 30 million rows and each row needed its own exponential smoothing model. So that's how it was designed and that's basically 30 million tiny models which on our legacy platform as you can see it last 16 hours.
So imagine if it went wrong halfway so you'd have to wait another 16 hours. Right? So that was such a hassle. So when we ported the exact same logic to data bricks, we wrapped the forecasting into a spark UDF. That's all we did. And thanks to Spark's distributed compute, the whole workload finished in just 35 minutes. That's a 28 speed up using the same instance type. And after seeing that kind of performance gain, we tried another high impact pipeline.
This time the multisimmer model. So this time this also uses 30 million rows and in the legacy platform it runs 1 hour. So initially we tried to just lift and shift the code into data bricks but we immediately hit a memory error possibly because of how the model works. So we adapted by wrapping the logic into a single row spark UDF and turned on autosaling. It ran but it took 16 hours. So that's way worse than what what we had in the legacy platform.
So we had ourselves thinking should we still continue working with data bricks if it makes things worse. So of course we didn't stop there. So next we then disabled auting and then fix the cluster size to 12 nodes. So somehow we we didn't have the time to figure out why auto scaling made it worse. So we just turn it off and then set it to 12 nodes. That alone brought run time down to about 3 hours. better than the previous iteration but still worse.
So finally we switch to using batch UDF processing rows into chunks instead of one at a time and got the best result just 17 minutes of run time. That's amazing. Same model logic but smarter execution. So this journey showed us that how you run your workloads in data bricks matters just as much as where you run them. So with the right tuning, fixed resources, batching and smart parallelization we unlocked massive gains in both speed and efficiency.
So now let's head on back to the prepaid CLV results. So again in the legacy platform it ran about 16 hours in data bricks 35 minutes. So in a regular legacy environment the model ran and took around 16 hours and cost us $200. In the data bricks POC we resulted it to 35 minutes and just $6. That's 194 savings in 10 months. Our 2026 Data Brick Summit ticket is already secured. So 194 10 we're already secured for next year.
Hopefully we see you guys again. So that's a 28 times speedup and 97% cost savings. Huge numbers. Clear proof that data bricks provides a significant edge in runtime efficiency and scalability. even with the same hardware specs. So on the other hand for the multisim runtime comparison again in our legacy platform about an hour in data brak just 17 minutes in the legacy platform it cost us 3. 51 just small compared to our CLV CLV pipeline.
But in data bricks it it got reduced to 1. 02 to 1. 63. So that's three times faster and 50 to 70% cheaper. So these POCs highlighted persistent issues, platform overhead, package limitations and minimal logging with airflow on the ML upside and clunky tools, restrictive auto ML and limited resources hindering model development as mentioned earlier. So the POC reinforce our need for a more scalable, cost effective and collaborative ML environment.
This experience was crucial in demonstrating how data bricks could help us overcome these hurdles, making it a catalyst for a shift to the platform. So performance was clearly there, but we wanted to know how does it actually feel for the people building and deploying models every day. So that's what we ask our developers and here's what they have to say. So this is a quote from a model developer from our company. So it's not just about speed.
It's just it's also about making development more collaborative streamlined and production ready. Here's another quote from Unreay, another model developer from our team. So underscores the value of having a unified platform. So in the past we had to pce together various tools for data exploration, model building, tracking and deployment often leading to silos, friction and inefficiencies. With data brakes, everything is integrated.
ML for tracking batch and realtime deployment options and a unified data catalog that streamlines access across teams. This allin-one setup has been key to accelerating our model development life cycle and boosting team productivity. Lastly, another quote from our model developers. So his experience highlights a major painpoint men of us faced before managing messy logs and setups. So with bricks hyper parameter tuning and experiment tracking are built in and seamless with ML flow everything stays organized and reproducible which is critical when handling over models to MLOPs.
So the results smoother workflows, less rework and faster development cycles. This kind of developer experience is what accelerates real world impact. So now let's talk about the what's under the hood or the architecture we had before and what we have now. So the current framework or what we have before was we had airfow as the orchestrator. So the raw data resided in snowflake and data transformations happen in snowflake.
And all these transformations resulted to an analytical base table which is what you would ingest in your model. So this this ABT or the analytical based model gets saved in an S3 bucket as an as a CSV. This then gets injested in our external um inferencing platform and then the results of the inference get saved as a CSV again in in S3 and then it gets manually injusted again in snowflake as a final table. So while functional this workflow introduces manual steps, higher latency and limited scalability especially with increasing data volumes and real time demands.
So in our modern setup we've replaced the CSV CSV base handups and manual processes with the seamless scalable architecture powered by data bricks waterflows and iceberg tables. So as you can see the orchestrator orchestrator now is data bricks workflows dat data and the transformation still happens in snowflake. But then this time the ABT gets saved as uh gets saved in iceberg format in an S3 bucket and then gets ingested in data bricks.
The inferencing now happens there. The inference table get saved also in iceberg. So this inference table is automatically accessible in snowflake as an external table eliminating manual steps and ensuring real time availability to business users. So this framework delivers automation, scalability and cost efficiency setting the foundation for bat scoring, future streaming and future streaming use cases and gen AI integration.
So now let's talk about the impact of a unified ML platform. So shifting from a fragmented tool set to an integrated ecosystem has delivered four major benefits. So the first simplified workflows. No more switching between tools. A single platform has streamlined our processes and reduced context switching. Number two, better collaboration. Shared notebooks and MLFO integration make it easier for data scientists and engineers to work together and accelerate development.
Number three, faster experimentation. Auto ML, MLOW and scalable compute have shorten our model development and deployment cycles. So lastly, optimized cost and performance. With pay as you go compute and efficient batch processing, we scale only when needed and pay only for what we use. So in short, data bricks has help us go from tool chaos to a cohesive efficient MLM environment. So now I'll hand you over again to talk about the road ahead.
Okay. To strengthen our emelops backbone uh we tackled three priorities. First is the full CICD for ML automated build test deploy pipelines with gated model promotion regression test and canary rollouts. Second, a governed feature store. So we're creating some POCs and some uh development for this Delta Iceberg backed with lineage versioning and online offline party to teams. So teams can reuse features instead of building each time.
And third, Genops wrapping our Gen POCs with rag pipelines, promp version control and GPU optimized serving. So LLM apps move from notebooks to production or Python codes to production with the same rigor as any of the classical models. Let me introduce you to an LLM uh model that we have now. is our internal knowledge based chatbot designed to unlock insights buried in past research and studies. So those studies normally are in PowerPoint or some other form of documentation.
Instead of digging through old slide decks and documents, users can simply ask and get direct conversational responses drawn from previous or prior analysis. It uses LLMs with rags and document embeddings to match natural language questions with relevant archived content. The result would be faster access to institutional knowledge, reduced duplication of effort and the smarter way to reuse our learnings. Chat based knowledge retrieval at scale.
So instead of days or even weeks to produce the results, it will be available in minutes. Okay. So as we come to the end we are we will share some of the key learnings that we have. So platform of choice matters migrating two data bricks will unify RML workflows, improve performance and drastically reduce cost. Batchoring at scale needs smart storage. Using iceberg table managed in snowflake or in data bricks will allow us to scale scoring and maintain versioned quarable outputs accessible to the business.
Standardization enables collaboration. shared ML flow gitbased CICD and unified pipelines via data brick workflows will help you will help us streamline development and reduce manual intervention observability is key to trust especially among the business entities model monitoring VSI PSI distribution accurac recall those things and other validation metrics will ensure we could reliably deploy and track model performance over time.
And lastly, business impact is the true north. Every tech decision was anchored on engagement, revenue uplift and subscriber experience, not just model metrics. These lessons aren't just technical takeaways. They're cultural shifts. As we move forward, we are building not just better models but a more connected, collaborative, and impactful AI practice that puts business value at the center of everything we do. So if you know if you want to know more about Globe you can go to this website or scan the QR code.
And in Globe we are building an AI powered telco and we're happy to swap notes with you. That's it. And think we still have 4 minutes. You can take one or two questions if there are. There are none. It's okay. I know you're ready to party. Let's go party. தேக்யூ

Learn more about the Databricks Data and AI platform.