Democratizing Data Engineering with Databricks and dbt at Ludia

Summary

• Democratized pipelines: Ludia empowered analysts and domain experts to build production-grade data pipelines using Databricks and dbt without relying solely on centralized data engineering.
• Cross-functional autonomy: The approach enables cross-functional teams to own their data, accelerating time to insight and improving data reliability across the organization.
• Gaming innovation: By democratizing data engineering, Ludia fostered a culture of data ownership that drives faster decision-making for game development, live ops and player engagement strategies.

Democratizing Data Engineering with Databricks and dbt at Ludia

Talk by Huntting Buckley, Global Games GTM Leader, Databricks ; Jean-Christophe Rodrigue, Sr Director of Analytics, Ludia
Ludia, a leading mobile gaming company, is empowering its analysts and domain experts by democratizing data engineering with Databricks and dbt. This talk explores how Ludia enabled cross-functional teams to build and maintain production-grade data pipelines without relying solely on centralized data engineering resources—accelerating time to insight, improving data reliability, and fostering a culture of data ownership across the organization.
Here’s more to explore:
Economist Impact: Unlocking enterprise AI: https://www.databricks.com/resources/webinar/unlocking-enterprise-ai
The State of Data + AI: https://www.databricks.com/resources/ebook/state-of-data-ai
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

How does Ludia democratize data engineering?

Ludia enables analysts and domain experts to build and maintain production-grade data pipelines using Databricks and dbt, reducing dependence on centralized data engineering teams.

What are the benefits of this approach?

Democratization accelerates time to insight, improves data reliability across teams, and fosters a culture of data ownership where every team can make data-driven decisions independently.

How does this apply to gaming?

In gaming, democratized data engineering enables faster analysis for product decisions, live operations optimization, player segmentation and merchandising adjustments across Ludia's portfolio.

What games does Ludia develop?

Ludia is known for major titles including Jurassic World Alive (a geolocation dinosaur game), Jurassic World the Game (a park builder) and Dragons Titans Uprising, with many more games in their portfolio.

Full transcript

Uh but no, without further ado, I would like to pass it on. Yes, I should be miked up. There it is. Uh thank you, Hunting. So my name is JC. I'm Ludia's senior director of analytics and this is our journey uh through data. Um so first who we are Luzia. We are Canadian-based uh mobile gaming studio. We are founded, we were founded in 2007, started on console, did some Wii game, Wii U, but quickly, you know, mobile became a dominant platform uh and we pivoted to free-to-play.
Um shortly after our foundation, we are known to build games and partner with big IP licensers. We work a lot where we're NBC Universal's uh main uh gaming partner and I think well right now we've done a lot of games over the years but our three biggest games right now are Jurassic World Alive, Jurassic World the game and Dragons uh Titans Uprising. So one is a geoloccation dinosaur midcore combat battle game. The other one is a park builder with more casual dinosaur battles.
And the one the dragons game is actually kind of a clone of Jurassic World the game uh but with the how to train your dragon. We tend to do that too. Clone and reskin our games you know. So this the next slide will be it is the vision of what I have. What does what is the purpose of data at Luzia? The purpose of data and the vision is to accelerate the growth of Ludia's gaming portfolio by leveraging data and analytics analytics and insights to their fullest.
Um below are the business units that we support. These business units should be pretty familiar to you. Just just curious by you know show of hands who who is from the gaming industry here. Okay. So lots of folks out media and entertainment maybe. Well, you know, welcome to my gaming talk, but this probably applies to you. Um, so we're not going to go a little bit into detail on two of these big uh business units that are critical to our product success just to give you some detail.
Well, actually, who here plays video games? Who spends a lot? No, I'm kidding. Uh, some people do spend a lot. So, for product and live ops, we are in the entertainment industry. You know, we need to launch great games. Um, we need to launch games that feel well balanced. The experience has to be thrilling, fair, and exciting. We need data to do this. We can't just use our gut and intuition because what your opinion is is not representative of the whole player base.
You know, we need segmentation. We do a lot of merchandising optimizations, whether they're done manually or uh automatically. All of this needs to be data informed. We run a store inside of a world. We have a monopoly inside the game. We decide what we sell uh typically at what price as well, but we um you know, I I'll just be clear here. We don't really uh segment and target different prices and discriminate. I think we're we're not that kind of studio.
Um we do other things, but but not that. Um content is king. It's important to have enough content to sustain your monetization. But if you have too much content and burn it too quickly, one, users get used to it and so you might not reap the maximum ROI on its release and two um you know they they they get used to it. So they always want more. It's a constant thread mill and content is expensive to make. So we want to have just enough and you need data to make that decision.
Uh forecasting and target setting monthly revenue forecasting. So, who here works in a company where the word forecast is mixed up with the word goal as well? You know, even hunting a data brick, you know, are we going to meet the forecast this month? I'm like, what do you mean we're going to meet the forecast? The forecast is a prediction based on what we think could happen. It's like the weather forecast, you know, it's like no rain because the CEO said so.
I like no it's the weather pattern you know and but it's the same we need to you know we need to set ambitious goals for our products and set up our teams for success so that they meet those goals but the goals are not forecast you know I just I just keep um repeating this but it's it's like a it's a negotiation between our product teams and data science and our CEO who no surprise really likes money uh he's great though best boss I've ever had on the uh and you know we do a lot of deep dive analysis to understand really what is the root cause of certain behaviors uh that we either think are are positive or or negative to our users.
This is Rexi, the most iconic dinosaur in the Jurassic World franchise. And um you know we uh we're going to talk about marketing. Marketing is about maximizing the product channel fit. We got to get the word out. Some games scale very well organically. you know, the buzz goes and people talk. Other games are more niche. Finding the users that are a good fit are are harder. Therefore, we need marketing. Marketing is about allocating resources to the right marketing channels uh and choosing the best creatives.
Right now, things move very fast in the world of creatives with Gen AI, all that stuff, influencers, Tik Tok, it's it's dizzying, you know, to stay on top of the pace. You have to optimize your bid strategy. If you bid too much, well then you're going to never recoup your investment. If you bid too little, then you might miss out on scale and let your competitors actually scoop up these eyeballs. Lastly, it's important also to, you know, have very good app store optimization, good conversion.
Uh maybe people from marketing here, marketing industry, advertising one. Does that make sense what I'm saying? You have gaming clients. No. Okay. Marketing is super tough, you know. Is so our data journey. So this brings us to I joined Lizad. This is 2021. This is our modern-ish lata stack. On the left we have our source systems. We have our games uh backend client server generating uh telemetry events. Adjust is our MMP for you know we've been partnering with them for a long time.
It you know gets the job done. Uh and we have our app stores Apple and Google that are also uh we pull data from their API. Okay, we ingest all this data uh through some you know we used to use Fiverr uh it works well but it cost money sometimes a lot of money um snowpipe also noticed that we were snowflake customer back then spoilers we're not snowflake customers anymore in case you have not guessed uh and we um we also had amplitude for ingestion but also consumption you know amplitude is a full end toend data uh product performance uh and insights platform.
So processing snowflake Jenkins. Ah what is Jenkins doing there? I don't know you know but he doesn't belong there. Poor Jenkins and consumption your usual suspect Tableau you know because Tableau is still a thing. Uh local Python development environment. So not really machine learning at scale but at least we you know cond and amplitude. And so you know we ran what are we happy about this the positive? There are few but we have to mention them.
It's simple and reliable cloud infrastructure. You know nothing went down. We didn't need to maintain anything really and in some aspects we had high level of self-service mostly because amplitude allows users to self-s serve very efficiently. uh I don't know what their backend cost is but it must be stratospheric to see how much you can pivot and get data in semi-real time to a level of granularity that goes from 10,000 ft to almost nothing in seconds.
Uh I think they are having high backend cost but you know I might be wrong. The negatives though, the negatives, well, we have this functional orchestration. Obviously, we have Jenkins, you know, uh, and Jenkins. What did Jenkins run? It ran a constellation of copypasted SQL scripts on Snowflake. Yeah, exactly right. You know, it's super dirty. Um, not the worst I've seen, but, you know, can do better. Uh, we have no data quality test and observability.
You know, I'd say that our testers are the people that look at our dashboard. Our CEO should not be a tester. You know, there's something weird in the dashboard. Ah, I'm sweating. What? Again, very paranoid. No data lineage. This is 2021. If you wanted data lineage, you had you had to buy a tool probably like a Monte Carlo, you know, to do like, you know, you know, invest invest more, you know, just to have your data lineage.
We had new CI/CD and very poor data governance due to all of the above. We had data silos, expensive total cost of ownership as well. You know, getting close to the seven figures for a studio excluding headcount on my team for a studio that was at the time a little bit around like 400 people. That's a considerable chunk of money, you know, make sure that we get our value. Um, the results I give ourselves a B+, you know, but I'm being generous in the day and age that a fivestar rating, you know, anything than less than a five star is, you know, criminal.
It's it's not very good. It's very painful. Uh the results is we have trust issues, data silos, same metric, different value depending on who you ask. And you ask the product team, they're going to tell you something. You have the marketing team, you ask the CEO. Uh it it was very tough, a lot of manual work. It's incredibly painful to backfill using Jenkins, you know, it's like full refresh. Ah, you know, too expensive date parameters.
Ah, they're not in the scripts. Okay, copy paste the script. go to the SQL editor and hold my beer. You know, it's kind of like Indiana Jones all in production. You know, that was it. 2021 very scary. So, the big question I asked myself and the team is how can we increase the quality, reliability, and trust we have in our data without incurring additional costs. That was uh like I don't want to spend more money on this clearly.
And neither did my boss. I'd uh already went to him once because we've exceeded the snowflake budget. And I was like, I need more money. He's like, I don't like that. I understand. So, we said, well, why don't we turn our data engineers and analysts into data stewards uh using DBT as the common language? And honestly, you know, I know a thing or two about data. I'm not some PhD data guru. I just I just thrive on delivering value to the business.
I've worked with several other uh cloud data platforms. The lowest common denominator between all of them is always been SQL, you know, and so analysts understand SQL. Engineers understand SQL. Everyone really understands SQL or has some level of mastery of SQL. And then there had been a consultant that came in and had built you know a DBT uh uh March and you know everyone really shot that project down because it wasn't perfect and there was really like an internal war at Ludia between the data analysts who were on the product and would sit very closely with product leadership and shoot down everything happening centrally uh from BI.
So I you know I flipped it upside down. I I saw the project I was like this is a really cool tech that's been improperly implemented. Let's do that again. So we did turn our engineers and analysts into data stewards and these are the benefits uh you know shared version control collaboration testing a lot of testing. It brought myself great peace of mind to finally have tests although they are a bit tedious in DBT back then they are also free you know that's the cool thing.
So we'll dive into the details of these benefits here. Um just pause here. Who uses DBT? DBT users in the audience. One, two, three. Okay. So half of the audience is not using DBT. DBT stands for data build tools. To the other people, please call me out on my Uh you know, and questions. Ask your tough questions. Uh we've been using it for a few years, but anyway. So shared data uh model development. So data engineers and data analysts work together to uh to optimize performance.
The analysts are always the best at knowing the data. They define the taxonomy, the tracking. The data engineers can't really stay on top of the taxonomy. The analysts always get the tough question. They sit at the forefront. So they know it all when it comes to implementing. Sometimes their SQL chops are not as high as the data engineers. But together it creates an you know a great combo for us. So SQL optimization things like you know I want the last row of this day.
Oh I'll do a window function and row and sort descending. It's like yes. Uh but it's highly ineffective. You know data engineer says you should use max buy if you don't use max buy. Max buy is awesome in data bricks. Uh it's much more efficient. You save money right there and it runs faster. uh the engineers you know serverless compute optimization one we use serverless coming from snowflake snowflake only has serverless so that's what we knew and then we we we met to data brick and data brick launched serverless compute you know about at the same time that we went to data brick uh but choosing the right warehouse size is critical because if you have a small warehouse you think you're saving money well the warehouse is actually um swapping to disk and it's ineffective so don't be afraid to build a regression model or a scatter plot of like size of warehouse and you know execution time.
You might be surprised. Uh so testing you know we didn't monitor freshness. We didn't really have a great alert on on our source systems. Now we have freshness monitoring. Uh but we test for game business logic and consistency like your revenue table aggregate should match the sum of the non-aggregate table. Sounds very basic but turns out it's not true you know so we need to realign and govern our data uh and check for the right level of granularity especially if we have sometimes more junior people tinkering with models you know CI/CD really helped uh with that we now have over 800 tests in our build uh and it's not enough we have now well dbt back then brought data lineage uh this is not new well now we have unity catalog which brings data lineage But who here is on Unity catalog?
Okay, about half the audience. Well, you know, back then this was a big breakthrough for us. We could trace our data and where it's started its journey to its destination. Uh Snowflake didn't have as well uh anything for lineage. So this was like turning on the lights for us. Uh data ops CI/CD engineers orchestrate the process. We are orchestrate our CI/CD and GitHub action. Back then we were uh clients of DBT cloud uh but we were mostly using it to orchestrate the build.
They have slim CI in there too. Watch out. If you tick that box and are not careful about what you select, your cloud build will explode because it's going to test way too much and then you're going to go and ask your boss for more money for cloud compute. Uh but it sounds so straightforward but now everyone needs an approval to emerge and deploy new code. You need to be accountable for what you deploy and someone has to you know sign off on your changes.
You cannot do this by yourself. No one deploys to main by themselves. We are very strict about this. Um this limits bad code propagation and now and also documentation. You know everything that's gold uh in our mark needs to be curated for self-service. We need to have very deep explanation of our dimensions and measures for clarity. This bonus using DBT is published back now to Unity catalog descriptions. Um, and it's extremely helpful.
You know, this will set us up for um a head start with AIB Genie. We have probably all been to AIB Genie talks uh you know over the last two days and this is the main thing is get your metadata right. So we already worked a lot on metadata with uh with DBT. We just didn't know. But, you know, we're gonna we're gonna score because of that. And now this brings us to 2022. Goodbye Jenkins. We now have DBT. We're very happy about this.
But this is not the end. This is far from the end. Um well then 2023, you know, gaming industry is booming. COVID is kind of still aroundish. And then Jens City purchases Ludia for mucho dollars. And you know, our founder is very happy. He leaves us with all the cash and he's like, "Have fun with Gem City." And we did have a lot of fun with Gem City. They first greeted us with um announcing that we're obviously not going to renew the snowflake contract because they have a very high commitment on data brick and are not using it uh to its full potential.
So they're like, "You're coming to data bricks." I was like, "Oh, okay. All the governance and the things that we did on Snowflake, are we going to lose all of this? you know how long is this going to take us? We didn't really have a choice. So we had to we packed our suitcase and moved in with Jam City. So we managed to migrate to data bricks in about six months while maintaining our governance on our data. Much of the work um you know for those who know DBT was related to reconfiguring all our DBT sources.
So all the raw or bronze layer of the medallion cut it off you know rip it and then that you know gem city welcomed us uh with their stack they have a in-house tracking SDK they have a schemas uh and using cafka confluent and you know so it's very rigid taxonomy which was not necessarily the case for us so that required a lot of work reimplementation and QA QA QA QA QA and track uh also your data quality over Once this was done, we restarted the DBT build.
The tests ran. Some tests failed. We fixed the data. But once this was done, we had reestablished our governance and kept the lights on uh on our data products that our development teams needed to make and operate our games. A little bonus um here is well, I should have moved to this slide. I'm I'm confused. I was seeing a slide. Little bonus here is we got uni we migrated to unity catalog with very little effort. So who here is on unity catalog?
Okay, so not that many people of course hunting. Uh but your vote doesn't count. Uh who here is considering a move to unity catalog? Who here is considering a move to unity catalog while also using dbt? DBT user is easy. Yeah. Yeah. Well, easy I say because we we just have to fork the build, reconfigure, you know, fork your DBT repo, make a DBT Unity catalog build, reconfigure it to work with your new sources. You might have a little bit of work to do on the sources, but then run it.
And then we had parallel builds running the nonUC build and we had the DBT use catalog very easy to QA and track for gaps. There were not really that many gaps. Honestly, it's the same source data, just a new build. So if you're on DBT considering a Unity catalog move, it's extremely easy. Um we're the first I think product vertical to adopt it at Jam City and many other product verticals uh because they're not using DBT.
Um you know struggle it's it's a big move you know if you don't have a system like DBT. Well then 2025 happened. This is fairly recent but uh my boss uh sits sits me down in his office with the rest of leadership team. He's like big news guys. I'm like, "What, boss? You're quitting?" He's like, "No, the opposite." I like, "What? We're going to buy back our studio from Jam City for um an undisclosed amount, which is less than what they paid for, but also the gaming industry also has been hit with tough times recently, sadly.
We're going to recover. Uh, but we reacquired a studio. Uh, so this is the whole Ludia leadership team in Montreal. And I am very privileged to work with these guys. Now we had to uh rehit our textbooks. This book practical lakehouse architecture was published in 2024. This is the first time we have the keys to the entire lakehouse. You know under Jam City I had to deal with a central data team and architects and stuff.
Right now we have full power on everything. Hunting Jack the data bricks team are you know we're in good hands. It's I wouldn't say it's that much of a steep learning curve but there's a lot of stake you know so we need to get it right. We started by my team and I reading this book. Um I think it's a very good book and it's not a bias towards one cloud provider. You might not all be on data brick. If you want to build a lakehouse, you can do it on AWS using AWS tech.
You can do it on GCP. By far my takeaway is that it's by far the easiest to do it on Snowflake. Uh not on Snowflake. My god, cut it. It's you they talk about snowflake but it's not the easiest. It's the by far the most incomplete way of building a lakehouse is on is on snowflake. Uh on data bricks it's it's it's so much easier. You know you get unity catalog you get the governance uh you get the compute you want to have you know serverless compute now you have it.
You want to have dedicated compute now you have it. You want to run data science at scale it's so easy to do it. uh everything is well connected and governed. So you know check out this book. Now this is 2025. We have some new things we have changed. You know the source systems haven't changed but the ingestion has changed. We got rid of amplitude. Um and we are now using CFKA confluent schema registry to govern our data at the ingestion ingesting a files because they're efficient and small.
Uh we now you know we're betting on DT hub. Who anyone here heard about DT hub? It's like yeah you use it. No. Okay. So, you know, I made a bet said uh I'm not going to write Python workbooks starting from importing requests to scrape web API. This is no. No, I don't want that. But I also, you know, need to decide if I want to build or buy. Should I go to an air bite, a fiverr to ingest or should we try this tech that's kind of um more like a framework for ingestion, a declarative framework for ingestion that we actually run um on GitHub using GitHub action. is very is very lightweight and effective.
It was not, you know, turnkey out of the box, but fairly easy to set up. So, I think it's cost effective for us right now. So, you know, ask me in six months what I'll say. Maybe I I hope I still say the thing. Storage processing, everything is data bricks. Now, we are now on Delta Life table 2 for um transformation. All our streaming data pipelines run on DT um and we orchestrate you know the batch stuff still on GitHub actions.
Uh although the new uh lake flow sounds very interesting. We're going to be sure to check that out. See what value we can add on uh with lakeflow consumption you know Tableau and data bricks and with data bricks it's it's going to we're very bullish on the IBI genie now especially with all the metadata work that we've done uh with uh with our dbt documentation. So the new lakehouse is designed with the following uh guiding principles.
These are the six guiding principles of databicks lakehouse. The first one is curation. Deliver you know for us curation means deliver high quality data gold layer. So everything that is in gold it needs to be ironclad and bulletproof. Um and we do it with unity catalog and all the governance and the data quality test. DBT is part of that. Um eliminate data silos and minimize data movement. Well, with lakehouse and data bricks, it's extremely easy.
We are well our lakehouse is on Google cloud platform. So most of everything lands in Google cloud storage buckets you know or CFKA CFKA topics and um it's very straightforward to load that data as bronze layer into data brick and then from there you know silver and gold everything lives inside of different uh workspaces depending on on business units and we can share tables between workspaces too but um these are not silos you know but they they just organize things democratize uh value creation honestly uh for us getting rid of Amplitude kind of turned us into a Tableau dashboard factory very undesirable um you know Tableau for self-service I wouldn't say like using a dashboard is self-service it's more like just using a dashboard it's not very powerful they have explorer roles but I feel like AI agents and BIG is is really the thing that we want the push forward.
That means we could also decrease our investment in Tableau for self-service uh and also we very uh you know eager to play with the updates on the data brick dashboards in general. um adopt a organizationwide data data governance again this is so easy to do with data bricks and unity catalog on other cloud platform it's a lot harder you know um open interfaces we work with a lot of licensers and you know have a duty to share data back to them we stop sending email reports and Google sheets or or excel files uh we now use delta share whenever it's possible now we're just like come to data bricks and just get the data here It's always fresh.
It's always good. Um, it's easy scaling with cost control. Well, so you know, we've done a few mistakes like turning on the CI/CD and having it run on everything that was very expensive uh cloud or I mean serverless wise it's expensive anyway, you know. So with data bricks, we have budget policies and we tag everything all our assets for cost control. So it's we get a lot of peace of mind honestly and we get to monitor everything that takes a long time to work.
Uh you need to make it a process. it doesn't you know come um instantly but assign owners to track budgets and performance and we you know we usually are able to decrease our costs by finding where are the bottlenecks especially in the ETLs and and accelerating execution even slow running dashboards you know we we find that I mean you know data brick has amazing system table now that tell you everything about your users and how they're using the platform so take advantage of this it's it's a no-brainer So the results honestly all of this work needs to impact the business and you know I I can't really dive into real hard numbers but I'll say that live ops is crucial for our success uh merchandising just the merchandising efforts that we do on a daily basis you know close the gap often with our revenue target which is 5 10% revenue lift monthly you quickly make your money back just with that um we get fast and accurate AutoML forecasts when we want to make monthly revenue forecasts.
AutoML isn't always, you know, perfect, but I really like that it runs a lot of time series models and checks for performance and recommends the best one. And then sometimes the best one is a profit-based model and then you just get the notebook and add your live ops events, seasons, and holidays as extra parameters and then you get an even better model uh fast. This allows us to set ambitious goals and have our teams really motivated to hit the target.
We're really clear with our teams, you know, if we if everyone hits the target, we'll have bonuses. That's part of, you know, extrinsic motivation. It's it's powerful. For marketing, it's about right sizing our investments. Uh resulting increased profitability. If you overspend, you never see your money back. If you don't have data and effective data operation, uh you're you're toast in this day and age with marketing and customer support.
Uh it it takes time, you know, to understand and analyze all that pile of of usually complaints. We very rarely get love letters, although some people have gotten married and have met through playing our games. It's beautiful story, but you know, it's not the typical story. Uh we're very bullish on using everything AI, you know, all the agents to summarize all these these these this feedback that we get uh to make our games better. you know, a one-star review really if we can avoid the few one-star reviews that we get due to crashes and other issues, it it results in higher organic install pipeline as well, which is really a crucial to our success.
So, you know, that's that's the end. I I and what I want you to remember here is that coming to data brick using dbt for us was a lot easier than we thought. You know, we did it in six months and now we have to do it all over again. Leaving Jam City's lakehouse, coming to our own lakehouse, and we're confident that we're going to do it in in six months or less. Depends when the game teams implement the new tracking SDK, but that we can control, you know, and I'd be, you know, grateful for your survey.
Finish, you know, please rate the talk and we have some time left. So ask all the tough questions. You know, I'm more than happy to answer them.

Learn more about the Databricks Data and AI platform.