Skip to main content

Building a Self-Service Data Platform With a Small Data Team

Summary

  • Self-service analytics: Dodo Brands empowered non-engineers to create over 1,420 data jobs, with 90% auto-generated using Jinja templates.
  • Scalable architecture: A pizza and coffee company with 1,200+ locations scaled to thousands of weekly active users without expanding data teams.
  • Tool democratization: Teams across the organization independently built analytical pipelines using Apache Superset and Databricks as the backbone.

Building a Self-Service Data Platform With a Small Data Team

Watch: Building a Self-Service Data Platform With a Small Data Team
Talk by Evgenii Dobrynin, Senior Data Engineer, Dodo Brands ; Gleb Lesnikov, Head of Architecture, Dodo Brands
Discover how Dodo Brands, a global pizza and coffee business with over 1,200 retail locations and 40k employees, revolutionized their analytics infrastructure by creating a self-service data platform. this video explores the approach to empowering analysts, data scientists and ML engineers to independently build analytical pipelines with minimal involvement from data engineers. By leveraging Databricks as the backbone of their platform, the team developed automated tools like a "job-generator" that uses Jinja templates to streamline the creation of data jobs. This approach minimized manual coding and enabled non-data engineers to create over 1,420 data jobs — 90% of which were auto-generated by user configurations. Supporting thousands of weekly active users via tools like Apache Superset. this video provides actionable insights for organizations seeking to scale their analytics capabilities efficiently without expanding their data engineering teams.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

FAQs

How did Dodo Brands reduce data engineering burden?

They created a job-generator tool using Jinja templates that automated 90% of data job creation, enabling analysts and data scientists to independently build pipelines without manual data engineer involvement.

What BI tools does Dodo Brands use?

The platform leverages Apache Superset for business intelligence, supporting thousands of weekly active users across the organization with real-time analytics.

How many retail locations does Dodo Brands operate?

Dodo Brands operates over 1,200 retail locations for their pizza (Dodo) and coffee (DrinKit) brands across 26 countries, primarily in EMEA (Europe, Middle East, and Africa).

What is the scale of Dodo Brands' data infrastructure?

With 40,000 employees, $1.5 billion in annual revenue, and data from millions of daily transactions, the platform handles e-commerce, ERP, point-of-sale, and digital payments systems.

Full transcript

0:00 Company overview and vision

Uh. I'm head of architecture at data brands and with me is debrin senior data engineer also data brands. Uh so what exactly is do brands? Uh we are relatively young food tech company comprised of two brands do pizza and drink it. Uh do the pizza is obviously pizza company. Yeah, we make pizza and drink it is our new coffee startup which by the way um we hope to open this year in Los Angeles. So if you're there um maybe even in Hollywood or something somewhere.
So um if if we open there and you see it um please uh take a look inside. So um as of today we have um around more than 100 more than 1,400 pies and coffee shops. We operate in 26 countries primarily in EMI region, Europe, Middle East and Asia and Africa and Asia. Um and our last stoping was in Mongolia. Uh we have more than 40,000 employees and $1. 5 billion in annual revenue. Um but our secret or not so secret sauce is that we are food tech company or as we sometimes say we're cyber company which is uh like we are offline and online company at the same time you know like cyber electronics and yeah so we build all of our software internally uh we have more than 350 engineers u a lot of services micro and monolith whatever and uh we have a quite a bit of requests per second because we handle all of our e-commerce and stuff um internally and ERP we collected a huge amount of data um and continue to collect and um but this brings a problem that we we have a really huge scope.
We have to build our own ERP, point of sales, fulfilling, accounting, software, HR. Uh we build our workforce management internally. We try to at least uh we build our point of sale. Um actually our point of sale software is an electron app with React UI. Um and we built it like 10 years ago already. So, and we uh do our updates to our point of sale and fulfilling like almost every day. Yeah, we try to do almost everything web- based in order to you know to be fast.
Um and not only that and we also have e-commerce, we handle our own digital payments. We have our own like digital payment system inside that uh handles uh around 30 integrations. Um and on top of that you are required to have right now ML and GI and all of this and this is all very expensive in also in terms of engineers. Um um and on top of that you also have to have analytics because you know the key point of building our own software is not only to you know improve the more work and you know do some disruption in uh uh in terms of food tech but also that you can u so so you could do an analytics on top of all of that data uh because every pizza every click is um uh we save it basically.
So yeah as a I'd like to say that we our system is um 50 OLTP uh you know online translation processing and 50% analytics. Um yeah that's why our first data stack was just our applications written in C. We wrote our first reports in uh using 100% GMO free organically sourced handwritten SQL queries with freerange D3GS and HTML. So yeah, that was our first reports and um and run them directly on our master database which regularly caused the times down times because of the raw queries and stuff you know and the uh uh high load and then in 2017 we decided that we have to be better and um we have to use something you know like proposely built for analytics and we utilized Azure SQL that warehouse um yeah then it was not in Syninnapse or whatever they have right now.

3:33 Data platform architecture

Um we also utilize data factory and PowerBI. Uh and it was fine but it was fine for only you know our inside analytics because uh we have a little bit of um interesting use case because we have because half of our uh application or half of our system is analytics. So we have to embed those analytics or like resell it. So, and PowerBI even PowerBI embedded wasn't uh going to cut it because it's really expensive if you have to show it to 40,000 people or like or even 10,000 people because you have to pay for every license and stuff and it's really expensive.
So, we um we decided that we have to do something something better. Also, I actually used data bricks in 2017 when it was like really really fresh and it only had like notebooks and ML and we used it for notebooks and ML. But uh then after some time we settled on uh we discovered some things uh did a few PCs and settled on data bricks as u our um you know for everything uh Azure a hubs for data movement uh Azure data explorer for serving data and Apache superset for our um you know for embedding for our dashboarding and abending it and we maintain our own you know fork with that we like integrate directly in our applications because as I said um writing handcrafted reports in HTML or JavaScript is kind of um it's just expensive.
So we wanted to you know take something open source integrate it in our system to allow everybody almost everybody to do a dashboard and uh you know deploy it to our production to everybody. Yeah. And because of the huge scope as I said we have a lot of the engineer heavy uh we wanted to create a platform to not uh also not to you know uh spend time as I said so we wanted to create a self-service data platform which will enable u users to create data jobs pipelines dashboards independently with uh almost no involvement from data engineers.
So they will create content and we will not spend any money on that and we will stay lean and by lean I mean u we only had and still have not much engineers. So we only have four net engineers for I think 250 I think or 200 uh engineers uh overall including front end back ends. So for our data team we only had four data engineers one back end and one front end and backended front end engineers were almost uh specifically for superset and uh you know serving APIs and the idea was that data engineers don't create data jobs or pipelines they create a platform to do it and we will utilize code out of generation engine plating to do it and uh most of data jobs and pipelines could be created by the users users themselves.
Um, if you're interested, if you know want to know more about how we integrated Apache superset into our platform and embedded it, um, here's the link to talk to our talk. Um, three years ago already. Yeah. And here to tell you how did we meet our goal is. Go ahead. Uh, yeah. Thank you. Uh, hello everyone. Uh. I'm senior data engineer in uh do brands and this is how all of this uh actually works. So uh one more time what is our main purpose?

7:52 Technical design and layers

Our main purpose is to be data driven but we simply uh cannot afford it to hire a giant data department and I guess this is this could be applied to a lot of businesses. But does this mean you cannot be data driven if you cannot afford to hire hundred of data engineers? And the answer is no. You could you could be data driven. uh you simply need to build a self-service uh platform which uh would make it possible for your users to make all these uh data related job themselves.
So we'll be able to create the data data jobs, data pipelines, data dashboards by themselves with very little or no involvement from the uh data engineering team. Uh this is our architecture. Uh as you can see it's a little bit simplified uh and it has only two main uh data sources. Of course we have much more of them. We have external APIs. We have Google Sheets etc. But these two data sources are let's say major because 90% of our workloads um sourced from there.
So the first one is Azure cloud MySQL database our oper operational transactional database and the second source is our microservices and as GL mentioned we have more than 80 of them. Uh both of these uh sources provides an event. So we mostly work with our data as an events. These events could be directly pushed into Aszure event hub topics by our microservices or they can be uh they can be sourced from MySQL database and transferred by CDC tool called called called Disio.
Uh Azure event hub topics in this case are kind of middleware here and the events from Azure event hub topics uh would be consumed by our uh spark streaming jobs uh from data bricks. Uh we have a medallion architecture there. I will speak about our uh data architecture with more details lately. Uh the most probably important point here is this uh little red arrow when the gold business data are transferred into uh Aszure data explorer MPP database and why would we use that?
We use this specific database because uh well basically for two reasons the first one is the cost effectiveness because uh data explorer are quite cheap to use and the second reason is speed because it's very very very fast. And the last piece here is Apache superset. As GLP mentioned, we previously have uh PowerBI tool for um for our visualization and dashboarding purposes. But to be honest, it's very uh it's very expensive.
Uh you need to pay for a lot of licenses. So this is why we decided to switch into superset and forked it and develop our own version. And this is our delta lake architecture. To be honest, this specific schema is a little bit outdated because now we are during the process of getting rid of landing layer when we store row data. We're trying to cut it off because now uh with a cloud architecture with a cloud-based architecture, you can simply use capturing features of your message broker.
For example, you can just store your uh row data into event hubs uh capture or kinesis capture uh if you use AWS or any kind of uh cloud message broker. Uh so in our case the first level would be bronze and the bronze layer we applying the schema we perform merging data and duplication. So we basically made it a a little bit human readable at least and the relationships between landing and bronze are usually one to one.
The second layer is silver and in silver we applied our data modeling technique. In our specific case it's data vault but uh it's not necessary to use data vault here. it uh could be I don't know it could be Kimble star schema snowflake schema inman or anchor modeling anything you see fit for your specific business case uh and here you usually has uh again onetoone relationship between bronze and silver or many to one if you would like to union some data and the last one is the gold the actual data marts business layer uh when you join some join some silver tables and uh the last step would be a serving again uh this process between uh the process of transferring your gold business data from uh from your cloud storage to your MP database to make it a little bit faster and a little bit cheaper.
Uh we mentioned our users a lot. We would like users to be able to do this and that and that. But who are our users actually? Uh I would split it into them into two groups. The first group is u tech users or data producers. Uh so it's mostly analytical teams. It could be BI. Uh it could be product analysts, data analysts. Uh or this could be our tech teammates but not do uh not data engineers. Uh and here I mean ML engineers, data scientists, software engineers.
So anyone who would like to work with data but uh these guys are not let's say major data engineers. And the second group are data consumers or our business users. Uh most of them the most important group probably are our franchising partners because we're a franchising company. We have a lot of franchising part partners around the world. Uh it's our own company management any level of it up to up to the sea level. Of course they would like to make the uh the decisions uh datadriven.
Uh our uh platform is also uh very important for our uh point of sales uh software and of course we use this for external uh external integrations via API. And if we will try to go a little bit uh to dive a little bit deeper, we have uh we based our data platform at two repos. The first one called job generator and it's uh let's call it a platform repo internal repo because as you can see only data engineering team commits here and it has two uh main purposes.

13:23 Repository and job workflow

It stores uh ginger templates for job generation and also job generator is responsible for parsing uh configs provided by users. These configs are stored into uh in a second second repo called data lake sources. So job generator would read this config parse them uh create a job and deploy this into the data bricks. And the second project is u called data lake sources. This is the user uh user repo. Uh users push this uh push their configs here and u also data lake sources provides uh some kind of user interface. uh we use GitHub action for this specific purposes and here both our users and data engineers can perform the commits and this is the typical user scenario for our data platform.
So let's say we have some user and the user would like to work with uh with the data. The data could be from my SQL operational database or this data could be in the form of events from some microservices. And the first question needs to be answered does this data aligned with NFRS with the non-functional requirements list. I would talk about them a little bit later and but basically the answer is no. The user would need to go to the responsible development team and fix it and make it aligned with NFRS.
But if everything is fine, if NFRS list is aligned, uh the user would need to uh create uh to choose proper generated template and uh deployment template. So uh his uh his job will basically could be generated and deployed. Uh he will need to push uh to create and push the couple of config files into data lake sources project and then run the related GitHub action. And that's basically it. Job generator under the hood. will part these configs uh deploy them uh deploy the uh generated job into the datab bricks and the user could validate the final results and we will go through the schema step by step.
So the first uh uh the first the first topic is NFRS nonfunctional requirements what is that actually is and why do we need them this is basically just a list of requirements your table or your event needs to be aligned with to be added into the data platform. So NFRS are kind of data contract let's say and uh the reasons why we need this is very simple we need them to make our jobs templatable uh I provided the very simple example of NFRS as you can see the points here are quite obvious you need to have a some kind of time stamp in your table or your event you need country ID if your business are country specific and you will need an event ID and the reasons why we need this are quite obvious uh it make it very easy to build a historical pipelines historical tables and uh to perform duplication operations and uh the next step uh is to choose the appropriate templates.
I provided a couple of examples here. This is the generator ginger template. I called it uh lending uh lending kafka. So this ginger template is responsible to create uh a job which will be consume simply consume events row events from events hub topic and store this into the cloud storage. And as you can see nothing very special here to be honest nothing basically to explain couple of variables couple of spark related configs Kafka related configs we import couple of common python functions for creating delta table etc.
Uh basically this is just the mix between uh a mix of pure python syntax just like here and ginger syntax. So couple of python functions kafka configs uh like uh offset topic name etc. And uh here's the Python code when we actually trigger our our Spark job. So that's it. This is this is the template. Uh this template is responsible for uh the process of job generation. And the second one users need to pick correctly uh is deployment template called lending continuous is this specific case.
Uh and again it's very simple. Just provide the ID of the necessary of the correct cluster. Install some libraries set up uh set up the timeout uh the timeout seconds the access control list and that's it. Uh the step three user would need to create and push job configs into the uh data lake sources repo. The two most important lines here are the first two uh data lake name it's basically just the name of storage account cloud storage account uh then we uh when the data will be uh stored and the template name in our case it's landing Kafka this is the template we observed previously on the previous slide and the second config again is just a couple of lines as you can see we just provided the name of the proper deployment template in our case it's landing continuously lipsynet again this one we already observed uh previously And we have a small demo of how does all of this actually works.
We will start with uh data engineering point of view. So how all of this works for data engineer. Let's take a look. And so here is our project uh the internal one job generator. And let's take a look on our templates. Let's choose one of them. Here it is again. template. We already observed this one previously. Nothing very special, just a mix between and Python. And here's the user's project data lake sources. Let's pick some job to generate.

18:22 Demonstrating job generation

We will choose some data domain B2B for example, lending layer and just choose one job. This is users provided configs again very simple. Uh this is the name of our uh deployment template where it is where is the folder and let's find it. Here it is. So we have users configs and we have our uh our templates. The first step we need to do is to install job generator as a as a dependency. We will need couple of seconds to do this.
And that's it. Job generator was installed. Uh and we have an access to job generator CLI. Let's try to create some job. Job generator generate. And the environment is B2B. The L uh data layer is landing. and the job name. Here it is. And that's basically it. The job was created immediately in a second and it was stored into generated jobs folder. Let's take a look at that. We have two files here. The first file is the Python file with your SP pispark job.
And take a look. Almost 600s lines of code were rendered created immediately in flight. Just imagine how much time would you spend doing all of this manually. hundreds and hundreds and hundreds of times even if you have some kind of example in front of you. Uh the second file is just a JSON responsible for deployment and by the way those recommendations are not mine. Uh and the step number four user would need to run the corresponding GitHub action.
As you can see we have a list of them here. Uh our would be uh generate generate and deploy job. And one more demo how does this looks from a user's point of view. So how does this works for our user? So our user is going to pick the GitHub action fill the parameters. Again this is the same we uh we did previously in the previous video just different data layer and different job name. We launch the GitHub action and let's take a look what's happening under the hood.
The Python was set up. The job generator was installed successfully. And we have our job generated and launched in datab bricks. And we even have the job ID, right? So we can go to datab bricks and take a look. Was the job actually actually deployed? Let's go to datab bricks workspace where it is. And yeah, here it is. This is our spark streaming job created by user without any involvement of data engineering team. It's successfully started and user right now is able to verify the results.
So this is the simple schema of how this actually work. You take your templates a couple of configs provided by user. Job generator would parse them. Uh it would create two files 1 py and white. json and uh job generator would push them uh to the datab to the datab bricks cluster to be executed. Uh as you can see we use rest api here. uh but you could or probably even should use as bundles because this is their more modern or more native instrument to do this.
We don't uh use asset bundles just because our project was created years ago and asset bundles simply didn't exist uh exist though time those times and uh this is the step five verify the results in data bricks uh we have our beautiful table here but it's not actually so beautiful because as you can see the all the data is binary right it's not very human readable doesn't look very useful can we somehow do this a little bit more useful so yeah this is a good question what about data transformation what if you uh need to do your data more human readable.

22:39 Data transformation steps

Could you do this? The answer is yes. All of the uh all of you need to do is to add more config called transform. py. It consists of two parts. The uh spark schema and the simple uh transform function of python. This is how it looks. Uh this is the first part of transform. py simple spark schema column names column types. That's that's basically it. And this is the second part the transformation function. we drop couple of them, select only the necessary, rename couple of them, etc.
So this is the bad news here. Your user would need to learn and to understand some basic level of Python to do this because your business logic is the only part of the job which is not boil boiler plate. It's uh it's very differ from time to time for this specific business case. The good news, the level of Python here is really basic. It's really elementary and if you are an enthusiast of modern generative like data bricks assistant, like chat GPT, like cloud, this is an amazing place to apply them.
They could be extremely useful for your users. And the only difference between landing and other data layers here as they simply add this transform file in the end of our template just like uh you can see at the uh this slide the last line of it. Uh and again the entire schema didn't uh didn't change so much. We only added uh one config here and the entire schema is still the same. We only added transform. py for bronze and silver layers and we replaced this with query SQL config on the gold layer.
We prefer our users to work with SQL on the gold layer because basically this is the most native way for them. They are mostly SQL experts not Python experts. So let's say this is their native language. But still the entire schema are still the same. Let's say about pros and cons. Every solution has its pros and cons. So is ours. The pros are the pros are quite obvious. This kind of solution could be implemented and maintained easily by a very small data team.

24:28 Results and conclusion

As a GLAB mentioned, we only have four anata engineers who implemented this project from scratch. Uh this approach aligns very well with u data mesh because one of the core principles of data mesh is to have your own uh self-service data platform and this is basically it. Um this is the real uh full self-service because users are able to implement almost all of their jobs themselves and to work the data from the very row level to the level of dashboards or serving endpoints or uh or machine learning models or anything else any any kind of output.
And u the last one is uh it's it leads us to the significant reduction of time to data. And this specific metric as you probably know is not very easy to calculate to be calculated precisely. Uh but again just imagine writing all of these pispark jobs by yourself from time to time hundreds and hundreds and hundreds of times. How much time would you spend? What's the cons? Uh the first one is yes your users would need to learn basic python and basic g commands like commit and push and pull.
But again the level of it are very very beginner. Um the second con is this solution is partially vendor locked in because first of all it was created having data bricks features in mind uh but uh still it's partially because you can switch relatively easy to open sourced version of spark using this solution. Yes you will lose a lot of beautiful datab bricks features but it's more or less easy to do. But if for some reason you would decide to switch to another managed solution like click house, like snowflake or something like that, that would be hard to do.
You will need to implement all of this mostly from scratch. And uh the third one is your data engineering team may and it's almost 100% will become a bottleneck for data modeling uh tasks basically just because your users are not data modeling experts. you will need to uh learn them to show them how to implement data vault, how to implement Kimble etc etc and this is the result and I would ask uh GIP to share our technical and business results.
Yeah. So as of now we have more than 3,000 users in superset which um u mostly comprise of like everybody pizza managers uh pizzeria managers like coffee like point of sales people marketing franchises um developers so we like utilize we we have a lot of people coming to this dboards and uh making all these uh dboards and jobs and uh pipelines are 85 contributors But six 76 of them aren't data engineers and are not in the data platform team.
So and they they have created 1,700 jobs already and uh 1,500 are created just by uh job narrator from templates. So and 250 are you know more more exquisite handcrafted. Yeah. So and uh for business side um we were able in these years we're able to drive dozens of profitable datadriven use cases uh for example we did um minimum order um dynamic minimum order pricing uh you know based on uh the time of day and the uh location and stuff uh utilizing data bricks and points and this was done almost entirely in uh data bricks and with our jobs um and our um e-commerce uh e-commerce people were just using uh rest APIs to go to connect to datab bricks endpoints um yeah and we generated a lot of revenue from that because we were able to allow people to order from for example uh uh more uh um like from more uh u far sites from the city for example uh because without minimum order without minimum order um they were were not just able to order and now they can.
So and u we also on boarded all the business uh domains um to our data platform also because of this was really easy to do. We're just okay you have an an analytics person just um here is the ducks you can do you can do it. Um and um based on all that uh our data team has grown but not um really much but uh still have grown and mostly it has grown with data engineers inside the domains. um mostly for the data vault and data modeling uh um tasks because it's kind of hard for um for not uh you know data uh people to do all this uh you know hubs satellites links um this is really hard.
So yeah and we and we got a and we got a DevOps engineer because why not? Um right now our main driver of the cost is data bricks. Um and the of course the storage uh we spend some uh also we spend some money for superset and uh data explorer but it's u less because it's just because it's very cheap. Uh yeah. Um but uh we also um know that we can drive even even that costs are also we can we can drive it down and it's kind of easy because of the job generator because we can just you know if we need uh to utilize some new virtual machine or job compute or serverless or some other thing uh we can just uh redeploy all the jobs from uh from scratch because we have all the code for it stored in one single repository.
Sorry. Um, thank you for your attention. Um, we are have some time to answer your questions if you Yeah, I guess we have uh 7 minutes. So, if there is many uh any questions, we will be glad to answer them.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.