Federated Data Analytics Platform

Summary

• Federated platform pattern: Databricks internally built a config-driven data pipeline framework using Metric Store and UC Metrics to democratize BI ownership while maintaining central governance.
• Massive engineering efficiency: The federated approach reduced effort from 100 traditional data engineers to just 2 platform engineers—proving the scalability of the pattern on the Databricks Data and AI platform.
• Distributed contributions, unified control: Enable hundreds of product teams to contribute metrics and BI while a small central team maintains quality, consistency, and performance standards.

Federated Data Analytics Platform

Talk by Neo Ni, Senior Software Engineer, Databricks ; Rohit Mathews, Senior Software Engineer, Databricks
Are you struggling to keep up with rapid business changes that demand constant updates to your data pipelines? Is your data engineering team growing rapidly just to manage this complexity? Databricks was not immune to this challenge either. Managing our BI with contributions from hundreds of Product Engineering Teams across the company while maintaining central oversight and quality posed significant hurdles. Join us to learn how we developed a config-driven data pipeline framework using Metric Store and UC Metrics that helped us reduce engineering effort — achieving the work of 100 classical data engineers with just two platform engineers.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

Chapters

• 0:00 Introduction and motivation
• 2:16 Defining the problem space
• 6:49 Performance and reliability
• 10:20 Implementing best practices
• 13:48 Addressing complex logic
• 17:50 Standardizing with tracing
• 20:23 Scaling bottlenecks
• 24:46 Transitioning to a platform
• 27:24 Metric definition architecture
• 31:37 Optimized deployment engine
• 34:19 Final results and ecosystem
• 36:18 Summary and conclusion

FAQs

What is a federated data analytics platform?

A federated platform allows distributed teams to contribute data and metrics while central governance teams maintain quality, consistency, and performance standards.

How did Databricks reduce 100 engineers to 2?

By building a config-driven framework using Metric Store and UC Metrics on the Databricks Data and AI platform, enabling product teams to self-serve while automation enforces quality.

What is UC Metrics?

UC Metrics (Unity Catalog Metrics) is a system on the Databricks Data and AI platform that tracks metric definitions, lineage, and governance across your entire organization.

Can this pattern work for other companies?

Yes, this federated pattern is applicable to any organization with multiple teams contributing to shared analytics infrastructure and BI initiatives.

Full transcript

0:00 Introduction and motivation

Hello and welcome to data and AI summit. together with my co-presenter Neo, we'll be talking about what a federated data analytics platform is, why we built one at data bricks, and why it might be the right fit for your organization. Let me take you back to 2001. An engineer named was working on a small site project. A web-based email client. The goal just to be able to search through his old emails. Nothing too fancy or ambitious.
It was a quiet attempt to solve a problem that he was facing daily. But when his co-workers started using it, something amazing happened. They loved it. They wanted more. They wanted it to be faster. They wanted better storage. They wanted better threading. And just like that, little side project became Gmail, a tool that redefineses how we think about emails. But why am I talking to you about Gmail at a talk about data pipelines?
Raise your hands if you have ever been part of a team putting together an ETL pipeline or a reporting stack. Phew, at least I'm in the right conference room this time. All right. Uh, now keep your hands up if you worked on it for, let's say, 3 months. Okay. How about a whole year or two? adding new features, augmenting it, tracking new metrics. Great. Neo and I need your CVS. Please leave it with us because you might have built the Gmail of ETL pipelines because just like Gmail, a successful ETL pipeline never really stays small.
People always want more from it. And Data Bricks was no exception to this. Stakeholders always wanted more. Some background here at Datab Bricks, one of our core principles is to let the data decide. More than any other company that I've worked for, data bricks strongly believes in removing the guesswork out of decision-m I mean we AB tested which AB test framework we want to use. To that end, a team of specialists at data bricks put together a proof of concept pipeline with the goal of answering the question, what if I wanted to know the adoption and usage of every product that data bricks ships and every single product capability within. a data bricks that meant instrumenting over,500 different product capabilities across hundreds of microservices interacting uh in a complex and nuanced fashion.

2:16 Defining the problem space

The proof of concept pipeline generated a table that did just that. Let's take a look at a couple of examples. Meet Anthony. By the way, this is what chap GPT thinks a member of the finance team at data bricks looks like. wants to understand the evolution of usage of a set of SKUs data bricks delta delta life tables and streaming. He also wants to understand the weekly active use weekly active users associated with the same.
Then there's a product manager for the SQL gateway team. We all have a at our company. Her questions are never that simple. team has just launched the ability to schedule SQL queries using the job service. And she wants to understand adoption of this new product capability. Ideally, she would like this data broken down by serverless and non-serverless. And she's only interested in those jobs that have processed at least 30 GB of data.
No weekend hobby jobs need apply. To answer question, we need to be able to reconstruct the life cycle of a workload from logs of hundreds of microservices, figuring out what services were involved, in what order, and how they interacted with each other. Databicks customers run about a half a billion unique workloads on data bricks daily. This means combing through about a pabyte of logs to answer that question. Each one contributing a small but important part of the full picture.
If there was a question you had spanning any combination of the hundreds of microservices at data bricks, the table generated by this proof of concept pipeline was built with the vision of answering it. And just like Gmail since inception this table gained massive traction at the company. Today this table as a single acts as a single source of truth for numerous use cases including sales compensation uh revenue forecasting abuse detection and much more.
It can provide adoption metrics on uh key it can provide key insights on adoption metrics like daily active users or weekly active users associated with a newly launched product capability, the number of new workspaces using a product on any given week or the percentage of revenue associated with any product line. Most organizations facing hyperrowth the way data bricks did pivot in one of two different directions at this point.
They either empower their product teams to build their own ETL pipelines or they hire an army of data engineers to constantly onboard new metrics to this platform. The former approach leaves you with siloed data sets that are hard to reconcile and the latter just doesn't scale. The pipeline at data bricks was headed down a combination of these two paths. I'm here to tell you the story of how we were able to pivot to a power this table using an army of exactly one data engineer.
The key was to adopt a platform mindset to hypers scale. But it wasn't always this rosy. Let me take you back one and a half years to when this table was powered by the proof of concept pipeline. First, it took 48 hours after the end of a business day to generate insights for that business day. Okay, it meant teams couldn't react quickly to changes in trends. Second, performance was deteriorating rapidly. Jobs that were once taking five hours now took a staggering 15 hours or more just from organic growth in data volumes.

6:49 Performance and reliability

More on the details of this later. I should also note that we had strict SLAs around data freshness for all the tables at data bricks. Anytime a table was not delivered in a timely fashion, it was counted as downtime. This table then time of only about 50%. No surprises here. If a 15-hour job fails at the 14-hour mark, you have set your pipeline behind by at least 14 hours. And that is assuming the job succeeds at all.
On top of that, the metrics from the reporting pipeline had an error margin of about 5 to 15%. This was on account of the complexity of cross referencing logs from these hundreds of microservices. Again, more on that later. Yet, despite this limitations, this table had gained massive traction at the company. It was answering so many valuable questions that people couldn't wait to onboard instrumenting their platform capability onto it.
The only real bottleneck, data engineering resources to keep up with the demand. So that's where we started. Delayed insights, long runtime, unreliable uptime, and a rising demand that far outpaced our ability to scale. In the coming slides, Neo and I will talk about how we used a platform mindset to hyperscale this platform from a proof of concept to a truly robust self-service and federated platform. As we progressed, we saw monumental improvements across the board.
We went from a pipeline supporting 100 metrics to over,200. Our system uptime skyrocketed from a mere 50% to a robust 99. 9%. The pipeline runtime end to end decreased from 24 hours to just under 6 hours. Our largest job, the one that I told you took 15 hours was now completing in less than 1 hour. All these changes led to a substantial 90% cost savings. And the best part, it didn't take an army of data engineers. It just took the right mindset and one data engineer.
These metrics aren't just numbers. They represent a transformation. One that fostered a true datadriven culture at data bricks. I would now like to invite my colleague Neo to kick off what our journey looked like. what could be resulting in this issue. Let's look deeper into the product architectural diagram that we put together after reviewing the code. So here is what we discovered. The PC pipeline was indeed a proof of concept pipeline.

10:20 Implementing best practices

The team had built exactly what the stakeholder was looking for, but not in a way that was meant to stand the test of time. The job didn't scale as expected. Doubling the compute resource didn't really double the speed. In fact, a 2x in data volume increase would cause a 4x in the job runtime slowdown. Meanwhile, the data team felt huge pressure. The pipeline got massive adoption that we didn't expect, but it was built by a specialist team that disconnected from the product team.
The product team could just go and change the content or the schema of their locks at any time and put a rug out from under us at any time. The incentive were clearly not align. Let's get serious for a moment. The architecture we inherited was a beast. 65 different job written in Pispark, Scala, SQL. We jumped in just in time and in later we'll be dealing with business logic written in R or even worse. Each job cross referencing logs from various of product services transforming that raw data into over a 100 intermediate table before delivering this gold table that everybody reads from.
At this point, we knew it was time to roll up our sleeves and apply some standard data engineering magic. With the goal of improving the uptime, we apply this classic data engineering playbook. First, we introduce dev and staging environments. This let us launch change safely and spot issue early. Next, we automate all the job batching and back fields. Instead of running a year-long job in one go, we break it up into smaller rolling window, something like amounts of 15 days at a time.
Next, we made sure that all the table were optimized before the downstream job touched the data. We reduce the union failure by having each job write to at most one table. That way, if a job failed, we will have lesser time lost by retrying the same. Last but not least, we put everything under source control and database terapform provider so that we always knew what changed and when. But here's the key point. You don't want to reinvent the wheel and apply this best practice by hand for each of the job.
That solution doesn't scale and it's easy for things to force real quick. What you need is a platform mindset, a way to package all this best practice into a standard container that can be reused across every job. That way all the new pipeline benefit from the same reliability right off the blocks. This change along with this platform first approach made a real difference. With them in place, the uptime of our pipeline shoot up from 50% to 90%.
All right, so we have a dramatic shift in the pipeline uptime, but the job was still very slow and the business logic was still very complex. Why? Let me try to answer it. But before that, let's do another quick poll. How many people think of measurement in terms of kilometer and centimeter? Great. A lot of engineer around the world. How many people think of measurement in terms of interest and fees? Looks like we got some fans of Home Depot and in home project DIY project at home.

13:48 Addressing complex logic

How many people think of measurement in terms of nanometer? Great. You guys are in the wrong conference room. The nanotech talk is two block two two block down that street and also you're here early by about two weeks. People think differently and that's exactly what happened when teams are generating their logs. Looking at the logs emitted by teams is like looking at the data collect in Celsius, Fahrenheit, meters, miles per gallon or whatever other combination you can think of. team are not speaking the same language and semantics and cross referencing that logs require building complex huristic logic that is difficult to comprehend in the best case and easy to break in the worst case scenario.
At this point you probably want to know what that expression could be. This, believe it or not, it's a simplified version of what the join class look like when cross referencing the logs. When when we try to stitch logs together, the code turns into a mess of substream and reject expression. Honestly, don't bother trying to read it. The point is this mess was everywhere. Over 60 file packed with this messy logic. Chaotic, but not without purpose.
But you might be wondering why it gets so messy in the first place. Let me break it down with a real example. Imagine a user kick off a workflow in their brakes. The job service emits locks and tag them with a job ID. Let's say job 123. Then they hand off to service A which locks using its own ID. Then the service B jumped in managed by the techiest team at their bricks whose ID are cryptic something like spark 471 123 attempt 2 well it has a start to tail and they want to leave no ambiguity.
Finally the service C jump enters into the picture. It it emits locks too but with no tag or context that trying back to the original workflow. This lock became often locks impossible to trace back to where it came from. When it's time to actually connect the dots, we end up with this messy logic. This is an actual screenshot of the cross referencing logic in action. As a data engineer, the minute you see a substream and reject expression in a join, you know you are in deep trouble.
If you see a combination of substream reject expression coles, you know for certain that you can put together a five course meal between the two round testing of your same code. If you are looking at the spark UI and trying to figure out where the data skew came from, you are not doing your team any favor. It was also not uncommon to see that the logic was so complex to join that it will require building and maintaining a lookup table to just cross referencing this logs.
And also what happened to the to the metric accuracy when you cannot re reliably cross referencing this logs instead of stitching logs together with this messy logic. What we really want is a unified standardized join key. So what do we do? Almost everyone in this room understand that storage is always cheaper than compute. If adding extra data to your loss locks can simplify your system, do it. Your finance team would appreciate that.

17:50 Standardizing with tracing

With that in mind, we create this concept called analytic context that can be reused by all the services. any contextual data such as the detail of the job, the workspace, the cluster would be automatically locked by the logging framework. Now the PA team would only need to worry about logging tags associated with their own services. Let's see what this will look like. In the same example of user triggering a workflow, the job service emits locks.
The self job service locks the workload ID as job one two three. Instead of the product team having to worry about persist that context, the logging framework is responsible for propagation this context across microservices and persist that context in every single log message. Here the workflow ID is locked in every product services. It is important to know that the analytic context must be set at the right granularity.
For instance, simply set the workspace ID in the context provide little to no information. It must be it must be set at the granularity that people want to do analytic ads at ERIC. Typically, this is every time a user triggering a workflow. By propagating and logging the universal analytic context, we enable the new way to correlate logs across all the product services. Now, regardless of which service generates that logs, we can easily trace every event related to a single workflow end to end using a single standardized format.
This not only streamline our business logic but also dramatically reduce the time spent on debugging an analysis and cross team investigation. This was a risky change that took a lot of testing to roll out but in the end the saving were clear tremendous cost saving and huge improvement in the operational uptime. In addition, there was a drastic improvement in the accuracy of those reported metrics. This can be credited to the replacement of heristic logic with this standardized join key.

20:23 Scaling bottlenecks

The explicit workload tagging enable us to cross reference accurately with those locks. This looks great and to be honest, this would have been a decent place to just leave the pipeline ads. The destination goal table was by far the most popular table at a bricks and after all it could answer any combination of adoption metrics no matter the products or the product features. But what did it actually take for a product team to onboard a new matrix to this most popular table?
Well, here is here is what the onboarding process really look like. It start with a product team founding a tickets. The data engineer would sit down with the product manager to dig into the requirements. Then the data team would collect all the requests and pick the most important three to five to prioritize. Then came another interview this time with the product engineer to figure out if the necessary logging were were in place.
If so, the data engineer will hunt through over 60 file to add the new code. Then we will run intensive test to make sure that nothing else break. We launch this change. We back field the data. This process require a lot of handoff and back and forth. The time from the product launch to actionable insights. It's about a quarter. This goal table was our northstar. The single source of truth for adoption metrics no matter the products or the product features.
But data bricks was expanding very fast. New dimensions and new products were coming from every directions. Organia growth, mergers and acquisition. Every new products, every new team meant a new matrix that need to be land in this table. Whenever a matrix was missing, people consider it as a data quality issue and all eyes turns to data team to fix it. We became the bottleneck. And here's the reality. Despite all the demand, we could only put at most one half engineer for this onboarding process at any given time.
Keeping pace with the company's explosive growth was a constant challenge. Beyond that, owning this gold table made us the omination engineer. We understand every product inside out and how every feature was measured. But that omnition was dangerous. The whole company depend on one or two engineer to understand how every product worked. If that person was out or just for vacation, the entire on boarding process could freeze.
The risk of being single point of failure had never been so real. And as a bonus, our omination engineer got another job title, senior slack data responder. Between maintaining this gold table, shipping new feature and onboarding new metrics, they will also expect to have to have answer for every single data question anyone ask. No matter how random, if you are in a data team, you know the feeling. Someone come to you pins you on Slack with a question that you have never even thought about before, but somehow you are expected to know the answer.
What we had at this point was a very pretty house of cart. People love using it, but it came with big risk. As things became more complex, adding new matrix became increasingly dangerous. A single mistake like a incorrect join or logic error could breaks matrix everywhere. And what if the prior team suddenly changed the way they lock data? the whole house could collapse since the product team didn't know how their locks fed into our matrix.
Any change put their quality at risk. Soon it became clear to the team that in the current model it will require an army of engineer to maintain and update the pipeline. And here's the kicker. Even with all those engineer, our matrix will still be fragile and our definition will still fall out of sync. Why? Because the ownership and incentive was still misaligned. We had to make a platform. The platform had to be self-service.

24:46 Transitioning to a platform

We had to federate the ownership. So, how did we get there? I will turn it over to for the best part of this talk. Thank you. All right. So what does it mean to be self-service? For one, teams should be able to onboard metrics with little to no intervention from the data team. Second, it should be easy for teams to onboard these metrics. Finally, it should be possible for stakeholders to discover these metrics, look up their definition, and find out exactly which team to ask questions to instead of bothering me.
What does it mean to be federated? Like any good system design, there are functional and non-functional considerations. Functionally, we would want the ownership and definition of these metrics to be federated to the product teams. In an ideal world, you also want to be able to capture the costs of evaluating these metrics and assign it back to the individual product teams. However, there are non-functional considerations as well.
Onboarding a new metric should not break any existing metric definitions. That is, the metrics should be isolated and self-contained. There should be correctness guarantees. The platform should be able to scale horizontally as more metrics are onboarded without compromising on job or SLAs. As a platform, it is also our responsibility to ensure the teams cannot onboard arbitrarily complex metric logic that can stall the entire pipeline.
There are a number of other considerations as well like how do you handle backfields for a newly launched metric. We won't have time to go into all of this today, but we'll see how adopting a platform mindset will help us hypers scale. The first step was to make it modular. Instead of a PISpark notebook or Scala notebook or a notebook evaluating n arbitrary metrics, each metric definition was encapsulated in a separate file.

27:24 Metric definition architecture

Each product team would then own one of these metric files and product teams could add more files in a self-service fashion. Let's take a closer look at what one of these files looks like. To keep things modular and safe, we built a config-driven interface. Essentially, a structured YAML contract that all metric definitions must follow. Each file starts by declaring the name of the metric. In this case, a serverless.
Teams can optionally provide a detailed description field which comes really handy if you plan to later layer on LLM capabilities. Imagine going into the datab bricks genie space and asking the question, "Hey Genie, can you tell me the total number of DBUs associated with SQL queries scheduled using the job service?" Break that down by serverless and not serverless. Oh, and only count jobs that are processed at least 30 GB of data. would be so happy.
Next, we define the source table. In this case, main. clustri. cluster events. We also define fields like date call and the workload ID call which tells the parser how to extract all these columns from the table. Next, let's look at the feature definition. This is where we specify exactly how the metric is calculated. In this example, it's a simple boolean aggregation. Select bool_rr of the is serverless tag from the cluster events table where the operation is cluster initialization.
The structure here is very intentional. We use specific fields like select from and where this isn't just for tidiness. It is a crucial design choice. Teams can't write arbitrary SQL and that constraint is key. It is the only thing standing between you and the eager intern onboarding a metric definition that cross joins 10 different tables. Great. Now that we understand how the metric is evaluated, let's take a quick second to talk about the engine.
The engine is written exactly once and is thoroughly unit tested and as such adding new metric definitions can never break existing metric definitions. Finally, we have the ownership section, aka the don't bother me section. This is where fields like data scientist owner and nge team owner are clearly defined. If you have questions about the isserverless metric, you know exactly who to reach out to. The nench cluster controls team for product related questions or neo. ndicks. com for complex data science questions.
This ownership model puts responsibility into the hands of the teams that understand the data best. It reduces ambiguity, streamlines communication, and empowers teams to own their metrics end to end. Finally, the feature explorer can be used by anyone at the company to discover all the metrics that are available, figure out who to redirect questions to, look up the definition in source code, and look up the SQL definition of the same.
In addition to offering users a clear and structured way to define metrics, we provide an onboarding notebook. Here, users can experiment with feature definitions and preview the resulting insights in terms of dollars, DPUs, or whatever other metric they might like. This hands-on environment helps teams validate the accuracy of the metrics and build metric definitions that meet their specific business needs. All right, now that we understand the highle interfaces, let's take a peek under the hood. a 10,000 ft overview if you will.

31:37 Optimized deployment engine

It all starts with the metric configs which are defined in YAML. The config parser validates the configurations to make sure that they're correctly formatted and adhere to our interface specifications. Then comes the CI stage. This is where we run triggerous unit tests and integration tests such as does the config evaluate to a valid SQL? Were all the columns and tables referenced in the configuration existing in the metas store or did they make typos?
Once these tests pass and the PR is merged, the deployment engine kicks in to intelligently create as many jobs and tasks in the orchestrator as necessary. What does it mean to intelligently create jobs? Given all the feature definitions, we have a few different options. On one end of the spectrum, we can create a really large job that evaluates all the metrics, but we have already seen the pitfalls of having a very large unit of failure.
On the other end, we can evaluate each metric in its own job. However, imagine the scenario where you have seven metrics sourced from the same table. You now have to parse the table seven times in seven different jobs. That's a little bit like sending each member of your family in a separate Uber to the airport. Since we use the strict interface, we can do something a little more intelligent. We can easily figure out all the metric definitions that can be evaluated in a single pass of the table.
And we just have to dynamically create one job for each set of metrics that can be created uh evaluated in a single pass of the table. All right, let's take a look at what the onboarding process looks like now. Remember the Arduous quarterlong process? Look at it. Now, one key improvement you would have seen is that only the product team is involved. There is no data team anymore. First, they create a config file. Next, they use our onboarding notebook to test and validate their metric definition. make sure that it meets their business needs.
Finally, they create a pull request and merge their changes into the system. And just like that, the metric is live, ready to generate insights. It's a dramatic shift from a quarter down to a single day. This slide encapsulates the achievements we made so far. Since launched, we scaled from 100 metrics to over,200, demonstrating its broad applicability and showcasing the platform's ability to grow with our needs. Thanks to metric isolation, our pipeline uptime has gone up from a precarious 97.

34:19 Final results and ecosystem

5% to a robust 99. 9%. Most impressively, we slashed the time to onboard a metric from an entire quarter down to a business day, fundamentally changing how quickly we can react to business needs. And we were able to do this with just one data engineering resource. All right, this is a great place to end the journey. Product teams that understand their features and the logging are in charge of metric definitions and their upkeep.
The data team which knows how to build robust platforms that scale is in charge of the same. The incentives and core expertises are aligned. But this is where the exciting part begins. You can now begin to add functionality that is written once and can be leveraged by anyone on the platform. Let's look at a few examples. We provide anomaly detection, forecasting, and explanability as a service so that we have an automated way of finding out when things go wrong, when it'll go wrong next, and which in turn to blame when it does.
We provide dozens of CAN metrics like daily active users, weekly active users, revenue out of the box for all the teams onboarding proper product instrumentation onto the platform. These aren't just nice to haves. They're key differentiators of a platform. We now have an ecosystem where we can author features once and benefit the entire organization. So let's recap and see why the platform model benefits every persona at your organization.

36:18 Summary and conclusion

Each team onboards metrics through simple isolated configs. The metric engine is written exactly once and thoroughly unit tested, meaning that new metrics cannot break existing ones. Behind the scenes, the platform is able to stitch together the life cycle of the workload using the analytics context. The intelligent optimizer is able to go through all the definitions, figure out exactly which metrics can be evaluated in a single pass of a table, and calculate all of them in one job.
All teams even with this binable config get dozens of metrics like daily active users, DBUS, dollars for free. The data engineering team is now focused on just add best practices. And the data engineering team can finally start building practices that meets with their core expertise. Management is happy because they are not spending all their time interviewing new candidates. Product teams and PMs are happy because they are no longer bottlenecked by the data team and they have reliable metrics.
The data team is happy because they can finally put their ML Corsera certificate to use instead of talking to PMS and responding and questions on Slack. And the org is happy because now we have a model and processes that scale with the company. So that is our journey from a single pipeline held together by tribal knowledge to a robust federated platform for scaling and ownership. We didn't need an army of engineers to get here.
It took just one and the right mindset. If your organization is feeling the growth weight of rapid growth, siloed metrics or data team bottlenecks, now is the time to shift your mindset. Build pipeline Stop building pipelines, build platforms. Thank you.

Learn more about the Databricks Data and AI platform.