Beyond Medallion: Building Contextual Data Aggregation at Disney - My pick

Summary

Disney built DATOS (Data and Facts, Or piece of information), a contextual aggregation layer within the commerce domain of Disney Plus, Hulu, and ESPN that delivers 50 million aggregated records daily with six nines fidelity by combining structured streaming and batch semantics in Spark.
DATOS solves the fundamental problem that no single commerce domain—payments, subscriptions, billing—has full context about a user transaction, so DATOS assembles that cross-domain context centrally to enable consistent answers to business questions and power analytics and AI agents.
Liquid Clustering optimizes high-volume Delta tables in the pipeline, Unity Catalog provides unified governance, and a deliberate shift from use-case-driven design to data-driven discovery accelerated product delivery and created a reusable foundation for downstream consumers.

Beyond Medallion: Building Contextual Data Aggregation at Disney - My pick

Learn how Disney evolved beyond traditional medallion architecture to build DATOS, a contextual aggregation layer powering commerce domain decisions across subscriptions, billing, and payments. By standardizing data ingestion and building semantic context into every event, DATOS delivers 50 million aggregated records daily with six nines fidelity while decoupling upstream and downstream dependencies.
this video explores how structured streaming and batch semantics in Spark work together to ensure no data is lost, how Liquid Clustering optimizes high-volume tables, and how Unity Catalog provides unified governance across the entire pipeline. Discover how shifting from use case driven thinking to data-driven discovery accelerates product delivery, enables consistent answers to business questions, and creates a foundation for agents and analytics at scale.
🤝

Chapters

FAQs

What is DATOS and why did Disney build it?

DATOS is a contextual aggregation platform built within Disney's D2C commerce domain to address the problem that no single service—payments, subscriptions, billing—has complete context about a user transaction or session. By assembling cross-domain context into 50 million aggregated records per day, DATOS enables consistent answers to business questions across Disney Plus, Hulu, and ESPN without each team rebuilding the same aggregations independently.

How does DATOS extend the medallion architecture?

DATOS builds on top of the standard medallion architecture rather than replacing it, adding a contextual aggregation layer that assembles rich cross-domain facts for downstream analytics and AI consumers. The key design shift described in this video is moving from building data for individual known use cases toward building contextually rich data that many future consumers can discover and use.

What does six nines fidelity mean in the context of the DATOS pipeline?

Six nines fidelity (99.9999% completeness) in the DATOS pipeline means the contextual aggregation layer loses fewer than one in a million records, ensuring commerce decisions and analytics are based on a complete and reliable dataset. Achieving this requires combining structured streaming for real-time event capture with batch semantics to recover any records not processed in the streaming path.

How does Liquid Clustering improve performance in the DATOS pipeline on Databricks?

Liquid Clustering is applied to high-volume tables in the DATOS pipeline to automatically optimize data layout for the query patterns commerce analytics teams use most frequently, without requiring manual partition key decisions. This allows the pipeline to maintain query performance as data volumes scale without periodic re-partitioning work or schema changes.

Full transcript

[00:08] Good afternoon everyone. Before we start, a quick disclaimer, a mandatory one. 5 seconds, that's it you get. And also please do share, subscribe, and hit that bell icon, please. Nah, it's not YouTube. You don't have to do that. We are data folks, the least we could do to Databricks is give feedback loop,
[00:25] right? Please do share your feedback and fill your service. My name is Rajendra. I lead team of engineers solving data problems at The Walt Disney Company. Today's my topic is beyond medallion.
[00:42] I think we can all agree that medallion is probably one of the most significant design pattern in the data world. We love it, we use it. As a matter of fact, my colleague here, Mark, right sitting right here, presented about Disney's architecture last year. But what if business has solutions that
[00:59] even a storage pattern like medallion cannot answer? Before we dive, let's see what's there in the store. I'll start with very shameless plugging about myself. Then I'll give brief intro about Datos, our platform in Disney that we built.
[01:15] And the problem statement that it is solving. And the decisions made and the architecture of Datos. Finally, I'll share some results and the outcomes of this architecture. And I'll have some time for Q&As. So, that's me at Pike's Peak Summit. Trust me, that is much easier than this.
[01:33] Uh I have 19 years of experience, out of which last 10 years I've been focusing on commerce solutions within Disney, streaming platforms of Disney Plus, Hulu, and ESPN. I wore many hats in the commerce world, solving billing, invoice, and tax solutions. Now, I'm focusing on data
[01:49] side of things. When I don't do any of these, you can find me on any hike or trail with my camera. Now that you know me, let's meet Datos. What is Datos? Datos is part of commerce domain within
[02:05] Disney's D2C org. Datos stands for data, facts, or piece of information. What exactly do we do? We put together data to provide contextual aggregation.
[02:21] I know the context has been the word of this summit. Everywhere I go, including yesterday's keynote, many of these booths, they are talking about context. Why is context so important? Let's find out. In our commerce domain,
[02:39] context for us means different things, right? Let's say for example, user has come to a Disney Plus to sign up for a subscription. To fulfill that request, we have so many domains and services working. But, what's interesting is none of these domains have full context of the
[02:54] request. Take for instance payment. Payment just needs to know how much to charge, which country or currency to use, and what kind of payment method to use like credit card or PayPal, so on and so forth, right? None of them have full context to give full picture.
[03:11] We at Datos, what we try to do is to put together all this data together and label it or whatever you call it. For instance, in this case, where user came to us to purchase subscription, we call it as purchase payment success. What that means is
[03:27] whoever is consuming data from Datos, and if they see this purchase payment success event, it is given that someone tried to purchase subscription with us. We are able to successfully charge the user. A subscription was created, and entitlement was given. There is no ambiguity to it. There are
[03:43] no questions asked. All the data that is there to provide context on that request is sort of bundled into single content or single event. So, why is this required? What is it solving, right?
[03:59] So, this single event is powering so many use cases. We made this event so use case agnostic that anyone who wants to know the context of this request, for instance, send purchase confirmation emails, subscription analytics, payment analytics,
[04:15] finance reporting, revenue sharing, everything could be done through single event because that is use case agnostic. So, on a bird's-eye view, this is how our Datos infrastructure looks like, right? You have multiple sources, a
[04:30] compute powerhouse sitting in the middle, aggregating all the data and publishing the data to all the consumers who wants to consume. But, why do we have to do that? This seems to be very obvious, right? So, let's look at the problem statement.
[04:52] Without Datos, before this is how it was. It's not like we didn't have data. It's not like we didn't have use cases. It's just that it was spaghetti of integrations. Many downstream consumers were integrating with many upstream sources or providers of data. What it
[05:07] does is a tightly coupled engineering problem. If someone has to change something in the upstream, they have to chase all the consumers of the data and create impact analysis. That itself will take sprints and sprints of time. So, it's not scalable to start with.
[05:24] More than that, the biggest issue was source of truth. For me, everything is red. I see everything as red here. For my wife, she can say red, cyan, magenta, and so on and so forth. In data world, that cannot be the case, right? We want a consistent
[05:40] source of truth. No one should be inferring a purchase payment success as a something else in the domain. that will not give consistent answers to our questions. So, that's the biggest problem more than the engineering problem to solve.
[05:58] On top of it, data is complex. If it was simple, I don't think we'll be sitting here listening to me, right? So, data we have a lot of heterogeneous sources. Data is coming through Kafka, through Kinesis. We have JSON store, we have data lake tables.
[06:14] And some of the data through streaming data, which is fine. I think the happiest scenario. Then we have static data, like in the in our case, offers, catalogs, etc. But, the static data doesn't mean it never changes. It has its own cadence. It could change once a week, twice a day, or once every 6
[06:31] months. We still have to account for those changes. Then we have hybrid data. Think of Amazon. When you're purchasing something on Amazon, we simply go and choose our payment method and finish our checkout. Or sometimes,
[06:46] we might use a new credit card and finish our checkout. In those cases, we are sort of creating a stream and hybrid static data together and the aggregators like us, we have to keep that in mind also to union all that data and see which one makes sense.
[07:02] And last but not least, APIs. As a last resort, sometimes we have to fall back to APIs because some of the data might not be falling into data lake in the given SLAs.
[07:19] With these challenges, we tried to set out few options. We tried different approaches, right? Coming from a Java background myself, I thought I can handle this by building some applications, calling multiple APIs, and solve this problem.
[07:35] Easier said than done. Point of time The APIs were only providing us point in time. Even though they are very low latency solutions, but still they were only providing point in time. We could not do any reconciliations or backfills with the APIs, and they have
[07:51] limited context. Obviously, for performance reasons, they don't want to put bunch of data in the API responses, right? And then most of these services are P0 services. We could not calling them for our backdoor or offline services use cases.
[08:06] So, we had to build parallel infrastructure to solve this use case. So, this one we tried, but we gave up on this. It's not an option for us. So, what did we do then?
[08:28] Obviously, like I said earlier, we love our medallion architecture. Instead of reinventing the wheel, we made that our foundation. What it did was we went to all of our upstream data producers and said, "Guys, let's standardize our data. Let's put our data into a single
[08:43] stream or very formatted data model." And also this architecture provide clear separation of concerns. We told them, "Don't worry about pipelines or putting the data into data data lake. That will be taken care by the architecture. You just worry about publishing that
[08:59] whatever data you have. And also this architecture gave us very good flexible processing capabilities. Especially, we can use the same data for streaming and batch semantics both. That's a very powerful tool, which I'll explain in a bit.
[09:20] Okay. Skipping. Now that we have formed our foundation, the next big task was to find our tooling and the platform that we want to use to build this architecture. While doing so, we had major four
[09:37] four major decisions that we have to make. This platform or tool have to support support heterogeneous sources. That means data should could be in data lake or it could be coming from Kinesis. It could be JSON store and whatever it is. Then it should provide us unified
[09:53] processing capabilities. That means it should not treat batch or stream as separate semantics. It should give us the capabilities of using the same code base to do both batch and streaming uh semantics. And cloud-native efficiency. The last thing we want to do
[10:09] is have a complete different org maintaining our infrastructure. We want that to be as managed as possible. Last but not least, future proof. We wanted to build skill capital within single platform instead of thinning ourselves learning bunch of different
[10:25] applications or uh tools. Spark, we Again, I don't want to start war here. We did explore Flink and Spark. At that time, for our use case, Spark seemed to be the better option, especially around batch semantics. I understand Flink 2.0
[10:41] Sorry if there are Flink purists here. Flink 2.0 did sort of merge that gap, but at that point, uh we felt Spark was our go-to solution and Databricks was a no-brainer. Especially for a team like us coming from application background,
[10:56] Databricks did give us that power and wings to quickly move towards data solutions. So, with this foundation and the decision that is made on the platform and tooling,
[11:12] a very high-level data source looks like this. We still have heterogeneous sources. We can't avoid that. How much ever we try to move everyone into our medallion architecture, we still have legacy systems, some outliers where we have to take them into account.
[11:28] But if you can see that, we have structured streaming and batch job on the left and right. They're both consuming the same data source and publishing the data. And what's more interesting is both are being served by the same code and logic. We are not treating them as a
[11:44] separate. We are not treating them as either this or that. The same code is serving streaming and bad job. Streaming is sort of providing the most muscle. That means it is processing most of the data that we have. And the bad job is sort of
[12:00] a crawler for us. It is making sure no data is left behind. If some case some messages were missed by the upstream teams or there were issues in the Kinesis or AWS infra or data bricks infra, our bad job is making sure the system is rectified and the data is there of
[12:16] highest fidelity. And then we're again giving the data to our medallion architecture. We are not steering away from the medallion. That is our foundation. That way anyone who is consuming the data from our Datos, they can either consume through Kinesis directly for real-time
[12:32] needs or near real-time needs. Or if someone has a batch processing needs, they can still use data lake. So, again, we are not moving away from medallion. We are just making this as a foundation and building on top of it for a contextually aggregated layer.
[12:52] Now that you know what you have, what Datos is, what the architecture and the decisions made, what did it change for us? There was a seismic shift in the way we were thinking. Earlier we were asking for what data you want, what use cases you want us to serve. Rather now
[13:08] we are simply saying what data you have to the upstream. We start We stopped chasing the use cases. We started hunting for the data. With this our velocity has increased. For most part, we had the
[13:23] answers to the questions that the team are asking us. So, we are not really sort of struggling to keep up with the project timelines and whatnot, because we are simply going to all the publishers of the data and say, "Guys, we want to consume your data. We want to aggregate
[13:38] the data contextually, and we want to put this." And since we are agnostic to any use cases, 90% of our questions are already answered in the data.
[13:55] As of now, this is where we stand. Daily we are publishing 50 million contextually aggregated records per day, serving 144 contexts, with a high fidelity of at minimum fidelity of six nines.
[14:14] So, what did we achieve with this architecture? Now, we have a resilient platform, where we are not treating a streaming and batch for either this or that or both of them are used we are using them in the same semantics to achieve us
[14:29] our goals of very high fidelity and self-healing of the any failures. Then, we have very highly observable platform, full lineage and monitoring across the data life cycle. What's interesting happened with this is
[14:44] we have become the heartbeat of our commerce data or our system itself. If we are missing single data, we know for sure some system in the upstream hasn't produced the data. And we are able to proactively reach out to them and say, "Hey guys, something is wrong. Please check this out." And we're
[15:00] able to capture it in QA itself. Then, scalable. Like I said, we stopped chasing use cases. We are sort of become data hunters, and we are aggregating data at scale. I think by this end of this year, we might be reaching
[15:17] 90 million records per day, contextually aggregated events. And then, the most important thing is consistent. One unified trusted source of truth for all. In a nutshell, Datos, we are not moving away from medallion.
[15:34] Medallion is our foundation. We are not treating either streaming or batch as separate enemies, or rather we are using both in tandem to achieve our goals. And contextual aggregation more than ever is important in this day and age. Because
[15:51] right now, because of our consistent data, we are able to provide this data to our agents and get consistent answers with this. Because a purchase payment success is purchase payment success across all the domains who are consuming this data. That is Datto's for you all. Thank you so much for attending this.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.