Sponsored by: AWS | Ripple: Well-Architected Data & AI Platforms - AWS and Databricks in Harmony
Summary
- Well-architected framework: AWS and Databricks align on principles of operational excellence, security, reliability, and cost optimization.
- Data governance synergy: Combined data governance and interoperability enable organizations to build robust, scalable platforms.
- Performance and cost savings: Ripple achieved 80% cost reductions and 55% performance improvements using Photon and Delta Lake.
Sponsored by: AWS | Ripple: Well-Architected Data & AI Platforms - AWS and Databricks in Harmony

Talk by Hari Rajendran, Staff Software Engineer, Ripple (Sponsor Speaker); Priyanka Adhia, Engineering Manager, Ripple (Sponsor Speaker); Rudy Chetty, Principal Solutions Architect, AWS (Sponsor Speaker)
Join us as we explore the well-architected framework for modern data lakehouse architecture, where AWS's comprehensive data, AI, and infrastructure capabilities align with Databricks' unified platform approach. Building upon core principles of Operational Excellence, Security, Reliability, Performance, and Cost Optimization, we'll demonstrate how Data and AI Governance alongside Interoperability and Usability enable organizations to build robust, scalable platforms. Learn how Ripple modernized its data infrastructure by migrating from a legacy Hadoop system to a scalable, real-time analytics platform using Databricks on AWS. this video covers the challenges of high operational costs, latency, and peak-time bottlenecks—and how Ripple achieved 80% cost savings and 55% performance improvements with Photon, Graviton, Delta Lake, and Structured Streaming.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements
FAQs
What is the well-architected framework?
The well-architected framework provides core principles for building reliable, secure, and cost-effective solutions. AWS and Databricks together demonstrate how Data and AI Governance alongside Interoperability and Usability enable organizations to build robust, scalable platforms.
How did Ripple modernize its data infrastructure?
Ripple migrated from legacy Hadoop systems to a scalable, real-time analytics platform using Databricks Data and AI platform on AWS. This modernization addressed high operational costs, latency issues, and peak-time bottlenecks in the old architecture.
What are the specific improvements Ripple achieved?
Ripple achieved 80% cost savings and 55% performance improvements using Photon, Graviton, Delta Lake, and Structured Streaming. These optimizations demonstrated the tangible benefits of a well-architected modern data platform.
How do AWS and Databricks work together?
AWS provides comprehensive data, AI, and infrastructure capabilities while Databricks offers a unified platform for analytics and AI. Together, they create a seamless experience for building modern data lakehouse architectures with strong governance.
Full transcript
All right, let's welcome to the stage our first talk today. Okay, folks. See, you're all excited cuz it's what? Day two, day three. But at AWS, it's always day one, right? Exactly. See, see, you know, so we'll get through these uh fancy first few slides. Forward-looking statement. We always look forward not backwards right acknowledge our past but we look towards the future. Another thing which we all love to fill out is surveys.
But look if we do a good job let us know. If we do a bad job let us know even more. And welcome to this presentation. It's called wellarchchitected data and AI platforms. We've also got a special appearance by a customer called Ripple. I'm joined by and who are going to come up and tell you their story of how they utilized data bricks and AWS in harmony. But to set the stage, let's give you some background, right? So you you know what's going on.
You know, data is everywhere. It's everywhere. It's being generated in more and more amounts every day, every second. Just think about the amount of data you use on your phone, right? Especially if you're on Wi-Fi. The problem comes in is that we've got all this data, but how do we take that data and gain insights? That's the important part is you're like, well, we could just examine it manually, maybe an Excel spreadsheet.
Even better, you could create a dedicated app. So, with that app, you're actually sitting there going, oh, this is going to get the insights I need I need to pull out from here. And even better, you can utilize a platform like AWS to help you do that. Look at those services. It's only a handful that I've shown there. S3, we've got Amazon Q, Quicksite. Essentially, you get your data into AWS. Then you can analyze it, generate insights, and as we like to say where I'm from, your uncle after that, right?
But if you've got data and you're here at the data and AI summit probably utilizing data bricks great platform recommend it even better is you can develop all these apps in tandem with AWS and data bricks but when you're developing them and I'm sure some of you who are technical folks you know this consistent app development is very tricky it's needed but it's tricky. So what do I mean by that? Well, you got many moving pieces.
You got your database. You got your other data that might not be in a database. Compute everything under the sun. So you're thinking, well, can I do this alone? You could. Or you could take all your team members together and you can get the app developed quicker, faster in a more uh stabilized or as we say standardized mechanism, right? You figure out how it works with your first workload. Your next workload's a bit better.
You keep iterating as you go forward, right? The way that you do that on AWS is through something we call the wellarchchitected framework, WFT for short. It's got six pillars. We'll go into them just now. But even better, apart from the standardized way of developing your apps on AWS, we have lenses, industry lenses. So it's like my lenses here. You can see they're kind of kind of thick, right? But you put this lens in between the actual well architected framework pillars and the workload you're running.
It gives you additional insights into how you could potentially improve your application, how you can make sure it's reliable, and so forth. And like I said, that partnership, we want to stress it, datab bricks and AWS, when you're putting your data into the databicks platform or storing it on AWS or both. That's how you can accelerate your workloads. And as I mentioned, why databicks on AWS? Well, it's the million-dollar question, right?
Well, think about it like this. Databicks was born on AWS, right? They've got a lot of experience developing and running their workloads there such that they provide that to customers like yourselves. Case in point here, innovations that come through data bricks, they usually debut on AWS first. That's cool because you get the latest and greatest. For example, here SAP or like I said where I'm from, we actually say SAP because SAPs is South African Police Services.
So if you say SAPs, they're like, "Oh, no, no, no, no. We don't want that. Don't want that." Right? In addition with this joint partnership you get vertical solutions remember those industries I mentioned all of those healthcare life sciences you can go through manufacturing IoT you name it it's been developed and the best part is you can learn from those other customers or partners to enhance your workloads you also get interoperability so like I said data bricks was born on AWS so it's kind of cloudnative and then you take a step tag.
It's like, oh, it's using the same services that I'm used to now. S3 simple storage service. You can store your data in a bucket. Just makes it that much easier and seamless, I should say, to integrate these features. In addition, you get a lower TCO. That's total cost of ownership. Examples here. You can see those two icons. One is EC2, specifically Graviton. We've got some 64-bit ARMbased CPUs. So if you convert your workloads to utilize that chipset architecture, you can actually lower your TCO just by utilizing that chipset.
In addition, you can take advantage of spot instances, fleets, all of the lovely features that have been baked into services like EC2, elastic compute cloud. You can also use something which is on the right there, uh S3 intelligent hearing. So remember the data that's sitting there. I'm telling you some workloads when people put their data there, they keep it. And they're like, well, this is 5 years old. Oh, we need it.
Okay, I understand. Let's do a compromise. Let's tar it so that after 5 years, it moves to a lower storage tier, which is more cost effective. So you still have your data, but it's less TCO. That's how you start improving it. Remember, it's an evolving workload. And the hot topic now I mean it's baked into the actual title of this summit genai you get access to Amazon Sage Maker AI and Amazon Bedrock. So if you want to integrate those in seamless like I said you can utilize foundation models large language models you know the ones that are general purpose but they get you to where you want to go.
Amazon Nova as well. Nova Sonic, you got access to all of that if you just spin it up on Bedrock. And one thing you might not know is data bricks was number one on the AWS marketplace. So a lot of folks are using it. I mean that's why you're here, right? I can see that guy's excited. Yeah, this Oh, you're all I can see it in your eyes, right? You're excited. But well architected framework. This is the last part we'll go over.
Well, maybe two more slides. This here is comprised of six pillars that you evaluate applications or workloads against such that they're developed in a consistent manner. Right? So, you know that oh, if I've gone through the well architected framework while developing this, there's a better chance my app's going to be stable, reliable, all of the above. First one here, operational excellence. Is the application able to be monitored run efficiently and in the sense that okay it's operating we've got more users now the next month they keep going it's operating well it's operating with excellency see that's where it comes in the next one which very important I I'd argue this should be number one security you don't want bad actors to have access to that data that you've you know created put into an S3 bucket or whatnot.
So, make sure that when you're developing applications, you adhere to what we call the principle of least privileged access, which means if user A needs to read from database B, just give them read access. You don't have to go the other way and give them read, write, delete, all of that CRUD access. No, bare minimum permissions. And that permeates throughout all your workloads, your applications. You also want applications to be reliable.
Say there's a disruption in one of uh your workloads. Can your application recover gracefully without your users even knowing that there was a disruption? That's what we mean by reliable. And it is possible utilizing multi- availability zones, multi- region. You could go all the way there. The fourth pillar, performance efficiency. Is your application performing how it should be as your workload, as your customer base, as everything that's involved evolves, right?
Efficient performance because you never know, maybe you started it two years ago, there's been feature enhancements, enhancements or new CPU uh types that came out in EC2. If you changed them over to that one, oh, that makes it more efficient. That's the beauty of the cloud is it's dynamic. It's evolving. You can change your workloads as your needs evolve, as your customers needs evolve and your business requirements.
It's never one and done. And we've talked we've talked a bit about this cost optimization. Like I said, new features come out, new services by switching to it or utilizing a different S3 bucket uh uh tier or S3 tier. Can you Oh, I can save some money here, be more cost effective. I'm telling you, people be like, "Oh, I'm saving money. Love you already. Thank you." Right. And the final one here is sustainability. We've got one world, one earth.
We all want to grow together, be consistent, be mindful of what we're doing from an environmental perspective. So, when you look at a specific region, you can check are the resources that are being consumed sustainable? are the energy uh resources clean. That's where sustainability comes in. And through all of this, right, like I said, data bricks was uh born on AWS is they took inspiration. They were like, we like this well architected framework.
We'd like to do something similar. So, they created the well architected lakehouse, right? I live in Seattle. There's a lot of lakeouses there. There's a lot of lot of water, too. We actually call it the Puget Sound. And for the longest time I kept listening outside my window when I moved like what where is I don't hear the sound like sound is actually the body of water. So well architected lakehouse and it is built on top of AWS W.
Now why do I have a snake there? It's to help you remember the acronym for well architected framework. The way I remember is corpse. You know it's a snake. It's like corpse. cost optimization, operational efficiency, reliability, performance efficiency, security, sustainability. So if you're ever going for an exam, that's how you remember the acronym for well architected framework. But like I said, data bricks built on top.
In this case, data and AI governance. That's the first pillar they added. So it's like, okay, well, is there oversight with uh the data and AI that is actually bringing value to my customers workloads? Yes. In addition, this is the other one, interoperability and usability. Can you actually utilize this uh lakehouse architecture to interact with other systems? Is it that seamless? As I've said a few times, that's how you do it.
And that's where that partnership with AWS and data bricks comes to the forefront. You're like, oh, I've got the well architected framework. I've got the datab bricks well architected lakehouse. If I develop with these two technologies or companies or workload providers, however you want to describe it, you know, I'm pretty much already set from the start. I've got this to work on. But look, I'm not here to market to you.
I'm here to tell you my experiences even better. This is where we bring up Ripple and we say, you know what, hear from a customer. So, not just from me, from the customer. So I'd like to welcome the Ripple engineering team one at a time to the stage. So I think you're up first Priyanka. Yes. Please come on. Please give a round warm Yes. Yes. Yes. Round warm round of applause. There you go. Thank you. Thank you. Hello everyone.
I'm Priyanka Adya, engineering manager at Ripple for the data team. Uh today I'm going to share the journey of how we migrated uh from Hadoop architecture the Ripple's legacy architecture to um more fast reliable platform and how we saved cost and how we got there using data bricks and AWS. At Ripple, we process high uh proc we have high frequency ledger transactions at global scale. But until now uh you know things were working uh but we hit so many roadblocks.
There were cost issues, there were performance issues, there were latency issues, you name it. And we know and we knew that we needed a change. We knew we needed modern architecture. So let's start with where we were our Hadoop based system and the fragmented crowd services and what problems uh we saw in in the legacy architecture. First the maintenance overhead was significant. Um you know we we faced performance bottleneck leg u latency issues we faced autoscaling failures. uh that meant high volume of support tickets and as uh we we spend lot of time firefighting.
So uh secondly we had real time processing issues right? So we couldn't have um uh Hadoop ledger transaction high ledger frequency transaction flowing in and we couldn't support realtime workloads. The third one was peak business hours especially you know during holiday times uh there were high throughput issues. Um and then lastly we had this fragmented architecture with multicloud um data fragmented everywhere leading to high cost and um you know um issues around uh supporting and maintenance.
So these were the challenges uh we had. Now let's let's take a look at um um the dimensions we um so with the new architecture modern framework right we were focused on four core dimensions. We wanted to optimize on cost, we wanted to optimize on latency, we wanted to optimize on performance and we opt wanted to optimize on high volume data using the modern architecture and AWS and datab bricks gave us that support and everything in terms of services um and um different um u uh I would say plethra of services that we needed to go for it.
So um let let's take a look at that. Um are we on? Yeah. So so the first dimension is cost. Um with AWS we saw flexible uh pay as you go model instances. We we had multiple instance type and we could use reserved instances. We could use uh wide range of instance type right sizing. We could use lambda which was serverless feature. Uh we also had u photon and gra graviton com uh uh combination which accelerated our query processing.
Uh the photon plus graviton really did the magic. We saw like 3x price performance um u improvements and it it just did it magic. Um and then lastly we also used graphana uh as well as we used uh datab bricks system tables to measure our daily cost uh as well as um how how how we could uh optimize on our cost savings that itself gave us about 80% reduction in cost. So the second dimension um let's take a look at the second dimension.
Um the second dimension was on the performance uh for performance optimization we wanted um data bricks uh which which gave us uh apache spark uh as the back as the backbone for our machine learning workloads. uh photon engine with vectorzed um execution help us like extremely um uh get to the extremely fast queries and we could save about 55% month- on month by moving to photon and then delta lake enabled uh along with the autoloader enabled the fast um continuous pipelines to get there then the third dimension was latency so AWS uh supported services like kinesis dynamob DB uh which we blended together with our APIs uh and that gave us like highly uh uh fast workloads and latency up to seconds.
Uh datab bricks also supports uh spark structured streaming um which which made uh our continuous workload scale at high speed and uh we could really see realtime workloads in in production which was a huge win. Uh and then lastly we leverage Delta live tables um um natively supported with autoloader for seamless end to end continuous workloads running in production. Now the fourth dimension was storage. Uh definitely storage was last piece of the puzzle and we wanted to get it right.
For storage, we had S3 and S3 was supported with intelligent taring uh with um um uh policies that could help us um you know um uh archive data, put it into cold storage, so on and so forth. Um and then also we had integration with lake formation and glue which help us made um you know federated uh architecture with right governance and best practices in place. And then lastly, uh, data bricks offered Delta Lake optimization capabilities, uh, which was, uh, excellent, uh, when when we saw, uh, um, things like Zeping, um, and then, uh, Zordering, u high volume of ingestion workloads coming through and we partitioned using those techniques uh, for data skipping, file compaction, so on and so forth.
And then lastly we had S3 as backup uh which eliminated our redundant storage uh and help us scale for high volume workloads. So with this I would like to invite who will walk us through the architecture and how we got there. guys I am rajindran u staff software engineer in ripple's data platform team I'll talk about the highle architecture that we have with our data platform now so uh we primarily have three injection methods uh primary being the API based injection which um uh which integrates tightly with all the microservices that we have for various business units um the service teams or the engineering teams push the data to our caspian uh to the data platform API uh which is actually backed by Amazon API gateway, Lambda functions, kindness data streams and fires and S3.
Um and also we have the direct file-based injection in terms of uh you know when the data volume is large and the data file formats are like CSV, park and JSON. Uh we allow cross account access for teams to directly upload the files to S3 buckets and also CDC based injection. Um all a lot of microservices within ripple use uh my Amazon Dynamob and uh which uh which puts the data into Kinesis data streams again through cross account access uh lambda function reads the data from kindness data streams and puts the data to fires and puts the data to S3 there and also depending upon the latency requirements that uh our business use cases dictate we either read the data from Kinesis data streams or from S3 three through structured spark streaming jobs or uh DT and uh which also moves the data to uh the Delta Lake which is principled on top of bronze, silver and gold tables quote unquote uh medallion architecture which most of us will be following here.
Um and once the data gets into the silver and gold tables uh we also have the reporting layer and AAI layers built on top of them. Um also we have observability nothing uh uh is complete without observability let's say and we have ETL stats dashboard to which measures the success failure rates um delta lake metrics uh which uh which talks about uh the size of the tables the size of S3 uh storage that the table contributes the vacuuming frequency um and other metrics there and also job CPU and IO metrics uh we launch all our uh uh compute on EC2 instances within our own uh account.
Uh so we also have the metrics uh captured for them as well. Um and also we have the other AWS infra for other workflows. Um we utilize EC2 instances, NAD gateway, SQS, SNS uh for monitoring notifications and our compute requirements. Okay. So let's talk about the business impact. Um so uh so Priyanka talked about uh what was lacking in our previous data platform and what happened when we switched to this new data platform.
Cost savings of more than $1 million was achieved in 2024 uh through uh both targeted optimizations uh for various cost drivers as well as uh platform optimizations in AWS and data bricks that we put in and improve developer productivity too. So uh in some cases the feature delivery time uh decreased from few weeks to few days and faster data delivery for stakeholders as well. Uh from batch uh pipelines that we had in our previous architecture.
Uh the data movement from injection API to uh go tables was now done within a few seconds and uh customers are now happy. So yeah, so how did we go about cost optimization and how did we achieve uh the million-doll savings that we uh that I highlighted last slide. So first we identified the various cost drivers uh within our platform. Um we observed that jobs DT pipeline SQL queries S3 storage allpurpose clusters some amount of model serving endpoint as well but uh that was not a lot of cost so I didn't include it here.
But once these major cost drivers were identified uh we were able to attribute these costs to various business units u and also we created dashboards for the various business units for all these cost drivers and we calculated efficiency scores for each of them. So which what actually says that how efficiently the resources are being utilized uh for example how efficiently are using the job how we efficiently have created a DT pipeline and we created a formula out of it which uh which actually gave a overall view to the business units on what to target and what to optimize and which pip which pipeline if you target you'll get better savings and all kinds of insights there.
So let's talk about the AWS optimizations, the platform optimizations. I think a portion of it was also covered by Priyanka. So uh we uh enable S3 intelligent tearing for our S3 buckets um and enable life cycle policies for all the S3 buckets that we own. So this actually had like 40% cost savings compared to the standard storage class and 20% savings with graviton EC2 instances uh compared to the x86 architecture uh instances that we were previously using um and also VPC gateway endpoints.
We have enabled it for our S3 bucket access so that uh all our EC2 EC2 instances can access data from our S3 buckets in a costefficient reliable and secured manner. And also we tracked the spark cluster usage as well. U and uh we made sure that you know we tagged all the clusters with the job ids and the cluster ids. We were able to roll it up to a particular job and DT pipeline and made sure that each and every job and DT pipeline were uh utilizing the proper instance types and the EBS uh uh uh volumes there as well and uh which would actually uh give us better uh you know uh cost uh uh per performance ratio.
Um and also we were big on spot instances uh for our executors. So we uh instead of using on demand we used the spot on fallback policy which I will talk about um which helped us save uh like around 10% of uh money compared to the on demand instances. Okay. Um let's talk about the cost savings that we apply through datab bricks optimization. So uh we also set up uh uh monitoring on the S3 growth of the datab bricks tables and we enabled periodic vacuum jobs.
Uh in some cases uh for example the actual table was like few gigabytes but previous versions of the data were several terabytes and that was occupying a lot of storage space in our S3. uh which we promptly identified and uh implemented periodically uh periodic vacuum jobs to clean them up and also optimizing this executed memory of clusters. By this what I mean is uh uh all stateful applications that we have uh the states to growth was managed through watermarking appropriate watermarking intervals and uh we made sure that executive memory did not blow out of proportions and we needed to add more executors all the time to satisfy the demand. um enabling autoscaling uh it's like a default policy within um uh data bricks.
So autoscaling was enabled so that when the uh load is less it can scale down and when the load is uh more it can scale up and also we utilize database cluster policy configurations as well. Uh so very deeply so we had multiple policies created for jobs and DT pipelines uh which uh which had to be attached uh when we when the jobs and DTS were started. Primarily these settings were uh the ones that contributed to a lot of savings.
Uh we enforced pipelines cluster shutdown delay setting to 2 minutes so that when the job when the DT pipeline completes uh it would shut down within uh 2 minutes. The cluster would shut down within 2 minutes. And also we set the max value for debuts per hour. uh so that uh if the user wants to uh launch bigger instance types and also more number of instances it was the total DBUS are bound by the limit that we set here and also we set the node type ID to graviton instances by default both in cluster policy as well as teraphform as well terapform modules as well um so that by default if there was no u cluster or or instance type defined for executors uh we use graviton instances and also So we use the uh spot with fallback as the uh default for our executed instances as I defined earlier.
So uh the data brick system tables utilization how did we utilize it? So we used it to track updates happening in the upstream tables and uh and based on that we scheduled our downstream pipelines. In some cases what we identified was the tables were getting updated uh every hour but the downstream pipelines were running every 5 minutes. Um which means that you know they are always getting the older data in spite of running every 5 minutes.
Right? So uh we in that we also adjusted the downstream pipeline scheduling based on the refresh rate of the upstream tables. And uh we also monitored the continuously running jobs in our lower environments and also jobs running for more than 14 days. And we alerted the job owners to take action to shut them down. And also we uh uh we also lowered the SQL varrows cost uh by monitoring long runninging resource intensive queries analyze the query plan and optimize the queries.
So what's the conclusion? So the conclusion is improved processing speeds u P90 low latency use cases uh like you know which is like most of the use cases that we have uh from injection API that I defined to the silver tables it was less than 2 minutes and for ultra low latency use cases uh from injection API to uh the silver tables it was less than 4 seconds. Um yeah higher reliability. So we utilized AWS uh uh u you know in underlying architecture.
So we expected no near zero data loss uh due to the robust free try monitoring mechanisms that we had in place and also reduced cost. So the cost in 2024 alone was 10 times lower compared 10% lower uh compared to 2023 uh with despite like 12x daily volume of data and 270 terabytes of extra data that we processed. Yeah. So lastly this is the last slide so uh lastly I would like to thank my team here. So thank you for your contributions towards building this new data platform here as well.
Thank you very much. Okay. Okay. Get the mic back up. Thank you. That was See, remember what I told you at the beginning, the milliondoll question. We answered it. They both answered it. They saved over $1 million. And you didn't realize that till the end of the presentation. But you can see we're not just telling you how to utilize our platforms together. We're showing you through an actual customer example and using Lambda, one of my favorite services.
Do you know why it is? Always reminds me of a game I used to play in my youth. You can see there's some gray hair here, right? Uh called Halflife. It's got that same symbol, the lambda symbol. So, think of this when you're developing your workloads. Relate the services to things that you can, you know, think of in your everyday life. The other example that was mentioned was the S3 intelligent tiering option where you can move data.
A simple way to remember that is maybe you have a house, right? You got your fridge. You got your favorite drinks in there. Maybe a Coke, Lacroy. Mine's topico. Got my Topo Chico there. But you've got so many Topo Chico. You want to store the rest of them somewhere else. So you store them in the fridge in your garage. That is S3 intelligent tearing. You still have access to both, but instead of going to your kitchen fridge, you have to go to your garage, take it out there. both still very cold and tasty beverages.
But before I digress into all of these uh analogies, I want to thank you for being here listening to us talk about not only the well architected framework that you can utilize on AWS, but also the two additional pillars that you can use with the well architected lakehouse. And how do you approach it? You dip your toes in the water at the lakehouse, get a feel for it. Okay, this is not so bad. Maybe you can dive in, you can go a bit further and that's where you start to build out your architecture.
Just like the diagram that showed. You start small, right? But you think big. It's one of our uh leadership principles is think big. So think of it that way. You can always look at the end game, but the fun part is figuring it out in between. Start with that small workload, build on top, and cost optimize performance efficiency. All of those well architected pillars and the lakehouse uh pillars as well. utilize them as your workload evolves cuz like I said, it's never one and done.
So with that, I know got a few minutes. I'm standing between you and probably lunch. I know I fed you. We fed your mind. Now I'm going to feed your belly. So go get your lunch on. Thank you again for joining us. Hopefully we'll see you again next year.
Learn more about the Databricks Data and AI platform.
The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.