Skip to main content

Let's Save Tons of Money With Cloud-Native Data Ingestion!

Summary

  • Cloud-native cost savings: Scribd redefined its data ingestion architecture using open source tools like kafka-delta-ingest, oxbow, and Airbyte to create an event-driven, reliable, and significantly cheaper ingestion pipeline.
  • No-jobs architecture: By leveraging third-party tools in concert with Databricks and Unity Catalog, Scribd eliminated the need for traditional batch jobs while maintaining efficiency and reliability.
  • Multi-source ingestion: The platform ingests data from AWS Aurora, SQS, Kinesis Data Firehose, and other sources into Delta Lake, providing a cost-effective unified approach to massive data querying.

Let's Save Tons of Money With Cloud-Native Data Ingestion!

Watch: Let's Save Tons of Money With Cloud-Native Data Ingestion!
Talk by Tyler Croy, Valued Employee, Scribd, Inc
Delta Lake is a fantastic technology for quickly querying massive data sets, but first you need those massive data sets! In this video we will dive into the cloud-native architecture Scribd has adopted to ingest data from AWS Aurora, SQS, Kinesis Data Firehose and more. By using off-the-shelf open source tools like kafka-delta-ingest, oxbow and Airbyte, Scribd has redefined its ingestion architecture to be more event-driven, reliable, and most importantly: cheaper. No jobs needed! Attendees will learn how to use third-party tools in concert with a Databricks and Unity Catalog environment to provide a highly efficient and available data platform. This architecture will be presented in the context of AWS but can be adapted for Azure, Google Cloud Platform or even on-premise environments.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What tools did Scribd use for cloud-native data ingestion?

Scribd used open source tools including kafka-delta-ingest, oxbow, and Airbyte in concert with Databricks and Unity Catalog. These tools enable efficient event-driven ingestion from multiple cloud sources.

What data sources does Scribd ingest?

The Scribd ingestion architecture ingests data from AWS Aurora, SQS, Kinesis Data Firehose, and more. The cloud-native approach consolidates data from these various sources into Delta Lake efficiently.

Apache Iceberg vs Delta Lake - which table format should I choose

Apache Iceberg and Delta Lake are both open-source table formats that bring structure and performance to data lakes, but they differ in their design philosophy and feature set. This video discusses how Delta Lake, developed by Databricks, focuses on ACID transactions and performance optimization, while Iceberg emphasizes compatibility and advanced features like schema evolution and hidden partitioning. The choice depends on your specific requirements for data consistency, query performance, and operational simplicity.

which platforms use Delta Lake format

The Databricks Data and AI platform uses Delta Lake format as its native table format, providing ACID transactions, time-travel capabilities, and unified batch and streaming processing. This video demonstrates how many modern data systems including Apache Spark, Presto, and other analytics engines support Delta Lake format for efficient data management. Adoption of Delta Lake across the ecosystem enables organizations to build lake-based data architectures with proven reliability and performance.

How do I save money on my Splunk bill?

Reducing Splunk bills requires analyzing your logging strategy, consolidating duplicate instrumentation, and optimizing the volume of data sent to monitoring systems. This video discusses how organizations can move non-critical logs to more cost-effective storage solutions while maintaining visibility into critical business events. By using Delta Lake and the Databricks Data and AI platform for centralized logging and analysis, enterprises can achieve better cost efficiency compared to specialized log aggregation services.

Full transcript

I'm going to go ahead and get started if everybody's okay with that. Thank you all for making the big long trek across the street up the escalator down the hall towards the bathrooms to my talk. Um, nobody told me that I was going to be bathroom adjacent, so I'm glad everybody made it all the way down here for the chat. I thought I would do something um a little bit different this year. My boss it loves a spreadsheet.
Um, and so I thought I would present in spreadsheet format which I assume would be something acceptable to anybody who would attend a talk talking about saving money. Um, so if I go to four I think it is there we go. Okay. So at a high level um we're going to talk a lot about a number of different things. I've tried to incorporate as many, you know, spreadsheet related graphics as possible. But to give you a sense of scale of what script, which is the company I work for and where a lot of this work comes from, is about, we have hundreds of millions of documents, which means the data and ML platform that we build and operate at script has a tremendous scale.
Over the last couple of years, with the advances in large language models and the utility of large language models, script takes a tremendous advantage of those things. And so the cost of operating our infrastructure at this scale has become a very important part of what I obsess about and think about on a day-to-day basis. Um, if you ever need cost explorer help, if you have billing usage or system table questions, I can answer those as well.
I've spent way more time than I ever thought I would just looking at data sets coming from AWS and data bricks. So this is the one audience interactive session or part of the session until we get to Q&A. The whole point of saving money, I want to see a show of hands, is so that we can give put money towards other things. So, who would rather give money to the 1999 Time magazine person of the year? That's Jeff Bezos. It was actually the first year that they called it person of the year and not man of the year.
So, personally, I don't want to give anyway. Um, the 2025 person of the year hasn't been named yet, but it might be. and do you want to give data bricks more money or 2006 time person of the year was you? And so the whole premise of this talk is that if we save money, maybe we'll get some more of that in our pockets or we can put it towards something useful. Like ultimately at the end of the day, we all have budgets and I would rather spend budget on something that's going to move the business forward as opposed to unnecessary baloney in the data ingestion process.
Um, if is time person of the year, I I hope data bricks gives me something something special because that would be really awesome that I predicted it. Um, so at a high level, a lot of data pipelines are kind of simple. It's you take data from somewhere, you bring it in, you land it typically in if you're following the medallion architecture, that's your bronze layer. It's structured, it's not refined, it's not clean, etc.
And then you do interesting stuff with it. One of the big things that I've pushed the data bricks team on in the past and I've worked a lot in the open source community around is this first step getting the data in is not interesting. That's taking bytes from one place and putting them into another. Once you get to the processing or the transformation or joining data sets that's where things like Apache Spark the data bricks runtime photon etc become really interesting.
And so if you think about what we spend our money on in the data bricks platform, we usually we script we you all spend a lot of money just getting the data there. And a lot of the work that I've I'm going to be presenting on is trying to remove that step because it's not differentiated. It's not interesting. Getting data into a delta table format or into an iceberg table format is pretty simple. Um it's not something that Apache Spark is well suited to do.
It's not something big distributed compute is well suited to do and you can do it in a much lower utility or much lower footprint than Apache Spark typically requires to operate. Um, a lot of my numbers are are very accurate. So everything is 100% true and correct in this presentation. So you can quote me on that. Um, so let's talk about Delta Lake as a format. Delta and iceberg are really what I'm mostly concerned about these days.
There are other lakehouse formats. I'm not interested in them. You may be interested in them. But at its core, Delta Lake is a parquet or it's parquet data files in orange as indicated by my helpful legend and a transaction log that sits in object storage next to it. The reason that this is important, and by the way, you should get this wonderful book. I contributed a chapter, but a bunch of other people did much more interesting things in that book.
If you want to know everything about Delta that there is to know, you should get that book or get data bricks to give you the book. Um there's actually a free PDF of it on script because there's a free PDF on delta. io. So you don't actually have to buy the book. Um saving you money. Excellent. So the transaction log is the interesting part about Delta. The data files could come from anywhere. And that was one of the big realizations that I had a couple of years ago when I started to see parquet become the standard format that a lot of data systems produce.
And once you know that a lot of data systems produce parquet, then the challenge becomes, well, how do I dress up the transaction log to match that? How do I gussy it up so that it's now queryable as a delta table and I'm not doing a list or a scan across S3 to ingest my data into a open lakehouse format. Iceberg is fairly similar. I'm not a super big expert on iceberg. I've never written anything about iceberg. Um, but functionally they are similar.
There's more metadata. There's a catalog with iceberg. There's other things. there's other stuff at the the sort of table format with iceberg but functionally they are the same that there's data files which are kind of who cares where they come from and then there's table metadata around them. So this these approaches that I'm going to talk about will apply for delta and iceberg as well. So at script when I started to look at the work that we had and the systems that we were interacting with what I found was a lot of things are actually already parquet capable I don't have a good buzz word for this but they can produce parquet and when most of the things are in parquet already spinning up a spark cluster to take that parquet and turn it into differently shaped parquet or differently written parquet with different tags or whatever um isn't really all that interesting.
So we already get it in part in almost all cases. Now there are some outliers and I can talk about those um if I have bonus time um where we have systems that might produce CSVs or JSONs. If you're working with a large language model, a lot of these or large language model as a service like an open AI or a Gemini or a data model serving or batch bedrock. A lot of those will produce line delimited JSON which is not parquet. the savvy listener may have noticed and getting those into a delta table format is requires a little bit more work but it still doesn't require a spark cluster.
Getting data in is the most simple thing that we can do. It's where we do querying and joints that we're that's where we should be spending money and leveraging distributed compute that we have in um Apache Sparkle. So this is a small selection of the data sources that we have at script. If you're in AWS, as script is, as many many companies are, there's a lot of things that already produce parquet. When I talked with our AWS account team about, I discovered in 2023 that produces parquet.
I was like, whoa, this is the craziest new feature. Stop the presses. Let's stand up at reinvent and talk about this. This is incredible. And then I actually went in preparing a different presentation and I looked, it's actually been around since 2020 and nobody told me. um you can turn park or tables into parquet with nothing more than a button click if you have console access or a step function or something like that.
We'll talk a little bit about that more in a second. Then there's another product called Kinesis data fire hose or formerly known as Kinesis data fire hose. We've dropped the Kinesis. It's now just data fire hose and data fire hose for the last number of years has supported what they call output conversion. And so you can take a streaming if you're not familiar with data fire hose um it's just it's like a Kafka. is a thing that you stream bytes into and at the other end you can get those messages out.
You can just flip data conversion on or output conversion on for data fire hose and you get parquet in an S3 bucket of your choosing. I'll talk a little bit more about the cost implications to that. But if you have smaller amounts of data um data fire hose with output conversions means you can take any real time source of data and blammo presto bango you've got an appendon delta table with pretty much one click. Um, SQS doesn't give you park A, but I got something for you for that.
Um, Airbite, and you should go down and talk to them. I did see them on the expo floor this morning. Airbite is an open source tool. If you're not familiar with it, um, you should check them out. Um, they have an S3 destination, which we use that produces Parquet natively. If you're on Google Cloud, you can just take any input and dump it into parquet. If you're on Azure, you can take any input, dump it into parquet.
You may be noticing theme Kafka. Kafka is kind of a pain in the ass for a lot of reasons, but in this one in particular, there's tools with Confluent that you can get. There's tools that you can get from your cloud provider that will produce parquetish things out the other side of a Kafka topic. Um, Script built one called Kafka Delta Injust. Uh, I did mention this up front because it it's not super relevant. Script created the Delta Lake bindings for Rust.
The whole reason we created the delta lake binics for rust was to build Kafka delta ingest which is an open source tool which reads data from Kafka and dumps it into a delta table. We can do better than that in 2025. We could do better than that, but there's not really a great way to take data out of Kafka other than running something that's going to consume basically do a while true take message take message take message right to parquet um in the Delta user Slack at least once a week someone comes in and says hey I've got a bunch of parquet how do I turn this into a delta table and usually the other contributors might be tired of me saying this I have a solution and it's to not write this in Python not to do anything anything too esoteric or interesting in your job per se, but to use this lambda which I created and I highly recommend it.
It's very quality code. It's called Oxbow. And the trick with Oxbow and why I I want more people to start using it is it relies on bucket notifications. So when a parquet file lands, it will go add that transaction log. So it builds on top of Delta Rust and it adds those transaction log entries as data comes in to um into your S3 bucket. The really interesting thing when I talk to a lot of data engineers is eventdriven data pipelines is something that is a little bit beyond where most people are thinking about what's possible.
But when you step back from the table format when you step back from how we traditionally build spark jobs orchestrate them with airflow etc. A lot of clouds give you bucket not or object notifications. They will tell you when data is created and then if all you need to do is add a little transaction log on top of that, you can build a very very lightweight event driven system that scales very horizontally um to ingest large amounts of data because the thing that's doing the delta right in this case is whatever's producing the parquet or it's doing the data right excuse me not the delta right but the little bitty lambda just says I know that this file exists and it should be in the transaction log.
That's the zen and art of Oxbow. It's incredibly simple. It's probably all said and done, it's maybe four or 500 lines of Rust. Um, which makes it extra cool because it's in Rust. But, um, it's very, very simple because the problem domain is actually quite simple once you remove the data file right from the equation. And so, here's uh, building a presentation in a spreadsheet is surprisingly difficult. Um, you'll see as we go through these sheets because there's no slides. um that I I started to figure out a little bit better ways to diagram things.
But this is an asky diagram because I couldn't figure out a better way. But in essence, we have an external service. What is producing the parquet data irrelevant? If you use fiverr, fiverr will produce parquet. If you use airbyte, if you use a lot of cloud native tools, I'm sure there's a myriad of integrations down on the expo floor which do produce parquet in some more form or fashion. You just have to land that into an S3 bucket.
Somebody somewhere has to write the data. And what makes this cheap is you already are running that something else to write the data to parquet. Then we receive a bucket notification in SQS. This is the cloud native part of the top. And then all Oxbow does is it's triggered by that message coming into SQS that says file exist here. It does a little bit of intelligence to infer the path to the delta log because that's a pretty important part of creating the append.
And then it writes to the delta table. What makes this really interesting to me and why I get like super jazzed up about it is when we think about data ingestion, a lot of people think about rows per second, messages per second, you know, gigabytes per second, like these very very large numbers. Most of the oxbow lambdas that exist that I have deployed are running on one vp vcpu or 0. 5 vcp. And so you have a delta table that is being produced by something with one vCPU of compute and something else like an or a Kinesis data fire hose that is producing the parquet files.
And so your your typical thinking of throughput for a data ingestion pipeline gets completely thrown out the window because something else is performing that right and you don't have to think about that. You think about what comes after it lands in your bronze layer. I should note while before I move on from this, the append part is important. Um, concurrent transactions on delta tables is a little bit of a tricky subject in S3.
We've had a lot of development there. But if you have data files that are landing, but you have some other process that is trying to do merges, upserts, deletes, etc. , you're going to scrub your delta table. Don't do that. The Medallion architecture plays a pretty big role in the success of this approach. If your delta table is append only and you can sell the delta table property called append only equals true then you only add rows to this table and then your silver and gold layer that's where you're going to be doing those joins and those merges those things where your big distributed computation are going to going to be valuable for you.
And so if you have a organically grown data platform like script does, you might not always be able to make this hardline rule of this is append only. This is your bronze layer. Only this one ingestion pipeline is going to going to write to that. In that case, you're kind of in a tough spot. It's difficult to it's difficult to fix fix that without or approach that from the oxbow or like this event driven architecture without changing your underlying data model to I would say more appropriately adopt bronze, silver, gold etc.
The Aurora export part is a little bit interesting. Uh earlier this year um I gave a webinar with um it was supposed to be with my colleague Kunthal. was actually sick that day and so I had to give I had to present our slides um uh to you know one person presenting both those slides getting data out of is pretty simple but structuring it in the appropriate way is a little bit tricky if you think of your classic uh delta table let's say you're partitioning on a date stamp or you're partitioning on like year or something like that your your tree is my delta table slash you know year equals 2025 slashstartk does not give two shits about that. is going to dump a huge mess of paret files into an S3 bucket and say good luck.
Have fun. And so there's another lambda that's this is also in the Oxbow repo if if you're interested that we use called the um restructure lambda. Fortunately, does give you some predictable prefixes on on its output. So you know that you know your landing zone slash slashzero slash I don't know why they put the zero at the beginning but there's a zero therez slash my database clusternames schema slashtable name slash star. parquet is what you have to then sort of translate into your parquet or your delta table structure in some form or fashion.
And so with the export flow the restructure lambda has a regular expression. Um if you're scared of those then be warned. um you have to use a regular expression to fish out the datas from that prefix you need and translate those to your delta table path. The nice thing about having that tool in your toolkit is this for in our architecture actually does our S3 bucket replication sort of by default as well. So we don't replicate from the landing zone into our delta table.
Um we have it write to a different S3 bucket. And the reason we don't let write to our data lake is because, as I mentioned, is just shooting infinity paret files of various shapes and sizes into an S3 bucket. And we don't want that in our data lake. And so we're also using this restructured lambda to take those paret files and make sure that they're going in the right locations and that only those files, not any of the temp files, the underscore files, or the other things that the export process might export goes into uh into our data lakeink.
Um, let make sure I got everything there. There's the oxbow part as well. I couldn't see it way over there. Um, because I'm presenting on a spreadsheet, I can't see it that much. So, I've got to look at the screen like everybody else. Um, but with export, this is probably the most complicated way of getting or most complicated workflow for getting data that comes from an external park system that I work with. I'm sure there are other more complicated ways to do it, but in our architecture, there's some step functions involved.
There may be another lambda. And at the end of the day, I personally I will prefer a couple of step functions in favor of a number of Spark jobs orchestrated by Airflow in a DAG that needs to trigger on some interval. Because one of the really great things about this approach is you can do two things with. You can export directly. you go say export these tables in paret or you can take a snapshot that you should already be producing on a semi-regular basis hopefully nightly if not more frequent and you just export the snapshot to parquet and so you have two ways to get data out of out of um out of but in our case we're taking the snapshots that we were already producing and our step function helps us identify when those snapshots are finished being created by the backplane and they can be exported to parquet All right, Kinesis data fire who's I have not used output conversion in anger in production.
So caveat about a lot of things that I'm about to say. Um but with out output conversion you can just say take this stream I think that's the name of the topic equivalent in data fires and produce parquet from that. There's some nuance to that configuration on how big your parquet files are or how frequently you might need that like flush interval or whatever your your batch might be from a from a streaming data pipeline standpoint.
But one thing that is really important to understand about output conversion is you pay for it. AWS will never let you get away for something with something for free. And so if you have let's say a thousand messages per second flowing through data fire hose that costs you know a hundred bucks. That's not actually what it cost. It cost 100 bucks. I don't know how much it cost. Output conversion on top of that usually costs like another hundred bucks.
And so for high throughput streams, I have walked away from systems before they went to production because the cost of output conversion with a very high throughput stream, we're talking like hundreds of messages per second, is just too expensive to where it no longer justifies the cost of using data fire hose. That's where a Kafka or or a system where you have more provision throughput is probably going to serve you better.
Um, unless you're tightly wound to data fire hose. But if you already have data fire hose and you already are incurring this cost of pushing data into it, enabling output conversion is a pretty simple from an opportunity cost, a labor cost standpoint to just produce those those data sets into uh into S3. And then Oxbow says, "Hey, there's a new new file. Let's go do that. Now, Airbite is something we have deployed in production, but we are a little bit behind at the booth downstairs.
They can tell you all about the wonderful new things that have come in the more recent releases of Airbite. I have not used those, but Airbite is a really great tool. It's open source, which means I like it more just by default. Um, but one of the really interesting things about Airbite is for a lot of data pipeline work that we're doing, especially around ingestion, it's not really that interesting to begin with. It's like we've got Google ads that have to come Google Ads data, Google Analytics data.
We've got this thirdparty provider that provides data here. We've got to go get it through a REST API, blah blah blah blah. It's really boring work that nobody likes to do. Anyways, Airbite gives you a lot of that connector marketplace to pull those in um into this sort of intermediate layer that they have and then plum that through to the S3 destination, the Azure, whatever storage destination that you might have and that supports the parket output.
One of the big criticisms and I went down to chat with them about air this morning. One of the big things that we've hit a snag with is large data sets. So looking at how airbite was implemented, I thought it was sim simultaneously very very cool but also terrifying because the way that it works is everything is JSON and it's piped like literal Unix pipes between two processes where one is writing your input data say from Google Analytics to JSON line delimited messages that it pipes to standard end of your output connector.
And so if your input source is hundreds of gigabytes of data of parquet coming from somewhere or from some other thirdparty data source, it's going to have to serialize all that to line delimited JSON and then go through, you know, the drinking straw over to the output connector and rewrite reerialize paret. So it was hugely inefficient. They've since made some changes to the protocol and this is what I was asking them about downstairs or I guess it's over there. um is they now have a way to get large files sort of out of band from one side to the other.
And so if you're taking large data files from an external system, that approach will probably allow you to take very large data sets out of one thing and put them into paret format in S3. Um so my biggest con for air bite I have been assured has been fixed. Um which is really exciting but as I said it's open source. Open source is great. Um, so I want to talk about Kafka Delta Injust because this is what sort of started our journey at script.
My colleague gave a presentation I think in 2022 and in 2022 we cut our data ingestion costs by switching from a Sparkbased workload to a Kafka delta ingest based workload by 90ish%. We saw a tremendous reduction in data ingestion. But the way that Kafka delta ingest works is we have to sort of pin a running container of Kafka delta ingest to a partition in Kafka because we've got to make sure that we get every message.
Ordering is not guaranteed in Kafka except on the partition level. And so you can only incrementally move your your consumer if you're looking at the partition. And so we had we threw out this big Spark infrastructure and we brought in these you know Kafka delta ingest rustbased containers that were running. But then we started to blow those out to where we were autoscaling these container groups out to run a large number of containers because we had so much data that we had to have kind of container per topic or container for groups of topics.
And one of the big downsides of this approach is if you're not familiar with the Kafka architecture is your table might have one consumer group. And whenever there's a consumer group change or your uh subscription changes in Kafka, all of your consumers in that consumer group need to rebalance. And so what we found is that as we scaled up our data through Kafka delta injust or through Kafka and then Kafka del Delta ingest, it wasn't the writing to delta, it wasn't the writing to parquet, it was actually just churn of autoscaling up and down to meet our needs that was causing the most um I say problems in our production environment.
And so some of what led me down this path was trying to fix this problem. Kafka delta injust just has too much consumer group churn because of the architecture of Kafka and the architecture of Kafka delta injust. So if you can avoid running a process to consume data and just let something else write it that's certainly my preference. One of my colleagues here, Anish, I'm gonna call out Anish. If you could wave at her, say, so you can ask him questions.
He's got a talk as well. He or his team and my team built another thing called SQS injust because I was, you know, really excited to do all cloudnative things all the time and lambda is a big part of that. And so what we identified was we had all some workloads that needed Kafka. They needed that extremely high throughput. We couldn't put them in data fire hose, but we had a lot of data sets that were streaming streaming, but they were streaming like in the dozens of messages per second or dozens of messages per minute.
Like we're talking about data volumes that you could solve with a bash script. And we had a number of those. We had a tremendous number of those. We probably had of our total topic count, I would say 75 80% of data streams were actually in this low throughput categorization. And so what we did is we ripped those out of Kafka for a lot of reasons and we started to put those into SQS. And one of the nice things about SQS is you can trigger your Lambda to just when messages arrive in SQS, it gets triggered with the messages that were in SQS.
And then what SQS does unlike Oxbow, it does the actual data wrink. So Oxbow just knows that a file exists and updates the delta transaction. SQS ingest is much more like Kafka delta ingest or your traditional data data ingestion pipelines in that it actually performs a data file. Right? Something that I learned along the way is a lambda function has a six megabyte argument invocation limit. And so that means every bite that the lambda was taking out of SQS could be at most six megabytes of data.
And if you think back to our transaction log, that meant that we were creating at most a six megabyte uncompressed parquet data file and a transaction log at the same time. And so for a number of these streams, we saw that our transaction volume exploded and or not yeah, our transaction volume exploded, but our data volume really wasn't that large because the model of lambda based invocation really wasn't cutting it for the amount of data that we were trying to push through.
We talked about this with our AWS team and one of the interesting things that was suggested by who who is our TAM or not TAM essay at AWS I give him credit because this was his idea was just to go grab more messages and I felt really dumb when he said that because God that is so simple I should have thought about you can also just go retrieve messages from SQS of course it's just an API and so what we actually use SQS for is or with is this sort of buffer more messages setting where The lambda invocation triggers that data is available to be read and then once it it receives you know we'll say our six megabytes of data but then it says I can run for let's say I have a timeout of two two minutes I can run for 60 more seconds to grab as many data any many rows as possible from SQS and then do the data file right and so that's how we've actually deployed it solve this transaction log volume problem is we just use the SQS lambda invocation as a signal and then go grab a bunch more data.
And the really novel thing about this is that means when there's no data there, the cost of our ingestion pipeline is zero. Which means we also were able to enable teams to just say like if you want a data pipeline, you need to give me an AR that can write to an SQS and you then you get a data pipeline. We no longer have to be involved in capacity planning in a lot of cases. We no longer have to be involved in cost discussions for new data pipelines.
We can just say if you want to get data from here to there just give us an ARM that can write to SQS and then if they're writing data we incur costs if they're not writing data if they forgot or they decided against it or the experiment failed and they turned it off there's no cost there's no ongoing cost and that is a pretty big deal for a company like Scriptur when an experiment as quickly as possible and not all not all of our experiments really go anywhere um so this is not real numbers.
Just to be clear, I'm not allowed to share real numbers. Um I will get yelled at. Um but when I think of our our two data ingestion flows, we've got streaming data and then we've got online application. The data coming from something like that. When we went as I mentioned to Kafka Delta Inest, we went from a sort of spark based approach to streaming data. We cut it tremendously. We cut that cost tremendously. when we brought it to SQS ingest or airte or not airbyte excuse me um oxbow we cut it even further like to take a data ingestion pipeline cut the cost by 90% and then two years later cut the cost of that by 90% I felt really good about that it seemed like a really great idea what was even better than that to me is the OLAP import became sort of a non-issue we don't really run any infrastructure aside from a couple lambdas to import our OLAP data we used to run replicas where we could sort of scoop Scoop, if you're familiar with Scoop SQ, um, effectively scoop all of our data every night in a big Spark job.
We would have our replicas autoscale up. We would copy terabytes and terabytes of data into our data lake and then shut everything down. It was huge operational complexity. There's a handful of lambdas and a couple step functions now involved in that. And so the operational cost being what it is, that small yellow line is less impressive to me than the fact that we have a system that no longer requires maintenance. We no longer are maintaining database replicas.
We're no longer worrying about replication lag. We're no longer worrying about any of the infrastructure standing up Spark jobs, orchestrating, you know, this has to happen, this has to happen, this has to happen. So the I would say the opportunity cost that we were spending on that system was much higher than the operational cost. And switching to just having give us parquet let us drive both of those costs down, which is really important.
Um, I have eight minutes and 30 seconds left. If you have questions and to give you a final graph, if your questions are formed as a question, they will almost certainly be answered. If they are formed as a comment, they might not be answered. Highly unlikely that I'll answer those. But if you have questions, I'm happy to. Thank you for your attention and thank you for stopping by.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.