Skip to main content

Crypto at Scale: Building a High-Performance Platform for Real-Time Blockchain Data

Summary

  • Real-time blockchain intelligence: Elliptic built a scalable, high-performance data platform on Databricks delivering actionable blockchain insights at scale.
  • Risk and compliance: Organizations use the platform to manage risk, investigate financial crime, and stay ahead of evolving threats in crypto markets.
  • Enterprise performance: The architecture integrates multiple technologies to provide fast, reliable intelligence for real-time crypto data analysis.

Crypto at Scale: Building a High-Performance Platform for Real-Time Blockchain Data

Watch: Crypto at Scale: Building a High-Performance Platform for Real-Time Blockchain Data
Talk by Ferran Cabezas Castellvi, Lead Data Engineer, Elliptic ; Matthew Moorcroft, Specialist Solutions Architect, Databricks
In today’s fast-evolving crypto landscape, organizations require fast, reliable intelligence to manage risk, investigate financial crime, and stay ahead of evolving threats. In this video we will discover how Elliptic built a scalable, high-performance Data Intelligence Platform that delivers real-time, actionable Blockchain insights to their customers. We’ll walk you through some of the key components of the Elliptic Platform, including the Elliptic Entity Graph and our User-Facing Analytics. Our focus will be put on the evolution of our User-Facing Analytics capabilities, and specifically how components from the Databricks ecosystem such as Structured Streaming, Delta Lake, and SQL Warehouse have played a vital role. We’ll also share some of the optimizations we’ve made to our streaming jobs to maximize performance and ensure Data Completeness. Whether you’re looking to enhance your streaming capabilities, expand your knowledge of how crypto analytics works or simply discover novel approaches to data processing at scale, this video will provide concrete strategies and valuable lessons learned.
Here’s more to explore:
Production ready data pipelines for analytics and AI: https://www.databricks.com/solutions/data-engineering
The Big Book of Data Engineering: https://www.databricks.com/resources/ebook/big-book-data-engineering-2nd-edition
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Elliptic's platform for crypto data?

Elliptic built a scalable, high-performance Data Intelligence Platform on Databricks delivering real-time, actionable blockchain insights to customers for risk management and compliance.

How does the platform manage real-time crypto threats?

The Databricks-based architecture processes blockchain data at scale, enabling organizations to detect financial crime, manage risk, and stay ahead of evolving threats in crypto markets.

What scale does the platform operate at?

The platform is designed for enterprise scale, combining many technologies and touchpoints to deliver fast, reliable blockchain intelligence to customers continuously.

Why is real-time blockchain intelligence important?

The fast-evolving crypto landscape requires organizations to act quickly on intelligence. Real-time insights from blockchain data enable faster risk assessment and compliance decision-making.

Full transcript

Hello everyone. Welcome to our session. We're going to get started. First off, I want to mention because we don't have the slide there. After the talk, please if you could fill out the survey, it helps us a lot. Um, but yeah, we're going to talk today about real-time crypto data intelligence at scale. I am part of data bricks. I work as a specialist solutions architect primarily focused on ditch engineering and streaming and basically this is going to be a little bit of a journey on the partnership that elliptic and dig has have over the past two years and actually we how we scaled this platform over time you'll see some mistakes we made along the way and how we improved them but yeah excited to talk about it.
Yep. Thanks. Hello everyone. and today as said we're gonna be talking about the evolution of the data platform and the and how we have built a real time platform. So the agenda for today we're going to a bit first talking about elliptic the history of it. We're going to be talking about database components that have enabled userf facing analytics. We're going to be explaining what is userf facing analytics. We're going to be touching about optimizing streaming jobs best practices and key takeaways.
So to start with what is elliptic at elliptic we help crypto businesses financial institutions and law enforcement agencies to fight financial crime in crypto space and just to give an example of that and show the products we have. So this is the risk management products that we have and essentially this allow us to you know screen an address or a transaction into into a product and then essentially with some risk rules that the customers are able to configure we we provide like a a risk score essentially right and we provide that on the sort of funds and the destination of funds.
Um so this product is powered through the elliptic entity graph. We're going to be touching a bit on this but essentially it's like nodes edges right nodes being blockchain addresses that are labeled unlabeled and the edge is essentially the flow of money and then why this problem is hard. So unlike traditional banking where everything is known like all entities are known in crypto you know everything is fuzzy. So we have whenever we have um a customer customer request essentially we have to run thousands of traversals in real time in order to provide insights to the customer right and on top of that we have 50 different blockchains with different characteristics that we have to integrate into our platform which makes this problem even harder.
The other product we have is the graph visualization product and this allows essentially single click crashion investigations to cover elite activity. This is powered through the elliptic entity graph and the userf facing analytics and as you can see on the picture on the left this graph essentially the user can see the UI right the nodes and the edges we can see in pink this like a unlabeled address. We don't have information.
We don't have a label for that address. But in the middle, we have a sanction entity, right? Like we receive this ent this label, this intelligence. We show it on the UI and the user is able to understand more what's happening. The other component that is very important for the graph visualization product is the userfacing analytics. And essentially this is a tab we have on the right that is able to provide more insights to a customer.
And this is key to have um an effective um investigation for for the customers. Key insights that we provide could be the you know given an address right a blockchain address we are able to provide a list of the different transactions that this address has been part of. We are also able to provide like total incoming value that a certain address has received the total going value that a certain address has ever sent.
And why is this a problem? A big problem userf facing analytics like it's a hard problem because essentially we have it requires a lot of query filters right as you can see on the on the tab on the right we have one two three four I don't know how many tabs we have but every tab that we have it requires different query filters and not only that but these query filters like are quering the history of all of blockchains that we provide that we support in the platform.
So 50 different blockchains, Bitcoin, Ethereum like 50 more, 40 47 more and all the history of it and I think it's around like 300 terabytes or something like that in total that we have to query in real time and provide these insights to our customers. So just to give a bit of history of elliptic platform and you know the different evolution of it we start very humble just supporting bitcoin ethereum we were having neo forj for graph database we were running traversals on it everything was in batch you know we had we were we were running like batch processes very expensive ones to dump all this data into into neo forj and essentially we had one neo forj for every blockchain which made it very complicated to scale essentially so As we start to grow and integrate more blockchains into our platform, this was a challenge, right?
And that's when in 2021, we put a lot of effort to build the scalable platform essentially that was blockchain agnostic powered by Dynamob and Lambda that could power both the entity graph and the userfacing analytics. So you can see here the diagram, right? the architecture diagram where data ingestion essentially it's blockchain data coming from our it's in Kinesis but essentially we have a team dedicated team that they are the domain experts for blockchain and they push all this data into Kinesis and this comes in real time we also have the intelligence right we have we have seen in the UI before where you know on the addresses we might have intelligence on it and we put a label right and this intelligence is essentially computed from our intelligence sim to do that, right?
We we use ML to do that as well. And just to focus on the top part of the architecture, right? We see the entity graph. We see the compute essentially is Lambda from AWS. We have the entity graph store in Dynamob. And then the grass reversal is done in a in-house grass server hosted in Kubernetes where every time we have a customer request again we run the you know thousand of traversals to provide you know real time insights in less than a second in less than a second to our customers and this powers the risk detection engine essentially.
The bottom part of the architecture we see the user userfacing analytics. It also reads the blockchain data and intelligence and store the this these tables into into Dynamob and again it's powered through the inhouse server and powers the graphics decision products that we have talked about. This is the real-time platform. It's important to say as well that the first time we have to integrate a new blockchain we have to do we with it in a spark essentially.
We use Spark to read all you know blockchain history and we dump all this data into into Dynamob. But we start to see that soon that the user facing this had a lot of challenges and one of them is what I mentioned before you know like it has a lot of tabs. It required a lot of query filters and suddenly we realized that this one not scalable right we had to add one index one table every time you want to power a new feature on userface analytics compute limitations so we're trying to do a big data job inside lambda and this was clearly not not not ideal so this prevented us from expanding and heavy maintenance architecture we were we were maintaining spark logic to initialize is and backfield all the blockchain data.
So this required like some custom you know like logic and code as well. We were like hosting all the real-time logic in lambda functional programming. So our engineers had to you know maintain two different repos super different and again this was very complex. Next milestone in our platform that I want to highlight is 2023 we fully migrated to databris giving us access to databris ecosystem. And then next thing important to highlight is 2023 blockchain transactions start hitting record heights.
And just to show some numbers here. So we can see here the graph which is like it's it grows from it's very small but it grows considerably incredible like from 2023 to 2025 and you can see that in January 2023 we ingested one billion transaction two years later we triple that the number of blockchain supported as well we doubled that and the number of structured data into a platform again we triple that. So our platform start to see like more blockchain data.
The costs were rising and we had to do something something to highlight here as well from the databicks timeline is that 2023 SQL warehouse rest API goes public preview and this is going to be useful for context for the next slides. And then we had to do something we had we we could not expand user facing analytics. We were blocked through the compute through the cost and we decided to do something right. We didn't have much engineering capacity back then and that's why we kind of do a kind of a a hack.
I don't want to say a hack but we we had to do something quick right and that's why you can see that we didn't touch the main architecture but with dotted lines we added extra computation extra storage and extra like rich computation to power userf facing analytics right so this quickly allowed us to expand the functionalities on userfacing analytics without committing much engineering resource essentially and again allowing us to expand right So just to touch on that we just essentially um stream all the data coming from Dynamo using sparse streaming storing it into delta and then quering it through SQL warehouse and then last milestone I want to highlight today of this evolution is that last move that we have seen you know like these dotted lines adding Dynamo and you know adding delta we did very in a reactive way you you know, we had to expand, we had to do something.
But then when we had time, when we had the buffer, when we had the engineering resources, we decided, okay, let's do something. Let's build the platform. Let's build a platform fully powered by structure streaming. We would ingest directly everything into Delta powered by SQL warehouse and this would power directly the userfacing analytics for our graph visualization. worth mentioning as well everything was governed by Unity catalog and that's that's pretty much the architecture we have nowadays so key wins of this architecture we have a streamlined development we have spark powering both batch and streaming right we have better data governance right now with unity catalog giving better access controls to jobs to data as well we have scalable compute so now we don't have the problems with lambda you know with factor streaming we can horizontally scale we can horizontally scale as well the SQL warehouse to query your data in Delta and yeah that's pretty much it from the key wins and now I'm going to be passing to who's going to be essentially touching on the key components from data bricks that have enabled userf facing analytics for us over to you thank you pan so yeah as pan was saying I'm going to be walking through the different components that we managed to work towards in order adopt data bricks features but also some of the thought process that went into deciding how we could actually make this scale.
The first thing I'm going to talk about is about the SQL warehouse which was a critical part of Ferrron's architecture and a lot of the challenges that SQL warehouses have in general are the fact that most of the times the cost of the product itself tends to be 25% of the overall cost that you have with entire platform and that is because you have to manage your infrastructure you need to manage the startup delays on running up a single warehouse idle time when it's not being used things like governance maintaining reliability and these like many more things.
Uh but the main challenge that elliptic had in terms of capacity was that they weren't dedicating time to actually building up the use cases but instead trying to manage the existing system which was not scaling. So some of the things that SQ serverless SQL enabled um elliptic was the fact that one of the things that Fernand mentioned was what which was the rest API that allow them to communicate with the SQL warehouse using a programming language without using directly SQL statement there. uh but also it allow them to iterate way faster.
And the reason they could iterate faster is because when they're using serverless, the startup times for the warehouses themselves tend to be in the seconds, which means that you could iterate, run it, run queries, turn it off without incurring significant cost. And this also increases productivity as well as lowering the TCO as you don't need to manage the infrastructure any longer. So we have the SQL warehouse but it's only going to be as good as we design the data layout and we're talking about terabyte level tables and as Pan is saying this is only going to grow over time.
So we need to be careful and think very uh well how we going to design that. So here we have an example of a query pattern that Elliptic has. They have many query patterns that they in their different use cases, but this is one of them where we have addresses which are a way to identify different wallets in crypto transactions and they want to find out for this specific wallet that what transactions has it participated in and how um like the being the USD value higher than 2,00 on the right hand side we have an example of what that could look like from a JSON perspective and having this this um query pattern and the data itself, we have to think about okay well what are the things that you're going to rely on to basically retrieve data from your tables.
So for the thought process part, we have a wallet address, which is a key that is very important when querying the data. And we're going to touch a little bit more about this because there's some things that about the hashes that aren't ideal for many situations, but in this use case particularly and in crypto in general is actually pretty important. The other aspect of it is the USD value, which is also very important considered that it's a common filter uh condition.
So we considered the different options that were available and just to remind you this was two years ago 2023 when we started the partnership and how we can actually scale with data bricks. So the options that we had available was hive style partitioning zording and liquid clustering which back then was in public preview and for that reason we ended up not using it because there were some limitations at the time around bigger tables.
Uh so we basically said like let's make sure we use something that we know is going to work uh and that way we avoid getting burned by any any of those limitations. So we went for hive style partition partitioning combined with Z ordering and this is a very common uh combination when working with big tables because they they complement each other. So we have high style partitioning which is really good for those low cardality columns and Z ordering which handles very well those high cardality columns such as USD value and things like that.
So for the hype style partitioning because the read of the wallets themselves were so important we had to find a way to simplify how we basically lower the cardality of the wallet so that we could basically partition it more efficiently. And for that we had an AR an algorithm that would help us basically truncate the address to specific characters so that we could spread the load of the data more evenly and that also allowed us to do partition pruning within data bricks with Z ordering.
We chose to use USD value for it which allows us to basically cluster the data more efficiently in term of the USD value column which also allows us to do file pruning. Still having gone through this and I'm actually going to talk a little bit more on the optimization part. We still had lot of skew where some partitions will have as low as 18 megabytes and some other partitions up to 130 GB and that is really really bad in terms of performance for queries that may basically read from those two.
So how do we get the data from our source to our SQL warehouses and basically build that data layout throughout that journey. So as mentioned we have the Kinesis connector and this Kinesis connector is a native connector that we have available within data bricks and the great thing about it is that it's paralyzed by shards but also it's managed for you. Having said this, there are two main threads that you need to think about when working with this connector, which is the record fetch buffer and batch execution.
The reason I'm mentioning this is because these play an important role in managing how much data you flow through your system and that is for actually going to talk a little bit more about that, but it plays an important role in making sure you give it stability to your pipeline and you avoid getting spikes of data that may cause the pipeline to lag. Moving to Delta Lake, the great thing about Delta Lake is that is a format that allows for streaming and batch, which is incredibly powerful.
And another great feature that it has is that it guarantees exactly once, which means that you can once you solve duplication, if you have any duplication at the source, once it's in Delta Lake, you don't need to think about potential duplicates there. As I mentioned with Kinesis, here we have as well two knobs that are very very important to consider while uh building pipelines and streaming that allow you to throttle how much data you read from your tables.
And it may sound counterintuitive where you're saying like why am I slowing down when I need to go faster, but you'll find later why that actually makes sense. So having said this, we've built the pipeline and we would like to say that we were done. But the reality of the situation was that there were a lot of issues that we still needed to solve in based on our original design and also because it the time frame of this lot of features were released from part of from data bricks liquid clustering actually went G which was one of the things we started revisiting.
Starting with existing situation that we had, we had high star partitioning and Z ordering like I mentioned before and this is a visualization of how that could look like where partitioning as you may know it offers uh physical boundaries within different folders to separate the data and liquid and Z ordering allows clustering between multiple ranges of USD value and basically combine the data into those same files. This improves the read times but it actually has some issues when treating it in this situation.
The main issues that you have here is the fact that partitioning partitioning is has physical boundaries and that causes you to have skew. I mentioned before that we have some folders that have around 80 megabytes other ones will have 130 gigabytes. Well, that actually provides in a little bit of inconsistent performance in your pipelines. Another issue that you have with Zingering specifically is that it causes some issues where you need to optimize and need to think in terms of maintenance because you have to specify what you want to optimize on.
And another important factor is that this is static. Once you decide how you're going to do the hive style partitioning, you can't change the columns that you've chosen without rewriting the entire table. And now we're talking about terabyte level tables. So rewriting every time you want to change it is not good. One other issue that we encountered with elliptic was concurrency. And with have star positioning and zed ordering, you get file level concurrency.
What that means is that if you have two pipelines that interacting with the same table, if those two pipelines touch the same file, it's going to cause one of those pipelines to fail. This is a situation that we had in elliptic. So that is a factor when we decided to revisit this as well. So let's talk about liquid cursor. You can see the visualization is a little bit different and the reason it's different is because with liquid clustering you don't no longer have physical boundaries for your data.
The algorithm itself that optimizes it will decide how to best cluster the data between the different columns that you've chosen. It also makes it way simpler where you now choose address and USD value for liquid clustering. You not need to you don't need to think about cardality any longer. So if you have a low cardality high cardity column it doesn't really matter. What matters most is that you follow your query patterns.
So before we mentioned that we're using wallet and USD value. Well, we look across by those. The great thing about it as well is dynamic. If your query patterns change or you have new queries that use different columns, you can actually add the new column, optimize, good to go. Another great implementation from liquid crushing is the fact that it offers rowle concurrency and that takes it a little bit level further where you have you can actually interact with the same file but as long as you're not touching the same row groups within the parquet file you're not going to get conflicts which reduces the hits when you're working with the same files by a significant margin.
The other thing that this solves is the fact that you no longer have skew because you're combining data between all these different uh columns and you don't have any physical boundaries. You now bring a little bit more stability in terms of how you're going to get performance in your in your tables as well. So now that we solved the later layout for these tables, let's actually talk about other optimizations that we considered when we revisited this.
And some of you may have already heard about deletion vectors. I'm going to briefly explain what they are and why they actually were created. So you may be aware that parquet is an append only format and that has some limitations especially in the new way of working with lakehouses where we don't only add data but we actually need to edit it or delete it and being paret in a pen only format that means that if you ever edit or delete you're going to have to rewrite the entire file.
In this situation, we're doing streaming. So, we are continuously rewriting these files, which is again inefficient. What deletion vectors does is actually creates a binary file that tracks the deletes that are happening within the park. And the engine itself with those will decide and amorticize th those deletion uh vectors to choose when is the right time to actually rewrite because it's going to be worth doing that compared to just adding it to the deletion vector.
And by doing this, we now improve dramatically the performance of doing merges because we no longer have to basically rewrite an entire file. We can simply create a new file and tag the deleted deleted vectors on on the older parket. Now that we're talking as well about merge operations that are happening within the pipeline itself. This is something that I've encountered many many times with customers and I'm not sure if people are very familiar with static file pruning or dynamic file pruning.
Is anybody familiar with it? Not many people which is unfortunate. So I'm going to actually explain why this is an issue and why you should be considering when you're building structure streaming pipelines. Static file pruning is a way of specifying your in your merge operations how you want the target table to behave. You're specifying when you're doing a merge operation that you want to limit the data that you're going to do the merge operation on.
The reason you need to do this is because when you're doing a merge operation where you're not specifying the filtering, the static filters that you want to use like we're doing in example on the left where we're specifying the time stamp that we care about. The scan operation on the target table will basically read the entirety of it. Meaning if you have a 150 terabyte table, you're going to read the entirety of the 150 terabyte table to figure out what exactly you want to join on.
Many many customers don't consider this when they're building their pipelines and they start complaining that the performance of it is actually lagging significantly which is something that you should be considering if you're doing it and in this example that's the way that we approached it as well within. Moving on to the right hand side where you have dynamic file pruning. This is actually an improvement that comes with using Photon which is another is a native engine with data bricks that you can use which makes the problems it solves the problem basically.
So instead of you having to think about doing file pruning yourself where you have to decide what columns you're going to do the filtering on the target table you simply do a merge operation which is pretty simple. So you simply say like I want to join by hash and in this case I want to join by Unix time stamp and under the hood it will actually do the logic that you would have to do on the left hand side. So it would basically be like okay this is the microbatch that you want to join on going to pass the columns that I'm going to join on to the target table and the target table will use that to basically do a filter and only bring in the data that you're going to need to do that merge operation.
So having talked about this, Fernand is actually going to talk a little bit further about how they thought about the data layout in terms of columns to optimize the dynamic file printing of the tables. Thank Matthew. So what I want to explain now is a bit how important is to choose a good clustering key and for that we're going to base with example on the right. You can see similar JSON that showed before where we have Unix time stamp.
The network is essentially the blockchain height is the block block height or the blockchain transaction hash random hash from is the sender to is the receiver asset is um intuitive already and USD value is the amount of money that the sender has sent to the receiver right and essentially what you want to do here is store these transactions in the transactions table right this transactions table is going to be around 15 terabytes data right we want to test how you know how we can can improve merging this microbatching from a structure streaming to this 15 terabytes transactions table the transactions table is a liquid cluster table we are using a cluster five workers I3 large as said we're going to be triggering trying to use dynamic f pruning so every cluster key that we have we have to make sure it's it's passing on the merge key and in this case the merge is the transaction hash network and unique stam sum which are the fields that uniquely identify a transaction.
So the first result or the first example that we're going to give is just setting the transaction hash as a cluster key right which is required for RIP performance right we have this table we have to query it through the hash so we cannot change that and by only putting transaction hash we can see that the number of files prune is is is nothing right we prune less than 1% of the files during merge and this results in a very slow merge right this result more than 20 minutes per microbatch But what happens if we add unique stam sum into the cluster key?
Unix times sum is an order column, right? And the results are amazing. You know, you can see here the number of file prunes goes from less than 1% and more than 99% making the merge into delta table right into a 15 terabyte delta table making this merge is less than one minute which is incredible results. That's it in terms of optimizations. We're going to show a bit best practices that we have in our architecture. Um, in order to do that, we have to do a bit of zoom in into the architecture that we were touching at the very end, right?
The the architecture we have currently and that's pretty much the architecture that we have currently in our platform. Zoomed in, right? We can see we have two streaming jobs, right? We had the first one is the transaction stream reading from two sources. The first one is blockchain data coming in and streaming right. We have 50 different kinesis reading but we have all the data in kinesis 50 different kinesis one for every blockchain and this is being read by the transaction stream and this stream is reading as well from another table.
We can see the backfield table. What is this backfield table? In this backfield table, every time we have to integrate a new blockchain into the platform, we just batch all of this data into this table. And essentially, it's just in one commit, we just have a lot of data, right? And then this stream is writing into a transactions table, right? This 15 terabytes table I mentioned before. And we write it there just to accomplish it potency.
We don't want to process the same transaction twice. And just by writing it there, we just ensure like all the transactions are just process one. And downstream of that we have the userf facing analytics stream which is reading the CDF from this table and it's also reading the CDF from another table in this case the intelligence table in this intelligence table similar as in the back field right or intel team they batch all of the intel the new assertions the new intelligence that they have created through ML manually whatever into this table and in batch right And this userfacing analytics stream reads both sources right and writes into four different aggregation tables that are needed for our userf facing analytics.
You can see here the scale more or less the size of each table. So the first best practice that we have to adopt is atomic rights especially on the user facing analytics the stream right what happens if we do the first two rightes on the first two tables and we have a failure on the on after that right after before writing to the transactions table three we would not have data completeness right we have some tables having some data other tables having another another data and then if we just restart the microbatch we would add into the transaction table one and transaction table to duplicate data, right?
We have deltas. We are adding deltas. Essentially, we would end up with, you know, like double counting. So, in order to do that, we implemented some strategies to whenever we have a failure, we're going to do some retro. We're going to discard all the microbi into S3 to process later and to quarantine it. And then we're going to roll back transaction table one, the aggregation table 211 with the delta roll back. And that ensure we have data completeness.
The other best practice we adopted here is graceful stopping. Right now when you have a spreaming job and you have to start stop it and start it just to do maintenance and change the HR for example we use Scala and we don't know at what point in time this this stream is and essentially we had to using spark listeners just you know detect whenever the microbatch has finished and we just use just spark context just stop the spar session and ensure you know like we always write whenever all the writing into different table has finished we don't stop the streaming mid midway.
The other perspective we have adopted is limiting input rate as has mentioned using these two configurations. And I'm going to just do a bit of deep dive into this one, right? And sorry, let me go back one sec. And before doing this deep dive, I'm going to give an example of what happens when we do an intelligence load, right? when we do an intelligence load into into the intelligence table and especially when we do a batch of let's say three three gigabytes of 10 gigabytes and we don't want the stream to be lagging right the stream is is configured to process in a certain capacity and we don't want a big batch to cause lag to the other tables.
So this is a an example of the Intel table from the uni from the Unity catalog essentially and you can see an example of an Intel load of 195 files and 3. 5 GB size. So by using the max bytes per trigger this is very interesting we have seen that thanks to delta um this big batch of 3. 5 gigabytes it is splitted into three small chunks three or more where it's split into into small chunks and for those you are not familiar with the checkpointing offsets right whenever we have a streaming job running in in in structure streaming or lit whatever we have this checkpointing folder we have this offsets folder folder that keeps track of what point in time we're reading from the from the delta table.
And on the right you can see three different microbatches. This case is microbatch 62, 63 and 64. And you can see it's a really little but you can see two reservoir versions. The first reservoa version is the version of the transactions delta table that the userfacing analytics stream is reading. And the second version is the version of the intelligence table that again the userf facing analytics stream is reading. And you can see that across all three different microbatches the index is exactly the same.
Meaning that we are processing this big batch into smaller chunks not causing any lag when when writing to different tables. So that's pretty much it on best practices, key takeaways of this talk. So we have seen that we had to do kind of a hack in 2023 when blockchains were rising. We had to add, you know, this streaming pipeline right into Delta. We have built a scalable platform proactively and ensuring we are ready for the next crypto wave. we know it's going to happen and we don't want you know to not have a platform that can process data scale databris has allowed us to focus more on our domain so components like SQL warehouse components like dynamic for pruning even spark you know when we trigger spark job we don't you know configure the memory we just rely on datrix to do that and we have seen that it's way better to do that than just writing the memory the driver like you forget about this just use spark and dri will figure out that for you and this allow us to focus more our domain on on what we what we care which is crypto and data processing with crypto.
We have seen that in a legal architecture we can power real near real-time features and probably this was not possible 10 years ago with parquet but now with all the components with data ecosystem this is possible and we have seen it with the userf facing analytics and lastly but not last data sharing I haven't mentioned anything about data sharing in this presentation but thanks to unity catalog apart from the governance this has allowed us to you know seamlessly sharing data thanks to delta sharing and I think that's pretty much it for today.
Thank you for attending. We are going to have happy to attend questions downstage. So yeah, thank you. Thank you.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.