From Datavault to Delta Lake: Streamlining Data Sync with Lakeflow Connect

Summary

• Data Vault migration: Australian Red Cross migrated from Azure SQL Datavault 2.0 to Unity Catalog using Lakeflow Connect.
• Multi-layer approach: Raw vault and business vault tables sync alongside view definitions converted from T-SQL to SQL.
• Validation at scale: Lakehouse Monitoring ensures data consistency and quality throughout migration and ongoing analytics.

From Datavault to Delta Lake: Streamlining Data Sync with Lakeflow Connect

Talk by Andrew Clarke, Senior AI/ML Engineer, Australian Red Cross Lifeblood ; Olivia Ren, Solution Architect , Databricks
In this video, we will explore the Australian Red Cross Lifeblood's approach to synchronizing an Azure SQL Datavault 2.0 (DV2.0) implementation with Unity Catalog (UC) using Lakeflow Connect. Lifeblood's DV2.0 data warehouse, which includes raw vault (RV) and business vault (BV) tables, as well as information marts defined as views, required a multi-step process to achieve data/business logic sync with UC. This involved using Lakeflow Connect to ingest RV and BV data, followed by a custom process utilizing JDBC to ingest view definitions, and the automated/manual conversion of T-SQL to Databricks SQL views, with Lakehouse Monitoring for validation. In this talk, we will share our journey, the design decisions we made, and how the resulting solution now supports analytics workloads, analysts, and data scientists at Lifeblood.
Here's more to explore:
Databricks named a leader in the 2024 Gartner® Magic Quadrant™for Cloud DBMS: https://www.databricks.com/resources/analyst-paper/databricks-named-leader-by-gartner
An open, unified approach to your data, BI and AI workloads: https://www.databricks.com/product/databricks-sql
See all the product announcements from Data + AI Summit: https://www.databricks.com/events/dataaisummit-2025-announcements

FAQs

What is Datavault 2.0?

Datavault 2.0 is a data warehouse architecture methodology with raw vault tables storing detailed historical data and business vault tables adding business logic. It provides flexibility and scalability for enterprise data warehousing.

What does Lakeflow Connect do?

Lakeflow Connect is a migration tool that automates data transfer from external systems into Unity Catalog on Databricks. Red Cross used it to ingest raw and business vault tables without manual coding.

How were views converted?

Views defining business logic were converted from T-SQL to Databricks SQL using a combination of automated scripts and manual adjustment. This ensured business logic transitioned correctly to the new platform.

How does Lakehouse Monitoring ensure quality?

Lakehouse Monitoring validates data quality, tracks schema changes, and monitors data freshness. Red Cross used it to confirm successful migration and ensure ongoing data integrity supporting analytics.

Full transcript

How's it going everybody? Welcome in. How's the summit going for everyone? Pretty solid. Pretty solid. Cool. Uh quick housekeeping thing. Uh we'll probably run the full length of the session here. So for any questions, just we got to turn the room over. So just step outside and ask uh and questions out there. Uh but yeah, happy excited to announce uh or introduce from Red Cross lifelood and Olivia Ren from Data Bricks to talk to you about LakeFlow Connect.
Uh so pass it over to them for joining us this afternoon. Um so we'll be talking about from data vault to Delta Lake streamlining data sync with Leglow Connect. Um it's a bit of background about lifeblood as there's a lot of people in the room who may not have uh interacted with us in the past. We're we're a funded by Australian governments to provide life-giving blood plasma transplantation and biological products including breast milk and freal matter transplantation to deliver worldleading health outcomes.
We have close to 3,800 people working at Lifeblood. From our donor center teams who support our generous donors who give to those in our manufacturing and quality teams who process blood, plasma and biological materials safely. Our many scientific and testing teams who ensure our products and services are worldleading and safe. And our various support functions across so many areas as we all work together and ensure we deliver a worldclass efficient and effective blood service for the health of our community.
We operate across 112 sites Australiawide and Lifeblood serves over 27. 6 million people stretched across 7 million square kilometers. Ultimately though, our more than 610,000 donors are our lifeblood. Without them, we couldn't deliver critical health services to our nation. It's the generosity of these donors that drives us to continually improve our world-class capability. In the last financial year, we have booked more than 2 and a half million donation appointments, resulting in 1.
6 million life-changing donations of blood, plasma, and platelets. Every day, we welcome collections from thousands of generous donors that provided the gift that can be replicated. Of course, every single donation that we receive is life-changing. But just to give you a further quick snapshot, in the last financial year alone, LifeLood has collected, processed, and delivered over 4,000 lers of breast milk to feed vulnerable babies.
Uh delivered almost 1. 1 million fresh blood products to support Australian hospitals and healthcare providers and supplied 869 tons of plasma to be fractionated by CCL bearing to then be turned into life-saving products to change the lives of patients. Everyone across our organization works tirelessly every day to support the Australian healthcare community. And our team in particular is dedicated to underpinning LifeBlood's purpose of turning life-giving donations into life-changing outcomes by using data and analytics to drive improvements, measure performance, and provide insights to every business unit in our organization.
Within lifeblood, we work as a center of excellence for analytics, insights, and machine learning. meaning we have the privilege of assisting analysts and subject matter experts with their analytics needs across the organization. Some examples of the types of analytics projects we provide are using cellular segmentation models to detect specific cells or to detect and measure red cells. We also support a variety of early trials into business as usual initiatives um such as lifeblood gives our rewards program for donors and initiatives to improve the donation experience such as lifetimes weight pred weight time predictions a trial solution that we ran in previous years to add value to the donation experience by using predictive analytics to anticipate delays and notify donors of any weight times before they arrive in the donor center.
We also support the analysis and planning of our 108 site donor center network through geospatial analytics to support uh to support our teams to ensure our collection network is reachable and effective for donors. Lastly, we provide an extensive catalog of analytics dashboards and other statistical analysis reports to support our lifeblood purpose. Of course underlying all of this is data governance and data availability to drive these um these initiatives and uh projects.
So today I'll be talking about a component of the next stage of our data platform which is as the talk started off syncing a data vault as the SQL server data warehouse into the lakehouse. But to get to that point and to provide balance to the decisions and explain some of the constraints we're working on under, I'll need to start with some context. At Lifeblood, we've been using data bricks for around 5 years now. Initially it was primarily for us an advanced analytics machine learning and data science platform but the scope has grown as have the data teams that use it including our data and engineering teams who use it for their near realtime data processing particularly for operational technology and regulatory validated pipelines and analysts across R&D business intelligence finance and other analysts in other business units including marketing use the platform in similar ways that we do But as a result of that growth, we operate under a distributed governance model in our data bricks platform.
And where there are items that overlap overlap across boundaries, we collaborate closely with our colleagues to develop the best solution to support not just our team but the analysts and users across the organization. The solution we built and we'll discuss today was a result of that collaborative process with the overarching goal to make our data vault warehouse available in data bricks for all users in the lowest friction highest cap highest capability way possible.
The data vault warehouse we are working with is a modern highquality implementation and integrates and combines some of life blood's most core applications data. It has built into it over a decade of established business process and logic and is a highly trusted and relied upon component of our data platforms. It was critical to the next stage of our data platform to bring our data vault warehouse and our lakehouse platform together.
So going beyond that headline goal of making data vault data available, what did we actually need? What were the requirements for the team really to support the analytics machine learning AI workloads on Azour data bricks we needed a replica or sync of the information m layer which is the equivalent to a gold layer in data vault from the SQL server data vault data warehouse and some of the technical requirements is that we need to be able to support daily and intraday sync at minimum um and intraday for us is typically an hourly refresh uh timeliness so the first sync of the day had to be achievable within two hours um to meet our start of day processes and would need to as the data vault warehouse wasn't initially designed for this load.
So we'd want to ensure that we had predictable and manageable load on source to not cause any upstream impacts as we're effectively making a replica of a trusted source. We needed to be able to maintain the quality with accuracy and completeness certainty and we had some certain constraints around scalability. So we expected that at a field buildout this would be 300 plus information arc objects or 2,000 plus raw vault business vault tables and that's a subset of the overall data vault warehouse that we have at lifeblood.
We also determined that full change history would not be required as point of time latest of time of sync is sufficient in terms of the data that we bring into the data lake. if possible would like more frequent sync. Who wouldn't? It's always good. Um, integrations into Unity Catalog and our existing processes including uh DT um integration into Unity Catalog observability particularly for error detection and downstream pipeline um recovery and we also needed support for schema evolution because we expected those tables to change from the source quite regularly.
So we at minimum needed new new columns to come through through schema evolution. Lastly, we had sort of the constraints around our implementation as the overall goal was to reduce the complexity of access to data. That was always going to be a key driver. But we also had to achieve this project within existing headcount and skill sets with potentially some skill up within the team. And somewhat of an important note is that we're primarily an analytics team.
We're not a ICT team. We're not an IT team. We're not data engineers. So it had to be manageable for our skill sets. Um we also needed the ongoing effort beyond the initial project um for tables changing at source to be manageable in a business only business as usual only effort. And lastly it was really important to us where there was sync failures that there would be a straight recovery from failures. So from that we took away those requirements.
We evaluated the connectivity methods available in data bricks and the CDC capabilities of Azour SQL server. Um we assessed our options and essentially we committed to exploring at minimum two pox to an initial trial. Um however before we go into the technical evaluation I'll hand over to talk more about data vault and how it's normally emulated on data bricks. Yep. Hi everyone. So data vault 2. 0 their pattern on datab bricks follows the enhanced medallion architecture.
So data is flowing on the screen through several layers. So first of all starting from the bronze layer you have your staging lending injection layer. So here you have data ingested from the raw source uh layer and also this data is being served as one source of truth and you're enabling that for the um full auditability and traceability and following by that we have the silver layer. So in this layer we have rollout and business vault.
So robout your data model is being constructed into the hops, links and satellites. Hubs are your um business keys and links are your relationships. Satellites are your descriptive data. And then in the business vault layer, this is where your business rules and your data quality checks has been applied. And optionally, you can choose to create point in time table and bridge table to help you for further analysis. Finally when we're reaching the um go layer which is also the presentation layer um in data vault pattern we call information m.
So in this case we're creating the customer centric views typically in views um for the reporting purpose. Thanks. So in our technical evaluation we needed to consider our options and seek a solution that would be fit for purpose for the next five years. Um and to meet that complexity constraint, we initially looked at options within our current tools and platforms as of November 2023 when we kicked off this project with as a preferred option with custom or external tools being a secondary consideration only.
At time at the time we had the following options. Firstly was raw only. So this would be just reusing the raw landed data from our existing data vault and redoing everything on top of that. So this would be effectively a full migration and replication away from um SQL server. This would involve significantly more effort than would be achievable within this project and it was well outside the constraints of this initiative.
So that wasn't really on the table unfortunately. So the second option we looked at was JDBC which we already used within data bricks quite extensively. This would be accessing the IM information mark layer directly from Missou data bricks. This was a potentially viable option for our power users but fell a bit short in terms of the ease of access for regular analysts and for that we sort of thought it wasn't achievable.
The third option we considered was lakehouse federation which had recently been released at around that point. This technology would actually suit quite a few use cases with smaller databases and warehouses and it was a big improvement on ease of access and integrations within data bricks and unity catalog. But we had some concerns about performance and particularly the predictable and manageable load on source um and the complexity that would be introduced for the analysts that would need to accept um their queries being pushed down to the SQL server and understanding what that impact would be and what the performance would be.
Lastly, we considered a a way to build build a custom watermark based CDC uh and run through Azour data factory. This would have been potentially viewp specific and while possible we thought this would be a lot of extra work to set up. Maintenance would be difficult and recovery from sync errors was going to be quite tricky in some cases. So none of those options we had natively in the platform at the time were quite meeting what we needed.
So we kept looking and we ended up with a pragmatic solution to use a SQL change tracking and that's where we get to here. So the decision was essentially rather than trying to directly bring in the information m or starting over with the raw data from scratch we explored replicating the silver layer. So the raw vault business vault model data which appeared quite possible with change tracking from Azour SQL server. At the time, the feature that would be eventually known as let flow connect was just starting to be mentioned in data bricks pro product roadmap.
But in advance of that being available to us, we built a basic change tracking ingestion process to fill that capability gap and allow us to build up the rest of the process process around it in advance. The largest change from the previous approaches which focus on moving IM directly is that rather than replicating that IM layer views from source would then sync raw vault business vault and then translate the TSQL views to data brick SQL which introduces some complexity there.
Um on the ZUA SQL server side there would be some additional load on the warehouse due to storage of the changes on targeted tables for the retention period. We're generally running a 7 days retention period um for change tracking. But the way that uh change tracking works is it also uses a technology we were already using within SQL server which is SQL replication. It just redirects those changes into change tracking tables in addition.
So the the performance load we thought would be quite manageable. And additionally querying the data out because we're only pulling out the changes since the last sync the query load would be would be much less than directly querying information m and would be really predictable and consistent because it would be in line with the amount of rows that they changed each day. So the overall architecture involved three main components which we'll go into further detail in the following section.
The initial driver of this is really the view extraction conversion and validation process uh in which we use JDBC to retrieve the view definitions from SQL server. So we're always in sync with what they're defining within SQL Server. Then we use a process to extract the table dependencies from the view and then we convert the TSQL to data bricks SQL and reregister the view within data bricks. The extracted table dependencies are then passed to the data replication process which is ASUS SQL change tracking and lakeflow connect and we run a separate process for data validation utilizing lakehouse federation lakehouse monitoring and some reports and dashboards we've built up to validate that our data is in sync with our existing data warehouse.
Internally we refer to this combined process as brickout. So stepping into that view extraction and conversion process which is that repeatable approach that isn't a once- off migration. We'll have to do this over and over again as the views uh change over time. Um this was to convert our TSQL to data SQL and prepare the required tables list for flow connect. Uh the key way that this run is essentially we've got a YAML file which is a list of views and that drives the whole process.
Um so that makes calls back to SQL to extract the view definitions and removes any characters that are invalable invalid for data brick SQL. It then extracts the required tables using reax from the view and we only enable change tracking and setup sync for the necessary tables to minimize the compute storage load on Azour SQL. This also has the side effect of reducing the number of leg flow connect gateways and injesting site pipelines we require because the current recommendation is around one gateway and ingestion pipeline pair for every 250 tables that you want to ingest for that list of tables.
We then check whether change tracking is already enabled by querying back to the database. Um and if it's not we then check for the prerequisites for change tracking. For us, this is mainly whether a primary key is set. Um, as in addition to the view definitions which are output from this process, we also output a list of tables that are not enabled and the pre-workers that are not met for investigation and remediation with the data warehouse team. the tables that we need to have enabled which we then pass to our DBAs in an automated process which then they pick up set up change tracking on the tables and apply the permissions we need to be able to read it and the tables to be synced which go directly to LakeFlow connect on this slide I've included a view translation example between Azour TSQL and data brick SQL so our original view and the translated version for data vault we found with we're working with a number of views now that it's reasonably straightforward to translate because of the structure nature of data vault and the consistent um table table structures.
So we still have to do some things though um and we can get away with using LLMs in some cases to speed this up. But we essentially parameterize the create and schema name so that the translated view can be used where needed and applied across different catalogs and schemas. We do a function rewrite of any TSQL specific functions to databick SQL. In the example slide, uh we've converted from cross apply to join lateral.
Um and then we also to give us some extra capabilities out of the the replica, we put our point in time controls. So we can easily add start of day, start of week, start of month, yesterday point in time views using delta travel. So this is essentially just appending a time step as at to the views as appropriate. And within Unity catalog, we structure our synced version. Um, so we've got brick out, which is our control files and our validation tables and our staging volume.
And then we structure the rest of the schemas similar to what you'd normally do in SQL Server with ROV vault business vault information m. And in addition, we also create schemas for point in time views registration. All right. So now we've got the tables, we've got the view definitions and we passed them across to Lakeflow connect for to complete the data sync. So as LakeFlow connect wasn't available when we started this project, we built an initial pock using a custom data bricks work uh process.
Even in a minimum B products state, it introduced significant complexity to break out. It was designed initially as a batch process and would have been difficult to change that to be a high frequency sync and overall we felt it would require significant additional effort to meet our medium to long-term requirements. I'll go into the extra steps that the pock implementation uh included in a very brief manner because this is now being retired out of our solution.
But essentially for each of the tables it would look into sync would get the current version um from Azour SQL. It would then detect the primary primary keys from the tables as would need those for merge statements later on in Unity catalog. It would then retrieve the minimum valid version effectively the minimum version of the database copy needed to be up to sync accurately. And then it's just a simple if statement of whether the minimum valid version is less than the latest sync version or not.
If it was true would initiate an initial snapshot of the table, writing that snapshot with a create or replace table into Unity catalog. And if it was false, we were certain that we could get the changes needed to bring the database version up to the latest update. So we write those changes into raw vault and business vault using the detected primary key as they merge into. And then lastly to finish off the process, we update the latest sync version to the current version of the timer sync and we're ready to run the process again.
As a contrast, setting up the data sync via Lexflow connect using the UI, but this is can also be achieved through the through an API process either through the SDK, asset bundles or the CLI. Um, is a once- off config of an ingestion gateway. Then you can add tables to the sync as required. You also need to set up a destination catalog and schema. And then the initial snapshots and streaming updates are handled in the gestion pipeline for you automatically.
In terms of how we run it in our environment, it's scheduling is handled the same as as any other database workflow on pipeline. And for maintenance, um, some schema evolution is supported, but for other changes when there is a sync issue, it just requires a full refresh of that table from the pipeline. So, it's really easy to recover from an error. I'll hand over to now to give us a more general background on the custom versus Lakeflow Connect change data capture and change tracking syncing in data bricks.
Yep. So before lakeflow connect as team experienced um engineering team normally needs to combine multiple tools to achieve this to bring the data from SQL server to lakehouse. So as an example a general pattern would look something like this. So I would have a extraction tool that is running on the virtual machine somewhere and then uh that means I need to think about what infrastructure I need to allocate for that right how do I scale for it and in terms of the software how can I make it up to date and following by that I might create a message bus that needs to subscribes to all the events from the source tables and later on I also need to create custom codes in my notebooks right to use structure streaming to load the data into the data tables.
While this solution works, it will probably going to take me a long time to set it up and also I need to worry about um the ongoing maintenance about it. And if anything goes wrong with the initial um source APIs, right, this whole thing could entirely break. So now with LakeFlow connect um I can forget forget about all that complexity in the previous slide. All I need to worry about is what data I want to bring to the lakehouse.
So how do we do that from a technical level? Um how do we bring the data from um outsource um external databases to lakehouse is going through the following steps. So starting from the step one, the injection gateway using a classic uh DT job is going to collect the utens uh it's going to collect the um credentials from UC and then using that credential to connect your um database and collect the data and save them and stage them for the latest version into your Unity catalog volume.
And from staging um you're using the civilist DT pipeline to actually digest the data putting the data into a streaming data table. So in this process there are three critical entities here. So we have the injection gateway. We also have the unity catalog in the middle this the volume and also the managed injection. The first one is the injection gateway. The gateway is a separate like pipeline that captures the metadata, the snapshots and the change logs from your databases, right?
And then that stage the data to the UC volumes. The UC volumes is secured using the same mechanism that you would use for other UC volumes. Uh and from the staging, the injection pipeline then collects the data, grab the data and merge it into the streaming tables. During this process, uh, real-time data replication finds the latest changes using the SQL server change tracking CT if the primary key is presented in the table.
If the primary key is not presented table, then we're using change data capture CDC. We are also locking down the permission by default. That means only the people have access to this pipeline would be able to run this pipeline. Thanks. So the last part of the process that we really need to cover is data validation. So we achieved that through using lake flow monitoring lakehouse monitoring and lakehouse federations to snapshot and validate our unity catalog gold layer and compare it to the Azilla SQL server information mark. um the lakeflow connect ingestion sync of ro vault and business vault obviously that's built on top of some really wellestablished um technologies in terms of Azour SQL change tracking it's been available from the SQL server for a number of years now so we're pretty ke confident we were going to get good data out of that and we'll be up to date however as we were converting information m views from TSQL to data brick SQL that process and its manual and automatic components could introduce errors and data of mismatches between our environments.
So this process is really to provide a check that we're in sync. So we use lakehouse federation as another connectivity method back to the ZQ SQL warehouse to create a baseline snapshot for comparison from information m in a SQL server. We then snapshot from the Unity catalog IM views into a validation table at the same point in time. This is necessary because uh lakehouse monitoring can't be run directly on views. It needs to be a table.
Um we'll then use lakehouse lakehouse monitoring to compare and create metrics for the baseline versus validation comparison. And as part of that process, it generates autogenerates a lake view dashboard for exploration of those merit metrics. However, for our use case, we also built an additional custom validation report to check specific requirements and thresholds for our data quality that we use for our decision- making about view um promotion between environments.
On this slide, I've included a example of a monitor dashboard. Um this is the dashboard that's automatically created when a monitor is run or updated if it's already existing. uh and this particular one is an example of a monitor dashboard for a snapshot table. The left side of the dashboard provides a list of metrics and statistics used in the tables and these are generally uh null value checks um distinct values those sorts of like overall statistical tests which you can then check to see your data as a overall is the same. uh the data validation report we built in addition to that it still uses the metrics from lakehouse monitoring but we do some additional processing we have flexibility to add additional things on top if we need to this report which I've included an example of on the slide um is an offline static HTML output that was key for us because it allows us to have a store validation of particular mile milestones within our development or at particular points in time that we can take offline and store um for long-term validation.
Um and it also lets us set up our consistent threshold and rules for reporting. We built this on top of a open source package called data validator that we used before for a couple of use cases. So it was pretty straightforward for us to adapt it. However, this this and the previous bit of work has only really scratched the surf of like house monitoring. So I'll hand over to now to speak more about the capabilities there and what will be likely to adopt at lifeblood in the near future.
Thanks. So lake house monitoring monitors all the tables in your lakehouse for different types of the tables. Uh we will have the statistics being calculated for the quality of the data in different um elements of the table. For example, if you're having a snapshot tables, the data quality matrix is calculated across all table with all the columns. If you're having time series tables, the data quality metrics is calculated across the timebased windows as well.
And you can also moving forward using machine learning models and having monitoring monitor your inference tables for any of the data drift or data performance. So as long as you have the inference table, have the input data and predictions there, we'll be able to help you monitor that. On top of that, there are a few functionalities or features I'd like to mention um that is out of box automatically done for you on data bricks uh if you choose to enable it.
So anomaly detection, you can easily monitor the data qualities of all the tables in a schema. um it automatically access the data qualities specifically evaluating the freshness and completeness of the table. So freshness means that they're tracking how recently the data has been updated and flagging stale table if the updates are not like up to date it's like delayed um and completeness are talking about uh we're checking on the table to if to see that if the ETL pipeline has loaded all the rows into the table and these insights are populated into the house indicator so you can um have the glance of the health of your catalog uh with one um dashboard.
So house indicator is enabled at the schema level and it will show up for each single table that is under that schema. As I mentioned, this is the dashboard that is automatically generated when you run the anomaly detection for the first time. And the dashboard uh will populate the insights for your scan schema and data owners can then use this dashboard to see okay what do I need to quickly react on? Can I set alerts when I see any of the tables being stale?
Um can I re like tell somebody to uh keep up the ETL pipeline to make it up to date? And it's also critical for data teams to understand what kind of sensitive data has been captured in the tables. Right? So data class classification automatically classifiies and tax the table um in your catalog and you can get alerts when the new PII information is caught in the column. And on top of all that, this is also automatically generated.
Um, so you can have this dashboard created out of box within databicks platform to help you present the CLA classification results. It shows you the number of the tables uh that has been classified uh the distribution of different classes of this catalog as well as over time as a time series table uh time series chart what the classification is looking like. Thank you. So, we'll just go wrap up into the outcomes and next steps.
Um, obviously we have a lot of work still to do in our monitoring approach, but uh we've got the minimum product in there at the moment. As a result of this project, we now have really a foundational capability to ingest and combine our datab data assets and other lakehouse data sets. For us, this will support quicker and less complex access to data in data bricks for our team and other analysts across lifeblood. And it will also reduce context switching between TSQL and pi spark and spark SQL for teams that have to use both.
Um, a key capability that we're able to deliver through this work is point in time views and materialized views to support fixed point in reporting requirements which was not a capability built into the data vault warehouse at the time. uh and quite importantly this provides a clear lineage and a single point of integration between Azour SQL data vault and data bricks as if we went down some of the other approaches would have multiple points of entry and it would be really hard to control and manage.
So with all that in mind, what are some overall insights after completing the stage of our data bricks journey? I mean we always take it as a have a plan but be flexible. Um it's a fastmoving platform. So when we started our design from making data vault available in our lakehouse to now we've adjusted our approach as new features were added and enhancements were made to the platform. It was really key for us to assess our requirements and our constraints and particularly the limitations of what the team could achieve before considering building custom.
And what worked really well for us was to build with what we had available now. And the team we had available now proved that it worked and would bring value but not overengineering the solution as we had reason to believe we may be able to replace some of that complexity in the near future. Thank you for your time and attention.

Learn more about the Databricks Data and AI platform.