Databricks Lakeflow: Cut Data Onboarding Time by 85 Percent

Summary

Corteva AgriScience, which supports the food supply for roughly 10% of the world's population across 35 global seed sites, reduced data source onboarding time from 30–45 days to just 4–7 days by implementing a metadata-driven architecture with Databricks, Lakeflow, and Lakeflow Connect.
Five reusable frameworks—metadata-driven batch ingestion, zero-copy edge-to-lakehouse streaming, configuration-driven data quality rules, incremental DDL for schema evolution, and Genie space deployment—now power over 5,000 tables across 15-plus source systems at Corteva.
A new framework for deploying Genie spaces and dashboards as code removes the IT gatekeeping bottleneck that typically affects organizations managing hundreds of business analytics assets without a systematic deployment path.

Databricks Lakeflow: Cut Data Onboarding Time by 85 Percent

Data source onboarding traditionally consumes weeks of engineering effort and manual coordination, becoming a bottleneck for enterprise data platforms. Corteva AgriScience reduced onboarding time from 30 to 45 days to just 4 to 7 days by implementing a metadata-driven architecture using Databricks, Lakeflow, and Lakeflow Connect. This case study presents five reusable frameworks that replaced ad hoc pipelines with standardized, configuration-driven processes.
Learn how to design metadata-driven batch ingestion frameworks that scale across disparate source systems, implement zero-copy edge-to-lakehouse streaming for real-time plant telemetry, apply configuration-driven data quality rules across domains, manage safe schema evolution through incremental DDL deployment, and empower business users to deploy Genie spaces and dashboards without IT gatekeeping. These frameworks now power 5,000 plus tables across 15 source systems at Corteva, reducing technical overhead and accelerating analytics delivery.
🤝

Chapters

FAQs

How did Corteva AgriScience reduce data onboarding time by 85%?

Corteva replaced ad hoc, unplanned pipelines with five standardized, metadata-driven frameworks built on Databricks Lakeflow, covering batch ingestion, streaming, data quality, schema evolution, and business asset deployment. This transformation reduced onboarding time from 30–45 days down to 4–7 days across 86 domains, 15-plus source systems, and 5,000-plus tables at 35 global seed sites.

What is zero-copy edge-to-lakehouse streaming and how does Corteva use it?

Zero-copy edge-to-lakehouse streaming is an ingestion pattern that delivers real-time telemetry data from manufacturing plants directly into the Databricks lakehouse without creating intermediate data copies. Corteva uses this approach to ingest plant telemetry from their 35 global seed sites in real time, enabling operational insights without the latency and cost of traditional batch pipelines.

How do configuration-driven data quality rules work in the Corteva framework?

In Corteva's framework, data quality rules are stored as configuration metadata rather than being hard-coded into individual pipelines, allowing teams to apply, update, and reuse quality checks across domains without modifying pipeline code. This blueprint approach standardizes data cleansing and validation across all 86 onboarded domains.

What is the Genie space deployment framework Corteva built and why is it needed?

Corteva created a framework to deploy Genie spaces and dashboards as code, addressing the common problem where organizations end up with hundreds or thousands of business analytics assets that have no systematic way to be versioned, tested, or deployed. This framework empowers business users to deploy their own analytics assets without requiring IT involvement.

Full transcript

[00:09] Hello everyone, thanks for being here. This is Mahul Ba. I'm a data and AI platform engineer at Cortiva Agri Science. And yeah, I'm a data and AI platform engineer at Cortiva Agri Science. I have 22 years of software development and architecture experience. Um at Cortiva I
[00:26] work on the enterprise datab bricks lakehouse uh driving the data and AI innovations. Thank you. Hi everyone, my name is Harshett. I am a senior forward deployed engineer here at datab bricks. This simply means that I
[00:43] get to work with all the data bricks customers hands-on to design, develop and deploy end-to-end data and AI solutions that address the challenges that our customers have. Yep.
[00:59] So the key message I want everyone to take here is how we cut the onboarding weight times by 85% using five primary frameworks that we have designed at Cortiva and uh to cut the onboarding time by 85%. We onboarded 86 domains
[01:14] across 15 plus source systems with 5,000 plus tables across 35 global seed sites. So we're going to talk about those five frameworks in detail. Uh in this session the key takeaway will be how we designed a reusable lake flow first ingestion
[01:32] pattern. How to apply metadatriven data quality blue blueprint manage safe schema evolution across environments. And how we created a new framework for deploying business assets like genie spaces and dashboards which is typically
[01:48] a problem where you have hundreds and thousands of dashboards and genaces and there's no good way to deploy them. We've created a framework around it. Uh we also have two de demos where we show you the lakeflow and streaming injection pipelines.
[02:06] So before I begin I wanted to ask this audience do we know who Cortiva Agri science is? Raise your hands. You know yeah cool. So Cortiva develops and sells high performance seeds using advanced genetics and biotechnology to improve yield resilience and farm productivity.
[02:23] Cortiva supports 10% of the world's population. This is pretty big. There are 800 million people on this planet that rely on Cortiva to put their food on their table every single day. Realize this is the entire population of US, Europe and Japan combined. So we do very
[02:38] important work in advancing the seed genetic technology. Now before we dive into how we did it, we want to go back to the past and we wanted to explain challenges that we encountered in the past. So we had a
[02:56] bunch of ad hoc and unplanned pipelines that were sprawling up right in an unorganic fashion and there was no shared architecture. There were no standards or there was no reuse that we had at Cortiva. We had multiple different source systems. We were
[03:12] spending a lot of time writing data engineering pipelines and that used to take at least 75% of the time writing the data engineering pipelines, plumbing, you know, fixing things here and there. Things were breaking all over the place. A developer was writing pipelines. Every everyone was writing pipelines in different ways. There was
[03:28] no standards. There were no uh, you know, there was no modeling instructions. There were no refinement. So we had pipelines named after the developers. So there was like a, you know, Scott's own pipeline created. We don't know why it's created. So we spent a lot of time writing data engineering pipelines. Um and actually you know uh
[03:46] there was also another problem that we had file based ingestion which means we had these files that used to be dropped in a specific location that we had to pick up and mostly these were SAP files. You know how SAP is right? It is it is a pretty monolithic system and the integrations used to break all the time.
[04:03] So we had that challenge as well. Pipelines were handbuilt. There were all kinds of connection logic, connection strings, sedd, key value pairs, you know, there no safety security at all. Knowledge lived in people. There was it was very difficult to get the tacet knowledge out and you know document it.
[04:20] So we also we always had to rely on tribal knowledge. This made the overall onboarding very difficult to begin with. So why is this still hard? There are still you know a lot of current challenges like most of our source systems don't have CDC enabled.
[04:35] uh and that's a pro problem because it forces us to do full loads each time versus incremental loads. There is one framework but there are many many different kinds of platforms. We have Azure SQL, we have Oracle, we have all kinds of custom databases things like that. Sedd should be defined in the
[04:52] governed layer whether it's a full or an incremental it should be defined in the metadata layer not buried inside the pipelines. uh there are a lot of issues we faced you know we still face where we have downstream impacts because you know someone changed the schema and it broke the pipeline. Now how did we redesign
[05:08] this whole thing? We create we we designed it to create a reusable framework so that we don't have to repeat ourself and eliminate duplicate data engineering pipelines. So um we wanted to move away from data swamps to a unified lakehouse architecture. We
[05:25] didn't want to spend 75% of the time re-engineering and writing the same kind of pipelines over and over again. So it we wanted more reuse and you know flexibility and every you know everything should be following a specific standard. That was our goal. So we wanted also had a we also had a
[05:42] governance governance and modeling first approach to developing our whole framework and architecture and we also wanted to streamline deployment pipelines uh and implement a metadatadriven architecture. Moving on to the next one. It's
[05:58] important to look at the past before we redesign the future. So this is how our past look like. So we used to take half a month just for source identification, schema design and modeling, right? One month was spent on writing data engineering pipelines, permissions,
[06:15] CI/CD deployments, testing, validation. This was a very inefficient process. Please raise your hands if you have been here or have seen this. Okay, cool. I think majority of us. So yeah, I mean we are all we have all been
[06:30] here and we have all seen this thing and uh let's see how we you know what the present looks like. So now in the present we spend less than a week to onboard a new data source. So what has made the difference?
[06:45] So again going away 75% of time was spent on writing data engineering pipelines and just re-engineering the same wheel again and again. Let's see what we changed. We created these five frameworks.
[07:01] Metadatri driven batch injection to begin with. This basically streamlines our data engineering pipelines. Instead of having hundreds of pipelines for every source system, we have one common framework to ingest data from multiple different kinds of disparate data
[07:16] sources. Edge to lakehouse streaming ingestion. We do real-time streaming for our IoT and sensor data. cleansing and modeling rules. Everything that goes from bronze to silver goes through a cleansing layer that's also embedded in
[07:34] our injection pipeline. A DDL deployment framework. A lot of times we've seen that the assets that we deploy, we keep deploying them to tabs again and again and again. We don't have to deploy a table that's already deployed. We don't have to alter a table if you want to make property changes and this already
[07:50] done in the past. So we we want to do incremental DDL deployments. So that that that's that framework. Harsh will talk about that but it's a pretty pretty cool framework that we have created. Genie spaces and dashboards deployment as I said the challenge now is an explosion of business assets which is
[08:06] genie spaces and dashboards. Everyone has that problem. Who here has hundreds and hundreds of genaces that the you know business is creating in production. Please raise your hands if you have seen that there's no governance. There is no process. We don't cannot tag the owners. Some of these half of these genie spaces
[08:22] no one uses these spaces and and no one is aware that such a solution exists. So they create their own again. So it so we solve that problem by by you know uh creating a framework around deploying deploying these business assets. So this is a high level overview of our
[08:39] unified lakehouse operating model. So we have you see here on the left hand side we have data from different sources web hooks we have you know FedEx web hook injection we have uh structured data coming from operational data sources
[08:55] like Oracle SQL and other custom databases we have semi-structured data documents coming from SharePoint and we also have plant and edge telemetry data which is our sensor based data at all plant sites across the world now the top three are managed by that metadata
[09:13] driven batch injection because they are at a scheduled frequency. We don't we don't need them most of the time real time for data analytics and AI and BI purposes. So that metadata you know batch injection it's driven by a configuration table or a metadata table. So you don't have to write anything new.
[09:29] You just have to add an entry to a config table and it'll pick that source up and ingest the data for you. This is the way we automate and we avoid that 75% you know timing you know time in rewriting those pipelines. The second one is the streaming based zerobus injection which is a more
[09:46] realtime solution that we have that we have built here at Cortiva. This helps us bring all the plant and sensor data in real time within our lakehouse. In the middle you see the medallion architecture. We have bronze, silver, gold. We call it raw, refined
[10:02] and presentation. That's just our own internal terminology. Number three is cleansing and modeling rules. You see right there in the middle there is a you know whenever data goes from bronze to silver we apply those cleansing and modeling rules. Again they are also defined in a common table where
[10:18] you don't have to rewrite them each time. It's all coded for it's all configured. Then we have an incremental DDL deployment framework which is your data definition language. So if you have create tables, alter tables, no creating, you know, adding table, adding
[10:33] columns, changing schema and applying security, everything is handled by that DDL deployment framework incrementally. So we don't reapply the same changes over and over again in our CI/CD process. And finally, the consumption layer, which is the AIBI dashboards. We
[10:50] have a framework as I mentioned on, you know, deploying Genie spaces and dashboards. And this framework is controlled by the business. At the end we have governance all around and uh we have unity catalog governance.
[11:05] So this is the overall highle architecture. There are a lot of parts and moving parts and pieces. We this is just a highle overview of our architecture here. Now I'll take you through the metadata driven batch injection first. So with
[11:20] this we have solved the boring part. What is the boring part? data engineering, right? We we don't want to spend 75% of time writing the pipelines over and over again. So that's and then we don't want to do all the plumbing work again and again applying, you know, right, you know, doing the CI/CD
[11:36] process, applying security permissions. It's all done in that framework. We don't want to write new data engineering pipelines to bring new source systems. So any any entry in the source systems goes to that config config table on the on the metadata layer. If you see on the diagram on the
[11:52] right hand side, this is this config rules. We have rules defined for every source system. How the data needs to be ingested. Is it sedd type one, sedd type two, full incremental? We have dynamic wear clauses that we define there. We have active flags. We we can enable and disable any source system at any given point in time without changing a single
[12:09] line of code. So again the data sources are on the top which is all kinds of data. So se you know structured, semistructured, streaming data, I mean service bus, web hooks etc. And then there's a common notebook which drives the whole the whole process. The common
[12:24] notebook is our compiler. It looks at the metadata table, reads the information, understands it and then processes it. Uh we have multiple lakeflow jobs that refer to the same common notebook. And then on on in the bottom we have the unity catalog. Everything is stored as delta tables for
[12:40] governance. So what we have achieved here is we have multiple source connectors metadata driven onboarding. So now we take only 4 to 7 days to onboard a new source system versus 30 to 45 days in the past. I will hand it over to hers to cover the
[12:56] next streaming framework. Y all right. Welcome everyone and thank you Mahul for covering everything before this.
[13:11] Now before I start talking about streaming and everything after this I want to mention a key important point uh an important point. Every architecture, every solution that you see on the screen, it has been kind of designed and
[13:26] tested against several key principles. And the principles are number one, customizability, number two, reusability. Number three, simplicity. Number four, scalability. And number five, the most important is speed to value. So when you
[13:43] have all of these things, all of these principles coming together into the design process of the solution, that's where you, you know, end up with a lot of wonderful frameworks that work together. Perfect. Now let's talk about edge to lakehouse stream injection.
[14:00] Mihul mentioned earlier the batch injection. So that's working out pretty well and we were like, okay, now we need to go a step ahead and create something for streaming. To do that we have we identified that well Corteva has a lot
[14:15] of manufacturing plant sites all across the world and all of these plant sites produce millions of data points every single day. Right? So if we can get all of the data on data bricks in a very streamlined fashion. Basically we can do
[14:30] a lot of use cases. We can put a lot of use cases business valuedriven use cases. things like predictive maintenance, predictive analytics, root cause analysis, uh performance optimization, so on and so forth. So this was the aim and to do this we
[14:47] identified that zero bus from uh which is part of lakeflow connect on data bricks is a very optimal solution and the primary reason for that is we agreed that the final target for all the data across all the sites would be data
[15:03] bricks. every consumer that needs to consume the data the insights then will pull the data from data bricks and so this is where the the zerobus based solution comes in and really out outshines everything else the other
[15:19] thing I want to mention and and we will have a demo on this towards the end so you will see everything just falling in uh falling in place together but I want to mention a couple of things here number one you'll see the the central block which says process in uh
[15:36] information network that's where the zerobus based solution stays right now you'll see in that block and you'll say hey but there are these so many components that I have to take care of not really because I have broken the architecture into all these individual
[15:51] components so you understand it better when it comes to actually deploying it on your plant sites or manufacturing sites it will all be packaged together as a container service or a docker that you can just deploy and it will run just fine. That's one. The other big thing is
[16:10] we uh we planned it we we customized this solution to have it work the way the manufacturing sites are currently working. So basically what this means is right now we have developed it for OPCUS
[16:27] server but if you had something like uh ignition you could very well just pull out everything that doesn't apply and you can still deploy and it would just work fine. So basically the whole the big idea was to make customized solutions uh a a lightweight solution
[16:44] easy to maintain solution that meets you where your data is produced to meet you where your factories and the manufacturing sites are and not have it the other way around. Right? So we are not trying to have that requirement that you have to
[17:00] completely change the way your factories operate just to send some data on data bricks. And so we'll go through all of this in uh very briefly uh towards the end. Now the last uh the the third one. So after we have done the
[17:16] batch injection and the stream injection, the next one is well how do we clean the data? How do we model the data? And so basically the the idea is simple. We want to create one single rules engine which is used across all the domains, right? And these rules are
[17:33] kind of driven by the business owners, right? And also the fact that if tomorrow your business requirements change, some logic changes and accordingly you want to change some sort of like transformation logic, you can come in and modify those rules in the
[17:48] configuration tables and none of the pipeline is impacted in the sense that you don't have to go back and rewrite the pipelines again and again or make any changes there. The last point is important because 5,000 plus tables across 10 15 domains are all working and
[18:04] using the same framework. The idea here is simple. You have the the raw table or the bronze table sitting on the left. You have the refined or the silver table sitting on the uh on on the right. We have our wonderful data architect Paul who maintains all of the configuration
[18:21] rules, business rules in his Excel file. We create and we ingest that Excel file as a data model table on data bricks. Perfect. The data model table is ready. Now we have all these batch jobs or streaming jobs which kind of run. The moment they trigger there is a common
[18:38] notebook which reads the com uh which reads the data model table. It also reads the the table from the raw layer and it identifies what kind of rules need to be applied depending on you know the source and the business logic. It
[18:53] performs those transformations. Maybe it's data type, you know, correction or column renaming, whatever. It performs all those transformations, pushes the logs in a separate table, and then it uh pushes all of that on the
[19:08] refine layer. Again, the idea is simple, but it allows us to really not rebuild the same thing again and again and use this same logic at across all the different domains that we have at Corteva. The last part is the or not the last
[19:25] part the fourth framework that we have is the incremental DDL deployment. Now we have covered a bunch of things right we have ingestion. So that covered batch injection that covered streaming injection. Then we looked at a little bit about how we do the common data modeling and how do how we have that
[19:42] layer sorted out. Perfect. We also want to now cover little bit of how we manage our CI/CD. So right when we started this journey one year ago, one of the key decisions we made is we will make sure that all the CI/CD best practices are also you
[19:59] know uh kind of adopted and to do that we en we made sure that there is no fat fingering or ad hoc random queries running in different environments. Right? So every query, every DDL, every DML, the the data manipulation language
[20:16] or the the the data definition language scripts, they all need to be part of your CI/CD process. So if you apply a certain let's say column name change or table property change, it not just applies to dev, but it also, you know, trickles down to test and prod as well.
[20:33] And so this means two things. Number one, if something fails, you know, you can trace it back and understand where the failure was. And the the bigger one I feel is we had the ability to kind of then make
[20:51] certain changes and make sure that the downstream pipeline or not downstream pipeline but the data engineering pipelines do not break in the higher environments because the person who you know executed the SQL query in dev forgot to do it in production. So that's
[21:07] a whole bunch of things that we took care of. Now again the architecture is pretty simple earlier but let me back up a little bit. So all all good we have the SQL scripts part of CI/CD. Everything is working fine. But one of
[21:23] the challenges that we immediately hit is because all the SQL files are now staying in Git repository and our deployment pipelines are running. all these SQL files get executed every single time and that's simply because there is no log awareness within the
[21:39] deployment pipeline itself. So what we essentially did is build this architecture again it's a very you know verbose architecture in this slide but the basic idea is you have all your SQL files staying in your git repository the
[21:54] CI/CD pipeline runs now before any SQL script can execute on data bricks it checks first what was the history and what is the net new DDLs that need to be executed once it has figured out it
[22:10] deploys it to whatever environment you are running it in. It also saves the logs of every single deployment in data bricks itself. So if something fails, you can go back and look at the exact output and then go fix it and then run
[22:26] it again. And so this is basically the framework that has been adopted across multiple teams within Corteva and it has supported you know thousand plus runs across different environments and this framework is something that
[22:42] came out of need because we when we deployed the CI/CD pipelines they would initially run for like 60 70 80 minutes and eventually time out. Now all of these files, all of these DDLs, they deploy within 8 or 10 minutes. And so
[23:00] it's kind of agnostic now to some extent to the number of files you add in your uh repository because it's no longer executing everything from the scratch. With that, I would pass it on to Mihol who will cover the next part.
[23:15] Thank you Harid. So as I mentioned we at Cortiva we are starting to see a genie space explosion um and a dashboard deployment dashboard explosion right people are creating dashboards and genie spaces every single day and there is no way to govern them manage them and
[23:32] that's why we needed a framework around it so Ali mentioned in the keynote speech that we have choice control and cost this is exact same terminology that applies to genie spaces and dashboards we have a choice now for business owners to pick and choose what genie spaces and
[23:48] dashboards they want deployed in their environment in the production environment. We have control because business owners control the ownership of these genie spaces, dashboards and permissions and we have cost factor because it eliminates all the cost and complexity of creating pull request for
[24:05] each and every asset genius or dashboards and you know downloading the serialized space JSON all through manual processes. It eliminates all that stuff and that's why it saves a lot of time. We don't need data engineers or you know your your technical people to deploy
[24:21] these genie spaces and dashboards. So that's why we created this framework. If you see on the left you have this business analyst or the author who creates or develops the genie spaces and dashboards adds a configuration entry in those tables and the framework will
[24:37] automatically and then he authors it in dev only. We have access to product cataloges in dev. So he can point and create assets in dev and then there is one click one push button where basically he can decide when he wants to deploy his asset when he's ready creating the asset he can click a button
[24:53] and deploy it everything to a data bricks app. So that goes through the whole dev test and prod it he doesn't need to go and you know go to the IT teams and wait for his you know whole you know wait for the backlogs and you know wait for the deployment pipeline to run he can do it himself. So we give the whole control and ownership to the the
[25:10] business owner. Now what this brings to the table is AI is governance first of all because we wanted we wanted to know who are who people know which uh which kind of business users are creating these genie spaces and dashboards in your organization. We wanted to identify
[25:26] the owners and we also wanted governance around it. This this brings the governance on the on the on the table. So so we have choice control and cost that we we derive out of this whole framework that I've created. Now I mean there's a lot of details and complexity in there. I don't want to go through all
[25:42] of that. But again if you see the whole uh you know once you add that entry to an app it will basically go and go through dev test and prod workspaces. You just have to do the business user just has to add that entry once and then it goes and modifies the space serialized space JSON and all the JSON
[25:57] files associated to the dashboard and automatically goes to and publishes that entry in the test and the prod catalogs and then deploys it through a job a lakeflow job. So with that we have covered genie spaces as well. So our five foundation pillars the metadatriven
[26:14] architecture that cuts the engineering effort data engineering effort by 75%. the cleansing rules and modeling which gives us the ability to model first and then you know and then do cleansing rules and then the third framework was the streaming batch injection zero bus injection framework. The fourth
[26:30] framework was the DDL deployment framework where we were able to cut the deployment time by you know by 8x. And the fifth framework is genius spaces and dashboard deployment which empowers the business users to go and basically deploy their assets without relying on IT teams
[26:47] with this as as we promised we have two demos. So we'll start those demos real quick. So this is a demo on um you know writing a lakeflow connect injection pipeline you know through program you know everything programmatically instead of by hand
[27:13] this is Mayu here from today I'm going to demo a framework that we have created here at cortiva we call it the lakeflow connectet managed metadata driven injection framework this framework is completely metadata driven. Instead of hard- coding pipelines, all ingestion behavior is controlled through
[27:28] two delta tables. One that defines what to ingest and the second one that defines how changes are being tracked. A single reusable notebook reads this metadata dynamically, constructs the pipeline configuration and executes it through APIs. The same logic seamlessly supports multiple source systems
[27:44] including SQL and Oracle and handles both full loads and incremental change tracking. Key advantage is scalability through simplicity. Onboarding a new table requires only an insert into the metadata table. No code changes are needed. No PRs and no redeployments.
[28:00] This approach enables rapid, consistent and highly scalable data injection across the platform. Let's take a deep dive. Traditionally, we create the injection pipelines by hand. We select the type of connection
[28:16] or system and then we select whether it's a CDC or query based capture. We select the catalogs and schemas and then the next screen we select the tables. Imagine there could be hundreds of tables, thousands of tables in one source system. Now this process is very manual and cumbersome. Imagine deploying
[28:32] it at scale across several environments. Let's say we have four workspaces dev test stage and prod. Doing this by hand in all four workspaces is not feasible. Uh so then that's why we came up with a metadata driven framework which can be
[28:49] managed by a single table where you can capture all your sources with their application names and source table names source catalog names schema destination catalog schemas connections sedd type one means full sedd type two means incremental here I have examples of an
[29:07] application that's hosted on Azure SQL database and the other application is a Oracle database. So I'm going to show you a demo for these two. The active flags where you can control what you want to ingest and what you don't by setting the flags. Um so this is a very simple lakeflow metadata config table.
[29:25] There's another table that we use for sedd2 where you pass in the primary key names for uh the sedd2 type change you know sd type two table names uh primarily for incremental loads. This is needed because the lake flow connect
[29:43] uses this uh for incremental injection to identify the changes. Now let's go to the job that creates and orchestrates all this. We have this spin job which has this notebook which is
[29:58] again reading off all the entries from the metadata config tables that I showed you. So and then the job has bunch of parameters that we pass to the common notebook. Now
[30:13] the common notebook is a single notebook that can ingest data from multiple different data sources provided the right parameters are passed through the jobs. So the architecture is that we'll have a job for source system which will
[30:30] call call the same common notebook with different parameters and the parameters will drive the injection pipeline. So we read the configuration. We have a bunch of helper API utilities to create or update the pipelines, start the pipelines. Um reading from the config
[30:46] tables for SD1 and SED2 building the pipeline config and then finally building the pipeline spec that's get that gets passed to the API. So if you look at the pipeline spec here, this is a single an example of pipeline spec for spin. This is nothing but a payload that
[31:01] we pass uh to when we create the the the injection pipeline. Now you will see that the the the SED type two uh spec will have primary key additional as an additional parameter that we need to pass. Uh for SD type one since it's a
[31:17] full load we don't need to pass the u primary key values. So finally it basically once it builds a pipeline spec it creates updates or runs the pipeline. So essentially if the pipeline doesn't exist it'll create a brand new one. If it already exists,
[31:33] it'll update it and then finally it will run the pipeline. So this notebook does all the magic based on the params that has been it has passed. Now we can also definitely schedule your triggers based on your frequency that you want to update and run the notebook. And
[31:49] the final output of it is this injection pipeline that gets created. U this is the latest run that we showing you. Um you can you can see a a nice DAG of all the tables that got ingested from source. It's easy to track errors here.
[32:05] You can look at specific errors. You can also look at performance. So this is a great uh interface which I call that. Similarly, our Oracle job for limbs essentially ingesting from Oracle database is looks pretty similar
[32:22] because it has all these parameters that we pass to the common notebook and the common notebook is what drives the whole ingestion pipeline. So this so this pipeline gets created for limbs and if you see we ingested 20 tables we can see
[32:37] how much how many records we ingested like 380 million records that we read here from an ingestion perspective and essentially this is what it does because basically it it's you know it's pipeline creation at scale and uh it's inest
[32:52] ingesting data at scale I would say and you could scale it across your enterprise uh it's it's a pretty slick framework very lightweight the two tables drive the ingestion across multiple different kinds of source systems and you can ingest directly into
[33:08] your bronze layer this way. Uh so the benefit that we have achieved out of this is our source system onboarding time has gone down from 30 to 40 from 30 or 45 days back now to 4 to 7 days. So this has helped us tremendously in our
[33:25] lakehouse journey and ingesting data from different sources. Thank you very much for watching the demo and have a nice rest of your day. All right, I'll very quickly play the whole edge to lakehouse streaming
[33:41] injection. So the end toend solution of what it for this demo we are going to look at five different components in this architecture. First is the opcua server which acts as a universal data translator and traffic controller in industrial environments. The second component is the opcua client which
[33:58] pulls the data from the server and then pushes it out to rabbit MQ. The third component here is rabbit rabbit MQ itself which acts as a temporary buffer in case there are network connectivity issues. The fourth component is the forwarding agent which uses the zerobus
[34:16] SDK to pull the data from rabbit MQ and then push it eventually to datab bricks. The fifth component here is JSON file which acts as a failover in case my Rabbit MQ service itself dies. Now let's switch over to the terminal so I can
[34:31] start each of these services. Now you do not have to do this manually. You can package all of these services as nice docker containers that you can then deploy on each of the each of your sites. Now once I have started all these services, what you will see is the data
[34:48] will start to show up in Rabbit MQ. You can see uh the numbers showing up. The publish rate is 9.4 or 10 records per second. If I switch over to my pipeline, the streaming pipeline is red running and it's processing the data actively.
[35:04] The nice thing about zerobus is that it allows you to get the data from your plant sites and ingest all of it as delta tables directly on data bricks. That means two things. You do not have to maintain separate injection pipelines
[35:20] and you do not have to maintain separate Kafka clusters. Now once you have this, let's dig deeper into the pipeline itself. We mentioned right in the beginning that the architectures that we have built are supposed to be metadata driven. What
[35:37] that means is if you wanted to add a new streaming source, you would add a new record in this table. The data model is governed by this particular table which defines what kind of transformation should be applied when I'm moving the data from my bronze layer to the silver
[35:54] layer. Similarly, data quality rules are defined in this particular table. Now, by doing all of this, what I have done is I have very effectively moved the data from my plant sites to the bronze layer to the silver layer. Next, I can
[36:10] create something like a dashboard. I mean again just for the sake of simplicity for this demo, I created this dashboard which shows a realtime view of my operations. So, this is the operations overview. The second part is the one that looks interesting. What I did is I trained a
[36:28] machine learning model and then registered it with Unity catalog and then have that model predict things like you know what is the average seed loss rate for my future batches, right? And so here you can see you can create these
[36:44] sort of artifacts uh for further consumption. Now as you can see this whole thing is pretty smooth but in reality this is not going to be so smooth. So the next question is hey what happens if you know
[36:59] I stop receiving the data from uh from my plant side. In such cases you can set up uh simple alerts on the data brick side. So that will take care of uh outages at least the notification part. And as you can see it can send me the
[37:16] the the notifications here. But well what happens on the plant side itself I do not want to lose my data if there is any internet connectivity issue. Right? And so let's look how how this architecture holds up under those scenarios. So I have this forwarding
[37:32] agent. I am going to kill this right. So let's say it's not able to send any more data and let's go back to rabbit MQ. So what you will see is once the forwarding agent is unable to send any data to datab bricks the rabbit in Q's buffer
[37:48] starts to fill up with these records and as you can see the number is rising. Let's say I'm I'm really unlucky and for some reason my rabbit MQ service also dies. The moment that happens that means I do not have a buffer. I do not have
[38:04] the ability to push the data to data bricks. And so that's where this JSON file comes into picture. So as you can see all the data that I'm unable to send to my buffer is getting pushed in this file as the ultimate failover. Now let's
[38:20] say I have resolved all the issues with rabbit MQ and it comes back up. The moment I do that, what you will see is it will train this backup file and it will bring all the data back to Rabbit and Q. And here you can see the number
[38:35] 482 which represents all the backups that I've had in that file. But that number will continue to grow because I haven't yet started or sorted out my forwarding agent. Right? And so the moment I do that you will see the forwarding agent comes live it will then
[38:53] drain the entire rabbit MQ uh buffer and then all of that data should show up here in the streaming table. So, it takes a couple of seconds, but right now you just saw it was idle. The table was sitting idle and it came back up.
[39:17] Yep. Let's look at the future. So, at Corta, we are continuously trying to push the boundaries and adopting new new new technologies and new things that come up in our way. So, we're going to expand Lake Flow Connect adoption to include multiple data sources, integrate Aback attribute-based access control
[39:32] plus govern tags for fine grain security, and we're also going to automate the data modeling process with Gen AI. We also want to leverage Genieontology and Genie1.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.