Sponsored by: Astronomer | Unlocking the Future of Data Orchestration: Introducing Apache Airflow® 3

Summary

Apache Airflow 3 capabilities: This video introduces Apache Airflow 3, bringing new flexibility, scalability and security to data orchestration with an intuitive upgraded UI on the Databricks Data and AI platform.
Enhanced security and isolation: Learn how Airflow 3 decouples task execution from database connections, enforcing task isolation and minimal-privilege access while meeting compliance standards on the Databricks Data and AI platform.
Remote execution and scheduling: Discover how Airflow 3 enables running tasks anywhere with remote execution and event-driven scheduling for hybrid-cloud and modern data environments on the Databricks Data and AI platform.

FAQs

What are the key benefits of Airflow 3?

Airflow 3 improves ease of use with an intuitive UI, DAG versioning and scheduler-integrated backfills. It also strengthens security through task isolation and enables ultimate flexibility with remote execution for heterogeneous environments on the Databricks Data and AI platform.

How does Airflow 3 improve security?

This video explains how Airflow 3 decouples task execution from database connections and enforces task isolation with minimal-privilege access. This architecture significantly reduces unauthorized data exposure risk while meeting compliance standards on the Databricks Data and AI platform.

What orchestration tools are available for data pipelines?

Apache Airflow 3 is a leading open-source orchestration platform. The Databricks Data and AI platform integrates with Airflow for data orchestration while providing native scheduling through workflows. Airflow excels for complex multi-system workflows where central orchestration is needed.

Which data orchestration tools are recommended?

Apache Airflow 3 leads the market with strong operator ecosystem and community support. Databricks Workflows provide tight integration with the Databricks Data and AI platform for notebook and SQL execution. Choose based on orchestration scope and existing platform investments.

What are the best data orchestration platforms?

Top platforms include Apache Airflow 3 for general orchestration, Databricks Workflows for tight integration with the Databricks Data and AI platform, and cloud-native services like AWS Step Functions and GCP Cloud Composer. The best choice depends on your architecture and whether you prioritize flexibility or native integration.

Full transcript

Good morning everyone. Thanks for coming here today. Um we're going to be starting the session in just a second here. But before we get going, uh just a quick reminder to please rate all the sessions that you attend. Uh there is a survey in uh in the app that you can access. uh gives all the speakers the notes of you know what you've thought of these sessions and how how uh relevant they are for you and where they might improve.
So it's a it's a big help for all of us. Um but just to start things off um we have Vicram Koka here from Astronomer who's going to be taking us through uh the future of data orchestration with Apache Airflow 3. So let's welcome him to the stage. Thank you Rich. Thank you. So before I uh get started and I'd love to actually chat about like you know what is data orchestration, what is Apache Airflow and about Airflow 3.
One quick question just because I don't want to repeat a lot of content if all of you are already aware of this. How many of you already use or in are using Airflow today? significant majority I would actually say in that case I'm actually going to go through the first part of the sections really quickly about data orchestration and about introducing airflow and kind of jump to airflow a lot of like you know the differences with airflow 3.
I assume that the vast majority of the folks here are using airflow 2 some incarnation thereof. Sounds great. Awesome. Cool. So um what just as context um I'm Vicram Koka I'm the chief strategy officer at astronomer. I'm also part of the airflow Apache Airflow comm uh committee. I'm on the PMC been contributing to airflow since roughly about 2019. So happy to answer any questions even about the history and like you know you kind of say what the heck were you guys thinking about various features if those are the questions and why we're actually making the changes today about what we've been working towards Airflow 3.
So again uh very interested in uh in your feedback since most of you are aware of data orchestration. I'm going to actually just kind of run through the initial aspects really really quickly. Some of the common examples we tend to use are like based on use cases. In this case, one of the first use cases is about an an energy company which wants to pull data together from a variety of different uh energy sources and being able to pull those together using a combination of data bricks data intelligence engine and then like dashboards for visualization.
Usual challenges in these situations as all of you are familiar with because you've already been using this is pulling together data from a variety of different sources, different APIs, different formats, data arriving at different times, inconsistency, needing to kind of do transformation before things happen and people actually want this data to be as available as timely as quickly as possible. So the general process of pulling stuff together from a variety of different sources is really where like airflow shines and um that's actually how we actually regard this.
It's a combination of scheduling combination of monitoring combination of actually applying the transformation logic at various stages typically in a lot of cases before you actually load the data. That's the classic ETL paradigm. A lot of cases after you loaded the data then being able to apply it through the ELT paradigm as well. And then respecting all the dependencies and handling and especially handling failures gracefully about you know what do you do if the data is inconsistent?
What do you do if the data is missing? What do you do? Do you use prior incarnations of data and all the programmatic flows is really where Airflow tends to shine. So this is again something you're all really really familiar with because the vast majority have actually been using Airflow. Couple of things I do actually love about this part because I've been working with Airflow for like six years is love kind of the growth overall but especially the variety of use cases which we're seeing companies work with um with Uber for like you know automating automating the supply and demand analysis being able to figure out pricing dynamically.
I remember chatting with Uber like I think the first large company I'd spoken with in September 2019. uh Stripe, somebody else has been working with uh Airflow uh for like a really long time. Uh clearly Apple has been doing a ton of stuff. They've actually spoken to the Airflow Summit as well. So has getting pabytes of data for a variety of different use cases including for AI and machine learning for automated for autonomous driving.
Uh we see Wix, Robin Hood, LinkedIn and a variety of other companies using Airflow for a wide wide variety of use cases. Now, one of the things I'd be tremendously interested in and please do reach out if there's something which you feel is dramatically different of in your particular sets of use cases which could benefit from additional capabilities in Airflow or some variations thereof. I'd personally love to hear this after the session. would uh really love to understand if you feel some of your use cases are not properly being represented as well.
As somebody who's been involved with Airflow from like a long time, I actually led the 2. 0 release of Airflow. We released that on the 17th of December 2020 and we just released Airflow 3 on the 22nd of April. So pretty excited by the growth uh from Airflow across the last four plus years. uh the growth in in downloads. I think the most recent numbers are actually significantly higher than this at this point. I think the last month was about 39 million downloads.
Uh I love the fact that we've got a really broad community. I think we've got over 3,200 contributors right now. And we have had a dramatic increase in the number of people contri contributing that to the project as part of Airflow 3. was thrilled to actually have over 330 contributors to Airflow uh 3 and like over 120 developers contributing to it from around the world which is one of the things I absolutely love but especially love seeing the uh close to 80,000 enterprises using Airflow from across the world which is a tremendous jump from over the last four years.
Also really love the fact that it's a very global community from AC from around the world. We've had people contributing and using Airflow from literally every place on the planet. And um I'm also thrilled as a result of this, the most recent set of work we've actually been doing is on internationalization. So in addition to English going forward in 3. 1 of Airflow, we're going to have support for multiple other languages.
Uh initial ones being German, Polish, Mandarin Chinese. Uh we've also had people contributing to a couple of other languages and like you know more coming up uh based on user demand and like interest in being able to support these couple of things which I've seen tremendously changing over the last few years as well. Airflow initially started and even like about two plus years ago I have said that the primary audience and in fact the vast vast majority of people using airflow were only data engineers were data engineers and only data engineers the common joke used to be airflow was built by data engineers for data engineers which actually was generally true a few years ago and uh one ex one thing I'm actually personally really excited about seeing over the last couple of years is about 30% of people now using Airflow are ML engineers and using it for MLOps and that's been a significant evolution in the kind of the demands of what they've actually wanted the additional use cases and how it actually gets used as well and therefore how we needed to evolve airflow to be able to support it the uh AI engineers has also been a significant growth about 10% of pe people using airflow today our AI engineers u significant ificant number of them using it for generative AI applications.
They have actually also brought up new use cases and new capabilities which they'd like to see to this particular project as well. So now that's been one of the key personas we've seen people using Airflow 4 and we're pretty excited to hear more feedback and uh about how we can actually make this experience better for all of you as well. The couple of interesting elements which I actually find in find different is about model training versus inference execution with the second two.
I mean this you know when I actually reference as to why we feel it's different with ETL transformations for data engineers there was one consistent set of flows we used to see for ML engineers who been primarily using it for model training and retraining and regeneration of data. That's been a a com a significant addition and capability which we've actually developed and deployed over a period of time with AI engineers.
What we're seeing more is less of model training but more of inference execution and the inference execution elements and like the reason thereof and what changes we need to make have really driven a lot of the capabilities recently within Airflow and again that's the reason why I'd love to kind of get your perspectives for the different use cases and the different personas as well. Nav uh and I'll touch on all of this as we kind of get through the use cases.
So key elements which you actually are pretty familiar with but again I'd love to hear about which areas you'd feel you'd like more capabilities as well. Clearly the pythonic development and pythonic uh dag writing is the first area which most people kind of said hey this is why I actually started using airflow along with the UI to be able to monitor it. The second set of capabilities which evolved after that were capabilities around higher language abstractions so that you didn't have to do the detail mechanics things like you know hey there might be a 100 files showing up how do I do this dynamically without actually having to write this and do a do a for loop or like you know do it dynamic task mapping really evolved for that being a parallel execution of a dynamic processing a dynamic set of inputs and very similar to that was higher levels of abstraction which we introduced for assets.
Originally we called it data sets that's now been significantly expanded into assets and then like the provider packages we haveund roughly about 97 providers I think at this point in time about 1,600 plus integrations distributed across those 19 provider packages and a lot of those integrations are things which are continuing to evolve. We'd love to hear people who say hey look either I want more provider packages or I want different incarnations or people I know we have also seen people wanting to contribute provider packages and uh telling me like at least me and other people on the PMC but you got to make this easier again more feedback love to kind of get this uh insights from all of you as well.
Now as I mentioned the airflow was really um has evolved tremendously. 10 years ago was when the initial incarnation from Airbnb after that was really the big milestone was getting this adopted as part of Apache top level project. Uh that was like a while back in 2019. uh as part of uh the evolution of airflow, it really started as being something which was widely adopted by people downloading it, data engineers downloading it onto their laptops, started delva pipelines and deploy it and that actually grew virally across the board.
Our big jump as part of Airflow 2. 0 was really making Airflow enterprise ready introduction of high availability active active active models, fully specified REST API and then starting to make the DAG writing more Pythonic with that HAS flow API. Since then, we've really focused on on uh a couple of different areas. One, making it more efficient and uh to be able to write things like differable operators, being able to make it more resilient, uh higher levels of abstraction.
I refer to dynamic tasks as being one of them. But then the other key element we started working on was significantly greater data awareness. Data aware scheduling automatically being able to trigger stuff because of data showing up conditional scheduling as a combination thereof were some of the key things which we really focused on. Our big jump as part of 3. 0 really started last year roughly about this time frame.
We started working on on figuring out what 3. 0 should happen as about April. We just released it on April 22nd with three main themes for 3. 0 So really the and this is actually what I'm going to jump into quite a bit more. Uh and really excited to share all of these capabilities with you and to hear about where else we can actually take this. The first key element was really driven based on the feedback from the community and people using Airflow which was hey we'd like to make this easier to use and so we focused a lot of work on intuitive experience.
The second key one is that made the reference about how how airflow grew tremendously through individual viral adoption. But then as enterprises started using it for business critical use cases for missionritical applications, they started asking for additional capabilities and enterprise level security. And that's kind of the key second driver really for Airflow 3 was better security. The third one is um far more flexibility as being able to write pipelines and integrate into systems.
And this is really was uh one of the key themes for Airflow 3 was being able to run tasks anywhere, anytime, and in any language. Far more flexibility as part of Airflow 3 as part of the intuitive experience. And all of these were driven based on community and user feedback. The first element was really being able to version DAGs. Again, DAGs uh Airflow is the way you write pipelines are Pythonic developer experience.
Therefore, you need to be able to version it over a period of time because these things evolve and how do you actually look at it? This has really been the number one feature asked for a significant amount of time. Second key thing was a much cleaner, absolutely new UI written from scratch. Huge effort, huge kudos to the team who got it done. We I think we had over 30 people across the world contributing to just the UI.
And then finally I referenced backfills or like you know said about MLOps is a use case backfills and a better support for backfills was really driven by a lot of people who had been working on uh MLOps and ML engineers for this particular case. DAG versioning consistent code execution. This has really been a key thing on the community wish list. just made that uh made that comment earlier. And um that code changes, right?
And because this is code, you may actually write if then loops, you might write additional tasks, you might actually delete tasks. And for people who have actually been debugging tasks or debugging DAGs because these things live for a long period of time, this was one of the most common things people have asked for. Now um I know I see people looking at it this in great detail but yes I mean actually this is available today in airflow 3.
We actually released the patch versions of airflow 3 as well. This morning we shipped 3. 0. 2 which is a second patch release on top of airflow 3. And uh DAG versioning is something been very very wellreceived. It's a groundup incarnation of how DAGs actually evolved. You can look at logs from prior prior tasks whether or not they've been still existing or deleted. You can look at code from prior versions. You can look at the current code.
You can look at like you know every single thing from an auditability, diagnosis, dab debugability perspective with DAG versioning. So really really thrilled with the job which the team has actually done on this front. Second key thing which I'm really thrilled about is the UI. The Airflow UI is one of the key reasons why a lot of people loved it. This has been completely rewritten based on React. Huge amount of work.
Lots of things have changed as part of it. We have tried to keep as many things as consistent which we think people liked like the grid view as being one of the core areas to kind of navigate from. Also added a lot of navigation. So that's far fewer clicks to be able to look at the t the status of a task uh within a particular uh DAG and be able to look at all of this uh directly. And the grid view is far more expanded as a result of this as well.
You can just kind of say, "Hey, um, I want to look at this because it was failing. It is failing today, but was this running yesterday? How do I look at this? How how do I go click on this and be able to go back and debug this?" We've had great view uh great reviews on this, including dark mode among other things. The ability to get to the actual errors and tasks and logging is much much faster than the way it was before.
So really excited about the changes on this front and I know that there's a lot of like requests coming in as well for additional features on this UI uh react based UI plugins are going to come as part of 3. 1 in August and that's one of the key things which are looking at being able to make this a lot more pluggable. This is the foundation for the internationalization stuff which I just talked about that PR got merged.
So we expect to see a few different languages support for this by August which is our next big release 3. 1 backfills at scale. talked about MLOps, talked about how Airflow is being getting used a lot more for ML and um this was one thing which was a little bit of an afterthought in Airflow 2 and before uh remember even chatting about this with like in 2020 I think and backfills are like now fundamentally built into Airflow.
Why do you need backfills right? You might actually have a newer model you and say, "Hey, I want to actually regenerate all the predictions or like look at the based on all the prior historical data for this new model. How do I actually make this available?" You want to actually go back and run an entire back fill for that particular reason. You might also run back fills because you introduced a new uh column in uh in your data warehouse and you say I want to populate this column based on running and like look at the historical elements and be able to populate this for going forward.
So backfills are a relatively common use case. Used to be less common with just ETL and ELT lot more common with MLOps and especially with AI. And now we've made this available through the API through the UI directly. You can specify what dates you want to be able to what date ranges you want to be able to backfill. You can also have concurrency and parallelism controls about how important it is, how many workers you want to actually use for backfills.
Be able to run this automatically. And this is probably second or third most requested feature in over the last couple of years because of the number of people using it for MLOps. So part of the key things of making it a full-fledged MLOps platform is a full-fledged support for backfills including the full-fledged reliability around this enterprise security. This is quite possibly sorry this is without a doubt actually my favorite feature for airflow 3 and the biggest reason why we needed to make a fundamental new version change.
Airflow historically has been a single package wherein the scheduleuler, the web server, the workers wherein your actual tasks and DAGs run and uh all the elements actually run and contact the the Airflow metadatabase. The Airflow metadatabase was used for state. So a DAG file processor reads it, reads the DAG from disk or like you know from a file system or GitHub, stores the serialized representation into the metadatabase.
The scheduleuler looks at that serialized representation in the metadatabase then kind of says hey this task needs to be scheduled at this time that's pulled by an available worker which then looks at the XCOM data or like you know connections and prior state from the metadatabase then runs it stores the state after the task into the airflow metadatabase and goes on. It's very airflow metadatabasecentric and chatty work perfectly well when you're actually being able to run this as individual engineers.
But for enterprises wherein there might be different engineers or different data analysts with different permissions and being able to look at this that became a security risk. I you could go potentially like if you're storing your credentials in the database, hey, can somebody actually else use this access it? And that became a a sec a security vulnerability for people wanting to share Airflow deployments across in a large enterprise and so on and so forth.
As part of Airflow 3, this is now decoupled. We have actually introduced a client server architecture so that workers or like userdefined code as airflow 3 and beyond no longer has direct database access. The way we have actually done this is introduce an API server. This API server is the only way user code can communicate with the airflow metadatabase. It's kind of uh with a particular JWTD token which is gives a much stronger security posture.
This API server is also used as a foundation for all the web server interactions because of it's all based on react. The react element kind of hits an API server as well. So it's got a very strong boundary for enforcement and for interaction and many many other things which I'll get into detail into further details. But this is a key reason why we had to introduce a major version change. Uh we actually take the semantic versioning very very seriously.
So this was actually required a lot of debate and a lot of thought before making this particular change. As part of this we are introducing a task execution interface which is the interface protocol kind of similar to HTTP between the tasks I the which are actually being run versus the API server on the back end. Okay. On the back end element I all the server components are now going to be known as airflow core. All the client end elements which are actually integrated into your workers i. e. your Python packages are now known as the task SDK.
The first task SDK which was shipped is a Pythonbased task SDK and that Python task SDK airflow core with this task execution interface has been shipped as part of Airflow 3. 0 and is going to be shipped continuously going forward. Our intention is when we cut new Airflow releases going so going on from now on they're going to be two versions. One is what is the server version. The second is what is the task SDK version.
And they can be upgraded independently based on what execution protocol they support which is going to make it really really easier for your Python dependency problem. I. e. your Python dependencies are no longer going to be reliant on any Python changes in your server i. e. Kubernetes and all that kind of stuff. Makes it completely independent. Makes it much better from a security profile. I. e. what CVE show up in a Python dependency for using the server versus what you need in the client.
Also really key for AI engineers because those packages change at such a frequent rate these days and you don't want your uh data teams which are dealing with it with dealing with ETL applications having to make those changes on the same frequency at all. So you can actually have completely different Python environments, different dependency models, different versioning schemes across the board, which is part of the reason why this is absolutely my favorite feature as part of making this change for Airflow 3 and part of the reason why it was why it was a big change and a big version change.
Okay, I think I probably talked about all of this already, which I tend to like to talk with pictures rather than words on the page. So that's just me. Now I actually said this is my favorite feature not just because of security- but because of what all it lets us do. One of the ways I actually define airflow 3 is being able to run tasks anywhere at any time in any language. The running at anywhere is really enabled because we now have an API server.
Task SDKs I actually mentioned are independent. They communicate not directly through the database but through an API server. This lets us have a very distributed model. I in the first place you can actually have tasks running locally on your local Kubernetes clusters or your VMs in the same network boundary. But also you can actually start having tasks running completely independently different clusters different clouds on your on-prem data centers communicating with an Airflow API server and the rest of the server components running in a central location comp.
Airflow at this point with this becomes a true multicloud hybrid cloud architecture and that is actually available today as part of Airflow 3. We are really excited by this. We believe that it's completely expands the kinds of things you can actually do with Airflow. You can actually run some tasks which are CPU based tasks on CPUs. You can run other tasks on GPUs completely differently. You can we actually support the whole rent a GPU model on a different cluster because you don't have to redo anything.
You don't have to set up airflow there. You don't have to set up your your different server components. You can just run these tasks over there on demand. Also enables a lot of like you know different kind of optimization techniques for cost optimization and running these tasks like potentially on your own local data centers as well because you don't want that data to kind of leave that particular premises. That could be either for privacy reasons, regulatory reasons, political reasons from different country boundaries and so on and so forth.
But that is one of the key reasons why we're really really excited by this capability which we call remote execution. Remote execution is actually available today and this is one of the foundational differences as part of Airflow 3 which we're pretty excited about. The second key thing which I love is being able to run at any time. Now we've actually done this based on um updating our framework honestly uh this is probably a long time coming.
We had actually introduced data awareness as data sets in 2. 4 I think which was a couple of years uh years ago. We have now expanded this to be uh higher levels of abstraction which we call data assets. There is now code associated with data assets which is what we call watchers. And now we support completely event-driven scheduling. event-driven scheduling in this model is like you know when you actually a file shows up or like a database table shows up or a message shows up in CFKA or a message system of any kind that can automatically trigger processing of that data as a DAG.
So DAGs can be fired based on a message arriving as part of data changes before that was available within Airflow. I can only detect changes based on a data set getting published in Airflow and then kind of triggering other data sets within Airflow. Now this is completely based on ecosystem changes of data showing up externally in either its vendor data or things like you know across the board like a message cube. Uh this is a common pattern which I've actually seen people saying hey we've already been bu building some kind of workarounds on around this.
Uh I've actually heard like you know multiple people building like you know data availability service publishing service to go across Airflow deployments. This is now kind of baked directly into Airflow today. Our first integration as part of 3. 0 we shipped was with uh AWS I think SQS uh but it is there's a CFKA is basically coming right now I think the PRs are in I think like you know Google pubsub and like you know other cloud messaging systems are going to come in they all have a common interface which we call the message uh Q interface so you can literally change just the plug-in or the configuration and then be able to work with a different messaging provider and as part of this airflow really actually does work from like you know being able run task at any time literally as soon as data shows up.
Uh we've had people with interest on this even for blockchain I when data shows up on a blockchain I want to actually done run visibility or analysis on crypto uh based on algorithms and show that the second key thing which we did with airflow is inference execution. This is one of the things I talked about AI engineers is different from MLOps. How do you actually do inference execution? So we enabled first of all support for inference execution.
You can run literally like you know non-data interval DAGs I. e. you can basically say this is not based on an interval. I just want to run this based on parameters. Look at my uh pre-trained model look at that data then come back with a new prediction based on the model I'm coming uh I've already trained for this inference execution we believe is exploding from a usage. So we're starting to see more and more um jobs or like you know tasks based on inference execution starting to happen and we just want to make this fundamentally easier with airflow as well.
I think I've actually touched on a lot of the use cases. I'm going to skip the very first one because most of you are really familiar with the uh ETL and ELT use cases. I've also touched on the backfills use case. One thing which I think is I find personally very interesting is how people have actually been using it in generative AI and for MLOps. Uh I actually like this use case. They spoke at the Airflow Summit from.
It is an automated it is an automated timekeeping platform for lawyers timebased professionals wherein you actually do a lot of work on your computer. You actually may have meetings. You may send out emails. uh you you have uh you're sending out updates into documents and sheets and you do all that work but then you have to actually send an invoice to your client about hey I did this work I took this long and I've got to send a a time sheet and an invoice there.
So the premise of law role was to say we're going to automate all of that for you now each of these processes are different because for lawyers it's regulated i. e. The billing into is really based on six minute intervals, right? And how often do any of us remember what we did in six minutes intervals to the entire day, right? So even if you try to do it the at the end of the day, they found that the quality or like in the accuracy drops tremendously.
If you try do to do it in the end of the week, it drops much much further even when you're talking about six minute increments. So you really need this ideally to be automated completely. Now there are two challenges with this. One is this is regulated which as anyone who's interacted with lawyers you probably say hey they charge so much. I'm really glad there is guidelines on this about what is billable versus what is non-billable and what you can actually bill for.
So for example setting up a meeting with a client is not a billable activity. You can't be kind of saying hey I'm going to charge the client for the amount of time I'll spend in scheduling time with you. However, the time spent in like actually reading their their document to prepare like an analysis is billable. I'm just and like if you actually if the lawyer shows up in court that is billable but also at a different rate.
So it's like you know many different elements. So you got to be able to look at the calendar entries, you look at emails, you look at documents and then you create an automated time sheet, right? And from that you create an invoice. So now there's two elements. One which is actually the creating of the time sheet of the invoice is an inference execution, right? That's what we just talked about. The it can also be a data interval if you're processing a whole bunch of stuff.
But if you say, "Hey, I just finished the meeting." Boom. It kind of shows up. That's kind of like very very simple element of stuff. Now, but the other thing is all lawyers are like, you know, different firms have different regulations. Different countries have different regulations. So for that you need to train the model. You need to train the model to be specific to that particular client or that particular um firm and to be able to actually report that.
So there's two streams of work they actually have in this case. One is which is model training in this case model fine-tuning is really more of the true case but it is you also pick which model works uh on a very very frequent basis and then you actually apply user activity to this train model to kind of come up with the uh with the data. So how does airflow 3 help? First you need to be able to detect right user activity on new data.
Be able to trigger a pipeline as soon as a meeting happens so that you can actually say again ideally as soon as the event happens if you can get if you can uh run this through the model say hey was this with for this case for this amount of time. The chance of an attorney being able to approve it and saying yep that looks right within like the span of an hour. Yeah, they'll probably be pretty accurate and they can probably correct it really quickly.
If this happens like you know the next day, the chance of them being able to say yes, that was right or wrong is much lower. So the key thing is this drives an event-based interaction really makes this a lot simpler, a lot more accurate for them as well. The second key thing is we just talked about retraining models on historical data. With generative AI, this is happening a heck of a lot more frequent. with they actually look at different models like you know whether it's llama or open AI or cloud they evaluate different models and they change models literally every week and that's a very common pattern that we have seen with people using generative AI at least within the context of airflow they're looking at multiple different models trying to figure out hey what is the cost of running this particular model what is the performance or the quality of running this particular model and then change that model very very quickly if those results change, right?
Again, the cost of running a GPD3 model versus a GPD4 model from a cost standpoint is is rather significant when you start adding up saying, hey, I'm doing this as part of my platform. What is my margin going to look like and what is the return ROI on this? So model retraining and being able to run backs based on this critical use case and that's why we see so many changes and that's the whole reason for backfills. The third element which we see is some of these are like compute intensive, right? especially when you're actually training and you really want to run that on GPUs whenever possible.
One of our other uh customers as u using airflow and astronomer is uh do this for climate change right you actually like you know for tracking um storms coming in and therefore like you know rerouting ships based on traffic and you can eat that on time and that's really real time stuff when you want to reroute traffic kind of going on sea based on like you know storms or winds and stuff and that can't just wait and for that being able to run it on GPUs because I want to get that data compressed running this really quickly is really really important and uh the selection of hey where is the GPA available how much price is it can I run this on remotely on that particular cluster is a critical need for being able to actually deliver the data on time when it's actually useful versus when it's it becomes if it's too late that data is completely useless.
Being able to have the remote uh execution capabilities so you can dispatch these run these on different places at real time and as quickly as possible to get get the data was critically useful for them in these sets of use cases which is part of the reason what drove us towards enabling these remote execution capabilities outside of a core environment. There's a few other use cases like this personalized uh newsletter generation wherein you can optimize and like an autom automate the gathering of data the selection of data and then the sending of data.
Again this is the notion of like the classic personalization level stuff uh which is again inference execu a combination of model training for like you know figuring out what somebody would want what somebody would like and then a combination of inference execution at scale based on the data availability and based on you know hey when this needs to happen so event-driven scheduling for real-time processing is something we've seen a lot of adoption a lot of interest but also this is where I frankly love a lot of feedback because the integration patterns for this seem to be going through the roof.
There's a lot of people asking for a lot of different kinds of integration here. So, I'd love to kind of get your your folks's perspective on this. Really love to listen on this. The parallel scalability for inference execution is driving a lot of demand and more scalability uh requirements as well. And then the other thing which we've seen a lot of is and actually this is one thing I really love in the UI now in the airflow 3 UI is depending on how you want to work you can look at your UI in the in the UI completely based on DAGs and tasks or in the same UI you can look at it based completely based on assets and the same things work.
So we're trying to be fitting with Airflow's philosophy. We try to be really non-opinionated. We say, "Hey, there people in your team or you individuals may want to look at it a certain way. Other people want to look at it a certain way." And we want to enable that flexibility across the board. So we're completely unopinionated. Now you might actually say, "We've made this more confusing." But again, we'd love to kind of get your feedback on this because we've tried to make that really, really simple.
Same thing with healthcare. the one key uh driver in healthcare which we see over and beyond this again the traceability auditability has all been really critical but for claim analysis and analytics one of the key things which we see a lot of is uh people not wanting that data to leave their particular cloud and be able to run that particular data and be able to run that particular transformation inside that particular environment.
So it is again remote execution for task isolation was a key part in that region running that data that's task in that particular region. traceability, being able to say, "Hey, look, I ran this using this code of this DAG at this time and that version." being able to kind of go back and prove that for an auditor or a regulator saying this is the model I used. This is the code I used to process that information in that year which created this particular prediction or this particular um chain or like you know response to this insurance for this particular patient is actually not only nice to have it's actually kind of critical for that particular set of use cases.
So we're seeing a lot of people um coming up with use cases in healthcare. We've seen like a lot of talks getting submitted in healthcare for not only patient tracking and analysis and claims but actually we're seeing a lot of stuff coming in even for inference i. e for automated uh kind of like an advice nurse capability early based on patient records as these are kind of getting built in into the system. And finally key takeaways we believe data orchestration is fundamental for being able to manage your data in an efficient scalable way.
We believe from our understanding what we see is Airflow is the de facto standard for data orchestration globally. See a huge amount of this across the world and we're really excited with Airflow 3 for all the changes in making the user experience better the better security through task isolation and the and the client server separation and all the flexibility which you're now able to build on top of this. Thank you for listening.
Love to hear questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.