Sponsored By: Astronomer | Unifying Data and AI Orchestration with Apache Airflow

Summary

A single LLM call is not production-ready AI—this video explains the four things missing from raw model code: resilience with retry logic, validation of model outputs, observability into what ran and why, and human-in-the-loop approval workflows.
Apache Airflow provides a unified orchestration control plane for both data pipelines and AI workflows, enabling multi-agent orchestration, AI-driven pipeline branching, dynamic task mapping for parallelized LLM inference, and an AI-as-judge pattern where a second model scores the first.
Astronomer's managed Airflow service on Databricks supports production AI patterns including deferred human approval operators, SQL-based data quality validation, and the 'mise en place' mindset of ensuring everything is prepared and governed before an AI workflow executes.

Chapters

FAQs

Why isn't a basic LLM API call production-ready?

This video identifies four gaps that make raw LLM code unfit for production: lack of resilience (no retry or failure handling), no output validation, no observability into what ran and why, and no mechanism for human review of sensitive decisions. Apache Airflow addresses all four by wrapping AI calls in a governed, observable pipeline.

What is the AI-as-judge pattern in Airflow?

The AI-as-judge pattern uses a second AI model to evaluate and score the output of a first model before that output is used downstream. This video demonstrates the pattern in the context of a customer review pipeline, where the scoring model validates response quality and flags low-confidence outputs for human review.

How does dynamic task mapping help with LLM inference in Airflow?

Dynamic task mapping in Airflow allows a single task definition to be automatically parallelized across a variable number of inputs at runtime. This video shows how this pattern enables efficient, parallelized LLM inference across large datasets without requiring separate task definitions for each input.

What is the 'mise en place' mindset for production AI pipelines?

Mise en place is a culinary concept meaning 'everything in its place before the rush,' applied in this video to the discipline of preparing data quality checks, governance controls, and observability before an AI workflow runs. Astronomer presents this as the operational mindset that separates AI experiments from reliable production systems.

Full transcript

[00:09] Okay. Hi everybody. Uh, thanks for being here. I'm going to go ahead and get started. I'm going to be talking today about unifying data and AI orchestration with Apache Airflow. Uh, just a show of hands. How many people here are being asked to implement AI workflows in their
[00:22] jobs? Okay, lots of hands. Me too. Uh well hopefully this talk will provide some insight on how you can make that easier. Uh before I dive into it give just a
[00:34] brief introduction. I'm Kenton Danis. I'm a director of developer relations at Astronomer. I've been using Apache Airflow since 2019 when I was a data engineer. So I have quite a lot of history with the project. I'm always really excited to talk to people about
[00:47] new ways that they can use Airflow. Uh when I'm not working, I'm typically out in the mountains trail running or skiing. So if you want to talk to me about either of those things, come and find me afterwards. But for now, I'll go ahead and dig in.
[01:01] Uh so for my talk today, the agenda is going to be I'm going to start with a bit of an overview on what I mean when I say unified data and AI orchestration, why that matters, why we should care about it. I'll talk about implementing
[01:13] those type of workflows with Apache Airflow and data bricks. I'll go through a couple of different examples to provide some inspiration on what this might look like um in a real world setting. And finally, I'll end with what
[01:25] I refer to as the mison plus mindset. So I'll explain what that means. Uh but basically that is going to be talking about how we can successfully productionize these data and AI projects.
[01:37] So starting with unified data and AI orchestration, why does this matter or why should we care about it? And I want to start with a question. So everybody take a look at this code for
[01:49] just a minute. It's pretty straightforward. Um, we are sending a prompt to some open AAI model to presumably write a support response to some sort of ticket or complaint. Uh, by
[02:01] a show of hands, who thinks that this code is ready for production? Okay, good. I didn't see any hands. Um, that is the right answer. So, the question here is what is missing?
[02:14] And there's four things that I want to cover. Uh, the first is resilience. So there's nothing in this code that will handle any failures. There's nothing that will retry if something goes wrong or even let us know that something has
[02:26] failed. Uh the second is validation. So here we're kind of blindly trusting the model output. Uh we don't know what format it's going to be in and we have no means of verifying consistency.
[02:39] Uh the third is approval. So if we are having the AI um draft a response to some sort of customer complaint, uh we probably want a human to look at that before we send it off. Um probably
[02:51] everyone has seen here uh what happens when those sort of things go sideways. It's not a place that any of us want to be. Uh and then the last thing is observability. So once this runs, we have no idea what happened. Neither does
[03:04] anybody else on your team. So these things are the gap between a prototype and a production AI system. And it's this gap that we're going to spend the rest of the talk discussing how we can
[03:16] close. Uh and just going back in time a little bit, who has seen this diagram before or one like it? Maybe. Yeah, hands. Um pretty classic. So this one is referring
[03:28] to machine learning code and what it takes to put it into production. Um again, kind of classic diagrams. the model is a pretty small box in the middle and everything around it is what you have to do to actually get a productionized use out of that model.
[03:41] Um, random aside, I studied grad uh data science in grad school and we spent a lot of time studying like the very nitty-gritty internals like the math of how a lot of machine learning models
[03:53] work. Um, and true to form with uh going through education, if you care a lot about theory, the minute you start working, you no longer care about that theory. Uh, so I don't think I've ever used that math since grad school. Um,
[04:06] when I started working as a data scientist, what I cared about was all of these boxes around it. So in practice, of course, the model is a very small part of productionizing. And the exact same picture applies to AI
[04:18] today. So of course LLMs are very impressive. You can do a lot of stuff with them, but they're a tiny fraction of what you actually need to ship if you're going to get something out of this in your day-to-day work. Um, you are going to have to deal with things
[04:31] like data ingestion, prompt and model config, um, data quality, token management. Depending on your company, that might be in the downwards or upwards direction. Um, but management nonetheless. Uh, you'll have to figure
[04:43] out what to do with your context, how to handle errors, how to insert your human judgment, how to observe all of this stuff. Um, so again, the hard parts haven't really changed. They just have new names within the context of AI.
[04:57] And that brings me to kind of the punchline of this section, which is that everyone has access to the same foundational models. We can all write a prompt to Claude or Gemini or whatever it is that we're using. Um, but your competitive differentiation is not going
[05:10] to come from that. Um, it's going to come from the operational and context layers around the model. Um, so of course here I'm going to argue that that operational and context layer should be Apache Airflow.
[05:24] Um so let's talk about how you can actually implement um productionized AI workflows with Airflow and data bricks. Uh how many folks here are already using Airflow or at least familiar with it?
[05:37] Okay, lots of hands. Um great. Well, for those of you who aren't, um very brief introduction. Uh Airflow is the world's most widely used open source tool for data orchestration. Uh it is a massive open source project. frequently sees
[05:50] more than 30 or 40 million monthly downloads. Uh it has over 3,000 contributors all around the world. Um Astronomer I work is of course heavily invested in open source airflow. We drive 100% of their releases on the
[06:03] project. Um and we have deep visibility into where it's headed. Uh and of course Airflow is used by massive Fortune 100 companies like the ones on this slide. Some of you might work at these companies or ones that could have been
[06:16] on this slide instead. And they're all taking an orchestration first approach. And I have a very brief again overview here into some of the concepts that drive Airflow. Just in case you are
[06:28] newer to Airflow, this is kind of the bare bones of what you need to know just to understand the rest of my talk. If you are interested in more Airflow introductory stuff, come find me afterwards. My team has lots of resources for you. Um, but this is the
[06:41] crash course. So within Airflow, you have a DAG that's just your data pipeline. Um it's shown on the left here uh as a graph and on the right in the code as the code that defines it. So all DAGs are written in Python code. Within
[06:55] that DAG, you're going to have tasks. So tasks are like a unit of work within your data pipeline. Um and those tasks are defined by operators and decorators. So those are the templates that define what the task actually does. So in this
[07:07] particular example, we've got a task decorator uh that defines a Python function and a bash decorator that defines a bash function. Um the example here is using the task flow API which is the most Pythonic way of writing DAGs.
[07:20] Just a note to say it's not the only way. Um and if you are used to defining Airflow code in a different way, the way that you're used to is also completely fine here for all of the use cases that I'm going to talk about.
[07:32] And then of course Airflow is great, but you also have to run it somewhere. Um there's many ways of running open source Airflow. Uh some teams are able to do that themselves. Um and that's great. uh other teams either don't have the
[07:45] resources to run airflow themselves or uh don't want to spend their time doing that. So if that is you um a managed service can be a great solution there. Um astronomer provides Astro which is
[07:57] our managed airflow service place where you can build run and observe your data pipelines um using airflow. Uh if you are interested in learning more about Astro and the features, come see us at our booth later or come find me after
[08:10] the talk. Um and I can answer any questions. We can give you a demo, anything like that. Okay. And then one of the best things about Airflow, of course, is that it can be used to orchestrate nearly any tool
[08:23] in the data ecosystem. Um, since I assume everybody here is probably using data bricks at least to some degree, I'm going to focus my use cases here on Airflow and data bricks, but just know that this is by far um this is far from
[08:36] the only way that you can implement AI workflows like this. Um, so you can swap out the tools as you need. Um, with Airflow and data bricks, there is of course a data bricks provider that has specialized operators for doing things
[08:49] like copying data into data bricks, running notebooks, running spark jobs, all that orchestration stuff that you need. Um, you will also likely use more generic providers really in any pipeline that you're implementing. So ones like
[09:02] the common SQL provider uh to do data transformations in whatever database you're using are really common and you will likely need to use those here. And in this section, I'm just going
[09:14] through some of the tools that you have in your toolbox that we're going to look at later um in our examples for implementing uh orchestration. And this is a really important one. Um and that's data quality. So everybody here has
[09:28] probably heard the garbage in garbage out um phrase as it relates to machine learning code or even just ETL pipelines. Any sort of data work um that applies to AI pipelines in the same way. So this is another place that you can
[09:41] leverage the common SQL provider in Airflow. It ships with um these operators listed here which just provide you various ways of sort of cutting and evaluating the data um in a table or multiple tables to check the quality um
[09:54] as it makes sense to your use case. But the pattern that I want you to remember here um is that you should be putting these up front as gates in your AI pipelines so that if any data does not
[10:06] meet your rules, it never gets to the LLM in inference part. Um that's a huge amount of risk that you can eliminate in production for basically no extra work.
[10:18] Um and then finally we'll look at the AI specific part of this. So for AI pipelines in Airflow, we now have the common AI provider which is under active development. It was originally released
[10:30] in March or April, I can't remember exactly, but just pretty recently a couple months ago. Um it has been being it is being developed quite rapidly. Um so I know I say check for latest versions here, but it's especially true
[10:43] in this case. Um when I submitted these slides, we were on 0.1.1. We're now well past that. Um, but the idea with this provider is that LLM and AI inference become first class citizens of your pipelines. Um, so you have various
[10:56] operators available to you. We're going to go through some of these in my examples. Um, but you can uh do all of your AI orchestration using this provider. That's pretty great.
[11:08] Of course, if all of the um kind of options that I've covered do not uh work for the use case that you are implementing um and there isn't a provider for whatever tool that you are working with, Airflow is really
[11:20] flexible. It's one of the best things about it. So, you can always drop down to plain Python um inside a regular task. So, if you can define it in Python, you can orchestrate it in Airflow uh and you can create custom
[11:32] tasks that way. I'll also throw in there that on the opposite side of that if you are not using Airflow currently and you're also not super comfortable in Python and this seems like kind of too much to learn to
[11:45] implement pipelines like this. Um this is something where some of the features that we have in Astro astronomers managed service can help. Um we do have our Astro IDE which has a built-in AI assistant. So you can offload some of
[11:58] your DAG authoring to that. Um, we also have a service called Blueprint which gives you a drag and drop um, interface for self-service DAG authoring. Uh, this is again something that we can talk to
[12:10] you more about at our booth if we're if you're interested. We have demos of these things so you can see a little more about how they work. Okay. And then of course uh when you're working with Airflow and any other tool,
[12:22] you will need to connect to it. I always sort of joke that this is the most annoying part of Airflow, but usually once you get it set up, it works pretty well. Um, if you are working with Airflow and data bricks, um, probably you're going to be using OOTH to do that
[12:34] connection. Uh, you do have other options. You can use something like a personal access token. We don't recommend that outside of like local dev or testing situations. Um, but again, you can work with kind of the needs of
[12:47] your organization in terms of connections there. And then that connection is probably going to look something like this. Um, so you will have again the connection to data bricks. um with whatever your host
[13:01] and connection method is. Um if you are using the common AI provider, you will also need a connection to uh a pedantic AI connection that will connect to wherever your model is hosted. That might be data bricks as well. So using
[13:14] like a foundation model um and you can see that in the host example in this code. Uh you can define these you know like this in code that would live in like an M file or something like that for your Airflow environment. You can of
[13:26] course also put them um directly into the Airflow UI in the connections manager there. Um or another plug um for Astro if you are using our managed service we do have
[13:38] an environment manager where you can um store centralized connections variables and other secrets for your Airflow environments. Um you can then apply them to multiple deployments and workspaces.
[13:50] One of the big benefits of this is that if you want to have the same connection defined for like dev and prod um so your dev one goes to your dev database or prod goes to your prod database um you can do that and override certain fields
[14:03] without changing the connection name so you don't have to make any changes to your airflow code um so again if that's something you're interested in come find me afterwards and then now that we know kind of some
[14:16] of the building blocks I want to talk about how we're going to put them all together And this is where I want to introduce this term called mison plas. Um this is a French phrase that means putting in place. Uh it is used in the
[14:28] context of professional kitchens and it refers to the setup that they do before cooking. Uh so things like chopping vegetables, making sauces, uh putting everything in order. And the concept
[14:40] here is that a professional kitchen does not or should not be improvising during a rush. You want to have everything in place before your first orders come in so you can be efficient and respond to a growing volume of orders as needed. Your
[14:53] data and AI pipeline should work the same way. So before you put anything into production, you need to have these concepts in place of resilience, validation, approval, and observability. Uh and you need to have that pre-built
[15:05] in again before you're shipping these things off. So, I want to um illustrate these concepts with two specific examples that we'll walk through again to provide inspiration on what this might look like
[15:18] in a um an actual use case. So, the first we're going to look at is customer review processing. Before I talk more about um the actual use case here, I want to first look at the pipeline because it's actually pretty straightforward. And you'll notice with
[15:31] AI orchestration, a lot of it looks exactly like the ETL pipelines or things like that that you might be used to. Um, so this is the DAG that we're going to be walking through. Um, the first step is it's going to pull all of these
[15:44] customer reviews from whatever database we're storing them in. So maybe Lakehouse or something like that. Um, we're then going to do some formatting on those reviews with Python. Uh, then we're going to run our LLM inference part. So we're going to use like a data
[15:57] bricks foundation model for that again from the common AI provider in Airflow. uh we'll then take the results of that, do some preparation, and then save all that analysis back to our database. So, Airflow here is going to be the unified
[16:11] data and AI orchestration layer underneath all of this. And looking at the use case for this a little bit more, our setup here is we have a fictional uh company that sells
[16:23] trips to space. So, you can book your family a vacation to the moon or Mars or something similar. Um, and of course they get customer reviews, so hopefully some of them are good. Or in this case, uh, somebody is saying that they paid
[16:36] $60,000 for, uh, two passengers to Mars and we're very disappointed that the food came in vacuum-sealed packages. Honestly, fair. Uh, so what the pipeline is going to do is it's going to take in
[16:48] um this review information and produce four outputs. So we're going to get an analysis summary of the review. uh if there is an image like there is in this case, we'll get a written summary of the
[17:00] image. Uh we'll get a sentiment assigned to the review. So in this case, that's negative. And we'll get a category assigned to the review as well. And the idea here is that we can use all of this information to help us process all of
[17:13] these reviews downstream. So we can route them to the right team based on the category or sentiment. Um we can do, you know, a summary and analysis to see how we're doing as an overall company, things like that.
[17:26] I also want to add an extra note on this um example in particular that all of the code for this example is actually publicly available. This is part of an AI orchestration with Airflow workshop that we run. So if you are interested in
[17:38] playing around with that um again come find me afterwards and I can point you in the right direction. Um and you do get to actually see all of the results of the AI pipelines here in an Airflow plugin. I also like to highlight this because while you're probably not going
[17:51] to be showing customer reviews for your actual jobs in an Airflow plugin, um the plugins interface in Airflow 3 is really powerful. Um and you can do a lot with it, you know, on top of Airflow's
[18:03] regular orchestration capabilities. So definitely something to look into. We have teams at Astronomer that do use this internally um for their own teams analysis. Okay, so now let's look some at the
[18:17] ingredients at the of this actual pipeline and we'll show kind of how it is more ready for production than the first code snippet that we looked at. So the first thing that we're doing here is defining a structured output. Um so this
[18:30] is that validation step. So here we're defining the paidantic base model and that's going to describe exactly what we want to get back from the LLM. So that's how we know that every time we're going to get those four things I mentioned,
[18:42] the sentiment, the category, uh the summary, and the image description. Um, so this will give us a consistent output every time again, so we always know what to expect, and we can ensure that any downstream tasks that are processing
[18:54] that information aren't going to fail because they get something unexpected. The second thing that I have in this example is automatic retries. Um, so that's defined in the default args on the DAG. So, we've set it to three here
[19:06] with a retry delay of 10 seconds. Um, this will apply to every task in your DAG. Um, this is some of the most basic functionality within Airflow. Um, and again, it's going to give you baseline resilience, you know, basically for
[19:19] free. Uh, from there, we're going to go ahead and get all of our reviews that we're going to process. So, this is super straightforward. Again, we're getting this data from a database, a lakehouse,
[19:31] or wherever it is that you're storing it. Um once you have a connection set up using uh the common SQL provider is very straightforward. So we use the SQL execute query operator to run a query and grab all of the data uh that we want
[19:43] to analyze. And then we can move on after we've formatted that to uh the LLM inference part. So again this is where we're going to use the common AI provider. And here we're going to use the task. Llm
[19:56] decorator. This is also pretty straightforward. So it's going to take a connection ID, the system prompt, the output type, um, which is your pyantic model. Uh, notice here that we are also setting the max active TI's per DAG run
[20:10] parameter. So that's going to help us control parallelism when we're trying to process a lot of reviews. I'm going to come back to that in just a minute. Um, I also want to point out one part of this that can be a little counterintuitive when using um, the LLM
[20:23] decorator, and that's that the function that you decorate is not where the actual inference is going to happen. Um the function is what prepares the context. So whatever you are termed from it becomes the prompt that the LLM sees.
[20:35] Um and the actual inference happens when Airflow runs this task, sends the prompt off to the LLM and gets a result back.
[20:47] And then I mentioned that we are going to try and do some parallelization here. So, when we're processing re customer reviews, we might not know on any given day or however often we're running this whether we're going to have say 50 reviews to process or 200 reviews to
[21:01] process, but we don't want to shove all of that into a single task um because that makes it really hard to rerun if something fails or um even look at the logs to see what's happening for a particular review. Um so, this is where
[21:14] we can use dynamic task mapping. Um, this is also kind of a breadandbut feature of Airflow. That means that at runtime, Airflow will take some information from upstream. So in this case, we would pass in how many reviews
[21:26] that we have and it'll create one instance of this analyze review task per element in that formatted list. Um, so we can fan out again based on the number of reviews and we can do that for every
[21:39] DAG run. Again, here we can set this MAC max active TIS per DAG run setting. Um, so that will cap how many of those runs can happen concurrently. Um, the goal here is that you don't want to uh
[21:53] overwhelm your Airflow infrastructure if you do have a day where there were a lot of reviews. Um, but you also still want to have that dynamic parallelism. In this particular case, we have set it to one, which I would say is very very
[22:05] conservative and probably not necessary, but of course, you know, you can work with your own Airflow in infrastructure to figure out kind of what is right there. Okay, so summary um of this use case and
[22:20] everything that we covered that again makes this more ready for production than the first uh code we looked at. So we're running um SQL against our data bricks data break datab bricks database
[22:32] uh to get all of the reviews that we're using. We are defining structured output so we have a validation. We know what to expect from the model coming out. Um, we've parallelized this at runtime. So, this is more scalable, can kind of fan
[22:45] in and out depending on the volume of reviews we need to process. Uh, we've added retries and retry delays, so we have some resiliency in case there are any failures. And of course, we're using the common AI provider to actually do
[22:57] the LLM inference, uh, which is pretty straightforward. Of course, if you need to, if you don't have a way of running this on Airflow yourself, you can run and observe it on Astro.
[23:10] Okay, cool. So now I want to look at a slightly more complex example. Um, and this is one about decision tracing. So here we're going to look at fetching pending support tickets and responding
[23:22] to them. So again, I'll come back to the use case in just a minute, but I want to look at the pipeline first. Um, so here we're going to get our support tickets again from our database. We're going to generate an AI response to them using our foundation model or whatever LLM
[23:36] we're using. Then we're going to run a second agent as a judge to rate that response. So I'll talk more in depth about that in just a minute. Um we'll format that request and then we're going to route this through a human approval step and then we'll process kind of that
[23:50] final decision. So the key thing on this slide again is the line at the bottom. We're saying that Airflow is the unified data and AI orchestration layer, but here it's also the context layer. So the goal with this use case is that every
[24:03] run of this pipeline is going to generate context about how and why a decision was made and then we're going to feed that into future runs so that we can use that context to improve our agents. So hopefully they get better at
[24:15] this over time. We have to intervene less. Okay. So I want to take a look a little bit more about this use case. Again we are responding to customer support tickets. This is a pretty high-risk
[24:28] highreward kind of use case. Um, support teams of course can be overwhelmed. It's really great to offload some of that work to um, an AI if you can, but it's also somewhere where we need a lot of guard rails. Um, how many people here
[24:41] have seen news headlines about when AI makes bad customer support decisions? Nods. Yeah. Um, it's kind of funny if it's not happening to you, it's not great that it can make the news. Um, obviously we do not want that to happen
[24:54] to us. So, we're going to again use Airflow to build in a lot of guard rails here. So, what we're going to do is we're going to feed our support ticket in as a JSON. Um, we're going to get out the AI summary and response. Again, the
[25:08] second AI's judgment on that response, a place for us to input our human judgment and a way of managing the decision trace again so that we can use it to improve the agent next time.
[25:20] So, for the human response, this is the most important guardrail and this is very very relevant for a lot of AI orchestration. So I want to spend a few minutes talking about uh the most relevant airflow feature here and that
[25:32] is called human in the loop. Um this is again what we're going to use so that we can input our own judgment in these pipelines. So human in the loop was released in airflow 3.1 uh which came out kind of in the fall of last year. Um
[25:46] and it allows you to put operators into your DAGs that will wait for a human to input something. We have different options there which I'll talk about in just a minute. um while that task is
[25:58] waiting it gets deferred which means it does not hold on to its worker slot. So these are pretty performant even if you have a lot of them. Um and then what you are actually entering um and what you
[26:10] see as the person who is you know reviewing some sort of response um before making a decision is totally up to you. So the form gets rendered in markdown. So in this particular case, we're including the text of the AI
[26:23] response, that second AI's rating, um, as well as our action options. So we'll look at each of those in a little bit more detail. Um, so this is as a human reviewer who's going to review again the
[26:35] AI's response to a support ticket. This is what I am going to see when I go into the Airflow UI. Um, so the first thing here is that we see um the AI's response to the ticket. Uh so the model has
[26:47] drafted a complete reply in this case to a customer that has been locked out of their account. Um and it looks pretty good. It's friendly, accurate, covers the unlock flow. Again, this is a pretty straightforward um sort of ticket. So
[27:00] hopefully we don't need to do too much. And then if we scroll down in the form, we will also see the second agent's review rating. So what we've instructed this agent to do is give an accuracy, a tone, a completeness, and helpfulness,
[27:13] as well as a list of suggested improvements. And again, that's all for the first agent. Um, the goal of using this pattern that we refer to as AI as a judge again is just to give the human reviewer a little extra signal to work
[27:25] with. If we have a lot of these that we need to look at and, you know, say like 30% are just pretty mundane things like customers getting locked out of their account, we probably don't need to do
[27:37] too much other than just give it a glance and say like, "Yeah, that looks good. Thumbs up." um versus a more complex thing where maybe the AI as a judge says like this response doesn't really seem like it's actually solving the issue or something like that. We
[27:50] know to pay more attention there. Um and then finally at the bottom of the form uh the human reviewing gets to pick an action. So in this case we've defined three different options. We can approve
[28:02] the response. Um we can enter our own manual response or we can escalate to our CR team. Um, so we have a free text field for um, the manual response if we choose to provide one. And then we have another free text field for the reason
[28:15] for our decision, which is probably the most important part in this whole pipeline. Um, so we are always asking why. If we approved the response, why did we do that? And maybe even more importantly, if we rejected the
[28:27] response, why did we do that? Um, that context is what is going to make the next run of your AI even better. Okay, so looking at how we've actually implemented this pipeline. Um, in this
[28:41] case, we are going to use the agent decorator in the common AI provider and that's going to work much like the LLM inference did in our past example. Um, so the task agent decorator is going to take a connection ID, a system prompt,
[28:53] an output type, and then importantly, it also has agent params. So this is where we can pass tools. Um, and tools are just Python functions, but they give your agent access to something, probably
[29:06] another system, so that that agent can create better responses or make better decisions, whatever it is. Um, so here we have um defined on the left uh a tool function ourself called read all decision traces. Um, this is what the
[29:20] agent is going to use to take the context that we've um gathered from previous runs of the DAG uh to make itself better over time. And then the AI or the agent on the right is the response generator. So it has access to
[29:34] that decision traces tool. We've also given it a check the road map tool um which presumably gets information about our product road map that might be useful in answering support tickets. Um and it can use those to generate the
[29:47] response that goes to the customer. Um and I want to kind of back up a little bit and talk about this tool set functionality more. um because the common AI provider in addition to um
[29:59] having the operators listed here and that I've spoken about previously also has some really powerful functionality and tool sets are one of those. Um so again tool sets allow um your agents to
[30:11] access another system or do something specific that you have defined in a Python function and it's going to make them better at their jobs. Um within the common AI provider there are a couple of tool sets that ship out of the box. So
[30:23] one example is the SQL tool set which is access to a database. There's a data fusion tool set. Um you can also turn any Airflow hook into a tool set. Um so there are over 350 hooks in Airflow. So
[30:35] again as you might imagine this gives you a ton of flexibility in terms of what you can allow your agents to do. I'll also call out here um that the summary stats here that are about the common AI provider are definitely out of date. All of these have been um
[30:48] increased now since I originally made these slides. So it's it's pretty great what you can already do with this provider. And just an example of what this might look like architecturally using some of
[31:00] these tool sets. So if you implemented something like you want an agent that can answer, you know, questions about your data for you, you can implement all of this with Airflow um in a pipeline. So you have some way of inputting that
[31:12] question and then you have this research agent that's going to find the answer for you. that agent is going to have access through tool sets to things like your database. So through the SQL tool set um to things like your iceberg
[31:25] tables and data bricks through the data fusion tool set it can access systems through MCP servers if you have those and then of course you can build in this human in the loop review so that um you're making sure that the output looks
[31:37] right. Um so again just a highle kind of architectural example of what this might look like. I also want to call out that when doing this sort of work with data bricks specifically, you do have a a couple of
[31:50] patterns that you can use for agents. Um, so one is the pattern that I've been talking about so far, which is where you define the agent in Airflow. So you're calling a model, but all of the context um and everything around it is managed
[32:03] within your Airflow DAG. So that's like if you wanted to use a foundation model or something like that. Um you can also have um all of the agent uh tooling around it hosted somewhere else. So if
[32:16] you wanted to use data bricks if you are or data bricks agent bricks if you already have an agent defined there and you're happy managing all of the context and kind of evaluation and serving of that agent within data bricks you can
[32:28] absolutely use that as well. You can still use airflow to call that um via an endpoint and use airflow to manage kind of passing data in and out. Um so both of these options are totally valid and can be used in the context of the um
[32:41] examples that I'm giving. All right. Now back to the original pipeline. Um so we have implemented our AI agents. Now we're on to our guard rails. Um so the human in the loop
[32:55] operator again is a really important piece of this. Um we have implemented this in this case with just the human in the loop operator which is the most flexible of the operators for this feature. Um it allows you to really
[33:08] implement kind of whatever you want um in terms of what you see in that form and what options you can give in. So in this case we've given it a subject a markdown body the action options um
[33:20] whether we're allowing multiple selections things like that. Um definitely uh do set the execution timeout. So this will allow you to fall back to a default or fail the pipeline if you don't have a default um if you
[33:33] haven't responded in time. So that can be an important piece of moving this to production. Um and then just another note for human in the loop if you want something simpler there are other more specialized
[33:45] operators that are a little easier to implement that do specific things. They're all named pretty aptly so it's pretty clear what they do. Um you have uh the entry operator which allows you to input into a form. Branching allows
[33:57] you to pick a branch in a DAG. Um and approval which is just thumbs up, thumbs down. And then finally al together I want to talk a little bit about the context engineering layer of this system. So
[34:10] remember here we have our first DAG that we just went through. We're also going to have a second DAG and that's this gather decision trace one. Um so that's going to be triggered whenever the first DAG finishes. um that can be through
[34:22] data aware scheduling and that's going to hit the airflow rest API for those human in the loop details. So that's going to pull all of the human decision and reasoning uh that we just talked about collecting so that we can make
[34:34] this uh improve over time. And I'll note that you don't need to fully understand this diagram and if you implement something like this yourself definitely your mileage may vary in terms of what this actually looks like. Um, but the
[34:47] takeaway here is that Airflow exposes a lot of context and you can use that to build, you know, genuinely contextaware pipelines. I'll also throw out there that if you are not familiar with Airflow, not
[35:00] already using it, um, and all of these ingredients that I've talked about sound great, but you're like, I have no idea how to use any of that. Um, you're not alone. So, lack of Airflow literacy across teams was one of the, um, highest
[35:12] cited challenges in the Airflow survey last year. Uh so for that this is another place where astronomer can potentially help out. Um so we provide additional tooling on top of airflow including our data engineering agent um
[35:25] which is called auto. So you can offload some of this DAG authoring to your AI co-workers. Uh again if this is something you're interested in come find me after see us at our booth. Okay, so finally just a recap um of the
[35:39] second recipe and the things that we looked at again that make this more ready for production than our uh first little code snippet. Um we are now implementing agents uh using the task agent decorator in the common AI
[35:51] provider. Uh we've done that using some tool functions. So our agents have lots of functionality uh that we give them so they can be the best at their job. Uh we have human in the loop so we can always have the final say over whether something is going well or not. And then
[36:04] we're gathering that orchestration context um with Airflow so that we can you know have a paper trail and we can make sure this is improving over time. We're not making the same mistakes over and over again.
[36:17] Okay. And then finally I want to bring this back to the mislass concept that I talked about at the beginning. Um so remember this picture um where we said the model is really the small box in the
[36:29] middle. Everything around it is what we've just talked about in these examples. That's what's going to get you into production. Um, and this is what Misen Plus is going to look like for AI. So, Airflow is
[36:41] going to fill out all of the surrounding pieces around the model. So, the data ingestion, the data quality, uh, the orchestration, the observability, the human judgment, and the context. So, the models are going to keep improving on
[36:53] their own. We'll be able to do more and more stuff with them. Um, but the context and operational layer is the part that is going to require intentionality from you so that you can build all of this and be ready before your rush hits.
[37:07] Okay, and that's it for me. I uh I think we're about out of time, but come find me afterwards if you have any questions.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.