Sponsored by: Astronomer | Your AI Strategy Has a Context Problem. Orchestration Solves It.

Summary

Astronomer's Chief Strategy Officer Vikram Koka explains that enterprise AI strategies fail not from a lack of data access but because AI agents lack temporal context—knowing when data was valid—a problem that orchestration systems like Apache Airflow are uniquely positioned to solve.
Apache Airflow is described as the de facto standard for data orchestration with over 3,800 contributors, 1,500 integrations, and 135 discrete systems, and is now used by 32% of surveyed users for ML workloads and a growing share for generative AI applications.
This video introduces a 360-degree agent lifecycle framework and identifies orchestration as the only layer where operational context—which pipelines ran, what changed upstream, and what humans corrected—accumulates continuously as a production record.

Chapters

FAQs

What is the AI context problem and why does data access alone not solve it?

The AI context problem is that agents making decisions based on data need to know not just what the data says but when it was valid, what changed upstream, and what corrections humans made. Simply giving an agent access to a data table does not provide that temporal and operational evidence, meaning agents can confidently act on stale or incorrect data and produce wrong outputs.

How does Apache Airflow provide context that AI agents need?

Airflow is the layer where pipeline execution history accumulates naturally—it records which pipelines ran, what data changed, what failed, and what operators corrected. This video argues that because orchestration systems are the source of this operational record, they are the natural place to harvest the up-to-date context that production AI agents depend on for accurate reasoning.

Why does Astronomer say data engineers are more critical than ever in the agentic era?

This video concludes that while AI has changed the data engineer's role from a bottleneck to a platform provider, the agentic era makes that platform role more critical rather than obsolete. Data engineers own the orchestration layer that generates the operational context agents need, and without well-maintained, trusted pipelines, AI agents will hallucinate or act on stale information.

What are the three knowledge buckets that AI agents need to function accurately?

This video identifies three knowledge buckets: public context (general knowledge the model was trained on), operational context (the live record of what actually happened in pipelines and systems), and semantic context (declared business definitions and metrics). Orchestration systems like Airflow are the primary source of operational context, which is the bucket most often missing in enterprise AI deployments.

Full transcript

[00:08] Hey everyone. Thank you for coming. And just wanted to start with the the part of the the due diligence in the presentation. A lot of these are forward-looking statements, but I would like to kind of walk you through this presentation. So, please do take a a minute at the end
[00:21] of the the session to actually complete your survey. We'd really appreciate feedback and would love to hear from you as part of the any questions, feedback, and comments you may actually have.
[00:33] My name is Vikram Koka. I'm the chief strategy officer at Astronomer, and I'm also on the Airflow project management committee. And I'm here to talk to you about uh AI strategy and how orchestration is
[00:45] related to AI strategy in this particular case with a particular focus on context and why we believe that orchestration is a critical element for solving and enabling AI applications to go to
[00:58] production. Airflow, for those of you who may not be aware, Apache Airflow is the de facto standard for data orchestration on the planet. We're actually now the most popular open source project from ASF. We
[01:12] passed Spark about a year ago, and with respect to number of contributors, we are over 3,800 contributors and with tremendous amount of activity with GitHub comments and the overall community. Now, how does this actually
[01:25] translate to you? The one of the key ways it translates to you as benefits is really the vast ecosystem in which Airflow works. We've got literally 1,500
[01:37] integrations, which manifest themselves as 135 discrete systems to which we integrate across the planet. This includes a large variety of databases, computing frameworks,
[01:50] systems of applications, cloud services, and so on and so forth across the world. And that's the key part of like us being an orchestration for the world's data across across the planet.
[02:03] All right? Now, Airflow has been the standard for data orchestration. We've kind of evolved this over a period of years. And now we're starting to move tremendously into AI, largely as a result of like people as pulling it into
[02:15] this. Now, the community as as the Airflow community, we've been designing for AI. We've had a lot of people starting to adopt Airflow as being the agentic orchestration mechanism for agentic applications. We're seeing 32%
[02:29] of people from the latest survey using it for ML. And actually a significant amount of people using it for generative AI as well. We also see a tremendous number of people based in this conference about over a third of people
[02:42] use Airflow with Databricks and with Spark as well for orchestrating Databricks jobs and Spark workflows. Now, from the context of data and how Airflow
[02:54] is evolved and how the role of a data engineer is evolved. Good old days, all of the requests for data pipelines would actually come in through the data engineer. Somebody would come up say, "Hey, I want this dashboard. I want to
[03:06] update this application." They would work with the data platform team and they would actually deploy to production through the data engineer. The downside, data engineers are always a bottleneck. The upside is once it
[03:18] actually landed in production, it was very reliable and it completely worked because it worked on the happy path of what the data engineering team and the platform team expected to happen with this particular pipeline.
[03:30] Fast forward to today, the role of the data engineer has been absolutely transformed. They've actually made available a system with a data platform, as in data lakes, data warehouses, computing platforms. And people are actually interacting with these using
[03:43] agents. Happens for us in our own company. We have the data team has exposed a complete data warehouse and set of frameworks. A lot of us actually just use cloud or other agent like mechanisms to query the data
[03:56] to query the data warehouse and access it. And absolutely, we think, "Okay, data engineers are no longer a bottleneck. Problem solved. All good to go. Let's go forward." And there's a lot more queries. We actually all celebrate
[04:09] the fact that there's a lot of queries being run, lots of data immediately being available to people. And everybody, all the way from engineering directors, our CEO himself, as well as
[04:21] sales people, are directly accessing the data, using it to get ready for customer calls and all that kind of stuff. The challenge, of course, is yes, access is completely democratized, but is the
[04:33] data actually ready? Are there Is the the conclusions from this, are they actually accurate? The volume is increasing, but has the accuracy really stayed consistent towards this level of
[04:46] volume which has dramatically increased? And the sad truth is, not always. People are actually accessing data, and this is a real example, right? I actually said about how my sales team is
[04:58] using the data directly, getting ready for customer calls. A typical thing is, "Okay, we're chatting with this customer. What is the revenue profile been over the last year? And, you know, do they have any support issues that we should be careful before kind of having
[05:11] a renewal conversation or an upsell conversation?" In this case, this is a real excerpt. They actually came up with, "Hey, this is actually the expansion. They originally landed about 100k, 150k, expanded twice, 50k each,
[05:24] and there's no support tickets in the last month." We think we run a pretty reliable platform, but sadly, that wasn't accurate. They I hallucinated in this case, and it actually missed a certain
[05:36] element of the data because the data wasn't up to date. It wasn't even there. It looked at the wrong field that hadn't yet been updated because that pipeline hadn't yet run for this particular period of time when they're actually
[05:48] done. Real life problem. And the challenge with AI is for all of you may work with it, agents are really confident, right? And they kind of reinforce that level of confidence. They
[06:01] kind of say, "Yep, you're absolutely right." Whereas if you actually typically ask a peer, most of them kind of doubt you at the first case. "Eh, how did he get this? Is it right?" Whereas you're interacting with an agent, AI is wonderful about making you feel really
[06:13] good about the answer you've already come up with and go forward with this. And therefore somebody kind of says, "Uh, nope, not so right." And that fundamentally is what we term a context problem. And the key thing I would actually say
[06:26] context is not semantics. Those are Those are absolutely different. Semantic data is about, "Hey, what is this particular field?" That is not sufficient for an agent to be able to give you the right answer
[06:39] when you need it. Because the key thing here for all of the people who work with the data, data is temporal. Data might be valid at a certain period of time, but it may not be valid for all time, especially when you need it. And that is
[06:53] one of the key things we tend to define as context. Which is, what is the logic behind the metric? And most importantly, is this the right metric to use? Is this the right field to use at this point in time when I actually need it? Is a key
[07:06] element of what we term context. It's like the timeliness and what to look at. Is it stale? What changed upstream? Maybe this is no longer what it actually means and it has evolved. And that is why context is a fundamental issue for
[07:20] making AI accurate to be able to actually use in conclusions going forward. And that is the challenge today. I mean, it's without a doubt it's something which we see every single day going forward.
[07:33] Which is part of the belief of like, you know, agentic systems really need accurate context. And this accurate context is actually a a created and is known to orchestration systems because
[07:45] orchestration systems know when something was actually updated. They actually know when something succeeded and when something failed and therefore is this valid and as to what time is
[07:57] this valid and what is the basis for that conclusion and the data created as a result of all the prior steps which have come to. Almost everyone who's worked in data, you know you're not kind of going directly from a set of data sources and
[08:10] coming up with an answer. You come up with intermediate products, some of which are refreshed at different periods of time to come up with the right conclusion which is valid at that point in time and that is really known to the
[08:22] orchestration system because it knows when those were run, what were run, what is the basis, what is the lineage and to come to this particular conclusion which is actually needed. And that's really
[08:34] what we define and the reason why we believe Airflow as a de facto orchestration standard is why it's relevant. Kind of work across all the different sources, all the different intermediate destinations towards
[08:47] finally getting to the answer which you really want towards the end of that. And that's really why context and orchestration creating context is critical for timely data which is important for accurate analysis
[09:01] for your agent existence going forward. Another real example. And this is a classic element of that again an agent is hallucinate and a LM can hallucinate
[09:14] really well. Actually ask an agent about, "Hey, what is this customer actually running?" We actually tend to know the system. We know what they're running, right? And it came back with an answer which is like, "Hey, they're running some" came up with
[09:26] a relatively generic answer and anyone who knows who's kind of interacted with the agents know that they actually tend to come up with really high-level generic answers very easily, quickly,
[09:38] and often. And then we kind of say, "Hmm, that's actually not what I'm looking for. I'm looking for something a lot more specific. Can you dig into details? Look at the orchestration data. Look at the analysis of the details. What are they
[09:50] running? How frequently are they running? How many inputs are they looking at?" And then you start getting into a lot more nuanced answer. "Hey, they're actually running a multi-tenant system. It is they're running pipelines
[10:03] for each one of their customers. Their backgrounds look clean, but 5% of all their customer data is customer data tags are failing. And they're not actually aware of it, and that's the key kind of the nuance, which
[10:16] is important with agents, because without asking it to be specific, an agent is very good at giving you a broad brush answer, typically which makes you feel good. Like, at least for me, and there is a
[10:31] there's multiple places where I've I've actually been using this in automating my own job. It's very easy to kind of feel like, you know, everything is working just fine without actually diving into the details. And that level of depth, which is what is actually
[10:44] going on, is the key next-level context to which needs to be given to an agent for it to give you answers which are accurate towards actually being insightful and useful, rather than a
[10:56] just a high-level answer. But what if agents could really do it all, right? Because the key thing is, if you give them enough saying, "Here is what I've actually asked you to do from
[11:10] a why." Right? The spec is pretty important. That gives you the analysis authoring towards I want you to come up with this for this particular criteria. I want you to then be able to kind of analyze and how to create that
[11:23] particular pipeline. But then every data engineer knows that there is data drift. It is not what you authored the first time. It is what you author after things
[11:38] go bump many times because incoming data changes. So, the ongoing maintenance is also context, which is important for an agent to understand over time to say how is this changed over time? What issues
[11:51] have we actually fixed? How have we had to investigate these particular issues and fix the validations towards kind of feeding that back to the original analysis of what does this data mean?
[12:03] When is it valid? When is it not valid? And how do I kind of do this complete full circle in being able to resolve problems, which is then when you actually get to an accurate answer, which can stand the test of time because
[12:16] we all know that you can actually deploy it. Two days later, you can say "What did you come up with?" And for anyone who's deployed a agent to production, that is a common problem
[12:28] we've all had. Right? Without a doubt, it's the answer changes over time, and I really believe that AI drift is magnified as a result of data drift. And the way to address that is with context
[12:41] and being able to kind of keep that context up to date as being the way to kind of happen. But, if we could actually build this entire life cycle, give the agent context of what you want, how have you initially authored it, what
[12:54] issues happened, how have you had to fix it, and then finally how have you had to investigate those particular problems, that 360 level life cycle gives the entire context towards giving an accurate picture, which stands a significant
[13:07] level of test of time. Which is really what you want in a repeatable execution for an agent. And this really depends on multiple levels. Yes, you need an
[13:19] agent harness, which is the reasoning leap. But, in addition to that, you need the evaluation step towards how do you actually validate this particular answer which it comes up with? How do you then apply deterministically the values which
[13:33] are created from this evaluation step for the reasoning which therefore makes it four distinct steps. Gather the data, apply the reasoning loop, apply the evaluations, and then
[13:46] run the evaluations, and then finally apply the result of this evaluation as part of your overall agent harness, which is why orchestration is at a higher level than the agent harness, which is really typically the reasoning
[13:58] loop itself. But, this relies on really three different sets of information. One which is public knowledge, which you might say, "Hey, what is the best practices for this particular domain? How do you author pipelines for this overall
[14:11] level?" Second is operational knowledge, which is how is this actually deployed in your own enterprise, in your own organization, and what is all the information needed to kind of say, "Hey, here are common problems." For example,
[14:24] we run pipelines differently for every customer. Running pipelines for one customer, if that fails, should not affect other customers. But, for overall aggregated needs for like, "Hey, how do I report my monthly ARR?"
[14:36] all customer data needs to be valid. So, different cases need to be factored into words that particular dependency matrix, and that is operational knowledge which you would supply the agent for saying,
[14:48] you know, "This is what is timely in this case. This is what is timely and up-to-date in this case, and what error tolerance is okay." And finally, your own knowledge about, "Hey, this is a semantic meaning, and
[15:00] when to use this data for different applications, when not to use this for data for different applications." You might say, "Hey, look, PII data is never to be considered. I want to avoid this list." Some other information from the
[15:12] customer, such as which location, which cloud they're running on, may actually be very valid and very important kind of factor in for financial ARR calculations, discounts, so on and so forth. Everything which kind of comes
[15:24] together towards actually making this happen making this happen. But once you actually have this full cycle towards these three buckets of knowledge, but then the full life 360 about being able to apply this, the ability to create
[15:38] this is absolutely real. And this is something which we've seen from it's a direct quote from one of our customers who actually said, "Look, you know, we actually implemented this for certain set of our pipelines and as a result of
[15:51] that knowledge, we could actually have the agents authored the pipelines going forward for this particular domain." Now, by no means is this am I going to make the same that this applies for all domains, but once you get that 360 level
[16:05] information about running certain set of domains, those can absolutely be updated and be able to kind of really made operationally real for these things to actually apply without a doubt.
[16:19] So, key takeaways. For everyone who actually felt that the data the data engineers job is done because of AI, I absolutely vehemently argue the opposite. I actually think the data engineering role has become even
[16:32] more important because they're not just helping people roll out individual pipelines. They've got a 10x multiple in enabling a lot of pipelines to be done by the entire organization in a very very quick high velocity fashion in
[16:46] being able to deploy these pipelines. So, one as a data architect as being the provider of the fundamental platform for this democratized agent authoring and agent analysis to happen, the data the
[16:59] data engineers role is even more critical than it has ever been before and I very firmly believe that at this stage. The architectural knowledge, the what is updated when, what can be relied
[17:11] at at what time is really that knowledge which is needed and is only known to the data engineer, but then they need to be able to build and share that with the rest of the organization for this to truly be effective.
[17:24] Second is based on just kind of the examples we've seen, based on my personal experience, but honestly, I really want to hear from everybody else as well. Agents without context just leads to chaos.
[17:37] Agents are especially good at this because they're so incredibly confident about actually giving you that answer. saying, you know, absolutely right. And we've all kind of worked with that had to kind of argue back. So, that context
[17:51] which is up-to-date, real, based on your organizational priorities, based on your knowledge, is critical for an agent to kind of be effective in production. versus being in dev. In dev is pretty
[18:03] easy because you can course correct it because you're saying, "Yep, you give me 80% of an of the right answer. I can correct the other 20. I can use this to go forward." But for production, you need this to be right pretty much all the time. And for that, up-to-date
[18:17] context up-to-date is critical for agent not to cause chaos because ultimately a bad answer is worse than no answer at all. With and that's really a key thing in this context to kind of be aware about.
[18:30] Finally, data teams are critical in this journey for agentic adoption. For AI agents to actually be able to be enabled and governed because the key thing here is also the fundamental knowledge of
[18:43] what do you want your AI agent to see from data? How do you actually govern this to make sure that they're looking at only the right data and the right data and be able to kind of govern this going forward. That's basically the key
[18:56] things which we believe very strongly is the kind of the key takeaways from the data engineering role as an evolution of the agentic role. Thank you so much. I'm Vikram Koka. Please do come reach me. I'm happy to
[19:08] take any questions after. And we are at a smaller. Please come find us at booth 500.

Learn more about the Databricks Data and AI platform.

The information provided herein is for general informational purposes only and may not reflect the most current product capabilities or configurations.