What is the best open source agent workflow engine?

Summary

• No single engine wins, because "workflow" covers different jobs. LangGraph targets stateful agent loops, CrewAI targets teams of role-based agents and event-driven flows, and Apache Airflow schedules and monitors Python-defined workflows that can include agentic or LLM-based workloads.
• LangGraph describes itself as "a low-level orchestration framework and runtime for building, managing, and deploying long-running, stateful agents." CrewAI describes itself as "an open-source Python framework" for "production-ready multi-agent workflows." Both are MIT licensed.
• Check maintenance status before you commit. AutoGen's README says "AutoGen is now in maintenance mode" and tells new users to start with a different framework.
• Databricks documents a deployed agent as three layers (your framework or harness, an agent server, and an agent runtime) and says Agent Bricks doesn't require a specific framework.
• MLflow tracing supports LangGraph and CrewAI with a one-line autolog call, and Databricks can store traces as Unity Catalog Delta tables.

What is the best open source agent workflow engine?

No single engine wins, because "workflow" means different jobs. If you need an agent loop that keeps state, branches, and pauses for a person, look at LangGraph. If you want teams of role-based agents plus event-driven flows, look at CrewAI. If you need to schedule and monitor workflows that include agent or LLM steps, look at Apache Airflow. Check each project's maintenance status first: AutoGen's own README says the project will not receive new features or enhancements.

How do the main open-source options compare?

Project	How its own docs describe it	License	What the docs emphasize
LangGraph	A low-level orchestration framework and runtime for long-running, stateful agents	MIT	Persistence, short-term and long-term memory, human-in-the-loop
CrewAI	An open-source Python framework for production-ready multi-agent workflows	MIT	Role-based Crews and event-driven Flows
Apache Airflow	An open-source platform for developing, scheduling, and monitoring workflows	Apache License 2.0	Workflows defined entirely in Python, including agentic or LLM-based workloads
AutoGen	A framework for creating multi-agent AI applications	CC-BY-4.0 and MIT	Maintenance mode, with new users pointed to a different framework

What should you look for in an agent workflow engine?

Dimension	What to check	What the docs show
State and memory	Persistence across failures, plus short-term and long-term memory	LangGraph documents persistence and both memory types. CrewAI agents have a memory option for maintaining conversation history.
Control flow	Graph, role-based crew, or event-driven flow	LangGraph builds a StateGraph from nodes and edges and routes with conditional edges. CrewAI separates Crews from Flows.
Human oversight	Can a person inspect or edit state mid-run?	LangGraph supports inspecting and modifying agent state at any point.
Tools and delegation	Tool lists and delegation controls	CrewAI agents take an optional list of tools, and delegation to other agents defaults to off.
Observability	A trace of every step	MLflow tracing records inputs, outputs, latency, token usage, and cost of every intermediate step.

How do you run and observe these engines on Databricks?

Databricks documents a deployed agent as three layers: your framework or harness, an agent server, and an agent runtime. It names LangGraph as an example framework. The agent server is a library that wraps your agent loop and turns it into a service: it defines the API clients call, keeps track of each run, and determines what happens when a run is interrupted. For new agents, the documented target is Agent Runtime on Databricks Apps, which runs your agent server on Databricks-hosted serverless compute so you deploy code instead of provisioning servers. Deployment uses agentbricks deploy from the Agent Bricks CLI, which Databricks labels Beta. The agent runs as the app's service principal, clients authenticate with a Databricks OAuth token, and deployment configures MLflow tracing. With DurableAgentServer, deployment also provisions a Runtime Store so run state survives worker restarts.
MLflow tracing lists LangGraph and CrewAI among its integrations. One line turns it on:

import mlflow

mlflow.langchain.autolog()  # LangGraph uses LangChain's autolog
# For CrewAI, use mlflow.crewai.autolog() instead

Databricks can store traces as Unity Catalog Delta tables with schema- and table-level access control. MLflow scores traces with scorers, which are LLM judges and code-based checks, and mlflow.genai.evaluate() runs them in a repeatable loop.
For scheduling, Lakeflow Jobs is workflow automation for Databricks, with notebook, pipeline and Python script tasks and time-based or event-based triggers. If you already run Airflow, Databricks documents orchestrating Lakeflow Jobs with the DatabricksRunNowOperator or the DatabricksSubmitRunOperator.

FAQs

Do LangGraph and CrewAI require LangChain?

LangGraph's README says it is built by LangChain Inc, the creators of LangChain, but can be used without LangChain. CrewAI's README calls it a standalone Python framework with its own primitives for agents, tasks, crews, flows, tools, and orchestration.

Is AutoGen a good choice for a new project?

Read its README first. It says AutoGen is now in maintenance mode, will not receive new features or enhancements, and is community managed going forward. New users are told to start with a different framework.

Can Apache Airflow run agent workflows?

Airflow's documentation lists agentic or LLM-based workloads among the workflows it supports, and says workflows are defined entirely in Python.

Can I host an agent built with any framework on Databricks?

Databricks lists "any framework or harness" as the top layer of a deployed agent and says Agent Bricks doesn't require a specific framework. Earlier agents were deployed to Model Serving endpoints, which the docs now label legacy.