MLflow vs LangSmith for agent evaluation and tracing

Summary

• MLflow is the open-source AI and ML lifecycle platform stewarded by Databricks: it traces LLM calls and agent workflows with OpenTelemetry-compatible spans, and evaluates agent quality through mlflow.genai.evaluate() with built-in LLM-as-judge scorers.
• LangSmith is a cloud-hosted tracing and evaluation platform by LangChain with broad framework support beyond LangChain itself (OpenAI, Anthropic, CrewAI, Pydantic AI, Vercel AI SDK) and automated production issue detection.
• Both capture spans, support dataset-based evaluation, and enable LLM-as-judge quality scoring. The key structural difference: MLflow is open source and runs as a managed service within the Databricks platform; LangSmith is a hosted SaaS product.
• For teams building on Databricks, MLflow connects agent evaluation to the full platform: Unity Catalog model registry with versioning and governance, managed experiment tracking, Model Serving, and Lakeflow Pipelines.

MLflow vs LangSmith for agent evaluation and tracing

Agent evaluation and tracing have become critical infrastructure for any team shipping LLM-based applications. Both MLflow and LangSmith address the same core problems: capture what an agent does at runtime, score the quality of its outputs, and help teams detect regressions before they reach users. They differ in architecture, licensing, ecosystem fit, and how much of the surrounding ML lifecycle each covers.

What is MLflow?

MLflow is an open-source platform (Apache 2.0 license) for managing the ML and AI lifecycle, stewarded by Databricks. For LLM and agent development, it provides MLflow Tracing for capturing agent runtime behavior and mlflow.genai.evaluate() for systematic quality assessment. On Databricks, MLflow runs as a fully managed service integrated with Unity Catalog, Model Serving, and experiment tracking.

What is LangSmith?

LangSmith is a cloud-hosted tracing and evaluation platform developed by LangChain. It provides full-stack observability for LLM applications: tracing agent runs in production, building evaluation datasets from live traffic, surfacing quality issues via automated detection, and collecting human annotation feedback. LangSmith supports frameworks beyond LangChain itself, including OpenAI, Anthropic, CrewAI, Pydantic AI, and Vercel AI SDK.

MLflow vs LangSmith: how they compare

Capability	MLflow (Databricks)	LangSmith
License and deployment	Open source (Apache 2.0); self-host or Databricks managed	Cloud-hosted service by LangChain
Trace capture	OpenTelemetry-compatible spans for LLM calls, retrieval, tool use	Full trace capture with support for LangChain, OpenAI, Anthropic, CrewAI, Pydantic AI, Vercel AI SDK
Auto-instrumented frameworks	LangChain and OpenAI via Databricks autologging	LangChain, OpenAI, Anthropic, CrewAI, Pydantic AI, and more
Evaluation API	mlflow.genai.evaluate() on trace datasets with built-in and custom scorers	Dataset-based evaluation with LLM judges and annotation queues
Built-in quality judges	Correctness, Completeness, Fluency (extensible with custom scorers)	LLM-as-judge evaluation (specific built-in judges vary by tier)
Production issue detection	Custom scorer-based quality monitoring across versions	LangSmith Engine automatically identifies recurring problems and diagnoses root causes
Model registry	Unity Catalog model registry: versioning, cross-workspace access, lineage, and role-based access control	No native model registry
ML lifecycle coverage	Full Databricks platform: experiments, models, pipelines, serving	Tracing and evaluation focus; external integrations for the rest
Human feedback	Feedback attached to traces via UI or programmatic API	Annotation queues with human reviewer workflows

How Databricks approaches agent evaluation with MLflow

Databricks positions MLflow as the central observation and evaluation surface for agents built on the platform.
MLflow Tracing auto-instruments LangChain and OpenAI applications to capture spans at every layer of an agent run: LLM completions, retrieval steps, tool invocations, and any custom functions you instrument. Each span records inputs, outputs, latency, and token counts. Traces are linked to MLflow experiment runs, so evaluation results sit alongside the training and configuration history of the model. See the MLflow documentation.
mlflow.genai.evaluate() runs systematic quality assessment across a dataset of traces. Built-in judges cover Correctness, Completeness, and Fluency; you add custom scorers for domain-specific criteria. Teams run evaluation over production traffic samples to catch regressions across agent versions. Results are stored as feedback records attached to traces and are visible in the MLflow UI. See Agent evaluation on Databricks.
Unity Catalog model registry governs the models behind your agents: versioned model artifacts with lineage back to training runs, role-based access control, and cross-workspace promotion. When an agent version passes evaluation, the underlying model transitions through the registry. This connects evaluation outcomes directly to deployment decisions.

FAQs

Can MLflow trace agents built outside LangChain?

Yes. MLflow Tracing provides a manual instrumentation API for any Python application and auto-instruments LangChain and OpenAI out of the box on Databricks. Any LLM call or agent step you decorate with @mlflow.trace is captured as a span in the trace tree.

Does LangSmith require using LangChain?

No. LangSmith integrates with OpenAI, Anthropic, CrewAI, Pydantic AI, Vercel AI SDK, and others. It was built by LangChain but operates as a framework-agnostic tracing and evaluation service.

How do custom evaluation scorers work in MLflow?

Custom scorers in mlflow.genai.evaluate() are Python functions that receive a trace (inputs, outputs, context) and return a numeric score or label. You register them once and reuse them across evaluation runs. Built-in judges (Correctness, Completeness, Fluency) handle the most common quality dimensions; custom scorers extend coverage for domain-specific criteria like domain accuracy, safety for a specific use case, or citation quality.

What governance does Databricks add to agent evaluation?

On Databricks, MLflow evaluation runs sit inside the workspace's Unity Catalog governance perimeter. The model versions being evaluated are registered in Unity Catalog with lineage, access control, and audit logs. Evaluation datasets stored in Delta tables inherit the same access policies as other governed data assets. This makes agent evaluation part of the organization's existing data governance posture, not a separate system.

Which tool is better for teams not on Databricks?

MLflow is open source and self-hostable, so it runs on any infrastructure. LangSmith is a hosted service, so teams that prefer not to run infrastructure may find it operationally simpler for the tracing and evaluation layer alone. The tradeoff is that LangSmith does not cover model registry, experiment tracking, or ML pipeline orchestration, so teams relying on it for the full ML lifecycle will need to assemble those from other tools.