What is AI harness engineering?

Summary

• AI harness engineering is designing everything in an AI agent except the model (the loop, tools, state, checks, and controls) so the agent works reliably in production.
• Databricks documentation defines the framework or harness as the agent code that runs the loop of reasoning, tool calls, and responses.
• Reliable harnesses plan for interruption: they persist run state, leave progress artifacts for the next session, and make repeated tool calls safe.
• Something other than the generator checks the work, and people approve consequential actions.
• On Databricks, Agent Bricks accepts any framework or harness and any model, with managed layers for serving, state, tracing, and governance.

What is AI harness engineering?

In common usage, the harness is everything in an AI agent except the model itself, and AI harness engineering is the practice of designing that system so an agent can do useful, reliable work. The harness runs the loop that calls tools, observes results, and decides when to respond, and it owns the context, state, permissions, and checks around that loop. Databricks documentation uses the term for the agent code itself: "The framework or harness is your agent code. It runs the loop that reasons with a model, calls tools, and decides when to respond."

What components make up a production agent harness?

The table maps each component of a production agent to its documented Databricks counterpart.

Layer	What it does	On Databricks
Framework or harness	Runs the agent loop	Your code, using a framework such as LangGraph or the OpenAI Agents SDK, or a meta-harness such as Omnigent (Beta)
Model access	Connects the loop to a model that reasons about each step	Unity Gateway, one API for frontier and open models, so you can switch models without changing agent code
Tools	Retrieve data or change another system	MCP servers, APIs, and sandboxes, including Databricks Sandbox (Beta)
Agent state	Holds the current session and memory across sessions	Managed agent sessions and managed agent memory (both Beta, backed by Lakebase)
Agent server	Wraps the loop in an HTTP server and handles durability	DurableAgentServer
Agent runtime	Provides hosting, identity, and scaling	Agent Runtime on Databricks Apps
Observability	Records each step and measures output quality	MLflow Tracing and evaluation
Governance	Controls which models, tools, and data the agent can use	Unity Catalog and Unity Gateway

What design principles make a harness reliable?

Define done before adding autonomy. Agree on what the agent should do and how you will measure quality. In one published pattern for long-running coding agents, the first session writes a feature list in which every item starts as failing, and an item is marked passing only after careful testing.
Make work resumable. A long task can span several context windows. That pattern also sets up the environment in the first session, then has each later session make incremental progress and leave a progress log and version-control history for the next one. On Databricks, a deployed DurableAgentServer keeps run status, events, and results through worker restarts, and with a recovery handler it starts a replacement attempt for an interrupted run. Recovery can run your agent code more than once, so make calls to external systems idempotent.
Verify with something other than the generator. Agents asked to evaluate their own work tend to praise it, and a standalone evaluator can be tuned to be skeptical, which is easier than making a generator critical of its own output. In a published planner, generator, and evaluator design, the planner expands a short prompt into a product spec, the generator builds against it, and the evaluator exercises the running application. That evaluator is worth its cost when the task sits beyond what the model does reliably alone. On Databricks, the counterpart is a scorer, either an LLM judge or a code-based check, run against datasets built from real failures.
Keep people at consequential steps. Agents can pause for human feedback at checkpoints, and stopping conditions such as a maximum number of iterations maintain control. An agent served by DurableAgentServer can pause for human input, such as approval of a tool call. Its invocation status becomes interrupted, and a new invocation with the same session_id and a resume field continues the run.
Add complexity from observed failures. Databricks guidance for an initial agent is to start with the least complex design that works. Then trace the agent, find an issue, write a scorer for it, fix the agent, re-run the evaluation, and monitor production traffic.

How does Databricks support harness engineering?

Agent Bricks is the Databricks agent developer platform for agents built with any framework or harness and any model. The Agent Bricks CLI (Beta) scaffolds an agent project from a framework template, and agentbricks deploy provisions the resources the agent needs and deploys it to a stable, authenticated endpoint on Agent Runtime. MLflow Tracing records each step, and you can store and govern traces at scale in Unity Catalog.

FAQs

Is harness engineering the same as prompt engineering?

No. Prompt engineering writes the instructions the model receives. Harness engineering also covers the tools the model can call, the state it keeps, the checks on its output, and the controls around it.

Which framework or harness should I start with on Databricks?

Databricks documents three starting points. Use a template for a new agent: the Agent Bricks CLI scaffolds projects for LangGraph and the OpenAI Agents SDK. Bring an agent you already built with either one. Or use a meta-harness such as Omnigent (Beta) to compose several harnesses, such as coding agents, into one agent. If you need custom endpoints, request formats, or protocols, the documentation points to your own agent server.

How do I know whether my harness is working?

Trace every run, then evaluate. A trace records the input, the final output, and each intermediate step with its inputs, outputs, latency, token usage, and cost. Turn representative failures into evaluation datasets, and run scorers on live traffic to catch regressions.