What Is An Agent?

An agent is a software system in which a model decides what to do next, not just what to say. It reads a goal, selects an action, calls a tool, observes the result, and uses that result to choose the next action. It repeats that cycle until a stopping condition is met. Around the model sits the code that makes this possible: instructions, tool definitions, memory, permissions, retries, and the loop itself. That surrounding system is often called the harness.

The distinction that matters is against two neighbors. A single model call takes an input and returns text. Whatever happens next is decided by your application code. A fixed pipeline calls a model, then a retriever, then a model again, in an order you wrote in advance. An agent is the case where the order is not fixed in advance, because the model itself chooses each next step from the options you gave it.

Be warned that the industry uses this word loosely. A prompt chain with one tool call attached is routinely marketed as an agent. So is a chatbot with a search button. The useful test is not how autonomous the marketing sounds, it is whether control flow depends on model output at runtime. If you can draw the execution path before the request arrives, you have a pipeline with a model in it.

Key takeaways

• An agent is defined by control flow: the model chooses the next action, calls tools, observes results, and continues until a stopping condition fires.
• The loop, not the model, is what you debug. A trace of an agent run is a sequence of decisions and observations, and most failures are visible as a wrong tool, a bad argument, or a loop that never terminates.
• Everything around the model matters as much as the model: tool definitions, instruction text, how context gets assembled, and what the agent is allowed to do.
• A single model call and a fixed pipeline are not agents, even when both call tools, because the execution path is decided by your code rather than by the model.
• Agents fail in ways single calls do not, including repeated identical tool calls, silent partial success, and correct answers reached by an unsafe or expensive path.

What the loop actually looks like

Strip away framework vocabulary and an agent run is a short cycle repeated a handful of times:

1. Assemble context. The instructions, the user goal, tool schemas, and whatever history or retrieved material the harness decides to include.
2. Model decides. The model returns either a final answer or a request to call a tool with specific arguments.
3. Execute. The harness runs the tool, enforces permissions, and captures whatever comes back, including errors.
4. Observe. The tool result is appended to the context.
5. Check for stop. Answer produced, step limit hit, budget exhausted, error policy triggered, or a human takes over. Otherwise, go back to step one.

In a trace, this shows up as a root span for the run with alternating LLM and tool spans beneath it. Reading those spans in order tells you what the agent believed at each decision point. The structure of an agent and the pieces it needs is mostly a question of what goes into step one and what options exist at step two.

The parts around the model

• Instructions. The system prompt and any policy text. This is where most behavioral tuning actually happens.
• Tools. Functions the model can call, described by a schema. The names, the descriptions, and the argument shapes are prompt surface, not just plumbing.
• State. What the agent carries between steps: conversation history, scratchpad notes, retrieved documents, intermediate results. How that state is stored, trimmed, and persisted is its own discipline.
• Control logic. Step limits, retry rules, error handling, and the stopping conditions. Usually plain code, and usually where runaway behavior gets contained.
• Permissions. What the agent may touch, and which actions require approval.

Reliability work concentrates in these pieces rather than in model choice, which is the argument behind treating the harness as an engineering surface with its own tests and its own failure budget.

How agents fail

Model quality is only one input. The failures that show up in production are mostly system failures:

• Wrong tool. The model picks a plausible tool that cannot answer the question, often because two tool descriptions overlap.
• Bad arguments. Right tool, malformed or hallucinated parameters. A date that does not exist, an ID from an earlier turn, a field the API does not have.
• Context loss. The relevant detail was three steps back and got trimmed out of the window.
• Looping. The same call repeated because the result was empty and nothing in the logic treats an empty result as terminal.
• Right answer, wrong path. The output is correct and the run took nine tool calls and forty seconds when it should have taken two. Output-only checks never catch this.

Because the path varies between runs, evaluating an agent means scoring the trajectory as well as the final answer. That is the practical case for tracing and evaluating agents together rather than treating an agent as a function you can unit test on inputs and outputs.

FAQ

Is an LLM with one tool call an agent?

Usually not, in any useful sense. If your code calls the model, sees a tool request, runs it, calls the model once more, and returns, the path is fixed and there is no loop. The label is not worth arguing about, but the distinction is: a system with a fixed path can be tested like normal software, and a system where the model picks the path cannot.

What is the difference between an agent and a chatbot?

A chatbot produces responses. An agent takes actions with effects outside the conversation: writing to a database, filing a ticket, sending a message, running code. Many products are both. The moment side effects enter, permissions and stopping conditions stop being optional.

What is an agent harness?

The harness is the runtime around the model: the loop, tool execution, context assembly, memory, permissions, retries, and safety checks. Two teams can run the same model on the same task and get very different reliability, and the harness is usually the reason.

How autonomous does something have to be to count?

There is no threshold anyone agrees on, and claiming one would be inventing precision that does not exist. Autonomy is a range, from an agent that must ask before every write to one that runs for an hour unattended. Describe where your system sits rather than reaching for the label.

How do you know when an agent should stop?

You decide, explicitly. Common stopping conditions are a produced final answer, a maximum number of steps, a token or dollar budget, a wall-clock timeout, repeated identical tool calls, or an unrecoverable tool error. An agent without an enforced stop condition eventually finds a way to run forever.