Agent architecture is the structural design of the system around the model: which components exist, what each one owns, how they connect, and how state moves between them. It covers the instruction layer, the tool interface, memory and state, retrieval, control flow, permissions, and the points where traces and evaluations attach.
It is a design-time question, not a description of a single run. Architecture says a planner hands a step list to an executor and the executor may call four tools. What happened on last Tuesday’s request is the run.
Most agent reliability work turns out to be architectural. When an agent picks the wrong tool, the fix is usually a narrower tool surface rather than a better prompt. When context goes missing at step six, the fix is a decision about who assembles context and from what. No amount of prompt editing resolves a structure problem.
Key takeaways
- Agent architecture describes structure: components, interfaces, control flow, and the movement of state. Runtime concerns like persistence and recovery sit on top of it.
- The core components are the model, the instruction layer, tool definitions, a state store, control logic, a permissions boundary, and instrumentation points.
- Common shapes trade adaptability for predictability: a single tool-calling loop, plan-then-execute, a router, a supervisor with sub-agents, or a fixed pipeline with one agentic step.
- Tool surface design is the architectural decision that moves accuracy most. Overlapping tool descriptions produce wrong-tool failures that look like model failures.
- Choose where control flow is decided, in code or by the model, deliberately and per step. Most working systems mix both.
The components an architecture has to place
Model layer. Which model handles which step. Routing a cheap model to classification and a stronger one to synthesis is an architectural choice with cost and latency consequences.
Instruction layer. System prompt, role definitions, policy text, and output format rules. Worth versioning, because changing it moves behavior more than most code changes do.
Tool interface. The set of callable functions, their schemas, and their descriptions. This is the agent’s entire action space. If two tools do similar things, or if one tool takes nine optional parameters, you have designed in an error rate.
State. What persists across steps and across sessions: conversation history, scratchpad reasoning, retrieved documents, intermediate results, task status. The architectural question is what is stored and who reads it. The runtime mechanics of persisting and restoring it are a separate topic, usually filed under agent state management.
Control logic. Loop conditions, step budgets, retry rules, error handling, and termination. This is where the difference between a demo and a system lives.
Permissions boundary. Which actions are read-only, which write, which require human approval. Should be enforced in the harness, never by asking the model nicely.
Instrumentation points. Where spans open, what gets recorded on them, and where evaluators run. Decide this late and you get traces showing only that a request went in and an answer came out.
Common architectural shapes
Single tool-calling loop. One model, one set of tools, iterate until done. The simplest thing that works and the right default. Fails on long tasks where context grows past what the model can use well.
Plan-then-execute. A planning step produces a sequence of steps, then an executor runs them. Makes the intended path inspectable before anything runs, and gives you a checkpoint for approval. Brittle when reality diverges from the plan, unless replanning is designed in.
Router. A classifier or small model sends the request to one of several handlers. Cheap and predictable. The failure mode is a misroute the handler cannot detect, so it answers confidently from the wrong place.
Supervisor with sub-agents. A coordinating agent delegates to specialists with their own tools and context. Buys separation of concerns and independent permissions, and costs you handoff fidelity.
Pipeline with an agentic step. Fixed code for everything except one bounded decision. Underrated. Most production systems that work look more like this than like a fully autonomous loop.
A survey of these architectural patterns and where each holds up is more useful than picking one on principle. The shapes are composable, and real systems usually nest one inside another.
Structural decisions that determine reliability
How wide is the action space. Every additional tool adds selection error. Grouping twelve narrow tools into four coherent ones often improves accuracy without touching the model.
Who assembles context. If retrieval, history, and tool output all append to one growing message list with no owner, you will hit a context problem you cannot debug. Naming the component responsible for what the model sees at each step is a real design decision, and the architecture and controls of the harness around the model is where that ownership usually lands.
Where termination is enforced. Model-decided stopping is a suggestion. Code-enforced limits are a guarantee. Production systems need both.
How state moves between components. Passing a full transcript is easy and expensive. Passing a summary is cheap and lossy. Passing structured fields is precise and rigid. Most handoff failures trace back to this choice.
Where evaluation attaches. Scoring only the final output tells you nothing about which component failed. Attaching evaluators to tool selection, retrieval, and the plan gives you an answer.
Agent architecture in classical AI
The phrase predates language models. Earlier AI literature used it for how a perceiving, acting entity is organized: reactive architectures that map percepts straight to actions, deliberative ones that maintain a world model and reason over it, and hybrid designs that layer fast reflexes under slower planning. That tension between reactive and deliberative is the same one behind the modern choice between a tight tool-calling loop and plan-then-execute. What changed is that the reasoning component is now a language model, and the components and tools that make up a modern agent are software engineering concerns rather than knowledge representation ones.
FAQ
What is the difference between agent architecture and an agent framework?
Architecture is the design. A framework is one implementation of it, with opinions about loops, state, and tool registration baked in. You can build the same architecture in two frameworks, or in neither. Picking a framework before deciding the architecture is a common way to inherit someone else’s design by accident.
How many tools should one agent have?
There is no clean number, and anyone quoting one is guessing. The practical signal is selection accuracy: if the agent starts choosing the wrong tool as you add more, you have passed the limit for that model and those descriptions. Consolidating overlapping tools or splitting into specialists with narrower sets both help.
Should the model or the code decide control flow?
Per step, whichever you can defend. Code for anything with a known correct order, compliance requirement, or hard cost ceiling. Model for genuinely open-ended steps. The useful question is not which is better but how much of your path is actually uncertain.
How does agent architecture differ from a multi-agent system?
Architecture is the design of any agent system, single or multiple. A multi-agent system is one architectural choice: several agents as participants. All multi-agent systems have an architecture, and most architectures are single-agent.
Where do you start when redesigning an existing agent?
With traces of real failures, grouped by which component was responsible: wrong tool, missing context, bad plan, unenforced limit. The distribution usually points at one structural problem rather than general quality, and that is the piece to redesign first.
