Agent orchestration is the coordination layer that decides what runs next. It routes a request to the right handler, sequences steps, delegates work to sub-agents, applies retries, and enforces the conditions under which everything stops. It applies both within a single agent, deciding which step follows a tool result, and across several agents, deciding which one gets the task.
Orchestration is the act of coordinating. It is worth separating from two terms it gets swapped with. A workflow is the specified path being followed. A multi-agent system is the set of participants doing the work. You can orchestrate a single agent through a fixed workflow with no second agent anywhere, and that is a perfectly normal system.
The reason it deserves its own name is accountability. When an agent produces a wrong answer, someone has to determine whether the prompt was bad, the tool returned garbage, retrieval missed, the wrong specialist was picked, or the loop simply never terminated. Those are different bugs with different fixes, and only the last two belong to orchestration. Systems where the coordination layer is scattered across callbacks and conditionals make that question unanswerable.
Key takeaways
- Orchestration answers one question at every point in a run: what executes next, and who decided.
- The decision can come from code or from the model, and the mix is a deliberate design choice rather than a framework default you inherit.
- Routing, sequencing, delegation, retries, and termination are the five things a coordination layer owns. Termination is the one teams skip and later regret.
- Orchestration is the act of coordinating; a workflow is the path; a multi-agent system is the set of participants. Keeping these separate makes failures attributable.
- If your traces do not show which component chose the next step, you cannot debug coordination failures, only guess at them.
Who decides what runs next
Every orchestration design sits somewhere on a line between two extremes.
Code-decided. A conditional, a state machine, or a graph you wrote. The path is knowable before the request arrives. Predictable, testable, cheap, and blind to anything you did not anticipate.
Model-decided. The model chooses the next tool, the next sub-agent, or that it is done. Adapts to inputs you never planned for, and introduces a decision you cannot unit test.
Working systems mix these. A common and sensible pattern is code-decided at the top level, where compliance and cost live, and model-decided inside a bounded step. The mistake is not choosing either one, it is not knowing which you have. A graph framework whose edges are all conditioned on model output is model-decided orchestration wearing a diagram.
The five responsibilities
Routing. Sending a request to the right handler, agent, or tool. Usually a classifier, a small model, or rules. Misroutes are quiet, because the wrong handler answers confidently rather than raising an error.
Sequencing. Deciding order, and what can run in parallel. Parallel fan-out cuts latency and complicates error handling, since you now need a policy for partial failure.
Delegation. Handing a subtask to a sub-agent with its own tools and context, then deciding what comes back. The interface between the two is where context goes missing.
Retries and error policy. What happens on a tool timeout, a malformed argument, an empty result. Retrying an identical call that failed deterministically is how loops turn into bills. Distinguish transient failures from real ones before you retry.
Termination. The stopping conditions: answer produced, step budget, token or dollar budget, wall-clock limit, repeated identical calls, human takeover. Enforce these in code. A model asked to decide whether it is finished is a model with an opinion, not a guarantee.
Frameworks, runtimes, and what you still own
Orchestration frameworks give you a vocabulary for graphs, handoffs, and state, and a runtime gives you execution, concurrency, and recovery. Both are useful and neither decides your policy. Routing rules, delegation boundaries, retry semantics, and budgets are still yours to specify. The workflows and architectures behind production agents matters most when something breaks, because the abstraction layer that made the code concise is also the layer hiding which step ran and why.
The corollary is instrumentation. Coordination failures are only visible if the trace records the decision, not just the outcome. A useful span for a routing step includes the candidates considered and the one chosen. A useful delegation span carries the context that crossed the boundary. Without those, a run that took eleven steps looks identical to one that took three, and you are left inspecting the final answer. This is the argument for treating coordination as something you observe in production rather than something you reason about from the code, and for testing the failure modes coordination introduces.
Where orchestration goes wrong
- No owner for termination. Each component assumes another one will stop the run. Nothing does.
- Retry storms. A failing dependency, a retry policy with no backoff and no cap, and a step budget large enough to let it run.
- Silent misroutes. The billing agent answers a security question. Nothing errored, so nothing alerted.
- Lossy handoffs. The sub-agent gets a one-line summary of a five-turn conversation and solves a subtly different problem.
- Duplicated work. Two branches retrieve the same documents and both pay for it, which is easy to miss until you look at cost per session.
- Over-orchestration. Twenty nodes of coordination around a task that one loop with three tools handled fine. The complexity is real, the benefit is not.
FAQ
What is the difference between agent orchestration and a workflow engine?
A workflow engine executes a defined graph of steps, usually deterministic and durable. Agent orchestration includes that, and adds steps whose next move is chosen by a model at runtime. Many teams run agents on top of a workflow engine specifically to get retries and recovery for free while keeping model-decided branches inside individual steps.
Do I need orchestration for a single agent?
Yes, though it may be twenty lines of code. A single agent still needs sequencing, a retry policy, and termination conditions. Orchestration becomes a named layer when the number of components grows, not when the second agent appears.
Should the orchestrator be an LLM?
Only where the routing decision is genuinely open-ended. A model as router costs a call, adds latency, and can misroute in ways a rule cannot. If the categories are known and stable, a classifier or plain rules are more accurate and easier to test. Reserve model-decided routing for cases where you cannot enumerate the options.
How do you evaluate orchestration separately from the agent?
Score the coordination decisions directly. Was the chosen route correct for the request. Did the handoff carry the information the sub-agent needed. Did the run terminate for the intended reason. These are trajectory-level checks, and they catch problems that a correct final answer would otherwise hide.
What does good orchestration look like in a trace?
Every decision point appears as a span with the inputs it saw and the choice it made, the path is reconstructible from the spans alone, and the reason the run ended is recorded. If you can answer “why did this step run” by reading the trace, without opening the source, the orchestration layer is doing its job.
