A self-improving harness is an agent harness that changes in response to what it observes about its own failures. The scaffolding around the model, meaning context assembly, tool definitions, retry policy, limits, and stopping conditions, is treated as the thing that gets better over time, driven by production traces and eval results rather than by someone remembering to revisit it.
Say the obvious thing first: this is emerging vocabulary describing a bet, not an established practice with agreed mechanics. Very few teams run a fully automated version. Most run a manual version and have no name for it. The term is worth having because it points at something specific and tractable, which is easy to lose in the broader conversation about AI systems that improve themselves.
The specific claim is this. The model is not the part you can change. The harness is. It is ordinary code, it ships on your schedule, it can be reviewed and reverted, and it accounts for most of the behavior users complain about. So if any part of an agent system is going to get systematically better without retraining anything, the harness is the realistic candidate.
Key takeaways
- The unit of improvement is the scaffolding, not the weights: context assembly, tool descriptions, retry behavior, limits, and stopping conditions.
- This is tractable precisely because it is normal software. Every change is diffable, reviewable, and revertible, which is not true of a model update.
- The loop needs three things that most agents lack: traces detailed enough to attribute a failure, a scored measure of whether a change helped, and a version on the harness itself.
- Automation is a spectrum. A human reading grouped failures weekly and editing tool schemas is the same loop with a person as the optimizer.
- The danger is a loop that optimizes against a flawed scorer. Without a held-out set and human review, the harness improves at scoring well rather than at working.
The loop, concretely
Observe. The agent emits traces with enough detail to say which step went wrong and what it was given: the assembled context, the tool called, the arguments, the tool’s response, the step count. Attribution is the hard part and it is upstream of everything else. A failure you cannot localize cannot drive a change.
Attribute. Group failures into categories rather than treating each as a one-off. Twelve sessions where the agent called the wrong tool with overlapping descriptions is one problem, not twelve. This grouping step is what turns telemetry into a work item, and it is the same move behind using telemetry from coding agents to drive changes back into the software.
Propose a change. A tighter tool description that states when not to use it. A stopping condition for the repeated-search case. A context policy that re-injects session constraints each turn instead of trusting conversation history. These are edits to the scaffolding and controls that make up the harness, not to a model.
Measure, then keep or revert. Run the eval set against the old and new harness with the model held constant. If the score does not move outside the noise band, you learned that this was not the bottleneck, which is a real result. Either way, record a harness version that stored eval results carry, or you cannot tell next month whether two numbers are comparable.
Why the harness rather than the model
Three reasons, all about tractability. You can inspect the change: a new tool description is a diff, while nobody can diff a model update, and the effect of one shows up as a shift across every behavior at once. You can revert in minutes, because harness changes ship through the same path as any code change, so the cost of a bad one is bounded by how fast you notice. And it is where the failures are: most agent incidents trace to the scaffolding, meaning truncated context, a tool description that invited the wrong call, a retry with no ceiling, no definition of done. That is the general argument for treating the harness as the main engineering surface of an agent, and it applies with more force once you start trying to improve things automatically, because the harness is the only surface where an automated change is safe to attempt.
How this differs from a self-improving agent
The two terms get used interchangeably and should not be. Draw the line by what changes.
A self-improving agent is usually described in terms of the agent’s own behavior improving: accumulated memory, learned strategies, feedback shaping future decisions, sometimes model updates. The subject of the sentence is the agent.
A self-improving harness makes the scaffolding the subject. The agent behaves differently because the code around it changed, not because it learned anything. The distinction matters operationally: harness changes are visible in a pull request, and agent-level learning is visible only as behavior drift you have to detect after the fact.
Closed-loop agents describe the plumbing that connects production signal back to a change, which is the mechanism both of the above depend on. A self-improving harness is one thing you can put on the end of that loop, and it is the least risky one, because the change lands in reviewable code.
Where it goes wrong
Optimizing against a broken scorer. If the judge rewards verbosity, an automated loop will discover verbosity. Any loop that edits the harness based on scores needs a held-out set the loop never optimizes against, plus periodic human review of what changed and why.
Overfitting to recent failures. A harness tuned on last week’s incidents can get worse at cases nobody complained about. Keep old cases in the set permanently; they are the regression guard. And keep every automated edit versioned and attributable to the observation that motivated it, or you end up with a system nobody can explain.
Confusing activity with progress. A loop that makes a change every day is not evidence of improvement. Most proposed changes should fail to move the score, and a loop where everything helps is usually measuring itself.
FAQ
Does a self-improving harness require automation?
No, and starting manual is the sensible path. Read grouped failures on a cadence, propose one harness change, measure it against a fixed eval set, keep or revert. That is the whole loop with a human as the optimizer. Automation changes who proposes the change, not the structure, and it raises the requirements on your eval set considerably.
How is this different from prompt optimization?
Prompt optimization improves the wording of a prompt. A harness includes the prompts and also the tool schemas, the context assembly policy, the retry rules, the limits, and the stopping conditions. Many agent failures cannot be fixed by any wording, which is why the larger surface is the useful unit.
Can the harness improve itself without human review?
Parts of it, with guardrails. Narrow, reversible changes measured against a held-out set are a reasonable place to allow automation. Anything touching permissions, spend limits, or which actions require human confirmation should not be automated, because those are the controls that bound the damage when something else in the loop is wrong.
How do I know the harness got better and not just different?
Hold the model fixed, run the same eval set against both harness versions, and repeat the run enough times to see the noise band before calling a difference real. Then check the slices, not the aggregate. A change that lifts the mean while degrading your highest-risk segment is a regression wearing a better average.
