A coding agent is an AI agent that reads a codebase, edits files, and runs commands to complete a software task in several steps. Coding agents matter in AI engineering because they act inside a real environment, so you have to measure the work they finish and control what they can touch.
For example, suppose you give a coding agent this task: “The checkout tests fail after the currency change. Make them pass.” The agent searches for the failing tests, reads the changed currency code, and suspects that rounding now happens before tax instead of after.
It edits the calculation and reruns the checkout tests. One test still fails because its expectation encoded the old behavior on purpose. A trustworthy run stops there and flags the conflict for a human instead of editing the test to match its own code.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why coding agents matter
A code assistant returns text for you to apply. A coding agent changes your working tree and runs your project’s own checks to see whether the change worked.
That shift changes what you need to manage:
- Scoping access: Decide which files, commands, and network targets the agent can use, because access defines both what it can help with and what it can harm.
- Defining done: Agree before the run on what counts as finished, usually named tests passing and a change that survives review.
- Debugging runs: Use the record of commands, edits, and test output to see why the agent took a path, not just what it produced.
- Comparing setups: Score the same tasks the same way when you change the model, the instructions, or the harness.
- Keeping humans in the loop: Route changes that are hard to reverse through review before they merge.
How a coding agent works
A coding agent receives a goal, such as fixing a failing test, adding a feature, or refactoring a module. It then works the goal as a loop of action and observation.
The loop usually looks like this:
- Explore: Search the repository to find relevant files.
- Read: Open those files and the tests that cover them.
- Edit: Propose or apply a change.
- Execute: Run the build, tests, or other commands.
- Observe: Read errors and output.
- Revise: Adjust the change and repeat, or stop and report.
This loop is what separates a coding agent from a chat assistant. The output is a changed working tree plus the evidence from running the project’s checks.
The parts under the loop
A model does the reasoning and writes the code. Tool calling connects the model to file operations, search, a terminal, version control, and sometimes a browser or issue tracker.
Standing instructions shape each step. These include repository conventions, build commands, and style rules. They are often supplied through instruction files such as AGENTS.md or through reusable skills.
The surrounding harness manages permissions, checkpoints, and how results are reported back. In most setups the harness also decides which actions need human approval.
What a recorded run looks like
When you trace a coding agent, each model call, file edit, and shell command becomes a span. Together the spans let you reconstruct why the agent chose a path.
Arize’s write-up on tracing open-source coding agents shows this in practice. The companion coding-harness tracing repository contains the instrumentation.
How to measure a coding agent
Measure completed tasks, not produced code. A practical primary measure is the fraction of assigned tasks finished to a standard you set in advance.
A task usually counts as finished when all of these hold:
- Relevant tests pass: The tests named for the task ran and passed.
- No regressions: Unrelated test suites still pass.
- Review passes: A human accepts the change without substantive rework.
- Scope holds: The change stays inside the task.
Supporting measures add detail. Track time from assignment to a reviewable change, the number of human interventions per task, and rework discovered after merge.
Lines of code, number of edits, and the agent’s own claim of success are weak measures. An agent can produce a large, confident diff that fails review.
Use a fixed task set
This is agent evaluation applied to software work. It works best with an evaluation harness: a fixed set of representative tasks, run under the same permissions and instructions, and scored the same way each time.
Include tasks the agent should decline or escalate, not only tasks it should complete. That way you measure restraint instead of assuming it.
A hypothetical scorecard
Suppose you run 40 representative tasks against two instruction sets. Each task is scored against the four conditions above.
| Measure | Instructions A | Instructions B |
|---|---|---|
| Tasks attempted | 40 | 40 |
| Finished to standard | 26 | 31 |
| Completion rate | 65% | 77.5% |
| Tasks where tests were never run | 6 | 1 |
Instructions B finishes 5 more tasks. Most of the gain comes from fewer runs that skipped verification. The “tests never run” row tells you which behavior changed, which a single completion rate would hide.
With only 40 tasks, a gap like this can still reflect noise. Check the uncertainty before you treat it as a real improvement. For more on that, see confidence interval.
How to debug coding agent runs
Debugging starts from the record of the run. Check which files were read before editing, which commands ran, what the tests actually said, and where the stated plan diverged from the actions.
This is everyday debugging of agents, with one advantage. The environment talks back through compilers and test suites.
Classify the failure before you fix it. Each class has a different fix:
- Exploration failure: The agent edited the wrong file or missed context it needed.
- Generation failure: The diagnosis was right, but the edit was bad.
- Verification failure: The agent never ran the tests, or misread their output.
- Scope failure: The agent changed something outside the task, which points to a permissions or instruction problem.
In production use, keep traces across many tasks so patterns surface. You might find one area of the repository that repeatedly misleads exploration, or one instruction that reliably leads to skipped verification.
For a shared vocabulary across runs, see agent failure modes.
Limitations of coding agents
Coding agents fail in ways that tests and benchmarks do not always catch:
- Untested behavior: The agent can pass the tests it ran while breaking behavior no test covers.
- Test tampering: It can edit a test to match its own bug.
- Untrusted instructions: It can follow instructions found in repository content that a human would recognize as untrusted.
- Loops: It can retry variations of a failing approach, using time and budget without progress.
- Oversized changes: It can produce a change too large for meaningful review.
- Benchmark gaps: A strong benchmark score measures performance on that benchmark’s tasks and harness, not reliability on your codebase.
Sandboxing, least-privilege permissions, protected branches, and required review are part of a coding agent’s design. They are not optional extras. For more on isolating execution, see agent sandbox.
Coding agent vs. code autocomplete
Autocomplete predicts the next piece of code inside an editor. It works one insertion at a time, and the human drives.
A coding agent takes a task-level goal. It explores, edits, runs, and revises across several steps, then hands you a finished change for review.
Frequently asked questions
How is a coding agent different from a coding chatbot?
A coding chatbot returns code as text, and you apply it. A coding agent applies the change itself, runs the project’s checks, and revises based on the results.
What should I give a coding agent access to first?
Start narrow. Give it read access to the repository, the ability to edit on a branch, and the commands needed to build and test.
Add deployment, network, or credential access only when you have a measured need. Keep irreversible actions behind human approval.
How do I know a coding agent finished a task?
Check it against the standard you set before the run. The relevant tests ran and passed, the suites you named show no regressions, the scope stayed inside the task, and the change passed human review.
The agent’s own summary is a report to verify. It is not the verification.
Can a coding agent work without human review?
Sometimes, for low-risk changes that are well tested and easy to reverse. Teams that allow this usually add tight permissions and monitoring.
The safer default is review before merge. Grant more autonomy per task type as measured completion and low rework justify it.
Why does my coding agent edit tests instead of fixing code?
Usually because nothing in its instructions or permissions stops it. State in the instructions that tests encode expected behavior, and flag test edits for review when you score runs.
Putting coding agents to work
Before you roll out a coding agent, write down three things: the permissions it has, the instructions it follows, and the standard that defines a finished task.
Build a small fixed task set that includes tasks the agent should escalate. Rerun it whenever you change the model, instructions, or harness.
Trace every run so you can see the files read, commands executed, and test output behind each change. Tools such as Arize AX can collect those traces so you can review failures by class instead of one run at a time.
