SWE-bench is a benchmark that tests whether a language model or coding agent can fix real software issues taken from GitHub. It is one of the most cited ways to compare coding agents, so knowing what a SWE-bench score does and does not mean helps you read vendor claims and plan your own evaluations.
For example, suppose your team is choosing between two coding agents. You run both on the same pinned SWE-bench subset of 100 tasks, with the same harness release and one attempt per task. Agent A resolves 80 tasks. You then review the 20 unresolved tasks by hand and sort them by failure type, because that review tells you more than the headline 80%.
The numbers in that example are hypothetical. This entry deliberately states no real model scores, because any score printed here would be out of date and tied to an unnamed setup by the time you read it.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why SWE-bench matters for coding agents
Most coding benchmarks test small, self-contained functions. SWE-bench tests something closer to real work: reading an issue, finding the right place in an existing codebase, and producing a patch that passes the project’s tests.
SWE-bench is useful for:
- Shortlisting models: Narrow a long list of models or agents to a few worth testing on your own code.
- Comparing agent scaffolds: Hold the model fixed and measure how much the surrounding agent harness changes results.
- Tracking progress over time: Rerun a pinned setup after a model or prompt change to see whether resolution improved.
- Studying failure types: Review unresolved tasks to learn where an agent breaks down.
A SWE-bench score is a controlled comparison. It is not a certificate that an agent is ready for your production codebase.
For more on agents that write and change code, see coding agent.
How SWE-bench works
The original benchmark, described at the official SWE-bench site, builds tasks from real GitHub issues and the merged pull requests that fixed them. The tasks come from Python repositories.
Each task gives the system two inputs: the issue description and a snapshot of the codebase from before the fix. The system must produce a patch.
How a task is judged
Each task keeps two sets of tests:
- Fail-to-pass tests: Tests that failed before the fix and should pass once the issue is resolved.
- Pass-to-pass tests: Tests that passed before the fix and must keep passing.
The harness applies the submitted patch in a prepared environment and runs both sets. The task counts as resolved only when every fail-to-pass test now passes and every pass-to-pass test still passes.
That test-based check makes the benchmark reproducible. It also limits what a score means. It verifies only the behavior the tests cover, in that repository snapshot, in that environment.
A sketch of the resolution rule
The following code shows the resolution rule in simplified form. It takes test results after the patch and returns whether the task is resolved. Real harnesses also build environments and parse test logs.
def is_resolved(fail_to_pass, pass_to_pass):
"""Each argument maps a test name to True if it passed after the patch."""
return all(fail_to_pass.values()) and all(pass_to_pass.values())
fixed_issue = {"test_parse_empty_header": True}
broke_nothing = {"test_parse_basic": True, "test_parse_unicode": True}
broke_something = {"test_parse_basic": True, "test_parse_unicode": False}
print("fixed issue, broke nothing: ", is_resolved(fixed_issue, broke_nothing))
print("fixed issue, broke a test: ", is_resolved(fixed_issue, broke_something))
print("issue still failing: ", is_resolved({"test_parse_empty_header": False}, broke_nothing))
Output:
fixed issue, broke nothing: True
fixed issue, broke a test: False
issue still failing: False
A patch that fixes the issue but breaks one covered test does not count. That rule is strict on purpose.
Variants and dataset releases
SWE-bench has grown into a family. Named variants and subsets differ in which repositories they draw from, how tasks are selected, whether humans verified them, and which modalities they include.
Names, task counts, and contents differ by release. The maintainers also revise datasets over time, including removing or fixing tasks found to be flawed. A result quoted without the variant and dataset date cannot be reproduced.
How to run and report a SWE-bench evaluation
To make a result comparable, pin every part of the setup and record it with the run.
Record these for every run:
- Dataset: The variant name and its version or access date.
- Harness: The release of the evaluation code used to run it.
- Model: The model and its settings.
- Scaffold: The agent harness around the model, including its tools and prompts.
- Attempt budget: The number of attempts allowed per task.
- Run date: When the run was executed.
A run configuration might look like this:
dataset: <variant name>, version/date pinned in the run config
harness: <evaluation harness release>, model settings recorded per run
attempts: 1 per task, no human intervention
result: resolved share reported per agent, with the unresolved tasks listed
follow-up: the 20 unresolved tasks are reviewed by hand for failure type
Report the resolved share with this pin list, plus cost and time per task. Keep the per-task outcomes so you can later break results down by repository or failure type.
The attempt budget matters. A single-attempt protocol and a multi-attempt protocol answer different questions, and mixing them in one comparison makes it invalid. For more on multi-attempt scoring, see pass@k. For the run discipline itself, see evaluation harness.
Worked example: reviewing unresolved tasks
Return to the hypothetical comparison from the opening. Agent A resolved 80 of 100 tasks. Suppose your hand review of the 20 unresolved tasks finds this:
| Failure type | Tasks | What it suggests |
|---|---|---|
| Patch was wrong | 11 | The agent misread the issue or fixed the wrong code |
| Fixed the issue but broke a covered test | 5 | The agent needs to run the existing tests before submitting |
| Environment or test problem | 4 | The harness or task needs attention, not the agent |
| Total | 20 |
These are three failures with three different fixes. A headline score of 80% hides all of that.
Review a sample of resolved patches too. A patch can pass the designated tests while hard-coding a value, weakening a check it was allowed to touch, or fixing the symptom in a way a maintainer would reject.
For more on connecting benchmark results to real work, see benchmark vs. production evaluation.
How to compare SWE-bench scores
The most common mistake is comparing two published scores from different pages. If the variant, dataset date, and harness don’t match, the gap between the scores may not reflect the models at all.
Before you treat a difference as real, check that both scores share:
- The same variant and dataset version.
- The same harness release.
- The same scaffold, or a clearly stated difference in scaffold.
- The same attempt budget.
- Run dates that fall after the same dataset revision.
Treat unmatched scores as marketing, not measurement.
Limitations
SWE-bench is a useful instrument, but it measures a narrow slice of software work.
- Training data contamination: Public benchmark tasks can end up in model training data. Older tasks may partly measure memorization. Dataset revisions and newer variants exist partly for this reason. See data leakage.
- Test-defined success: Resolution can reward narrow patches and miss regressions outside the covered tests.
- Environment sensitivity: A harness or dependency change can move a score with no model change at all.
- Mixed protocols: Single-attempt and multi-attempt results answer different questions and should not appear in one comparison.
- Stale releases: A score from an older dataset release does not describe behavior on the current one.
SWE-bench vs. production coding quality
SWE-bench tasks are self-contained issues in public Python repositories, with a test-defined finish line. Production work is messier.
Real tickets are often ambiguous. Private codebases have their own conventions. Changes go through review, infrastructure can be flaky, and many changes have no existing test that decides whether they are correct.
A system can lead a benchmark and still produce patches your reviewers reject for style, scope, or risk. Use SWE-bench to shortlist, then evaluate on tasks from your own repositories with agent-native evaluation and an evaluation dataset drawn from real work.
Frequently asked questions
What does a SWE-bench score actually mean?
A SWE-bench score is the share of tasks in a specific dataset release whose patches passed the fail-to-pass tests without breaking the pass-to-pass tests. That result holds for a specific harness, model, scaffold, and attempt budget. Nothing more general follows without further evidence.
Why do two leaderboards show different SWE-bench scores for the same model?
Usually the variant, dataset version, harness, scaffold, or attempt budget differs. The runs may also have happened on different dates after a dataset revision. Match the pin list before treating the gap as real.
Should I build a separate evaluation for each SWE-bench version?
No. Keep one benchmark practice and pin the version inside each run’s record. A new release is a new measurement condition, not a new concept.
Can SWE-bench replace testing on our own codebase?
No. SWE-bench can shortlist candidate models or scaffolds cheaply. Adoption decisions need tasks from your repositories, with your tests and review standards as the criteria.
Is SWE-bench only for Python?
The original benchmark draws its tasks from Python repositories. Later variants differ in repositories and modality, so check the variant’s own documentation for what it covers.
Putting SWE-bench to work
When you run SWE-bench, record the pin list with every result: variant, dataset version or date, harness release, model and settings, scaffold, attempt budget, and run date. Keep per-task outcomes, not just the headline share.
Review the unresolved tasks and a sample of the resolved ones. Sort failures by type so each one points to a specific fix.
Then move to your own code. Trace your coding agent’s runs on real tickets so you can see each tool call and patch attempt. Arize Phoenix can capture those traces and attach evaluation results to them. For how benchmarks fit into a wider program, see Arize’s LLM evaluation resource and the LLM evaluation glossary entry.
