AI Coding Agent Benchmark 2026: Which Agents Finish Repository Tasks Reliably?

Published on August 20, 2026 by Steven Jones

A useful coding agent benchmark should answer a harder question than “Can this model solve a GitHub issue?” It should show whether a deployed agent can take on an unfamiliar repository task, produce an acceptable diff, recover from its own mistakes, and finish without turning one ticket into three rounds of human repair.

That means testing the entire agent configuration: product, model, reasoning settings, repository instructions, permissions, tools, and execution limits. For this 2026 benchmark framework, the important outcomes are accepted task rate, first-pass success, repeated-run reliability, retries, elapsed time, review burden and effective cost per accepted task. Bug fixes, constrained refactors, test generation, dependency upgrades, and multi-file features should all be represented because agents often look very different once work extends beyond a single obvious failing function.

Benchmark question	What should count	What should not count as success
Did the agent finish?	Hidden verifier passes, and the diff is acceptable without manual code edits	The agent says “done”, or its own visible tests pass
Was it reliable?	Repeated clean runs converge on accepted solutions	One impressive run selected from several failures
Was it efficient?	Total wall time, agent steps, test runs and cost across failed attempts	Token price or subscription price in isolation
Was the change reviewable?	Required behaviour changes with limited unrelated churn	A passing diff that disables checks, rewrites unrelated code or edits the grader

Quick verdict: there is no defensible universal winner from the public leaderboards alone. Claude Code and Codex CLI both deserve a place in a serious repository benchmark, while editor-native agents such as Cursor should be tested in their actual product workflow rather than reduced to the model behind them. The agent that wins for a team is the one that produces the most accepted work for the lowest combined retry, review, and compute cost across that team’s task mix.

Current public benchmark evidence points to a shortlist, not a winner

The table below is an evidence snapshot from 20 August 2026. These rows are useful signals, but they are not a head-to-head league table because the tasks, harnesses and experimental units differ.

Evidence	Configuration	Reported result	Useful interpretation
DeepSWE v1.1	Claude Opus 5 max effort on mini-swe-agent	74% ± 4%; $11.84 average task cost	Top-tier long-horizon model performance, but not a Claude Code product score
DeepSWE v1.1	GPT-5.6 Sol max effort on mini-swe-agent	73% ± 3%; $8.39 average task cost	Similar success interval with lower reported average cost in the same harness
Terminal-Bench 2.0	Codex CLI with GPT-5.5	82.2% ± 2.2%	Strong verified terminal-agent execution, not a pure repository-feature benchmark
RuBench	Claude Code with Opus 4.8	78.7% across 25 repository tasks	Useful deployed-product evidence, but the sample and language setting limit generalisation
RuBench	Codex CLI with GPT-5.5	66.7% across 25 repository tasks	Meaningful comparison inside that study, not proof of a universal gap

---

---

The accepted diff is the result, not the agent’s final message

Coding agents are unusually easy to benchmark badly because they can look successful before the repository is healthy. A polished summary can follow a partial implementation. A green unit test can coexist with a broken integration path. An agent can even “fix” a task by weakening the test, changing a fixture or bypassing validation.

DIY AI should therefore use an accepted diff as the primary unit of success. A run is accepted only when the final workspace passes the hidden behavioural verifier, keeps the repository’s pre-existing checks green, preserves protected interfaces and contains no prohibited changes. If a human must edit production code before merging, the autonomous run failed even if the repair is trivial.

This is stricter than a test-pass metric, but it reflects the work developers actually care about. A task that takes 12 minutes of agent time and another 25 minutes of review and repair is not equivalent to a 16-minute run that can be merged as-is.

Why a SWE-bench percentage cannot tell you which coding agent to buy

SWE-bench remains useful because it made repository-level software engineering measurable. The problem is interpretation. A leaderboard row can represent a model, an agent scaffold, a reasoning setting or a combined product configuration. Those are different experimental units.

The latest DeepSWE v1.1 snapshot clarifies the distinction. It runs models through the same mini-swe-agent harness across 113 original long-horizon tasks. On 13 August 2026, Claude Opus 5 at max effort was at 74% ± 4%, with an average reported task cost of $11.84, while GPT-5.6 Sol at max effort was at 73% ± 3% and $8.39, respectively. The success intervals overlap, while the cost difference is material. That is useful model evidence, but it does not prove that Claude Code beats Codex CLI, or vice versa, because neither commercial agent product is the variable being isolated.

Terminal-Bench 2.0 gives another signal. Its verified leaderboard reported Codex CLI with GPT-5.5 at 82.2% ± 2.2% on 23 April 2026. That tells us Codex CLI is a strong terminal agent configuration, but Terminal-Bench includes broader terminal and system tasks. It should not be treated as a direct forecast for a dependency migration in a large TypeScript monorepo.

RuBench is more interesting for product evaluation because it tests deployed CLI agents plus model configurations on recent repository tasks. Claude Code with Opus 4.8 resolved 78.7% of its 25 tasks, while Codex CLI with GPT-5.5 resolved 66.7%. The study also warned that the sample was too small to statistically resolve many close differences. More importantly, an experimental Claude Code configuration silently fell back to another model on 5 of 25 tasks. That is exactly why a 2026 coding agent benchmark should record the observed product configuration, not just the model the tester intended to run.

Use two benchmark tracks, or you will confuse product quality with model quality

A fair evaluation needs two separate tracks. Do not average them into one score.

Track	Question it answers	Configuration rule
Product reality	Which coding agent should a developer actually use?	Run each product with its recommended or normal production configuration, then record the exact model, plan, permissions and tool access
Harness isolation	How much does the agent software itself help or hurt?	Hold the underlying model, reasoning effort, task, repository and network permissions constant wherever the products allow it

The product-reality track is the commercial answer. Claude Code, Codex CLI, and Cursor are not interchangeable wrappers, so their native context handling, editing tools, approval flow, and recovery behaviour should be included in the measurement.

The harness-isolation track is diagnostic. Model-agnostic products are particularly useful here because you can reduce one source of variance. Our Kilo Code review explains why provider freedom is valuable but also why model routing, indexing and permissions can make two installations of the same agent behave very differently.

The five repository tasks that expose different failure modes

A useful suite does not need hundreds of tasks before it tells you anything. Start with five carefully designed categories, then add repetitions. The important part is that each task has a hidden acceptance oracle and enough ambiguity to require repository exploration.

1. Bug fix: Can the agent find the actual fault?

Seed a realistic defect where the visible error points near the problem but not directly at it. The final fix should touch a small number of production files and preserve existing behaviour outside the failing path.

A good example is a parser, validation or state-management bug with one obvious failing case and two hidden edge cases. Do not give the agent the exact test that reveals the root cause. Grade the final behaviour, not whether the patch resembles a reference fix.

2. Constrained refactor: can it change structure without changing behaviour?

Ask the agent to extract a module, split an oversized service or replace an internal pattern while preserving the public API. The hidden verifier should cover old behaviour, imports, serialisation, and at least one less obvious call path.

Refactors are where “helpful” agents often create review debt. They rename unrelated symbols, clean up nearby code, introduce a new abstraction, or move files that didn’t need moving. Those changes may be reasonable in isolation, but they make the benchmark task worse because the requested surface was narrow.

3. Test generation: grade whether the tests catch bugs, not how many tests exist

Line coverage is a weak metric for evaluating generated tests. Instead, prepare hidden mutants or deliberately altered implementations and measure whether the agent’s new tests reject them while still passing on the correct implementation.

Six focused tests that kill the seeded mutants are more valuable than a 400-line test file that repeats happy paths. This also catches agents that generate assertions about implementation details rather than behaviour.

4. Dependency upgrade: force the agent to deal with the migration, not dodge it

Pin a repository to an older dependency, then ask for an upgrade that introduces one or two known API changes. The acceptance checks should require the new version, a valid lockfile, a successful build and unchanged application behaviour.

Explicitly fail runs that pin to the old version, suppress the failing feature, or remove the check. Dependency work is valuable because it tests documentation use, package tooling, multi-file edits and the agent’s ability to distinguish a migration problem from an application bug.

5. Multi-file feature: make the agent carry one requirement through the stack

Add a small feature that touches configuration, domain logic, an API or UI boundary and tests. Give it one trap, such as a feature flag, an optional field, or a permission check, that can be missed if the agent only follows the obvious happy path.

This is the closest of the five tasks to normal feature work. It tests whether the agent maintains a consistent mental model across multiple file reads and edits, rather than solving a single, isolated function.

Run each task three times from a clean checkout

One run is a demo. Three independent runs begin to reveal reliability.

Every repetition should start from the same pinned commit in a fresh worktree or container, with no conversation history, no previous diffs and no warmed project-specific agent memory unless warm-state behaviour is the thing being tested. Keep the prompt byte-for-byte identical. Record the agent version, model version, reasoning setting and date.

Do not report “best of three”. Report all three outcomes. A task that passes once and fails twice is a 33% reliable task for that configuration, even if the successful diff is excellent.

This is also where differences between agents become more useful than a single average. One agent may solve three categories almost every time and repeatedly fail dependency work. Another may be less consistent overall but much stronger on refactors. Teams should see the category pattern rather than a flattened badge.

Separate self-correction from retries

The word “retry” is often used to mean three different things, which makes benchmark reports hard to compare.

Event	Definition	How to count it
Self-correction	The agent runs a check, sees failure and changes approach inside the same session	Count agent steps and verification commands, not as a new trial
Independent repetition	The same task is run again from a clean checkout with the original prompt	A separate benchmark trial used for reliability
Repair attempt	A failed run receives error output or human feedback and gets another chance	Keep outside pass@1 and report as repair success

This separation prevents an agent with a long internal loop from looking identical to one that lands the correct patch immediately. It also allows a second, useful workflow test: after a first-pass failure, give the agent only the verifier output, without a human diagnosis, and measure whether it can recover. That tests repair skill without contaminating the autonomous result.

Measure effective cost per accepted task, not advertised token price

Agent cost is a system metric. The cheap model can be the expensive agent if it repeatedly rereads context, runs the same test suite and needs another session after a wrong implementation.

For API-billed configurations, record the provider-reported inference cost for every attempt, including failed ones. Then calculate:

Effective cost per accepted task = total measured agent cost across all trials ÷ number of accepted tasks.

For subscription products, do not invent a precise per-task API cost from the monthly fee. Publish any exposed credits or usage units separately, then show an amortised plan cost only if the observation window is long enough to make that number meaningful.

Review time deserves its own column. A recurring practical problem with long agent sessions is that activity looks productive while the run repeatedly reprocesses repository context and accumulates a larger diff. Capturing agent steps, full-suite test runs and human review minutes makes that inefficiency visible even when the final patch eventually passes.

Elapsed time needs a baseline, or slow test suites distort the ranking

Record total wall-clock time because that is what a developer experiences, but also record baseline verifier time. If a repository’s full integration suite takes eight minutes, an agent should not be punished as though all eight minutes were reasoning overhead.

A useful timing report contains time to first meaningful diff, time spent in agent execution, time spent running repository checks and total time to acceptance. If an agent launches parallel workers, record concurrency as well. Otherwise, a parallel system can look dramatically faster while consuming far more total compute.

A passing test suite is still a failure if the diff cheats

The grader needs hard failure rules before the first run. Add them after seeing results, and the benchmark becomes subjective.

• Changing hidden verifier files is an automatic failure.
• Disabling or deleting required tests results in an automatic failure unless the task explicitly requests test maintenance.
• Downgrading a dependency to avoid the requested migration is an automatic failure.
• Removing validation, authentication or safety checks to make a failing path pass is an automatic failure.
• Edits outside the allowed repository should be blocked by the sandbox, not merely noted after the run.

Unnecessary but harmless code churn should be recorded separately as scope debt rather than always causing failure. That gives reviewers a way to distinguish a correct but noisy diff from a clean implementation without turning style preference into hidden grading logic.

Repository instruction files should be controlled, not assumed, to improve results

AGENTS.md and CLAUDE.md files are now common in agentic development, but they are another experimental variable. A July 2026 two-agent study covering 288 evaluated runs found no statistically significant improvement in correctness from always-on or selectively retrieved context files for Claude Code and Codex across the tested repositories. Some efficiency behaviour changed, but correctness did not reliably move.

The implication for benchmarking is not “delete your repository instructions”. It is to keep their use explicit. In a product-reality track, allow the agent to consume the repository instructions exactly as the product normally does. In a harness-isolation track, provide equivalent information to every agent or remove it from all of them.

Oversized instruction files can also make a benchmark less representative. Real repositories contain conventions, but they do not usually contain a hand-authored solution strategy for the exact hidden task.

IDE agents need to be benchmarked in the IDE surface you actually use

A Cursor agent running inside its native editor is not the same experiment as a headless CLI wrapper around the same underlying model. The editor contributes retrieval, file editing, diff review, command execution and often background-agent behaviour.

The same warning applies to products with several agent surfaces. TRAE, for example, separates conventional IDE assistance from more autonomous modes and cloud workflows. Our TRAE AI review explains why long-session reliability and recovery should be judged differently from short editor interactions.

For a product benchmark, choose one named surface and stick to it. “Cursor”, “TRAE”, or “Copilot” is not a reproducible configuration by itself.

The benchmark manifest should make every run reconstructable

Store configuration beside the result, not in a note written after testing. A minimal manifest should include the following fields:

task_id
category
repository_commit
prompt_hash
agent_name
agent_version
requested_model
observed_model
reasoning_effort
permission_mode
network_access
mcp_servers
repository_instruction_mode
max_wall_time
max_agent_steps
trial_number
verifier_version
pricing_snapshot_date

The observed model field is worth keeping even when it is difficult to obtain. Product routing, safety fallbacks and plan-level model substitution can otherwise make a later reproduction look inexplicably different.

Do not let an expensive agent win by brute force

Every agent should receive the same wall-time ceiling and a documented step or budget ceiling. Without stop rules, a system that takes four times longer can eventually grind through more failures and appear “more capable”, even though it would be a poor production choice.

Publish timeout failures as failures. Do not exclude them as inconvenient infrastructure noise if the agent itself has exhausted the allowed budget. Provider outages and verifier failures are different and should be marked unscored, with the trial rerun later.

A sensible first benchmark uses one generous but finite task budget, then a secondary cost curve. The cost curve answers a useful question the headline pass rate cannot: how much extra success do you buy by letting the agent keep working?

Use public leaderboards to choose the shortlist, then benchmark your repository

Public benchmarks are excellent for filtering obviously weak configurations and seeing where frontier systems cluster. They are much weaker at predicting which agent will be easiest to merge in a specific codebase. A mature Python service, a TypeScript monorepo and a WordPress plugin place different demands on repository discovery, build tooling, tests and dependency management.

If DIY AI had a budget to establish three baseline agents first, Claude Code and Codex CLI should be the two terminal anchors, followed by the editor-native agent most representative of the target audience. Add model-agnostic agents when the goal shifts from buying advice to isolating harness behaviour.

The winner should minimise review debt, not maximise activity

The most useful final table should not be a decorative 9.4-versus-9.2 rating. Show raw task outcomes by category, then accepted-task rate, first-pass acceptance, three-run consistency, median time to acceptance, effective cost per accepted task, review minutes and scope violations.

A team shipping a mature backend may accept a slower agent if its diffs are consistently narrow and reviewable. A prototype team may prefer a faster agent that occasionally needs repair. A large monorepo may care more about repeated context efficiency and targeted test selection than a small service does.

That is why the best coding agent benchmark is deliberately harder to summarise than a public leaderboard. Repository engineering is not a capability. The agent has to find the right code, change the right amount, verify the right behaviour, recover when it is wrong and stop before its own process becomes the expensive part of the task.

AI coding agent benchmark FAQ

What is the best benchmark for AI coding agents?

There is no single benchmark that covers every production concern. SWE-bench-style evaluations are useful for resolving repository issues. DeepSWE adds original long-horizon tasks and behavioural verifiers, and Terminal-Bench tests broader terminal execution. For tool selection, a private repeated-run suite based on your repository patterns is more actionable than any one public score.

How many times should a coding agent run each benchmark task?

Three clean repetitions are a practical minimum for an editorial benchmark because they expose obvious variance without multiplying cost excessively. More repetitions yield greater statistical power, especially when the two agents are close. Never select only the best run.

Should coding agents be allowed to edit tests?

Only when the task explicitly requires test changes. Otherwise, protect hidden verifiers and treat the deletion, weakening, or bypassing of required checks as a failure. For test-generation tasks, grade the new tests against hidden mutants or seeded faults rather than trusting coverage alone.

How should the coding agent cost be compared?

Use an effective cost per accepted task. Include failed attempts and retries. Report subscription allocation separately from measurable API or credit usage, and add review time as its own metric instead of pretending token spend captures the whole cost.

Should Claude Code, Codex CLI and Cursor use the same model in a benchmark?

Run both kinds of tests if possible. A product-reality benchmark should use each tool in a realistic, supported configuration. A harness-isolation benchmark should hold the model and reasoning settings constant where the products allow it. Do not merge those tracks because they answer different questions.