BenchmarkActivecategory · codingfamily · swe-bench · variant Verified

SWE-bench Verified

swebench.com

resolve real GitHub issues (500 human-validated instances)

data quality57

Updated 6 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 158
Models: 82
Current leader: gpt-5 71.8%

Filters

Comparability group

Trust level

Views

Score history · Undisclosed 24 rows

Undisclosed

70.4%date=2025-06-10 · board=Verified · system=Augment Agent v1 · submission=20250610_augment_agent_v110 Jun 2025
70.6%date=2025-05-19 · board=Verified · system=TRAE · submission=20250519_trae19 May 2025
65.8%date=2025-04-15 · board=Verified · system=OpenHands · submission=20250415_openhands15 Apr 2025
56.6%date=2025-04-05 · board=Verified · system=SWE-Rizzo · submission=20250405_swe-rizzo_claude375 Apr 2025
65.4%date=2025-04-05 · board=Verified · system=Amazon Q Developer Agent · submission=20250405_amazon-q-developer-agent-20250405-dev5 Apr 2025
65.4%date=2025-03-16 · board=Verified · system=Augment Agent v0 · submission=20250316_augment_agent_v016 Mar 2025
53.2%date=2025-01-20 · board=Verified · system=Bracket.sh · submission=20250120_Bracket20 Jan 2025
41.6%date=2025-01-12 · board=Verified · system=ugaiforge · submission=20250112_ugaiforge12 Jan 2025
62.8%date=2025-01-10 · board=Verified · system=Blackbox AI Agent · submission=20250110_blackboxai_agent_v1.110 Jan 2025
58.2%date=2024-12-13 · board=Verified · system=devlo · submission=20241213_devlo13 Dec 2024
57%date=2024-12-08 · board=Verified · system=Gru · submission=20241208_gru8 Dec 2024
55%date=2024-12-02 · board=Verified · system=Amazon Q Developer Agent · submission=20241202_amazon-q-developer-agent-20241202-dev2 Dec 2024

Back to the leaderboard

Frontier over time · resolved · board=Verified · system=OpenHands

71.8%gpt-5 OpenAI Official board7 Aug 2025
70.4%Claude 4 Sonnet Anthropic Official board24 May 2025
65.8%Undisclosed Official board15 Apr 2025
60.8%4x Scaled Official board3 Feb 2025
53%CodeAct v2.1 (claude-3-5-sonnet-20241022) Anthropic Official board29 Oct 2024

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 9 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	gpt-5ClosedOpenAI · GPT 5	71.8%	Official board	group defaults	leader	7 Aug 2025	swebench.comT2	History
2	Claude 4 SonnetAnthropic · Claude 4	70.4%	Official board	group defaults	Partially comparable-1.40 pt	24 May 2025	swebench.comT2	History
3	qwen3-coder-480b-a35b-instructOpen weightsAlibaba Group · Qwen3-Coder	69.6%	Community	group defaults	Partially comparable-2.20 pt	5 Aug 2025	swebench.comT2	History
4	Undisclosed	65.8%	Official board	group defaults	Partially comparable-6.00 pt	15 Apr 2025	swebench.comT2	History
5	Kimi K2 0711Open weightsMoonshot AI · Kimi	65.4%	Official board	group defaults	Partially comparable-6.40 pt	16 Jul 2025	swebench.comT2	History
6	4x Scaled	60.8%	Official board	group defaults	Partially comparable-11.0 pt	3 Feb 2025	swebench.comT2	History
7	CodeAct v2.1 (claude-3-5-sonnet-20241022)Anthropic · Claude 3.5	53%	Official board	group defaults	Partially comparable-18.8 pt	29 Oct 2024	swebench.comT2	History
8	Qwen3 Coder 30B A3B InstructOpen weightsQwen · Qwen3	51.6%	Community	group defaults	Partially comparable-20.2 pt	5 Aug 2025	swebench.comT2	History
9	Devstral Small 1.0Open weightsMistral AI · Devstral	46.8%	Official board	group defaults	Partially comparable-25.0 pt	20 May 2025	swebench.comT2	History

9 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Known limitations: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Paper: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Website: swebench.com
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium

Active group

resolved · board=Verified · system=OpenHands

9 models · 9 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench, SWE-bench Verified

slug swe-bench-verified · bench_01M293SPEGA1XMKH0SMTFTCK2R

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.