BenchmarkActivecategory · codingfamily · swe-bench · variant Verified

SWE-bench Verified

swebench.com

resolve real GitHub issues (500 human-validated instances)

data quality57

Updated 6 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 158
Models: 82
Current leader: Claude 4 Sonnet 66.6%

Filters

Comparability group

Trust level

Views

Score history · claude-35-sonnet 11 rows

claude-35-sonnet

62.8%date=2025-02-28 · board=Verified · system=EPAM AI/Run Developer Agent · model_tag=claude-3-5-sonnet-2024102228 Feb 2025
63.4%date=2025-02-06 · board=Verified · system=AgentScope · model_tag=claude-3-5-sonnet-202410226 Feb 2025
60.2%date=2025-01-10 · board=Verified · system=Learn-by-interact · model_tag=claude-3-5-sonnet-2024102210 Jan 2025
55.4%date=2024-12-12 · board=Verified · system=EPAM AI/Run Developer Agent · model_tag=claude-3-5-sonnet-2024102212 Dec 2024
50.8%date=2024-12-02 · board=Verified · system=Agentless-1.5 · model_tag=claude-3-5-sonnet-202410222 Dec 2024
51.8%date=2024-11-25 · board=Verified · system=Engine Labs · model_tag=claude-3-5-sonnet-2024102225 Nov 2024
54.2%date=2024-11-08 · board=Verified · system=devlo · model_tag=claude-3-5-sonnet-202410228 Nov 2024
39.6%date=2024-10-29 · board=Verified · system=EPAM AI/Run Developer Agent · model_tag=claude-3-5-sonnet-2024102229 Oct 2024
49%date=2024-10-22 · board=Verified · system=Tools · model_tag=claude-3-5-sonnet-2024102222 Oct 2024
40.6%date=2024-10-16 · board=Verified · system=Composio SWEkit · model_tag=claude-3-5-sonnet-2024102216 Oct 2024
33.6%date=2024-06-20 · board=Verified · system=SWE-agent · model_tag=claude-3-5-sonnet-2024102220 Jun 2024

Back to the leaderboard

Frontier over time · resolved · board=Verified · system=SWE-agent

66.6%Claude 4 Sonnet Anthropic Official board22 May 2025
62.4%Claude 3.7 Sonnet w/ Review Heavy Anthropic Official board25 Feb 2025
33.6%claude-35-sonnet Anthropic Official board20 Jun 2024
22.4%GPT-4 (1106) OpenAI Official board2 Apr 2024

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 8 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	Claude 4 SonnetAnthropic · Claude 4	66.6%	Official board	group defaults	leader	22 May 2025	swebench.comT2	History
2	Claude 3.7 Sonnet w/ Review HeavyAnthropic	62.4%	Official board	group defaults	Partially comparable-4.20 pt	25 Feb 2025	swebench.comT2	History
3	SWE-agent-LM-32BQwen	40.2%	Official board	group defaults	Partially comparable-26.4 pt	11 May 2025	swebench.comT2	History
4	Devstral Small 1.1Mistral AI · Devstral	38%	Official board	group defaults	Partially comparable-28.6 pt	25 Jul 2025	swebench.comT2	History
5	claude-35-sonnetClosedAnthropic · Claude 35	33.6%	Official board	group defaults	Partially comparable-33.0 pt	20 Jun 2024	swebench.comT2	History
6	gpt-4oClosedOpenAI · GPT 4	23.2%	Official board	group defaults	Partially comparable-43.4 pt	28 Jul 2024	swebench.comT2	History
7	GPT-4 (1106)OpenAI · GPT 4	22.4%	Official board	group defaults	Partially comparable-44.2 pt	2 Apr 2024	swebench.comT2	History
8	claude-3-opusClosedAnthropic · Claude 3	15.8%	Official board	group defaults	Partially comparable-50.8 pt	2 Apr 2024	swebench.comT2	History

8 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Known limitations: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Paper: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Website: swebench.com
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium

Active group

resolved · board=Verified · system=SWE-agent

8 models · 8 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench, SWE-bench Verified

slug swe-bench-verified · bench_01M293SPEGA1XMKH0SMTFTCK2R

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.