BenchmarkActivecategory · codingfamily · swe-bench · variant Lite

SWE-bench Lite

swebench.com

resolve real GitHub issues (300-instance subset)

data quality57

Updated 6 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 78
Models: 26
Current leader: Claude 4 Sonnet 56.7%

Filters

Comparability group

Trust level

Views

Score history · gpt-4o 14 rows

gpt-4o

28%date=2024-11-17 · board=Lite · system=reproducedRG · model_tag=gpt-4o-2024-08-0617 Nov 2024
31.33%date=2024-11-11 · board=Lite · system=CodeShellTester · model_tag=gpt-4o-2024-05-1311 Nov 2024
32%date=2024-10-28 · board=Lite · system=Agentless-1.5 · model_tag=gpt-4o-2024-05-1328 Oct 2024
25.33%date=2024-09-25 · board=Lite · system=HyperAgent · model_tag=gpt-4o-2024-08-0625 Sept 2024
29.67%date=2024-08-08 · board=Lite · system=Agentless + RepoGraph · model_tag=gpt-4o-2024-08-068 Aug 2024
18.33%date=2024-07-28 · board=Lite · system=SWE-agent · model_tag=gpt-4o-2024-05-1328 Jul 2024
34%date=2024-07-23 · board=Lite · system=Bytedance MarsCode Agent · model_tag=gpt-4o-2024-05-1323 Jul 2024
27.67%date=2024-07-06 · board=Lite · system=SIMA · model_tag=gpt-4o-2024-05-136 Jul 2024
27.33%date=2024-06-30 · board=Lite · system=Agentless · model_tag=gpt-4o-2024-05-1330 Jun 2024
38%date=2024-06-27 · board=Lite · system=AbanteAI MentatBot · model_tag=gpt-4o-2024-05-1327 Jun 2024
30.67%date=2024-06-21 · board=Lite · system=AutoCodeRover · model_tag=gpt-4o-2024-05-1321 Jun 2024
24.67%date=2024-06-17 · board=Lite · system=Moatless Tools · model_tag=gpt-4o-2024-05-1317 Jun 2024

Back to the leaderboard

Frontier over time · resolved · board=Lite · system=SWE-agent

56.7%Claude 4 Sonnet Anthropic Official board26 May 2025
48%Claude 3.7 Sonnet Anthropic Official board26 Feb 2025
23%claude-35-sonnet Anthropic Official board20 Jun 2024
18%GPT-4 (1106) OpenAI Official board2 Apr 2024

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 6 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	Claude 4 SonnetAnthropic · Claude 4	56.7%	Official board	group defaults	leader	26 May 2025	swebench.comT2	History
2	Claude 3.7 SonnetClosedAnthropic · Claude	48%	Official board	group defaults	Partially comparable-8.67 pt	26 Feb 2025	swebench.comT2	History
3	claude-35-sonnetClosedAnthropic · Claude 35	23%	Official board	group defaults	Partially comparable-33.7 pt	20 Jun 2024	swebench.comT2	History
4	gpt-4oClosedOpenAI · GPT 4	18.3%	Official board	group defaults	Partially comparable-38.3 pt	28 Jul 2024	swebench.comT2	History
5	GPT-4 (1106)OpenAI · GPT 4	18%	Official board	group defaults	Partially comparable-38.7 pt	2 Apr 2024	swebench.comT2	History
6	claude-3-opusClosedAnthropic · Claude 3	11.7%	Official board	group defaults	Partially comparable-45.0 pt	2 Apr 2024	swebench.comT2	History

6 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Known limitations: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Paper: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Website: swebench.com
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium

Active group

resolved · board=Lite · system=SWE-agent

6 models · 6 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench Lite

slug swe-bench-lite · bench_01M293SPG45DHPX5C1HZD84RJY

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.