BenchmarkActivecategory · codingfamily · swe-bench · variant Lite

SWE-bench Lite

swebench.com

resolve real GitHub issues (300-instance subset)

data quality57

Updated 5 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 78
Models: 26
Current leader: Claude 4 Sonnet 58.3%

Filters

Comparability group

Trust level

Views

Score history · claude-35-sonnet 13 rows

claude-35-sonnet

51.33%date=2025-09-11 · board=Lite · system=Isea · model_tag=claude-3.5-sonnet-2024102211 Sept 2025
46%date=2025-06-19 · board=Lite · system=KGCompass · model_tag=claude-3-5-sonnet-2024102219 Jun 2025
42.67%date=2025-05-09 · board=Lite · system=Lingxi · model_tag=claude-3-5-sonnet-202410229 May 2025
39%date=2025-01-14 · board=Lite · system=Moatless Tools · model_tag=claude-3-5-sonnet-2024102214 Jan 2025
41%date=2025-01-13 · board=Lite · system=OrcaLoca + Agentless-1.5 · model_tag=claude-3-5-sonnet-2024102213 Jan 2025
37%date=2025-01-04 · board=Lite · system=Patched.Codes Patchwork · model_tag=claude-3-5-sonnet-202410224 Jan 2025
41.33%date=2024-12-20 · board=Lite · system=PatchKitty-0.9 · model_tag=claude-3-5-sonnet-2024102220 Dec 2024
44.67%date=2024-12-07 · board=Lite · system=Kodu-v1 · model_tag=claude-3-5-sonnet-202410227 Dec 2024
40.67%date=2024-12-02 · board=Lite · system=Agentless-1.5 · model_tag=claude-3-5-sonnet-202410222 Dec 2024
47.33%date=2024-11-22 · board=Lite · system=devlo · model_tag=claude-3-5-sonnet-2024102222 Nov 2024
38.33%date=2024-11-17 · board=Lite · system=Moatless Tools · model_tag=claude-3-5-sonnet-2024102217 Nov 2024
26.67%date=2024-06-23 · board=Lite · system=Moatless Tools · model_tag=claude-3-5-sonnet-2024102223 Jun 2024

Back to the leaderboard

Frontier over time · resolved · board=Lite · system=KGCompass

58.3%Claude 4 Sonnet Anthropic Community6 Sept 2025
46%claude-35-sonnet Anthropic Community19 Jun 2025
36.7%DeepSeek V3 DeepSeek Community9 Jun 2025

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 3 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	Claude 4 SonnetAnthropic · Claude 4	58.3%	Community	group defaults	leader	6 Sept 2025	swebench.comT2	History
2	claude-35-sonnetClosedAnthropic · Claude 35	46%	Community	group defaults	Partially comparable-12.3 pt	19 Jun 2025	swebench.comT2	History
3	DeepSeek V3Open weightsDeepSeek · DeepSeek	36.7%	Community	group defaults	Partially comparable-21.7 pt	9 Jun 2025	swebench.comT2	History

3 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 14 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 14 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Known limitations: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 14 h agomedium
Paper: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 14 h agomedium
Website: swebench.com
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 14 h agomedium

Active group

resolved · board=Lite · system=KGCompass

3 models · 3 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench Lite

slug swe-bench-lite · bench_01M293SPG45DHPX5C1HZD84RJY

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.