BenchmarkActivecategory · codingfamily · swe-bench · variant Verified

SWE-bench Verified

swebench.com

resolve real GitHub issues (500 human-validated instances)

data quality57

Updated 5 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 158
Models: 82
Current leader: Qwen3 Coder 30B A3B Instruct 60.4%

Filters

Comparability group

Trust level

Views

Score history · Qwen3 Coder 30B A3B Instruct 3 rows

Qwen3 Coder 30B A3B Instruct

52.2%date=2025-09-01 · board=Verified · system=EntroPO + R2E · model_tag=Qwen3-Coder-30B-A3B-Instruct1 Sept 2025
60.4%date=2025-09-01 · board=Verified · system=EntroPO + R2E · model_tag=Qwen3-Coder-30B-A3B-Instruct1 Sept 2025
51.6%date=2025-08-05 · board=Verified · system=OpenHands · model_tag=https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct5 Aug 2025

Back to the leaderboard

Frontier over time · resolved · board=Verified · system=EntroPO + R2E

1 leader change recorded, all dated 1 Sept 2025 — the frontier line needs at least two distinct dates. The corpus is young: every result was first observed on the same day, so leader changes will separate in time as sources are re-crawled.

60.4%Qwen3 Coder 30B A3B Instruct Qwen Official board1 Sept 2025

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 1 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	Qwen3 Coder 30B A3B InstructOpen weightsQwen · Qwen3 · best of 2 rows	60.4%	Official board	group defaults	leader	1 Sept 2025	swebench.comT2	History

1 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 5 h agomedium
Known limitations: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Paper: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium
Website: swebench.com
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 15 h agomedium

Active group

resolved · board=Verified · system=EntroPO + R2E

1 models · 2 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench, SWE-bench Verified

slug swe-bench-verified · bench_01M293SPEGA1XMKH0SMTFTCK2R

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.