BenchmarkActivecategory · codingfamily · swe-bench · variant Lite

SWE-bench Lite

swebench.com

resolve real GitHub issues (300-instance subset)

data quality57

Updated 6 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 78
Models: 26
Current leader: Undisclosed 39.3%

Filters

Comparability group

Trust level

Views

Score history · Undisclosed 17 rows

Undisclosed

44%date=2025-03-10 · board=Lite · system=CodeFuse-CGM · submission=20250310_codefuse-cgm10 Mar 2025
55%date=2025-01-14 · board=Lite · system=Isoform · submission=20250114_Isoform14 Jan 2025
35.67%date=2025-01-04 · board=Lite · system=CodeFuse-AAIS · submission=20250104_codefuse-aais4 Jan 2025
49%date=2024-12-20 · board=Lite · system=Blackbox AI Agent · submission=20241220_blackboxai_agent_v120 Dec 2024
48.67%date=2024-12-08 · board=Lite · system=Gru · submission=20241208_gru8 Dec 2024
48.33%date=2024-11-27 · board=Lite · system=Globant Code Fixer Agent · submission=20241127_globant_codefixer_agent27 Nov 2024
39.33%date=2024-09-12 · board=Lite · system=Bytedance MarsCode Agent · submission=20240912_marscode-agent-dev12 Sept 2024
30%date=2024-09-08 · board=Lite · system=AIGCode Infant-Coder · submission=20240908_infant_gpt4o8 Sept 2024
35%date=2024-08-29 · board=Lite · system=Isoform · submission=20240829_Isoform29 Aug 2024
38.33%date=2024-08-20 · board=Lite · system=Honeycomb · submission=20240820_honeycomb20 Aug 2024
35.67%date=2024-08-11 · board=Lite · system=Gru · submission=20240811_gru11 Aug 2024
34%date=2024-08-06 · board=Lite · system=SuperCoder2.0 · submission=20240806_SuperCoder2.06 Aug 2024

Back to the leaderboard

Frontier over time · resolved · board=Lite · system=Bytedance MarsCode Agent

39.3%Undisclosed Community12 Sept 2024
34%gpt-4o OpenAI Community23 Jul 2024

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 2 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	Undisclosed	39.3%	Community	group defaults	leader	12 Sept 2024	swebench.comT2	History
2	gpt-4oClosedOpenAI · GPT 4	34%	Community	group defaults	Partially comparable-5.33 pt	23 Jul 2024	swebench.comT2	History

2 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Known limitations: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Paper: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Website: swebench.com
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium

Active group

resolved · board=Lite · system=Bytedance MarsCode Agent

2 models · 2 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench Lite

slug swe-bench-lite · bench_01M293SPG45DHPX5C1HZD84RJY

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.