BenchmarkActivecategory · codingfamily · swe-bench · variant Multimodal

SWE-bench Multimodal

swebench.com/multimodal

resolve visual JavaScript issues (screenshots + code)

data quality57

Updated 6 h ago · first seen 11 Sept 2026

Metric: resolved · % ↑
Current results: 22
Models: 10
Current leader: o3 36.0%

Filters

Comparability group

Trust level

Views

Score history · gpt-4o 6 rows

gpt-4o

30.37%date=2025-05-31 · board=Multimodal · system=GUIRepair · model_tag=gpt-4o-2024-08-0631 May 2025
3.09%date=2024-10-06 · board=Multimodal · system=Agentless · model_tag=gpt-4o-2024-08-066 Oct 2024
6%date=2024-10-06 · board=Multimodal · system=RAG · model_tag=gpt-4o-2024-08-066 Oct 2024
9.28%date=2024-10-06 · board=Multimodal · system=SWE-agent JavaScript · submission=20241006_SWE-agent_JS_gpt4o6 Oct 2024
11.99%date=2024-10-06 · board=Multimodal · system=SWE-agent · submission=20241006_SWE-agent_gpt4o6 Oct 2024
12.19%date=2024-10-06 · board=Multimodal · system=SWE-agent Multimodal · submission=20241006_SWE-agent_M_gpt4o6 Oct 2024

Back to the leaderboard

Frontier over time · resolved · board=Multimodal · system=GUIRepair

36.0%o3 OpenAI Official board1 Jul 2025
33.9%o4-mini OpenAI Official board31 May 2025

Includes closed rows (history). A point is emitted whenever a result beats every earlier result of the same group, ordered by evaluated_at when the source publishes it, else observed_at.

Leaderboard 4 models

Select models with +, then Compare.

Leaderboard
#	Model	Score	Trust	Configuration	vs leader	Evaluated	Source	Actions
1	o3ClosedOpenAI · OpenAI o-series	36.0%	Official board	group defaults	leader	1 Jul 2025	swebench.comT2	History
2	o4-miniClosedOpenAI · OpenAI o-series	33.9%	Official board	group defaults	Partially comparable-2.13 pt	31 May 2025	swebench.comT2	History
3	gpt-4.1ClosedOpenAI · GPT 4.1	31.1%	Official board	group defaults	Partially comparable-4.84 pt	31 May 2025	swebench.comT2	History
4	gpt-4oClosedOpenAI · GPT 4	30.4%	Official board	group defaults	Partially comparable-5.61 pt	31 May 2025	swebench.comT2	History

4 results

One row per canonical model — its best current row inside this comparability group (effort variants are folded into the model). Bars are relative to the page's best score. “vs leader” reads comparability: partially comparable = same task, conditions differ (reasoning effort, temperature, judge). Rules →

Definition

Task: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium
Metric range: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Variant: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Harness: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Comparability note: Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 6 h agomedium
Website: swebench.com/multimodal
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 16 h agomedium

Active group

resolved · board=Multimodal · system=GUIRepair

4 models · 4 current rows

Comparability

Comparable same task and conditions · Partially comparable same task, conditions differ (effort, temperature, judge) · Not comparable different variant or metric

Also known as

SWE-bench MM, SWE-bench Multimodal

slug swe-bench-multimodal · bench_01M293SPGC8HWGJ1NKFWQ04G29

Rows are grouped by comparability group = canonical metric × config_key (hash of the task-defining configuration keys: variant, evaluator, harness, shots, pass regime…). A leaderboard shows one row per canonical model: its best current row inside the group. Reasoning effort, temperature or judge differences keep rows in the same group but mark them partially comparable. delta_rank compares with the ranking built from the closed (previous) rows of the same group.