Skip to content
AI Atlas
BenchmarkActivecategory · coding

SWE-bench Verified

swebench.com

resolve real GitHub issues (500 human-validated instances)

quality57

Updated 8 h ago · first seen 11 Sept 2026

bench_01M293SPEGA1XMKH0SMTFTCK2R

Metric
resolved · %
Direction
Higher is better
Results
180
Leader
Claude Opus 4.5 79.2%

Score history · Gemini 2.5 Flash 1 row

Not enough history to chart — a single observation (28.73% on 26 Jul 2025). Rows under different configurations count separately; the list below shows each one.

  • 28.73%date=2025-07-26 · board=Verified · system=mini-SWE-agent · model_tag=gemini-2.5-flash26 Jul 2025

Back to the full leaderboard

Leaderboard 180 current results

Select models with +, then open Compare.

Leaderboard
#ModelScoreConfigEvaluatedSourceActions
101#101CodeAct v2.1 (claude-3-5-sonnet-20241022)Anthropic53%date=2024-10-29 · board=Verified · system=OpenHands · submission=20241029_OpenHands-CodeAct-2.1-sonnet-2024102229 Oct 2024swebench.comT2 History
102#102Claude 3.7 SonnetAnthropic52.8%date=2025-07-20 · board=Verified · system=mini-SWE-agent · model_tag=claude-3-7-sonnet-2025021920 Jul 2025swebench.comT2 History
103#103Gemini 2.0 Flash (v20241212-experimental)Google DeepMind52.2%date=2024-12-12 · board=Verified · system=Google Jules · submission=20241212_google_jules_gemini_2.0_flash_experimental12 Dec 2024swebench.comT2 History
104#104Qwen3 Coder 30B A3B InstructQwen52.2%date=2025-09-01 · board=Verified · system=EntroPO + R2E · model_tag=Qwen3-Coder-30B-A3B-Instruct1 Sept 2025swebench.comT2 History
105#105claude-35-sonnetAnthropic51.8%date=2024-11-25 · board=Verified · system=Engine Labs · model_tag=claude-3-5-sonnet-2024102225 Nov 2024swebench.comT2 History
106#106claude-3-5-sonnet-20241022Anthropic51.6%date=2025-01-22 · board=Verified · system=AutoCodeRover-v2.1 · model_tag=claude-3-5-sonnet-2024102222 Jan 2025swebench.comT2 History
107#107Qwen3 Coder 30B A3B InstructQwen51.6%date=2025-08-05 · board=Verified · system=OpenHands · model_tag=https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct5 Aug 2025swebench.comT2 History
108#108claude-35-sonnetAnthropic50.8%date=2024-12-02 · board=Verified · system=Agentless-1.5 · model_tag=claude-3-5-sonnet-202410222 Dec 2024swebench.comT2 History
109#109Undisclosed50%date=2024-10-28 · board=Verified · system=Solver · submission=20241028_solver28 Oct 2024swebench.comT2 History
110#110Undisclosed50%date=2024-11-25 · board=Verified · system=Bytedance MarsCode Agent · submission=20241125_marscode-agent-dev25 Nov 2024swebench.comT2 History
111#111MultipleAnthropic49.2%date=2024-11-05 · board=Verified · system=nFactorial · model_tag=claude-3-5-sonnet-202410225 Nov 2024swebench.comT2 History
112#112claude-35-sonnetAnthropic49%date=2024-10-22 · board=Verified · system=Tools · model_tag=claude-3-5-sonnet-2024102222 Oct 2024swebench.comT2 History
113#113MultipleAnthropic48.6%date=2024-10-25 · board=Verified · system=Composio SWE-Kit · model_tag=claude-3-5-sonnet-2024102225 Oct 2024swebench.comT2 History
114#114MultipleAnthropic47.2%date=2024-11-06 · board=Verified · system=AppMap Navie v2 · model_tag=claude-3-5-sonnet-202410226 Nov 2024swebench.comT2 History
115#115TTS(Bo8)Qwen47%date=2025-06-16 · board=Verified · system=Skywork-SWE-32B · model_tag=Qwen2.5-Coder-32B-Instruct16 Jun 2025swebench.comT2 History
116#116Devstral Small 1.0Mistral AI46.8%date=2025-05-20 · board=Verified · system=OpenHands · model_tag=mistralai/Devstral-Small-250520 May 2025swebench.comT2 History
117#117MultipleAnthropic46.6%date=2024-10-23 · board=Verified · system=Emergent E1 · model_tag=claude-3-5-sonnet-2024102223 Oct 2024swebench.comT2 History
118#118claude-3-5-sonnet-20241022Anthropic46.2%date=2024-11-08 · board=Verified · system=AutoCodeRover-v2.0 · model_tag=claude-3-5-sonnet-202410228 Nov 2024swebench.comT2 History
119#119Co-PatcheR46%date=2025-05-28 · board=Verified · system=PatchPilot · submission=20250528_patchpilot_Co-PatcheR28 May 2025swebench.comT2 History
120#120Undisclosed45.4%date=2024-09-24 · board=Verified · system=Solver · submission=20240924_solver24 Sept 2024swebench.comT2 History
121#121Undisclosed45.2%date=2024-08-24 · board=Verified · system=Gru · submission=20240824_gru24 Aug 2024swebench.comT2 History
122#122Frogmini 14B 251045%date=2025-11-10 · board=Verified · system=FrogMini-14B-2510 · model_tag=FrogMini-14B-251010 Nov 2025swebench.comT2 History
123#123o4-miniOpenAI45%date=2025-07-26 · board=Verified · system=mini-SWE-agent · model_tag=o4-mini-2025041626 Jul 2025swebench.comT2 History
124#124gemini-2.0-flash-expGoogle44.2%date=2025-01-18 · board=Verified · system=CodeShellAgent · model_tag=gemini-2.0-flash-exp18 Jan 2025swebench.comT2 History
125#125Kimi K2 0711Moonshot AI43.8%date=2025-08-07 · board=Verified · system=mini-SWE-agent · model_tag=Kimi-K2-Instruct7 Aug 2025swebench.comT2 History
126#126Undisclosed43.6%date=2024-09-20 · board=Verified · system=Solver · submission=20240920_solver20 Sept 2024swebench.comT2 History
127#127Amazon.nova Premier v1:042.4%date=2025-05-27 · board=Verified · system=Amazon Nova Premier 1.0 · model_tag=amazon.nova-premier-v1:027 May 2025swebench.comT2 History
128#128o3-miniOpenAI42.4%date=2025-02-14 · board=Verified · system=Agentless Lite · model_tag=o3-mini14 Feb 2025swebench.comT2 History
129#129DeepSWE-Preview42.2%date=2025-06-29 · board=Verified · system=R2E-Gym · model_tag=https://huggingface.co/agentica-org/DeepSWE-Preview29 Jun 2025swebench.comT2 History
130#130DeepSeek V3 0324DeepSeek42%date=2025-08-06 · board=Verified · system=SWE-Exp · model_tag=DeepSeek-V3-03246 Aug 2025swebench.comT2 History
131#131MultipleAnthropic41.6%date=2024-10-30 · board=Verified · system=nFactorial · model_tag=claude-3-5-sonnet-2024102230 Oct 2024swebench.comT2 History
132#132Undisclosed41.6%date=2025-01-12 · board=Verified · system=ugaiforge · submission=20250112_ugaiforge12 Jan 2025swebench.comT2 History
133#133Llama3-SWE-RL-70BMeta AI41.2%date=2025-02-26 · board=Verified · system=Agentless Mini · model_tag=Llama3-SWE-RL-70B26 Feb 2025swebench.comT2 History
134#134claude-35-sonnetAnthropic40.6%date=2024-10-16 · board=Verified · system=Composio SWEkit · model_tag=claude-3-5-sonnet-2024102216 Oct 2024swebench.comT2 History
135#135MultipleAnthropic40.6%date=2024-11-13 · board=Verified · system=Nebius AI · model_tag=Llama 3.113 Nov 2024swebench.comT2 History
136#136Claude Haiku 3.5Anthropic40.6%date=2024-10-22 · board=Verified · system=Tools · model_tag=claude-3-haiku-2024030722 Oct 2024swebench.comT2 History
137#137Undisclosed40.6%date=2024-08-20 · board=Verified · system=Honeycomb · submission=20240820_honeycomb20 Aug 2024swebench.comT2 History
138#138SWE-agent-LM-32BQwen40.2%date=2025-05-11 · board=Verified · system=SWE-agent · model_tag=Qwen 2.511 May 2025swebench.comT2 History
139#139claude-35-sonnetAnthropic39.6%date=2024-10-29 · board=Verified · system=EPAM AI/Run Developer Agent · model_tag=claude-3-5-sonnet-2024102229 Oct 2024swebench.comT2 History
140#140gpt-4.1OpenAI39.58%date=2025-07-26 · board=Verified · system=mini-SWE-agent · model_tag=gpt-4.1-2025041426 Jul 2025swebench.comT2 History
141#141gpt-4oOpenAI38.8%date=2024-10-28 · board=Verified · system=Agentless-1.5 · model_tag=gpt-4o-2024-05-1328 Oct 2024swebench.comT2 History
142#142Undisclosed38.8%date=2024-07-21 · board=Verified · system=Amazon Q Developer Agent · submission=20240721_amazon-q-developer-agent-20240719-dev21 Jul 2024swebench.comT2 History
143#143gpt-4oOpenAI38.4%date=2024-06-28 · board=Verified · system=AutoCodeRover · model_tag=gpt-4o-2024-05-1328 Jun 2024swebench.comT2 History
144#144Devstral Small 1.1Mistral AI38%date=2025-07-25 · board=Verified · system=SWE-agent · model_tag=devstral-small-250725 Jul 2025swebench.comT2 History
145#145Qwen2.5 Coder 32B InstructQwen38%date=2025-06-16 · board=Verified · system=Skywork-SWE-32B · model_tag=Qwen2.5-Coder-32B-Instruct16 Jun 2025swebench.comT2 History
146#146Undisclosed37%date=2024-06-17 · board=Verified · system=Factory Code Droid · submission=20240617_factory_code_droid17 Jun 2024swebench.comT2 History
147#147gpt-5-nanoOpenAI34.8%date=2025-08-07 · board=Verified · system=mini-SWE-agent · model_tag=gpt-5-nano-2025-08-077 Aug 2025swebench.comT2 History
148#148claude-35-sonnetAnthropic33.6%date=2024-06-20 · board=Verified · system=SWE-agent · model_tag=claude-3-5-sonnet-2024102220 Jun 2024swebench.comT2 History
149#149Qwen2.5 (7B + 72B)Qwen32.8%date=2025-03-06 · board=Verified · system=SWE-Fixer · model_tag=Qwen 2.56 Mar 2025swebench.comT2 History
150#150gpt-4oOpenAI32.6%date=2024-06-12 · board=Verified · system=MASAI · model_tag=gpt-4o-2024-08-0612 Jun 2024swebench.comT2 History
151#151Undisclosed32%date=2024-11-20 · board=Verified · system=Artemis Agent v1 · submission=20241120_artemis_agent20 Nov 2024swebench.comT2 History
152#152Undisclosed31.6%date=2024-10-07 · board=Verified · system=nFactorial · submission=20241007_nfactorial7 Oct 2024swebench.comT2 History
153#153Qwen2.5 (7B + 72B)Qwen30.2%date=2024-11-28 · board=Verified · system=SWE-Fixer · model_tag=Qwen 2.528 Nov 2024swebench.comT2 History
154#154Lingma SWE-GPT 72b (v0925)OpenAI28.8%date=2024-10-02 · board=Verified · system=Lingma Agent · submission=20241002_lingma-agent_lingma-swe-gpt-72b2 Oct 2024swebench.comT2 History
155#155Gemini 2.5 FlashGoogle28.73%date=2025-07-26 · board=Verified · system=mini-SWE-agent · model_tag=gemini-2.5-flash26 Jul 2025swebench.comT2 History
156#156gpt-4oOpenAI27%date=2024-10-16 · board=Verified · system=EPAM AI/Run Developer Agent · model_tag=gpt-4o-2024-08-0616 Oct 2024swebench.comT2 History
157#157gpt-4oOpenAI26.2%date=2024-06-15 · board=Verified · system=AppMap Navie · model_tag=gpt-4o-2024-05-1315 Jun 2024swebench.comT2 History
158#158gpt-oss-120bOpenAI26%date=2025-08-07 · board=Verified · system=mini-SWE-agent · model_tag=gpt-oss-120b7 Aug 2025swebench.comT2 History
159#159Undisclosed25.8%date=2024-10-01 · board=Verified · system=nFactorial · submission=20241001_nfactorial1 Oct 2024swebench.comT2 History
160#160Undisclosed25.6%date=2024-05-09 · board=Verified · system=Amazon Q Developer Agent · submission=20240509_amazon-q-developer-agent-20240430-dev9 May 2024swebench.comT2 History
161#161Lingma SWE-GPT 72b (v0918)OpenAI25%date=2024-09-18 · board=Verified · system=Lingma Agent · submission=20240918_lingma-agent_lingma-swe-gpt-72b18 Sept 2024swebench.comT2 History
162#162gpt-4oOpenAI24%date=2024-08-20 · board=Verified · system=EPAM AI/Run Developer Agent · model_tag=gpt-4o-2024-08-0620 Aug 2024swebench.comT2 History
163#163gpt-4.1-miniOpenAI23.94%date=2025-07-20 · board=Verified · system=mini-SWE-agent · model_tag=gpt-4.1-mini-2025041420 Jul 2025swebench.comT2 History
164#164gpt-4oOpenAI23.2%date=2024-07-28 · board=Verified · system=SWE-agent · model_tag=gpt-4o-2024-05-1328 Jul 2024swebench.comT2 History
165#165MCTS Refine 7B23.2%date=2025-06-27 · board=Verified · system=MCTS-Refine-7B · model_tag=MCTS-Refine-7B27 Jun 2025swebench.comT2 History
166#166GPT-4 (1106)OpenAI22.4%date=2024-04-02 · board=Verified · system=SWE-agent · model_tag=gpt-4-1106-preview2 Apr 2024swebench.comT2 History
167#167gpt-4oOpenAI21.62%date=2025-07-20 · board=Verified · system=mini-SWE-agent · model_tag=gpt-4o-2024112020 Jul 2025swebench.comT2 History
168#168Llama 4 Maverick InstructMeta AI21.04%date=2025-07-20 · board=Verified · system=mini-SWE-agent · model_tag=llama-4-maverick-instruct20 Jul 2025swebench.comT2 History
169#169Lingma SWE-GPT 7b (v0925)OpenAI18.2%date=2024-10-02 · board=Verified · system=Lingma Agent · submission=20241002_lingma-agent_lingma-swe-gpt-7b2 Oct 2024swebench.comT2 History
170#170claude-3-opusAnthropic15.8%date=2024-04-02 · board=Verified · system=SWE-agent · model_tag=claude-3-opus-202402292 Apr 2024swebench.comT2 History
171#171Gemini 2.0 FlashGoogle13.52%date=2025-07-26 · board=Verified · system=mini-SWE-agent · model_tag=gemini-2.0-flash26 Jul 2025swebench.comT2 History
172#172Lingma SWE-GPT 7b (v0918)OpenAI10.2%date=2024-09-18 · board=Verified · system=Lingma Agent · submission=20240918_lingma-agent_lingma-swe-gpt-7b18 Sept 2024swebench.comT2 History
173#173Llama 4 Scout InstructMeta AI9.06%date=2025-07-20 · board=Verified · system=mini-SWE-agent · model_tag=llama-4-scout-instruct20 Jul 2025swebench.comT2 History
174#174Qwen2.5 Coder 32B InstructQwen9%date=2025-08-03 · board=Verified · system=mini-SWE-agent · model_tag=Qwen2.5-Coder-32B-Instruct3 Aug 2025swebench.comT2 History
175#175claude-3-opusAnthropic7%date=2024-04-02 · board=Verified · system=RAG baseline · model_tag=claude-3-opus-202402292 Apr 2024swebench.comT2 History
176#176claude-2Anthropic4.4%date=2023-10-10 · board=Verified · system=RAG baseline · model_tag=claude-210 Oct 2023swebench.comT2 History
177#177GPT-4 (1106)OpenAI2.8%date=2024-04-02 · board=Verified · system=RAG baseline · model_tag=gpt-4-1106-preview2 Apr 2024swebench.comT2 History
178#178SWE-Llama 7BMeta AI1.4%date=2023-10-10 · board=Verified · system=RAG baseline · model_tag=SWE-Llama10 Oct 2023swebench.comT2 History
179#179SWE-Llama 13BMeta AI1.2%date=2023-10-10 · board=Verified · system=RAG baseline · submission=20231010_rag_swellama13b10 Oct 2023swebench.comT2 History
180#180GPT-3.5OpenAI0.4%date=2023-10-10 · board=Verified · system=RAG baseline · submission=20231010_rag_gpt3510 Oct 2023swebench.comT2 History

Scores are reported as published, with their evaluation configuration (harness, prompting, judge). The bar is relative to the best score on this page. Results with different configs are not directly comparable — see methodology.

The config filter matches a value inside each result's configuration (server-side, `config=` on the API). Chips are the values shared by several rows on the first page; per-model identifiers are not offered.