Timeline of Anthropic

claude-35-sonnet scores 6.19% on SWE-bench Multimodal

claude-35-sonnet scores 11.41% on SWE-bench Multimodal

Claude Sonnet 3.5 scores 11.99% on SWE-bench Multimodal

Claude Sonnet 3.5 scores 12.19% on SWE-bench Multimodal

claude-35-sonnet scores 25.34% on SWE-bench Multimodal

Claude 3.7 Sonnet scores 30.56% on SWE-bench Multimodal

Claude 3.7 Sonnet scores 31.33% on SWE-bench Multimodal

Claude Sonnet 4 scores 34.43% on SWE-bench Multimodal

Claude 4 Sonnet scores 35.59% on SWE-bench Multimodal

claude-2 scores 3% on SWE-bench Lite

claude-3-opus scores 4.33% on SWE-bench Lite

claude-3-opus scores 11.67% on SWE-bench Lite

Multiple scores 21.67% on SWE-bench Lite

claude-35-sonnet scores 23% on SWE-bench Lite

Multiple scores 23.67% on SWE-bench Lite

GPT-4o & Claude 3 Opus scores 26.33% on SWE-bench Lite

CodeAct v1.8 scores 26.67% on SWE-bench Lite

claude-35-sonnet scores 26.67% on SWE-bench Lite

claude-3-5-sonnet-20241022 scores 30% on SWE-bench Lite

Multiple scores 36% on SWE-bench Lite

claude-35-sonnet scores 37% on SWE-bench Lite

claude-35-sonnet scores 38.33% on SWE-bench Lite

claude-35-sonnet scores 39% on SWE-bench Lite

claude-35-sonnet scores 40.67% on SWE-bench Lite

claude-35-sonnet scores 41% on SWE-bench Lite

Multiple scores 41% on SWE-bench Lite

claude-35-sonnet scores 41.33% on SWE-bench Lite

Multiple scores 41.67% on SWE-bench Lite

CodeAct v2.1 (claude-3-5-sonnet-20241022) scores 41.67% on SWE-bench Lite

claude-35-sonnet scores 42.67% on SWE-bench Lite

claude-35-sonnet scores 44.67% on SWE-bench Lite

claude-35-sonnet scores 46% on SWE-bench Lite

Multiple scores 47% on SWE-bench Lite

claude-35-sonnet scores 47.33% on SWE-bench Lite

Claude 3.7 Sonnet scores 48% on SWE-bench Lite

Multiple scores 48.33% on SWE-bench Lite

Multiple scores 49% on SWE-bench Lite

claude-35-sonnet scores 51.33% on SWE-bench Lite

Multiple scores 51.67% on SWE-bench Lite

Claude 4 Sonnet scores 56.67% on SWE-bench Lite

Claude 4 Sonnet scores 58.33% on SWE-bench Lite

Multiple scores 60% on SWE-bench Lite

Claude 4 Sonnet scores 60.33% on SWE-bench Lite

claude-2 scores 4.4% on SWE-bench Verified

claude-3-opus scores 7% on SWE-bench Verified

claude-3-opus scores 15.8% on SWE-bench Verified

claude-35-sonnet scores 33.6% on SWE-bench Verified

claude-35-sonnet scores 39.6% on SWE-bench Verified

Multiple scores 40.6% on SWE-bench Verified

Claude Haiku 3.5 scores 40.6% on SWE-bench Verified

claude-35-sonnet scores 40.6% on SWE-bench Verified

Multiple scores 41.6% on SWE-bench Verified

claude-3-5-sonnet-20241022 scores 46.2% on SWE-bench Verified

Multiple scores 46.6% on SWE-bench Verified

Multiple scores 47.2% on SWE-bench Verified

Multiple scores 48.6% on SWE-bench Verified

claude-35-sonnet scores 49% on SWE-bench Verified

Multiple scores 49.2% on SWE-bench Verified

claude-35-sonnet scores 50.8% on SWE-bench Verified

claude-3-5-sonnet-20241022 scores 51.6% on SWE-bench Verified

claude-35-sonnet scores 51.8% on SWE-bench Verified

Claude 3.7 Sonnet scores 52.8% on SWE-bench Verified

CodeAct v2.1 (claude-3-5-sonnet-20241022) scores 53% on SWE-bench Verified

claude-35-sonnet scores 54.2% on SWE-bench Verified

claude-35-sonnet scores 55.4% on SWE-bench Verified

Claude Sonnet 4 scores 57% on SWE-bench Verified

Multiple scores 57.2% on SWE-bench Verified

Multiple scores 58.2% on SWE-bench Verified

claude-35-sonnet scores 60.2% on SWE-bench Verified

swe-search scores 62.2% on SWE-bench Verified

Claude 3.7 Sonnet w/ Review Heavy scores 62.4% on SWE-bench Verified

claude-35-sonnet scores 62.8% on SWE-bench Verified

Claude 3.7 Sonnet scores 63.2% on SWE-bench Verified

claude-35-sonnet scores 63.4% on SWE-bench Verified

Claude 4 Sonnet scores 64.93% on SWE-bench Verified

Claude 3.7 Sonnet scores 66.4% on SWE-bench Verified

Claude 4.5 Haiku scores 66.6% on SWE-bench Verified

Claude 4 Sonnet scores 66.6% on SWE-bench Verified

claude-4-opus scores 67.6% on SWE-bench Verified

Multiple scores 68.2% on SWE-bench Verified

Multiple scores 70% on SWE-bench Verified

Multiple scores 70.2% on SWE-bench Verified

Claude 4 Sonnet scores 70.4% on SWE-bench Verified

Multiple scores 70.4% on SWE-bench Verified

Claude 4.5 Sonnet scores 70.6% on SWE-bench Verified

Claude 4 Sonnet scores 70.8% on SWE-bench Verified

Multiple scores 71% on SWE-bench Verified

Multiple scores 71.2% on SWE-bench Verified

Claude 4 Sonnet scores 71.2% on SWE-bench Verified

Claude 4.5 Sonnet scores 71.4% on SWE-bench Verified

Claude 4 Sonnet scores 72.4% on SWE-bench Verified

Multiple scores 73% on SWE-bench Verified

claude-4-opus scores 73.2% on SWE-bench Verified

Multiple scores 73.8% on SWE-bench Verified

Claude Opus 4.5 scores 74.4% on SWE-bench Verified

Multiple scores 74.4% on SWE-bench Verified

Multiple scores 74.6% on SWE-bench Verified

Claude 4 Sonnet scores 74.6% on SWE-bench Verified

Claude 4.5 Sonnet scores 74.8% on SWE-bench Verified

Claude Sonnet 4 scores 74.8% on SWE-bench Verified

Multiple scores 75.2% on SWE-bench Verified

Claude Opus 4.6 scores 75.6% on SWE-bench Verified

Multiple scores 75.6% on SWE-bench Verified

Multiple scores 76.4% on SWE-bench Verified

Claude Opus 4.5 scores 76.8% on SWE-bench Verified

Multiple scores 76.8% on SWE-bench Verified

Claude 4 Sonnet scores 76.8% on SWE-bench Verified

Claude Opus 4.5 scores 79.2% on SWE-bench Verified

Claude Opus 4.5 scores 79.2% on SWE-bench Verified

claude-2 scores 1.96% on SWE-bench (full test split)

claude-3-opus scores 3.79% on SWE-bench (full test split)

claude-3-opus scores 10.51% on SWE-bench (full test split)

claude-35-sonnet scores 18.13% on SWE-bench (full test split)

claude-3-5-sonnet-20241022 scores 24.89% on SWE-bench (full test split)

CodeAct v2.1 (claude-3-5-sonnet-20241022) scores 29.38% on SWE-bench (full test split)

Claude 3.7 Sonnet scores 33.83% on SWE-bench (full test split)

Multiple scores 44.25% on SWE-bench (full test split)

Claude Opus 4.5 scores 52.62% on SWE-bench (full test split)

Claude 4.5 Haiku scores 64.7% on SWE-bench Multilingual

Claude 4.5 Sonnet scores 67% on SWE-bench Multilingual

Claude Opus 4.5 scores 70.7% on SWE-bench Multilingual

Claude Opus 4.6 scores 72% on SWE-bench Multilingual

Claude Fable 5.1 Max Effort scores 72.988% on LiveBench

Claude Fable 5.1 Max Effort scores 89.501% on LiveBench

Claude Fable 5.1 Max Effort scores 80.276% on LiveBench

Claude Fable 5.1 Max Effort scores 97.007% on LiveBench

Claude Fable 5.1 Max Effort scores 66.061% on LiveBench

Claude Fable 5.1 Max Effort scores 86.375% on LiveBench

Claude Fable 5.1 Max Effort scores 91.692% on LiveBench

Claude Fable 5.1 Max Effort scores 83.414% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 72.034% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 79.659% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 66.033% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 94.317% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 50.505% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 81.828% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 89.192% on LiveBench

Claude 4.8 Opus Thinking Max Effort scores 76.224% on LiveBench

Claude Fable 5 xHigh Effort scores 75.771% on LiveBench

Claude Fable 5 xHigh Effort scores 90.684% on LiveBench

Claude Fable 5 xHigh Effort scores 80.538% on LiveBench

Claude Fable 5 xHigh Effort scores 95.986% on LiveBench

Claude Fable 5 xHigh Effort scores 62.172% on LiveBench

Claude Fable 5 xHigh Effort scores 85.992% on LiveBench

Claude Fable 5 xHigh Effort scores 89.654% on LiveBench

Claude Fable 5 xHigh Effort scores 82.971% on LiveBench

Claude Sonnet 5 xHigh Effort scores 63.858% on LiveBench

Claude Sonnet 5 xHigh Effort scores 74.97% on LiveBench

Claude Sonnet 5 xHigh Effort scores 71.741% on LiveBench

Claude Sonnet 5 xHigh Effort scores 92.942% on LiveBench

Claude Sonnet 5 xHigh Effort scores 59.394% on LiveBench

Claude Sonnet 5 xHigh Effort scores 80.68% on LiveBench

Claude Sonnet 5 xHigh Effort scores 88.692% on LiveBench

Claude Sonnet 5 xHigh Effort scores 76.04% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 63.312% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 83.27% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 69.893% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 89.317% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 48.99% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 78.184% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 88.673% on LiveBench

Claude 4.6 Opus Thinking High Effort scores 74.52% on LiveBench

Claude Haiku 3.5 scores 2.27% on Terminal-Bench

Claude Haiku 3.5 scores 10.11% on Terminal-Bench

Claude Haiku 3.5 scores 24.56% on τ²-bench

Claude Haiku 3.5 scores 45.61% on MMMU-Pro

Claude Haiku 3.5 scores 42.79% on IFBench

Claude Haiku 3.5 scores 3.63% on Humanity's Last Exam

Claude Haiku 3.5 scores 40.81% on GPQA

Claude Haiku 3.5 scores 8.94 on Artificial Analysis Intelligence Index

claude-fable-5-1-low scores 85.02% on Terminal-Bench

claude-fable-5-1-low scores 40.4% on Terminal-Bench

claude-fable-5-1-low scores 56.71% on SciCode

claude-fable-5-1-low scores 48.89% on Humanity's Last Exam

claude-fable-5-1-low scores 88.08% on GPQA

claude-fable-5-1-low scores 47.04 on Artificial Analysis Intelligence Index

claude-35-sonnet scores 3.69% on Humanity's Last Exam

claude-35-sonnet scores 59.9% on GPQA

claude-35-sonnet scores 7.88 on Artificial Analysis Intelligence Index

claude-opus-5-xhigh scores 88.01% on Terminal-Bench

claude-opus-5-xhigh scores 46.46% on Terminal-Bench

claude-opus-5-xhigh scores 83.99% on MMMU-Pro

claude-opus-5-xhigh scores 55.67% on SciCode

claude-opus-5-xhigh scores 54.4% on Humanity's Last Exam

claude-opus-5-xhigh scores 93.74% on GPQA

claude-opus-5-xhigh scores 49.65 on Artificial Analysis Intelligence Index

claude-3-7-sonnet-thinking scores 21.21% on Terminal-Bench

claude-3-7-sonnet-thinking scores 54.68% on τ²-bench

claude-3-7-sonnet-thinking scores 48.3% on IFBench

claude-3-7-sonnet-thinking scores 9.67% on Humanity's Last Exam

claude-3-7-sonnet-thinking scores 77.17% on GPQA

claude-3-7-sonnet-thinking scores 17.71 on Artificial Analysis Intelligence Index

claude-sonnet-5-low scores 2.53% on Terminal-Bench

claude-sonnet-5-low scores 50.12% on SciCode

claude-sonnet-5-low scores 21.92% on Humanity's Last Exam

claude-sonnet-5-low scores 24.74 on Artificial Analysis Intelligence Index

Claude 3 Haiku scores 0.76% on Terminal-Bench

Claude 3 Haiku scores 21.05% on τ²-bench

Claude 3 Haiku scores 30.81% on MMMU-Pro

Claude 3 Haiku scores 36.12% on IFBench

Claude 3 Haiku scores 4.08% on Humanity's Last Exam

Claude 3 Haiku scores 37.37% on GPQA

Claude 3 Haiku scores 5.58 on Artificial Analysis Intelligence Index

claude-opus-5-low scores 76.4% on Terminal-Bench

claude-opus-5-low scores 26.26% on Terminal-Bench

claude-opus-5-low scores 79.83% on MMMU-Pro

claude-opus-5-low scores 49.19% on SciCode

claude-opus-5-low scores 43.42% on Humanity's Last Exam

claude-opus-5-low scores 88.89% on GPQA

claude-opus-5-low scores 39.79 on Artificial Analysis Intelligence Index

Claude Sonnet 5 scores 80.52% on Terminal-Bench

Claude Sonnet 5 scores 14.14% on Terminal-Bench

Claude Sonnet 5 scores 77.28% on MMMU-Pro

Claude Sonnet 5 scores 54.28% on SciCode

Claude Sonnet 5 scores 41.29% on Humanity's Last Exam

Claude Sonnet 5 scores 91.11% on GPQA

Claude Sonnet 5 scores 38.36 on Artificial Analysis Intelligence Index

claude-opus-4-5-thinking scores 46.97% on Terminal-Bench

claude-opus-4-5-thinking scores 89.47% on τ²-bench

claude-opus-4-5-thinking scores 74.05% on MMMU-Pro

claude-opus-4-5-thinking scores 57.96% on IFBench

claude-opus-4-5-thinking scores 30.12% on Humanity's Last Exam

claude-opus-4-5-thinking scores 86.57% on GPQA

claude-opus-4-5-thinking scores 29.1 on Artificial Analysis Intelligence Index

Claude Sonnet 4.6 scores 46.21% on Terminal-Bench

Claude Sonnet 4.6 scores 79.53% on τ²-bench

Claude Sonnet 4.6 scores 70.58% on MMMU-Pro

Claude Sonnet 4.6 scores 41.16% on IFBench

Claude Sonnet 4.6 scores 13.35% on Humanity's Last Exam

Claude Sonnet 4.6 scores 79.9% on GPQA

Claude Sonnet 4.6 scores 24.69 on Artificial Analysis Intelligence Index

claude-opus-4-7-non-reasoning scores 54.55% on Terminal-Bench

claude-opus-4-7-non-reasoning scores 73.98% on τ²-bench

claude-opus-4-7-non-reasoning scores 76.36% on MMMU-Pro

claude-opus-4-7-non-reasoning scores 43.61% on IFBench

claude-opus-4-7-non-reasoning scores 33.32% on Humanity's Last Exam

claude-opus-4-7-non-reasoning scores 88.48% on GPQA

claude-opus-4-7-non-reasoning scores 30.93 on Artificial Analysis Intelligence Index

claude-fable-5-1-medium scores 88.01% on Terminal-Bench

claude-fable-5-1-medium scores 44.95% on Terminal-Bench

claude-fable-5-1-medium scores 56.37% on SciCode

claude-fable-5-1-medium scores 53.8% on Humanity's Last Exam

claude-fable-5-1-medium scores 88.59% on GPQA

claude-fable-5-1-medium scores 49.06 on Artificial Analysis Intelligence Index

Claude Fable 5.1 scores 91.39% on Terminal-Bench

Claude Fable 5.1 scores 52.02% on Terminal-Bench

Claude Fable 5.1 scores 63.08% on SciCode

Claude Fable 5.1 scores 59.13% on Humanity's Last Exam

Claude Fable 5.1 scores 93.74% on GPQA

Claude Fable 5.1 scores 53.37 on Artificial Analysis Intelligence Index

Claude Opus 4.5 scores 40.91% on Terminal-Bench

Claude Opus 4.5 scores 86.26% on τ²-bench

Claude Opus 4.5 scores 71.21% on MMMU-Pro

Claude Opus 4.5 scores 42.99% on IFBench

Claude Opus 4.5 scores 13.16% on Humanity's Last Exam

Claude Opus 4.5 scores 81.01% on GPQA

Claude Opus 4.5 scores 23.68 on Artificial Analysis Intelligence Index

Claude Opus 4.6 scores 48.48% on Terminal-Bench

Claude Opus 4.6 scores 84.8% on τ²-bench

Claude Opus 4.6 scores 72.54% on MMMU-Pro

Claude Opus 4.6 scores 44.63% on IFBench

Claude Opus 4.6 scores 19.09% on Humanity's Last Exam

Claude Opus 4.6 scores 84.04% on GPQA

Claude Opus 4.6 scores 26.35 on Artificial Analysis Intelligence Index

claude-4-opus scores 43.27% on IFBench

claude-4-opus scores 6.16% on Humanity's Last Exam

claude-4-opus scores 70.1% on GPQA

claude-4-opus scores 16.61 on Artificial Analysis Intelligence Index

claude-opus-4-6-adaptive scores 46.21% on Terminal-Bench

claude-opus-4-6-adaptive scores 92.11% on τ²-bench

claude-opus-4-6-adaptive scores 75.43% on MMMU-Pro

claude-opus-4-6-adaptive scores 53.13% on IFBench

claude-opus-4-6-adaptive scores 39.94% on Humanity's Last Exam

claude-opus-4-6-adaptive scores 89.6% on GPQA

claude-opus-4-6-adaptive scores 31.95 on Artificial Analysis Intelligence Index

claude-opus-5-high scores 87.64% on Terminal-Bench

claude-opus-5-high scores 45.96% on Terminal-Bench

claude-opus-5-high scores 82.43% on MMMU-Pro

claude-opus-5-high scores 55.44% on SciCode

claude-opus-5-high scores 52.83% on Humanity's Last Exam

claude-opus-5-high scores 93.74% on GPQA