Timeline of Meta AI

SWE-Llama 7B scores 1.33% on SWE-bench Lite

SWE-Llama 13B scores 1.2% on SWE-bench Verified

SWE-Llama 7B scores 1.4% on SWE-bench Verified

Llama 4 Scout Instruct scores 9.06% on SWE-bench Verified

Llama 4 Maverick Instruct scores 21.04% on SWE-bench Verified

Llama3-SWE-RL-70B scores 41.2% on SWE-bench Verified

SWE-Llama 7B scores 0.7% on SWE-bench (full test split)

SWE-Llama 13B scores 0.7% on SWE-bench (full test split)

muse-spark-1-3-xhigh scores 78.004% on LiveBench

muse-spark-1-3-xhigh scores 82.794% on LiveBench

muse-spark-1-3-xhigh scores 79.565% on LiveBench

muse-spark-1-3-xhigh scores 95.95% on LiveBench

muse-spark-1-3-xhigh scores 64.091% on LiveBench

muse-spark-1-3-xhigh scores 81.062% on LiveBench

muse-spark-1-3-xhigh scores 89.654% on LiveBench

muse-spark-1-3-xhigh scores 81.589% on LiveBench

Muse Spark 1.2 scores 74.325% on LiveBench

Muse Spark 1.2 scores 78.575% on LiveBench

Muse Spark 1.2 scores 76.458% on LiveBench

Muse Spark 1.2 scores 91.203% on LiveBench

Muse Spark 1.2 scores 57.576% on LiveBench

Muse Spark 1.2 scores 77.541% on LiveBench

Muse Spark 1.2 scores 90.005% on LiveBench

Muse Spark 1.2 scores 77.955% on LiveBench

Muse Spark 1.1 scores 69.638% on LiveBench

Muse Spark 1.1 scores 74.342% on LiveBench

Muse Spark 1.1 scores 72.548% on LiveBench

Muse Spark 1.1 scores 87.145% on LiveBench

Muse Spark 1.1 scores 58.535% on LiveBench

Muse Spark 1.1 scores 77.159% on LiveBench

Muse Spark 1.1 scores 87.731% on LiveBench

Muse Spark 1.1 scores 75.3% on LiveBench

Muse Spark 1.2 scores 80.15% on Terminal-Bench

Muse Spark 1.2 scores 7.07% on Terminal-Bench

Muse Spark 1.2 scores 57.41% on SciCode

Muse Spark 1.2 scores 45.46% on Humanity's Last Exam

Muse Spark 1.2 scores 90.4% on GPQA

Muse Spark 1.2 scores 39.8 on Artificial Analysis Intelligence Index

Llama-3.1-405B scores 6.82% on Terminal-Bench

Llama-3.1-405B scores 19.01% on τ²-bench

Llama-3.1-405B scores 39.05% on IFBench

Llama-3.1-405B scores 3.98% on Humanity's Last Exam

Llama-3.1-405B scores 51.52% on GPQA

Llama-3.1-405B scores 7.29 on Artificial Analysis Intelligence Index

Llama-3.1-70B scores 3.03% on Terminal-Bench

Llama-3.1-70B scores 15.2% on τ²-bench

Llama-3.1-70B scores 34.42% on IFBench

Llama-3.1-70B scores 4.47% on Humanity's Last Exam

Llama-3.1-70B scores 40.91% on GPQA

Llama-3.1-70B scores 6.6 on Artificial Analysis Intelligence Index

Muse Glimmer 30B scores 51.69% on Terminal-Bench

Muse Glimmer 30B scores 0.51% on Terminal-Bench

Muse Glimmer 30B scores 74.34% on MMMU-Pro

Muse Glimmer 30B scores 44.91% on SciCode

Muse Glimmer 30B scores 21.96% on Humanity's Last Exam

Muse Glimmer 30B scores 83.54% on GPQA

Muse Glimmer 30B scores 18.07 on Artificial Analysis Intelligence Index

llama-2-chat-70b scores 5.21% on Humanity's Last Exam

llama-2-chat-70b scores 32.73% on GPQA

llama-2-chat-70b scores 5.3 on Artificial Analysis Intelligence Index

llama-3-2-instruct-11b-vision scores 0.76% on Terminal-Bench

llama-3-2-instruct-11b-vision scores 14.62% on τ²-bench

llama-3-2-instruct-11b-vision scores 29.31% on MMMU-Pro

llama-3-2-instruct-11b-vision scores 30.41% on IFBench

llama-3-2-instruct-11b-vision scores 5.53% on Humanity's Last Exam

llama-3-2-instruct-11b-vision scores 22.12% on GPQA

llama-3-2-instruct-11b-vision scores 5.41 on Artificial Analysis Intelligence Index

llama-2-chat-13b scores 4.77% on Humanity's Last Exam

llama-2-chat-13b scores 32.12% on GPQA

llama-2-chat-13b scores 5.3 on Artificial Analysis Intelligence Index

Llama-3.2-3B scores 21.05% on τ²-bench

Llama-3.2-3B scores 26.19% on IFBench

Llama-3.2-3B scores 5.35% on Humanity's Last Exam

Llama-3.2-3B scores 25.45% on GPQA

Llama-3.2-3B scores 5.7 on Artificial Analysis Intelligence Index

Llama 3.1 8B scores 0.76% on Terminal-Bench

Llama 3.1 8B scores 1.5% on Terminal-Bench

Llama 3.1 8B scores 16.37% on τ²-bench

Llama 3.1 8B scores 28.57% on IFBench

Llama 3.1 8B scores 5.29% on Humanity's Last Exam

Llama 3.1 8B scores 25.86% on GPQA

Llama 3.1 8B scores 6.93 on Artificial Analysis Intelligence Index

Llama-3.2-1B scores 0% on Terminal-Bench

Llama-3.2-1B scores 0% on τ²-bench

Llama-3.2-1B scores 22.79% on IFBench

Llama-3.2-1B scores 5.46% on Humanity's Last Exam

Llama-3.2-1B scores 19.6% on GPQA

Llama-3.2-1B scores 4.83 on Artificial Analysis Intelligence Index

llama-3-instruct-8b scores 0% on Terminal-Bench

llama-3-instruct-8b scores 0% on τ²-bench

llama-3-instruct-8b scores 24.56% on IFBench

llama-3-instruct-8b scores 5.06% on Humanity's Last Exam

llama-3-instruct-8b scores 29.6% on GPQA

llama-3-instruct-8b scores 4.83 on Artificial Analysis Intelligence Index

Llama 4 Maverick scores 6.82% on Terminal-Bench

Llama 4 Maverick scores 7.87% on Terminal-Bench

Llama 4 Maverick scores 0% on Terminal-Bench

Llama 4 Maverick scores 17.84% on τ²-bench

Llama 4 Maverick scores 62.14% on MMMU-Pro

Llama 4 Maverick scores 42.99% on IFBench

Llama 4 Maverick scores 31.71% on SciCode

Llama 4 Maverick scores 4.91% on Humanity's Last Exam

Llama 4 Maverick scores 67.07% on GPQA

Llama 4 Maverick scores 9.3 on Artificial Analysis Intelligence Index

Llama 3.3 70B scores 3.03% on Terminal-Bench

Llama 3.3 70B scores 4.87% on Terminal-Bench

Llama 3.3 70B scores 26.61% on τ²-bench

Llama 3.3 70B scores 47.07% on IFBench

Llama 3.3 70B scores 3.56% on Humanity's Last Exam

Llama 3.3 70B scores 49.8% on GPQA

Llama 3.3 70B scores 7.66 on Artificial Analysis Intelligence Index

Muse Spark 1.3 scores 84.27% on Terminal-Bench

Muse Spark 1.3 scores 33.33% on Terminal-Bench

Muse Spark 1.3 scores 58.8% on SciCode

Muse Spark 1.3 scores 48.7% on Humanity's Last Exam

Muse Spark 1.3 scores 93.54% on GPQA

Muse Spark 1.3 scores 48.17 on Artificial Analysis Intelligence Index

llama-3-2-instruct-90b-vision scores 39.48% on MMMU-Pro

llama-3-2-instruct-90b-vision scores 4.49% on Humanity's Last Exam

llama-3-2-instruct-90b-vision scores 43.23% on GPQA

llama-3-2-instruct-90b-vision scores 6.41 on Artificial Analysis Intelligence Index

llama-3-instruct-70b scores 0.76% on Terminal-Bench

llama-3-instruct-70b scores 0% on τ²-bench

llama-3-instruct-70b scores 37.07% on IFBench

llama-3-instruct-70b scores 4.52% on Humanity's Last Exam

llama-3-instruct-70b scores 37.88% on GPQA

llama-3-instruct-70b scores 5.45 on Artificial Analysis Intelligence Index

Llama 4 Scout scores 1.52% on Terminal-Bench

Llama 4 Scout scores 3.75% on Terminal-Bench

Llama 4 Scout scores 0% on Terminal-Bench

Llama 4 Scout scores 15.5% on τ²-bench

Llama 4 Scout scores 52.95% on MMMU-Pro

Llama 4 Scout scores 39.52% on IFBench

Llama 4 Scout scores 21.3% on SciCode

Llama 4 Scout scores 3.78% on Humanity's Last Exam

Llama 4 Scout scores 58.69% on GPQA

Llama 4 Scout scores 6.45 on Artificial Analysis Intelligence Index

muse-spark-1-3-xhigh scores 85.39% on Terminal-Bench

muse-spark-1-3-xhigh scores 16.67% on Terminal-Bench

muse-spark-1-3-xhigh scores 82.02% on MMMU-Pro

muse-spark-1-3-xhigh scores 59.72% on SciCode

muse-spark-1-3-xhigh scores 47.45% on Humanity's Last Exam

muse-spark-1-3-xhigh scores 94.14% on GPQA

muse-spark-1-3-xhigh scores 45.16 on Artificial Analysis Intelligence Index

Muse Spark 1.1 scores 77.9% on Terminal-Bench

Muse Spark 1.1 scores 6.06% on Terminal-Bench

Muse Spark 1.1 scores 58.8% on SciCode

Muse Spark 1.1 scores 46.2% on Humanity's Last Exam

Muse Spark 1.1 scores 89.8% on GPQA

Muse Spark 1.1 scores 34.27 on Artificial Analysis Intelligence Index

muse-spark scores 45.45% on Terminal-Bench

muse-spark scores 62.17% on Terminal-Bench

muse-spark scores 91.52% on τ²-bench

muse-spark scores 80.52% on MMMU-Pro

muse-spark scores 75.92% on IFBench

muse-spark scores 40.69% on Humanity's Last Exam

muse-spark scores 88.38% on GPQA

muse-spark scores 31.3 on Artificial Analysis Intelligence Index

llama-65b scores 5.03 on Artificial Analysis Intelligence Index

llama-2-chat-7b scores 5.33% on Humanity's Last Exam

llama-2-chat-7b scores 22.73% on GPQA

llama-2-chat-7b scores 5.71 on Artificial Analysis Intelligence Index

Llama 4 Maverick scores 99.1% on Aider polyglot

Llama 4 Maverick scores 15.6% on Aider polyglot