BenchmarkActivecategory · instruction-following
IFBench
precise instruction following with novel constraints
quality51
Updated 9 h ago · first seen 11 Sept 2026
bench_01M293SPGX8Q1PSHHMPKC9R37H
- Metric
- accuracy · %
- Direction
- Higher is better
- Results
- 450
- Leader
- grok-4-3-medium 83.33%
Score history · Claude Opus 4.7 1 row
Not enough history to chart — a single observation (58.64% on 11 Sept 2026). Rows under different configurations count separately; the list below shows each one.
- 58.64%aa_slug=claude-opus-4-7 · evaluator=Artificial Analysis · index_version=4.311 Sept 2026
Leaderboard 450 current results
Select models with +, then open Compare.
| # | Model | Score | Config | Evaluated | Source | Actions |
|---|---|---|---|---|---|---|
| #201tri-21b-think-previewTrillion Labs | 47.14% | aa_slug=tri-21b-think-preview · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #202Llama 3.3 70BMeta AI | 47.07% | aa_slug=llama-3-3-instruct-70b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #203qwen3-5-27b-non-reasoningAlibaba Group | 46.94% | aa_slug=qwen3-5-27b-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #204grok-3SpaceXAI | 46.94% | aa_slug=grok-3 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #205glm-4-7-flash-non-reasoningZ.ai (Zhipu AI) | 46.26% | aa_slug=glm-4-7-flash-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #206cogito-v2-1-reasoningDeep Cogito | 46.26% | aa_slug=cogito-v2-1-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #207gpt-5-5-non-reasoningOpenAI | 46.12% | aa_slug=gpt-5-5-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #208qwen3-235b-a22b-instruct-2507Alibaba Group | 46.05% | aa_slug=qwen3-235b-a22b-instruct-2507 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #209lfm2-24b-a2bLiquid AI | 45.85% | aa_slug=lfm2-24b-a2b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #210grok-3-mini-reasoningSpaceXAI | 45.85% | aa_slug=grok-3-mini-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #211deepseek-v4-pro-0424-non-reasoningDeepSeek | 45.78% | aa_slug=deepseek-v4-pro-0424-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #212qwen3-6-27b-non-reasoningAlibaba Group | 45.71% | aa_slug=qwen3-6-27b-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #213midm-250-pro-rsnsftKorea Telecom | 45.58% | aa_slug=midm-250-pro-rsnsft · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #214gpt-5-mini-minimalOpenAI | 45.58% | aa_slug=gpt-5-mini-minimal · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #215gpt-5-minimalOpenAI | 45.58% | aa_slug=gpt-5-minimal · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #216gemma-4-26b-a4b-non-reasoningGoogle | 45.44% | aa_slug=gemma-4-26b-a4b-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #217Claude Sonnet 4Anthropic | 45.37% | aa_slug=claude-4-sonnet · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #218gemma-4-12b-non-reasoningGoogle | 45.17% | aa_slug=gemma-4-12b-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #219qwen3-vl-30b-a3b-reasoningAlibaba Group | 45.1% | aa_slug=qwen3-vl-30b-a3b-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #220gpt-5-chatgptOpenAI | 45.03% | aa_slug=gpt-5-chatgpt · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #221Claude Opus 4.6Anthropic | 44.63% | aa_slug=claude-opus-4-6 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #222ring-2-6-1tinclusionAI | 44.56% | aa_slug=ring-2-6-1t · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #223ring-1tinclusionAI | 44.56% | aa_slug=ring-1t · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #224qwen3-5-35b-a3b-non-reasoningAlibaba Group | 44.49% | aa_slug=qwen3-5-35b-a3b-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #225Magistral Small 1.2Mistral AI | 44.35% | aa_slug=magistral-small-2509 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #226granite-4.1-30bIBM | 44.35% | aa_slug=granite-4-1-30b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #227kimi-k2-6-non-reasoningMoonshot AI | 44.29% | aa_slug=kimi-k2-6-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #228gemma-4-E4BGoogle | 44.22% | aa_slug=gemma-4-e4b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #229Qwen3 MaxQwen | 44.15% | aa_slug=qwen3-max · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #230GLM 4.5Z.ai (Zhipu AI) | 44.08% | aa_slug=glm-4.5 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #231Claude 3.7 SonnetAnthropic | 44.01% | aa_slug=claude-3-7-sonnet · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #232LFM2.5-1.2B-InstructLiquid AI | 43.81% | aa_slug=lfm2-5-1-2b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #233kimi-k2-5-non-reasoningMoonshot AI | 43.67% | aa_slug=kimi-k2-5-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #234claude-opus-4-7-non-reasoningAnthropic | 43.61% | aa_slug=claude-opus-4-7-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #235gemini-2-5-flash-preview-09-2025Google | 43.54% | aa_slug=gemini-2-5-flash-preview-09-2025 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #236glm-4-6-reasoningZ.ai (Zhipu AI) | 43.4% | aa_slug=glm-4-6-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #237qwen3-omni-30b-a3b-reasoningAlibaba Group | 43.4% | aa_slug=qwen3-omni-30b-a3b-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #238claude-4-opusAnthropic | 43.27% | aa_slug=claude-4-opus · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #239ring-flash-2-0inclusionAI | 43.27% | aa_slug=ring-flash-2-0 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #240gpt-5-1-non-reasoningOpenAI | 43.2% | aa_slug=gpt-5-1-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #241deepseek-v3-2-0925DeepSeek | 43.13% | aa_slug=deepseek-v3-2-0925 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #242longcat-flash-liteLongCat | 43.06% | aa_slug=longcat-flash-lite · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #243gpt-4.1OpenAI | 42.99% | aa_slug=gpt-4-1 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #244Claude Opus 4.5Anthropic | 42.99% | aa_slug=claude-opus-4-5 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #245Magistral Medium 1.2Mistral AI | 42.99% | aa_slug=magistral-medium-2509 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #246Llama 4 MaverickMeta AI | 42.99% | aa_slug=llama-4-maverick · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #247Claude Haiku 3.5Anthropic | 42.79% | aa_slug=claude-3-5-haiku · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #248mimo-v2-5-pro-non-reasoningXiaomi | 42.72% | aa_slug=mimo-v2-5-pro-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #249Qwen3 VL 235B A22B InstructQwen | 42.65% | aa_slug=qwen3-vl-235b-a22b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #250Claude Sonnet 4.5Anthropic | 42.65% | aa_slug=claude-4-5-sonnet · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #251claude-sonnet-4-6-non-reasoning-low-effortAnthropic | 42.38% | aa_slug=claude-sonnet-4-6-non-reasoning-low-effort · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #252jt-35b-flashChina Mobile | 42.04% | aa_slug=jt-35b-flash · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #253Claude Haiku 4.5Anthropic | 42.04% | aa_slug=claude-4-5-haiku · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #254Seed-OSS-36B-InstructByteDance | 41.9% | aa_slug=seed-oss-36b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #255lfm2-5-1-2b-thinkingLiquid AI | 41.84% | aa_slug=lfm2-5-1-2b-thinking · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #256gemini-2-5-flash-lite-preview-09-2025Google | 41.84% | aa_slug=gemini-2-5-flash-lite-preview-09-2025 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #257MiniMax-M1-80kMiniMax | 41.77% | aa_slug=minimax-m1-80k · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #258Kimi K2 0905Moonshot AI | 41.7% | aa_slug=kimi-k2-0905 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #259Olmo-3-7B-ThinkAllen Institute for AI | 41.5% | aa_slug=olmo-3-7b-think · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #260deepseek-v3-1-reasoningDeepSeek | 41.5% | aa_slug=deepseek-v3-1-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #261qwen3-30b-a3b-instruct-reasoningAlibaba Group | 41.5% | aa_slug=qwen3-30b-a3b-instruct-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #262Kimi K2 0711Moonshot AI | 41.5% | aa_slug=kimi-k2 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #263Grok Build 0.1xAI | 41.36% | aa_slug=grok-code-fast-1 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #264ernie-5-0-thinking-previewBaidu | 41.36% | aa_slug=ernie-5-0-thinking-preview · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #265MiniMax-M1-40kMiniMax | 41.22% | aa_slug=minimax-m1-40k · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #266DeepSeek V3.1 TerminusDeepSeek | 41.22% | aa_slug=deepseek-v3-1-terminus · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #267Claude Sonnet 4.6Anthropic | 41.16% | aa_slug=claude-sonnet-4-6 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #268nova-2-0-omniAmazon Web Services | 41.09% | aa_slug=nova-2-0-omni · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #269DeepSeek V3 0324DeepSeek | 41.02% | aa_slug=deepseek-v3-0324 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #270k2-v2-lowMBZUAI Institute of Foundation Models | 40.95% | aa_slug=k2-v2-low · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #271Qwen3 14BQwen | 40.54% | aa_slug=qwen3-14b-instruct-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #272nova-2-0-liteAmazon Web Services | 40.54% | aa_slug=nova-2-0-lite · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #273gemma-4-e4b-non-reasoningGoogle | 40.54% | aa_slug=gemma-4-e4b-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #274qwen3-coder-480b-a35b-instructAlibaba Group | 40.48% | aa_slug=qwen3-coder-480b-a35b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #275Gemini 2.0 FlashGoogle | 40.2% | aa_slug=gemini-2-0-flash · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #276mimo-v2-flashXiaomi | 39.93% | aa_slug=mimo-v2-flash · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #277qwen3-vl-8b-reasoningAlibaba Group | 39.86% | aa_slug=qwen3-vl-8b-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #278Mistral Medium 3.1Mistral AI | 39.8% | aa_slug=mistral-medium-3-1 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #279Qwen3 Next 80B A3B InstructQwen | 39.66% | aa_slug=qwen3-next-80b-a3b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #280k-exaone-non-reasoningLG AI Research | 39.59% | aa_slug=k-exaone-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #281deepseek-r1DeepSeek | 39.59% | aa_slug=deepseek-r1 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #282Llama 4 ScoutMeta AI | 39.52% | aa_slug=llama-4-scout · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #283llama-3-3-nemotron-super-49bNVIDIA | 39.46% | aa_slug=llama-3-3-nemotron-super-49b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #284Mistral Medium 3Mistral AI | 39.25% | aa_slug=mistral-medium-3 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #285Qwen3 VL 32B InstructQwen | 39.18% | aa_slug=qwen3-vl-32b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #286olmo-3-1-32b-instructAllen Institute for AI | 39.18% | aa_slug=olmo-3-1-32b-instruct · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #287ernie-4-5-300b-a47bBaidu | 39.12% | aa_slug=ernie-4-5-300b-a47b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #288Llama-3.1-405BMeta AI | 39.05% | aa_slug=llama-3-1-instruct-405b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #289Gemini 2.5 FlashGoogle | 38.98% | aa_slug=gemini-2-5-flash · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #290deepseek-r1-0120DeepSeek | 38.98% | aa_slug=deepseek-r1-0120 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #291gpt-5-4-mini-non-reasoningOpenAI | 38.84% | aa_slug=gpt-5-4-mini-non-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #292QwQ-32BAlibaba Group | 38.78% | aa_slug=qwq-32b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #293qwen3-235b-a22b-instruct-reasoningAlibaba Group | 38.71% | aa_slug=qwen3-235b-a22b-instruct-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #294granite-4.1-8bIBM | 38.64% | aa_slug=granite-4-1-8b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #295gpt-4.1-miniOpenAI | 38.3% | aa_slug=gpt-4-1-mini · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #296llama-3-1-nemotron-ultra-253b-v1-reasoningNVIDIA | 38.16% | aa_slug=llama-3-1-nemotron-ultra-253b-v1-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #297llama-3-3-nemotron-super-49b-reasoningNVIDIA | 38.1% | aa_slug=llama-3-3-nemotron-super-49b-reasoning · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #298olmo-2-32bAllen Institute for AI | 38.1% | aa_slug=olmo-2-32b · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #299nova-proAmazon Web Services | 38.1% | aa_slug=nova-pro · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History | |
| #300Devstral 2Mistral AI | 38.1% | aa_slug=devstral-2 · evaluator=Artificial Analysis · index_version=4.3 | — (obs. 11 Sept 2026) | artificialanalysis.aiT2 | History |
Scores are reported as published, with their evaluation configuration (harness, prompting, judge). The bar is relative to the best score on this page. Results with different configs are not directly comparable — see methodology.
Definition
- Category
- instruction-following
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 9 h agomedium
- Task
- precise instruction following with novel constraints
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 9 h agomedium
- Metric
- accuracy · %
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 9 h agomedium
- Direction
- Higher is better
- Paper
- https://arxiv.org/abs/2507.02833
Source:AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2observed 9 h agomedium
Each value shows its source, tier and observation time. Missing rows mean no source stated them. How results are recorded →
- Evaluated models
- grok-3-mini-reasoning, gemma-4-12B, grok-4-3-low, GPT-5.1-Codex Mini, Qwen3 Coder Next, KAT-Coder-Pro V2, nova-2-0-lite-reasoning-medium, lfm2-24b-a2b +442(450 total)
As of
Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.
Claim history
5 claims · 5 properties
Categorycategory1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| instruction-following | → current | current | AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2 | medium | curated |
Metricmetric1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| accuracy | → current | current | AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2 | medium | curated |
Paperpaper1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://arxiv.org/abs/2507.02833 | → current | current | AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2 | medium | curated |
Tasktask1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| precise instruction following with novel constraints | → current | current | AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2 | medium | curated |
Unitunit1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| % | → current | current | AI Atlas curated registry (YAML, versioned in git, every entry carries its source URL)T2 | medium | curated |
Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →
| Source | Document | Type | Tier | Last observed | Snapshots |
|---|---|---|---|---|---|
| Artificial Analysis | artificialanalysis.ai/leaderboards/models | leaderboard | T2· Quality secondary | 8 h ago | 1 |
Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.