IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
Updated 2 h ago · first seen 11 Sept 2026
paper_01M294WYEF63VYT7JKS39D78E7
- Published
- 9 Sept 2026
- T2 · 4 h ago
- arXiv
- 2609.10539
- T2 · 4 h ago
Abstract
A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect.
Authors 5
Yiling Ma, Yilun Zhao, Sihong Wu, Manasi Patwardhan, Arman Cohan
Specification
- Official page
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 4 h agomedium
- arXiv id
- 2609.10539
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 4 h agomedium
- Github repo
- Yiling-Ma/IdeaAMBIG
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 4 h agomedium
- Hf paper url
- https://huggingface.co/papers/2609.10539
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 4 h agomedium
- Github stars
- 0
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 2 h agomedium
- Hf comments
- 1
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 2 h agomedium
- Upvotes
- 0
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 2 h agomedium
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 4 h agomedium
- Published
- 9 Sept 2026
Source:Hugging Face Hub (public pages, model cards, papers)T2observed 4 h agomedium
Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →
Provenance
Attributed facts
11
Source tiers
T211
Freshest observation
2 h ago
Conflicts
None
No models linked to this paper yet.
No relations recorded.
As of
Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.
Claim history · Abstract
Abstractabstract1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| A research idea may be novel, coherent, and scientifically plausible, yet its proposed method may remain insufficiently specified for faithful implementation. We study the codification readiness of implementation-facing research-method specifications, defined by whether they provide sufficient methodological information for a competent implementer or coding agent to construct the intended method without unsupported assumptions. We construct evidence-grounded specifications and their supported resolutions from papers, codebases, issue threads, and reproduction artifacts. We introduce IdeaAMBIG, a benchmark of 660 evidence-grounded instances: 163 real-world gaps from reproducibility reports and GitHub issues, and 497 controlled synthetic gaps injected into codification-ready references. IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Defect localization receives only the specification, whereas clarification additionally receives the annotated defect. Across 13 LLMs, the best model achieves 9.6% Macro Defect Recovery Rate on real-world instances but 80.6% Macro Clarification Action Success Rate when given the defect. In an oracle study, supplying the gold resolution raises the downstream codification-ready rate from 14% to 98%. Across all evaluated models, defect localization is the main bottleneck, with stronger clarification given the defect. | → current | current | Hugging Face Hub (public pages, model cards, papers)T2 | medium | deterministic |
Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →
New paper: IdeaAMBIG: Benchmarking Implementation-Critical Gaps in Research-Idea Specifications
huggingface
| Source | Document | Type | Tier | Last observed | Snapshots |
|---|---|---|---|---|---|
| Hugging Face Hub (public pages, model cards, papers) | huggingface.co/papers | listing | T2· Quality secondary | 2 h ago | 3 |
Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.