New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models — Paper

Abstract

A model first sees an image from one physical measurement experiment, such as how far a block coasted, and must answer a question about a new trial, such as whether the block will pass a target after a fixed push. The initial experiment may provide enough information to answer, or the model may need another measurement, such as the object's mass, friction, restitution, or spring stiffness. We study whether vision language models can decide when to answer immediately and, when more evidence is needed, which experiment to perform. Current physical reasoning benchmarks usually evaluate only the final answer, so they do not directly measure this decision-making ability. We introduce a controlled evaluation where each problem provides one measurement image and four possible physical worlds created by combining two possible masses and two possible values of another relevant property. The model must either stop and answer or select the cheapest additional experiment that can resolve the question. We construct matched problem pairs where changing either the observed measurement or the question changes the optimal action. Since all possible worlds and experiment costs are known, we can explicitly determine the optimal choice. Across six open models and 144 physical parameter sets, direct responses repeat the same action for 95.1% to 100% of image pairs even when the correct action changes. Brief reasoning improves action switching, but the best model makes both decisions correctly for only 5.9% of image pairs. Additional analysis reveals failures in measurement interpretation, physical reasoning, and response formatting. By evaluating evidence selection separately from final answers, our benchmark reveals limitations in physical reasoning that conventional answer accuracy can overlook.

Authors 7

Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar, Shaswati Saha, Yuxuan Jiang, Siyuan Li, Qiheng Wang

Specification

Official page: arxiv.org/abs/2609.11022
Source:arXiv (Atom API + RSS)T1observed 50 min agohigh
Arxiv announce type: new
Source:arXiv (Atom API + RSS)T1observed 49 min agohigh
arXiv id: 2609.11022
Source:arXiv (Atom API + RSS)T1observed 50 min agohigh
Categories: cs.CV, cs.AI, cs.CL, cs.LG
Source:arXiv (Atom API + RSS)T1observed 50 min agohigh
PDF: arxiv.org/pdf/2609.11022
Source:arXiv (Atom API + RSS)T1observed 50 min agohigh
Primary category: cs.CV
Source:arXiv (Atom API + RSS)T1observed 50 min agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 50 min agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

49 min ago

Conflicts

None

No models linked to this paper yet.

Authors: Sourajit Saha, Shubhashis Roy Dipta, Nobin Sarwar

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Categories

1 claims · 1 propertiesShow all properties

Categoriescategories1

Claim history for Categories
Value	Valid from → to	Status	Source	Confidence	Extractor
cs.CV, cs.AI, cs.CL, cs.LG	11 Sept 2026 21:02 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
Property changedPaperNew Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models: arxiv announce type changed from cross to new
Arxiv announce typecross→newarxiv
rss.arxiv.org
11 Sept 2026
New paperPaperNew Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
New paper: New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Models
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.CV	feed	T1· Official	49 min ago	1
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.LG	feed	T1· Official	50 min ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.