Beyond Benchmarks: Using VLMs to Reveal Systematic Classification Failures Under Real World Conditions
Updated 5 h ago · first seen 11 Sept 2026
paper_01M294H1SVXH8DK275SBX4YN6Q
- Published
- 11 Sept 2026
- T1 · 5 h ago
- arXiv
- 2609.11126
- T1 · 5 h ago
- Category
- cs.CV
- T1 · 5 h ago
Abstract
Verification and validation (V&V) of classification models is crucial to enable a wide range of sensor processing applications. Currently, the V&V process relies on time-consuming manual inspection of erroneous samples to find meaningful patterns. This work explores the use of Vision Language Models (VLMs) to speed up this laborious process. VLMs are trained to embed images into a semantically meaningful vector representation, from which human-interpretable systematic errors can be distilled. Deploying such VLM-based methods in a defence context introduces two major challenges: (1) the defence domain is underrepresented in the training data of VLMs, and (2) surroundings and context are less diverse than for other domains. This study provides an initial assessment of the suitability of VLM-based methods for V&V of defence applications. We propose a VLM-based error slice detection (ESD) method that independently groups and labels systematic errors made by a classification model. We demonstrate that this method is able to identify operationally-relevant artificially added perturbations in a non-military dataset. In a military context, our method clusters and describes images based on their surroundings, but also exhibits overlap between cluster descriptions. We further investigate the difference in embedding variation between our military and non-military dataset, which remains a topic of interest. Although the results do not yet warrant fully automated V&V through VLM-based ESD, they show that VLMs could be used to accelerate V&V processes in the future.
Authors 6
Dieuwertje Alblas, Alma M. Liezenga, Jan Erik van Woerden, Fedor Taggenbrock, Dalia Aljawaheri, Klamer Schutte
Specification
- Official page
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
- Arxiv announce type
- new
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
- arXiv id
- 2609.11126
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
- Categories
- cs.CV, cs.AI, cs.ET
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
- Primary category
- cs.CV
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
- Published
- 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 5 h agohigh
Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →
Provenance
Attributed facts
9
Source tiers
T19
Freshest observation
5 h ago
Conflicts
None
No models linked to this paper yet.
As of
Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.
Claim history
Official pageofficial_url1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://arxiv.org/abs/2609.11126 | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
Abstractabstract1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| Verification and validation (V&V) of classification models is crucial to enable a wide range of sensor processing applications. Currently, the V&V process relies on time-consuming manual inspection of erroneous samples to find meaningful patterns. This work explores the use of Vision Language Models (VLMs) to speed up this laborious process. VLMs are trained to embed images into a semantically meaningful vector representation, from which human-interpretable systematic errors can be distilled. Deploying such VLM-based methods in a defence context introduces two major challenges: (1) the defence domain is underrepresented in the training data of VLMs, and (2) surroundings and context are less diverse than for other domains. This study provides an initial assessment of the suitability of VLM-based methods for V&V of defence applications. We propose a VLM-based error slice detection (ESD) method that independently groups and labels systematic errors made by a classification model. We demonstrate that this method is able to identify operationally-relevant artificially added perturbations in a non-military dataset. In a military context, our method clusters and describes images based on their surroundings, but also exhibits overlap between cluster descriptions. We further investigate the difference in embedding variation between our military and non-military dataset, which remains a topic of interest. Although the results do not yet warrant fully automated V&V through VLM-based ESD, they show that VLMs could be used to accelerate V&V processes in the future. | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
Arxiv announce typearxiv_announce_type1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| new | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
arXiv idarxiv_id1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 2609.11126 | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
Categoriescategories1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| cs.CV, cs.AI, cs.ET | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
PDFpdf_url1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| https://arxiv.org/pdf/2609.11126 | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
Primary categoryprimary_category1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| cs.CV | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
Publishedpublished_at1
| Value | Valid from → to | Status | Source | Confidence | Extractor |
|---|---|---|---|---|---|
| 11 Sept 2026 | → current | current | arXiv (Atom API + RSS)T1 | high | deterministic |
Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →
- New paperPaperBeyond Benchmarks: Using VLMs to Reveal Systematic Classification Failures Under Real World Conditions
New paper: Beyond Benchmarks: Using VLMs to Reveal Systematic Classification Failures Under Real World Conditions
arxiv
| Source | Document | Type | Tier | Last observed | Snapshots |
|---|---|---|---|---|---|
| arXiv (Atom API + RSS) | rss.arxiv.org/rss/cs.CV | feed | T1· Official | 5 h ago | 1 |
Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.