ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance — Paper

Abstract

As LLM agents move from answering questions to carrying out procedures, failures can be unwarranted rather than visibly wrong: the final response looks acceptable even though the system skipped the check, branch, dependency, or invariant that made the answer justified. Output-only evaluation sees the answer, and trace-aware judging sees activity, but neither identifies which obligations were active for the query. We introduce CONTRACTEVAL, a diagnostic framework for making those active obligations explicit. It represents procedural instructions as query-active obligations and matches them against response or trace evidence, turning omissions, wrong branches, ordering errors, extra actions, invariant breaches, and output-contract violations into distinct conformance failures. On a controlled suite of audited procedural contracts, output-only and trace-aware LLM judges miss many injected structural failures; under gold expected and observed graphs, ContractEval detects and localizes all of them. LLM-backed extraction preserves much of this signal but remains calibration-sensitive. ContractEval is therefore not a compliance guarantee; it makes procedural conformance auditable rather than implicit in final-answer quality.

Authors 4

Praphul Singh, Shanu Kumar, Akshat Agarwal, Ganesh Kumar

Specification

Official page: arxiv.org/abs/2609.09458
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh
Arxiv announce type: new
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh
arXiv id: 2609.09458
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh
Categories: cs.AI
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh
PDF: arxiv.org/pdf/2609.09458
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh
Primary category: cs.AI
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 8 h agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

8 h ago

Conflicts

None

No models linked to this paper yet.

Authors: Praphul Singh, Shanu Kumar, Akshat Agarwal

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Authors

1 claims · 1 propertiesShow all properties

Authorsauthors1

Claim history for Authors
Value	Valid from → to	Status	Source	Confidence	Extractor
Praphul Singh, Shanu Kumar, Akshat Agarwal, Ganesh Kumar	11 Sept 2026 21:03 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
New paperPaperContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance
New paper: ContractEval: Query-Conditioned Execution Matching for Procedural Instruction Conformance
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.AI	feed	T1· Official	7 h ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.