Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue — Paper

Abstract

As AI systems integrate into online spaces, differentiating them from humans in conversations is increasingly important. We present Inverse Turing Bench, a benchmark that evaluates LLMs and other models on their ability to differentiate humans and AI in multi-turn text. The benchmark provides a collection of paired dialogue transcripts, wherein one dialogue is between two humans and the other is between a human and an AI. The task is to correctly identify which dialogue is human-only vs. human-AI. We evaluated a preliminary set of models against this benchmark, and found that GPTZero, Claude Opus-4.6, and GPT-5.5 achieve the highest accuracy: 89.41%, 77.92%, and 75.94% respectively. Our results suggest that statistical approaches to detection have semantic blind spots, but semantic approaches are susceptible to persona-prompting. Our work speaks to the Inverse Turing Test and motivates human-AI differentiation as a critical capability for AI systems. Our live benchmark can be found at https://huggingface.co/spaces/roc-hci/Inverse-Turing-Bench-Leaderboard.

Authors 4

William Hager, Ishika Rathi, Masum Hasan, Cameron Jones

Specification

Official page: arxiv.org/abs/2606.21844
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Arxiv announce type: replace
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
arXiv id: 2606.21844
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Categories: cs.CL, cs.CY
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
PDF: arxiv.org/pdf/2606.21844
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Primary category: cs.CL
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

2 h ago

Conflicts

None

No models linked to this paper yet.

Authors: William Hager, Ishika Rathi, Masum Hasan

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Official page

1 claims · 1 propertiesShow all properties

Official pageofficial_url1

Claim history for Official page
Value	Valid from → to	Status	Source	Confidence	Extractor
https://arxiv.org/abs/2606.21844	11 Sept 2026 21:03 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
New paperPaperInverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue
New paper: Inverse Turing Bench: Evaluating Language Models as Judges of Human vs. AI Dialogue
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.CL	feed	T1· Official	43 min ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.