Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training — Paper

Abstract

LLM agents for sequential decision tasks are often post-trained with trajectory-level outcome labels, but such labels provide little supervision for preserving multiple successful branches from the same decision state. We study this problem as successful strategy coverage: how broadly a model realizes distinct successful strategies under a fixed rollout budget. We present Direct Diversity Optimization (DDO), an offline post-training method that combines Divergence-Tree Collection (DTC) with the Reference-Relative Target-Odds Objective (RTO). DTC constructs state-aligned branch sets rooted at shared decision states, and RTO trains the model to match reference-relative targets over successful alternatives. DDO achieves the strongest task success and successful strategy coverage among the compared post-training methods across BabyAI, BabaIsAI, and WebShop. It also achieves the highest recovery rate after local action replacement and higher task success and coverage than successful-only imitation and decoding-time diversification controls.

Authors 5

Junwon Ko, Dong-Jae Lee, Minchan Kwon, Sunghyun Baek, Junmo Kim

Specification

Official page: arxiv.org/abs/2609.10052
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Arxiv announce type: cross
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
arXiv id: 2609.10052
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Categories: cs.CL, cs.AI, cs.LG
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
PDF: arxiv.org/pdf/2609.10052
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Primary category: cs.CL
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh
Published: 11 Sept 2026
Source:arXiv (Atom API + RSS)T1observed 2 h agohigh

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance

Attributed facts

9

Source tiers

T19

Freshest observation

2 h ago

Conflicts

None

No models linked to this paper yet.

Authors: Junwon Ko, Dong-Jae Lee, Minchan Kwon

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Published

1 claims · 1 propertiesShow all properties

Publishedpublished_at1

Claim history for Published
Value	Valid from → to	Status	Source	Confidence	Extractor
11 Sept 2026	11 Sept 2026 21:03 UTC → current	current	arXiv (Atom API + RSS)T1	high	deterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →

11 Sept 2026
New paperPaperDirect Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
New paper: Direct Diversity Optimization for Diverse Successful Trajectories in Preference Post-Training
arxiv
rss.arxiv.org

Full timeline →

Source documents
Source	Document	Type	Tier	Last observed	Snapshots
arXiv (Atom API + RSS)	rss.arxiv.org/rss/cs.AI	feed	T1· Official	2 h ago	1

Tier 1 = official/primary, 2 = quality secondary, 3 = community, 4 = unverified. Every snapshot is archived; see all sources and the methodology.