Skip to content
AI Atlas
PaperActive

BRACE: Anchored Bellman-Residual Correction for Stale Critics in Asynchronous RL

arxiv.org/abs/2609.09783

Explore graph
quality89

Updated 3 h ago · first seen 11 Sept 2026

paper_01M294GMQQQZT52D563DMDS98R

Published
11 Sept 2026
T1 · 3 h ago
arXiv
2609.09783
T1 · 3 h ago
Category
cs.LG
T1 · 3 h ago

As of

Rewind the record: see this entity's attributes exactly as AI Atlas knew them on a given day.

Claim history · Abstract

1 claims · 1 propertiesShow all properties

Abstractabstract1

Claim history for Abstract
ValueValid from → toStatusSourceConfidenceExtractor
Asynchronous reinforcement learning has become the standard way to scale training for language models, but the resulting policy lag biases the critic toward the stale behavior policy. Existing work on asynchronous LLM training corrects the actor and leaves this bias unaddressed, while the off-policy value correction of classical RL does not carry over to long-horizon agentic tasks, since a short correction horizon leaves the regression target free of the reward and a long one lets the product of importance ratios drift exponentially with the trajectory length. We propose BRACE, an anchored Bellman-residual correction for stale value models. BRACE bounds the correction horizon to a prefix of policy tokens and anchors a constant-weight Monte-Carlo tail beyond it, which separates policy correction from reward propagation. BRACE improves mean@1 on BrowseComp-Plus by $2.4\%$ over the strongest baseline, runs $2.46\times$ faster per step than synchronous training, and remains stable $50$ updates off-policy. → currentcurrentarXiv (Atom API + RSS)T1highdeterministic

Claims are temporal and append-only: a new observation closes the previous claim (valid_to) instead of overwriting it. Conflicting claims from different sources are kept side by side and flagged — never averaged. Methodology →