Generalization Guarantees on Data-Driven Tuning of Gradient Descent with Langevin Updates

Abstract

We study learning to learn through the lens of hyperparameter tuning. We propose the Langevin Gradient Descent Algorithm (LGD), which approximates the mean of the posterior distribution defined by the loss function and regularizer of a regression task with convex objective. For classification tasks, the LGD algorithm estimates the posterior probabilities of each class on the test set. We prove the existence of an optimal hyperparameter configuration for which the LGD algorithm achieves the Bayes' optimal solution for squared loss on regression tasks, and for which LGD closely approximates the posterior probabilities for well-specified classification tasks. Subsequently, we study generalization guarantees on meta learning optimal hyperparameters for the LGD algorithm from a given set of tasks in the data-driven setting. For a number of parameters $d$ and hyperparameter dimension $h$, we show a pseudo-dimension bound of $O(dh)$, up to logarithmic terms under mild assumptions on LGD. This matches the dependence of the bounds on number of parameters obtained in prior work for linear regression using the elastic net, which only allows for $h=2$ hyperparameters, and extends their bounds to regression on convex loss. Compared to bounds on regularized logistic regression that allow for only $h=1$ hyperparameter, our bounds improve greatly on the dependence on samples per task at the cost of worse dependence on the number of parameters by accounting for hardware-aware procedures. Finally, we show empirical evidence of the success of LGD and the meta learning procedure for few-shot learning on linear and logistic regression using synthetically created datasets.

Authors 4

Saumya Goyal, Rohith Rongali, Ritabrata Ray, Barnab\'as P\'oczos

Specification

Official page

arxiv.org/abs/2604.13130

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Arxiv announce type

replace

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

arXiv id

2604.13130

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Categories

cs.LG, stat.ML

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

PDF

arxiv.org/pdf/2604.13130

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Primary category

cs.LG

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Published

11 Sept 2026

Source:arXiv (Atom API + RSS)·T1·observed 5 h ago·high

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance