On the Generalization Benefit of Noise in Stochastic Gradient Descent

Samuel Smith, Erich Elsen, Soham De

Type: Preprint

Publication Date: 2020-01-01

Citations: 17

DOI: https://doi.org/10.48550/arxiv.2006.15081

View Publication

Locations

arXiv (Cornell University) - View
DataCite API - View

Similar Works

Action	Title	Year	Authors
+	Strength of Minibatch Noise in SGD	2021	Ziyin Liu Kangqiao Liu T. Mori Masahito Ueda
+	On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima	2016	Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy Ping Tang
+	On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima	2016	Nitish Shirish Keskar Dheevatsa Mudigere Jorge Nocedal Mikhail Smelyanskiy Ping Tang
+	Understanding Generalization and Stochastic Gradient Descent	2017	Samuel Smith Quoc V. Le
+	The Impact of the Mini-batch Size on the Variance of Gradients in Stochastic Gradient Descent	2020	X. Qian Diego Klabjan
+	Three Factors Influencing Minima in SGD	2017	Stanisław Jastrzȩbski Zachary Kenton Devansh Arpit Nicolas Ballas Asja Fischer Yoshua Bengio Amos Storkey
+ PDF Chat	Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent	2024	Hideyuki Umeda Hideaki Iiduka
+	Stochastic Gradient Descent: Going As Fast As Possible But Not Faster	2017	Alice Schoenauer Sebag Marc Schoenauer Michèle Sébag
+	On the Noisy Gradient Descent that Generalizes as SGD	2019	Jingfeng Wu Wenqing Hu Haoyi Xiong Jun Huan Vladimir Braverman Zhanxing Zhu
+	On the different regimes of Stochastic Gradient Descent	2023	Antonio Sclocchi Matthieu Wyart
+	Special Properties of Gradient Descent with Large Learning Rates	2022	Amirkeivan Mohtashami Martin Jaggi Sebastian U. Stich
+	Interplay Between Optimization and Generalization of Stochastic Gradient Descent with Covariance Noise.	2019	Yeming Wen Kevin Luk Maxime Gazeau Guodong Zhang Harris Chan Jimmy Ba
+	Stochastic Gradient Descent with Large Learning Rate.	2020	Kangqiao Liu Ziyin Liu Masahito Ueda
+	The Marginal Value of Momentum for Small Learning Rate SGD	2023	Runzhe Wang Sadhika Malladi Tianhao Wang Kaifeng Lyu Zhiyuan Li
+	Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent	2018	Xiaowu Dai Yuhua Zhu
+	Towards Better Generalization: BP-SVRG in Training Deep Neural Networks	2019	Hao Jin Dachao Lin Zhihua Zhang
+ PDF Chat	Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum	2025	Ken‐ichi Kamo Hideaki Iiduka
+	On the Overlooked Structure of Stochastic Gradients	2022	Zeke Xie Qian-Yuan Tang Zheng He Mingming Sun Ping Li
+	An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise	2019	Yeming Wen Kevin Luk Maxime Gazeau Guodong Zhang Harris Chan Jimmy Ba
+	A Bayesian Perspective on Generalization and Stochastic Gradient Descent	2017	Samuel Smith Quoc V. Le

Works That Cite This (9)

Action	Title	Year	Authors
+	The Impact of Neural Network Overparameterization on Gradient Confusion and Stochastic Gradient Descent	2019	Karthik Abinav Sankararaman Soham De Zheng Xu Wei Huang Tom Goldstein
+	Improved generalization by noise enhancement	2020	T. Mori Masahito Ueda
+ PDF Chat	Survey: Leakage and Privacy at Inference Time	2022	Marija Jegorova Chaitanya Kaul Charlie Mayor Alison Q. O’Neil Alexander Weir Roderick Murray‐Smith Sotirios A. Tsaftaris
+	Privacy-Preserving Domain Adaptation of Semantic Parsers	2023	Fatemehsadat Mireshghallah Yu Su Tatsunori Hashimoto Jason Eisner Richard Shin
+ PDF Chat	Neural network enhanced measurement efficiency for molecular groundstates	2023	Dmitri Iouchtchenko Jérôme F. Gonthier Alejandro Perdomo‐Ortiz Roger G. Melko
+ PDF Chat	Shift-curvature, SGD, and generalization	2022	Arwen V. Bradley Carlos Alberto Gomez-Uribe Manish Reddy Vuyyuru
+	Active Learning of Discrete-Time Dynamics for Uncertainty-Aware Model Predictive Control	2022	Alessandro Saviolo Jonathan Frey Abhishek Rathod Moritz Diehl Giuseppe Loianno
+	Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks	2020	Soham De Samuel Smith
+ PDF Chat	Deep Visual Navigation under Partial Observability	2022	Bo Ai Wei Gao Vinay David Hsu

Works Cited by This (29)

Action	Title	Year	Authors
+	Momentum and optimal stochastic search	1993	Genevieve Orr Todd K. Leen
+	Some methods of speeding up the convergence of iteration methods	1964	B. T. Polyak
+	A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets	2012	Nicolas Le Roux Mark Schmidt Francis Bach
+	Bayesian Learning via Stochastic Gradient Langevin Dynamics	2011	Max Welling Yee Whye Teh
+ PDF Chat	A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks	2019	Umut Şimşekli Levent Sagun Mert Gürbüzbalaban
+	Stochastic Gradient Descent as Approximate Bayesian Inference	2017	Stephan Mandt Matthew D. Hoffman David M. Blei
+ PDF Chat	Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks	2018	Pratik Chaudhari Stefano Soatto
+	Stochastic modified equations and adaptive stochastic gradient algorithms	2015	Qianxiao Li Cheng Tai E Weinan
+	The Step Decay Schedule: A Near Optimal, Geometrically Decaying Learning Rate Procedure For Least Squares	2019	Rong Ge Sham M. Kakade Rahul Kidambi Praneeth Netrapalli
+	Introductory Lectures on Convex Optimization: A Basic Course	2014	Ю Е Нестеров