Alert button

Predictive Local Smoothness for Stochastic Gradient Methods

Add code
Bookmark button
Alert button
May 23, 2018
Jun Li, Hongfu Liu, Bineng Zhong, Yue Wu, Yun Fu

Figure 1 for Predictive Local Smoothness for Stochastic Gradient Methods
Figure 2 for Predictive Local Smoothness for Stochastic Gradient Methods
Figure 3 for Predictive Local Smoothness for Stochastic Gradient Methods
Figure 4 for Predictive Local Smoothness for Stochastic Gradient Methods

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Stochastic gradient methods are dominant in nonconvex optimization especially for deep models but have low asymptotical convergence due to the fixed smoothness. To address this problem, we propose a simple yet effective method for improving stochastic gradient methods named predictive local smoothness (PLS). First, we create a convergence condition to build a learning rate which varies adaptively with local smoothness. Second, the local smoothness can be predicted by the latest gradients. Third, we use the adaptive learning rate to update the stochastic gradients for exploring linear convergence rates. By applying the PLS method, we implement new variants of three popular algorithms: PLS-stochastic gradient descent (PLS-SGD), PLS-accelerated SGD (PLS-AccSGD), and PLS-AMSGrad. Moreover, we provide much simpler proofs to ensure their linear convergence. Empirical results show that the variants have better performance gains than the popular algorithms, such as, faster convergence and alleviating explosion and vanish of gradients.

* 14 pages, 7 figures  
View paper onarxiv iconopen_review iconOpenReview

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon