Alert button
Picture for Sheng-Wei Chen

Sheng-Wei Chen

Alert button

BDA-PCH: Block-Diagonal Approximation of Positive-Curvature Hessian for Training Neural Networks

Add code
Bookmark button
Alert button
Feb 20, 2018
Sheng-Wei Chen, Chun-Nan Chou, Edward Y. Chang

Figure 1 for BDA-PCH: Block-Diagonal Approximation of Positive-Curvature Hessian for Training Neural Networks
Figure 2 for BDA-PCH: Block-Diagonal Approximation of Positive-Curvature Hessian for Training Neural Networks
Figure 3 for BDA-PCH: Block-Diagonal Approximation of Positive-Curvature Hessian for Training Neural Networks
Figure 4 for BDA-PCH: Block-Diagonal Approximation of Positive-Curvature Hessian for Training Neural Networks

We propose a block-diagonal approximation of the positive-curvature Hessian (BDA-PCH) matrix to measure curvature. Our proposed BDAPCH matrix is memory efficient and can be applied to any fully-connected neural networks where the activation and criterion functions are twice differentiable. Particularly, our BDA-PCH matrix can handle non-convex criterion functions. We devise an efficient scheme utilizing the conjugate gradient method to derive Newton directions for mini-batch setting. Empirical studies show that our method outperforms the competing second-order methods in convergence speed.

* Correct the author's name 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon