Alert button

meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

Add code
Bookmark button
Alert button
Oct 31, 2017
Xu Sun, Xuancheng Ren, Shuming Ma, Houfeng Wang

Figure 1 for meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Figure 2 for meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Figure 3 for meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
Figure 4 for meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

We propose a simple yet effective technique for neural network learning. The forward propagation is computed as usual. In back propagation, only a small subset of the full gradient is computed to update the model parameters. The gradient vectors are sparsified in such a way that only the top-$k$ elements (in terms of magnitude) are kept. As a result, only $k$ rows or columns (depending on the layout) of the weight matrix are modified, leading to a linear reduction ($k$ divided by the vector dimension) in the computational cost. Surprisingly, experimental results demonstrate that we can update only 1--4\% of the weights at each back propagation pass. This does not result in a larger number of training iterations. More interestingly, the accuracy of the resulting models is actually improved rather than degraded, and a detailed analysis is given. The code is available at https://github.com/jklj077/meProp

* Accepted by The 34th International Conference on Machine Learning (ICML 2017)  
View paper onarxiv icon

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon