Alert button
Picture for KyungHyun Cho

KyungHyun Cho

Alert button

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

Add code
Bookmark button
Alert button
Dec 11, 2014
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio

Figure 1 for Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Figure 2 for Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Figure 3 for Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Figure 4 for Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

In this paper we compare different types of recurrent units in recurrent neural networks (RNNs). Especially, we focus on more sophisticated units that implement a gating mechanism, such as a long short-term memory (LSTM) unit and a recently proposed gated recurrent unit (GRU). We evaluate these recurrent units on the tasks of polyphonic music modeling and speech signal modeling. Our experiments revealed that these advanced recurrent units are indeed better than more traditional recurrent units such as tanh units. Also, we found GRU to be comparable to LSTM.

* Presented in NIPS 2014 Deep Learning and Representation Learning Workshop 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Not All Neural Embeddings are Born Equal

Add code
Bookmark button
Alert button
Nov 13, 2014
Felix Hill, KyungHyun Cho, Sebastien Jean, Coline Devin, Yoshua Bengio

Figure 1 for Not All Neural Embeddings are Born Equal
Figure 2 for Not All Neural Embeddings are Born Equal
Figure 3 for Not All Neural Embeddings are Born Equal

Neural language models learn word representations that capture rich linguistic and conceptual information. Here we investigate the embeddings learned by neural machine translation models. We show that translation-based embeddings outperform those learned by cutting-edge monolingual models at single-language tasks requiring knowledge of conceptual similarity and/or syntactic role. The findings suggest that, while monolingual models learn information about how concepts are related, neural-translation models better capture their true ontological status.

* 4 pages plus 1 page of references 
Viaarxiv icon
Github IconAccess Paper or Ask Questions
Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon