Alert button

Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game

Add code
Bookmark button
Alert button
Jul 17, 2018
Matthias Dorfer, Florian Henkel, Gerhard Widmer

Figure 1 for Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
Figure 2 for Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
Figure 3 for Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game
Figure 4 for Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon

Score following is the process of tracking a musical performance (audio) with respect to a known symbolic representation (a score). We start this paper by formulating score following as a multimodal Markov Decision Process, the mathematical foundation for sequential decision making. Given this formal definition, we address the score following task with state-of-the-art deep reinforcement learning (RL) algorithms such as synchronous advantage actor critic (A2C). In particular, we design multimodal RL agents that simultaneously learn to listen to music, read the scores from images of sheet music, and follow the audio along in the sheet, in an end-to-end fashion. All this behavior is learned entirely from scratch, based on a weak and potentially delayed reward signal that indicates to the agent how close it is to the correct position in the score. Besides discussing the theoretical advantages of this learning paradigm, we show in experiments that it is in fact superior compared to previously proposed methods for score following in raw sheet music images.

* Published in the Proceedings of the 19th International Society for Music Information Retrieval Conference, Paris, France, 2018  
View paper onarxiv icon

Share this with someone who'll enjoy it:

Twitter IconFacebook IconLinkedin IconWhatsapp IconMessenger Icon