Recasting Gradient-Based Meta-Learning as Hierarchical Bayes

Jan 26, 2018

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, Thomas Griffiths

Meta-learning allows an intelligent agent to leverage prior learning episodes as a basis for quickly improving performance on a novel task. Bayesian hierarchical modeling provides a theoretical framework for formalizing meta-learning as inference for a set of parameters that are shared across tasks. Here, we reformulate the model-agnostic meta-learning algorithm (MAML) of Finn et al. (2017) as a method for probabilistic inference in a hierarchical Bayesian model. In contrast to prior methods for meta-learning via hierarchical Bayes, MAML is naturally applicable to complex function approximators through its use of a scalable gradient descent procedure for posterior inference. Furthermore, the identification of MAML as hierarchical Bayes provides a way to understand the algorithm's operation as a meta-learning procedure, as well as an opportunity to make use of computational strategies for efficient inference. We use this opportunity to propose an improvement to the MAML algorithm that makes use of techniques from approximate inference and curvature estimation.
Jan 26, 2018

Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, Thomas Griffiths

**Click to Read Paper**

Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm

Feb 14, 2018

Chelsea Finn, Sergey Levine

Feb 14, 2018

Chelsea Finn, Sergey Levine

* ICLR 2018

**Click to Read Paper**

* ICRA 2017. Supplementary video: https://sites.google.com/site/robotforesight/

**Click to Read Paper**

* NIPS 2016, Deep Reinforcement Learning Workshop, Barcelona, Spain. See https://cs.stanford.edu/~woodward/ for the poster and a short video description of the paper

**Click to Read Paper**

Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL

Dec 18, 2018

Anusha Nagabandi, Chelsea Finn, Sergey Levine

Dec 18, 2018

Anusha Nagabandi, Chelsea Finn, Sergey Levine

* Project website: https://sites.google.com/berkeley.edu/onlineviameta

**Click to Read Paper**

* First two authors contributed equally. Supplementary results available at https://sites.google.com/view/probabilistic-maml/

**Click to Read Paper**

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Jul 18, 2017

Chelsea Finn, Pieter Abbeel, Sergey Levine

Jul 18, 2017

Chelsea Finn, Pieter Abbeel, Sergey Levine

* ICML 2017. Code at https://github.com/cbfinn/maml, Videos of RL results at https://sites.google.com/view/maml, Blog post at http://bair.berkeley.edu/blog/2017/07/18/learning-to-learn/

**Click to Read Paper**

Unsupervised Learning for Physical Interaction through Video Prediction

Oct 17, 2016

Chelsea Finn, Ian Goodfellow, Sergey Levine

A core challenge for an agent learning to interact with the world is to predict how its actions affect objects in its environment. Many existing methods for learning the dynamics of physical interactions require labeled object information. However, to scale real-world interaction learning to a variety of scenes and objects, acquiring labeled data becomes increasingly impractical. To learn about physical object motion without labels, we develop an action-conditioned video prediction model that explicitly models pixel motion, by predicting a distribution over pixel motion from previous frames. Because our model explicitly predicts motion, it is partially invariant to object appearance, enabling it to generalize to previously unseen objects. To explore video prediction for real-world interactive agents, we also introduce a dataset of 59,000 robot interactions involving pushing motions, including a test set with novel objects. In this dataset, accurate prediction of videos conditioned on the robot's future actions amounts to learning a "visual imagination" of different futures based on different courses of action. Our experiments show that our proposed method produces more accurate video predictions both quantitatively and qualitatively, when compared to prior methods.
Oct 17, 2016

Chelsea Finn, Ian Goodfellow, Sergey Levine

* To appear in NIPS '16; Video results, code, and data available at: http://www.sites.google.com/site/robotprediction

**Click to Read Paper**

Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

May 27, 2016

Chelsea Finn, Sergey Levine, Pieter Abbeel

Reinforcement learning can acquire complex behaviors from high-level specifications. However, defining a cost function that can be optimized effectively and encodes the correct task is challenging in practice. We explore how inverse optimal control (IOC) can be used to learn behaviors from demonstrations, with applications to torque control of high-dimensional robotic systems. Our method addresses two key challenges in inverse optimal control: first, the need for informative features and effective regularization to impose structure on the cost, and second, the difficulty of learning the cost function under unknown dynamics for high-dimensional continuous systems. To address the former challenge, we present an algorithm capable of learning arbitrary nonlinear cost functions, such as neural networks, without meticulous feature engineering. To address the latter challenge, we formulate an efficient sample-based approximation for MaxEnt IOC. We evaluate our method on a series of simulated tasks and real-world robotic manipulation problems, demonstrating substantial improvement over prior methods both in terms of task complexity and sample efficiency.
May 27, 2016

Chelsea Finn, Sergey Levine, Pieter Abbeel

* International Conference on Machine Learning (ICML), 2016, to appear

**Click to Read Paper**

Learning Compact Convolutional Neural Networks with Nested Dropout

Apr 10, 2015

Chelsea Finn, Lisa Anne Hendricks, Trevor Darrell

Apr 10, 2015

Chelsea Finn, Lisa Anne Hendricks, Trevor Darrell

* 4 pages, 2 figures. Accepted as a workshop contribution at ICLR 2015

**Click to Read Paper**

Unsupervised Visuomotor Control through Distributional Planning Networks

Feb 14, 2019

Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, Chelsea Finn

Feb 14, 2019

Tianhe Yu, Gleb Shevchuk, Dorsa Sadigh, Chelsea Finn

* Videos available at https://sites.google.com/view/dpn-public/

**Click to Read Paper**

One-Shot Hierarchical Imitation Learning of Compound Visuomotor Tasks

Oct 25, 2018

Tianhe Yu, Pieter Abbeel, Sergey Levine, Chelsea Finn

Oct 25, 2018

Tianhe Yu, Pieter Abbeel, Sergey Levine, Chelsea Finn

* Video results available at https://sites.google.com/view/one-shot-hil

**Click to Read Paper**

Few-Shot Goal Inference for Visuomotor Learning and Planning

Sep 30, 2018

Annie Xie, Avi Singh, Sergey Levine, Chelsea Finn

Sep 30, 2018

Annie Xie, Avi Singh, Sergey Levine, Chelsea Finn

* Videos available at https://sites.google.com/view/few-shot-goals

**Click to Read Paper**

Unsupervised Meta-Learning for Reinforcement Learning

Jun 12, 2018

Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, Sergey Levine

Jun 12, 2018

Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, Sergey Levine

**Click to Read Paper**

A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models

Nov 25, 2016

Chelsea Finn, Paul Christiano, Pieter Abbeel, Sergey Levine

Generative adversarial networks (GANs) are a recently proposed class of generative models in which a generator is trained to optimize a cost function that is being simultaneously learned by a discriminator. While the idea of learning cost functions is relatively new to the field of generative modeling, learning costs has long been studied in control and reinforcement learning (RL) domains, typically for imitation learning from demonstrations. In these fields, learning cost function underlying observed behavior is known as inverse reinforcement learning (IRL) or inverse optimal control. While at first the connection between cost learning in RL and cost learning in generative modeling may appear to be a superficial one, we show in this paper that certain IRL methods are in fact mathematically equivalent to GANs. In particular, we demonstrate an equivalence between a sample-based algorithm for maximum entropy IRL and a GAN in which the generator's density can be evaluated and is provided as an additional input to the discriminator. Interestingly, maximum entropy IRL is a special case of an energy-based model. We discuss the interpretation of GANs as an algorithm for training energy-based models, and relate this interpretation to other recent work that seeks to connect GANs and EBMs. By formally highlighting the connection between GANs, IRL, and EBMs, we hope that researchers in all three communities can better identify and apply transferable ideas from one domain to another, particularly for developing more stable and scalable algorithms: a major challenge in all three domains.
Nov 25, 2016

Chelsea Finn, Paul Christiano, Pieter Abbeel, Sergey Levine

* NIPS 2016 Workshop on Adversarial Training. First two authors contributed equally

**Click to Read Paper**

End-to-End Training of Deep Visuomotor Policies

Apr 19, 2016

Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel

Apr 19, 2016

Sergey Levine, Chelsea Finn, Trevor Darrell, Pieter Abbeel

* updating with revisions for JMLR final version

**Click to Read Paper**

Self-Supervised Visual Planning with Temporal Skip Connections

Oct 15, 2017

Frederik Ebert, Chelsea Finn, Alex X. Lee, Sergey Levine

Oct 15, 2017

Frederik Ebert, Chelsea Finn, Alex X. Lee, Sergey Levine

* accepted at the Conference on Robot Learning (CoRL) 2017

**Click to Read Paper**

Learning a Prior over Intent via Meta-Inverse Reinforcement Learning

Oct 10, 2018

Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn

Oct 10, 2018

Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, Chelsea Finn

**Click to Read Paper**

Time Reversal as Self-Supervision

Oct 02, 2018

Suraj Nair, Mohammad Babaeizadeh, Chelsea Finn, Sergey Levine, Vikash Kumar

Oct 02, 2018

Suraj Nair, Mohammad Babaeizadeh, Chelsea Finn, Sergey Levine, Vikash Kumar

* 7 pages, 10 figures

**Click to Read Paper**

Universal Planning Networks

Apr 04, 2018

Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, Chelsea Finn

Apr 04, 2018

Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, Chelsea Finn

* Videos available at https://sites.google.com/view/upn-public/home

**Click to Read Paper**