The Pachinko Allocation Machine (PAM) is a deep topic model that allows representing rich correlation structures among topics by a directed acyclic graph over topics. Because of the flexibility of the model, however, approximate inference is very difficult. Perhaps for this reason, only a small number of potential PAM architectures have been explored in the literature. In this paper we present an efficient and flexible amortized variational inference method for PAM, using a deep inference network to parameterize the approximate posterior distribution in a manner similar to the variational autoencoder. Our inference method produces more coherent topics than state-of-art inference methods for PAM while being an order of magnitude faster, which allows exploration of a wider range of PAM architectures than have previously been studied.

**Click to Read Paper**
Topic models are one of the most popular methods for learning representations of text, but a major challenge is that any change to the topic model requires mathematically deriving a new inference algorithm. A promising approach to address this problem is autoencoding variational Bayes (AEVB), but it has proven diffi- cult to apply to topic models in practice. We present what is to our knowledge the first effective AEVB based inference method for latent Dirichlet allocation (LDA), which we call Autoencoded Variational Inference For Topic Model (AVITM). This model tackles the problems caused for AEVB by the Dirichlet prior and by component collapsing. We find that AVITM matches traditional methods in accuracy with much better inference time. Indeed, because of the inference network, we find that it is unnecessary to pay the computational cost of running variational optimization on test data. Because AVITM is black box, it is readily applied to new topic models. As a dramatic illustration of this, we present a new topic model called ProdLDA, that replaces the mixture model in LDA with a product of experts. By changing only one line of code from LDA, we find that ProdLDA yields much more interpretable topics, even if LDA is trained via collapsed Gibbs sampling.

**Click to Read Paper*** Supplementary material attached as Ancillary File; in PKDD 2016: European Conference on Machine Learning and Knowledge Discovery in Databases

**Click to Read Paper**

A Subsequence Interleaving Model for Sequential Pattern Mining

Nov 11, 2016

Jaroslav Fowkes, Charles Sutton

Nov 11, 2016

Jaroslav Fowkes, Charles Sutton

* 10 pages in KDD 2016: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

**Click to Read Paper**

Semi-Separable Hamiltonian Monte Carlo for Inference in Bayesian Hierarchical Models

Jun 15, 2014

Yichuan Zhang, Charles Sutton

Jun 15, 2014

Yichuan Zhang, Charles Sutton

**Click to Read Paper**

* Appears in Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence (UAI2005)

**Click to Read Paper**

* Appears in Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence (UAI2007)

**Click to Read Paper**

Bayesian inference for queueing networks and modeling of internet services

Apr 15, 2011

Charles Sutton, Michael I. Jordan

Apr 15, 2011

Charles Sutton, Michael I. Jordan

* Annals of Applied Statistics 2011, Vol. 5, No. 1, 254-282

* Published in at http://dx.doi.org/10.1214/10-AOAS392 the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

**Click to Read Paper**

Often we wish to predict a large number of variables that depend on each other as well as on other observed variables. Structured prediction methods are essentially a combination of classification and graphical modeling, combining the ability of graphical models to compactly model multivariate data with the ability of classification methods to perform prediction using large sets of input features. This tutorial describes conditional random fields, a popular probabilistic method for structured prediction. CRFs have seen wide application in natural language processing, computer vision, and bioinformatics. We describe methods for inference and parameter estimation for CRFs, including practical issues for implementing large scale CRFs. We do not assume previous knowledge of graphical modeling, so this tutorial is intended to be useful to practitioners in a wide variety of fields.

* 90 pages

* 90 pages

**Click to Read Paper*** Published as a conference paper at ICLR 2015

**Click to Read Paper**

Clustering with a Reject Option: Interactive Clustering as Bayesian Prior Elicitation

Jun 08, 2016

Akash Srivastava, James Zou, Charles Sutton

Jun 08, 2016

Akash Srivastava, James Zou, Charles Sutton

**Click to Read Paper**

A Convolutional Attention Network for Extreme Summarization of Source Code

May 25, 2016

Miltiadis Allamanis, Hao Peng, Charles Sutton

May 25, 2016

Miltiadis Allamanis, Hao Peng, Charles Sutton

* Code, data and visualization at http://groups.inf.ed.ac.uk/cup/codeattention/

**Click to Read Paper**

Latent Bayesian melding for integrating individual and population models

Oct 30, 2015

Mingjun Zhong, Nigel Goddard, Charles Sutton

Oct 30, 2015

Mingjun Zhong, Nigel Goddard, Charles Sutton

* 11 pages, Advances in Neural Information Processing Systems (NIPS), 2015. (Spotlight Presentation)

**Click to Read Paper**

Probabilistic Programming with Densities in SlicStan: Efficient, Flexible and Deterministic

Nov 02, 2018

Maria I. Gorinova, Andrew D. Gordon, Charles Sutton

Nov 02, 2018

Maria I. Gorinova, Andrew D. Gordon, Charles Sutton

**Click to Read Paper**

Learning Continuous Semantic Representations of Symbolic Expressions

Jun 10, 2017

Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, Charles Sutton

Jun 10, 2017

Miltiadis Allamanis, Pankajan Chanthirasegaran, Pushmeet Kohli, Charles Sutton

* Accepted to ICML 2017

**Click to Read Paper**

ColNet: Embedding the Semantics of Web Tables for Column Type Prediction

Nov 04, 2018

Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton

Nov 04, 2018

Jiaoyan Chen, Ernesto Jimenez-Ruiz, Ian Horrocks, Charles Sutton

* AAAI 2019

**Click to Read Paper**

Ratio Matching MMD Nets: Low dimensional projections for effective deep generative models

May 31, 2018

Akash Srivastava, Kai Xu, Michael U. Gutmann, Charles Sutton

May 31, 2018

Akash Srivastava, Kai Xu, Michael U. Gutmann, Charles Sutton

* Code: https://github.com/akashgit/RM-MMDnet

**Click to Read Paper**

A Survey of Machine Learning for Big Code and Naturalness

May 05, 2018

Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, Charles Sutton

May 05, 2018

Miltiadis Allamanis, Earl T. Barr, Premkumar Devanbu, Charles Sutton

* Website accompanying this survey paper can be found at https://ml4code.github.io

**Click to Read Paper**

Interpreting Deep Classifier by Visual Distillation of Dark Knowledge

Mar 11, 2018

Kai Xu, Dae Hoon Park, Chang Yi, Charles Sutton

Mar 11, 2018

Kai Xu, Dae Hoon Park, Chang Yi, Charles Sutton

**Click to Read Paper**