We consider problems involving groups of data, where each observation within a group is drawn from a mixture model, and where it is desirable to share mixture components both within and between groups. We assume that the number of mixture components is unknown a priori and is to be inferred from data. Such problems occur often in practice, e.g. in the problem of topic discovery in document corpora, each document is treated as a group of data items (bag of words), and each topic corresponds to a mixture component. In this setting, sharing components between groups simply means that topics can occur across a number of documents, allowing dependencies across documents (groups) to be modeled effectively as well as conferring generalization to new documents (groups).
The hierarchical Dirichlet process (HDP) is a Bayesian solution to this problem, utilizing both hierarchical and nonparametric modeling ideas. Each group is modeled using a Dirichlet process (DP) mixture, which provides a nonparametric prior for the number of components within each group. To facilitate sharing components between groups, we consider a hierarchical extension where the common base distribution for the DPs is itself distributed according to a DP. Such a base distribution being discrete, the group specific DPs necessarily share atoms, hence the mixtures for different groups necessarily share components.
We discuss a variety of representations for the HDP, as well as Markov chain Monte Carlo algorithms for posterior inference. We report experimental results on three text corpora showing the effective and superior performance of the HDP over previous models.
Technical Report: Hierarchical Dirichlet processes. Teh, Jordan, Beal and Blei (2004). UC Berkeley Department of Statistics, TR 653. Can be obtained at: http://www.cs.berkeley.edu/~ywteh/research/npbayes

While supervised kernel techniques involving a single latent function (or discriminant) are well understood and powerful methods have emerged, much less is known about models which combine a number of latent functions. We describe a generic way of generalizing the sparse Bayesian Gaussian process Informative Vector Machine (IVM) to such multi process models, emphasizing the key techniques which are required for an efficient solution (exploiting matrix structure, numerical quadrature).
We apply our method to the multi-way classification problem, obtaining a scheme which scales essentially linearly in the number of datapoints and classes. We show how kernel parameters can be learned by empirical Bayesian techniques. We argue that a good solution for the multi-class problem leads to schemes for larger structured graphical models such as conditional random fields.
Joint work with Michael Jordan.

The growing amount of scientific data has led to the increased use of computational discovery methods to understand and interpret them. However, most work has relied on knowledge-lean techniques like clustering and classification learning, which produce descriptive rather than explanatory models, and it has utilized formalisms developed in AI or statistics, so that results seldom make contact with current theories or scientific notations. In this talk, I present an approach to computational discovery that encodes explanatory scientific models as sets of quantitative processes, simulates these models' behavior over time, incorporates background knowledge to constrain model construction, and induces the models from time-series data. I illustrate this framework on data and models from Earth science and microbiology, two domains in which explanatory process accounts occur frequently. In closing, I describe our progress toward an interactive software environment for the construction, evaluation, and revision of such explanatory scientific models.
This talk describes joint work with Kevin Arrigo, Nima Asgharbeygi, Stephen Bay, Andrew Pohorille, and Jeff Shrager.
http://cll.stanford.edu/~langley/

We propose an algorithm for selectively removing examples from the training set using probabilistic estimates related to editing algorithms (Devijver and Kittler, 1982). The procedure creates a separable distribution of training examples with minimal impact on the decision boundary position. It breaks the linear dependency between the number of SVs and the number of training examples, and sharply reduces the complexity if SVMs during both the training and prediction stages.

Computers don't grasp meaning in any deep, human sense. From a computer's point of view, words are meaningless bits of information to be processed with speed and precision, but without any genuine interpretation of content. We envision a new class of machines that understand the meaning of information in more human-like ways by grounding knowledge in the physical world and in the machines' own goals. Towards this long term goal, our group develops conversational robots and other situated systems designed to communicate with human partners using natural language. In this talk I will highlight recent experimental results and applications in multimodal human-machine interaction. I will also discuss an emerging theoretical framework for connecting language, action, and perception which has relevance both for designing situated systems and for cognitive modeling.
Professor Roy's and the Cognitive Machines Group Web Page are at:
http://www.media.mit.edu/cogmac

The selection and analysis of differentially expressed gene profiles (markers) helps associate a biological phenotype with its underlying molecular mechanisms and provides valuable insights into the structure of pathways and cellular regulation. However, analyzing and interpreting a given list of gene markers to glean useful biological insights can be extremely challenging. This is in part due to the difficulty of objectively evaluating how well members of a given a pathway or functional class of interest (Gene Set) are represented in the markers list. To address this problem we introduce a statistical methodology called Gene Set Enrichment Analysis (GSEA) for determining whether a given Gene Set is over-represented or enriched in a Gene List of markers ordered by their correlation with a phenotype or class distinction of interest. The method is based upon a score computed as the maximum deviation of a random walk (in the same spirit as the Kolmogorov-Smirnov statistic) and uses permutation testing to assess significance. When multiple Gene Sets are tested simultaneously we propose two approaches to address the multiplicity: Validation GSEA which controls the Familywise error rate (FWER) and Discovery GSEA which controls the False Discovery rate (FDR). The utility of this procedure will be illustrated on two biological problems: validating a mouse model of lung cancer and finding chromosomal dislocations for myeloid leukemia.

In this talk I will take a new look at Shannon's Information theory from the Machine Learning perspective . I will argue that Shannon's theory provides a compelling mechanism for quantifying the fundamental tradeoff between complexity and accuracy, by unifying the source and channel coding theorems into one principle which I call the "Information Bottleneck" (IB). This unified view of the coding theorems can shed new light on the question of "relevant data representation" and suggests new algorithms for extracting such representations from co-occurrence statistics. It also provides new ways of thinking about neural coding and neural data analysis. When applied to the analysis of human language it reveals new - possibly universal - scaling law that may reflect the way words are acquired in natural languages. The IB principle has new interesting extensions that deal with multivariate data using Graphical models and multi-information; allow adding irrelevant side information; and extract nonlinear continuous dimension reduction that preserve information (SDR). I will describe some of those extensions as time allows.
More information and many related papers can be found at:
http://www.cs.huji.ac.il/labs/learning/Papers/IBM_list.html

Scene analysis is currently the most exciting problem to work on in sensory processing. Scene analysis entails decomposing sensory inputs into a combinatorial explanation that can be used to predict new inputs. The problem is old and can be traced back to Helmholtz and more fuzzily to the ancient Greeks (who thought that fire, etc, was the combinatorial explanation). However, only recently do we have 1) Efficient algorithms that turn exponential-time combinatorial inference and learning algorithms into "linear time" approximate algorithms; and 2) Fast computers that can process sensory data repeatedly and quickly enough that grad students stick around to look at the results. In this talk, I'll describe research on visual and auditory scene analysis, being done in my group at the University of Toronto (PSI-Lab), using graphical models and efficient, approximate inference algorithms. The focus will be on the generative modeling approach and why this approach holds the most promise for "solving" the problem of sensory scene analysis.

For serious reading, check out the following tutorial paper: http://www.psi.toronto.edu/~frey/stuff/tutorial.ps.gz

For fun videos, audio clips, gene expression array results, etc, see the web pages of Nebojsa Jojic, my postdocs (Romer Rosales, Quaid Morris) and my current students (Kannan Achan, Anitha Kannan, Chris Pal).

We present Hilbert space embeddings of dynamical systems and embeddings generated via dynamical systems. This is achieved by following the behavioural framework invented by Willems, namely by comparing trajectories of states. As important special cases we recover the diffusion kernels of Kondor and Lafferty, generalised versions of directed graph kernels of Gartner, novel kernels on matrices and new similarity measures on Markov Models. We show applications of our method to Dynamical texture recognition problems from computer vision.

Many recent techniques in learning over biological sequences implicitly embed sequences into a Euclidean space in order to take advantage of strong margin based learning algorithms. However, these embeddings often do not take advantage of the rich biological intuitions that have motivated development of Hidden Markov Model style biological sequence models and have lead to great successes in computational biology. In this talk, we present a new biological motivated sequence embedding. We discuss several of the formal properties of the embedding include its connection to local sequence alignment. One of the key features of the embedding is that a sequence is embedded along with its homologues or neighboring sequences. The distance between two sequences is defined by the distance between close neighbors of the sequences. We demonstrate application of the embedding to several applications. We apply the embedding to learning protein secondary structure and protein family classification. We also show how the embedding can be used for aligning two sequences based on their homologues. Finally we discuss how due to the properties of the embedding, we can efficiently compute the nearest neighbors. Joint work with Sagi Snir.

Bayesian inference is often associated with the concept of subjective probability and prior distributions. We have a different attitude, in which inference proceeds from a posited (not "elicited") probability model, and then the implications of these inferences are compared to existing data, new data, and other subject-matter knowledge. The resulting approach to statistics places the flexibility of complex probability modeling in a scientific context of falsifiable hypotheses. In practice, it means that we can solve lots of problems without agonizing over philosophical questions of subjectivity. We illustrate with several examples in social science and public health.

Most of the study of Bayesian prediction procedures is premised on some strong assumptions regarding the prior distribution. In particular, an assumption that always needs to be made is that the prior distribution assigns a non-zero probability to the correct model. In practice, however, we often have to restrict the set of models (in the support of the prior) in order to make it feasable to compute the posterior average. As a result, we often can't assume that the correct model is in our set and the standard Bayesian theory cannot be applied.
In this work we show a classification procedure which uses model averaging and can be interpreted as a Bayesian procedure. We show that this procedure has some desirable properties when it is applied to *any* source of IID examples, regardless of whether or not the source distribution is related to the models in the support of the prior. Our main result is that the predictions made by this procedure are very stable with regard to the choice of the random training set.
This stability property has some far-reaching implications on a variety of issues, including Bias-Variance decomposition of classification error, the "curse of dimensionality" and Bagging.
This is joint work with Yishay Mansour and Rob Schapire

An informal talk but should be very interesting!

We present novel simple appearance and shape models that we call epitomes. The epitome of an image is its miniature, condensed version containing the essence of the textural and shape properties of the image. As opposed to previously used simple image models, such as templates or basis functions, the size of the epitome is considerably smaller than the size of the image or object it represents, but the epitome still contains most constitutive elements needed to reconstruct the image. A collection of images often shares an epitome, e.g., when images are a few consecutive frames from a video sequence, or when they are photographs of similar objects. A particular image in a collection is defined by its epitome and a smooth mapping from the epitome to the image pixels. When the epitomic representation is used within a hierarchical generative model, appropriate inference algorithms can be derived to extract epitome from a single image or a collection of images and at the same time perform various inference tasks, such as image segmentation, motion estimation, object removal and super-resolution. We have also had some preliminary success in other domains (audio waveforms and MID files, for example).

Joint work with Brendan Frey and Anitha Kannan.

Constructing tight bounds on the future error rate of a classifier has historically been a rather difficult task. This task has been accomplished for support vector machines. I'll discuss the intuitions behind a couple bounds, and presents some empirical results. With bounds this tight, direct bound optimization algorithms are motivated, for which I'll also present results.

In this talk I'll present a new class of image models we call Probabilistic Montages. Probabilistic Montages represent an image using a grid of smaller sub-images. Each sub-image is then modeled as arising from a transformed and cropped version of one of a collection of possible slightly larger images. We describe the probability distribution relating the way in which sub-images are cropped and transformed with respect to the other locations on the grid using various graphical models. In this talk I'll illustrate a number of variations of the probabilistic montage. I'll present an EM algorithm for estimating parameters in Bayesian tree structured montages and show how probabilistic montages are applicable to various classification and segmentation problems. In particular, I will illustrate the use of a tree structured model for recognition tasks applied to full body images of people in motion.

Joint work with Brendan Frey and Nebojsa Jojic

Using a social network consisting of 10^4 human individuals as a motivating example, I will recall some generic features occuring in various classes of large networks such as "small world" properties and power laws. Going beyond aggregated measures, I will present a somewhat mysterious representation of the self-similar architecture of the network based on its adjacency spectrum. A model reproducing both aggregated and self-similar features of the network will also be discussed. I will conclude by showing that such self-similarity is present in several, though not all, real-world networks.

Joint work with Gabor Csanyi