Dictionary Learning on a Manifold

Slides:

Advertisements

Similar presentations

Bayesian Spatial and Functional Data Analysis Using Gaussian Processes Alan E. Gelfand Duke University (with contributions from J. Duan, D. Dunson, M.

Advertisements

Sinead Williamson, Chong Wang, Katherine A. Heller, David M. Blei

Xiaolong Wang and Daniel Khashabi

Markov Chain Sampling Methods for Dirichlet Process Mixture Models R.M. Neal Summarized by Joon Shik Kim (Thu) Computational Models of Intelligence.

MAD-Bayes: MAP-based Asymptotic Derivations from Bayes

Bayesian dynamic modeling of latent trait distributions Duke University Machine Learning Group Presented by Kai Ni Jan. 25, 2007 Paper by David B. Dunson,

Combining Information from Related Regressions Duke University Machine Learning Group Presented by Kai Ni Apr. 27, 2007 F. Dominici, G. Parmigiani, K.

Multi-Task Compressive Sensing with Dirichlet Process Priors Yuting Qi 1, Dehong Liu 1, David Dunson 2, and Lawrence Carin 1 1 Department of Electrical.

EE462 MLCV Lecture Introduction of Graphical Models Markov Random Fields Segmentation Tae-Kyun Kim 1.

Decoupling Sparsity and Smoothness in the Discrete Hierarchical Dirichlet Process Chong Wang and David M. Blei NIPS 2009 Discussion led by Chunping Wang.

Classical inference and design efficiency Zurich SPM Course 2014

Belief Propagation on Markov Random Fields Aggeliki Tsoli.

Bayesian Robust Principal Component Analysis Presenter: Raghu Ranganathan ECE / CMR Tennessee Technological University January 21, 2011 Reading Group (Xinghao.

BAYESIAN INFERENCE Sampling techniques

Image Super-Resolution Using Sparse Representation By: Michael Elad Single Image Super-Resolution Using Sparse Representation Michael Elad The Computer.

Image Denoising via Learned Dictionaries and Sparse Representations

Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.

Nonparametric Bayes and human cognition Tom Griffiths Department of Psychology Program in Cognitive Science University of California, Berkeley.

CSE 221: Probabilistic Analysis of Computer Systems Topics covered: Statistical inference.

Nonparametric Bayesian Learning

Statistical Critical Path Selection for Timing Validation Kai Yang, Kwang-Ting Cheng, and Li-C Wang Department of Electrical and Computer Engineering University.

(1) A probability model respecting those covariance observations: Gaussian Maximum entropy probability distribution for a given covariance observation.

Noise Estimation from a Single Image Ce Liu William T. FreemanRichard Szeliski Sing Bing Kang.

Learning the structure of Deep sparse Graphical Model Ryan Prescott Adams Hanna M Wallach Zoubin Ghahramani Presented by Zhengming Xing Some pictures are.

Inferring structure from data Tom Griffiths Department of Psychology Program in Cognitive Science University of California, Berkeley.

Markov Random Fields Probabilistic Models for Images

A ROBUST B AYESIAN TWO - SAMPLE TEST FOR DETECTING INTERVALS OF DIFFERENTIAL GENE EXPRESSION IN MICROARRAY TIME SERIES Oliver Stegle, Katherine Denby,

School of Electrical & Computer Engineering Image Denoising Using Steerable Pyramids Alex Cunningham Ben Clarke Dy narath Eang ECE November 2008.

Learning to Sense Sparse Signals: Simultaneous Sensing Matrix and Sparsifying Dictionary Optimization Julio Martin Duarte-Carvajalino, and Guillermo Sapiro.

Fields of Experts: A Framework for Learning Image Priors (Mon) Young Ki Baik, Computer Vision Lab.

Variational Inference for the Indian Buffet Process

Bayesian Multivariate Logistic Regression by Sean O’Brien and David Dunson (Biometrics, 2004 ) Presented by Lihan He ECE, Duke University May 16, 2008.

The Infinite Hierarchical Factor Regression Model Piyush Rai and Hal Daume III NIPS 2008 Presented by Bo Chen March 26, 2009.

A Hierarchical Nonparametric Bayesian Approach to Statistical Language Model Domain Adaptation Frank Wood and Yee Whye Teh AISTATS 2009 Presented by: Mingyuan.

Statistical Models for Partial Membership Katherine Heller Gatsby Computational Neuroscience Unit, UCL Sinead Williamson and Zoubin Ghahramani University.

Characterizing the Function Space for Bayesian Kernel Models Natesh S. Pillai, Qiang Wu, Feng Liang Sayan Mukherjee and Robert L. Wolpert JMLR 2007 Presented.

Image Decomposition, Inpainting, and Impulse Noise Removal by Sparse & Redundant Representations Michael Elad The Computer Science Department The Technion.

Markov Chain Monte Carlo for LDA C. Andrieu, N. D. Freitas, and A. Doucet, An Introduction to MCMC for Machine Learning, R. M. Neal, Probabilistic.

by Ryan P. Adams, Iain Murray, and David J.C. MacKay (ICML 2009)

Stick-breaking Construction for the Indian Buffet Process Duke University Machine Learning Group Presented by Kai Ni July 27, 2007 Yee Whye The, Dilan.

Zhilin Zhang, Bhaskar D. Rao University of California, San Diego March 28,

Bayesian Density Regression Author: David B. Dunson and Natesh Pillai Presenter: Ya Xue April 28, 2006.

Dirichlet Distribution

Improving Image Matting using Comprehensive Sampling Sets CVPR2013 Oral.

The Phylogenetic Indian Buffet Process : A Non- Exchangeable Nonparametric Prior for Latent Features By: Kurt T. Miller, Thomas L. Griffiths and Michael.

Multi-label Prediction via Sparse Infinite CCA Piyush Rai and Hal Daume III NIPS 2009 Presented by Lingbo Li ECE, Duke University July 16th, 2010 Note:

Observation vs. Inferences The Local Environment.

Single Image Interpolation via Adaptive Non-Local Sparsity-Based Modeling The research leading to these results has received funding from the European.

Sparse Approximate Gaussian Processes. Outline Introduction to GPs Subset of Data Bayesian Committee Machine Subset of Regressors Sparse Pseudo GPs /

Hierarchical Beta Process and the Indian Buffet Process by R. Thibaux and M. I. Jordan Discussion led by Qi An.

The Nested Dirichlet Process Duke University Machine Learning Group Presented by Kai Ni Nov. 10, 2006 Paper by Abel Rodriguez, David B. Dunson, and Alan.

APPLICATIONS OF DIRICHLET PROCESS MIXTURES TO SPEAKER ADAPTATION Amir Harati and Joseph PiconeMarc Sobel Institute for Signal and Information Processing,

Jianchao Yang, John Wright, Thomas Huang, Yi Ma CVPR 2008 Image Super-Resolution as Sparse Representation of Raw Image Patches.

From Sparse Solutions of Systems of Equations to Sparse Modeling of Signals and Images Alfred M. Bruckstein (Technion), David L. Donoho (Stanford), Michael.

Institute of Statistics and Decision Sciences In Defense of a Dissertation Submitted for the Degree of Doctor of Philosophy 26 July 2005 Regression Model.

A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation Yee W. Teh, David Newman and Max Welling Published on NIPS 2006 Discussion.

Bayesian Semi-Parametric Multiple Shrinkage

Reducing Photometric Redshift Uncertainties Through Galaxy Clustering

An Infinite Factor Model Hierarchy Via a Noisy-Or Mechanism

Marked Point Processes for Crowd Counting

Accelerated Sampling for the Indian Buffet Process

The Chinese University of Hong Kong

A Non-Parametric Bayesian Method for Inferring Hidden Causes

Kernel Stick-Breaking Process

Presented by: Mingyuan Zhou Duke University, ECE Feb 22, 2013

Contrasts & Statistical Inference

Improving K-SVD Denoising by Post-Processing its Method-Noise

Contrasts & Statistical Inference

CS639: Data Management for Data Science

Contrasts & Statistical Inference

Presentation transcript:

Dictionary Learning on a Manifold 1Mingyuan Zhou, 2Hongxia Yang, 3Guillermo Sapiro, 2David Dunson and 1Lawrence Carin 1Department of Electrical & Computer Engineering, Duke University 2Department of Statistical Science, Duke University 3Department of Electrical & Computer Engineering, University of Minnesota Zhou 2011

Outline Introduction Dependent Hierarchical Beta Process (dHBP) Dictionary Learning with dHBP Image Interpolation and Denoising Conclusions

Introduction: Background Dictionary learning and sparse coding Sparse factor analysis model (Factor/feature/dish/dictionary atom) Indian Buffet process and beta process

Introduction: Background Beta process and Bernoulli process (Thibaux & Jordan AISTATS2007) Indian Buffet process (Griffiths & Ghahramani 2005)

Introduction: Motivation Exchangeability assumption is not true Image interpolation with BPFA (Zhou et al. 2009) 80% pixels are missing at random

Introduction: Covariate Dependence Dependent Dirichlet process MacEachern (1999), Duan et al. (2007), Griffin & Steel (2006) Non-exchangeable IBP Phylogenetic IBP (Miller, Griffiths & Jordan 2008) Dependent IBP (Williamson, Orbanz & Ghahramani 2010) Bayesian density regression (Dunson & Pillai 2007)

Review of Beta Process (Thibaus & Jordan 2007) A beta process is a positive Levy process, whose Levy measure lives on and can be expressed as If the base measure is continuous, then is drawn from a degenerate beta distribution parameterized by If , then

Dependent Hierarchical Beta Process Random walk matrix dHBP Covariate-dependent correlations

Dictionary Learning with dHBP

Missing Data and Outliers Full data: Observed: , Missing: Sparse spiky noise Recoverd data:

MCMC Inference Independence chain Metropolis-Hastings Slice sampling Gibbs sampling

Experiments Image interpolation Image denoising Missing pixels Locations of missing pixels are known Image denoising WGN + sparse spiky noise Amplitudes unknown Locations of spiky noise are unknown Covariates: patch spatial locations

Image Interpolation: BP vs. dHBP BP: 26.9 dB dHBP: 29.92 dB 80% pixels missing at random

Image Interpolation: BP vs. dHBP Observed BP recovery dHBP recovery Original BP atom usage dHBP atom usage dHBP recovery BP dictionary dHBP dictionary Dictionary atom activation probability map

Image Interpolation: BP vs. dHBP dHBP recovery Observed (20%) BP recovery dHBP recovery Original

Image Interpolation: BP vs. dHBP dHBP recovery Observed (20%) BP recovery dHBP recovery Original

Spiky Noise Removal: BP vs. dHBP Original image dHBP dictionary dHBP denoised image Noisy image (WGN + Sparse Spiky noise) BP dictionary BP denoised image

Spiky Noise Removal: BP vs. dHBP Original image dHBP dictionary dHBP denoised image Noisy image (WGN + Sparse Spiky noise) BP dictionary BP denoised image

Future Work Landmark-dHBP Locality constraint for manifold learning J landmarks J << N Locality constraint for manifold learning Covariates: cosine distance between samples Dictionary atoms look like the data Variational inference, online learning Other applications Super-resolution Deblurring Video background & foreground modeling

Conclusions A dependent hierarchical beta process is proposed Efficient hybrid MCMC inference is presented Encouraging performance is demonstrated on image-processing applications