Presentation is loading. Please wait.

Presentation is loading. Please wait.

Graph Based Multi-Modality Learning

Published byYenny Iskandar Modified over 6 years ago

Similar presentations

Presentation on theme: "Graph Based Multi-Modality Learning"— Presentation transcript:

1 Graph Based Multi-Modality Learning
Hanghang Tong; Jingrui He Carnegie Mellon University Mingjing Li Microsoft Research Asia

2 Outline Motivation Graph-based Semi-supervised learning
Methods The Linear Fusion Scheme The Sequential Fusion Scheme Justifications Similarity Propagation Bayesian Interpretation Graph-based un-supervised learning Experimental Results Conclusion 2018/11/17 ACM/Multimedia 2005

3 Motivation Multi-Modality in Multimedia Traditional methods Video:
Digital Image: color vs. Web Image: content vs. surrounding text Traditional methods Linear combination; super-kernel… Co-Training… Multi-view version of EM, DBSCAN… All Vector Model based ! 2018/11/17 ACM/Multimedia 2005

4 Motivation (conts) Two Key issues A more recent hot topic
How to learning within each modality How to combine… A more recent hot topic Graph-based learning Spectral Cluster; Eigen Map Manifold Ranking… Explore graph-based method in the context of multi-modality! 2018/11/17 ACM/Multimedia 2005

5 Notation n data points c classes, Two modalities: a and b
One Affinity Matrix for each modality: for modality a (nxn) for modality b (nxn) Labeling Matrix: (nxc) Vectorial Function: (nxc) Learning task: s 2018/11/17 ACM/Multimedia 2005

6 Basic Idea What is a ‘good’ vectorial function F? How to?
As consistent as possible with Info from modality a Info from modality b Info from Labeled points Y How to? Take into account the various constraints by optimization 2018/11/17 ACM/Multimedia 2005

7 Linear Fusion Scheme Optimization strategy Optimization Solution
Constrains. from modality b Constrains. from modality a Optimization strategy Optimization Solution Iterative form Closed form Constrains. from Labels Y Converge 2018/11/17 ACM/Multimedia 2005

8 Sequential Fusion Scheme
Constrains. from modality a and Y Optimization strategy Optimization Solution Iterative form Closed form Constrains. from modality b and Converge 2018/11/17 ACM/Multimedia 2005

9 Similarity Propagation
Taylor expansion (linear fusion) Similar result for sequential form Initial Label > Further propagate similarity by a and b; > Fuse the result by weighted sum > Propagate Y by a and b; > Fuse the result by weighted sum 2018/11/17 ACM/Multimedia 2005

10 Bayesian Interpretation
Optimal F by MAP (linear form): Assuming: Conditional pdf Prior by modality a Prior by modality b Fuse prior by The above setting leads to… 2018/11/17 ACM/Multimedia 2005

11 Extension to Un-Supervised Case
Compare For one modality: For two modalities (linear form): Graph Laplacian Learning Linear Form: Sequential From: Feed it the spectral cluster or embedding… Quite similar ! Independent on Y ! 2018/11/17 ACM/Multimedia 2005

12 Experimental Results: Coral Image
Sequential Form Linear Form Treat as one modality Color Hist Texture 2018/11/17 ACM/Multimedia 2005

13 Experimental Results: Web Image
Linear Form Sequential Form Treat as one modality Content Fea Surrounding Text 2018/11/17 ACM/Multimedia 2005

14 Conclusion Study multi-modality learning by graph based method
Propose two schemes for semi-supervised learning Extend them to un-supervised learning 2018/11/17 ACM/Multimedia 2005

15 Q&A The End Thanks 2018/11/17 ACM/Multimedia 2005

Download ppt "Graph Based Multi-Modality Learning"

Similar presentations

Consistent Bipartite Graph Co-Partitioning for High-Order Heterogeneous Co-Clustering Tie-Yan Liu WSM Group, Microsoft Research Asia 2005.11.11 Joint work.

Consistent Bipartite Graph Co-Partitioning for High-Order Heterogeneous Co-Clustering Tie-Yan Liu WSM Group, Microsoft Research Asia Joint work.
Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki 2008-11629.

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Weakly supervised learning of MRF models for image region labeling Jakob Verbeek LEAR team, INRIA Rhône-Alpes.

Weakly supervised learning of MRF models for image region labeling Jakob Verbeek LEAR team, INRIA Rhône-Alpes.
1 Manifold Alignment for Multitemporal Hyperspectral Image Classification H. Lexie Yang 1, Melba M. Crawford 2 School of Civil Engineering, Purdue University.

1 Manifold Alignment for Multitemporal Hyperspectral Image Classification H. Lexie Yang 1, Melba M. Crawford 2 School of Civil Engineering, Purdue University.
Spectral graph reduction for image and streaming video segmentation Fabio Galasso 1 Margret Keuper 2 Thomas Brox 2 Bernt Schiele 1 1 Max Planck Institute.

Spectral graph reduction for image and streaming video segmentation Fabio Galasso 1 Margret Keuper 2 Thomas Brox 2 Bernt Schiele 1 1 Max Planck Institute.
Semi-supervised Learning Rong Jin. Semi-supervised learning  Label propagation  Transductive learning  Co-training  Active learning.

Semi-supervised Learning Rong Jin. Semi-supervised learning  Label propagation  Transductive learning  Co-training  Active learning.
Large Scale Manifold Transduction Michael Karlen Jason Weston Ayse Erkan Ronan Collobert ICML 2008.

Large Scale Manifold Transduction Michael Karlen Jason Weston Ayse Erkan Ronan Collobert ICML 2008.
Video Enhancement with Super-resolution 694410100 陳彥雄.

Video Enhancement with Super-resolution 陳彥雄.
Unsupervised Feature Selection for Multi-Cluster Data Deng Cai et al, KDD 2010 Presenter: Yunchao Gong Dept. Computer Science, UNC Chapel Hill.

Unsupervised Feature Selection for Multi-Cluster Data Deng Cai et al, KDD 2010 Presenter: Yunchao Gong Dept. Computer Science, UNC Chapel Hill.
Lecture 21: Spectral Clustering

Lecture 21: Spectral Clustering
Modeling Pixel Process with Scale Invariant Local Patterns for Background Subtraction in Complex Scenes (CVPR’10) Shengcai Liao, Guoying Zhao, Vili Kellokumpu,

Modeling Pixel Process with Scale Invariant Local Patterns for Background Subtraction in Complex Scenes (CVPR’10) Shengcai Liao, Guoying Zhao, Vili Kellokumpu,
Graph Based Semi- Supervised Learning Fei Wang Department of Statistical Science Cornell University.

Graph Based Semi- Supervised Learning Fei Wang Department of Statistical Science Cornell University.
Locally Constraint Support Vector Clustering

Locally Constraint Support Vector Clustering
A Unified View of Kernel k-means, Spectral Clustering and Graph Cuts Dhillon, Inderjit S., Yuqiang Guan, and Brian Kulis.

A Unified View of Kernel k-means, Spectral Clustering and Graph Cuts Dhillon, Inderjit S., Yuqiang Guan, and Brian Kulis.
A Unified View of Kernel k-means, Spectral Clustering and Graph Cuts

A Unified View of Kernel k-means, Spectral Clustering and Graph Cuts
Sparse Solutions for Large Scale Kernel Machines Taher Dameh CMPT820-Multimedia Systems Dec 2 nd, 2010.

Sparse Solutions for Large Scale Kernel Machines Taher Dameh CMPT820-Multimedia Systems Dec 2 nd, 2010.
1 Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval 9-April, 2005 Steven C. H. Hoi *, Michael R. Lyu.

1 Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval 9-April, 2005 Steven C. H. Hoi *, Michael R. Lyu.
Iterative Set Expansion of Named Entities using the Web Richard C. Wang and William W. Cohen Language Technologies Institute Carnegie Mellon University.

Iterative Set Expansion of Named Entities using the Web Richard C. Wang and William W. Cohen Language Technologies Institute Carnegie Mellon University.
MANISHA VERMA, VASUDEVA VARMA PATENT SEARCH USING IPC CLASSIFICATION VECTORS.

MANISHA VERMA, VASUDEVA VARMA PATENT SEARCH USING IPC CLASSIFICATION VECTORS.
Efficient Spatiotemporal Grouping Using the Nyström Method Charless Fowlkes, U.C. Berkeley Serge Belongie, U.C. San Diego Jitendra Malik, U.C. Berkeley.

Efficient Spatiotemporal Grouping Using the Nyström Method Charless Fowlkes, U.C. Berkeley Serge Belongie, U.C. San Diego Jitendra Malik, U.C. Berkeley.

Similar presentations

Ads by Google