Presentation is loading. Please wait.

Presentation is loading. Please wait.

Online Multiple Kernel Classification Steven C.H. Hoi, Rong Jin, Peilin Zhao, Tianbao Yang Machine Learning (2013) Presented by Audrey Cheong Electrical.

Published byMarvin Kenneth Underwood Modified over 9 years ago

Similar presentations

Presentation on theme: "Online Multiple Kernel Classification Steven C.H. Hoi, Rong Jin, Peilin Zhao, Tianbao Yang Machine Learning (2013) Presented by Audrey Cheong Electrical."— Presentation transcript:

1 Online Multiple Kernel Classification Steven C.H. Hoi, Rong Jin, Peilin Zhao, Tianbao Yang Machine Learning (2013) Presented by Audrey Cheong Electrical & Computer Engineering MATH 6397: Data Mining

2 Online Multiple Kernel Classification (OMKC) Background - Online Online learning Learns one instance at a time and predicts labels for future instances Learner is given an instance Learner predicts the label of the instance Learner is given the correct label Learner refines its prediction mechanism 2

3 Online Multiple Kernel Classification (OMKC) Background – Multiple Kernel Composed of two online learning algorithms: Perceptron algorithm (Rosenblatt 1958) Type of linear classifier Learns a classifier for a given kernel Hedge algorithm (Freund and Schapire 1997) Combines classifiers by linear weights Perceptron Hedge 3

4 Online Multiple Kernel Classification (OMKC) Perceptron algorithm 4

5 Online Multiple Kernel Classification (OMKC) Hedge algorithm 5

6 Online Multiple Kernel Classification (OMKC) Notations 6

7 Online Multiple Kernel Classification (OMKC) Proposed framework 7

8 Online Multiple Kernel Classification (OMKC) Algorithms Deterministic approach: all kernels are used Stochastic approach: a subset of kernels are used 8 Deterministic Stochastic Deterministic Stochastic Update Combination

9 Online Multiple Kernel Classification (OMKC) OMKC (D,D) 9 … Kernel classifiers : Prediction: Combined Prediction: … Deterministic update Deterministic combination Deterministic Stochastic Deterministic Stochastic Update Combination

10 Online Multiple Kernel Classification (OMKC) OMKC (S,S) 10 … Kernel classifiers : Prediction: Combined Prediction: … Stochastic update Deterministic Stochastic Deterministic Stochastic Update Combination Stochastic combination

11 Online Multiple Kernel Classification (OMKC) Experimental setup 11 binary datasets

12 Online Multiple Kernel Classification (OMKC) Experimental setup 12

13 Online Multiple Kernel Classification (OMKC) Evaluation of the deterministic OMKC algorithm Comparison of the deterministic OMKC algorithm with three Perceptron based algorithms Perceptron : the well-known Perceptron baseline algorithm with a linear kernel (Rosenblatt 1958; Freund and Schapire 1999) Perceptron(u) : another Perceptron baseline algorithm with an unbiased/uniform combination of all the kernels Perceptron(*): an online validation procedure to search for the best kernel among the pool of kernels (using the first 10 % training examples), and then apply the Perceptron algorithm with the best kernel OM-2: a state-of-the-art online learning algorithm for multiple kernel learning (Jie et al. 2010; Orabona et al. 2010) 13

14 Online Multiple Kernel Classification (OMKC) Evaluation of the deterministic OMKC algorithm 14 < > <

15 Online Multiple Kernel Classification (OMKC) Average mistake rate (20 runs) 15

16 Online Multiple Kernel Classification (OMKC) Number of support vectors (20 runs) 16

17 Online Multiple Kernel Classification (OMKC) Kernel weights 17

18 Online Multiple Kernel Classification (OMKC) 18

19 Online Multiple Kernel Classification (OMKC) Time Efficiency 19 Decreases as size increases

20 Online Multiple Kernel Classification (OMKC) Conclusion All the OMKC algorithms usually perform better than the regular Perceptron algorithm with an unbiased linear combination of multiple kernels the Perceptron algorithm with the best kernel found by validation the state-of-the-art online MKL algorithm The deterministic combination strategy usually performs better Stochastic updating strategy improves computational efficiency without decreasing the accuracy significantly 20

21 21 1)How many kernel classifiers were used in the stochastic combination? 2)How was the number of support vectors determined? Should the support vectors be given in terms of the number of support vectors per kernel classifier? Did support vectors overlap between kernel classifiers?

22 Online Multiple Kernel Classification (OMKC) References Hoi, S. C. H., Jin, R., Zhao, P., & Yang, T. (2012). Online Multiple Kernel Classification. Machine Learning, 90(2), 289–316. doi:10.1007/s10994-012-5319-2 22

23 Online Multiple Kernel Classification (OMKC) Algorithm 1 All kernels are used 23 Deterministic Stochastic Deterministic Stochastic Update Combination Normalize the weights Update Combination

24 Online Multiple Kernel Classification (OMKC) Algorithm 1 → 2 24 Stochastic combination Deterministic update 17: Update Combination Deterministic Stochastic Deterministic Stochastic

25 Online Multiple Kernel Classification (OMKC) Algorithm 2 → 3 25 Deterministic Stochastic Deterministic Stochastic Update Combination

26 Online Multiple Kernel Classification (OMKC) Algorithm 2 → 3 26 Deterministic Stochastic Deterministic Stochastic Deterministic combination Stochastic update Update Combination

27 Online Multiple Kernel Classification (OMKC) Algorithm 4 27 Deterministic Stochastic Deterministic Stochastic Stochastic update Stochastic combination Update Combination

Download ppt "Online Multiple Kernel Classification Steven C.H. Hoi, Rong Jin, Peilin Zhao, Tianbao Yang Machine Learning (2013) Presented by Audrey Cheong Electrical."

Similar presentations

Koby Crammer Department of Electrical Engineering

Koby Crammer Department of Electrical Engineering
Recognizing Human Actions by Attributes CVPR2011 Jingen Liu, Benjamin Kuipers, Silvio Savarese Dept. of Electrical Engineering and Computer Science University.

Recognizing Human Actions by Attributes CVPR2011 Jingen Liu, Benjamin Kuipers, Silvio Savarese Dept. of Electrical Engineering and Computer Science University.
Integrated Instance- and Class- based Generative Modeling for Text Classification Antti PuurulaUniversity of Waikato Sung-Hyon MyaengKAIST 5/12/2013 Australasian.

Integrated Instance- and Class- based Generative Modeling for Text Classification Antti PuurulaUniversity of Waikato Sung-Hyon MyaengKAIST 5/12/2013 Australasian.
On-line learning and Boosting

On-line learning and Boosting
Particle swarm optimization for parameter determination and feature selection of support vector machines Shih-Wei Lin, Kuo-Ching Ying, Shih-Chieh Chen,

Particle swarm optimization for parameter determination and feature selection of support vector machines Shih-Wei Lin, Kuo-Ching Ying, Shih-Chieh Chen,
Learning on Probabilistic Labels Peng Peng, Raymond Chi-wing Wong, Philip S. Yu CSE, HKUST 1.

Learning on Probabilistic Labels Peng Peng, Raymond Chi-wing Wong, Philip S. Yu CSE, HKUST 1.
Robust Moving Object Detection & Categorization using self- improving classifiers Omar Javed, Saad Ali & Mubarak Shah.

Robust Moving Object Detection & Categorization using self- improving classifiers Omar Javed, Saad Ali & Mubarak Shah.
Linear Learning Machines  Simplest case: the decision function is a hyperplane in input space.  The Perceptron Algorithm: Rosenblatt, 1956  An on-line.

Linear Learning Machines  Simplest case: the decision function is a hyperplane in input space.  The Perceptron Algorithm: Rosenblatt, 1956  An on-line.
Sparse vs. Ensemble Approaches to Supervised Learning

Sparse vs. Ensemble Approaches to Supervised Learning
1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.

1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.
2D1431 Machine Learning Boosting.

2D1431 Machine Learning Boosting.
Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficiency with boostrap sampling: Every example.

Boosting Rong Jin. Inefficiency with Bagging D Bagging … D1D1 D2D2 DkDk Boostrap Sampling h1h1 h2h2 hkhk Inefficiency with boostrap sampling: Every example.
Generic Object Detection using Feature Maps Oscar Danielsson Stefan Carlsson

Generic Object Detection using Feature Maps Oscar Danielsson Stefan Carlsson
Announcements  Project proposal is due on 03/11  Three seminars this Friday (EB 3105) Dealing with Indefinite Representations in Pattern Recognition.

Announcements  Project proposal is due on 03/11  Three seminars this Friday (EB 3105) Dealing with Indefinite Representations in Pattern Recognition.
1 Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval 9-April, 2005 Steven C. H. Hoi *, Michael R. Lyu.

1 Integrating User Feedback Log into Relevance Feedback by Coupled SVM for Content-Based Image Retrieval 9-April, 2005 Steven C. H. Hoi *, Michael R. Lyu.
Linear Learning Machines  Simplest case: the decision function is a hyperplane in input space.  The Perceptron Algorithm: Rosenblatt, 1956  An on-line.

Linear Learning Machines  Simplest case: the decision function is a hyperplane in input space.  The Perceptron Algorithm: Rosenblatt, 1956  An on-line.
Sparse vs. Ensemble Approaches to Supervised Learning

Sparse vs. Ensemble Approaches to Supervised Learning
Genetic Algorithm What is a genetic algorithm? “Genetic Algorithms are defined as global optimization procedures that use an analogy of genetic evolution.

Genetic Algorithm What is a genetic algorithm? “Genetic Algorithms are defined as global optimization procedures that use an analogy of genetic evolution.
Text Classification Using Stochastic Keyword Generation Cong Li, Ji-Rong Wen and Hang Li Microsoft Research Asia August 22nd, 2003.

Text Classification Using Stochastic Keyword Generation Cong Li, Ji-Rong Wen and Hang Li Microsoft Research Asia August 22nd, 2003.
Selective Sampling on Probabilistic Labels Peng Peng, Raymond Chi-Wing Wong CSE, HKUST 1.

Selective Sampling on Probabilistic Labels Peng Peng, Raymond Chi-Wing Wong CSE, HKUST 1.

Similar presentations

Ads by Google