1. Protein Fold Recognition with Relevance Vector Machines Patrick Fernie COMS 6772 Advanced Machine Learning 12/05/2005

2. Relevance Vector Machine A Bayesian treatment of a generalized linear model Yields a formulation similar to that of a Support Vector Machine Hyperparameters Instead of Margin/Costs

3. Relevance Vector Machine SVMRVM Hard Binary Outputs or Point Estimates Probabilistic Outputs Requires a Mercer Kernel Can Use Arbitrary Kernel Must Determine Suitable Cost and Insensitivity Values “Nuisance” Values Automatically Determined Sparse (USPS ~2500) Sparser USPS (~316!)

4. Relevance Vector Machine Can’t Use qp() Must solve iteratively (Sequential Minimization Optimization) As we iterate, many hyperparameters (α i ) values become arbitrarily large; allows pruning.

5. Relevance Vector Machine Faster Algorithm (Still not SVM fast) Minimizes Number of Active Kernel Functions to Reduce Computation Time Analytic Approach to Pruning/Adding Basis Functions

6. Protein Fold Recognition Protein Structure Families Many Fold Families Not Necessarily Directly Related by Protein Sequence

7. Protein Fold Recognition Prime Situation for Machine Learning Techniques! NN, SVM, etc. Large Number of Classes

8. Protein Fold Recognition 27 Fold Families Train Many 2-Class Classifiers One vs. Others – False Positives One vs. Others – False Positives Unique One vs. Others – Like One vs. Others, with Another Round of Training Unique One vs. Others – Like One vs. Others, with Another Round of Training All vs. All – Requires a Lot of Classifiers! All vs. All – Requires a Lot of Classifiers!

9. RVMs & Protein Folds Why RVMs? Probabilistic Outputs Probabilistic Outputs Sparsity (useful only in assessment) Sparsity (useful only in assessment) True Multiclass Prediction True Multiclass Prediction No Need to Find “Nuisance” Parameters No Need to Find “Nuisance” Parameters

10. Issues/Future Work Optimize RVM Classification Implement True Multiclass Reduced Greediness and Sequential Convergence Optimization Novel Kernels?

11. References M. Tipping, “The Relevance Vector Machine”, http://www.relevancevector.com M. Tipping, “Sparse Bayesian Learning and the Relevance Vector Machine”, JMLR, 2001 1:211- 244. M. Tipping and A. Faul, “Fast Marginal Likelihood Maximisation for Sparse Bayesian Models”, http://www.relevancevector.com C. Ding and I. Dubchak, “Multi-class Protein Fold Recognition Using Support Vector Machines”, http://www.kernel-machines.org