1. Trading Convexity for Scalability Marco A. Alvarez CS7680 Department of Computer Science Utah State University

2. Paper Collobert, R., Sinz, F., Weston, J., and Bottou, L. 2006. Trading convexity for scalability. In Proceedings of the 23rd International Conference on Machine Learning (Pittsburgh, Pennsylvania, June 25 - 29, 2006). ICML '06, vol. 148. ACM Press, New York, NY, 201-208.

3. Introduction Previously in Machine Learning  Non-convex cost function in MLP Difficult to optimize Work efficiently  SVM are defined by a convex function Easier optimization (algorithms) Unique solution (we can write theorems) Goal of the paper  Sometimes non-convexity has benefits Faster == training and testing (less support vectors)  Non-convex SVMs (faster and sparser)  Fast transductive SVMs

4. From SVM Decision function Primal formulation  Minimize ||w|| so that margin is maximized  w is a combination of a small number of data (sparsity)  Decision boundary is determined by the support vectors Dual formulation s.t.

5. SVM problem Number of support vectors increases linearly with L Cost attributed to one example (x,y): From:

6. Ramp Loss Function Given: Outliers Non SV

7. Concave-Convex Procedure (CCCP) Given a cost function: Decompose into a convex part and a concave part Is guaranteed to decrease at each iteration

8. Using the Ramp Loss

9. CCCP for Ramp Loss

10. Results

11. Speedup

12. Time and Number of SVs

13. Transductive SVMs

14. Loss Function Cost to be minimized:

15. Balancing Constraint Necessary for TSVMs

16. Results

17. Training Time

18. Quadratic Fit