1. 1 PEGASOS Primal Efficient sub-GrAdient SOlver for SVM Shai Shalev-Shwartz Yoram Singer Nati Srebro The Hebrew University Jerusalem, Israel YASSO = Yet Another Svm SOlver

2. 2 Support Vector Machines QP form: More “natural” form: Empirical loss Regularization term

3. 3 Outline Previous Work The Pegasos algorithm Analysis – faster convergence rates Experiments – outperforms state-of-the-art Extensions kernels complex prediction problems bias term

4. 4 Previous Work Dual-based methods Interior Point methods Memory: m 2, time: m 3 log(log(1/  )) Decomposition methods Memory: m, Time: super-linear in m Online learning & Stochastic Gradient Memory: O(1), Time: 1/  2 (linear kernel) Memory: 1/  2, Time: 1/  4 (non-linear kernel) Typically, online learning algorithms do not converge to the optimal solution of SVM Better rates for finite dimensional instances (Murata, Bottou)

5. 5 PEGASOS Subgradient Projection A_t = S Subgradient method |A_t| = 1 Stochastic gradient

6. 6 Run-Time of Pegasos Choosing |A t |=1 and a linear kernel over R n  Run-time required for Pegasos to find  accurate solution w.p. ¸ 1-  Run-time does not depend on #examples Depends on “difficulty” of problem ( and  )

7. 7 Formal Properties Definition: w is  accurate if Theorem 1: Pegasos finds  accurate solution w.p. ¸ 1-  after at most iterations. Theorem 2: Pegasos finds log(1/  ) solutions s.t. w.p. ¸ 1- , at least one of them is  accurate after iterations

8. 8 Proof Sketch A second look on the update step:

9. 9 Proof Sketch Lemma (free projection): Logarithmic Regret for OCP (Hazan et al’06) Take expectation: f(w r )-f(w * ) ¸ 0  Markov gives that w.p. ¸ 1-  Amplify the confidence

10. 10 Experiments 3 datasets (provided by Joachims) Reuters CCAT (800K examples, 47k features) Physics ArXiv (62k examples, 100k features) Covertype (581k examples, 54 features) 4 competing algorithms SVM-light (Joachims) SVM-Perf (Joachims’06) Norma (Kivinen, Smola, Williamson ’02) Zhang’04 (stochastic gradient descent) Source-Code available online

11. 11 Training Time (in seconds) PegasosSVM-PerfSVM-Light Reuters27720,075 Covertype68525,514 Astro- Physics 2580

12. 12 Compare to Norma (on Physics) obj. value test error

13. 13 Compare to Zhang (on Physics) Objective But, tuning the parameter is more expensive than learning …

14. 14 Effect of k=|A t | when T is fixed Objective

15. 15 Effect of k=|A t | when kT is fixed Objective

16. 16 I want my kernels ! Pegasos can seamlessly be adapted to employ non-linear kernels while working solely on the primal objective function No need to switch to the dual problem Number of support vectors is bounded by

17. 17 Complex Decision Problems Pegasos works whenever we know how to calculate subgradients of loss func. (w;(x,y)) Example: Structured output prediction Subgradient is  (x,y’)-  (x,y) where y’ is the maximizer in the definition of

18. 18 bias term Popular approach: increase dimension of x Cons: “pay” for b in the regularization term Calculate subgradients w.r.t. w and w.r.t b: Cons: convergence rate is 1/  2 Define: Cons: |A t | need to be large Search b in an outer loop Cons: evaluating objective is 1/  2

19. 19 Discussion Pegasos: Simple & Efficient solver for SVM Sample vs. computational complexity Sample complexity: How many examples do we need as a function of VC-dim ( ), accuracy (  ), and confidence (  ) In Pegasos, we aim at analyzing computational complexity based on, , and  (also in Bottou & Bousquet) Finding argmin vs. calculating min: It seems that Pegasos finds the argmin more easily than it requires to calculate the min value