1. Toric Modification on Machine Learning Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology

2. Agenda Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary

3. Agenda Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary

4. What is the generalization error? The true model Data Learning model Predictive model i.i.d. MLE MAP Bayes Generalization Error

5. Algebraic geometry connected to learning theory in the Bayes method. The formal definition of the generalization error. The true model DataLearning model Predictive model

6. Algebraic geometry connected to learning theory in the Bayes method. The formal definition of the generalization error. Another Kullback divergence : The true model DataLearning model Predictive model

7. Algebraic geometry connected to learning theory in the Bayes method. The formal definition of the generalization error. Another Kullback divergence : The true model DataLearning model Predictive model Machine Learning / Learning Theory Algebraic Geometry

8. The zeta function has an important role for the connection. Zeta function : h(w) holomorphic function C-infinity func. with compact support h(w) Machine Learning / Learning Theory Algebraic Geometry Analytic func.

9. The zeta function has an important role for the connection. Zeta function : Machine Learning / Learning Theory Algebraic Geometry holomorphic function Prior distribution Kullback divergence

10. The largest pole of the zeta function determines the generalization error. Machine Learning / Learning Theory Algebraic Geometry Asymptotic Bayes generalization error [ Watanabe 2001]

11. Agenda Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary

12. Calculation of the zeta function requires well-formed H(w). : Polynomial form : Factorized form Resolution of singularities

13. The resolution of singularities with blow-ups is an iterative method. Blow-ups

14. The resolution of singularities with blow-ups is an iterative method. Blow-ups Kullback divergence in ML is complicated and has high dimensional w :

15. The bottleneck is the iterative method. Blow-ups Applicable function Computational cost Any function Large

16. Toric modification is a systematic method for the resolution of singularities. Blow-upsToric Modification Applicable function Computational cost Any function Large

17. Agenda Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary

18. Newton diagram is a convex hull in the exponent space Toric Modification

19. The borders determine a set of vectors in the dual space. Toric Modification

20. Add some vectors subdividing the spanned area. Toric Modification

21. Selected vectors construct the resolution map. Toric Modification

22. Non-degenerate Kullback divergence The condition to apply the toric modification to the Kullback divergence For all i : Toric Modification

23. Toric modification is “systematic”. Toric Modification Blow-ups Blow-up The search space will be large. We cannot know how many iterations we need. The number of vectors is limited.

24. Toric modification can be an effective plug-in method. Blow-upsToric Modification Applicable function Computational cost Any function Large Non-degenerate function Limited

25. Agenda Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary

26. An application to a mixture model Mixture of binomial distributions The true model: Learning model:

27. The Newton diagram of the mixture

28. The resolution map based on the toric modification

29. The generalization error of the mixture of binomial distributions : Factorized form The zeta function: : Generalization error : Polynomial form

30. Agenda Learning theory and algebraic geometry Two forms of the Kullback divergence Toric modification Application to a binomial mixture model Summary

31. The Bayesian generalization error is derived on the basis of the zeta function. Calculation of the coefficients requires the factorized form of the Kullback divergence. Toric modification is an effective method to find the factorized form. The error of a binomial mixture is derived as the application.

32. Thank you !!! Toric Modification on Machine Learning Keisuke Yamazaki & Sumio Watanabe Tokyo Institute of Technology

34. Asymptotic form of G.E. has been derived in regular models. Essential assumption: Fisher information matrix