1. RIKEN Brain Science Institute
MaxEnt 07’
Information Geometry of
MaxEnt Principle
Shun-ichi Amari
RIKEN Brain Science Institute

2. Dual Affine Connections
Information Geometry
Systems Theory
Information Theory
Statistics
Neural Networks
Combinatorics
Physics
Information Sciences
Math. AI
Riemannian Manifold
Dual Affine Connections
Manifold of Probability Distributions

3. Information Geometry ?
Riemannian metric
Dual affine connections

4. Manifold of Probability Distributions

5. Manifold of Probability Distributions

6. Invariance 1. Invariant under reparameterization
2. Invariant under different representation

7. Riemannian metric—Fisher information
Two Structures
Riemannian metric—Fisher information
Affine connection
-- geodesic, straight line
how curved is the manifold?

8. Riemannian Structure

9. Kullback-Leibler Divergence
quasi-distance

10. KL-divergence and Riemannian Structure
relation
Fisher information matrix

11. Affine Connection
covariant derivative
straight line

12. Renyi-Tasallis
Exponential connection
Entropy
KL-divergence
Mixture connection
Levi-Civita (Riemannian)

13. Affine Connections
e-geodesic
m-geodesic

14. Duality Y X Y X
Riemannian geometry:

15. Independent Distributions

16. Dually flat manifold
S = {p(x), x discrete}

17. Dually Flat Manifold 1. Potential Functions 2. Divergence
---convex (Legendre transformation)
2. Divergence
KL-divergence
3. Pythagoras Theorem
4. Projection Theorem

18. Projection Theorem
m-geodesic
e-geodesic

19. Applications to Statistics
curved exponential family:
: estimation
: testing

20. High-Order Asymptotics
:Cramér-Rao

21. Other Applications Systems theory Information theory Neuromanifold
Belief propagation
Boosting (Murata-Eguchi)
Higher-order correlations
Mathematics --- Orlicz space (Pistone, Gracceli)
Physics ---
Amari-Nagaoka, Methods of Information Geometry, AMS & Oxford U., 2000
Amari, Differential-Geometrical Methods of Statistics, Springer, 1985
Kass and Vos, Geomtrical Foundations of Asymptotic Inference, Wiley, 1997
Murrey and Rice, Differential Geometry and Statistics, Chapman, 1993

22. Exponential Family : dually flat
Two coordinate systems

23. Exponential Family example (1) : discrete distributions
Negative entropy

24. example (2) : Gaussian distributions
example (3) : AR model

25. Legendre transformation

26. Divergence
Pythagorean Theorem
m-flat
e-flat

27. Divergence and Entropy
equi-divergence: equi-entropy

28. Dual Foliation
Pythagorean theorem

29. Maximum Entropy

30. Simple Example : independence

31. Simple example : Gaussian

32. Time Series

33. Geometry
Potentials

34. Stochastic Realization

35. Dual Problem

36. Rényi-Tsallis entropy
Manifold of positive measures m(x)

37. Entropy (alpha-entropy) is a fundamental quantity It is given rise to from a fundamental geometrical structure. KL-divergence is derived therefrom.