Presentation is loading. Please wait.

Presentation is loading. Please wait.

Information Geometry and Neural Netowrks

Published byLewis Reynolds Modified over 6 years ago

Similar presentations

Presentation on theme: "Information Geometry and Neural Netowrks"— Presentation transcript:

1 Information Geometry and Neural Netowrks
Shun-ichi Amari RIKEN Brain Science Institute Orthogonal decomposition of rates and (higher-order) correlations Synchronous firing and higher correlations Algebraic singularities caused by multiple stimuli Dynamics of learning in multiplayer perceptrons

2 Dual Affine Connections
Information Geometry Systems Theory Information Theory Statistics Neural Networks Combinatorics Physics Information Sciences Math. AI Riemannian Manifold Dual Affine Connections Manifold of Probability Distributions

3 Information Geometry ? Riemannian metric Dual affine connections

4 Manifold of Probability Distributions

5 Two Structures Riemannian metric and affine connection
Fisher information

6 Riemannian Structure

7 Affine Connection covariant derivative straight line

8 Independent Distributions

9 Neural Firing ----firing rate ----covariance higher-order correlations
orthogonal decomposition

10 of Higher-Order Correlations
Information Geometry of Higher-Order Correlations ----orthogonal decomposition Riemannian metric dual affine connections Pythagoras theorem Dual geodesics

11 Correlations of Neural Firing
firing rates correlations orthogonal coordinates

12 　　　００１１０００１０１１０１ ０１００１００１１０１００ ０１０１１０１００１０１０ firing rates: correlation—covariance?

13 Independent Distributions

14 Pythagoras Theorem p q r D[p:r] = D[p:q]+D[q:r] p,q: same marginals
correlations D[p:r] = D[p:q]+D[q:r] q r p,q: same marginals r,q: same correlations independent estimation correlation testing invariant under firing rates

15 No pairwise correlations,
Triplewise correlation …… …… ……

16 Pythagoras Decomposition of KL Divergence
only pairwise independent

17 Higher-Order Correlations

18 Synfiring and Higher-Order Correlations
Amari, Nakahara, Wu, Sakai

19 Population and Synfire
Neurons

20 Population and Synfire

21

22

23

24 Synfiring

25 r Bifurcation : independent---single delta peak pairwise correlated
higher-order correlation！ r

26 RIKEN Brain Science Institute
Field Theory of Population Coding Shun-ichi Amari RIKEN Brain Science Institute Collaborators: Si Wu 　Hiro Nakahara

27 Population Coding and Neural Field

28 Population Encoding x f (z-x) r(z) z

29 Noise b

30 Probability Model

31 Fisher information Cramer-Rao

32 Fourier Analysis

33 Fisher Information

34

35 Dynamics of Neural Fields
Shaping Detecting Decoding

36 How the Brain Solves Singularity in Population Coding
S. Amari and H. Nakahara RIKEN Brain Science Institute

37

38 Neural Activity

39 Parameter Space

40

41

42

43

44 synfiring resolves singularity

45

46 synfiring mechanism common multiplicative noise

47 S.Amari and H.Nagaoka, Methods of Information Geometry AMS &Oxford Univ Press, 2000

48 Mathematical Neurons

49 Multilayer Perceptrons
ｙ

50 Multilayer Perceptron
neuromanifold space of functions

51 Neuromanifold Metrical structure Topological structure

52 Riemannian manifold

53 Geometry of singular model
Ｗ

54 Gaussian mixture

55 Topological Singularities

56 singularities

57 Singularity of MLP---example

58 Backpropagation ---gradient learning

59 Information Geometry of MLP
Natural Gradient Learning : S. Amari ; H.Y. Park

60 ｙ

61

62 2 hidden-units

63 Dynamics of Learning

64 The teacher is on singularity

65 The teacher is on singularity

66

67

68

69

Download ppt "Information Geometry and Neural Netowrks"

Similar presentations

What is the neural code? Puchalla et al., 2003. What is the neural code? Encoding: how does a stimulus cause the pattern of responses? what are the responses.

What is the neural code? Puchalla et al., What is the neural code? Encoding: how does a stimulus cause the pattern of responses? what are the responses.
SHAPE THEORY USING GEOMETRY OF QUOTIENT SPACES: STORY STORY SHAPE THEORY USING GEOMETRY OF QUOTIENT SPACES: STORY STORY ANUJ SRIVASTAVA Dept of Statistics.

SHAPE THEORY USING GEOMETRY OF QUOTIENT SPACES: STORY STORY SHAPE THEORY USING GEOMETRY OF QUOTIENT SPACES: STORY STORY ANUJ SRIVASTAVA Dept of Statistics.
Random Variables ECE460 Spring, 2012.

Random Variables ECE460 Spring, 2012.
1 Statistical Mechanics of Online Learning for Ensemble Teachers Seiji Miyoshi Masato Okada Kobe City College of Tech. Univ. of Tokyo, RIKEN BSI.

1 Statistical Mechanics of Online Learning for Ensemble Teachers Seiji Miyoshi Masato Okada Kobe City College of Tech. Univ. of Tokyo, RIKEN BSI.
Artificial Spiking Neural Networks

Artificial Spiking Neural Networks
Reading population codes: a neural implementation of ideal observers Sophie Deneve, Peter Latham, and Alexandre Pouget.

Reading population codes: a neural implementation of ideal observers Sophie Deneve, Peter Latham, and Alexandre Pouget.
WAGOS Conformal Changes of Divergence and Information Geometry Shun-ichi Amari RIKEN Brain Science Institute.

WAGOS Conformal Changes of Divergence and Information Geometry Shun-ichi Amari RIKEN Brain Science Institute.
Support Vector Machines (and Kernel Methods in general)

Support Vector Machines (and Kernel Methods in general)
Supervised and Unsupervised learning and application to Neuroscience Cours CA6b-4.

Supervised and Unsupervised learning and application to Neuroscience Cours CA6b-4.
Optimizing number of hidden neurons in neural networks

Optimizing number of hidden neurons in neural networks
Artificial Intelligence Statistical learning methods Chapter 20, AIMA (only ANNs & SVMs)

Artificial Intelligence Statistical learning methods Chapter 20, AIMA (only ANNs & SVMs)
Spike-triggering stimulus features stimulus X(t) multidimensional decision function spike output Y(t) x1x1 x2x2 x3x3 f1f1 f2f2 f3f3 Functional models of.

Spike-triggering stimulus features stimulus X(t) multidimensional decision function spike output Y(t) x1x1 x2x2 x3x3 f1f1 f2f2 f3f3 Functional models of.
November 2, 2010Neural Networks Lecture 14: Radial Basis Functions 1 Cascade Correlation Weights to each new hidden node are trained to maximize the covariance.

November 2, 2010Neural Networks Lecture 14: Radial Basis Functions 1 Cascade Correlation Weights to each new hidden node are trained to maximize the covariance.
Spike Train decoding Summary Decoding of stimulus from response –Two choice case Discrimination ROC curves –Population decoding MAP and ML estimators.

Spike Train decoding Summary Decoding of stimulus from response –Two choice case Discrimination ROC curves –Population decoding MAP and ML estimators.
Supervised Learning Networks. Linear perceptron networks Multi-layer perceptrons Mixture of experts Decision-based neural networks Hierarchical neural.

Supervised Learning Networks. Linear perceptron networks Multi-layer perceptrons Mixture of experts Decision-based neural networks Hierarchical neural.
MACHINE LEARNING 12. Multilayer Perceptrons. Neural Networks Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

MACHINE LEARNING 12. Multilayer Perceptrons. Neural Networks Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)
CSC321: Introduction to Neural Networks and Machine Learning Lecture 20 Learning features one layer at a time Geoffrey Hinton.

CSC321: Introduction to Neural Networks and Machine Learning Lecture 20 Learning features one layer at a time Geoffrey Hinton.
Introduction to Artificial Neural Network and Fuzzy Systems

Introduction to Artificial Neural Network and Fuzzy Systems
-Gaurav Mishra -Pulkit Agrawal. How do neurons work Stimuli  Neurons respond (Excite/Inhibit) ‘Electrical Signals’ called Spikes Spikes encode information.

-Gaurav Mishra -Pulkit Agrawal. How do neurons work Stimuli  Neurons respond (Excite/Inhibit) ‘Electrical Signals’ called Spikes Spikes encode information.
GEO7600 Inverse Theory 09 Sep 2008 Inverse Theory: Goals are to (1) Solve for parameters from observational data; (2) Know something about the range of.

GEO7600 Inverse Theory 09 Sep 2008 Inverse Theory: Goals are to (1) Solve for parameters from observational data; (2) Know something about the range of.

Similar presentations

Ads by Google