A Statistical Mechanical Analysis of Online Learning: Can Student be more Clever than Teacher ? Seiji MIYOSHI Kobe City College of Technology

2 Background (1) Batch Learning –Examples are used repeatedly –Correct answers for all examples –Long time –Large memory Online Learning –Examples used once are discarded –Cannot give correct answers for all examples –Large memory isn't necessary –Time variant teacher

3 Background (2) TeacherStudent

4 Simple Perceptron Output Inputs Connection weights +1

5 Background (2) TeacherStudent Learnable Case

6 Background (3) Teacher Student Unlearnable Case （ Inoue & Nishimori, Phys. Rev. E, 1997) （ Inoue, Nishimori & Kabashima, TANC-97, cond-mat/ , 1997)

7 Background (4) Hebbian Learning Perceptron Learning

8 Model (1) Moving Teacher Student True Teacher A

9 Model (2) Length of Student Length of Moving Teacher A B J

10 Model (3) A B J

11 Output Inputs Connection weights Simple Perceptron Linear Perceptron

12 Model (3) Linear Perceptrons with Noises A B J

13 f g Model (4) Squared Errors Gradient Method A B J

14 ErrorGaussian Generalization Error A B J

15 Differential equations for order parameters

16 f g Model (4) Squared Errors Gradient Method A B J

17 B m+1 = B m + g m x m + Nr B m+1 = Nr B m + g m y m Ndt Nr B m+2 = Nr B m+1 + g m+1 y m+1 Nr B m+Ndt = Nr B m+Ndt-1 + g m+Ndt-1 y m+Ndt-1 Nr B m+Ndt = Nr B m + Ndt N(r B +dr B ) = Nr B + Ndt dr B / dt =

18 Differential equations for order parameters

19 Sample Averages

20 Differential equations for order parameters

21 Analytical Solutions of Order Parameters

22 Differential equations for order parameters

23 ErrorGaussian Generalization Error A B J

24 Dynamical Behaviors of Generalization Errors η J ＝ 1.2 η J ＝ 0.3

25 Dynamical Behaviors of R and l η J ＝ 1.2η J ＝ 0.3

26 Analytical Solutions of Order Parameters

27 Steady State

28 ηJηJ 20

29 Conclusions Generalization errors of a model composed of a true teacher, a moving teacher, and a student that are all linear perceptrons with noises have been obtained analytically using statistical mechanics. Generalization errors of a student can be smaller than that of a moving teacher, even if the student only uses examples from the moving teacher.